ENH: implement editable wheels without having to run meson install

Exploring the idea of using a custom loader to avoid having to run meson install. This works for Python modules. Supporting data files is possible but requires some more work.

Working on this one question arises: what should we do with the Python bytecode cache files? Should we allow them and store them next to the source files, either in the source or build directory, or should we disallow them? Having them in the build directory does not bother me, but I'm not sure we should have them stored in the source directory. Implementing a loader that diverts the bytecode cache in a place disjoint from the source file is also an option, but seems a lot of work.

Exploring the idea of using a custom loader to avoid having to run meson install. This works for Python modules. Supporting data files is possible but requires some more work.

Is there a reason we're avoiding meson install here. Seems a bit like we're re-inventing the wheel.

Working on this one question arises: what should we do with the Python bytecode cache files? Should we allow them and store them next to the source files, either in the source or build directory, or should we disallow them? Having them in the build directory does not bother me, but I'm not sure we should have them stored in the source directory. Implementing a loader that diverts the bytecode cache in a place disjoint from the source file is also an option, but seems a lot of work.

I would prefer excluding them. They have caused trouble in the past in causing a bad magic number ImportError.

Is there a reason we're avoiding meson install here?

I kind of have the same question, but from a slightly different perspective. The two things that are known to be problematic right now are running tests and setting breakpoints directly in an IDE. For both, my impression was that installing symlinks and pointing the IDE at the install dir is the one thing that will help. This rewrite is orthogonal to that, right? I don't have a good feel for these custom loaders, but they can't fix the IDE issue was my impression.

We avoid meson install because it creates more problems that it solves. There is nothing to reinvent here: the code that matches files from the source directory and the build directory to the module structure is trivial. Installing symlinks is not an option because it is not supported by the wheel format. What we can install is a module finder that invokes a symlink factory, but that has more complexity of the approach in this PR and most of the ones of the current install based approach. Note that the approach in this PR does not use a custom module loader, but a custom module finder, which then uses the regular Python module loader to load the module content.
I don't know how IDEs set breakpoints in source code, but assuming they work similarly as how plain pdb works, the symlink solution does not solve the problem with breakpoints, unless the IDE loads the file via the symlink name. The reason is that for setting a breakpoint on a file location, the file path needs to be relative to the Python path.

It would be very useful to have some input from IDE developers about what's their idea about supporting setting breakpoints in the source code of editable wheels.

Installing symlinks is not an option because it is not supported by the wheel format

That's not relevant here - the idea was to install symlinks directly; the wheel only contains the loader.

unless the IDE loads the file via the symlink name.

Indeed, that's the one thing that will work - edit the installed directory in the IDE (which then also edits the original sources, because symlink). Still not great, but should work.

It would be very useful to have some input from IDE developers about what's their idea about supporting setting breakpoints in the source code of editable wheels.

I'm not sure how to get that. IDE developers probably won't care. This is only a problem for editable wheels + out-of-place builds, which is rare.

We avoid meson install because it creates more problems that it solves.

Can you elaborate more on this?

We avoid meson install because it creates more problems that it solves.

Can you elaborate more on this?

Well, the biggest problem I see is that it breaks the fundamental promise of editable wheels, namely that the code that is execute is the code that you are editing inside your project. With editable installations based on an installed tree, the code being executed is a copy of the code you are editing. One major consequence is that if you step through the execution in a debugger in your IDE the source code you see is not the source code in your project. If you quit the debugger and modify the code that you are staring at it in the IDE, your edits will be lost the next time you execute your code.
Another class of problems is keeping the installed tree and the source tree synchronized. Either you wipe the installed tree at every rebuild as you propose in #282 or you need some more complex form of file tracking. For small projects this is maybe not an issue, but for things like SciPy is a major slowdown: a Python code only change takes a few hundred of milliseconds to be rebuild (just the time for ninja to recognize that there is nothing to do) but installing takes several seconds.
Finally, the current code that uses the installed tree for the editable install is much more complex than the code that does not use it and does not suffer form the two class of problems above.
If you can live with the two class of problems above, you maybe don't even need editable wheels: simply integrate building a regular wheel and installing it into your test runner. Quickly clobbered together and untested make solution:
PYTHON ?= python
test:
    $(PYTHON) -m build --no-isolation --wheel -Cbuilddir=build
    $(PYTHON) -m pip install dist/*.whl
    $(PYTHON) -m pytest tests/

That's not relevant here - the idea was to install symlinks directly; the wheel only contains the loader.

This runs into the problem of keeping the symlinks synchronized. Either you walk the tree and synchronize the symlinks with the installation introspection data or you wipe them all and recreate them. Both approaches are deadly slow. Recreating them also most likely ends up in a whole lot of fun problems on Windows if an editor as the file open via the symlink name.  I'm also not sure how a whole lot IDEs would react at the now not so remote possibility of having the same file open via two different paths.
Also, does Windows support symlinks in all configurations? I'm pretty sure my Windows systems at work are locked down to do not allow symlinks.

It would be very useful to have some input from IDE developers about what's their idea about supporting setting breakpoints in the source code of editable wheels.

I'm not sure how to get that. IDE developers probably won't care. This is only a problem for editable wheels + out-of-place builds, which is rare.

Then I'm not sure we should put a lot of thought and effort to support this use case then.
Please note that setuptools also lists what I'm proposing in this PR as one way they can use to implement editable installations, thus I don't think it is an esoteric approach.

Please note that setuptools also lists what I'm proposing in this PR as one way they can use to implement editable installations, thus I don't think it is an esoteric approach.

Oh, I believe you and I wasn't implying that it was. I just don't understand how this works from reading the diff. Which is fine - I'm happy to not be too involved in editable installs. I know editable installs are important for adoption, but in SciPy/NumPy we have a dev.py CLI interface which is nicer to use, so we don't have an urgent need.

This seems reasonable, but it has a very big drawback, it'll break code that relies on the module layout, without providing any way to fix it.
My initial idea was to do basically this, but via the Traversable interface. While it would still break code that uses __file__, we could point people to the importlib.resources API, which would still work as expected. The behavior with __file__ would actually be exactly what we wanted, as that's how you fix the IDEs issue -- I don't think there's probably any way around it -- it'd just be unfortunate that it trips code relying on __file__, but I guess that's exactly the problem with relying on __file__.

Supporting data files is possible but requires some more work.

How? I don't see how it can be done without going back to our current approach of just installing them :sweat_smile:
I mean, you'd be installing less stuff

This runs into the problem of keeping the symlinks synchronized. Either you walk the tree and synchronize the symlinks with the installation introspection data or you wipe them all and recreate them. Both approaches are deadly slow.

Why? You can just save the previous intro-installed.json, diff it against the new one, and then only update exactly what changed.

Recreating them also most likely ends up in a whole lot of fun problems on Windows if an editor as the file open via the symlink name.

I'd say opening a file for edition and triggering a rebuild would be out of scope, it's a very niche case, but we can warn about it in the docs. We already do not support multiple interpreters trying to load the project at the same time.
Anyway, from my experience open files on Windows actually raise a permission error when trying to perform actions like this, so while that might not be the case here -- I haven't tried it -- it'd be my bet :rofl:

Please note that setuptools also lists what I'm proposing in this PR as one way they can use to implement editable installations, thus I don't think it is an esoteric approach.

They have control of the layout though, which we do not, hence the use of meson install. I think the scenario is slightly different.

I just don't understand how this works from reading the diff

It "finds" the modules in place at their original locations (source and/or build directories), instead of installing the project somewhere and letting the normal finders find them there.

Supporting data files is possible but requires some more work.

How?

Using a module loader that has a files() method that returns a Traversable object that fakes a filesystem layout using install location information. Progress on this is slower than I would like because the RsourceLoader API is rather convoluted and the three layers of backward compatibility API adaptation it is built upon are not really easy to see through.
Also filesystem access based on the path returned by the __file__ module attribute works in the (I think most common case) in which the module and the file being accessed are both non generated at build time and present in the right relative location in the source directory. Namely, if I have a package with this source structure:
example/
  data.py
  data.txt

and in data.py I have
import pathlib
with open(pathlib.Path(__file__).parent / 'data.txt', 'r') as f:
   data = f.read()

everything works. Because __file__ for the example.data module points to the file in the source directory. However, it breaks if data.txt is generated at build time because it would not exist in the source directory.

I'd say opening a file for edition and triggering a rebuild would be out of scope, it's a very niche case,

IIUC what @rgommers and @lithomas1 are saying, that is precisely what they expect people to do to use their fancy IDE debuggers on the installed editable wheel. With a symlink approach, at least, the executed file and the source file are the same file, but with the installed tree approach this is not the case. In my opinion this is a recipe for a lot of confusion and frustrated users. With the installed tree approach, users would need to have two copies of the same file open in their IDE: one from the installed tree for using in the debugger and another where to edit. Fun!

Why? You can just save the previous intro-installed.json, diff it against the new one, and then only update exactly what changed.

That's right. However, in case of not using symlinks you also need to compare file content for all the installed files.  Symlinks are not supported on all platforms.

It "finds" the modules in place at their original locations (source and/or build directories), instead of installing the project somewhere and letting the normal finders find them there.

If that indeed works, that is conceptually the best thing to do I'd say. It hurts my brain a little to think through how that works and what the corner cases are, but I'd love to test this out for NumPy and SciPy (lots of data files and __file__ usage though - can get rid of the latter if really needed, not of the former).

It installs a "meta path finder" in sys.meta_path https://docs.python.org/3/library/sys.html#sys.meta_path via a .pth file installed as part of the editable wheel. AFAIK, there are no strange corner cases regarding loading modules. importlib.resources support is on the way. Files lookup via the __file__ attribute has the constraints explained above.
I would encourage you to test this. AFAICT, it works, except for support for importlib.resources, but if your package does not use it you cannot notice it.

Well, the biggest problem I see is that it breaks the fundamental promise of editable wheels, namely that the code that is execute is the code that you are editing inside your project. With editable installations based on an installed tree, the code being executed is a copy of the code you are editing. One major consequence is that if you step through the execution in a debugger in your IDE the source code you see is not the source code in your project. If you quit the debugger and modify the code that you are staring at it in the IDE, your edits will be lost the next time you execute your code.
Another class of problems is keeping the installed tree and the source tree synchronized. Either you wipe the installed tree at every rebuild as you propose in #282 or you need some more complex form of file tracking. For small projects this is maybe not an issue, but for things like SciPy is a major slowdown: a Python code only change takes a few hundred of milliseconds to be rebuild (just the time for ninja to recognize that there is nothing to do) but installing takes several seconds.

I can live with 1 and accept it as a limitation of edtiable wheels. It's not great, but it's work-aroundable.
2 is the big problem, since it leads to completely incorrect results, which is why I opened #282 in the first place.

BTW, I'm happy to test this, but I would like to see some tests first as an indication that this is ready, and that the approach isn't going to change anymore.
You're free to pull my test from #282.
I'm also curious to see how this will interact with install scripts (https://mesonbuild.com/Installing.html#custom-install-script).

I would like to see some tests first as an indication that this is ready

There are already tests for editable wheels in the meson-python test suites. The CI passes so this approach works enough to satisfy the tests. I'm working on adding more tests. I'll makes sure to cover also the cases covered in #282

Using a module loader that has a files() method that returns a Traversable object that fakes a filesystem layout using install location information. Progress on this is slower than I would like because the RsourceLoader API is rather convoluted and the three layers of backward compatibility API adaptation it is built upon are not really easy to see through.

Ah, we are on the same page then! This was the plan, the current solution was mean to be semi-temporary, I got confused because the titled says "alternative", but I now get what you mean.
I have a WIP implementation locally. I'll try to finish it and add it to your code after I get the higher priority stuff sorted out.

However, in case of not using symlinks you also need to compare file content for all the installed files.

Checking the size plus a efficient binary comparison algorithm would be enough to handle this. But I think the only OS where symlinks are problematic is Windows, and IMO it'd be super reasonable to require users to have symlinks enabled for editable installs.

I have a WIP implementation locally. I'll try to finish it and add it to your code after I get the higher priority stuff sorted out.

If you drop your implementation into a scratch git branch somewhere I can try to take it from there.

importlib.resources is really not up to the quality of the rest of the standard library, and it seems to gt worst with newer releases, but not without way to many WTF moments, I got the implementation of the editable wheels support for importlib.resources working, at least in simple tests. Supporting (at least in the sense that things do not break when importlib.resources is not used) Python releases 3.8 and earlier may require some not so elegant tricks.

I would like to write some more tests for this, but the implementation is now ready for review. PTAL.

I think this is finished now. I'm happy of how the code looks like and the test coverage should be good. @rgommers @lithomas1 It would be great if it could get some real life testing.

I think this is finished now. I'm happy of how the code looks like and the test coverage should be good. @rgommers @lithomas1 It would be great if it could get some real life testing.

Awesome. I'm on holiday for another week, but I asked the SciPy maintainers who use editable installs to test this branch.

I think this is finished now. I'm happy of how the code looks like and the test coverage should be good. @rgommers @lithomas1 It would be great if it could get some real life testing.

Thanks for pushing this through. I'll be able to test this over the weekend.

This doesn't work for me.
I see, currrently
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/thomasli/pandas/pandas/__init__.py", line 22, in <module>
    from pandas.compat import is_numpy_dev as _is_numpy_dev  # pyright: ignore # noqa:F401
ModuleNotFoundError: No module named 'pandas.compat'

Printing out sys.meta_path I get
[<_pandas_editable_loader.MesonpyMetaFinder object at 0x7fe99462d6a0>, <_distutils_hack.DistutilsMetaFinder object at 0x7fe994285610>, <class '_frozen_importlib.BuiltinImporter'>, <class '_frozen_importlib.FrozenImporter'>, <class '_frozen_importlib_external.PathFinder'>]

so the loader is getting loaded correctly.
Dug in a little more, and
>>> sys.meta_path[0].find_spec("pandas") # works
ModuleSpec(name='pandas', loader=<_pandas_editable_loader.SourceFileLoader object at 0x7fe9942afcd0>, origin='/Users/thomasli/pandas/pandas/__init__.py', submodule_search_locations=[])
>>> sys.meta_path[0].find_spec("pandas.compat") # doesn't work

So it looks like support for dotted imports is broken?

Interesting. I'm pretty sure I tested this and it was working. I'll investigate.

Indeed it works for me. There may be something specific to the module layout or something. Where can I find the meson build definitions for pandas?

I've tried using the build definition from here https://github.com/pandas-dev/pandas/pull/49115 but Meson does not like this
inc_np = include_directories(incdir_numpy)

../../../pandas/meson.build:11:0: ERROR: Tried to form an absolute path to a dir in the source tree.

Hm, it works for me. Maybe numpy isn't installed properly on your machine? (The include directory is calculated from python -c 'import os; os.chdir(".."); import numpy; print(numpy.get_include())'
(You also have to be in the base pandas dir for this to work I think).
Maybe it's easier to reproduce this without pandas? The directory that is giving the import error is installed with install_subdir.
Here is the relevant part of intro-install_plan.json
    "/Users/thomasli/pandas/pandas/compat": {
      "destination": "{py_platlib}/pandas/compat",
      "tag": null
    },

I have to change it to explicitly return a relative path:
incdir_numpy = run_command(py,
  [
    '-c',
    'import os; import numpy; print(os.path.relpath(numpy.get_include()))'
  ],
  check: true
).stdout().strip()

then it works.

I see why it does not work. Fixing it for pandas.compat is easy, but there are instances like pandas._libs where install_subdir() and install paths contribute to place files in the same destination folder. I didn't think about this use case. Fixing those case requires some work and some thinking. I'll have a look tomorrow.

@eli-schwartz Is there a way to know from Meson introspection data if an entry in intro-install_plan.json refers to a directory? I would like to avoid to have to stat() all entries to figure it out.

@lithomas1 I got it working with test cases using install_subdir() but I still have troubles testing with pandas. I get an error importing pandas.core.methods but looking at the package I would say that is expected: there isn't an __init__.py in pandas/pandas/core/methods/ and I don't see what in the build definition puts one there. In pandas/pandas/meson.build there is a stray fs.copyfile('__init__.py') that seems to be an incomplete attempt to copy an empty init.py` in some folders. Does https://github.com/pandas-dev/pandas/pull/49115 need to be updated?

@eli-schwartz Is there a way to know from Meson introspection data if an entry in intro-install_plan.json refers to a directory? I would like to avoid to have to stat() all entries to figure it out.

Directories are stored as:
{
  "targets": {},
  "python": {
    "path/to/src/t/foo.py": {
      "destination": "{py_purelib}/foo.py",
      "tag": "python-runtime"
    }
  },
  "data": {
    "path/to/src/t/builddir/meson-private/foo.pc": {
      "destination": "share/pkgconfig/foo.pc",
      "tag": "devel"
    }
  },
  "install_subdirs": {
    "path/to/src/t/include": {
      "destination": "{prefix}/include",
      "tag": null
    },
    "path/to/src/t/mymod": {
      "destination": "{py_purelib}/mymod",
      "tag": null
    }
  }
}

Note this is grouped as install_subdirs.

@lithomas1 I got it working with test cases using install_subdir() but I still have troubles testing with pandas. I get an error importing pandas.core.methods but looking at the package I would say that is expected: there isn't an __init__.py in pandas/pandas/core/methods/ and I don't see what in the build definition puts one there. In pandas/pandas/meson.build there is a stray fs.copyfile('__init__.py') that seems to be an incomplete attempt to copy an empty __init__.py in some folders. Does pandas-dev/pandas#49115 need to be updated?

That directory(pandas/core) is installed with install_subdir. The fs.copyfile should only be for the top level init.

Then, there is no __init__.py in it and it cannot be imported.

Then, there is no __init__.py in it and it cannot be imported.

I don't know what pandas wants to do, but that statement is false in general, because it just means the package is a namespace package.

From the import error it seems pandas tried to import the pandas.core.methods, not the modules contained in it. But I may have misread the error. Let me check.

I have it in the source tree locally. I've just merged main again into that branch, can you try pulling again?

Pulling from the PR branch again, everything works. Can you give it another spin?
However, @eli-schwartz made me realize that this probably does not handle namespace packages correctly. I need to look into it.

It works for me, thanks for working on this.

Namespaces support was indeed missing. I added it and also added tests to check that it works.

FYI I opened #102097, which would make this implementation easier by a landslide. The earliest it could get in is 3.12, though.

I just scanned quickly through the issue. I agree that the current interfaces require a bit too much boiler plate and re-implementing some non that obvious code. However, also implementing the Traversable interface is not free of tedious code and non-obvious corner cases which even the support classes introduced in Python 3.11 get wrong. I didn't file bugs for CPython only because the status of importlib.resources is so bad that I would not know where to start.

Please do though! There's so much complexity here that we need to keep track of that it's usually hard to take a step back and notice those kind of issues, so feedback is definitely highly appreciated.

Gentle ping here. Is the plan to merge this in its current state?
Just checking to see if I need to pick up my meson PR to pandas again.

I think there is general agreement that this approach is better than the one currently implemented in meson-python. I don't know of any issues with this implementation. This PR is ready to be merged, I just fixed a merge conflict with main, and is awaiting a review. Or I can merge it as it is and fix any issue in follow-up patches.

Uhm, two test failures on Windows. I don't immediately understand the cause. I'll have a look.

Test failure on Windows is Meson failing to compile the Cython code in the test package. The test failure is triggered by the addition of support for Cython depfiles in Meson 1.0.1. @eli-schwartz Do you have any idea of what is going on here https://github.com/mesonbuild/meson-python/actions/runs/4291541241/jobs/7476854821#step:9:653 ? Somehow Cython is trying to compute a relative path and fails.

Ah! The source directory and the build directory are on two separate Windows drives in the test. This seems to do not be supported. Windows is weird.

Ah! The source directory and the build directory are on two separate Windows drives in the test. This seems to do not be supported. Windows is weird.

I ran into the same thing just last week. One has to do try: os.path.relpath(...); except ValueError: os.path.abspath(...). Luckily the two drives scenario does happen on GHA, otherwise it'd be a silent bug wherever you use relpath for anything outside of your own control.

@rgommers I understand, but the code that fails is in Cython. It seems that on Windows having a build directory on a different drive than the source directory is not supported.

Hmm, I think that that is supposed to work, probably a bug.

I would not know where the bug is, though. The depfile records relative paths between the build artifact and the source code that generated it. On Windows it is not possible to generate relative paths between different drives. Maybe absolute paths need to be used in this case. I don't know where the requirement for relative paths comes from. @eli-schwartz maybe knows more. If having the source tree and the build tree on the same drive is a more general requirement, maybe Meson can at least warn about it.

I don't know where the requirement for relative paths comes from. @eli-schwartz maybe knows more. If having the source tree and the build tree on the same drive is a more general requirement, maybe Meson can at least warn about it.

Meson catches the case of source tree and build tree on different drives, and uses absolute paths everywhere. I'm pretty sure that's what compilers which emit depfiles do too :) but I don't have a Windows machine to test.
This is a cython bug, cython should do the same thing: https://github.com/cython/cython/pull/5279

Thanks @eli-schwartz I've just added a work-around for the moment

I don't think releasing 0.13 with one implementation of the editable wheel support and switch to another implementation for 0.14 makes much sense, also considering that editable wheels support is the highlight of 0.13. Everyone agrees that this approach is the one that should be taken. Still I would like someone else to have a look at the implementation.

Okay, I'll try to finish my review this week, sorry for the delay.

@lithomas1 did you have a chance to try this is combination with setting break-points in an IDE?

@lithomas1 did you have a chance to try this is combination with setting break-points in an IDE?

Yeah, looks like it works perfectly there.

@dnicolodi please do not merge pull requests after a review before at least giving the reviewers a chance to reply to feedback and/or re-review the parts that changed.

Sorry. The last time you said that one round of review was enough. I felt like all your comments have been addressed. I must have misunderstood.


STORM-350: Upgrade to newer version of disruptor

This replaces #750 because I rebased.
@HeartSaVioR I think I found the bug that you saw.  @kishorvpatil was doing scale testing and found an NPE issue.  There was a race condition when a read was timing out and I badly tried to read after the timeout instead of just returning.

@revans2
I'm sorry but could you elaborate, please?
NPE could be occurred that we were reading same offset two or more times since we set offset to null after reading.
I don't know Disruptor deeply, but as far as I understand, I think consumeBatchWhenAvailable() seems not be thread safe unless _barrier.waitFor() forces wait for other thread to read datas first, since there're timing issue between modification of cursor and getting current cursor before calling _barrier.waitFor().
If consumeBatchWhenAvailable() is not thread-safe, revised consumeBatch() is not thread-safe, too.

Hi  @revans2   do you test new version disruptor ?   Would you know how much performance improvement ?  thanks you for you work~

@HeartSaVioR
The original version of consumeBatch looked like this.
    public void consumeBatch(EventHandler<Object> handler) {
        consumeBatchToCursor(_barrier.getCursor(), handler);
    }

where as consumeBatchWhenAvailable will wait for the cursor to be available.
 _barrier.waitFor(nextSequence)

I am not sure what happened in the disruptor code between the old version and the new version, but simply getting the cursor from the barrier does not guarantee that it has been committed.  It may still be being updated.  Perhaps this has always been a problem but we lose the race a lot more frequently with the newer code.
When I originally wrote the disruptor upgrade code I copied/modified the code so that we could do batching on the receiver side, not on the sender side.  This didn't help as much as I had hoped, but I also left in some code that when the timeout happened it would try to do something similar to what consumeBatch did, so we could read as much of the batch as possible.  This also resulted in the same issue, but it would only show up when a timeout happened on read, meaning the queue was empty for 1 ms, and right at that moment something was inserted into the queue.  Surprisingly with word count on 16 workers it only takes about 10 mins to reproduce the issue.  My tests were pushing things as fast as possible so they never let any queue stay empty for more then 1 ms.
The fix was to remove the special handling in the Timeout, and just say there was nothing to process so loop around and try again.

@darionyaphet I have seen only small changes to the performance of storm.  It is close to nothing really, but in my micro-benchmarks I see about between a 6% and 26% improvement in maximum throughput.
The big reason for upgrading is that the APIs are improved so that we can support #765.  In that I have seen a maximum throughput for a single node more then double. Making a faster version of wordcount go from 10,000 sentences/second fully counted to 22,000, on my Mac Book Pro laptop.  If you include acking and everything else that happens that is going from 320,000 tuples/second sent+processed to 704,000 tuples/second.  Be aware that you are not likely to see this in real life on a single topology, as the messaging layer is likely to slow things down, and wordcount is a very light weight processing.  Real world processing is usually more intensive.
But this is just the beginning we are working on benchmarks and subsequent optimizations all throughput the critical path of storm.

LGTM. +1.
@HeartSaVioR would you like to revisit and see Bobby's changes have addressed you comment. We have been running this in our staging for some time now and I would like to pull this one in if you have no objections.

@revans2 @kishorvpatil
I was seeing failed tuples constantly so I couldn't vote +1.
I'm suspecting that auto back pressure is in effect since throughput drops continuously after some minutes launched perftest.
I'll re-test with disabling auto back pressure.

@HeartSaVioR  We are not seeing any tuple loss, so there as way can we reproduce it? Some details on what topology, cluster setup and/or h/w to help us reproduce and investigate might help.

@HeartSaVioR It may because of your send and receive buffer settings. What is the value of those settings for worker and executor?

@kishorvpatil @binhnv @revans2
Here's my storm.yaml.
worker.childopts: "-Dfile.encoding=UTF-8 -Xmx768m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/daum/logs/storm/heapdump.hprof -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=1%ID% -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false"
topology.backpressure.enable: false
topology.transfer.buffer.size: 32
topology.executor.send.buffer.size: 16384
topology.executor.receive.buffer.size: 16384

I used topology.backpressure.enable, topology.transfer.buffer.size, topology.executor.send.buffer.size, topology.executor.receive.buffer.size as default but no luck.
I changed these values as http://www.michael-noll.com/blog/2013/06/21/understanding-storm-internal-message-buffers but also no luck.
Failed tuples appear within 3 minutes, which is same as we were seen from #134 (origin STORM-350).

I'm using VMs (IaaS on my office), but without Disruptor 3.3.2 it doesn't show failed tuples.
I ran perftest on command line.

storm jar storm_perf_test-1.0.0-SNAPSHOT-jar-for-0.10.0-rc-with-dependencies.jar com.yahoo.storm.perftest.Main  --ack --ackers 3 --bolt 4 --name test -l 1 -n 1 --workers 3 --spout 3 --testTimeSec 900 -c topology.max.spout.pending=1092 --messageSize 10

Here's log.
1034 [main] INFO  c.y.s.p.Main - Adding in 3 spouts
1081 [main] INFO  c.y.s.p.Main - Adding in 4 bolts
1231 [main] INFO  b.s.StormSubmitter - Generated ZooKeeper secret payload for MD5-digest: -6427640641721598663:-8827787475099997913
1236 [main] INFO  b.s.s.a.AuthUtils - Got AutoCreds []
1297 [main] INFO  b.s.StormSubmitter - Uploading topology jar storm_perf_test-1.0.0-SNAPSHOT-jar-for-0.10.0-rc-with-dependencies.jar to assigned location: /data1/storm/nimbus/inbox/stormjar-38afac0d-a0d8-49a6-8529-7705d89280e9.jar
1326 [main] INFO  b.s.StormSubmitter - Successfully uploaded topology jar to assigned location: /data1/storm/nimbus/inbox/stormjar-38afac0d-a0d8-49a6-8529-7705d89280e9.jar
1326 [main] INFO  b.s.StormSubmitter - Submitting topology test_0 in distributed mode with conf {"storm.zookeeper.topology.auth.scheme":"digest","storm.zookeeper.topology.auth.payload":"-6427640641721598663:-8827787475099997913","topology.workers":3,"topology.acker.executors":3,"topology.debug":false,"topology.max.spout.pending":1092}
1520 [main] INFO  b.s.StormSubmitter - Finished submitting topology: test_0
status	topologies	totalSlots	slotsUsed	totalExecutors	executorsWithMetrics	time	time-diff ms	transferred	throughput (MB/s)
WAITING	1	12	0	13	0<PHONE_NUMBER>611	0	0	0.0
WAITING	1	12	3	13	13<PHONE_NUMBER>611	30000	72600	0.02307891845703125
WAITING	1	12	3	13	13<PHONE_NUMBER>612	30001	608340	0.1933796318931296
RUNNING	1	12	3	13	13<PHONE_NUMBER>611	29999	783820	0.24917797349429877
RUNNING	1	12	3	13	13<PHONE_NUMBER>611	30000	200520	0.06374359130859375
RUNNING	1	12	3	13	13<PHONE_NUMBER>611	30000	17140	0.005448659261067708
RUNNING	1	12	3	13	13<PHONE_NUMBER>612	30001	0	0.0
RUNNING	1	12	3	13	13<PHONE_NUMBER>611	29999	17980	0.0057158785989480905
RUNNING	1	12	3	13	13<PHONE_NUMBER>611	30000	0	0.0
RUNNING	1	12	3	13	13<PHONE_NUMBER>611	30000	18180	0.005779266357421875
RUNNING	1	12	3	13	13<PHONE_NUMBER>611	30000	0	0.0
RUNNING	1	12	3	13	13<PHONE_NUMBER>611	30000	17620	0.005601247151692708
RUNNING	1	12	3	13	13<PHONE_NUMBER>611	30000	0	0.0
RUNNING	1	12	3	13	13<PHONE_NUMBER>611	30000	16800	0.005340576171875
RUNNING	1	12	3	13	13<PHONE_NUMBER>611	30000	0	0.0
RUNNING	1	12	3	13	13<PHONE_NUMBER>611	30000	17500	0.005563100179036458
RUNNING	1	12	3	13	13<PHONE_NUMBER>611	30000	0	0.0
RUNNING	1	12	3	13	13<PHONE_NUMBER>611	30000	17840	0.005671183268229167
RUNNING	1	12	3	13	13<PHONE_NUMBER>611	30000	0	0.0
RUNNING	1	12	3	13	13<PHONE_NUMBER>611	30000	18260	0.005804697672526042
RUNNING	1	12	3	13	13<PHONE_NUMBER>611	30000	0	0.0
RUNNING	1	12	3	13	13<PHONE_NUMBER>612	30001	17980	0.0057154975530763555
RUNNING	1	12	3	13	13<PHONE_NUMBER>611	29999	0	0.0
RUNNING	1	12	3	13	13<PHONE_NUMBER>611	30000	18000	0.0057220458984375
RUNNING	1	12	3	13	13<PHONE_NUMBER>611	30000	0	0.0
RUNNING	1	12	3	13	13<PHONE_NUMBER>611	30000	17720	0.005633036295572917
RUNNING	1	12	3	13	13<PHONE_NUMBER>611	30000	0	0.0
RUNNING	1	12	3	13	13<PHONE_NUMBER>611	30000	18720	0.005950927734375
RUNNING	1	12	3	13	13<PHONE_NUMBER>611	30000	0	0.0
RUNNING	1	12	3	13	13<PHONE_NUMBER>612	30001	18640	0.005925298909307189
RUNNING	1	12	3	13	13<PHONE_NUMBER>611	29999	0	0.0
RUNNING	1	12	3	13	13<PHONE_NUMBER>612	30001	18160	0.005772716104775674
RUNNING	1	12	3	13	13<PHONE_NUMBER>611	29999	0	0.0

It nearly stops functioning but I couldn't find any warning / error from worker logs.
It may be nice if we can share perftest jar / binary dist file which applies STORM-350 (with Disruptor 3.3.2) / binary dist file which doesn't apply STORM-350 (with Disruptor 2.10.4).

@HeartSaVioR I ran the patch with few topologies including acking. But don't see failed tuples. Can you tell me which topology you are running.

@HeartSaVioR I am seeing similar things to what @harshach is seeing.  I really want to trace this down and fix it.  How many nodes do you have?  Which daemons are running on which nodes? What is version of java you are running? What OS are you running on?  Can you share some information about the hardware, I know it is VMs but number of cores and frequency would be good.  What is the network connection between the nodes?
The failures you are seeing look like what I would see when ZK or the network would get overloaded.  The heartbeats could not make it to ZK and so it didn't show any change in the data some of the time, but with only 3 workers and none of them getting rescheduled I find that hard to believe.  Can you share any of the logs?  Have you tried to run zktop to see if any of the nodes in the ensemble are showing signs of slowness.  Have you looked to see if the network and disk utilization?

@harshach
I ran performance test using https://github.com/yahoo/storm-perf-test that @revans2 created.
@revans2

3 machines (All machines are VM)
CPU:

lscpu says

Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                2
On-line CPU(s) list:   0,1
Thread(s) per core:    2
Core(s) per socket:    1
CPU socket(s):         1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 45
Stepping:              7
CPU MHz:               1995.244
BogoMIPS:              3990.48
Hypervisor vendor:     Xen
Virtualization type:   para
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              20480K
NUMA node0 CPU(s):     0,1

RAM: 8 GB
JDK: Oracle JDK <IP_ADDRESS>
OS: RHEL 6.3
Processes

ZK is installed for all 3 VMs
Nimbus, UI are installed for the first VM
Supervisor, DRPC, Logviewer are installed for all 3 VMs

I didn't check CPU / Memory / network / disk utilization. I'll re-run perftest and check that when I'm available.
I'll re-run perftest without this patch and share the result.

Btw, I can't get idle PMs so I have to run perftest via VMs. :(

@HeartSaVioR I am still not seeing any errors.
I just pushed the exact version of the code that I tested with dfba63e6e3768085a3545cf70c78993586409a25 I created 3 VMs that are similar to yours, but with less memory because my VM quota on our IaaS didn't have that much memory left.
$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.3 (Santiago)
$ java -version
java version "1.7.0_51"
$ cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 13
model name      : QEMU Virtual CPU version (cpu64-rhel6)
stepping        : 3
cpu MHz         : 2094.950
cache size      : 4096 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 4
wp              : yes
flags           : fpu de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm unfair_spinlock pni cx16 hypervisor lahf_lm
bogomips        : 4189.90
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 13
model name      : QEMU Virtual CPU version (cpu64-rhel6)
stepping        : 3
cpu MHz         : 2094.950
cache size      : 4096 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 4
wp              : yes
flags           : fpu de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm unfair_spinlock pni cx16 hypervisor lahf_lm
bogomips        : 4189.90
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

I modified the perf-test to also print the number of failed tuples so I would not have to keep refreshing the UI all the time. I just pushed that to the perf-test repo https://github.com/yahoo/storm-perf-test/commit/96d65792be02d9d6f203dc20c329a40e6a9302f0
I ran the same command multiple times, each with a very similar result.
908  [main] WARN  b.s.u.NimbusClient - Using deprecated config nimbus.host for backward compatibility. Please update your storm.yaml so it only has config nimbus.seeds
1084 [main] INFO  c.y.s.p.Main - Adding in 3 spouts
1126 [main] INFO  c.y.s.p.Main - Adding in 4 bolts
1267 [main] INFO  b.s.StormSubmitter - Generated ZooKeeper secret payload for MD5-digest: -5117308553277668052:-5039407269887585982
1273 [main] INFO  b.s.s.a.AuthUtils - Got AutoCreds []
1273 [main] WARN  b.s.u.NimbusClient - Using deprecated config nimbus.host for backward compatibility. Please update your storm.yaml so it only has config nimbus.seeds
1310 [main] WARN  b.s.u.NimbusClient - Using deprecated config nimbus.host for backward compatibility. Please update your storm.yaml so it only has config nimbus.seeds
1361 [main] WARN  b.s.u.NimbusClient - Using deprecated config nimbus.host for backward compatibility. Please update your storm.yaml so it only has config nimbus.seeds
1403 [main] INFO  b.s.StormSubmitter - Uploading topology jar ./storm_perf_test-1.0.0-SNAPSHOT-jar-with-dependencies.jar to assigned location: /home/ME/apache-storm-0.11.0-SNAPSHOT/storm-local/nimbus/inbox/stormjar-e1d3ca41-9ffe-466d-81dd-33e7f90b8367.jar
1420 [main] INFO  b.s.StormSubmitter - Successfully uploaded topology jar to assigned location: /home/ME/apache-storm-0.11.0-SNAPSHOT/storm-local/nimbus/inbox/stormjar-e1d3ca41-9ffe-466d-81dd-33e7f90b8367.jar
1420 [main] INFO  b.s.StormSubmitter - Submitting topology test_0 in distributed mode with conf {"storm.zookeeper.topology.auth.scheme":"digest","storm.zookeeper.topology.auth.payload":"-5117308553277668052:-5039407269887585982","topology.workers":3,"topology.acker.executors":3,"topology.debug":false,"topology.max.spout.pending":1092}
1530 [main] INFO  b.s.StormSubmitter - Finished submitting topology: test_0
status  topologies      totalSlots      slotsUsed       totalExecutors  executorsWithMetrics    time    time-diff ms    transferred     throughput (MB/s)       total Failed
WAITING 1       12      0       0       0<PHONE_NUMBER>986   0       0       0.0     0
WAITING 1       12      3       13      10<PHONE_NUMBER>986   30000   69560   0.022112528483072918    0
WAITING 1       12      3       13      10<PHONE_NUMBER>986   30000   1012280 0.32179514567057294     0
WAITING 1       12      3       13      11<PHONE_NUMBER>986   30000   1346940 0.4281806945800781      0
WAITING 1       12      3       13      11<PHONE_NUMBER>986   30000   1419380 0.45120875040690106     0
WAITING 1       12      3       13      11<PHONE_NUMBER>986   30000   1315180 0.41808446248372394     0
WAITING 1       12      3       13      11<PHONE_NUMBER>987   30001   1324780 0.4211221828901276      0
WAITING 1       12      3       13      11<PHONE_NUMBER>986   29999   1377400 0.43787826374811456     0
WAITING 1       12      3       13      11<PHONE_NUMBER>986   30000   1432580 0.45540491739908856     0
WAITING 1       12      3       13      11<PHONE_NUMBER>987   30001   789260  0.25089063396780004     0
WAITING 1       12      3       13      11<PHONE_NUMBER>987   30000   1307960 0.41578928629557294     0
WAITING 1       12      3       13      12<PHONE_NUMBER>986   29999   1420280 0.451509903031924       0
...
WAITING 1       12      3       13      13<PHONE_NUMBER>986   30000   1330760 0.42303721110026044     0
RUNNING 1       12      3       13      13<PHONE_NUMBER>986   30000   1421300 0.45181910196940106     0
RUNNING 1       12      3       13      13<PHONE_NUMBER>986   30000   1403780 0.44624964396158856     0
RUNNING 1       12      3       13      13<PHONE_NUMBER>986   30000   1325140 0.4212506612141927      0
RUNNING 1       12      3       13      13<PHONE_NUMBER>986   30000   1328120 0.42219797770182294     0
RUNNING 1       12      3       13      13<PHONE_NUMBER>986   30000   1338720 0.425567626953125       0
RUNNING 1       12      3       13      13<PHONE_NUMBER>986   30000   1396000 0.4437764485677083      0
RUNNING 1       12      3       13      13<PHONE_NUMBER>986   30000   1400840 0.44531504313151044     0
RUNNING 1       12      3       13      13<PHONE_NUMBER>987   30001   1417260 0.4505198032298663      0
RUNNING 1       12      3       13      13<PHONE_NUMBER>986   29999   1238620 0.393759819256345       0
RUNNING 1       12      3       13      13<PHONE_NUMBER>986   30000   1400860 0.44532140096028644     0
RUNNING 1       12      3       13      13<PHONE_NUMBER>986   30000   1429200 0.4543304443359375      0
RUNNING 1       12      3       13      13<PHONE_NUMBER>986   30000   1339820 0.4259173075358073      0
RUNNING 1       12      3       13      13<PHONE_NUMBER>986   30000   1432580 0.45540491739908856     0
RUNNING 1       12      3       13      13<PHONE_NUMBER>986   30000   1373020 0.43647130330403644     0
RUNNING 1       12      3       13      13<PHONE_NUMBER>986   30000   1380660 0.4388999938964844      0
RUNNING 1       12      3       13      13<PHONE_NUMBER>986   30000   1489820 0.4736010233561198      0
RUNNING 1       12      3       13      13<PHONE_NUMBER>986   30000   1414680 0.44971466064453125     0
RUNNING 1       12      3       13      13<PHONE_NUMBER>987   30001   1403300 0.44608218666474136     0
RUNNING 1       12      3       13      13<PHONE_NUMBER>986   29999   1357460 0.4315392971595147      0
RUNNING 1       12      3       13      13<PHONE_NUMBER>986   30000   1418080 0.4507954915364583      0
RUNNING 1       12      3       13      13<PHONE_NUMBER>986   30000   1361900 0.4329363505045573      0
RUNNING 1       12      3       13      13<PHONE_NUMBER>987   30001   1376360 0.43751847676041006     0
RUNNING 1       12      3       13      13<PHONE_NUMBER>987   30000   979760  0.3114573160807292      0
RUNNING 1       12      3       13      13<PHONE_NUMBER>986   29999   1476040 0.4692361205334449      0
RUNNING 1       12      3       13      13<PHONE_NUMBER>986   30000   1394800 0.4433949788411458      0
RUNNING 1       12      3       13      13<PHONE_NUMBER>986   30000   1454180 0.46227137247721356     0
RUNNING 1       12      3       13      13<PHONE_NUMBER>986   30000   1420840 0.45167287190755206     0
RUNNING 1       12      3       13      13<PHONE_NUMBER>986   30000   1478660 0.47005335489908856     0
RUNNING 1       12      3       13      13<PHONE_NUMBER>987   30001   1402460 0.4458151667568112      0
RUNNING 1       12      3       13      13<PHONE_NUMBER>986   29999   1386660 0.44082203659718344     0
1741535 [main] INFO  c.y.s.p.Main - KILLING test_0

It takes a very long time for the event logger bolt to send enough metrics that the code registers that it is up and starts the test, so each test usually lasts about 30 mins.  I got no failures anywhere during the testing.
The following numbers were all collected on the nimbus/ui/zk/supervisor/logviewer node.  I also launched the topology from that node, so it would probably be the most overloaded.  All the other nodes were running a supervisor/logviewer/zk
top - 14:29:54 up 57 min,  4 users,  load average: 5.21, 4.50, 3.23
Tasks: 132 total,   2 running, 130 sleeping,   0 stopped,   0 zombie
Cpu(s): 48.0%us, 42.2%sy,  1.7%ni,  0.2%id,  0.5%wa,  0.0%hi,  7.3%si,  0.2%st
Mem:   4055232k total,  3161188k used,   894044k free,   123084k buffers

This is the output of running zktop.py against the zk ensemble (IP addresses and Host names changed).
Ensemble -- nodecount:32 zxid:0x10000086b sessions:16

ID SERVER           PORT M    OUTST    RECVD     SENT CONNS MINLAT AVGLAT MAXLAT
0  NIMBUS  2181 F        0     3895     3901     5      0      1     33
1  SUPER1  2181 L        0     2054     2054     3      0      1     22
2  SUPER2  2181 F        0    11358    11376     8      0      0     29

CLIENT           PORT S I   QUEUED    RECVD     SENT
NIMBUS   54133 0 1        0      330      330
NIMBUS   54629 0 1        0     1313     1313
NIMBUS  55899 0 0        0        1        0
SUPER1    41270 0 1        0     1305     1305
NIMBUS   54130 0 1        0      511      511
NIMBUS   49674 1 1        0      357      357
NIMBUS   51440 1 0        0        1        0
SUPER2   60290 1 1        0     1305     1305
SUPER2   51100 2 1        0     1147     1151
NIMBUS   53264 2 1        0     1148     1152
NIMBUS   53088 2 1        0     7937     7940
NIMBUS   54851 2 0        0        1        0
SUPER1    60972 2 1        0      216      216
NUMBUS   53263 2 1        0      217      217
SUPER1    60973 2 1        0     1148     1152
SUPER2   51098 2 1        0      216      216

Everything there looks OK.
Disk utilization while the test is running is between 0.6% and 0.7%.  Network maxed out at 23054.9 kbps in and 21443.2 kbps out.  Gigabit should be able to handle theoretically 128 MB/sec in and out so the 22/23 MB/sec should totally be within that range.  If somehow the connection slipped to 100 Mbit then this would possibly explain what you are seeing, but it is just a guess.
I have screen shots of the UI too, but I don't think it will add much.  I am now going to try and turn on the logging metrics consumer and see if I can get a measure for the GC/etc as another possible issue.
storm.yaml
worker.childopts: "-Dfile.encoding=UTF-8 -Xmx768m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/heapdump.hprof -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=1%ID% -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false"
topology.backpressure.enable: false
topology.transfer.buffer.size: 32
topology.executor.send.buffer.size: 16384
topology.executor.receive.buffer.size: 16384

storm.zookeeper.servers:
  - "NIMBUS"
  - "SUPER1"
  - "SUPER2"

nimbus.host: "NIMBUS"

command line I ran
./apache-storm-0.11.0-SNAPSHOT/bin/storm jar ./storm_perf_test-1.0.0-SNAPSHOT-jar-with-dependencies.jar com.yahoo.storm.perftest.Main --ack --ackers 3 --bolt 4 --name test -l 1 -n 1 --workers 3 --spout 3 --testTimeSec 900 -c topology.max.spout.pending=1092 --messageSize 10 | tee run.txt

@revans2 @kishorvpatil
I don't want to block this since you already did enough tests, and other PR depends on this.
+1
Since our new IaaS takes over legacy IaaS, I'll test it with new VMs when I complete set up new Storm cluster.

+1 as well.

Thanks @HeartSaVioR @harshach. I will pull this in.


fix bug: do not select the lastest snapshot/pit with --latest flag

I think an easy fix is to sort the snapshots before selecting the first one. I'm running cohesity 6.8.2 and snapshot list start with the oldest one (smallest snapshotTimestampUsecs).
if snapshots is not None and len(snapshots) > 0:
            if snapshots[0]['snapshotTimestampUsecs'] > latestSnapshotTimeStamp:
                latestSnapshot = snapshots[0]          <-- WE JUST SELECT THE FIRST ONE, WHY ? DO WE PRESUME ALREADY SORTED ?
                latestSnapshotTimeStamp = snapshots[0]['snapshotTimestampUsecs']
                latestSnapshotObject = object

same thing for pointsForTimeRange
if logRange['timeRanges'][0]['endTimeUsecs'] > latestLogPIT:
                    latestLogPIT = logRange['timeRanges'][0]['endTimeUsecs']
                if latest:
                    pit = logRange['timeRanges'][0]['endTimeUsecs']  <-- WE JUST SELECT THE FIRST ONE, WHY ? DO WE PRESUME ALREADY SORTED ?
                    break
                else:
                    if logRange['timeRanges'][0]['endTimeUsecs'] > desiredPIT and logRange.timeRanges[0]['startTimeUsecs'] <= desiredPIT:
                        pit = desiredPIT
                        break

Fixed. Thanks.


KAFKA-12495: allow consecutive revoke in incremental cooperative assignor in connector

jira: https://issues.apache.org/jira/browse/KAFKA-12495
Allow consecutive revoke in incremental cooperative assignor in connector, to fix the issue that when new members joined right after revocation round, it causes uneven distribution. (please check the jira for better understanding) What I did are:

remove canRevoke variable, since we allow consecutive revoking (as lone as delay == 0) now
compute currentWorkerAssignmentWithoutDuplication, to remove duplicated connectors/tasks from currentWorkerAssignment, so that we can use it in performTaskRevocation method, to compute if we need to revoke more connectors/tasks in this round by computing if the remaining assignments in each worker is higher than totalTasks/totalWorkers
we also passed the configured assignment into performTaskRevocation instead of activeAssignment, so that we can compute the correct expected max assignment number (totalSize / workerSize), instead of the activeTotalSize / workerSize, because the activeTotalSize doesn't include the newAssignments, which will cause the wrong computation, and cause uneven rebalance, or need more round of revoking rebalance.
Improve readability for tests by adding final assignment visualization for each phase of rebalance

With the change of (2) and (3), we can still make sure the revocation is always correct no matter if this is a consecutive revoking, or we have duplicated assignment.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

@kkonstantine @rhauch @ramesh-muthusamy , could you help review this PR to fix the uneven distribution in incremental cooperative assignor in connector? Thanks.

@kkonstantine , could you please check this PR when available? Thank you.

@kkonstantine , could you check this PR? Or I should find another guy to review this PR? Thanks.

@kkonstantine , I suddenly found this is a V3.0 blocker bug. Could you help take a look. Thanks.

@kkonstantine , call for review and comments. Thanks.

@C0urante , thanks for your comments. TBH, I need some time to revisit the code (since it's long time ago...), and answering your comments later. Thank you.

Thanks @showuon. In that case, I can file separate issues for a lot of the comments I've made here, and we can try to keep this PR as focused as possible for the sake of moving forward.

@C0urante , sure, please file another PR for other comments. And thanks for the comments. However, I'm still concerned that @kkonstantine doesn't like the current solution, and would like to have another proposal as mentioned here. So, I think we still need to get his approval before we can continue. WDYT?
cc @kkonstantine  , we need your suggestions here, please!

@kkonstantine , sorry to keep pinging you, but we need your advice before we can continue. Thanks.

@showuon FWIW, I've got a refactoring PR up for the testing logic that we might take a look at in the meantime: https://github.com/apache/kafka/pull/11974
Hopefully this should make writing and reviewing tests for changes like in this PR easier in the future.

@showuon FYI, I've just opened https://github.com/apache/kafka/pull/12019, which should address KAFKA-12495 and some other issues with rebalancing, but without using consecutive revocations.

Great! I'll take a look next week. Also cc @kkonstantine . Thanks.

@showuon @C0urante @kkonstantine What is the status of this PR? As far as I understand this PR might resolve KAFKA-8391, KAFKA-12283 and KAFKA-12495. Is this correct? Those tickets block the 3.2.0 release.

@showuon @C0urante @kkonstantine What is the status of this PR? As far as I understand this PR might resolve KAFKA-8391, KAFKA-12283 and KAFKA-12495. Is this correct? Those tickets block the 3.2.0 release.

@cadonna , it's correct (Only KAFKA-8391 is not 100% sure). So far, we are waiting for @kkonstantine 's comments about what's the better solution for this issue, since he was concerned about the current solution, and has a better solution for that (commented here).

@showuon @C0urante @kkonstantine What is the status of this PR? As far as I understand this PR might resolve KAFKA-8391, KAFKA-12283 and KAFKA-12495. Is this correct? Those tickets block the 3.2.0 release.

Hi, It very probably also resolves KAFKA-10413.

Thanks for working on this fix @showuon. Apologies for taking me so long to return here.
My main concern is related to the proposed change to apply consecutive rebalances that will perform revocations.
The current incremental cooperative rebalancing algorithm, is using two consecutive rebalances in order to move tasks between workers. One rebalance during which revocations are happening and one during which the revoked tasks are reassigned. Although clearly this is not an atomic process (as this issue also demonstrates) I find that it's a good property to maintain and reason about.
Allowing for consecutive revocations that happen immediately when an imbalance is detected might mean that the workers overreact to external circumstances that have caused an imbalanced between the initial calculation of task assignments of the revocation rebalance and the subsequent rebalance for the assignment of revoked tasks. Such circumstances might have to do with rolling upgrades, scaling a cluster up or down or simply might be caused by temporary instability. We were first able to reproduce this issue in integration tests by the test that is currently disabled.
My main thought was that, instead of risking shuffling tasks too aggressively within a short period of time and open the door to bugs that will make workers oscillate between imbalanced task assignments continuously and in a tight loop, we could use the existing mechanism of scheduling delayed rebalances to program workers to perform a pair of rebalanced (revocation + reassignment) soon after an imbalance is detected. Regarding when an imbalance is detected, the good news is that the leader worker sending the assignment during the second rebalance of a pair of rebalances knows that it will send an imbalanced assignment (there's no code to detect right now that but can be easily added just before the assignment is sent). The idea here would be to send this assignment anyways, but also schedule a follow up rebalance that will have the opportunity to balance tasks soon with our standard pair of rebalances that works dependably as long as no new workers are added or removed between the two rebalances. We can discuss what is a good setting for the delay. One obvious possibility is to reuse the existing property. Adding another config just for that seems unwarranted. To shield ourselves from infinite such rebalances the leader should also keep track of how many such attempts have been made and stop attempting to balance out tasks after a certain number of tries. Of course every other normal rebalance should reset both this counter and possibly the delay.
I'd be interested to hear what do you think of this approach that is quite similar to what you have demonstrated already but potentially less risky in terms of changes in the assignor logic and how aggressively the leader attempts to fix an imbalance.

@kkonstantine Thank you for your thoughts! From a 3.2.0 release perspective your proposal seems to be a change that we should postpone to a later release since feature freeze and code freeze has passed. Or is this a regression? If it is a regression is there a quick intermediate fix that we can include into 3.2.0 to unblock the release? If it is not a regression, I would propose to link the corresponding tickets one to each other and then to move them to the next release. Ideally with an assignee.

My comment is put in jira ticket. Thanks.


Declaration of non-initialized variable using no-semicolons coding style breaks syntax highlighting (VS Code)

Bug Report
🔎 Search Terms
syntax highlighting tokenizer
Declaration of non-initialized variable using no-semicolons coding style breaks syntax highlighting.
Typing semicolon or adding a line break fixes it

VSCode
Version: 1.77.0 (user setup)
Typescript: 5.0.2, 5.0.3

Duplicate of  #971


Depend on openff-toolkit-base?

Comment:
Would you consider a change of 's/openff-toolkit/openff-toolkit-base/'? It'd be nice to be able to use NAGL alongside RDKit and/or OpenEye Toolkits but without AmberTools.
I poked around the repo (of the source code, not the feedstock) and didn't find any obvious AmberTools dependency. Not sure if there are other technical details.

Sweet - I'm happy to handle this myself if you haven't already started

Ah no please go ahead -- thank you!

Fixed by #11!


prefer nerdctl over crictl for containerd if available

Many thanks for this useful tool.
We prefer to use containerd's native nerdctl CLI tool over crictl which isn't even available on the k8s hosts.
This PR will use it if available and fallback to crictl otherwise.
Tested on our k0s cluster.
Could you please take a look?

By the way, there is also containerd's internal ctr CLI tool, which I didn't consider to include in this PR. Because it comes with an unsupported interface as its description says:

ctr is an unsupported debug and administrative client for interacting
with the containerd daemon. Because it is unsupported, the commands,
options, and operations are not guaranteed to be backward compatible or
stable from release to release of the containerd project.

Moreover, it doesn't support anything like --format or --output for its ctr containers info command, which would add yet another dependency on some json parsing CLI tool.

Also, could you add nerdctl support in bottom of README.md, in Supported technologies chapter? With description of how cli utilities will be chosen in case of containerd?

I did my best, feel free to suggest your preferred wording :-)

Also, could you add nerdctl support in bottom of README.md, in Supported technologies chapter? With description of how cli utilities will be chosen in case of containerd?

I did my best, feel free to suggest your preferred wording :-)

LGTM, thx :)

@miminar
kubectl krew upgrade nsenter

New version with the feature :)

Great, thanks a lot!


MySql AllXml-column is too small

What steps will reproduce the problem?

Get an error on a page with a lot of text in the error message.

What is the expected output? What do you see instead?
The error is logged to the database but is truncated so I can't open it later on when I visit elmah.axd.
What version of the product are you using? On what operating system?
Elmah 1.2, self compiled with a few modifications.
Additional information
The MySql AllXml-column is of type "text" which only handles up to 16kb of text which is enough for most cases but not all. It should be changed to "mediumtext" instead to be sure that the whole error log fits in it.

Originally reported on Google Code with ID 248
Reported by karl.sjogren on 2011-08-19 21:08:38

This issue was moved to elmah/MySqlErrorLog#1


react-select 2.4.x causes cypress test failures

We have pinned our dependency to 2.3.0 until we can resolve this issue.

We're on 3.0.x now as of #1344
@mitchellhamilton should we close this out?


Tight layout

Please use tight layout for all plots.
For example, the scatter plots have a lot of white empty space

scatter are fixed, closing for now.


Re-index documents in batch

Is there a way to re-index a bunch of documents in batch?
My situation is:
I have a model called Sku, which belongs to a Product
When i update a Product, i want to re-index associated skus
I have looked into docs, api...but index_document method seems to be very simple, maybe there's another method to perform that

You can call the import method with a specific scope, see https://github.com/elastic/elasticsearch-rails/blob/master/elasticsearch-model/lib/elasticsearch/model/importing.rb#L29-L102
(Sorry for missing your question for so long)

Hello @karmi, no problem...
I think the import with scope does not work for me, as it deletes the records out of scope
Like...ids 1, 2, 3, 4, 5 are already indexed, and i want to reindex only ids 1, 2, 3, 4, using import with scope, it deletes my id 5...i mean, if i understood correctly the behaviour of import

It doesn't actually delete record with ID 5 -- it will just reindex 1, 2, 3, 4 (when they included in records found by the the scope of course). Closing this then?

Very nice! Didn't know that...
Thanks!


Add codecov

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@0b89046). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master     #111   +/-   ##
=========================================
  Coverage          ?   61.94%
=========================================
  Files             ?       20
  Lines             ?      975
  Branches          ?      125
=========================================
  Hits              ?      604
  Misses            ?      347
  Partials          ?       24

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0b89046...651f6c0. Read the comment docs.

m(


Sort fix

This fixes a bug where the optimized sorting routines weren't used on ARM64 (where purego=true).
I've also added benchmarks for indirect sorting, and added tests for pivot selection to make sure we aren't triggering worst case quicksort behavior.

Big improvement in cachette/lsm's sorting benchmarks, which I rigged up to sort 100K items that are randomly ordered and ordered ascending and descending:
name                            old time/op  new time/op  delta
BufferSort_sortedAscending-10   23.9ms ± 3%   3.4ms ± 2%   ~     (p=0.100 n=3+3)
BufferSort_sortedDescending-10  16.7ms ± 3%   3.6ms ± 2%   ~     (p=0.100 n=3+3)
BufferSort_random-10            9.54ms ± 7%  9.19ms ± 2%   ~     (p=0.700 n=3+3)


Not working for 2022?

Tried both on macOS and Windows (only Chrome, no FF yet) and the plugin is greyed out.
Tried re-installing multiple times.
Is it a me problem?

Hmmm, for me it's working in Chrome (both home Win10 PC and work Win10 laptop) and Firefox (work WIn10 laptop). I've not heard any reports from other users or colleagues either. Also tried removing and reinstalling it in Chrome just now, that also works for me.
Could you investigate a little bit more, share any kind of logs or other details that might be helpful?

Oh,it runs on "run_at": "document_end".
I just didn't scroll enough.

Haha 😅 phew glad it worked out! Thanks for getting back to us. Happy puzzling!


can't see suggestions together with Lsp

this is my packer.lua file
...

    use {
        "zbirenbaum/copilot.lua",
        cmd = "Copilot",
        event = "InsertEnter",
        config = function()
            require("copilot").setup({
                suggestion = { enabled = false },
                panel = { enabled = false },
            })
        end,
    }

    use {
        "zbirenbaum/copilot-cmp",
        after = { "copilot.lua" },
        config = function ()
            require("copilot_cmp").setup({
                event = { "InsertEnter", "LspAttach" },
                fix_pairs = true,
            })
        end
    }

...

this is my cmp_config.lua file
local cmp = require('cmp')

print("Loading Copilot-cmp")

cmp.setup({
  sources = {
    { name = "copilot", group_index = 1, keyword_pattern = "." },
    { name = "nvim_lsp", group_index = 2 },
    { name = "buffer", group_index = 3 },
  },

  completion = {
    keyword_length = 2,
  },

  experimental = {
      ghost_text = true,
  }
})

when I'm writing code, only lsp suggestions showed, like in the next photo.

when there is no suggestions left, like the following

if I press tab, the copilot suggestions appear.

what's the problem, how can I get all suggestions together, without pressing tab explicitly and when there are other suggestions.

same problem

I think I have the same problem with no copilot suggestions appearing in the cmp menu

My phenomenon is that sometimes only copilot appears in the menu, and everything else I need is cleared

My phenomenon is that it doesn't appear at all

Same here!

Same here

same here

for me, cmp never contains suggestions from copilot (if I turn on suggestions and auto_trigger, I can see the ghost text containing the copilot autocompletion). tried solutions from all the issues with this problem but nothing works. I'm using nvchad, if that is relevant.

for me, cmp never contains suggestions from copilot (if I turn on suggestions and auto_trigger, I can see the ghost text containing the copilot autocompletion). tried solutions from all the issues with this problem but nothing works. I'm using nvchad, if that is relevant.

NVM solved it. I was not setting the nvim-cmp options correctly. solved it  by doing:
  {
    "hrsh7th/nvim-cmp",
    event = "VeryLazy",
    config = function()
      local opts = require "plugins.configs.nvim-cmp"
      require("cmp").setup(opts)
    end
  },

Still happening

wow, its been a year but there isn't any solution
I switched to copilot.vim, and suggest all the people here the same, cuz maintainer doesn't care this issue 🤣


Add create_directory method for jail to allow creating subdirs

Add the ability to create subdirs in the jail so we can create files in the subdirs.
I considered the option of working out from the file path that it included a subdir and then automagically creating the subdir struct necessary.
BUT I decided against that as it would loose the specificity of the test code and perhaps make it unclear of exactly what was happening and when. I think this might be useful to have very clearly and simply defined in a test harness.

I need this to test functionality of adding secrets from a subdirectory.
My intended PR for that is here: https://github.com/nitnelave/figment_file_provider_adapter/pull/2

Thank you! Merged with some changes in https://github.com/SergioBenitez/Figment/commit/affdb62d50c56571bc58ecc733e3818a9e29b6a3. Will release a new version with this method shortly.

Released in 0.10.12.


Package the app

@Mingze for the deploy we need to be sure that we can deploy the app in a tomcat environment. My suggestion is to package the app using the same approach than the cluster is following (https://github.com/PRIDE-Cluster/cluster-web-app)

done


qa for comment incentive 2

qa

/start

/start

/comment-incentive @me505 true

please work
please


Is the pretrained model weight availabel now?

awesome job! amzaing!!
But may I ask that will you release a pretrained weight and if it will, how long will it takes?

The pretrained weights are already available! They download automatically when you launch the boltz predict command. Here's also a direct link

Hi @jwohlwend - Congrats on the release! I'm VB from the open source team at Hugging Face. Like @ValMystletainn I also had troubles finding the model weights, so I uploaded them over to the Hugging Face Hub over a community org: https://huggingface.co/boltz-community/boltz-1
I also opened a PR here: https://github.com/jwohlwend/boltz/pull/2 adding download support directly from the model repo too. Let me know if this works for you, I'm happy to add you as the admin or to transfer the weights to your namespace if you want! 🤗

@Vaibhavs10 That's awesome, thanks for setting this up. Could you please make me the sole admin? My huggingface username is jwohlwend

Done! You should’ve received an invite. I made you an admin and demoted my
role (feel free to remove me as well)
Sent from my iPhone, please excuse my typos!
On Mon 18. Nov 2024 at 21:50, Jeremy Wohlwend @.***>
wrote:

@Vaibhavs10 https://github.com/Vaibhavs10 That's awesome, thanks for
setting this up. Could you please make me the sole admin? My huggingface
username is jwohlwend
—
Reply to this email directly, view it on GitHub
https://github.com/jwohlwend/boltz/issues/1#issuecomment-2484102636, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AEOREK3CF6GPKAXDVZCVLJT2BJHJTAVCNFSM6AAAAABR6DUZAGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOBUGEYDENRTGY
.
You are receiving this because you were mentioned.Message ID:
@.***>

Amazing, thanks a lot!!

Weights are now on huggingface, and this is what is used by default in the repo! Thank you @Vaibhavs10 !!


isPaused method

mmm I'm not sure this is the best way to do it, I think a getPlayerStatus method would be much better. This method should get the status directly from the Javascript player.
Also the YouTubeListener interface already has a onStateChange method, this could be used to keep track of the status, from outside the player.


tweaks to three of the manuscript files

@lawinslow added ckan stuff - Couldn't find a paper or anything to cite for ckan, so just the URL

Cool. Thanks!


Add rwmutex to connection established.

We need this to prevent data race happened when running more than one router in the same application process, i got it in my unit test
Below is the data race error message that fixed by this PR
WARNING: DATA RACE
Read at 0x00c000188031 by goroutine 30:
  github.com/weaveworks/mesh.(*LocalConnection).isEstablished()
      /Users/ibk/pathgo/src/github.com/weaveworks/mesh/connection.go:122 +0x3e
  github.com/weaveworks/mesh.(*Peer).encode()
      /Users/ibk/pathgo/src/github.com/weaveworks/mesh/peers.go:532 +0x244
  github.com/weaveworks/mesh.(*localPeer).encode()
      /Users/ibk/pathgo/src/github.com/weaveworks/mesh/local_peer.go:133 +0x92
  github.com/weaveworks/mesh.(*Peers).encodePeers()
      /Users/ibk/pathgo/src/github.com/weaveworks/mesh/peers.go:398 +0x240
  github.com/weaveworks/mesh.(*topologyGossipData).Encode()
      /Users/ibk/pathgo/src/github.com/weaveworks/mesh/router.go:311 +0x74
  github.com/weaveworks/mesh.(*gossipSender).deliver()
      /Users/ibk/pathgo/src/github.com/weaveworks/mesh/gossip.go:144 +0xac
  github.com/weaveworks/mesh.(*gossipSender).run()
      /Users/ibk/pathgo/src/github.com/weaveworks/mesh/gossip.go:105 +0x207

Previous write at 0x00c000188031 by goroutine 20:
  github.com/weaveworks/mesh.(*LocalConnection).actorLoop()
      /Users/ibk/pathgo/src/github.com/weaveworks/mesh/connection.go:358 +0x3dd
  github.com/weaveworks/mesh.(*LocalConnection).run()
      /Users/ibk/pathgo/src/github.com/weaveworks/mesh/connection.go:256 +0xc13

Goroutine 30 (running) created at:
  github.com/weaveworks/mesh.newGossipSender()
      /Users/ibk/pathgo/src/github.com/weaveworks/mesh/gossip.go:94 +0x238
  github.com/weaveworks/mesh.(*gossipChannel).makeGossipSender()
      /Users/ibk/pathgo/src/github.com/weaveworks/mesh/gossip_channel.go:126 +0x11c
  github.com/weaveworks/mesh.(*gossipChannel).makeGossipSender-fm()
      /Users/ibk/pathgo/src/github.com/weaveworks/mesh/gossip_channel.go:122 +0x5f
  github.com/weaveworks/mesh.(*gossipSenders).Sender()
      /Users/ibk/pathgo/src/github.com/weaveworks/mesh/gossip.go:247 +0x178
  github.com/weaveworks/mesh.(*gossipChannel).senderFor()
      /Users/ibk/pathgo/src/github.com/weaveworks/mesh/gossip_channel.go:122 +0xc7
  github.com/weaveworks/mesh.(*gossipChannel).SendDown()
      /Users/ibk/pathgo/src/github.com/weaveworks/mesh/gossip_channel.go:93 +0x4c
  github.com/weaveworks/mesh.(*Router).sendAllGossipDown()
      /Users/ibk/pathgo/src/github.com/weaveworks/mesh/router.go:208 +0x133
  github.com/weaveworks/mesh.(*localPeer).handleAddConnection()
      /Users/ibk/pathgo/src/github.com/weaveworks/mesh/local_peer.go:191 +0x581
  github.com/weaveworks/mesh.(*localPeer).doAddConnection.func1()
      /Users/ibk/pathgo/src/github.com/weaveworks/mesh/local_peer.go:108 +0x68
  github.com/weaveworks/mesh.(*localPeer).actorLoop()
      /Users/ibk/pathgo/src/github.com/weaveworks/mesh/local_peer.go:143 +0x144

Goroutine 20 (running) created at:
  github.com/weaveworks/mesh.startLocalConnection()
      /Users/ibk/pathgo/src/github.com/weaveworks/mesh/connection.go:99 +0x4c7
  github.com/weaveworks/mesh.(*localPeer).createConnection()
      /Users/ibk/pathgo/src/github.com/weaveworks/mesh/local_peer.go:98 +0x2a3
  github.com/weaveworks/mesh.(*connectionMaker).attemptConnection()
      /Users/ibk/pathgo/src/github.com/weaveworks/mesh/connection_maker.go:375 +0x21f

Thanks for this!  I see the race very clearly from your report.  (there was already a comment noting it)
However, the "actor" model used in much of this package is explicitly chosen to avoid the use of mutexes, therefore I will not merge this right away.
Further thought and consideration is required.

However, the "actor" model used in much of this package is explicitly chosen to avoid the use of mutexes, therefore I will not merge this right away.

didn't know about it, i'll investigate it further

@iwanbk if you like videos you can watch me explain the whole idea here https://www.gophercon.co.uk/videos/2016/an-actor-model-in-go/

I'll close this rather than leave it hanging around: it needs more work.  Happy to re-open if you come back to it.


Prepare docs and Swagger definition for automatically generated API docs

- What I did
This is some preparation for automatically generating API documentation from a Swagger definition. See https://github.com/docker/docker/issues/22931.
It is the Engine half of this documentation pull request. This pull request should be merged before the documentation pull request.
It consists of a bunch of things:

Removed the API reference for the latest API version. This is now all in api/swagger.yaml, which the documentation is automatically generated from.
Removed all the other API documentation from this repository. The old versions barely ever change, and all the other stuff is not tied to any code change, so they should probably just live in docker/docker.github.io.
Renamed "Remote API" to "Engine API". See https://github.com/docker/docker/issues/28319 for details.
Update api/swagger.yaml with some more documentation and some recent API changes.

- A picture of a cute animal (not mandatory but encouraged)

/cc @dnephin @johndmulhausen

So excited about this, thanks @bfirsh. Slowly, we are killing all these hand-rolled markdown files and I couldn't be happier. To be getting such nice-looking API docs is just an added bonus!
cc @mstanleyjones

Is there an easy way to generate these docs for preview / reviewing pull requests?

@thaJeztah I think the easiest way is to copy swagger.yaml into the right place in the documentation repo and run the documentation locally.
That could be documented somewhere I suppose. Perhaps a "API documentation" section in the contributing docs?

If we want this included in 1.13, then these changes need cherry-picking to the 1.13 branch. I see no reason why not to include it – it's purely a documentation change.
Following an offline discussion with @thaJeztah, I'm also going to add some docs about how to write and test the API docs.

Adding to the milestone, but /cc @vieux

@thaJeztah Documentation added! https://github.com/bfirsh/docker/blob/prepare-api-docs-for-swagger-docs/project/API.md

LGTM

There we go!

@bfirsh can you create a PR against the 1.13.x branch directly ? this one is tricky to cherry-pick since it's 5 commits and the version changed in master

@bfirsh nevermind I got it

@vieux Nice, thank you! Appreciate that, because I realise it was non-trivial...

@bfirsh https://github.com/docker/docker/blob/master/docs/api/version-history.md has 2 broken links, 1.25 and 1.26

@vieux They will exist once https://github.com/docker/docker.github.io/pull/606 exists. :)
Well, 1.26 won't, but the 1.13 branch doesn't have that broken link.


"Segmentation fault" on Vercel when used inside Next.js getStaticProps (via Plaiceholder)

First of all, thank you! Sharp was the fundamental step in the creation of plaiceholder, and I'm extremely grateful for the work you put into maintaining this library.
I recently stumbled across an issue using sharp on Vercel after Next.js implemented their WASM-based approach - I was no longer able to use sharp inside getStaticProps in Vercel's environment. I can build the project perfectly locally, but when deployed to Vercel, the following issue occurs.
In the example provided, sharp is used inside my library plaiceholder.
I appreciate this is both a question/issue that might be a bit out of your remit, but I would be really grateful any guidance you could offer here, as infra/deployments are a bit out of my area of expertise. Is there something I can add to sharp in plaiceholder to make it more portable? or is this something that could be resolved on the sharp side?
Related Issues

https://github.com/joe-bell/plaiceholder/issues/60
https://github.com/vercel/next.js/issues/23531

Are you using the latest version? Is the version currently in use as reported by npm ls sharp the same as the latest version as reported by npm view sharp dist-tags.latest?
Uses<EMAIL_ADDRESS>(a peerDependency of plaiceholder)
What are the steps to reproduce?
Clone the minimal reproduction, and deploy to Vercel.
Alternatively, view the logs of the latest deployment.
What is the expected behaviour?
Given that sharp runs successfully as a custom pre-build step on Vercel, I would expect sharp to run successfully in getStaticProps
Are you able to provide a minimal, standalone code sample, without other dependencies, that demonstrates this problem?

Details for the Next.js/Vercel reproduction can be found in the relevant issue
Details for sharp usage can be found in plaiceholder's packages directory

Are you able to provide a sample image that helps explain the problem?
N/A
What is the output of running npx envinfo --binaries --system?
Running inside Vercel, I get the following:
09:18:42.578 | > npx envinfo --binaries --system
-- | --
09:18:43.661 | npx: installed 1 in 1.004s
09:18:44.051 | System:
09:18:44.051 | OS: Linux 4.14 Amazon Linux 2
09:18:44.051 | CPU: (4) x64 Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz
09:18:44.051 | Memory: 24.10 GB / 29.94 GB
09:18:44.051 | Container: Yes
09:18:44.051 | Shell: 4.2.46 - /bin/bash
09:18:44.051 | Binaries:
09:18:44.051 | Node: 14.15.4 - /node14/bin/node
09:18:44.051 | Yarn: 1.22.10 - ~/.yarn/bin/yarn
09:18:44.051 | npm: 6.14.10 - /node14/bin/npm

Update: the Vercel team took a look at this and seem to believe it might be something on the sharp side:
https://github.com/vercel/next.js/issues/23531#issuecomment-815589234

My best guess would be that this relates to the recent introduction of worker threads to Next.js.
The backtrace with __nptl_deallocate_tsd looks like something is attempting to dlclose a shared library whilst it is still in use.
https://sharp.pixelplumbing.com/install#worker-threads

The main thread must call require('sharp') before worker threads are created to ensure shared libraries remain loaded in memory until after all threads are complete.

There's some background to this at #2263.

Thanks for the details! I'm quite new to the concept of Node.js worker threads, so apologies if I'm missing something obvious here...
Would "calling before worker threads" be either:

Something I add to getStaticProps():export const getStaticProps = async () => {
  require('sharp');

  // an external function which uses `sharp`
  const images = await getPlaceholders();

  return {
    props: { images },
  };
};

Something to add to my next.config.js?
or something I need to ask the Next.js team to support on?

I think this logic is already running inside a worker thread by the time getStaticProps() is called.
When Next.js removed sharp, they also removed the use of require('sharp') from the main thread, so we need to find a new "hook" that allows you to make the call to require('sharp') in the main thread.
I've been able to reproduce locally, and it looks like simply creating the following, basic next.config.js fixes it.
require('sharp');
module.exports = {};

If Next.js provides a build-time setupStaticPropsOnTheMainThread() hook, that might be a cleaner place to add this, but I'm not sure such a thing exists.

Wow you're absolutely right, that worked perfectly.
I'll pass on this message to the Next.js team and see what they'd recommend (config vs. the proposed hook).
Thanks so much for your help on this @lovell

@lovell is there a way to fix this so requiring sharp from the main thread isn't required? We're having this issue in Vercel with Parcel as well. All Parcel plugins run inside worker threads, and we don't really have a clean way to also require sharp from the main thread before our workers start within our plugin system.

@devongovett Node.js uses reference counting internally when it calls dlopen and dlclose. It might be possible subvert that check and keep the count above zero to prevent the dlclose by calling require('sharp'), then removing its entry from require.cache, then requiring it again.
The logic in https://github.com/sindresorhus/import-fresh might help here, e.g. (untested):
require('sharp');
const sharp = require('import-fresh')('sharp');


Added in Calendar Modul the Option coloredSymbolOnly

If set to true, only the symbol is colored and the title is still white. Downwards compatible. No changes needed if old behaviour should persist.

Hi, thanks for your contribution. Unfortunately modules/default/calendar/calendar.js has some conflicts. Could you please rebase and merge your changes? Thanks!

Will try. This is my 1st ever contribution. Ever to an opensource project. I used the master/stable branch. I assume this is the problem? Lets see.
Von meinem iPhone gesendet

Am 30.05.2018 um 08:46 schrieb Michael Teeuw<EMAIL_ADDRESS>Hi, thanks for your contribution. Unfortunately modules/default/calendar/calendar.js has some conflicts. Could you please rebase and merge your changes? Thanks!
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

I just found out that this feature is already available in the develop
branch. I am confused. Anyhow. Nothing to do on your end. Keep up the good.
work. Thank you.
2018-05-31 1:10 GMT+02:00 Ralf Klüber<EMAIL_ADDRESS>
Will try. This is my 1st ever contribution. Ever to an opensource project.
I used the master/stable branch. I assume this is the problem? Lets see.
Von meinem iPhone gesendet
Am 30.05.2018 um 08:46 schrieb Michael Teeuw<EMAIL_ADDRESS>Hi, thanks for your contribution. Unfortunately modules/default/calendar/
calendar.js has some conflicts. Could you please rebase and merge your
changes? Thanks!
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/MichMich/MagicMirror/pull/1300#issuecomment-393048808,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGHLqxUd8NXDxenJO6rccd_eHD9E-OWFks5t3kBUgaJpZM4USOJq
.

--
<EMAIL_ADDRESS>


KAFKA-4203: Align broker default for max.message.bytes with Java producer default

Review by @hachikuji

lgtm

LGTM