[WIP] Atop fusion

The objective here is to build atop operations symbolically, and then fuse as much as we can before generating graphs.  I'm separting this into three parts:

[x] Have atop insert a symbolic Mapping object into the sharedict, rather than a full dictionary
[ ] Pick those symbolic mappings out during the collections->graph phase, and arrange them into groups that can be fused together
[ ] Perform the fusion on each group

The first and third parts of this seem fine.  I'm running into trouble with the second piece.  We don't have information about which symbolic mappings depend on the others.  It's not hard to back this out in the case of a group of symbolic mappings, but it's not easy to figure out when non-atop layers depend on atop layers.  This is important to make sure that we don't fuse those away.
Example
y = (x + 1 + 2).sum()
z = (x + 1).max()
dask.compute(x, y)

In this case we want to atop-fuse x + 1 as a single group, and then go on from there.  We don't want to fuse all of x + 1 + 2, because then the .max() operation won't have access to the correct intermediate variable.
So when we collect together all of the layers of the sharedicts into a big sharedict:
>>> sharedict.merge(x.dask, y.dask)
{'x-layer': {...}
 'add-1-layer': symbolic-dict,
 'add-2-layer': symbolic-dict,
 'sum-layer': {...},
 'max-layer': {...}}

We can look at all of the sharedicts and establish dependencies between them (this is cheap to do given the metadata that we have.  However we can't easily determine that max-layer depends on add-1-layer without diving through all of add-1-layer, which might be expensive.

@jcrist if you have any thoughts on how to determine which atop-mapping layers need to be in the graph I would welcome those thoughts.  No biggie if nothing comes to mind though.
I'm still pondering this.  I keep coming to conclusions like "we need to track more metadata throughout all array operations" though.  I'm still hoping that we discover an easier approach.

So, another option I'll mention would be to create a fully symbolic layer.  This is obviously much larger in scope, but also feels cleaner long-term.
I think that by relying on the logic in dask array for single-chunked arrays we can probably bootstrap a symbolic system pretty quickly.  If the original goal is to support XArray users then maybe there is a well-scoped set of operations (atop, reductions, slicing) that suffices near-term.  The hardest part might be to get XArray to dispatch on it cleanly.
As a separate effort I might spend half a day to cook something up and post it for review by XArray folks, who might be willing to contribute a bit.

https://github.com/mrocklin/symbolic-array/issues/1

Small update here.  I've been working a bit on the function to fuse atop metadata.  This now steals the SubGraphCallable implementation from #3979 and changes around the kind of metadata we store for indices (lists of tuples rather than a dict).  This helps us capture the full atop system.  If anyone takes a look at this (not particularly recommended at this point) I recommend avoiding the stuff around TOP and really just focus on the rewrite_atop code, which at this point is the main contribution of this PR, and a possible solution to point three above "Perform the fusion on each group".

I'll think more on this, but my first thought is that we can pretty quickly generate a set of all keys that are naively produced by not-fusing the symbolic graphs (e.g. what keys are the outputs of a atop operation). Given that, we can then run get_dependencies on all non-symbolic graphs to establish which atop operations need to have their outputs exposed. This isn't ideal, but may not be too terrible.

In [1]: %time _ = {('add-1238rh1hf', i, j) for i in range(1000) for j in range(1000)}
CPU times: user 236 ms, sys: 64.1 ms, total: 300 ms
Wall time: 299 ms

Looks like we can generate fake keys at around 300ns each.  This is probably ok to start.  I'll give this approach a try next.

So, barring a creative solution (which I think still has a chance of arising) I think that there are two comprehensive solutions to this problem:

We explicitly maintain a high level dependency tree between layers as we create the ShareDict task graph.  This would mean the introduction of new code everywhere in the dask array codebase, replacing lines like the following:
return Array(sharedict.merge((name, new_dsk), x.dask, y.dask), ...

with
return Array(sharedict.merge((name, new_dsk), x.dask, y.dask, dependencies={name: {x.name, y.name}}), ...

This is annoying, and introduces a barrier to entry for newcomers to dask array, but might not be that big of an endeavor.  It's probably about as expensive as the introduction of ShareDicts.

Develop full-blown symbolic arrays.  This adds a whole new layer and so increases complexity, but it also solves a few other problems.  It's a larger cost and a larger benefit.

I plan to investigate the cost of solution 1 if people aren't immediately against it.

Current status
In [1]: import dask.array as da

In [2]: x = da.ones((10000, 10000), chunks=(100, 100))  # 10k tasks

In [3]: x + 1  # first time we do this there is an import cost
Out[3]: dask.array<add, shape=(10000, 10000), dtype=float64, chunksize=(100, 100)>

In [4]: %time y = (x + 1 + 2 + 3 + 4 + 5).sum()
CPU times: user 43.2 ms, sys: 81 µs, total: 43.3 ms
Wall time: 42.7 ms

In [5]: {k: type(v) for k, v in y.dask.dicts.items()}
Out[5]:
{'sum-0835c592afa6e4de1d89b9732362e174': dask.array.core.TOP,
 'add-3671ae89980c3cca48838ce544bb1e88': dask.array.core.TOP,
 'add-55f5b5b6c1cbffbaa12d4507dce07055': dask.array.core.TOP,
 'add-22945ef41b1b2fe3c286cbb76000e4d2': dask.array.core.TOP,
 'add-896fceccb3deb065f83c186e1589b2e1': dask.array.core.TOP,
 'add-2177746a32fb5b5813f7ac1597dd2d15': dask.array.core.TOP,
 'ones-c7ae0f68c19a1208e37a3458b1ff57b7': dict,
 'sum-partial-6b66031dc6f0b5c86401e14be4fd4546': dict,
 'sum-partial-4e0bc23bb549115040e576da58146208': dict,
 'sum-partial-49160e27d19fb3eb85e8ef39a65286d2': dict,
 'sum-partial-67d2f13ffc06e757d108dac5d6414597': dict,
 'sum-partial-2b86a42bd086517e3a76d84a5df690fb': dict,
 'sum-partial-eaa2d6be5f9b7f2a02964533d3b5d27f': dict,
 'sum-aggregate-865384c3faa5f5fdb2247a10a82a0d14': dict}

In [6]: y.dask = da.optimization.optimize_atop(y.dask)

In [7]: {k: type(v) for k, v in y.dask.dicts.items()}
Out[7]:
{'sum-aggregate-865384c3faa5f5fdb2247a10a82a0d14': dict,
 'sum-partial-eaa2d6be5f9b7f2a02964533d3b5d27f': dict,
 'sum-partial-2b86a42bd086517e3a76d84a5df690fb': dict,
 'sum-partial-67d2f13ffc06e757d108dac5d6414597': dict,
 'sum-partial-49160e27d19fb3eb85e8ef39a65286d2': dict,
 'sum-partial-4e0bc23bb549115040e576da58146208': dict,
 'sum-partial-6b66031dc6f0b5c86401e14be4fd4546': dict,
 'sum-0835c592afa6e4de1d89b9732362e174': dask.array.core.TOP,
 'ones-c7ae0f68c19a1208e37a3458b1ff57b7': dict}

In [8]: y.dask.dicts['sum-0835c592afa6e4de1d89b9732362e174'].dsk
Out[8]:
{'sum-0835c592afa6e4de1d89b9732362e174': (<function dask.compatibility.apply(func, args, kwargs=None)>,
  functools.partial(<function sum at 0x7fdb046eeae8>, dtype=dtype('float64')),
  ['add-3671ae89980c3cca48838ce544bb1e88'],
  {'axis': (0, 1), 'keepdims': True}),
 'add-3671ae89980c3cca48838ce544bb1e88': (<function _operator.add>,
  'add-55f5b5b6c1cbffbaa12d4507dce07055',
  5),
 'add-55f5b5b6c1cbffbaa12d4507dce07055': (<function _operator.add>,
  'add-22945ef41b1b2fe3c286cbb76000e4d2',
  4),
 'add-22945ef41b1b2fe3c286cbb76000e4d2': (<function _operator.add>,
  'add-896fceccb3deb065f83c186e1589b2e1',
  3),
 'add-896fceccb3deb065f83c186e1589b2e1': (<function _operator.add>,
  'add-2177746a32fb5b5813f7ac1597dd2d15',
  2),
 'add-2177746a32fb5b5813f7ac1597dd2d15': (<function _operator.add>, '_0', 1)}

This seems to handle the fail cases I can come up with.  It's not maximally efficient (we have an opportunity for diamond fusion here that we're not taking advantage of), but seems pretty good.
This also provides a model for future high-level optimization, both in dask.array and in dask.dataframe (I'll write up a separate issue).
@jcrist if you have a chance to try this out sometime soon I would appreciate it.

Also, this currently breaks many things.  I wouldn't bother reviewing it critically yet.  We're still just at a high-level review stage.

https://github.com/dask/dask/issues/4038

OK, so I've cleaned things up so that most of the corner cases work.  I had to screw around a bit to capture things like delayed values in kwargs/args.  We're currently suboptimal there (though no worse than before).  This probably happens in a couple other places as well.  We're also a bit dangerous in that the tokens we use "_1", "_2", etc. can be literals in the graph (working on things now).
Generally though I think that this is ready for review, at least from a design perspective.  I think it's feasible to go in short-term solution until a larger high-level-graph implementation is built.

Tests pass

@jcrist this could use your review

We're also a bit dangerous in that the tokens we use "_1", "_2", etc. can be literals in the graph (working on things now).

Perhaps use a less common format? I'd be comfortable without checking user arguments as long as our string format was sufficiently unlikely. Perhaps _top_#1, _top_#2, etc...?

We're also a bit dangerous in that the tokens we use "_1", "_2", etc. can be literals in the graph (working on things now).

Perhaps use a less common format? I'd be comfortable without checking user arguments as long as our string format was sufficiently unlikely. Perhaps top#1, top#2, etc...? The regex checking you're doing now is fine, but it'd be nice to avoid if possible.

I'm ok with the current solution.  Short term I expect that we'll continue to manipulate these things a bit and it's nice to have them be short and readable.

Interestingly, _get_recusive is the bottleneck in the following computation:
from dask.distributed import Client
client = Client(processes=False)

import dask.array as da
x = da.ones((10000, 10000), chunks=(100, 100))

for i in range(10):
    x = x + 1

y = x.sum()

y.compute()

I wouldn't be surprised to learn that a little bit of codegen would be useful in the near future.

I think I've handled everything so far @jcrist another round next week would be welcome.  I'm not sure how best to document the internals of the atop optimization stuff.  It is, as usual with atop work, somewhat dense.  I feel like I would be writing several lines of comments for every line of code in order to properly convey things (although maybe that's best?).
@shoyer you may also want to take a look if you have time (though the gufunc thing possibly needs you more :) )

Interestingly, _get_recusive is the bottleneck in the following computation:

Ah, nm.  The computation I was running wasn't showing up in profiles.  I can make things slightly faster with the following implementaiton for SubgraphCallable, but it's something like 60s->55s total for this computation
def get_simple(dsk, out, memory=None):
    from .local import _execute_task
    memory = memory or {}
    order = toposort(dsk)
    for key in toposort(dsk):
        task = dsk[key]
        result = _execute_task(task, memory)
        memory[key] = result
    return memory[out]

Overall this looks good to me. Thanks for being patient and pushing this through.

I wouldn't be surprised to learn that a little bit of codegen would be useful in the near future.

In the original implementation I used codegen for SubgraphCallable, but removed it for fuse_subgraphs. In atop fusion it makes sense, because the same SubgraphCallable object will be called multiple times (so some upfront cost is acceptable), where in fuse_subgraphs each fusion chain generates a new instance of SubgraphCallable. I think what you have is fine for now, but I may want to experiment with code generation again once this is in.

Planning to merge tomorrow

Hi, I'm not sure I'm getting it.
>>> x = da.from_array([1, 2, 3, 4], chunks=-1)
>>> y = x + 1 + 2
>>> dict(y.dask)

{('add-32f70143b9954b35e4ac9ca1969c3754', 0): (subgraph_callable,
                                               ('array-9ba4167a79c237389dcca8293f5d7898',
                                                0),
                                               1),
 ('add-7d66dc08a76776926cf37c7b929b6c41', 0): (subgraph_callable,
                                               ('add-32f70143b9954b35e4ac9ca1969c3754',
                                                0),
                                               2),
 ('array-9ba4167a79c237389dcca8293f5d7898', 0): array([1, 2, 3, 4])}

I was expecting to see a single node that merged the two operations within it?


Mismatch between UI and API for version information

Describe the bug
Version visible on UI reports 404 when using the API to get detailed information
To Reproduce
Steps to reproduce the behavior:

Open https://www.nuget.org/packages/microsoft.identity.client/
See versions available (including those hidden under 'Show more'), includes 4.26.0
Open https://api.nuget.org/v3/registration3/microsoft.identity.client/4.26.0.json - get 404

Other versions, e.g. https://api.nuget.org/v3/registration3/microsoft.identity.client/2.6.2.json, return expected information
Expected behavior
Every version in the UI should have detailed information available at the registration3 endpoint.

Hello, apologies for the inconvenience here. This is actually by design. The base URL you have there ( https://api.nuget.org/v3/registration3/) was deprecated long ago and we have since moved to a new base URL. The new base URL that corresponds to this old one is: https://api.nuget.org/v3/registration5-semver1/.
I should also note that this registration3 base URL and the new registration5-semver1 only contain SemVer 1.0.0 packages. Over 95% of users and clients today support SemVer 2.0.0 so I would recommend using the base URL that contains SemVer 2.0.0 packages as well: https://api.nuget.org/v3/registration5-gz-semver2/.
If you are encountering this problem it is likely because you hardcoded the base URL into your script or code. This is not advisable because we change these base URLs from time to time (perhaps due to new metadata backfills or other administrative actions). To dynamically discover the current base URL, you should go to the service index (https://api.nuget.org/v3/index.json) and look up the base URL with a type RegistrationsBaseUrl/3.6.0.
It might help clear things up if you want to read through our API docs:
https://docs.microsoft.com/en-us/nuget/api/overview
The registration (package metadata) resource is documented specifically here:
https://docs.microsoft.com/en-us/nuget/api/registration-base-url-resource
If you are writing your code in .NET, you can consider using the NuGet client SDK. Here's an example on how to enumerate versions and metadata using the .NET SDK:
https://docs.microsoft.com/en-us/nuget/reference/nuget-client-sdk#list-package-versions
This handles the service (base URL) discovery that I mentioned above and can save you a lot of trouble of writing it yourself.

I should also note that hard-coding against the "registration leaf" URL is a bit risky since it's only by implementation detail that the URL is formatted: {base URL}/{lower ID}/{lower, normalized version}.json. The official NuGet client does not use this pattern and it is not documented to be exactly like this all of the time.
https://docs.microsoft.com/en-us/nuget/api/registration-base-url-resource#registration-leaf

On nuget.org, the URL for the registration leaf document coincidentally contains the package version. However, this assumption should never be made by a client since server implementations are free to change the shape of the URL as long as the parent document has a valid link.

That being said, if your code only uses NuGet.org's V3 API and no other package source, you are free to take the risk and depend on this NuGet.org implementation detail. It's possible we'll change it in the future but currently we have no plans to do so.

Thank you for the quick and helpful reply! I had found the registration3 URL via a google search, and didn't realize I was working with a deprecated endpoint.
The particular item I'm writing currently is not in .NET. I'll make an immediate switch to use the new URL you note, and eventually switch over to avoid the risk on the implementation detail. Thanks


Evaluation Results with MOT16

Dear @ZQPei
I evaluated the performance using MOT16 training datasets and got the following result, just wondering is it reasonable?
IDF1   IDP   IDR  Rcll  Prcn  GT MT  PT  ML   FP    FN IDs    FM  MOTA  MOTP IDt IDa IDm
MOT16-02 24.9% 54.0% 16.2% 25.2% 83.7%  54  5  19  30  873 13348  70   190 19.9% 0.316  75  13  19
MOT16-04 40.3% 65.2% 29.1% 37.8% 84.6%  83  6  42  35 3279 29586 124   600 30.6% 0.292 100  30  16
MOT16-05 35.4% 44.9% 29.3% 54.8% 84.0% 125 27  65  33  712  3079  64   155 43.5% 0.300 138   5  82
MOT16-09 40.6% 51.6% 33.4% 58.5% 90.6%  25  6  17   2  321  2180  59    94 51.3% 0.271  59   4  10
MOT16-10 32.0% 46.2% 24.5% 41.9% 79.0%  54  9  21  24 1376  7151 132   321 29.7% 0.304 132  15  32
MOT16-11 36.7% 46.6% 30.3% 57.6% 88.7%  69 13  23  33  672  3888  35    87 49.9% 0.257  55   5  26
MOT16-13 23.5% 42.5% 16.3% 25.5% 66.5% 107  6  40  61 1472  8530 131   379 11.5% 0.352 146  22  47
OVERALL  34.8% 54.8% 25.5% 38.6% 83.0% 517 72<PHONE_NUMBER> 67762 615  1826 30.2% 0.295 705  94 232
The MOTA is between 11.5% (MOT16-13) to 51.3%(MOT16-09). I'm using yolov3.weights. The highest performance was relatively close to the deepsort author claimed 61.4%. However, the MOTP value is much worse than the author claimed 79.1%. here I got 0.352. Wondering is it correct?
I noticed many people ask for the original MOT16 videos to test. I downloaded the videos from their website but found the result quite strange(many result are either nan or negative). Then I found these videos have been resized to half width and half height. I therefore convert the image frames downloaded from MOT16 dataset into videos and use these videos as input for our program and got the answer as above.

Hi guy, can you clarify how you produce result to test on MOT16? This repo only contains code for demo, but as far as I know, the MOT16 requires results in csv-text file.
Could you please share your code for evaluating?
By the way, which type of MOT devkit did you use? The Matlab one or the py-motmetrics one?
Thanks!

Dear @ZQPei
I evaluated the performance using MOT16 training datasets and got the following result, just wondering is it reasonable?
IDF1 IDP IDR Rcll Prcn GT MT PT ML FP FN IDs FM MOTA MOTP IDt IDa IDm
MOT16-02 24.9% 54.0% 16.2% 25.2% 83.7% 54 5 19 30 873 13348 70 190 19.9% 0.316 75 13 19
MOT16-04 40.3% 65.2% 29.1% 37.8% 84.6% 83 6 42 35 3279 29586 124 600 30.6% 0.292 100 30 16
MOT16-05 35.4% 44.9% 29.3% 54.8% 84.0% 125 27 65 33 712 3079 64 155 43.5% 0.300 138 5 82
MOT16-09 40.6% 51.6% 33.4% 58.5% 90.6% 25 6 17 2 321 2180 59 94 51.3% 0.271 59 4 10
MOT16-10 32.0% 46.2% 24.5% 41.9% 79.0% 54 9 21 24 1376 7151 132 321 29.7% 0.304 132 15 32
MOT16-11 36.7% 46.6% 30.3% 57.6% 88.7% 69 13 23 33 672 3888 35 87 49.9% 0.257 55 5 26
MOT16-13 23.5% 42.5% 16.3% 25.5% 66.5% 107 6 40 61 1472 8530 131 379 11.5% 0.352 146 22 47
OVERALL 34.8% 54.8% 25.5% 38.6% 83.0% 517 72<PHONE_NUMBER> 67762 615 1826 30.2% 0.295 705 94 232
The MOTA is between 11.5% (MOT16-13) to 51.3%(MOT16-09). I'm using yolov3.weights. The highest performance was relatively close to the deepsort author claimed 61.4%. However, the MOTP value is much worse than the author claimed 79.1%. here I got 0.352. Wondering is it correct?
I noticed many people ask for the original MOT16 videos to test. I downloaded the videos from their website but found the result quite strange(many result are either nan or negative). Then I found these videos have been resized to half width and half height. I therefore convert the image frames downloaded from MOT16 dataset into videos and use these videos as input for our program and got the answer as above.
Hi guys, can you clarify how you get the test result on MOT16?you can communicate with me through my email<EMAIL_ADDRESS>

Dear @ZQPei
I evaluated the performance using MOT16 training datasets and got the following result, just wondering is it reasonable?
IDF1 IDP IDR Rcll Prcn GT MT PT ML FP FN IDs FM MOTA MOTP IDt IDa IDm
MOT16-02 24.9% 54.0% 16.2% 25.2% 83.7% 54 5 19 30 873 13348 70 190 19.9% 0.316 75 13 19
MOT16-04 40.3% 65.2% 29.1% 37.8% 84.6% 83 6 42 35 3279 29586 124 600 30.6% 0.292 100 30 16
MOT16-05 35.4% 44.9% 29.3% 54.8% 84.0% 125 27 65 33 712 3079 64 155 43.5% 0.300 138 5 82
MOT16-09 40.6% 51.6% 33.4% 58.5% 90.6% 25 6 17 2 321 2180 59 94 51.3% 0.271 59 4 10
MOT16-10 32.0% 46.2% 24.5% 41.9% 79.0% 54 9 21 24 1376 7151 132 321 29.7% 0.304 132 15 32
MOT16-11 36.7% 46.6% 30.3% 57.6% 88.7% 69 13 23 33 672 3888 35 87 49.9% 0.257 55 5 26
MOT16-13 23.5% 42.5% 16.3% 25.5% 66.5% 107 6 40 61 1472 8530 131 379 11.5% 0.352 146 22 47
OVERALL 34.8% 54.8% 25.5% 38.6% 83.0% 517 72<PHONE_NUMBER> 67762 615 1826 30.2% 0.295 705 94 232
The MOTA is between 11.5% (MOT16-13) to 51.3%(MOT16-09). I'm using yolov3.weights. The highest performance was relatively close to the deepsort author claimed 61.4%. However, the MOTP value is much worse than the author claimed 79.1%. here I got 0.352. Wondering is it correct?
I noticed many people ask for the original MOT16 videos to test. I downloaded the videos from their website but found the result quite strange(many result are either nan or negative). Then I found these videos have been resized to half width and half height. I therefore convert the image frames downloaded from MOT16 dataset into videos and use these videos as input for our program and got the answer as above.

Hi guys, can you clarify how you  got the test result   on MOT16?we can communicate through email<EMAIL_ADDRESS>

Hi guy, can you clarify how you produce result to test on MOT16? This repo only contains code for demo, but as far as I know, the MOT16 requires results in csv-text file.
Could you please share your code for evaluating?
By the way, which type of MOT devkit did you use? The Matlab one or the py-motmetrics one?
Thanks!

Hi guys,do you know how to get the test result on MOT16?how to run the script  yolov3_deepsort_eval.py?

Hi guy, can you clarify how you produce result to test on MOT16? This repo only contains code for demo, but as far as I know, the MOT16 requires results in csv-text file.
Could you please share your code for evaluating?
By the way, which type of MOT devkit did you use? The Matlab one or the py-motmetrics one?
Thanks!

Hi guys,do you know how to get the test result on MOT16?I got some error when i  run the script yolov3_deepsort_eval.py

Dear @ZQPei
I evaluated the performance using MOT16 training datasets and got the following result, just wondering is it reasonable?
IDF1 IDP IDR Rcll Prcn GT MT PT ML FP FN IDs FM MOTA MOTP IDt IDa IDm
MOT16-02 24.9% 54.0% 16.2% 25.2% 83.7% 54 5 19 30 873 13348 70 190 19.9% 0.316 75 13 19
MOT16-04 40.3% 65.2% 29.1% 37.8% 84.6% 83 6 42 35 3279 29586 124 600 30.6% 0.292 100 30 16
MOT16-05 35.4% 44.9% 29.3% 54.8% 84.0% 125 27 65 33 712 3079 64 155 43.5% 0.300 138 5 82
MOT16-09 40.6% 51.6% 33.4% 58.5% 90.6% 25 6 17 2 321 2180 59 94 51.3% 0.271 59 4 10
MOT16-10 32.0% 46.2% 24.5% 41.9% 79.0% 54 9 21 24 1376 7151 132 321 29.7% 0.304 132 15 32
MOT16-11 36.7% 46.6% 30.3% 57.6% 88.7% 69 13 23 33 672 3888 35 87 49.9% 0.257 55 5 26
MOT16-13 23.5% 42.5% 16.3% 25.5% 66.5% 107 6 40 61 1472 8530 131 379 11.5% 0.352 146 22 47
OVERALL 34.8% 54.8% 25.5% 38.6% 83.0% 517 72<PHONE_NUMBER> 67762 615 1826 30.2% 0.295 705 94 232
The MOTA is between 11.5% (MOT16-13) to 51.3%(MOT16-09). I'm using yolov3.weights. The highest performance was relatively close to the deepsort author claimed 61.4%. However, the MOTP value is much worse than the author claimed 79.1%. here I got 0.352. Wondering is it correct?
I noticed many people ask for the original MOT16 videos to test. I downloaded the videos from their website but found the result quite strange(many result are either nan or negative). Then I found these videos have been resized to half width and half height. I therefore convert the image frames downloaded from MOT16 dataset into videos and use these videos as input for our program and got the answer as above.

I got the same result as u. Have u solved the problem?

Hi everyone, I managed to figure out how to change the code in yolov3_deepsort_eval.py such that it will run correctly. I realized that the documentation on how to do this is a bit lacking, so I'll chip in on how this can be done.

nowadays when you download the MOT16 data file from here, the "video" actually comes as a sequence of frames stored in MOT16/train/MOT16-XX/img1. You need to convert this into a video file for the tracker to track items in it. The resulting video (converted from frames in MOT16/train/MOT16-XX/img1) is the video that is being referred to on line 34 in yolov3_deepsort_eval.py (video_path = data_root+"/"+seq+"/video/video.mp4"). Change video_path to the path where your video is stored.

This is probably obvious, but on line 81 (data_root = 'data/dataset/MOT16/train/') data_root needs to refer to the root directory of the folder that you download from here.

This is probably obvious too, but for some reason, on line 36 (with VideoTracker(cfg, args, video_path, result_filename) as vdo_trk:) for some reason there are 4 arguments. This is incorrect, since the VideoTracker class in yolov3_deepsort.py only takes 3 arguments. The extra (incorrect) argument in this case is the last one, result_filename.  Two files are generated as a result of running the VideoTracker (from yolov3_deepsort.py) for an input video. These two files (results.avi and results.txt) are stored in the path that is provided in the argument "--save_path" (line 65). On line 65 (parser.add_argument("--save_path", type=str, default="./demo/demo.avi")) change default="./demo/demo.avi" to whichever path you want results.avi and result.txt to be stored. Note that if this is not changed, by default only demo.avi (which is results.avi in yolov3_deepsort.py) will be generated. If a path (as opposed to path to a specific file) is specified, then VideoTracker will generate and store the results of tracking as results.avi and results.txt, by default, as specified in line 59, 60 in yolov3_deepsort.py
(59: self.save_video_path = os.path.join(self.args.save_path, "results.avi")
60: self.save_results_path = os.path.join(self.args.save_path, "results.txt")).
Another work-around, of course, would be to specify a path as a flag when calling the function from the command line (i.e., python yolov3_deepsort_eval.py --save_path path/you/want/to/store/results/in).

Lastly, looking into line 33 (result_filename = os.path.join(result_root, '{}.txt'.format(seq))) one will notice that result_filename is actually meant to refer to the results.txt file that I mentioned above in (4). i.e., the results.txt file that is generated as a result of running the VideoTracker. You have to modify result_filename such that it refers to the path at which results.txt is stored for a given video. For instance, if --save_path is "./output/", then result_filename="./output/results.txt"

One major issue I noticed is that, even if the user passes in a path that does not lead to any legitimate results.txt file, the program does not give any indication (error/warning), that the user is running the program incorrectly. Instead, the mistake is reflected as a bunch of 0%s and nan's in the summary printed at the end of the program.
I hope this helps! If I'm missing anything, do leave a comment below!

     IDF1  IDP  IDR Rcll Prcn GT MT PT ML    FP FN IDs  FM  MOTA MOTP IDt IDa IDm

MOT16-01 0.0% 0.0% nan% nan% 0.0%  0  0  0  0  3620  0   0   0 -inf%  nan   0   0   0
MOT16-03 0.0% 0.0% nan% nan% 0.0%  0  0  0  0 54495  0   0   0 -inf%  nan   0   0   0
MOT16-06 0.0% 0.0% nan% nan% 0.0%  0  0  0  0  9982  0   0   0 -inf%  nan   0   0   0
MOT16-07 0.0% 0.0% nan% nan% 0.0%  0  0  0  0  7471  0   0   0 -inf%  nan   0   0   0
MOT16-08 0.0% 0.0% nan% nan% 0.0%  0  0  0  0  7087  0   0   0 -inf%  nan   0   0   0
MOT16-12 0.0% 0.0% nan% nan% 0.0%  0  0  0  0  4734  0   0   0 -inf%  nan   0   0   0
MOT16-14 0.0% 0.0% nan% nan% 0.0%  0  0  0  0  7451  0   0   0 -inf%  nan   0   0   0
OVERALL  0.0% 0.0% nan% nan% 0.0%  0  0  0  0 94840  0   0   0 -inf%  nan   0   0   0
I ran the yolo_deepsort_eval.py, but only got the FP. would you please what went wrong? thanks a lot!

MOT16-01 0.0% 0.0% nan% nan% 0.0% 0 0 0 0 3620 0 0 0 -inf% nan 0 0 0
MOT16-03 0.0% 0.0% nan% nan% 0.0% 0 0 0 0 54495 0 0 0 -inf% nan 0 0 0
MOT16-06 0.0% 0.0% nan% nan% 0.0% 0 0 0 0 9982 0 0 0 -inf% nan 0 0 0
MOT16-07 0.0% 0.0% nan% nan% 0.0% 0 0 0 0 7471 0 0 0 -inf% nan 0 0 0
MOT16-08 0.0% 0.0% nan% nan% 0.0% 0 0 0 0 7087 0 0 0 -inf% nan 0 0 0
MOT16-12 0.0% 0.0% nan% nan% 0.0% 0 0 0 0 4734 0 0 0 -inf% nan 0 0 0
MOT16-14 0.0% 0.0% nan% nan% 0.0% 0 0 0 0 7451 0 0 0 -inf% nan 0 0 0
OVERALL 0.0% 0.0% nan% nan% 0.0% 0 0 0 0 94840 0 0 0 -inf% nan 0 0 0
I ran the yolo_deepsort_eval.py, but only got the result of FP. would you please tell me what went wrong? thanks a lot!

MOT16-01 0.0% 0.0% nan% nan% 0.0% 0 0 0 0 3620 0 0 0 -inf% nan 0 0 0
MOT16-03 0.0% 0.0% nan% nan% 0.0% 0 0 0 0 54495 0 0 0 -inf% nan 0 0 0
MOT16-06 0.0% 0.0% nan% nan% 0.0% 0 0 0 0 9982 0 0 0 -inf% nan 0 0 0
MOT16-07 0.0% 0.0% nan% nan% 0.0% 0 0 0 0 7471 0 0 0 -inf% nan 0 0 0
MOT16-08 0.0% 0.0% nan% nan% 0.0% 0 0 0 0 7087 0 0 0 -inf% nan 0 0 0
MOT16-12 0.0% 0.0% nan% nan% 0.0% 0 0 0 0 4734 0 0 0 -inf% nan 0 0 0
MOT16-14 0.0% 0.0% nan% nan% 0.0% 0 0 0 0 7451 0 0 0 -inf% nan 0 0 0
OVERALL 0.0% 0.0% nan% nan% 0.0% 0 0 0 0 94840 0 0 0 -inf% nan 0 0 0
I ran the yolo_deepsort_eval.py, but only got the result of FP. would you please tell me what went wrong? thanks a lot!

hi, I got the same result as u. Have u solved the problem?

Dear @ZQPei
I evaluated the performance using MOT16 training datasets and got the following result, just wondering is it reasonable?
IDF1 IDP IDR Rcll Prcn GT MT PT ML FP FN IDs FM MOTA MOTP IDt IDa IDm
MOT16-02 24.9% 54.0% 16.2% 25.2% 83.7% 54 5 19 30 873 13348 70 190 19.9% 0.316 75 13 19
MOT16-04 40.3% 65.2% 29.1% 37.8% 84.6% 83 6 42 35 3279 29586 124 600 30.6% 0.292 100 30 16
MOT16-05 35.4% 44.9% 29.3% 54.8% 84.0% 125 27 65 33 712 3079 64 155 43.5% 0.300 138 5 82
MOT16-09 40.6% 51.6% 33.4% 58.5% 90.6% 25 6 17 2 321 2180 59 94 51.3% 0.271 59 4 10
MOT16-10 32.0% 46.2% 24.5% 41.9% 79.0% 54 9 21 24 1376 7151 132 321 29.7% 0.304 132 15 32
MOT16-11 36.7% 46.6% 30.3% 57.6% 88.7% 69 13 23 33 672 3888 35 87 49.9% 0.257 55 5 26
MOT16-13 23.5% 42.5% 16.3% 25.5% 66.5% 107 6 40 61 1472 8530 131 379 11.5% 0.352 146 22 47
OVERALL 34.8% 54.8% 25.5% 38.6% 83.0% 517 72<PHONE_NUMBER> 67762 615 1826 30.2% 0.295 705 94 232
The MOTA is between 11.5% (MOT16-13) to 51.3%(MOT16-09). I'm using yolov3.weights. The highest performance was relatively close to the deepsort author claimed 61.4%. However, the MOTP value is much worse than the author claimed 79.1%. here I got 0.352. Wondering is it correct?
I noticed many people ask for the original MOT16 videos to test. I downloaded the videos from their website but found the result quite strange(many result are either nan or negative). Then I found these videos have been resized to half width and half height. I therefore convert the image frames downloaded from MOT16 dataset into videos and use these videos as input for our program and got the answer as above.

I get similar results.The MOTA and MOTP are much worse than the author's.Have you found out the reason？
THANKS

Dear @ZQPei
I evaluated the performance using MOT16 training datasets and got the following result, just wondering is it reasonable?
IDF1 IDP IDR Rcll Prcn GT MT PT ML FP FN IDs FM MOTA MOTP IDt IDa IDm
MOT16-02 24.9% 54.0% 16.2% 25.2% 83.7% 54 5 19 30 873 13348 70 190 19.9% 0.316 75 13 19
MOT16-04 40.3% 65.2% 29.1% 37.8% 84.6% 83 6 42 35 3279 29586 124 600 30.6% 0.292 100 30 16
MOT16-05 35.4% 44.9% 29.3% 54.8% 84.0% 125 27 65 33 712 3079 64 155 43.5% 0.300 138 5 82
MOT16-09 40.6% 51.6% 33.4% 58.5% 90.6% 25 6 17 2 321 2180 59 94 51.3% 0.271 59 4 10
MOT16-10 32.0% 46.2% 24.5% 41.9% 79.0% 54 9 21 24 1376 7151 132 321 29.7% 0.304 132 15 32
MOT16-11 36.7% 46.6% 30.3% 57.6% 88.7% 69 13 23 33 672 3888 35 87 49.9% 0.257 55 5 26
MOT16-13 23.5% 42.5% 16.3% 25.5% 66.5% 107 6 40 61 1472 8530 131 379 11.5% 0.352 146 22 47
OVERALL 34.8% 54.8% 25.5% 38.6% 83.0% 517 72<PHONE_NUMBER> 67762 615 1826 30.2% 0.295 705 94 232
The MOTA is between 11.5% (MOT16-13) to 51.3%(MOT16-09). I'm using yolov3.weights. The highest performance was relatively close to the deepsort author claimed 61.4%. However, the MOTP value is much worse than the author claimed 79.1%. here I got 0.352. Wondering is it correct?
I noticed many people ask for the original MOT16 videos to test. I downloaded the videos from their website but found the result quite strange(many result are either nan or negative). Then I found these videos have been resized to half width and half height. I therefore convert the image frames downloaded from MOT16 dataset into videos and use these videos as input for our program and got the answer as above.

I get similar results.The MOTA and MOTP are much worse than the author's.Have you found out the reason？
THANKS

MOTA

14/5000
The authors used this motmetrics(python library) as an evaluation indicator.
MOTP Is described as
Metric MOTP seems to be off. To convert compute (1 - MOTP) * 100. MOTChallenge benchmarks compute MOTP as percentage, while py-motmetrics sticks to the original definition of average distance over number of assigned objects [1].
I think so

you can look at my blog：https://blog.csdn.net/dbdxwyl/article/details/118308750

Is there any FPS requirements when converting pictures to video？i got the video from those images but the are nan or 0.0% too.

MOT16-01 0.0% 0.0% nan% nan% 0.0% 0 0 0 0 3620 0 0 0 -inf% nan 0 0 0
MOT16-03 0.0% 0.0% nan% nan% 0.0% 0 0 0 0 54495 0 0 0 - inf% nan 0 0 0
MOT16-06 0.0% 0.0% nan% nan% 0.0% 0 0 0 0 9982 0 0 0 -inf% nan 0 0 0
MOT16-07 0.0% 0.0% nan% nan% 0.0% 0 0 0 0 7471 0 0 0 -inf% 나노 0 0 0
MOT16-08 0.0% 0.0% 나노% 나노% 0.0% 0 0 0 0 7087 0 0 0 -inf% 나노 0 0 0
MOT16-12 0.0% 0.0% 나노% 나노 % 0.0% 0 0 0 0 4734 0 0 0 -inf% nan 0 0 0
MOT16-14 0.0% 0.0% nan% nan% 0.0% 0 0 0 0 7451 0 0 0 -inf% nan 0 0 0
전체 0.0% 0.0 % nan% nan% 0.0% 0 0 0 0 94840 0 0 0 -inf% nan 0 0 0
yolo_deepsort_eval.py를 실행했는데 FP라는 결과만 나왔다. 무엇이 잘못되었는지 말씀해 주시겠습니까? 정말 감사합니다!

안녕, 나는 u와 같은 결과를 얻었습니다. 문제를 해결하셨나요?

Hello, did you solve this problem? Could you tell me your way to solve?


Fix mapped memory image data update issue

The problem
Some applications update data of images which is in mapped memory. For such application, it shows corruption if replaying the captured trace file with -m realign option.
The solution
The root cause of the problem is related process doesn't have the handling for updating mapped memory image data. The commit added the handling.
Result
Tested the target application with the build of the pull request on the same or different GPU/Driver system with -m realign option, it fixed the issue.

CI gfxreconstruct build queued with queue ID 179073.

CI gfxreconstruct build # 4102 running.

CI gfxreconstruct build # 4102 passed.

The rest looks good to me.

CI gfxreconstruct build queued with queue ID 255783.

CI gfxreconstruct build # 4826 running.

CI gfxreconstruct build # 4826 passed.


A Suggestion for Adding a Screenshot Feature to BingGPT

Thanks for your work!
Could you please add a new feature that supports taking a screenshot of the entire chat page and generating a downloadable image?
Because there are many chrome extensions that enable this feature, I wonder if BingGPT can also add this feature. This feature would be helpful for recording and sharing conversations. I would appreciate it if you could consider adding this feature.
Thank you for your time and attention.

Thanks for your advice. The screenshot feature will be launched in the next version. However, since Bing has set a strict Content Security Policy in the header of this page, it will be very troublesome to execute other JS codes on this page, so the current screenshot function will only call the Electron API to screenshot the entire window, and cannot get entire conversation.

BingGPT 0.1.7 has been released, which supports exporting full conversation as image.


Add docs for defining upper-case variables on nested Settings models

Change Summary
The reason explained here: https://github.com/samuelcolvin/pydantic/issues/3936#issuecomment-1152903692
Related issue number
Add docs for #3936
Checklist

[ ] Unit tests for the changes exist
[ ] Tests pass on CI and coverage remains at 100%
[ ] Documentation reflects the changes where applicable
[ ] changes/<pull request or issue id>-<github username>.md file added describing change
(see changes/README.md for details)
[x] My PR is ready to review, please add a comment including the phrase "please review" to assign reviewers

please review

please update

Thanks @hramezani 🚀


Threads domains

There should be an addition of the domains for Threads by Meta Platforms (formerly Facebook, Inc) in these https://github.com/jmdugan/blocklists/tree/master/corporations/facebook lists.

Do you know of any other than threads.net?

Do you know of any other than threads.net?
Sorry, no, I was hoping to acquire that information through this project actually, guessing that someone involved in it had already set up some methodology for extracting domains associated with organizations through e.g. the WHOIS-database and/or DNS or such.

Reopen with additional domain names if you would like to add them to the list.


Toggle Word wrap doesn't work on mac os when cursor is on text

Does this issue occur when all extensions are disabled?: Yes/No

VS Code Version:
OS Version: Mac OS Big Sur 11.3.1

Steps to Reproduce:

Focus cursor on text in editor
[alt]z gives print ˀ

Let's track in https://github.com/microsoft/vscode/issues/41024 . The root cause is a well known Chromium issue -- https://bugs.chromium.org/p/chromium/issues/detail?id=887519


TextureSystem option "flip_t" will flip vertical direction of texture lookups

This is helpful for renderers that adhere to the convention that the t=0 texture coordinate is the visible "bottom" of the texture when displayed.

In the udim case, where does the flip occur? Per tile?

Yeah, udim resolution happens before the coordinate flip. So effectively, it's flipping the orientation of the texture within each file. but the tile IDs remain the same (that's what pairs the texture file with the geometry parts).

That is, when both this change and the udim one are merged. They are actually orthogonal, the merges don't conflict on any line!

I could finally test the PR (sorry for the delay). It works.
Thanks!

Hi, just to clarify, are you referring to just the flip_t stuff, or did you also try out the UDIM patch?

Hi, just to clarify, are you referring to just the flip_t stuff, or did you also try out the UDIM patch?
I tried both together, I merged the PRs locally. The flip_t works. Our test suite passes.
I also tested UDIM and it seems correct, but I don't have any complex scene to test with,
only simple things and only using the UDIM tag.

Awesome!
I'll merge both, then. If there are situations where the UDIM work is less than ideal, we can always adjust it later.

Shouldn't the results returned in dresultdt also be inverted if flip_t is true?

We flip the input dtdx and dtdy (see lines 1010). Does that not achieve this properly? No, I suppose not, since the computation of daccumdt (in the sample functions) doesn't appear to consider the sign of the input derivatives.
I believe you are correct. Stay tuned, I'll submit a fix today.

Fix for the bug @oliverbjames pointed out in #1462

That was quick!
Thanks, Larry


[pio][stm32f103c8] Cannot compile, ezTime.h:70:2: error: redeclaration of 'ERROR'

Hello !
I am trying to use the library on an stm32f103c8 (Bluepill) stm board using PlatformIO, it seems that the simple fact of including the library makes the compilation to fail because of a redefinition of the ERROR enum.
How to reproduce
platformio.ini:
[env:bluepill_f103c8]
platform = ststm32
board = bluepill_f103c8
framework = arduino
upload_protocol = stlink
build_flags = -D EZTIME_ETHERNET
lib_deps =
    Ethernet
    ezTime

src/main.cpp:
#include <Arduino.h>
#include <ezTime.h>
#include <Ethernet.h>

void setup() {
}

void loop() {
}

Compilation log

Compilation log

<command-line>:0:0: note: this is the location of the previous definition
In file included from src/main.cpp:8:0:
/home/thomas/.platformio/lib/ezTime_ID5751/src/ezTime.h:70:2: error: redeclaration of 'ERROR'
  ERROR,
  ^~~~~
In file included from /home/thomas/.platformio/packages/framework-arduinoststm32/cores/arduino/stm32/stm32_def.h:28:0,
                 from /home/thomas/.platformio/packages/framework-arduinoststm32/cores/arduino/stm32/clock.h:43,
                 from /home/thomas/.platformio/packages/framework-arduinoststm32/cores/arduino/wiring_time.h:23,
                 from /home/thomas/.platformio/packages/framework-arduinoststm32/cores/arduino/wiring.h:38,
                 from /home/thomas/.platformio/packages/framework-arduinoststm32/cores/arduino/Arduino.h:32,
                 from src/main.cpp:1:
/home/thomas/.platformio/packages/framework-arduinoststm32/system/Drivers/CMSIS/Device/ST/STM32F1xx/Include/stm32f1xx.h:167:3: note: previous declaration 'ErrorStatus ERROR'
   ERROR = !SUCCESS
   ^~~~~
In file included from /home/thomas/.platformio/lib/ezTime_ID5751/src/ezTime.cpp:3:0:
/home/thomas/.platformio/lib/ezTime_ID5751/src/ezTime.h:70:2: error: redeclaration of 'ERROR'
  ERROR,
  ^~~~~
In file included from /home/thomas/.platformio/packages/framework-arduinoststm32/cores/arduino/stm32/stm32_def.h:28:0,
                 from /home/thomas/.platformio/packages/framework-arduinoststm32/cores/arduino/stm32/clock.h:43,
                 from /home/thomas/.platformio/packages/framework-arduinoststm32/cores/arduino/wiring_time.h:23,
                 from /home/thomas/.platformio/packages/framework-arduinoststm32/cores/arduino/wiring.h:38,
                 from /home/thomas/.platformio/packages/framework-arduinoststm32/cores/arduino/Arduino.h:32,
                 from /home/thomas/.platformio/lib/ezTime_ID5751/src/ezTime.cpp:1:
/home/thomas/.platformio/packages/framework-arduinoststm32/system/Drivers/CMSIS/Device/ST/STM32F1xx/Include/stm32f1xx.h:167:3: note: previous declaration 'ErrorStatus ERROR'
   ERROR = !SUCCESS
   ^~~~~
/home/thomas/.platformio/lib/ezTime_ID5751/src/ezTime.cpp: In function 'void {anonymous}::triggerError(ezError_t)':
/home/thomas/.platformio/lib/ezTime_ID5751/src/ezTime.cpp:51:45: warning: comparison between 'enum ezDebugLevel_t' and 'enum ErrorStatus' [-Wenum-compare]
  #define err(args...)   if (_debug_level >= ERROR) _debug_device->print(args)

[.... more warnings ....]

Do you have any idea how to fix that ? If you need more info feel free to ask !
Cheers

Not sure why this was closed. I have the same error on an Arduino Portenta, which is also ststm32.


Unable to build PotreeConverter

How to use Web-LOD-Converter instead of the PotreeConverter, could you please share the detail process？

Hello! PotreeConverter is designed to convert our Lidar-scanned point cloud into a multi-resolution structure for training. Meanwhile, Web-LOD-Converter is employed to convert the training outputs (such as level_0.ply, level_1.ply,...) into the octree.bin format for web rendering. We have tested building PotreeConverter in Windows; however, Linux has not been tested yet. We will carry out the test of PotreeConverter in Linux.

Thank you for your patient reply, there is indeed a problem compiling PotreeConverter on linux, looking forward to the author updating the version running on linux

have you solved the problem,I have the same problem.

Thank you for your patient reply, there is indeed a problem compiling PotreeConverter on linux, looking forward to the author updating the version running on linux

I also encountered the same problem on linux. Have you solved this problem?


api: migrate cron delete to sqlalchemy

[!WARNING]
This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

#133
#132
#131  👈
#130
#123 : 1 other dependent PR (#124 )
#122
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @isaacharrisholt and the rest of your teammates on  Graphite


The base sample project is invalid after the background

Hello,
In the base sample project, after about five counter increments in the background of the application, the subsequent notification bar will not be incremented and there will be no print

The permission isIgnoringBatteryOptimizations  is true, but the app still doesn't work in the background。

Hello, @willbetheone
Did you add the REQUEST_IGNORE_BATTERY_OPTIMIZATIONS permission in the manifest?
<manifest>
    <!-- Add permissions -->
    <uses-permission android:name="android.permission.REQUEST_IGNORE_BATTERY_OPTIMIZATIONS" />
</manifest>

If the battery optimization ignore request prompt does not appear, you can try directly request as follows:
Future<void> _requestPermissions() async {
   // This function requires `android.permission.REQUEST_IGNORE_BATTERY_OPTIMIZATIONS` permission.
   await FlutterForegroundTask.requestIgnoreBatteryOptimization();
}

There may be restrictions on background services depending on the device manufacturer. Please let me know the devices you tested.
And please check if it works correctly in the simulation as well.

Hello, @Dev-hwang
It's a pleasure to receive your message. I did the test demo on OnePlus 9R.

Hello , @Dev-hwang
I just tried it on the simulator and it worked fine.
Thanks for the reply.
Next steps
For the OnePlus 9R mobile phone, additional permissions are required. How can we handle it better? Do you have any good suggestions?
-- Zander

@willbetheone
This issue seems to be related to the manufacturer of OnePlus devices.
I couldn't find a way to programmatically go to the specific settings(power consumption controls) page. It seems to be a settings page for OnePlus devices.
so, it seems necessary to provide a guide for OnePlus users to go to the settings page and enable background.
I'm sorry that I couldn't be more helpful 😢


Add Robert Szwajkowski to Contributors list

Hi  @bszwajk, I'm quite elated about your pull request. I wanna evolve this project to addresses various problems faced by first-time contributors. I'd love to learn about your journey in open source community, the problems, pain points you had etc.
Could you explain how you felt when you went through the tutorial, made a pull request and learned that I merged it?
We’ve recently added social share  to our web app. Could you please go to https://roshanjossey.github.io/first-contributions/#social-share and share your first contribution to open source?
Also, check out projects with easy issues while you’re there.

Hello, @Roshanjossey,
I felt pretty comfortable going through the tutorial. I'm very much used to the command line and I have some prior familiarity with git from a few simple personal projects.
The hard part for me which has yet to be addressed is jumping into doing some actual open source work. I'd love to get involved with a project and I'm confident in my abilities as a developer, but social anxiety sucks. This was a good step 0 though so I have some idea of what I'm doing when I decide to stop dragging my feet. So, thank you for your project. :)

I’d urge you to continue contributing to other open source projects.
We also have a slack team to help/mentor people trying to contribute to open source projects.
You can join the team by going to https://goo.gl/PM1Jdh


Make sure examples are consistent between languages

Description
We should make sure we have the same examples for all languages and also that the have a similar structure. This makes it easier to use the same docs for the same example and also makes sure we try to support all languages equivalently. It may even help with DX as we can easily see if some examples are to verbous.
Are you planning to do it yourself in a pull request?
Maybe.

Started creating a list, but I think we should delay this until we have a basic version of the new docs running. As we the example structure will change


Remove version: CrystalDewWorld.CrystalDiskInfo.ShizukuEdition version 8.12.12

[Automated] It returns code over 400 in all urls
Microsoft Reviewers: Open in CodeFlow

URL: https://osdn.net/frs/redir.php?m=ipconnect&f=crystaldiskinfo%2F76210%2FCrystalDiskInfo8_12_12Shizuku.exe
Response Error: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.
(Automated message - build 923)


Document dependencies

Thanks for this. Just a note, this seems to depend on  qt6-base-dev (ubuntu) / qt6-qtbase-devel (fedora).
Also might be helpful to just point people to an install.sh like this, that creates the systemd service too:
#!/bin/bash

# Variables
SERVICE_NAME="fakwin.service"
SERVICE_DIR="$HOME/.config/systemd/user"
SERVICE_PATH="$SERVICE_DIR/$SERVICE_NAME"
EXECUTABLE="$HOME/.local/bin/fakwin"

# Build the binary
mkdir -p build
cd build
cmake ..
make
mkdir -p "$(dirname "$EXECUTABLE")"
mv fakwin "$EXECUTABLE"

# Ensure the service directory exists
mkdir -p "$SERVICE_DIR"

# Create the service file
cat << EOF > "$SERVICE_PATH"
[Unit]
Description=Run fakwin binary to fix shutdowns on custom WM plasma.

# For running before shutdown
DefaultDependencies=no
Before=shutdown.target reboot.target halt.target poweroff.target

[Service]
ExecStart=$EXECUTABLE
Restart=always
Type=simple

[Install]
# run before shutdown
WantedBy=shutdown.target
EOF

# Reload systemd to recognize the new service
systemctl --user daemon-reload
# Enable the service to start on boot or before shutdown
systemctl --user enable "$SERVICE_NAME"
# Start the service immediately (optional)
systemctl --user start "$SERVICE_NAME"

Hey, thanks for the point about dependencies. I'll add a note for the same. Thank you for the script as well, but I would rather let people choose how they want to run this script (whether they want to use a systemd service or something else).
And in any case, this is a just a temporary fix. I would rather wish that users complain to the upstream, urging them to fix it, than make the hacky fix a simple click-to-install solution. :)

Fair point!


Particle flickering issue

You can see how the particles are bugging out here, and go black and flicker. This happens with spawners to
https://user-images.githubusercontent.com/63996164/172275422-05b4d158-aa00-463a-85f2-82c2b710ffb9.mp4
forge: 40.1.25
Dynamic lights reforged: v1.3.3

I have the same issue. When breaking blocks, there are often black particles. The issue is very similar to this thread: https://forums.minecraftforge.net/topic/107173-black-particles-when-breaking-blocks-and-slight-lag-as-it-calculates-the-black-particles/ - but the issue was Jade in this post. Jade works fine for me, but after testing, I found out that this Mod was the problem. I removed it, and it works, however I really like the mod and hope this will be fixed.

Seems that my issue was related to Oculus


throw error java.io.FileNotFoundException: res/drawable-ldpi/title_bar_shadow.9.png

What steps will reproduce the problem?
1. I create a android application use android 2.2
2. I copy all resources from CatalogActivity to my application
3. I copy QuickActionActivity from CatalogActivity to my application
4. I modified some configure to make it compiled correctly
What is the expected output? What do you see instead?
when I start the application , I can see the ActionBar when I click the button.
but the application cann't start,error: "Caused by:       java.io.FileNotFoundException: res/drawable-   ldpi/title_bar_shadow.9.png"
What version of the product are you using? On what operating system?
version:  android:versionCode="1"  android:versionName="1.0"
operating system: windows xp

The problem is you forgot to read the README file ... You are not using GreenDroid and integrating it in your project correctly.

And for those who read the README file and still is getting the error?

Just answering my own question: my problem was in the Manifest.
I wasn't setting the GreenDroid Theme on my GDActivity, instead of that i was using Theme.NoTitleBar.
Now everything is ok.


Choice of num_warps

I wonder why you use 32 (256 threads/block instance) here instead of deciding based on hidden size? Thanks.
https://github.com/linkedin/Liger-Kernel/blob/dd86cbd2092177681acf75643ded1b23a785a816/src/liger_kernel/ops/fused_linear_cross_entropy.py#L95

Another thing to note with this, is that this hardcoded num_warps of 32 causes errors when trying to run on AMD Instinct accelerators, since they have a warp size of 64 rather than 32, and this ends up exceeding the max threads per block:
https://github.com/triton-lang/triton/issues/4128#issuecomment-2200944647
https://github.com/linkedin/Liger-Kernel/issues/231

Yeah I think this needs some more adaptive tuning. Though Triton generally seems to give good performance with brute-force parameters

32 is the best tuned value i found for llama. did you observe better performance number if you use a different number?

Did you choose this based on hidden dim? Thanks.

The hardcoded num_warps of 32 causes this error on AMD Instinct accelerators, which I managed to avoid on my end by reducing it to 16 instead. I was subsequently able to full finetune Qwen2.5 72B at 32k ctx length and Mistral Large 123B at 16k ctx on 8xMI300X with all Liger Kernel optimizations enabled.
Perhaps it would be possible to automatically detect when an AMD Instinct accelerator is in use and adjust num_warps to a suitable value accordingly? However I'm not sure if this reduced value of 16 is optimal performance-wise on these GPUs.

It seems the the block size/vocab(32k or 64k at max) size will be much larger than whatever number of threads you have, so better set num_warps to the maximum available on your hardware?

makes sense. feel free to send a pr for AMD!

It seems the the block size/vocab size(32k or 64k at max) will be much larger than any number of threads you have, so better set num_warps to the max available on your hardware?

I found that in ROCm/vllm and vllm fork, the restrict the num_warps to be either 4 or 8 in actual kernel, so maybe these numbers can be a reference to tune the num_warps on AMD.
https://github.com/search?q=repo%3AROCm%2Fvllm+num_warp&type=code&p=3


Use of container secrets needs amending

Problem Statement
The section about container secrets in readme.md might need updating. The way Docker Engine (tested with 20.10.12 on Ubuntu 20.04 LTS) handles secrets is different from podman (tested with podman 3.3.1 on Oracle Linux 8.5).
Docker Engine
For secrets to be available you need to initialise a (single node) Docker Swarm. Otherwise it's not possible to create the secret:
$ sudo docker secret create oracle-passwd ~/.pwd
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.

Once the swarm is initialised it is possible to add the secret:
$ sudo docker swarm init
Swarm initialized: current node (1rzudangjm68u30vx51227zf6) is now a manager.

To add a worker to this swarm, run the following command:

    docker swarm join --token SWMTKN-1-5ezmfxh6tzi7v4789vi0l8gjq2m1n1zzyiuluets9cawph849w-0soqtexxpbd3blrsygp7qchpi <IP_ADDRESS>:2377

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

Now it's possible to create a secret:
$ sudo docker secret create oracle-passwd ~/.pwd
dkh6i7frg9cstd8zn7izflbjm
$ sudo docker secret ls
ID                          NAME            DRIVER    CREATED          UPDATED
dkh6i7frg9cstd8zn7izflbjm   oracle-passwd             11 seconds ago   11 seconds ago

Without passing the secret on the command line /run/secrets/* does not contain any data and the container fails to start:
$ sudo docker run --name some-oracle -e ORACLE_PASSWORD_FILE=/run/secrets/oracle-passwd --rm -it gvenzl/oracle-xe:21-slim
CONTAINER: starting up...
/opt/oracle/container-entrypoint.sh: line 70: /run/secrets/oracle-passwd: No such file or directory

The error message in container-entrypoint.sh refers to a missing /run/secrets/oracle-passwd file
Unfortunately it seems impossible to pass the secret to docker run:
$ sudo docker run --name some-oracle -e ORACLE_PASSWORD_FILE=/run/secrets/oracle-passwd --rm -it --secret oracle-passwd gvenzl/oracle-xe:21-slim
unknown flag: --secret
See 'docker run --help'.

Passing secrets requires a service to be started:
$ sudo docker service create --name some-oracle -e ORACLE_PASSWORD_FILE=/run/secrets/oracle-passwd --secret oracle-passwd gvenzl/oracle-xe:21-slim
6qkawq9wg3r25uv6ashoajzs1
overall progress: 1 out of 1 tasks
1/1: running   [==================================================>]
verify: Service converged
$ sudo docker service ls
ID             NAME          MODE         REPLICAS   IMAGE                      PORTS
6qkawq9wg3r2   some-oracle   replicated   1/1        gvenzl/oracle-xe:21-slim

$ sudo docker container exec $(sudo docker ps --filter name=some-oracle -q) ls -l /run/secrets
total 4
-r--r--r-- 1 root root 7 Dec 14 15:09 oracle-passwd

Podman
The use of podman greatly simplifies matters. It is possible to create a secret without the need of any auxiliary construct. A simple podman secret create some-secret ~/.pwd will do. The secret can be passed as an argument to podman run as in
$ podman run --name some-oracle \
  --secret oracle-password \
  -e ORACLE_PASSWORD_FILE=/run/secrets/oracle-password \
  -d docker.io/gvenzl/oracle-xe:21-slim

Unless the secret is passed to either Docker service or podman it won't be accessible in /run/secrets/*

Hi @martincarstenbach,
Thanks a lot for this great write-up on how to use secrets with Docker. After some research, I found that the different containerization technologies use rather different approaches for handling secrets (Docker, Podman, Kubernetes).
I feel that the ReadMe for these images on how to use them is not the right place to educate people on how the secrets mechanism in all the different containerization technologies works. Also because Docker Hub has a 20k character limit for the ReadMe as I found out some months ago. I have therefore opted for linking to the official documentation of said three containerization technologies above, rather than documenting the details on how to create and use secrets for all three technologies.
Thx,


Update monitor-instances-health-check.md

0-100 should be 1-100 and also the text in sync (missed that part at last edit)

@msangapu-msft
Can you review the proposed changes?
When the changes are ready for publication, add a #sign-off comment to signal that the PR is ready for the review team to merge.
#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team

#sign-off


Banish the evil wizard Sebastian

The new exporter has weakened him. All we need to break his curse once and for all is a simple deep equals comparator. Hear my call, knight @jmalloc, and aid me in this noble task!
Deep within the labyrinth of Phony™, in a crypt known only as "EqualToMatcher", lies a scrawl of cursed runes:
        $comparator = $this->comparatorFactory
            ->getComparatorFor($this->value, $value);

        try {
            $comparator->assertEquals($this->value, $value);
        } catch (ComparisonFailure $e) {
            return false;
        }

        return true;

A brave hero need only make it to the crypt, and replace the runes with a spell of his own design, and the evil wizard's bond will be broken. Will you be the hero this land needs?

@jmalloc jam pls

How do I compared web?


[PE-D][Tester E] UG: In the User Guide there is an extra empty line after features in table of contents.

Extra empty line after the Features bullet point makes the User Guide look slightly less neat.

Labels: severity.VeryLow type.DocumentationBug
original: zlch/ped#2

Can be fixed


Problem with AllWidgets_30.wxg

When I try to show design window in wxglade with AllWidgets_30.wxg using python 3.6  wxglade crashes. The AllWidgets_28.wxg works.

What OS and wxPython version do you have?
On Windows with 4.0.1 things seem to work fine.

Windows 10, python 3.6 from Anaconda, wxpython 3.0.  Do I need to upgrade to wxpython 4.01?
Thanks,
Scott
From: Dietmar Schwertberger<EMAIL_ADDRESS>Sent: Tuesday, April 17, 2018 2:02 PM
To: wxGlade/wxGlade
Cc: Alchemist45; Author
Subject: Re: [wxGlade/wxGlade] Problem with AllWidgets_30.wxg (#316)
What OS and wxPython version do you have?
On Windows with 4.0.1 things seem to work fine.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub https://github.com/wxGlade/wxGlade/issues/316#issuecomment-382105387 , or mute the thread https://github.com/notifications/unsubscribe-auth/AOdUsPYBBpE35_63SQzAkZMVE96i-BADks5tpjwhgaJpZM4TY23r .  https://github.com/notifications/beacon/AOdUsJamMMWt0d-UnMifT-PO1_6sYtZTks5tpjwhgaJpZM4TY23r.gif

This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

Yes, please try with 4.0.1.
Usually wxPython 3 does not work with Python 3.6.
Either this is a pre-release of wxPython Phoenix when it had the 3 revision number from wxWidgets or it's something very, very special to Anaconda.
The wxPython Phoenix version number was set to 4.x only very late to differentiate it from Classic which is typically 2.8.x or 3.0.2.

Thanks.  The AllWidgets_30.wsg now displays and outputs useful python code that runs just fine.
Regards,
Alchemist45
From: Dietmar Schwertberger<EMAIL_ADDRESS>Sent: Tuesday, April 17, 2018 2:57 PM
To: wxGlade/wxGlade
Cc: Alchemist45; Author
Subject: Re: [wxGlade/wxGlade] Problem with AllWidgets_30.wxg (#316)
Yes, please try with 4.0.1.
Usually wxPython 3 does not work with Python 3.6.
Either this is a pre-release of wxPython Phoenix when it had the 3 revision number from wxWidgets or it's something very, very special to Anaconda.
The wxPython Phoenix version number was set to 4.x only very late to differentiate it from Classic which is typically 2.8.x or 3.0.2.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub https://github.com/wxGlade/wxGlade/issues/316#issuecomment-382121742 , or mute the thread https://github.com/notifications/unsubscribe-auth/AOdUsO7mqzjQv1D04KuZ81SGsG5IV4oTks5tpkkIgaJpZM4TY23r .  https://github.com/notifications/beacon/AOdUsCAwKzeOC4vAZQ5wSeeOQmYRSjAAks5tpkkIgaJpZM4TY23r.gif

This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

Fine. Please give feedback if you observe issues in the documentation or program.


A couple of nitpicks (bundled together so as to not flood the tracker with minor issues)

The material icons that were recently added to the mix (as mentioned here) are not really fit for dark themed toolbars, because they're very dark themselves. It would be best if they were remade into something more theme-agnostic, like the input mode icons currently are (which fit in both light and dark themes due to their opaque orange background).

In the list of includes for IBus provided in src/BUILD.ibus, the directory "lib/glib-2.0/include/**" should be added as it is used by the GLib package on Arch Linux (and possibly in other distros as well) for the glibconfig.h header.

Hi Alexander,
Thank you for the feedback. Please feel free to file an issue per topic. We also prefer one issue per one topic.
1. font
7654223979067e3c5ae52ea238d43880c5508f85 adds variant icons under white/ and outlined/.
If they don't work, please add screenshots of your environment.
2. iBus build
We only support the Docker environment, so it is probably necessary to modify some build files for other environments as you mentioned.
We updated the document with a list of files to be probably modified.
https://github.com/google/mozc/blob/master/docs/build_mozc_in_docker.md#build-mozc-on-other-linux-environment

To be honest, I didn't expect such a quick response to such minor issues so I didn't treat this as a proper "bug" report but rather as a kind of a to-do memo. My fault, I know, so I'll refrain from doing it in the future.

font
The new icons are usable in a dark environment, and especially the outlined ones can fit nicely with both light and dark backgrounds, so they can serve as the default. Thanks. For future reference, here's a a screenshot of how they look like on my system:

iBus build
Ah, I wasn't aware that the supported way of building Mozc is in a Docker container. OK, I understand. For the record though, the only modifications needed for building on an Arch system was the Qt path in config.bzl and the glib path in BUILD.ibus. The former was explicitly mentioned in a comment as being configured "for Debian" and thus in need of manual configuration on other systems, while the latter was not, so I thought it was an omission. The new documentation piece looks perfectly adequate.

iBus build

Thanks for the background. I added comments to BUILD.ibus and other BUILD files for third party libraries. I hope it helps too.


amplify pull with parameters behind a proxy failed.

Before opening, please confirm:

[X] I have installed the latest version of the Amplify CLI (see above), and confirmed that the issue still persists.
[X] I have searched for duplicate or closed issues.
[X] I have read the guide for submitting bug reports.
[X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.

How did you install the Amplify CLI?
npm
If applicable, what version of Node.js are you using?
14.18.0
Amplify CLI Version
6.3.1
What operating system are you using?
Amazon Linux 2(Docker Desktop for Windows - wsl2)
Amplify Categories
interactions
Amplify Commands
pull
Describe the bug
To get the remote environment, I ran the following amplify pull(with parameters) command behind a proxy.
This command has returned error. The result was that below stack.
$ env | grep -iE "^http[s]"
http_proxy=http://xxx.xxx.xxx:3128
https_proxy=http://xxx.xxx.xxx:3128
HTTP_PROXY=http://xxx.xxx.xxx:3128
HTTPS_PROXY=http://xxx.xxx.xxx:3128

$ amplify pull --appId xxxxxx --envName dev
Failed to pull the backend.
request to https://e7auv6no3g.execute-api.us-east-1.amazonaws.com/wave3Prod/AppState/?appId=xxxxxx failed, reason: connect ECONNREFUSED <IP_ADDRESS>:443
FetchError: request to https://e7auv6no3g.execute-api.us-east-1.amazonaws.com/wave3Prod/AppState/?appId=xxxxxx failed, reason: connect ECONNREFUSED <IP_ADDRESS>:443
    at ClientRequest.<anonymous> (/usr/lib/node_modules/@aws-amplify/cli/node_modules/node-fetch/lib/index.js:1483:11)
    at ClientRequest.emit (events.js:400:28)
    at ClientRequest.emit (domain.js:475:12)
    at TLSSocket.socketErrorListener (_http_client.js:475:9)
    at TLSSocket.emit (events.js:400:28)
    at TLSSocket.emit (domain.js:475:12)
    at emitErrorNT (internal/streams/destroy.js:106:8)
    at emitErrorCloseNT (internal/streams/destroy.js:74:3)
    at processTicksAndRejections (internal/process/task_queues.js:82:21)

However, amplify init and amplify pull without parameters works.

But execute amplify init and amplify pull without parameters in Interactive works fine.
$ amplify pull
? Select the authentication method you want to use: (Use arrow keys)
❯ AWS profile
  AWS access keys

GraphQL schema(s)
No response
Log output

# Put your logs below this line

Additional information
Looking the error log, this error occurred at calling request by node-fetch.
    at ClientRequest.<anonymous> (/usr/lib/node_modules/@aws-amplify/cli/node_modules/node-fetch/lib/index.js:1483:11)

I found the location of the node-fetch call that was causing the error when amplify pull(with parameters) and applied the following patch.
$ diff -u /usr/lib/node_modules/@aws-amplify/cli/node_modules/amplify-provider-awscloudformation/lib/utils/admin-helpers.js{.original,}
---<EMAIL_ADDRESS>2021-10-24 22:47:37.009666700 +0000
+++<EMAIL_ADDRESS>2021-10-24 22:49:02.369666700 +0000
@@ -44,10 +44,12 @@
     return await getAdminStsCredentials(idToken, region);
 }
 exports.getTempCredsWithAdminTokens = getTempCredsWithAdminTokens;
+const hpa = __importDefault(require("https-proxy-agent"));
+const hpa2 = new hpa.default("http://xxx.xxx.xxx.xxx:3128");
 async function getAdminAppState(appId, region) {
     var _a;
     const appStateBaseUrl = (_a = process.env.AMPLIFY_CLI_APPSTATE_BASE_URL) !== null && _a !== void 0 ? _a : exports.adminBackendMap[region].appStateUrl;
-    const res = await node_fetch_1.default(`${appStateBaseUrl}/AppState/?appId=${appId}`);
+    const res = await node_fetch_1.default(`${appStateBaseUrl}/AppState/?appId=${appId}`, {agent: hpa2});
     return res.json();
 }
 async function getAdminCognitoCredentials(idToken, identityId, region) {
@@ -186,4 +188,4 @@
         appStateUrl: 'https://3ne6skqg0g.execute-api.us-west-2.amazonaws.com/wave4Prod',
     },
 };

and execute amplify pull --appId xxxxxx --envName dev command then it works.
? Select the authentication method you want to use: (Use arrow keys)
❯ AWS profile
  AWS access keys

Do you have the HTTP_PROXY or HTTPS_PROXY environment variable configured?

Thanks @cjihrig.
Yes, own container environment variables is below.
$ env | grep -iE http[s]*
http_proxy=http://xxx.xxx.xxx:3128
https_proxy=http://xxx.xxx.xxx:3128
HTTP_PROXY=http://xxx.xxx.xxx:3128
HTTPS_PROXY=http://xxx.xxx.xxx:3128

@tamutamu
Hello, Thank you for working for this issue. The PR has merged so I suppose this issue was solved.
Is there any reason for this issue is still opened?


Count months correctly

https://crt.sh/?id=13469453&opt=cablint
"ERROR: BR certificates must be 39 months in validity or less"
This certificate is valid from the last day of month 1 (Feb 29th) until the penultimate day of month 39 (May 30th).  Therefore, its validity period does not exceed 39 months.
This bit in cablint.rb seems to be the problem:
if c.not_after.day > c.not_before.day
months += 1
end
I think it should consider the number of days left in the month after ".day" instead of the integer value of ".day".

I would argue that 1 month after February 29 would be March 29, not March 30 or March 31.

I guess you can also look at this the reverse way, and that 1 month before March 30 is February 29.
But I think the right way to look at it it to start from the start date and add the 39 months, and not try to subtract the 2 dates.

Oracle's ADD_MONTHS function works like this:
https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions004.htm
"If date is the last day of the month or if the resulting month has fewer days than the day component of date, then the result is the last day of the resulting month. Otherwise, the result has the same day component as date."
Since the BRs don't currently mandate a particular method of counting months, I reckon that any popular method should be considered valid.

This should be fixed in 710234c22ddcad71be4d6e701cc66ed7638ea89c.  certlint now uses days rather than months and the number of days is calculated to be the maximum number that would ever make up x months.


Support searchable ChangeVersion

Coverage decreased (-0.02%) to 68.963% when pulling 1649e8f0228d37684a81fb65fc796121677c5bac on versionsa into 39599a25dac2cfec6d222c96162464187d87b600 on master.


fix: Tier module fixes and improvements

Description
This PR aims to add a number of improvements and features to the Tier module, as well as to fix multiple bugs in the existing logic.
Tasks

[x] Implement CancelUnlocking keeper and msg server.
[x] Implement LockupKeyToAddressesAtHeight and UnlockingLockupKey to store unlocking lockups as delAddr/valAddr/creationHeight/.
[x] Implement MustIterateUnlockingLockups to iterate unlocking lockups.
[x] Add GetLockupAmount and removeLockup.
[x] Fix issues with the existing Unlock logic and use TokensFromSharesTruncated for the unlocking lockup amount calculation to prevent discrepancies in shares calculations.
[x] Fix issues with the existing CompleteUnlocking logic.
[x] Fix issues with the existing GetAllLockups logic.
[x] Add distribution keeper to the Tier module and expected keepers.
[x] Add creation_height field to UnlockingLockupRequest.
[x] Improve the test TierKeeper implementation.
[x] Add tests for Lock, Unlock, Redelegate, CompleteUnlocking, CancelUnlocking, and MintCredit.
[x] Add mocks for all expected keepers.
[x] Add mock tests for Lock, Unlock, and Redelegate.
[x] Add CreateTestValidator and CreateTestEncodingConfig utils for testing.
[x] Add tests for MustIterateUnlockingLockups and IterateLockups.
[x] Document changes to RewardRates Rate param to make it clear that we use integers to represent percentages with 2 decimal precision (e.g. a rate of 150 represents 1.50).

Changes to be included in the upcoming PR:

Prorate credit logic improvements and related tests
Events for all tier msgs
E2e tests for the lock, unlock, complete/cancel unlocking flow
Changes related to custom rewards distribution and sliding scale APY
Slashing logic improvements for the developer stake and related tests

Codecov Report
Attention: Patch coverage is 58.70647% with 166 lines in your changes missing coverage. Please review.

Project coverage is 4.79%. Comparing base (e5f5cfe) to head (534237d).

Files with missing lines
Patch %
Lines

api/sourcehub/tier/v1beta1/query.pulsar.go
2.04%
48 Missing :warning:

x/tier/keeper/keeper.go
62.38%
30 Missing and 11 partials :warning:

x/tier/types/keys.go
0.00%
28 Missing :warning:

testutil/keeper/tier.go
88.67%
8 Missing and 4 partials :warning:

x/tier/keeper/lockup.go
83.56%
11 Missing and 1 partial :warning:

x/tier/types/query.pb.gw.go
0.00%
12 Missing :warning:

x/tier/keeper/msg_server.go
0.00%
8 Missing :warning:

x/tier/keeper/credit.go
76.92%
3 Missing :warning:

x/tier/keeper/grpc_query.go
66.66%
1 Missing :warning:

x/tier/module/module.go
0.00%
1 Missing :warning:

Additional details and impacted files
@@           Coverage Diff            @@
##             dev      #8      +/-   ##
========================================
+ Coverage   4.48%   4.79%   +0.30%
========================================
  Files        211     211
  Lines      90595   90927     +332
========================================
+ Hits        4067    4359     +292
- Misses     86313   86322       +9
- Partials     215     246      +31

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Lagging Replica can Become Master, Leading to Split-Brain

mysql-operator version: docker.io/bitpoke/mysql-operator:v0.6.1
In one of our MySQL clusters operated by mysql-operator, we lost several days of data, because a lagging replica (limesurvey-mysql-2) became the master. The replica was lagging, because it didn't replicate anymore. However, somehow this replica later became the new master. The lost data exists on the old master (limesurvey-mysql-1), but not on the new master (limesurvey-mysql-2), i.e. we have a split-brain.
From the log
2022-11-24T14:19:45.473746Z 39031 [Note] 'CHANGE MASTER TO FOR CHANNEL '' executed'. Previous state master_host='limesurvey-mysql-1.mysql.limesurvey', master_port= 3306, master_log_file='mysql-bin.000008', master_log_pos= 16830944, master_bind=''. New state master_host='_', master_port= 3306, master_log_file='', master_log_pos= 4, master_bind=''.

it seems to me that the master role changed at this point (2022-11-24T14:19:45).
Due to this incident, I wanted to ask:

What are the rules regarding which replica can become the master?
Is there any mechanism in place to ensure that a replica that doesn't have all transactions cannot become the master?
Does such a mechanism exist as an optional feature? If yes, how to enable it?
Is it possible to enforce synchronous replication, to make sure there's always at least one replica that is up-to-date with the current master?

Maybe this can be mitigated by configuring a non-zero value for FailMasterPromotionOnLagMinutes in the orchestrator (because zero here means infinity). How to do this through mysql-operator?


Fix performance regression in block_executor

How to make your PRs look good: Introduce a performance regression without benchmarking (#4301), followed by another PR which improves performance a bit (#4536).
The improvements in #4536 were real, but ignore that #4301 introduced a regression. The regression was caused by the block_executor previously relying on thread executors which eagerly created threads for bulk execution (run_now = true), whereas the new underlying executor restricted_thread_pool_executor only creates task descriptions. In addition the restricted_thread_pool_executor did hierarchical spawning. Both of these seem to have caused the performance regression (fixing just one of them is not enough). This is based on the stream benchmark and @m-diers observations on his application.
In this PR I've extended the block_executor constructor to take priority, stacksize, and hint to be passed on to the underlying executor. The default priority is high, which forces immediate thread creation, and thus restores the old behaviour (mostly, the old threads were normal priority). I've also removed the hierarchical spawning from the restricted_thread_pool_executor.
The question is if this (controlling thread creation with the priority) nicely models what we want. I think it is a reasonable design as I wouldn't want to expose something like the run_now parameter to users. I also wouldn't want to force bulk execution to always create threads. The notion of a tight numerical parallel for loop like those in the stream benchmark being high priority feels more or less right (we want all of the for loop tasks to be executed asap without being interrupted by otherwise spawned regular tasks).
This is a draft as I want to redo the benchmarks in #4301 and #4536.

Finally some benchmarks (and with this I think this is ready to be merged modulo linting; I've rebased on the latest master):
Stream
For stream we were doing well with block executor in 1.4.1 until I made it worse on master. This branch basically restores the old performance for stream.
Large arrays
1.4.1 block_executor
Memory per array = 781.25 MiB (= 0.762939 GiB).
Function    Best Rate MB/s  Avg time     Min time     Max time
Copy:           76450.7     0.021574     0.021431     0.023405
Scale:          75844.3     0.021736     0.021602     0.022370
Add:            86259.7     0.028615     0.028491     0.029447
Triad:          87665.2     0.028129     0.028034     0.029529

This branch block_executor
Function    Best Rate MB/s  Avg time     Min time     Max time
Copy:           76818.0     0.021468     0.021328     0.022025
Scale:          75794.6     0.021769     0.021616     0.022544
Add:            86359.0     0.028640     0.028458     0.029616
Triad:          87852.2     0.028096     0.027974     0.028775

All of these numbers are mostly within noise thresholds. No progress, but no regression either.
Small arrays
1.4.1 block_executor
Memory per array = 0.078125 MiB (= 7.62939e-05 GiB).
Function    Best Rate MB/s  Avg time     Min time     Max time
Copy:             439.3     0.000460     0.000373     0.002166
Scale:            431.8     0.000462     0.000379     0.001871
Add:              665.4     0.000466     0.000369     0.002544
Triad:            670.9     0.000447     0.000366     0.002006

This branch block_executor
Memory per array = 0.078125 MiB (= 7.62939e-05 GiB).
Function    Best Rate MB/s  Avg time     Min time     Max time
Copy:             698.9     0.000331     0.000234     0.001073
Scale:            695.0     0.000334     0.000236     0.001046
Add:             1024.0     0.000333     0.000240     0.001153
Triad:           1011.8     0.000327     0.000243     0.001027

Here there's a small improvement (big in the worst case times; average/min improvement is not as dramatic, but still better), but in general HPX does badly with small arrays.
Sorry I don't have actual graphs for this. That would be something nice to have in the future to compare to other stream benchmarks.
Future overhead
This is a bit more interesting. I added hierarchical spawning to the thread_pool_executor without thinking much about the threshold at which it actually does hierarchical spawning. I made it do hierarchical spawning if there are at least two chunks for a worker thread. I've updated the default threshold to be 6 based measurements on daint, but it's also customizable now as the best value is likely to change from system to system. With 6 it'll do hierarchical spawning for half of the tasks on average (the default static chunk size creates 4-8 chunks per worker thread). With this most of the funny behaviour in https://github.com/STEllAR-GROUP/hpx/pull/4301#issuecomment-575627988 is gone.


Space

pair testing and coding for space class

Looks good, love the name of the slot instance variable.


Update deprecated playground function

XCPSetExecutionShouldContinueIndefinitely() should now be XCPlaygroundPage.currentPage.needsIndefiniteExecution = true as the former was deprecated.

Thanks for the PR @paulthorsteinson. Rather than merging this in, I removed the playground completely in 7582758d due to instability reasons. Thanks again.


Apb-base now uses the ansible-running base image

https://github.com/ansibleplaybookbundle/apb-base/pull/36
Changes:

playbooks need to be moved from /opt/apb/actions to /opt/apb/projects
copy inventory to /opt/apb/inventory/hosts

@rthallisey i see the PR was merged. anything else to do here?


Can't connect to local image remote

I was trying to copy an image from the local remote to a remote host with and got back the following error:
error: Get https://<IP_ADDRESS>:8443/1.0/images/793122da8bfc432366179bf38bda488750e41e9d9499d85e3343f62b60dbf16c?secret=6dddf908dfd433a087fc0ea528e94126c8df87e49d9b7993269a9b905b6b235d: Unable to connect to: <IP_ADDRESS>:8443
The URL for the local remote is listed as: unix://
curl -s -k --cert ~/.config/lxc/client.crt --key ~/.config/lxc/client.key https://<IP_ADDRESS>:8443/1.0 | jq .metadata.auth
returns: "trusted"
lxc launch <image> <remote>: results in the same error
Does the local remote have to be visible publicly to copy images from it and launch containers from images on it even if the copy and launch command are processed on it?
Local Config
  driver: lxc
  driverversion: 2.0.0
  kernel: Linux
  kernelarchitecture: x86_64
  kernelversion: 4.4.0-22-generic
  server: lxd
  serverpid: 14451
  serverversion: 2.0.0
  storage: zfs
  storageversion: "5"
config:
  core.https_address: '[::]:8443'
  core.trust_password: true
  storage.zfs_pool_name: lxd
public: false

Remote Config
  driver: lxc
  driverversion: 2.0.0
  kernel: Linux
  kernelarchitecture: x86_64
  kernelversion: 4.4.0-22-generic
  server: lxd
  serverpid: 4197
  serverversion: 2.0.0
  storage: zfs
  storageversion: "5"
config:
  core.https_address: '[::]:8443'
  core.trust_password: true
  storage.zfs_pool_name: lxd
public: false

Debug Info
lxc image copy local:<image> <remote>:

DBUG[05-10|17:53:22] Raw response: {"type":"sync","status":"Success","status_code":200,"metadata":{"api_extensions":[],"api_status":"stable","api_version":"1.0","auth":"trusted","config":{"core.https_address":"[::]:8443","core.trust_password":true,"storage.zfs_pool_name":"lxd"},"environment":{"addresses":["<IP_ADDRESS>:8443","[2605:e000:304d:ef00:842:2220:e21f:f9d8]:8443","[2605:e000:304d:ef00:3c4e:f4e1:1de6:ef13]:8443","[2605:e000:304d:ef00:97ef:c925:f365:54cb]:8443","<IP_ADDRESS>:8443","<IP_ADDRESS>:8443","[fd43:697c:1d49:de2::1]:8443"],"architectures":["x86_64","i686"],"certificate":"-----BEGIN CERTIFICATE----------END CERTIFICATE-----\n","driver":"lxc","driver_version":"2.0.0","kernel":"Linux","kernel_architecture":"x86_64","kernel_version":"4.4.0-22-generic","server":"lxd","server_pid":14451,"server_version":"2.0.0","storage":"zfs","storage_version":"5"},"public":false}}

DBUG[05-10|17:53:22] Raw response: {"type":"sync","status":"Success","status_code":200,"metadata":{"name":"studentfiles-pro","description":"studentfiles-pro","target":"793122da8bfc432366179bf38bda488750e41e9d9499d85e3343f62b60dbf16c"}}

DBUG[05-10|17:53:22] Raw response: {"type":"sync","status":"Success","status_code":200,"metadata":{"aliases":[{"name":"studentfiles-pro","description":"studentfiles-pro"}],"architecture":"x86_64","cached":false,"filename":"","fingerprint":"793122da8bfc432366179bf38bda488750e41e9d9499d85e3343f62b60dbf16c","properties":{"description":"student files prodution application","version":"20160510"},"public":false,"size":314738043,"auto_update":false,"created_at":"0001-01-01T00:00:00Z","expires_at":"0001-01-01T00:00:00Z","last_used_at":"2016-05-10T21:20:11.231338639Z","uploaded_at":"2016-05-10T13:46:19-07:00"}}

DBUG[05-10|17:53:22] Posting null
 to http://unix.socket/1.0/images/793122da8bfc432366179bf38bda488750e41e9d9499d85e3343f62b60dbf16c/secret
DBUG[05-10|17:53:22] Raw response: {"type":"async","status":"Operation created","status_code":100,"metadata":{"id":"6bcd804f-47d6-4a8d-adca-d5333ef6cd52","class":"token","created_at":"2016-05-10T17:53:22.616748006-07:00","updated_at":"2016-05-10T17:53:22.616748006-07:00","status":"Running","status_code":103,"resources":{"images":["/1.0/images/793122da8bfc432366179bf38bda488750e41e9d9499d85e3343f62b60dbf16c"]},"metadata":{"secret":"bc451f284382b63f505b98e6fed2c643d93fb86718b94d89bbe141b0836a4258"},"may_cancel":true,"err":""},"operation":"/1.0/operations/6bcd804f-47d6-4a8d-adca-d5333ef6cd52"}

DBUG[05-10|17:53:22] Raw response: {"type":"sync","status":"Success","status_code":200,"metadata":{"api_extensions":[],"api_status":"stable","api_version":"1.0","auth":"trusted","config":{"core.https_address":"[::]:8443","core.trust_password":true,"storage.zfs_pool_name":"lxd"},"environment":{"addresses":["<IP_ADDRESS>:8443","[2605:e000:304d:ef00:842:2220:e21f:f9d8]:8443","[2605:e000:304d:ef00:3c4e:f4e1:1de6:ef13]:8443","[2605:e000:304d:ef00:97ef:c925:f365:54cb]:8443","<IP_ADDRESS>:8443","<IP_ADDRESS>:8443","[fd43:697c:1d49:de2::1]:8443"],"architectures":["x86_64","i686"],"certificate":"-----BEGIN CERTIFICATE----------END CERTIFICATE-----\n","driver":"lxc","driver_version":"2.0.0","kernel":"Linux","kernel_architecture":"x86_64","kernel_version":"4.4.0-22-generic","server":"lxd","server_pid":14451,"server_version":"2.0.0","storage":"zfs","storage_version":"5"},"public":false}}

DBUG[05-10|17:53:22] Posting {"auto_update":false,"public":false,"source":{"certificate":"-----BEGIN CERTIFICATE----------END CERTIFICATE-----\n","fingerprint":"793122da8bfc432366179bf38bda488750e41e9d9499d85e3343f62b60dbf16c","mode":"pull","protocol":"","secret":"bc451f284382b63f505b98e6fed2c643d93fb86718b94d89bbe141b0836a4258","server":"https://<IP_ADDRESS>:8443","type":"image"}}
 to https://ec2-<ipaddress>.us-west-1.compute.amazonaws.com:8443/1.0/images
DBUG[05-10|17:53:22] Raw response: {"type":"async","status":"Operation created","status_code":100,"metadata":{"id":"93ad7f48-fb4a-4c1a-8dc6-5e3167081c1b","class":"task","created_at":"2016-05-11T00:53:22.810371664Z","updated_at":"2016-05-11T00:53:22.810371664Z","status":"Running","status_code":103,"resources":null,"metadata":null,"may_cancel":false,"err":""},"operation":"/1.0/operations/93ad7f48-fb4a-4c1a-8dc6-5e3167081c1b"}

DBUG[05-10|17:53:22] 1.0/operations/93ad7f48-fb4a-4c1a-8dc6-5e3167081c1b/wait
DBUG[05-10|17:53:42] Raw response: {"type":"sync","status":"Success","status_code":200,"metadata":{"id":"93ad7f48-fb4a-4c1a-8dc6-5e3167081c1b","class":"task","created_at":"2016-05-11T00:53:22.810371664Z","updated_at":"2016-05-11T00:53:22.810371664Z","status":"Failure","status_code":400,"resources":null,"metadata":null,"may_cancel":false,"err":"Get https://<IP_ADDRESS>:8443/1.0/images/793122da8bfc432366179bf38bda488750e41e9d9499d85e3343f62b60dbf16c?secret=bc451f284382b63f505b98e6fed2c643d93fb86718b94d89bbe141b0836a4258: Unable to connect to: <IP_ADDRESS>:8443"}}

error: Get https://<IP_ADDRESS>:8443/1.0/images/793122da8bfc432366179bf38bda488750e41e9d9499d85e3343f62b60dbf16c?secret=bc451f284382b63f505b98e6fed2c643d93fb86718b94d89bbe141b0836a4258: Unable to connect to: <IP_ADDRESS>:8443

That would be because your remote host is unable to connect to <IP_ADDRESS>:8443 to fetch the image.

So either make the local visible publicly or export from local and import
into remote.
It would be nice to be able to copy and launch pushing from local if you're
on the local machine.
On May 10, 2016 6:25 PM, "Stéphane Graber"<EMAIL_ADDRESS>wrote:

That would be because your remote host is unable to connect to
<IP_ADDRESS>:8443 to fetch the image.
—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
https://github.com/lxc/lxd/issues/2014#issuecomment-218338477

Yep, that's #553

Super, thx
On May 10, 2016 7:20 PM, "Stéphane Graber"<EMAIL_ADDRESS>wrote:

Yep, that's #553 https://github.com/lxc/lxd/issues/553
—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
https://github.com/lxc/lxd/issues/2014#issuecomment-218345081


metrics: add Memstats runtime metrics

This includes metrics on things like GC pause times. The metric calculation comes from http://pythonic.zoomquiet.io/data/20131112090955/index.html, but I'm not 100% sure I got everything translated correctly.

Lovely!


ShuffledDataset's get() called twice per iteration in BatchDataloaderIterator

Describe the bug
get() for ShuffledDataset is called twice for each self.dataset.get(self.current_index) in BatchDataloaderIterator
This method is called twice:
https://github.com/tracel-ai/burn/blob/b429cc39c19780e561b697f9b970c04c29969fa5/crates/burn-dataset/src/transform/random.rs#L44-L50
, when this method is called:
https://github.com/tracel-ai/burn/blob/b429cc39c19780e561b697f9b970c04c29969fa5/crates/burn-core/src/data/dataloader/batch.rs#L161
I add two info!() in the get() method of random.rs and dataset.rs. The number of log of the former is twice of the later.
2024-03-06T18:29:35.381089Z  INFO burn_dataset::transform::random: type of self.dataset: "alloc::sync::Arc<dyn burn_dataset::dataset::base::Dataset<regression::dataset::DiabetesItem>>"
2024-03-06T18:29:35.381105Z  INFO burn_dataset::transform::random: origial index: 0 new index: 120
2024-03-06T18:29:35.381109Z  INFO regression::dataset: get 120
2024-03-06T18:29:35.381111Z  INFO burn_dataset::transform::random: type of self.dataset: "burn_dataset::dataset::sqlite::SqliteDataset<regression::dataset::DiabetesItem>"
2024-03-06T18:29:35.381114Z  INFO burn_dataset::transform::random: origial index: 120 new index: 411
2024-03-06T18:29:35.381138Z  INFO burn_core::data::dataloader::batch: current index: 1
2024-03-06T18:29:35.381142Z  INFO burn_dataset::transform::random: type of self.dataset: "alloc::sync::Arc<dyn burn_dataset::dataset::base::Dataset<regression::dataset::DiabetesItem>>"
2024-03-06T18:29:35.381145Z  INFO burn_dataset::transform::random: origial index: 1 new index: 12
2024-03-06T18:29:35.381147Z  INFO regression::dataset: get 12
2024-03-06T18:29:35.381149Z  INFO burn_dataset::transform::random: type of self.dataset: "burn_dataset::dataset::sqlite::SqliteDataset<regression::dataset::DiabetesItem>"
2024-03-06T18:29:35.381152Z  INFO burn_dataset::transform::random: origial index: 12 new index: 262
2024-03-06T18:29:35.381167Z  INFO burn_core::data::dataloader::batch: current index: 2
2024-03-06T18:29:35.381170Z  INFO burn_dataset::transform::random: type of self.dataset: "alloc::sync::Arc<dyn burn_dataset::dataset::base::Dataset<regression::dataset::DiabetesItem>>"
2024-03-06T18:29:35.381173Z  INFO burn_dataset::transform::random: origial index: 2 new index: 81
2024-03-06T18:29:35.381176Z  INFO regression::dataset: get 81
2024-03-06T18:29:35.381178Z  INFO burn_dataset::transform::random: type of self.dataset: "burn_dataset::dataset::sqlite::SqliteDataset<regression::dataset::DiabetesItem>"
2024-03-06T18:29:35.381180Z  INFO burn_dataset::transform::random: origial index: 81 new index: 72
2024-03-06T18:29:35.381196Z  INFO burn_core::data::dataloader::batch: current index: 3
2024-03-06T18:29:35.381205Z  INFO burn_dataset::transform::random: type of self.dataset: "alloc::sync::Arc<dyn burn_dataset::dataset::base::Dataset<regression::dataset::DiabetesItem>>"
2024-03-06T18:29:35.381208Z  INFO burn_dataset::transform::random: origial index: 3 new index: 50
2024-03-06T18:29:35.381211Z  INFO regression::dataset: get 50
2024-03-06T18:29:35.381213Z  INFO burn_dataset::transform::random: type of self.dataset: "burn_dataset::dataset::sqlite::SqliteDataset<regression::dataset::DiabetesItem>"
2024-03-06T18:29:35.381216Z  INFO burn_dataset::transform::random: origial index: 50 new index: 169

To Reproduce

Expected behavior
It should only be called once.
Screenshots

Desktop (please complete the following information):

OS: [e.g. iOS]
Browser [e.g. chrome, safari]
Version [e.g. 22]

Smartphone (please complete the following information):

Device: [e.g. iPhone6]
OS: [e.g. iOS8.1]
Browser [e.g. stock browser, safari]
Version [e.g. 22]

Additional context
My guess is the get() is binded to the ShuffledDataset itself. And the self.dataset.get(*index) call itself again.

This issue is a bit old sorry, was just going through the issues and stumbled upon it.
It seems you are using the simple regression example, which uses the ShuffledDataset.
When using a dataloader with a random seed for shuffling, the provided dataset is also wrapped by a ShuffledDataset.
The symptom you're observing is just that the first ShuffledDataset::get(index) call comes from the dataloader, which calls the regression dataset's get(index), which in turn calls the self.dataset.get(index) (which is a ShuffledDataset).
The number of items actually retrieved by the dataset should still correspond to its length, it's just the debug info you added that seems to be leading you down the wrong path 🙂
If you really think there is a bug somewhere, please let us know. Otherwise I will close this issue.

Thanks for that clarification. I have refactored my code a lot, so it's not issue for me now. I close it.


ui enhancement of become a member page

Describe the feature
there are 2 different colors then the whole background is blank which is not looking good i will try to fix it
Add ScreenShots

Before

After
Record

[X] I agree to follow this project's Code of Conduct
[X] I'm a GSSOC'24 contributor
[X] I want to work on this issue

Do you need someone to fix this issue or what?

Do you need someone to fix this issue or what?

No i have already completed this issue under gssoc'24


Inspector causes huge lags in large scenes after 4.1.stable

Godot version
v4.1.stable.mono.official [970459615]
System information
Godot v4.1.stable.mono - Windows 10.0.19045 - Vulkan (Forward+) - dedicated NVIDIA GeForce GTX 1050 Ti (NVIDIA; <IP_ADDRESS>79) - AMD Ryzen 3 3200G with Radeon Vega Graphics (4 Threads)
Issue description
I have a scene with a large amount of InstanceMesh3D, StaticBody3D which are in a Map node. This scene does not cause any lags for me, but as soon as I expand the Map node, my scene starts to lag and it is impossible to work with the project at all. Before that I moved my project from 4.0.2.stable to 4.1.stable. The previous version did not cause such problems.
Steps to reproduce
Create a scene of over +500 MeshInstance3D with any shape that is in a Node3D node. Try to open and close Node3D and you will see the performance drop.
Minimal reproduction project
N/A

Can you please upload an MRP so that other contributors can attempt to reproduce the issue? Without an MRP, it is unlikely anyone will be able to help you

Sure, here's my project

I can't speak for any problem in the Inspector, since this is a very simple scene, composition wise, just Node3Ds. But the scene tree starts to lag like hell the more down you scroll (and gets back to snappy once you scroll back to top). Not sure if this didn't happen in 4.0.3, checking that now. But it's pretty bad regardless.

I tried the above MRP on 4.0.1 and 4.1, on my pc (Arch Linux, 1080Ti)
There is a noticeable lag in 4.1 which does not exist in 4.0.1 (which doesn't make it impossible to edit, but lower hardware)

The issue is not related to the editor, any Tree control is affected. Here's one with 10000 items, that exhibits same behavior:
GUITree.zip

I can't speak for any problem in the Inspector, since this is a very simple scene, composition wise, just Node3Ds. But the scene tree starts to lag like hell the more down you scroll (and gets back to snappy once you scroll back to top). Not sure if this didn't happen in 4.0.3, checking that now. But it's pretty bad regardless.
Edit: Can confirm that 4.0.3 is buttery smooth throughout the whole tree.

Thanks for confirming that this is a bug. By the way, is it possible to find the commit that changed the rendering or loading of nodes in the scene tree?

@dalexeev From the behavior and from the changeset of your PR, I think the issue is rooted in the fact that we now need to compute something about every preceding tree item, whereas before we would ignore items which are not currently on screen.
Which is admittedly a pretty typical issue for this kind of control and this kind of situation. Since offset is uneven now, cannot be calculated by using the number of items before the first visible one, we need to cache something.

After investigating the profiler data, I tentatively think we should try caching Tree::get_column_minimum_width(). I will try to do this tomorrow.

After investigating the profiler data, I tentatively think we should try caching Tree::get_column_minimum_width(). I will try to do this tomorrow.

Thank you so much. How I can get this fix or wait untill 4.2.devx version?

How I can get this fix or wait untill 4.2.devx version?

I opened PR #79325 which you can test. Yes, this is a change for 4.2, but if the PR is merged and considered safe enough, it will be cherry-picked for a 4.1.x maintenance release.


[Autocomplete] Add hint demo

preview:  https://deploy-preview-37496--material-ui.netlify.app/material-ui/react-autocomplete#hint
https://github.com/mui/material-ui/assets/60743144/8ed102c5-cb1c-4aba-9e26-2035a4f6b087

[x] I have followed (at least) the PR section of the contributing guide.

@siriwatknp Tried to add same demo to joy Autocomplete but somehow i couldn't able to get it working. Here is the sandbox if you want to try it out.
https://codesandbox.io/s/smoosh-snowflake-w9xn7e?file=/demo.js

@siriwatknp Tried to add same demo to joy Autocomplete but somehow i couldn't able to get it working. Here is the sandbox if you want to try it out.
https://codesandbox.io/s/smoosh-snowflake-w9xn7e?file=/demo.js

For Joy UI, you have to create a custom wrapper component and replace the default one. Here https://codesandbox.io/s/cool-shannon-5lzl7m?file=/demo.js