Improve UI-thread animation performance on busy screens (~10x reduction depending on your use-case)

This change can be a significant reduction in UI thread load for animations when a controller drives many animations that will come in and go out of existance. 100x are easily possible depending on how many things are popping on and off the screen (e.g. hundreds).
WARNING: This is a somewhat breaking change (as shown by the test-change), due to listeners only being notified once (in first-insertion order) rather than once every registration. I'd be more than happy to make the behavior configurable on the AnimationController, so that folks could opt into the new behavior or construct an AnimationController depending on what their use-case demands.
Tangentially, I'd also be interested in understanding the motivation for the implemented behavior a bit better. Given that listeners can be mutating, preserving the order seems quite intuitive. Whereas multiple registrations may make sense, I just can't think of a good use-case. I can easily think of unintentional gotchas like advancing an animation multiple times per step/frame. (EDIT: it's also surprising that despite its ordering, it's not a stack, i.e. you push new listeners to the end but you remove them from the front,which makes working with multiple registrations conceptually probably hard to work with :woman_shrugging: )
CONTEXT: I'm using a library that draws markers onto a map and then depending on the zoom-level it will cluster (collapse/expand) these markers. This results often in tens or hundreds of markers being animated (position & opacity) with a single AnimationController. After this animation no-longer visible markers will be taken off screen resulting in large numbers of listeners being removed. With the implementation at HEAD, my App is spending a majority of its UI-thread time (tens of milliseconds per frame) just deleting listeners. I'd argue that a generic tool like AnimationController should not micro-optimize for the case N=1.
I also did a performance comparison based on N to support the proposal. I tried to model realistic use cases, i.e. add/remove, notifiyListeners() w/o any _set invalidation, with invalidation, and a cycle of notifylisteners/rm/add. The benchmark code can be found here: https://gist.github.com/ignatz/dba6293d0fef41e3c6e1ed3c51364de6 .
2 add/remove                       SAME ObserverList vs MyObserverList                 takes:      1.0x, saved:       -0%   (0:00:01.373926 vs 0:00:01.376916)
2 add/remove                       SAME HashedObserverList vs MyHashedObserverList     takes:      1.0x, saved:       -0%   (0:00:02.170364 vs 0:00:02.177635)
2 add/remove                     SLOWER ObserverList vs MyHashedObserverList           takes:      0.6x, saved:      -64%   (0:00:01.375987 vs 0:00:02.254769)

2 notifyListeners                  SAME ObserverList vs MyObserverList                 takes:      1.0x, saved:        1%   (0:00:01.183753 vs 0:00:01.167889)
2 notifyListeners                FASTER HashedObserverList vs MyHashedObserverList     takes:      1.6x, saved:       38%   (0:00:01.703029 vs 0:00:01.054944)
2 notifyListeners                FASTER ObserverList vs MyHashedObserverList           takes:      1.1x, saved:       10%   (0:00:01.173942 vs 0:00:01.061108)

2 notifyListeners self-mod         SAME ObserverList vs MyObserverList                 takes:      1.0x, saved:        1%   (0:00:00.413579 vs 0:00:00.409012)
2 notifyListeners self-mod       FASTER HashedObserverList vs MyHashedObserverList     takes:      1.2x, saved:       16%   (0:00:00.861495 vs 0:00:00.725754)
2 notifyListeners self-mod       SLOWER ObserverList vs MyHashedObserverList           takes:      0.6x, saved:      -79%   (0:00:00.410618 vs 0:00:00.734759)

2 mix nl/rm/add                  SLOWER ObserverList vs MyObserverList                 takes:      1.0x, saved:       -4%   (0:00:00.382266 vs 0:00:00.398285)
2 mix nl/rm/add                  FASTER HashedObserverList vs MyHashedObserverList     takes:      1.3x, saved:       25%   (0:00:00.696625 vs 0:00:00.522181)
2 mix nl/rm/add                  SLOWER ObserverList vs MyHashedObserverList           takes:      0.8x, saved:      -25%   (0:00:00.363710 vs 0:00:00.454669)

4 add/remove                       SAME ObserverList vs MyObserverList                 takes:      1.0x, saved:       -1%   (0:00:03.210935 vs 0:00:03.240136)
4 add/remove                       SAME HashedObserverList vs MyHashedObserverList     takes:      1.0x, saved:       -2%   (0:00:04.000753 vs 0:00:04.088699)
4 add/remove                     SLOWER ObserverList vs MyHashedObserverList           takes:      0.9x, saved:      -18%   (0:00:03.457464 vs 0:00:04.062586)

4 notifyListeners                SLOWER ObserverList vs MyObserverList                 takes:      0.8x, saved:      -22%   (0:00:01.857049 vs 0:00:02.266333)
4 notifyListeners                FASTER HashedObserverList vs MyHashedObserverList     takes:      1.3x, saved:       21%   (0:00:03.031229 vs 0:00:02.395746)
4 notifyListeners                FASTER ObserverList vs MyHashedObserverList           takes:      1.2x, saved:       14%   (0:00:03.723434 vs 0:00:03.213373)

4 notifyListeners self-mod       SLOWER ObserverList vs MyObserverList                 takes:      0.8x, saved:      -28%   (0:00:03.809080 vs 0:00:04.857361)
4 notifyListeners self-mod       FASTER HashedObserverList vs MyHashedObserverList     takes:      1.3x, saved:       24%   (0:00:03.694135 vs 0:00:02.797687)
4 notifyListeners self-mod       FASTER ObserverList vs MyHashedObserverList           takes:      1.3x, saved:       23%   (0:00:03.691153 vs 0:00:02.850335)

4 mix nl/rm/add                  SLOWER ObserverList vs MyObserverList                 takes:      0.9x, saved:       -6%   (0:00:01.665852 vs 0:00:01.759793)
4 mix nl/rm/add                    SAME HashedObserverList vs MyHashedObserverList     takes:      1.0x, saved:        1%   (0:00:01.430239 vs 0:00:01.422435)
4 mix nl/rm/add                  FASTER ObserverList vs MyHashedObserverList           takes:      2.6x, saved:       62%   (0:00:03.076555 vs 0:00:01.176919)

7 add/remove                     FASTER ObserverList vs MyObserverList                 takes:      1.2x, saved:       16%   (0:00:08.613052 vs 0:00:07.209422)
7 add/remove                       SAME HashedObserverList vs MyHashedObserverList     takes:      1.0x, saved:        1%   (0:00:08.334085 vs 0:00:08.291406)
7 add/remove                       SAME ObserverList vs MyHashedObserverList           takes:      1.0x, saved:       -1%   (0:00:07.939038 vs 0:00:07.982456)

7 notifyListeners                FASTER ObserverList vs MyObserverList                 takes:      1.1x, saved:       11%   (0:00:03.537551 vs 0:00:03.162703)
7 notifyListeners                FASTER HashedObserverList vs MyHashedObserverList     takes:      1.2x, saved:       19%   (0:00:04.266710 vs 0:00:03.441823)
7 notifyListeners                  SAME ObserverList vs MyHashedObserverList           takes:      1.0x, saved:        2%   (0:00:03.562112 vs 0:00:03.489668)

7 notifyListeners self-mod       FASTER ObserverList vs MyObserverList                 takes:      1.3x, saved:       20%   (0:00:07.683331 vs 0:00:06.141342)
7 notifyListeners self-mod       FASTER HashedObserverList vs MyHashedObserverList     takes:      1.1x, saved:        6%   (0:00:02.777192 vs 0:00:02.607791)
7 notifyListeners self-mod       FASTER ObserverList vs MyHashedObserverList           takes:      2.3x, saved:       57%   (0:00:06.138749 vs 0:00:02.623466)

7 mix nl/rm/add                  SLOWER ObserverList vs MyObserverList                 takes:      1.0x, saved:       -4%   (0:00:01.527072 vs 0:00:01.591417)
7 mix nl/rm/add                  FASTER HashedObserverList vs MyHashedObserverList     takes:      1.1x, saved:       10%   (0:00:01.424173 vs 0:00:01.274727)
7 mix nl/rm/add                  FASTER ObserverList vs MyHashedObserverList           takes:      1.8x, saved:       43%   (0:00:01.718774 vs 0:00:00.972161)

10 add/remove                      SAME ObserverList vs MyObserverList                 takes:      1.0x, saved:        1%   (0:00:06.705254 vs 0:00:06.652850)
10 add/remove                    SLOWER HashedObserverList vs MyHashedObserverList     takes:      0.9x, saved:       -8%   (0:00:06.127039 vs 0:00:06.634606)
10 add/remove                    FASTER ObserverList vs MyHashedObserverList           takes:      1.1x, saved:        6%   (0:00:06.621233 vs 0:00:06.242699)

10 notifyListeners               SLOWER ObserverList vs MyObserverList                 takes:      0.8x, saved:      -21%   (0:00:02.542935 vs 0:00:03.065810)
10 notifyListeners               FASTER HashedObserverList vs MyHashedObserverList     takes:      1.2x, saved:       20%   (0:00:03.038072 vs 0:00:02.441563)
10 notifyListeners               FASTER ObserverList vs MyHashedObserverList           takes:      1.1x, saved:       10%   (0:00:02.708075 vs 0:00:02.429304)

10 notifyListeners self-mod      SLOWER ObserverList vs MyObserverList                 takes:      0.6x, saved:      -63%   (0:00:04.636833 vs 0:00:07.569618)
10 notifyListeners self-mod      SLOWER HashedObserverList vs MyHashedObserverList     takes:      0.7x, saved:      -44%   (0:00:01.926254 vs 0:00:02.768144)
10 notifyListeners self-mod      FASTER ObserverList vs MyHashedObserverList           takes:      3.9x, saved:       75%   (0:00:19.460013 vs 0:00:04.935028)

10 mix nl/rm/add                   SAME ObserverList vs MyObserverList                 takes:      1.0x, saved:       -3%   (0:00:03.122031 vs 0:00:03.207405)
10 mix nl/rm/add                 FASTER HashedObserverList vs MyHashedObserverList     takes:      1.1x, saved:       12%   (0:00:02.104239 vs 0:00:01.853132)
10 mix nl/rm/add                 FASTER ObserverList vs MyHashedObserverList           takes:      1.7x, saved:       43%   (0:00:03.167834 vs 0:00:01.817996)

100 add/remove                   FASTER ObserverList vs MyObserverList                 takes:      2.1x, saved:       53%   (0:00:29.435696 vs 0:00:13.740486)
100 add/remove                   SLOWER HashedObserverList vs MyHashedObserverList     takes:      1.0x, saved:       -4%   (0:00:01.810957 vs 0:00:01.874549)
100 add/remove                   FASTER ObserverList vs MyHashedObserverList           takes:      7.0x, saved:       86%   (0:00:13.562779 vs 0:00:01.932154)

100 notifyListeners              FASTER ObserverList vs MyObserverList                 takes:      1.1x, saved:       11%   (0:00:00.824788 vs 0:00:00.735041)
100 notifyListeners              FASTER HashedObserverList vs MyHashedObserverList     takes:      1.1x, saved:       10%   (0:00:00.780762 vs 0:00:00.701437)
100 notifyListeners              FASTER ObserverList vs MyHashedObserverList           takes:      1.1x, saved:        8%   (0:00:00.797545 vs 0:00:00.732023)

100 notifyListeners self-mod     FASTER ObserverList vs MyObserverList                 takes:      1.1x, saved:       11%   (0:00:24.091372 vs 0:00:21.355631)
100 notifyListeners self-mod     FASTER HashedObserverList vs MyHashedObserverList     takes:      1.0x, saved:        4%   (0:00:00.560706 vs 0:00:00.536952)
100 notifyListeners self-mod     FASTER ObserverList vs MyHashedObserverList           takes:     42.2x, saved:       98%   (0:00:22.614892 vs 0:00:00.536188)

100 mix nl/rm/add                  SAME ObserverList vs MyObserverList                 takes:      1.0x, saved:       -1%   (0:00:00.399696 vs 0:00:00.403067)
100 mix nl/rm/add                FASTER HashedObserverList vs MyHashedObserverList     takes:      1.1x, saved:       12%   (0:00:00.168415 vs 0:00:00.148730)
100 mix nl/rm/add                FASTER ObserverList vs MyHashedObserverList           takes:      2.7x, saved:       63%   (0:00:00.395535 vs 0:00:00.145706)

1000 add/remove                  SLOWER ObserverList vs MyObserverList                 takes:      1.0x, saved:       -3%   (0:00:43.742194 vs 0:00:45.164800)
1000 add/remove                    SAME HashedObserverList vs MyHashedObserverList     takes:      1.0x, saved:       -0%   (0:00:00.596296 vs 0:00:00.596309)
1000 add/remove                  FASTER ObserverList vs MyHashedObserverList           takes:     74.4x, saved:       99%   (0:00:44.349347 vs 0:00:00.595768)

1000 notifyListeners               SAME ObserverList vs MyObserverList                 takes:      1.0x, saved:       -3%   (0:00:00.281672 vs 0:00:00.289646)
1000 notifyListeners             FASTER HashedObserverList vs MyHashedObserverList     takes:      1.1x, saved:        7%   (0:00:00.251050 vs 0:00:00.234575)
1000 notifyListeners             FASTER ObserverList vs MyHashedObserverList           takes:      1.2x, saved:       17%   (0:00:00.283716 vs 0:00:00.235989)

1000 notifyListeners self-mod    SLOWER ObserverList vs MyObserverList                 takes:      1.0x, saved:       -3%   (0:01:03.477013 vs 0:01:05.411053)
1000 notifyListeners self-mod    SLOWER HashedObserverList vs MyHashedObserverList     takes:      0.6x, saved:      -78%   (0:00:00.188581 vs 0:00:00.336313)
1000 notifyListeners self-mod    FASTER ObserverList vs MyHashedObserverList           takes:    348.7x, saved:      100%   (0:01:05.049408 vs 0:00:00.186552)

1000 mix nl/rm/add                 SAME ObserverList vs MyObserverList                 takes:      1.0x, saved:       -1%   (0:00:00.123481 vs 0:00:00.124426)
1000 mix nl/rm/add               FASTER HashedObserverList vs MyHashedObserverList     takes:      1.1x, saved:        5%   (0:00:00.050968 vs 0:00:00.048402)
1000 mix nl/rm/add               FASTER ObserverList vs MyHashedObserverList           takes:      2.5x, saved:       60%   (0:00:00.121606 vs 0:00:00.048307)

Pre-launch Checklist

[ ] I read the Contributor Guide and followed the process outlined there for submitting PRs.
[ ] I read the Tree Hygiene wiki page, which explains my responsibilities.
[ ] I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
[ ] I signed the CLA.
[ ] I listed at least one issue that this PR fixes in the description above.
[ ] I updated/added relevant documentation (doc comments with ///).
[ ] I added new tests to check the change I am making, or this PR is test-exempt.
[ ] All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

A flutter-tools profile screenshot from 3.13.9:

@gaaclarke really interesting PR, somehow seems related to some small changes you do that radically improve things.

WARNING: This is a somewhat breaking change (as shown by the test-change)

I am a little careful of changing the behavior here. This is used in lots and lots of places. It looks like the change is not failing any customer tests, which is a good start. I'll see if I can run google's internal tests on this to see if we can get a better understanding of how breaking this would be.

Appreciated. I agree that we need to respect Hyrum's law here.
That said, the implemented semantics seem rather ad-hoc. For multiple registrations are virtually impossible to use correctly especially when listeners mutate the list of listeners. I'm also not convinced that the updated semantics are great. I'd say they're about as bad.
Personally, I'd argue that multiple registrations should be rejected (at least for the animations use-case). For a listener its trivial to run code N times w/o having to register itself N-times. Then again, this discussion probably goes beyond the scope of this PR.

I also did a performance comparison based on N to support the proposal

I scrolled through the data and I can't see a clear pattern. It seems to be a wash, sometimes the new implementation is faster, sometimes its slower. I am actually not sure whether this is a clear win?

Yes and no but really yes :). It's a bit hard to see given the various dimensions and quite a bit of trial-by-trial variation (I'm going to blame GC and my notebook's thermals :) ). Note that when the new cases are flagged as slower, it's only a very small percentage. In most cases it's just the same +/- noise, I merely picked a conservative threshold for labeling as SLOWER vs SAME. In these cases looking at the absolute runtime can also provide a sense of how noisy the measure may be. I did share the benchmark code, so you can simply run it yourself (the script has zero deps).
Let me try to break it down:
There are 3 comparisons:

ObserverList before and after
HasedObserverList before and after
Improved HasedObserverList against original ObserverList (as a before and after of the second commit)

Then we have two more dimensions:
A. N number of elements. It's worth pointing out that ObserverList as implemented behaves entirely differently for N <3 and N >= 3 elements (Otherwise I would have reduced the number of data points)
B. different test cases.
1. & 2.
The changes covered by 1. and 2. are strict improvements. They don't change any semantics and thus I pulled them out into a separate commit. They should be fairly straightforward. the overriden toList is a noticable improvement in the general case over falling back to the Iterator.toList's default implementation. It is used within notifyListeners() to make a copy of he list (to account for listeners which may mutate the list of listeners), so you'll hit the code path on every frame independent of your access patterns. The change to ObserverList is very subtle and only makes a difference if you try to delete missing listeners, which in real world will probably hardly ever make a difference.
I pulled the "toList()" case out into a separate benchmark to make this clearer:
10 toList()                      FASTER HashedObserverList vs MyHashedObserverList     takes:      2.0x, saved:       50%   (0:00:20.598112 vs 0:00:10.332040)
1000 toList()                    FASTER HashedObserverList vs MyHashedObserverList     takes:      1.4x, saved:       28%   (0:00:01.271092 vs 0:00:00.912649)

(With N=1000 probably a lot more noisy than N=10, due to it's significantly shorter runtime. I only wanted to spend so much time tweaking the iteration numbers for every combo)
3.
"3." covers the changes within the second commit, i.e. moving from ObserverList ot HashedObserver list for animations. This is where the real meat is, i.e. 100+x improvements depending on the number of listeners. At the end of the day it's just array vs hash map. No magic involved.
My assumption is that whoever did the current implementation assumed that AnimationControllers will only drive a small, static set of animations. In these cases, sure the array is slighly faster. In the general case: "a changing, not necessarily small set of Animations", the array is atrocious.
You can see that the ObserverList implementation tries to compensate for it managing a separate, potentially stale, hash set.... Unfortunately, it doesn't help for:

Removals, you still need to traverse the entire list and move elements
For mutating listeners it kills the performance on every frame.  When the hash set gets invalidated and notifyListeners may rebuild the hash set many times per frame. It's horrendous.

(Outside the scope of this change but the attempted band aid on ObserverList is dangerous. For certain access patterns it can help. For others it completely destroys performance. It's important to note that the access pattern is user-defined and the punishment is completely opaque to users).
That's where you get numbers like:
1000 add/remove                  FASTER ObserverList vs MyHashedObserverList           takes:     74.3x, saved:       99%   (0:00:45.577018 vs 0:00:00.613640)

1000 notifyListeners self-mod    FASTER ObserverList vs MyHashedObserverList           takes:    357.4x, saved:      100%   (0:01:06.012280 vs 0:00:00.184690)

Generally, I don't think we should get too hung up on the performance differences for small N that are well within the margin of error. At the end of the day, we'd be discussing 10us vs 10.5us on an already cheap operation, when in reality  we're seeing 10us vs 3ms in the N=1000 case.
As personal data point, my specific use-case that surfaced this issue has hundreds of animations popping on and off screen. These gains are not theoretical for me. At this point, I'm actually shipping a forked version of flutter incorporating these changes significantly reducing frame-drop.

Regarding 1 (Changes to ObservableList.remove): In practice, this sounds like regression to me. I would expect that the case where you remove a non-existent listener is very, very rare (I would almost consider it a bug in the code if somebody does that). So, optimizing for that is odd to me - since you will pay for the "optimization" in the far more common case with an additional check.
Regarding 2 (Optimizing HashedObserverList.toList): This does seem like a nice improvement and the benchmarks appear to back that up. We could probably land just this change without much trouble.
Regarding 3 (Changing AnimationLocalListenersMixin): This is actually less risky than I initially thought since it only affects this one mixin. For some reason I initially thought this would effect all ChangeNotifiers. We still need to run this change through our google testing suite to see if this would be a breaking change.
 I'll keep you posted.

Also, can you please resolve the issues reported by the analyze check? Thanks!

Regarding 1 (Changes to ObservableList.remove): In practice, this sounds like regression to me. I would expect that the case where you remove a non-existent listener is very, very rare (I would almost consider it a bug in the code if somebody does that). So, optimizing for that is odd to me - since you will pay for the "optimization" in the far more common case with an additional check.

Maybe all use-cases are buggy (in which case, we should probably surface it, e.g. assert or at least log), maybe there are some valid edge cases :woman_shrugging: . I'm not sure that ignoring it is the best way to deal with it in either case.
Meanwhile, it's the typical insurance-risk-severity evaluation: the check is basically free and the cost independent of the size, whereas invalidating the internal optimization is non-obvious to users and very costly especially for larger sets. In terms of cost, given any modern CPU with speculative execution and accurate branch prediction, I would expect this check to take <1ns :handwave:.
Personally, I can very much live w/o this "optimization", I'm merely surprised by your cost argument in the light of the benchmark results. I could certainly buy in your bug argument, however one should probably look into how to surface such bugs (independent of this change). As I was rambling earlier, I think there are other semantics that might be unintentional like FIFO (rather than LIFO) removals of duplicate listeners. Maybe duplicate listeners in general should be considered a bug? :woman_shrugging:  There would be nice benefits to it too, e.g. one could use cheaper data structure that don't need to keep track of insertion order.

Regarding 2 (Optimizing HashedObserverList.toList): This does seem like a nice improvement and the benchmarks appear to back that up. We could probably land just this change without much trouble.

Agreed, it shouldn't cause any surprises.

Regarding 3 (Changing AnimationLocalListenersMixin): This is actually less risky than I initially thought since it only affects this one mixin. For some reason I initially thought this would effect all ChangeNotifiers. We still need to run this change through our google testing suite to see if this would be a breaking change.

Thanks! Independently, it could be interesting to evaluate if other constructs would also benefit from other data structures but ultimately it will always depend on your use case.

Also, can you please resolve the issues reported by the analyze check? Thanks!

Once I get word from you on how  we will or won't proceed, I'll urgently address any issues.
Thanks!

This is not causing any issues internally at google. I think we should be able to land this if the reported analyzer issues get resolved.

This is not causing any issues internally at google. I think we should be able to land this if the reported analyzer issues get resolved.

Happy to hear. Thanks!

This PR is still failing the analyzer check. Can you please take a look? Thanks.

/b/s/w/ir/x/w/flutter/packages/flutter/lib/src/animation/listener_helpers.dart:94: trailing U+0020 space character
you have a space (" ") at somewhere where it shouldn't be.

Thanks. Analyze seems to pass now and I rebased against latest main

could you add a benchmark for this? It also would be nice to see the performance improvements this PR introduces.

I like the idea. Even more so for continuous performance monitoring. It's probably less likely that someone would stumble upon a benchmark and than decides to just implement their own AnimationListener.
I couldn't find any other micro-benchmarks in this repo. That's also why I hadn't added mine. I did share them above:  https://gist.github.com/ignatz/dba6293d0fef41e3c6e1ed3c51364de6.
I had hoped that this change would be validated by existing e2e benchmarks but if you feel adding micro-benchmarks makes sense we can talk about it. That said, I understand that you folks are busy, and therefore I'd like to avoid expanding this simple change further. It has already been more than an OKR cycle.

If you're looking for microbenchmarks: https://github.com/flutter/flutter/tree/master/dev/benchmarks/microbenchmarks

Added a microbenchmark following the example of dev/benchmarks/microbenchmarks/lib/foundation/change_notifier_bench.dart.
Could you also provide me a link to a dashboard where I can see/monitor the continuous evolution of them microbenchmarks? A Google search didn't bring anything up and I'm wondering how this setup is being put to use.

Ideally the microbenchmark should be merged before the change so we can gauge the performance improvement.
dashboard:
https://flutter-flutter-perf.skia.org/e/?begin=1706138370&end=1706224770&queries=sub_result%3DMatrixUtils_affine_transformPoint_iteration%26test%3Dmicrobenchmarks
It seems there are a couple of omitted type annotations in the benchmark.
   info • Missing type annotation • dev/benchmarks/microbenchmarks/lib/foundation/observer_list_bench.dart:24:10 • always_specify_types
   info • Missing type annotation • dev/benchmarks/microbenchmarks/lib/foundation/observer_list_bench.dart:24:33 • always_specify_types
   info • Missing type annotation • dev/benchmarks/microbenchmarks/lib/foundation/observer_list_bench.dart:25:7 • always_specify_types
   info • Missing type annotation • dev/benchmarks/microbenchmarks/lib/foundation/observer_list_bench.dart:28:11 • always_specify_types
   info • Missing type annotation • dev/benchmarks/microbenchmarks/lib/foundation/observer_list_bench.dart:29:11 • always_specify_types
   info • Missing type annotation • dev/benchmarks/microbenchmarks/lib/foundation/observer_list_bench.dart:39:9 • always_specify_types
   info • Missing type annotation • dev/benchmarks/microbenchmarks/lib/foundation/observer_list_bench.dart:40:14 • always_specify_types
   info • Missing type annotation • dev/benchmarks/microbenchmarks/lib/foundation/observer_list_bench.dart:66:10 • always_specify_types
   info • Missing type annotation • dev/benchmarks/microbenchmarks/lib/foundation/observer_list_bench.dart:66:33 • always_specify_types
   info • Missing type annotation • dev/benchmarks/microbenchmarks/lib/foundation/observer_list_bench.dart:67:7 • always_specify_types
   info • Missing type annotation • dev/benchmarks/microbenchmarks/lib/foundation/observer_list_bench.dart:70:11 • always_specify_types
   info • Missing type annotation • dev/benchmarks/microbenchmarks/lib/foundation/observer_list_bench.dart:71:11 • always_specify_types
   info • Missing type annotation • dev/benchmarks/microbenchmarks/lib/foundation/observer_list_bench.dart:81:9 • always_specify_types
   info • Missing type annotation • dev/benchmarks/microbenchmarks/lib/foundation/observer_list_bench.dart:82:14 • always_specify_types

Ideally the microbenchmark should be merged before the change so we can gauge the performance improvement.

I feel like now you're just pulling my leg after plastering this PR with benchmark results. Surely to gauge the improvement one can simply run it.
BEFORE

flutter: ================ FORMATTED ==============
flutter: notifyListeners:ObserverList:success (1 callbacks): 0.2 µs per iteration
notifyListeners:ObserverList:success (10 callbacks): 5.4 µs per iteration
notifyListeners:ObserverList:success (100 callbacks): 771.6 µs per iteration
notifyListeners:ObserverList:success (500 callbacks): 17936.7 µs per iteration
notifyListeners:ObserverList:fail (1 callbacks): 0.1 µs per iteration
notifyListeners:ObserverList:fail (10 callbacks): 8.5 µs per iteration
notifyListeners:ObserverList:fail (100 callbacks): 990.3 µs per iteration
notifyListeners:ObserverList:fail (500 callbacks): 24984.9 µs per iteration    <<<< !
notifyListeners:HashedObserverList (1 callbacks): 0.3 µs per iteration
notifyListeners:HashedObserverList (10 callbacks): 2.2 µs per iteration
notifyListeners:HashedObserverList (100 callbacks): 25.6 µs per iteration
notifyListeners:HashedObserverList (500 callbacks): 174.8 µs per iteration
notifyListeners:AnimationController (1 callbacks): 0.1 µs per iteration
notifyListeners:AnimationController (10 callbacks): 5.9 µs per iteration
notifyListeners:AnimationController (100 callbacks): 723.1 µs per iteration
notifyListeners:AnimationController (500 callbacks): 16763.6 µs per iteration <<<< !

AFTER

flutter: ================ FORMATTED ==============
flutter: notifyListeners:ObserverList:success (1 callbacks): 0.2 µs per iteration
notifyListeners:ObserverList:success (10 callbacks): 5.1 µs per iteration
notifyListeners:ObserverList:success (100 callbacks): 697.3 µs per iteration
notifyListeners:ObserverList:success (500 callbacks): 16868.7 µs per iteration
notifyListeners:ObserverList:fail (1 callbacks): 0.1 µs per iteration
notifyListeners:ObserverList:fail (10 callbacks): 4.2 µs per iteration
notifyListeners:ObserverList:fail (100 callbacks): 350.9 µs per iteration
notifyListeners:ObserverList:fail (500 callbacks): 8564.4 µs per iteration <<<< !
notifyListeners:HashedObserverList (1 callbacks): 0.2 µs per iteration
notifyListeners:HashedObserverList (10 callbacks): 2.0 µs per iteration
notifyListeners:HashedObserverList (100 callbacks): 24.7 µs per iteration
notifyListeners:HashedObserverList (500 callbacks): 171.9 µs per iteration
notifyListeners:AnimationController (1 callbacks): 0.2 µs per iteration
notifyListeners:AnimationController (10 callbacks): 1.8 µs per iteration
notifyListeners:AnimationController (100 callbacks): 18.8 µs per iteration
notifyListeners:AnimationController (500 callbacks): 99.4 µs per iteration <<<< !

dashboard:
https://flutter-flutter-perf.skia.org/e/?begin=1706138370&end=1706224770&queries=sub_result%3DMatrixUtils_affine_transformPoint_iteration%26test%3Dmicrobenchmarks

Sweet. It's reassuring to see that you're monitoring the numbers.

It seems there are a couple of omitted type annotations in the benchmark.

Funnily, when adding a Linux target to run the benchmarks it also added a analysis_options.yaml, which shadowed the parent analyzer configuration :woman_shrugging: . There are certainly a lot of types now.

Still LGTM. Could you sync the brach to head to fix the CI failure?

Done

The new benchmark will have to be registered here:

Did the registration.

I agree with @LongCatIsLooong: Best practice is to pull the benchmark (and its registration) out into a separate PR, submit that first, and then a day later submit this change to make sure the benchmark works properly and this change has the desired effect.

Not a problem. The changes and benchmark addition are in separate commits. Feel free to cherry-pick the benchmark additions first, happy to rebase after.

I've spent countless hours complying with any remotely reasonable request. I've provided extensive benchmarks from the get go. We all know and acknowledge that his is a small change with significant performance impact.

Best practice is to pull the benchmark (and its registration) out into a separate PR, submit that first, and then a day later submit this change to make sure the benchmark works properly and this change has the desired effect.

This sounds more like an opinion than a best practice. I'm always down for a good argument, just not big on process for the sake of process. AFAICT, there's nothing to be gained from this proposal unless you believe the CI server must execute the benchmarks to be trusted.
If you think I should spend a few more weeks on back and forts with moving goal posts: fine. I you don't have the bandwidth to spend 1minute on a cherry-pick: fine. If you don't want to improve Flutter's performance: fine. If you're ok with the opportunity cost for your users: fine. If you're comfortable with disrespecting your contributors' time: fine.
If you change your mind: also fine. Otherwise thank you for you time and good luck!

I am sorry if you misinterpreted my statement. That wasn't my intention. And there is no disrespecting anyones time. Getting any change landed sometimes requires a couple of feedback cycles and back-and-forth discussions before they are ready to land. That's just part of open-source team work.
We do appreciate your contribution, but we also expect contributors to get changes over the line and not just "throw them over the fence" to have someone else finish them when they are done with the fun bits.

I'm not sure how to read this. Are you talking generally? Are you talking about the same PR? Maybe some context faded over time? Are you insinuating?
Personally, I didn't think that addressing every request, answering everybquestion, writing and running benchmarks for hours were the fun bits. I'm sorry you feel like I'm just throwing code over the fence.
In any case thanks for explaining team work to me. I wish you had given me a rational instead   🤷♀

I still hope that we can land this with your help.

Good luck and thank you for your time

Hey @ignatz, if you allow me, I have pushed the benchmarks to a different PR so your contribution can be landed as part 1 (the benchmark) and part 2 (the improvements).

Hey @bernaferrari. Thanks for jumping in, much appreciated.

I don't know how to remove the benchmark PR and merge this, could you do that? @ignatz else I can reopen the PR without it...

I don't know how to remove the benchmark commit and merge this, could you do that? @ignatz else I can reopen the PR without it...

Of course, done. That's also what I offered before. In case you're curious, can be done with git rebase. There's also git cherry-pick in case you want to selectively pull in a single commit w/o creating an independent one.
To be clear, I'm happy to help with virtually any request I can remotely reason about.  In general, most engineers will be allergic to authority arguments. Doesn't feel like teamwork, feels at best like someone wanting to check the "demonstrates leadership at the next level" staff-level box and triggers my inner Hixie.
I love Flutter :heart:

Greetings from PR triage, checking in on this, it looks like the benchmark will need to be checked back in before we can land this PR?
For reference, https://github.com/flutter/flutter/pull/143799 was reverted in https://github.com/flutter/flutter/pull/143946.
Performance improvements like this are difficult to test, and therefore difficult to catch when they regress. The benchmark is intended to establish a baseline before this lands so we know (1) it has made an improvement on that baseline as expected, and then (2) it persists so we will know if it regresses in the future.
Also, reading through the comments here, just wanted to provide a friendly tone check. We should all be working together respectfully, per our code of conduct. 🙂

I'll reland for the third time the benchmarks. It is hard because they are not checked by the CI, so bugs slided two times and I got tired/traumatized of getting things reverted. Then I forgot, but I'll do it again shortly.

Thanks @bernaferrari. I appreciate you volunteering your time to help with this change. 🙏

Since this was closed, I filed https://github.com/flutter/flutter/issues/146211 to follow up on this.

This (finally) got merged! 🥳


trigger pattern mentioned in the paper

Hi,
Is the trigger pattern ∆ mentioned in the paper predefined?

Not exactly sure if I get the question...
If you are referring to how we initiate delta in the reverse engineering process, we use random values as start.

Thanks, that's what I refer to. I think I misunderstood the Reverse Engineering Process in the paper.


Custom gradient tape-based fit() function is slower than Tf2 fit() and has memory leaking

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 20.04.1 LTS
TensorFlow version (use command below): v2.3.0-54-gfcc4b966f1 2.3.1
Python version: Python 3.8.5
CUDA/cuDNN version: CUDA Version 11.0.207
GPU model and memory: NVIDIA Quadro RTX 5000, 16 GB GDDR6

I wrote a custom fit() function using gradient tape in TF2. The @tf.function decorator has been used in a few specific locations to disable eager execution, which is supposed to speed up the whole code.
My custom fit has three main issues:
1-It's a lot slower than its TF2 counterpart (decorator @tf.function on the train_batch function does not improve the speed, while the decorator placed above the fit() function does). The time indicated on the output (see image attached below) of the custom fit() is wrong for some reason, in reality it is constant, around 10 seconds per each epoch.
2-Memory usage does increase linearly (see plot at the bottom).
3-Code gets killed even before reaching the maximum amount of GPU memory usage (see code output at the bottom).
Below you can find the complete code, you should be able to run it directly, as it does not require any data or any additional line of code.
import tensorflow as tf
import tensorflow.keras as keras
from tensorflow.keras.layers import Input, Dense, Dropout, Conv1D, Flatten, MaxPooling1D
from tensorflow.keras.models import Model
import numpy as np
import warnings
import psutil
import time
import matplotlib.pyplot as plt

### DEFINE NEURAL NETWORK
def baseline_network():
    input = Input(shape=(128, 11))
    layer = Conv1D(64, 5, activation='relu', padding='same')(input)
    layer = Flatten()(layer)
    layer = Dense(1024, activation='relu')(layer)
    output = Dense(257, activation=None)(layer)

    model = Model(inputs=input, outputs=output)
    model.compile(optimizer='adam',
                  loss='mse',
                  metrics=['mse'])

    return model

### DATA GENERATOR
def generator():

    # initialize numpy tensors
    X_out = np.zeros([6000, 128, 11])
    Y_out = np.zeros([6000, 257])

    ### CREATE DATA BATCHES ###
    while True:
        for i in range(32):

            yield X_out, Y_out

### CUSTOM FIT USING GRADIENT TAPE
class runGradientTape:

    def __init__(self, model, cost_function, iterator_train, steps_train, max_epochs, batch_size):

        self.model = model
        self.cost_function = cost_function
        self.iterator_train = iterator_train
        self.steps_train = steps_train
        self.max_epochs = max_epochs
        self.batch_size = batch_size

    def lossWrapper(self):

<EMAIL_ADDRESS>        def lossFunction(y_true, y_pred):
            # calculating loss and squeezing single dimensions away
            loss = tf.squeeze(self.cost_function(y_pred, y_true))
            # calculate mean over batches
            loss = tf.reduce_mean(loss)
            # return the loss
            return loss
        # returning the loss function as handle
        return lossFunction

    @tf.function
    def train_batch(self, x_batch_train, y_batch_train):
        # run gradient taping
        with tf.GradientTape() as tape:
            y_hat = self.model(x_batch_train, training=True)
            loss_value = self.loss_fn(y_batch_train, y_hat)

        # calculate gradients
        grads = tape.gradient(loss_value, self.model.trainable_variables)
        self.model.optimizer.apply_gradients(zip(grads, self.model.trainable_variables))

        return loss_value

    @tf.function
    def fit(self):

        # others
        warnings.filterwarnings("ignore")

        self.loss_fn = self.lossWrapper()
        progbar = tf.keras.utils.Progbar(self.steps_train)

        memory_used = []

        # ITERATE OVER EPOCHS
        for epoch in range(self.max_epochs):

            print('Epoch = %d/%d' %(epoch+1, self.max_epochs))

            # iterate over steps in training generator
            for i in range(self.steps_train):

                x_batch_train, y_batch_train = next(self.iterator_train)

                loss_value = self.train_batch(x_batch_train, y_batch_train)
                progbar.update(i+1)

                memory_used.append(psutil.virtual_memory().used / 2 ** 30)
                #print('   memory used: ', memory_used[-1])

        #plt.plot(memory_used)
        #plt.title('Memory usage vs batch')
        #plt.savefig('mem_usage')

# parameters
steps_train = 100
n_epochs = 5
batch_size = 32
baseline =  baseline_network()
iterator_train = generator()

# TF2 fit()
print("TF2 fit")
start_time = time.time()
history = baseline.fit(iterator_train, steps_per_epoch=steps_train, epochs=n_epochs)
stop_time = time.time()
print("time elapsed (TF2 fit): ", stop_time-start_time)

#reset parameters
del baseline, iterator_train
baseline =  baseline_network()
iterator_train = generator()

# custom fit
print("Custom fit")
start_time = time.time()
run_gradient_tape = runGradientTape(baseline, keras.losses.MSE, iterator_train, steps_train, n_epochs, batch_size)
run_gradient_tape.fit()
stop_time = time.time()
print("time elapsed (custom fit): ", stop_time-start_time)

Some of the suggestions I tried so far:
1- gs.collect() does not give any improvement in this case.
2- profiling the code using tensorboard (check profile and memory usage below).
Could you please give me any suggestion on how to solve the three previously mentioned problems?
Code output

Memory usage

Timing:

@ymodak
I have tried in colab with TF version 2.4, 2.5.orc1 and nightly version and noticed that session is being crashed. Please, find the gist here. Thanks!

@ymodak
I have tried in colab with TF version 2.4, 2.5.orc1 and nightly version and noticed that session is being crashed. Please, find the gist here. Thanks!

Hi, I believe it has something to do with the RAM available in Colab. I do not use this service but, as far as I understood, maximum RAM should be around 11-12 GB, please correct me if I am wrong.
The code crashes around batch 121-143. If you see the linear memory usage plot, we should be around that value.
However, my code does not crash on my GPU, but it gets killed at the very end, after the very last batch is fed. You may want to just reduce the number of epochs to 1, and see if this problem still occurs in Colab.

Tried to reproduce this and saw that @tf.function is not marked correctly:
Because @tf.function's first execution is expensive (which is used to do function tracing), marking the custom loop function as a @tf.function will cause the whole training process to be under tracing, which is slow and uses extra mem, eventually killing the training process when the memory are used up.
The convention is to mark the single training step as a @tf.function, so only the first training batch will have the graph tracing overhead (if the data type and shape remain the same for the remaining data).
Please see this colab as an example. We see that the custom fit runs about the same as (or slightly faster than) the tf2 fit, and no apparent memory increase after the custom training.


Possible fix for #1

This is untested, just quickly written done. Basically, you want to test everything for existence (if (confiObj.theme && confObj.theme == "x") {) or assign defaults.
I believe Object.assign({}, {...}, config) should result in the values in {} being defaults and config overwriting them.
Untested!

@MatthiasWinkelmann I fixed the issue, just restart the app or reload plugins


Invalid V1 Local Team IDs

Problem
Note: The below only applies to LOCAL teams and not ldap or saml teams.
In IAM V1, team IDs did not conform to our alphaneumeric restrictions, so teams that are migrated will have some problems if they contain spaces or special characters.
Problems arise in both the API in the UI for teams that were migrated from V1 with characters that are now invalid.
curl -k -H "api-token: $TOK" https://a2-dev.test/apis/iam/v2/teams/unique team name! | jq
<curl errors>
{
  "error": "no team found with id \"unique\"",
  "code": 5,
  "message": "no team found with id \"unique\"",
  "details": []
}

In the UI:

Solutions
This is a bit tricky to fix because if we want to completely automate fixing this issue for customers, we need to update invalid IDs in the policies as well. This would involve multiple services and cereal which is always a non-trivial amount of engineering effort.
Solution 1: Do nothing
This issue only affects local teams which are in general not very utilized compared to ldap and saml teams by customers (outside of the admins team which is not affected).
We could just note this issue in the changelog and move on, helping any customers that might encounter it fix their teams up. This issue will not affect new teams, but only existing v1 teams that have invalid characters.
Solution 2: Note the brokenness
If a user does have invalid teams, we could do a few things to let the user know with varying degrees of effort.
a. Low effort: Log the broken teams (either on migration or periodically). This will be not the most ideal UX but easy to implement.
b. Fairly low effort: Add a warning in the list view of the UI for affected teams. They would have to re-create the teams and the policies but that would be a higher visibility place to note what is up to the end users.
Solution 3: Partial Fix
Medium effort: Fix the team's IDs but not update the policies. We'd still want to note to them somehow that they need to go fix their policies. Just fixing up the IDs would be non-trivial but not a huge amount of work.
Solution 4: Automated Fix
Large effort: Update IDs to be valid and resolve any resulting name conflicts. We'd also need to then update the policies via cereal with the new IDs with logic for conflicting IDs. This would present a few challenges and would need to be maintained long term. This would be a pretty significant amount of engineering effort and some decent testing to get correct.

i support solution 1, or solution 2 a. considering time and priority of other work.
im pretty worried about the ux of solution 2 b. and 3.

we have decided on solution 1: do nothing.


undefined references to static pthread things

cds/threading/details/pthread_manager.h:78: undefined reference to `cds::threading::pthread::Manager::Holder::m_key'
cds/gc/hzp/hzp.h:558: undefined reference to `cds::gc::hzp::GarbageCollector::HelpScan(cds::gc::hzp::details::HPRec*)
cds/gc/hzp/hzp.h:413: undefined reference to `cds::gc::hzp::GarbageCollector::classic_scan(cds::gc::hzp::details::HPRec*)
cds/gc/hzp/hzp.h:408: undefined reference to `cds::gc::hzp::GarbageCollector::inplace_scan(cds::gc::hzp::details::HPRec*)

The solution is to link with the objects in obj/gcc-amd64-linux-64/release.


Re-design Tekton Resource Pruner to delete only Pods and PVCs

Pull Request
Description
The change replaces removal of the whole PipelineRun and all of it's resources with removal of only Pods and PVCs.
This allows preserving history of PipelineRuns in Tekton Dashboard.
Also this enables automated removal of workspace volumes, for example EBS Volumes in AWS when AWS EBS CSI Controller is used.
Type of change

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)

How Has This Been Tested?

Chart linting performed with helm lint.
The pipelines-library helm chart was checked with Chart Testing ct lint.
All the tekton-resource-pruner-related templates of the  pipelines-library helm chart were rendered with this helm template command:helm template tekton-resource-pruner -n $namespace \
./charts/pipelines-library -s templates/resources/pruner/* > /tmp/tekton-resource-pruner.yaml

The rendered templates were deployed to a Kubernetes cluster:kubectl apply -f /tmp/tekton-resource-pruner.yaml -n $namespace

The job was observed to launch both on schedule and manually several times with the correct actions performed and no errors in log output.

Checklist:

[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes

Hi @NikolayMarusenko,
I've messed up history on the https://github.com/epam/edp-tekton/pull/38.
This is the new one with squashed commits.


Cannot set TypedArrayData in XMLHttpRequest.RequestData

This code:
import           JavaScript.TypedArray
import qualified JavaScript.Web.XMLHttpRequest as XHR

setData :: Uint8Array -> XHR.Request -> XHR.Request
setData u8 req = req { XHR.reqData = XHR.TypedArrayData u8 }

Produces this error message:
Couldn't match kind ‘AnyK’ with ‘*’
Expected type: ghcjs-base-<IP_ADDRESS>:JavaScript.TypedArray.Internal.Types.SomeTypedArray
                 e 'ghcjs-base-<IP_ADDRESS>:GHCJS.Internal.Types.Immutable
  Actual type: Uint8Array
In the first argument of ‘XHR.TypedArrayData’, namely ‘u8’
In the ‘XHR.reqData’ field of a record

As far as I can tell, this is the intended purpose of XHR.TypedArrayData, so I'm not sure why it doesn't work.

I'm seeing the same issue. @mgsloan any ideas?

Hmm, I'm not sure what's going wrong there.  One possible hack would be to coerce u8, should be representationally equivalent.

Yeah, I ended up slapping an unsafeCoerce on to get around it shortly after commenting on the issue, but there's definitely a bit of an impedance mismatch at the type level that appears to make XHR.TypedArrayData not usable without coercion.

got bit by this today


Question: MDC and udp-multicast benchmark

Is there any plan to provide a multicast benchmark,  like 1:n pub-sub pattern in realtime market data usecase.

This is now support. See https://github.com/real-logic/benchmarks/blob/master/scripts/aeron/remote-echo-mdc-benchmarks.


Fixed stats in Analytics section with KDA price per day

Now the TVL and Volume graph will display the right KDA price related to the selected day or period


Error while installing

I got an error from python.exe while installing
"The code execution cannot proceed because libpython3.9.dll was not found"
even though im using python 3.11??
sorry if this is some dumb error on my part, im new to python

nvm lmao i forgot to remove chocolatey
that fixed it


Fix FlatList jumping

Part of the Comment Linking project
Main issue: https://github.com/Expensify/App/issues/20282
Doc section: https://docs.google.com/document/d/1v-ZaIRTZL5LIsyPWB0IopBNNgCajf5WC1OA8cVKtd8I/edit#bookmark=id.bry74gx446k1
Project: Comment Linking

Address the noticeable jumps in the list when scrolling and simultaneously adding items from the bottom of the list. This fix will be applied on both native and web sides.
Manual Test Steps
TBD
#Automated Tests
TBD – probably none

cc @perunt

I re-tested this with a large dataset, and I can confirm that it is performing significantly better. I noticed that in version 0.72.1, the react-test-renderer was removed from the runtime, which seems to have improved things. When using a single item render per batch, it functions kinda ok(similar to previous performance levels). As such, I don't believe we need to make any changes here. And we can hop on the next steps.
Here video
@roryabraham , wdyt?

Sounds good. Hopefully we'll get much better performance (and no scroll-jumps) when we integrate WishList

We ended up having to revert this

Here's one of the regressions it caused - https://github.com/Expensify/App/issues/31143 - should I keep that issue open, or would you prefer to handle it on this one? Not quite sure what our default process is for that

Since the issue no longer exists we can close it, we'll just add a note on the next PR that need to make sure it doesn't reproduce that

Oops, didn't mean to close - we're still working on adding a PR here

Hey, it was merged a while ago. There will be another enhancement for the scrolling experience, but the original issue has been fixed. I guess we can close it


Update to arkworks-0.4.0

arkworks v0.4.0 introduces unbelievable amount of breaking changes, and they are clearly moving on full speed. Any improvements (and bug fixes) are going to build on v0.4.0 and up.
To be able to take advantage of those improvements and fixes, Hyperplonk needs to update its arkworks dependencies to v0.4.0 as soon as possible.

Also, I'm already working on it 😎
PR soon

Also, I'm already working on it 😎
PR soon
Thank you Ivan!


Fix Philips Hue Smart Button device automation triggers for custom clusters

Since https://github.com/zigpy/zha-device-handlers/pull/664 was merged, we now have additional zha_events.
This PR now makes use of them as device automation triggers.
This fixes https://github.com/home-assistant/core/issues/44170 as the on_hold event is now "exposed" in the Home Assistant GUI (as well as all other Philips emulated multi-click features).
The Hue Smart Buttons works by switching between sending the on and off_with_effect.
However, the PhilipsBasicCluster and PhilipsRemoteCluster (which were added to the Smart Button in https://github.com/zigpy/zha-device-handlers/pull/664 as well) are/were only meant for the 4 button Philips Dimmers which have separated off and on buttons.
As keeping the on and off_with_effect would cause confusion and break the emulated multi-clicking implementation, the off_with_effect has been removed. (SHORT_PRESS, TURN_ON): {COMMAND: COMMAND_ON} is replaced by (SHORT_PRESS, TURN_ON): {COMMAND: "on_press"}.
In most cases this will not break any automations, as (basically nobody uses the Smart Button atm and) the most used automation in Home Assistant (simply telling if the button was pressed) has two triggers: on and off. Since off will be removed and on also called when the button is pressed a second time, these automations will still work fine.
Only if a user is actually relying on having two distinct automations which are called in an alternative fashion, the user will need to create an input_boolean in Hass and switch with each click.
All in all, I think it makes more sense to implement it like this:

because of https://github.com/zigpy/zha-device-handlers/pull/664 being merged and having a consistent implementation with the Hue Dimmer remotes creates a better user experience (in Hass GUI)
and because a user expects a stateless button to not alternative between on and off

These changes have been tested and my automations also didn't break because of the reasons mentioned above (alternation of on/off as two triggers for the same automation is now only on)

Codecov Report

Merging #688 (f32442a) into dev (61dedab) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##              dev     #688   +/-   ##
=======================================
  Coverage   83.48%   83.48%
=======================================
  Files         171      171
  Lines        3814     3814
=======================================
  Hits         3184     3184
  Misses        630      630

Impacted Files
Coverage Δ

zhaquirks/philips/rom001.py
100.00% <ø> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 61dedab...f32442a. Read the comment docs.

Pull Request Test Coverage Report for Build 453788937

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 83.482%

Totals

Change from base Build 453670052:
0.0%

Covered Lines:
3184

Relevant Lines:
3814

💛  - Coveralls


Added input action handler interface to receive action events regardless of the input source

Overview
Added a new input action handler interface to listen for input action events regardless of input source:
public interface IMixedRealityInputActionHandler : IMixedRealityBaseInputHandler
{
	void OnActionStarted(BaseInputEventData eventData);
	void OnActionEnded(BaseInputEventData eventData);
}

Also added an input action handler script for convenience:

And an input actions example:

Changes

Fixes: #4006  .

@jbienzms , do you mind having a look at it and tell me what you think? I'll add the interfaces for actions that contain data soon but it should be easy enough to imagine how they will look like.

Talking to @killerantz Interactable isn’t actually meant to be used as the main way to listen to events. As I understand, the main power of interactable is in configuring visual states of buttons in response to events, not to hook up functionality in response to events.
Glad to see there is a way people may use instead of interactable.

I think this is what I've been looking for all along. As long as I can get the source of the event, than this is what I was expecting from the Input Actions system. I wanted to build Interactable off of action events.
It would be great for Interaction Mappings to have a better link to the actions they invoke. or the eventData to include the InteractionMapping that invoked it as a property.

@jbienzms , do you mind having a look at it and tell me what you think? I'll add the interfaces for actions that contain data soon but it should be easy enough to imagine how they will look like.

Soo sorry this has taken me so long. I'm in Berlin at hacks this week and didn't have a chance to look. Reviewing now.

In this PR it looks like this could be accomplished in 2 ways:
Look at BaseInputEventData.Source
Try cast the BaseInputEventData to the source-specific event data.
Am I correct about both of those working? If so, I think we're set in this regard.

@jbienzms , you're totally correct.

/azp run mrtk_PR

It would be great for Interaction Mappings to have a better link to the actions they invoke. or the eventData to include the InteractionMapping that invoked it as a property.

@killerantz , agreed. Do you mind opening a feature request for that?

/azp run mrtk_PR

/azp run mrtk_PR

Checking this in with support only for digital input actions. Created #4576 to add support for non-digital ones, as there's a bit more work involved than expected.


why i failed to install gulp?

npm ERR! Error: ENOENT, chmod 'D:\Tools\Dev\Now\npm_dir\node_modules\gulp\bin\gulp0.js'
but there is no such file there, what's wrong?

Your node and npm install is insanely out of date.    Installation problems should be reported to npm.


nerdctl push images to harbor too slow

Description
containerd : v1.7.1
nerdctl version : 1.7.1
k8s version : 1.27.5
I am install nerdctl tool in k8s node, and I pull images is normal, but push images to harbor is too slowly
What's the reason I can't find it

What can I do to speed up the push
Steps to reproduce the issue

download nerdctl
nerdctl push images to harbor
is slow

Describe the results you received and expected
Push speed only B/s
What version of nerdctl are you using?
v.1.7.1
Are you using a variant of nerdctl? (e.g., Rancher Desktop)
None
Host information
No response

after this image push success
this images I only try it

在这张图片推送成功后，这张图片我只尝试了，这张redis图片大小是49MB，但使用380s,
and I use docker push is normal

I tried investigating the issue, but the harbor's a little slow.

The Redis image takes 17.2 seconds, but it is not as slow as 380 seconds.


Migration tool sometimes reports files in project as unmigrated dependencies

Testing the migration tool with an external project, it's reporting files within the project itself as unmigrated dependencies.  This should never happen.
It seems that this happens when an input file has compilation errors; in this case, the migration tool skips it (to avoid the compilation errors leading to a crash).  This causes it to get confused and not realize that the file with an error is one of the files-to-be-migrated.  The reason this happened in my instance is that I had "git clone"d a project that had generated files, and I didn't think to run the code generation step prior to migrating.
Probably what should happen is that if the project has both compilation errors and unmigrated dependencies, the migration tool should report the compilation errors, since (a) they are a more serious problem, and (b) they could interfere with the tool's ability to correctly compute whether there are unmigrated dependencies.

Unassigning myself because I'm not actively working on the migration tool right now.

As of 1c7fe7196dd7a542f7f91cd6ea2ac81e1a13146c, the null safety migration tool has been removed from active development and retired. No further work on the tool is planned.
If you still need help, or you believe this issue has been closed in error, please feel free to reopen.


Develop

Successfully tested changes with small subset of samples in preparation for Major 2024 release.

good to go from my perspective, pending deciding whether to include {source} or not.


Optimize HUNTER_BINARY_DIR layout

For now to avoid path too long error external directory with short path can be specified: HUNTER_BINARY_DIR. But looks like Visual Studio 8 2005 use relative paths for sources, so in fact it becomes even more longer:

Path to sources: /<hunter-root>/_Base/<hunter-id>/<config-id>/<toolchain-id>/Build/<package>/Source
Path to build: /<hunter-binary-dir>/<package>/Build
Relative: /../../../../ (level up to <hunter-root>) + <hunter-binary-dir>/<package>/Build

Possible solution is to unpack sources to HUNTER_BINARY_DIR too:

Path to sources: /<hunter-binary-dir>/<package>/Source
Path to build: /<hunter-binary-dir>/<package>/Build

Note: how to deal with "unpack-only" packages from /<hunter-root>/_Base/Download/<package>/Unpacked?
Test it on visual studio 8 2005 with winsparkle project. See bug: https://github.com/ruslo/hunter/issues/121

---
Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/12375553-optimize-hunter_binary_dir-layout?utm_campaign=plugin&utm_content=tracker%2F1032790&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F1032790&utm_medium=issues&utm_source=github).

Path to sources:  /<hunter-root>/_Base/<hunter-id>/<config-id>/<toolchain-id>/Build/<package>/Source

Possible solution is to unpack sources to HUNTER_BINARY_DIR too:
Path to sources: /<hunter-binary-dir>/<package>/Source

What i instead of unpacking them at a different place we unpack then at the usual place and symlink them to a simpler path and update the symlink as necessary?

Path to sources: ```/<hunter-binary-dir>/<package>/Source```
Path to build: ```/<hunter-binary-dir>/<package>/Build```

And also I suggest the paths being something like /<hunter-binary-dir>/Builds/<package> to keep hunter root somewhat clean when multiple packages are used.

What if instead of unpacking them at a different place we unpack them at the usual place and symlink them to a simpler path and update the symlink as necessary?

what is the difference? Source directory will be removed after build so we not using more space

Sorry I meant to include the other directories (Build) and so on, it could shorten the path length while still maintaining hunter-id, config-id and toolchain-id which I assume you need?

Sorry I meant to include the other directories (Build) and so on, it could shorten the path length while still maintaining hunter-id, config-id and toolchain-id which I assume you need?

Ok.
Also I was thinking about do experiment with short path like C:\hunter and if it will work do remove HUNTER_BINARY_DIR feature because we can build package in C:\hunter, then upload binaries and reuse them in path of any long.

Might be relevant, mklink (symlinking on windows), its a part of windws starting from Vista as far as I am aware.


Internalize our ideviceinstaller dependency

Right now we depend on ideviceinstaller on Mac for iOS (and effectively for Android, see #1893).
It's the only uncommon direct dependency we have now other than XCode. As such, it's the only reason to set up homebrew.
If we could somehow internalize this dependency, it would remove all our homebrew issues and our ideviceinstaller issues, and make life easier for devs.

Actually I just discovered you also need ios-deploy.

We can put the binaries in cloud storage and download them.

Going to close this. We can deal with this in the context of streamlining our install process.


Failed to purge the temporary install directory

Describe the bug
I'm getting an error when running ddev composer create drupal-composer/drupal-project:8.x-dev --stability dev --no-interaction
Additional context
https://stackoverflow.com/questions/54693417/failed-to-purge-the-temporary-install-directory
Non related stuff
@rfay happy to see you back in my Drupal related life 😸

So great to see you too. Sorry for this regression. Odd that it happened, odd that the test doesn't catch it.

Thanks so much for the report, @amitaibu , #1486 should fix this up. Turns out it was only a warning anyway, but it's always cool when you can fix a bug by removing code!


How to Download All Video Quality Audio and Subtitles

I tried with some codes, m3u8 didn't work when I clicked, how can I download it like in the source file
url :
https://video.gumlet.io/64b89649aefe6ff644827c6c/64b896fa2c2bc76a3c82150a/main.m3u8

N_m3u8DL-RE https://video.gumlet.io/64b89649aefe6ff644827c6c/64b896fa2c2bc76a3c82150a/main.m3u8 -sv all -sa all -ss all

mp4 srt downloaded as m4a, I want it to download as in the photo
https://imgur.com/a/VDuQizd


Questsadmin give   - 'Quest Not Found'

When trying to give a Quest to a player when the Quest name includes spaces, I get the following message 'Quest Not Found'.
I've tried surrounding the Quest name in " and ' but no luck.

Yea, I can confirm that this has been an issue for a loooong time. I've been considering making a request for a solution to allow specific selection of quests that you are wanting.
Here is the work-around I've found, but which has it's limitations (hence needing my idea). You can give a quest via questadmin if you only use 1 unique word that is in your quest. For example, we have a quest called Bronze of the Gods. If we try to type in "/questadmin give  Bronze of the Gods", it doesn't work. HOWEVER, if we type in "/questadmin give  Bronze" then it works no problem!
The limitation is if you have multiple quests with the same name. For example we have a quest chain that forks, and the player picks a path... What that means is we have zero unique words to solo out the quest, as both forks end in Part-1, Part-2, Part-3 and each part has the same phrase before it. So puttin in Part-1 would simply select one and not the other.
My suggestion is to make each quest carry with it the number that it is given in "/quests list" and that is it's unique identifier in /questadmin as well. That, or allow for tab-completing quest names, but that no doubt would also solve being able to type in the full name shrug


Newly created volume fails to mount with fsck errors

kubectl get pods
NAME                                                             READY     STATUS              RESTARTS   AGE
pvc-93202023-1896-11e8-b8a8-96000007f375-ctrl-7fb796f666-tq7gj   2/2       Running             0          44m
pvc-93202023-1896-11e8-b8a8-96000007f375-rep-6bd59bcb55-49l56    1/1       Running             0          44m
pvc-93202023-1896-11e8-b8a8-96000007f375-rep-6bd59bcb55-gcf74    1/1       Running             0          1h
wordpress-55cbcdd99b-nrh5v                                       0/1       ContainerCreating   0          46m

kubectl describe pod wordpress-55cbcdd99b-nrh5v
Name:           wordpress-55cbcdd99b-nrh5v
Namespace:      default
<snip>
Volumes:
  wordpress-persistent-storage:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  wp-pv-claim
    ReadOnly:   false
<snip>
Events:
  Type     Reason                 Age                From                                   Message
  ----     ------                 ----               ----                                   -------
  Normal   Scheduled              46m                default-scheduler                      Successfully assigned wordpress-55cbcdd99b-nrh5v to worker-02
  Warning  FailedAttachVolume     46m                attachdetach-controller                Multi-Attach error for volume "pvc-93202023-1896-11e8-b8a8-96000007f375" Volume is already exclusively attached to one node and can't be attached to another
  Normal   SuccessfulMountVolume  46m                kubelet, worker-02  MountVolume.SetUp succeeded for volume "default-token-ppvzm"
  Warning  FailedMount            1m (x20 over 44m)  kubelet, worker-02  Unable to mount volumes for pod "wordpress-55cbcdd99b-nrh5v_default(8b9c9987-1899-11e8-b8a8-96000007f375)": timeout expired waiting for volumes to attach/mount for pod "default"/"wordpress-55cbcdd99b-nrh5v". list of unattached/unmounted volumes=[wordpress-persistent-storage]

kubectl get svc
NAME                                                TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                                         AGE
maya-apiserver-service                              ClusterIP   <IP_ADDRESS>     <none>        5656/TCP                                                        4d
pvc-93202023-1896-11e8-b8a8-96000007f375-ctrl-svc   ClusterIP   <IP_ADDRESS>    <none>        3260/TCP,9501/TCP                                               5h

Used the curl commands to query for the volume status using the cluster-ip <IP_ADDRESS>, which showed that volume - controller and replica's were functional.  (Refer #1275)
After seeing that volume was online, checked the openebs volume controller logs:
kubectl logs pvc-93202023-1896-11e8-b8a8-96000007f375-ctrl-7fb796f666-tq7gj
Same can be seen from the kubectl cluster-info dump
==== START logs for container pvc-93202023-1896-11e8-b8a8-96000007f375-ctrl-con of pod default/pvc-93202023-1896-11e8-b8a8-96000007f375-ctrl-7fb796f666-tq7gj ====
<snip... adding first replica and setting to RW>
time="2018-02-23T13:02:33Z" level=info msg="Connecting to remote: <IP_ADDRESS>:9502"
time="2018-02-23T13:02:33Z" level=info msg="Opening: <IP_ADDRESS>:9502"
time="2018-02-23T13:02:33Z" level=info msg="Adding backend: tcp://<IP_ADDRESS>:9502"
time="2018-02-23T13:02:33Z" level=info msg="Set replica tcp://<IP_ADDRESS>:9502 to mode RW"

<snip... adding second replica and setting to RW>
time="2018-02-23T13:02:37Z" level=info msg="Connecting to remote: <IP_ADDRESS>:9502"
time="2018-02-23T13:02:37Z" level=info msg="Opening: <IP_ADDRESS>:9502"
time="2018-02-23T13:02:37Z" level=info msg="Adding backend: tcp://<IP_ADDRESS>:9502"
time="2018-02-23T13:02:38Z" level=info msg="Set replica tcp://<IP_ADDRESS>:9502 to mode RW"
time="2018-02-23T13:02:38Z" level=info msg="Update peer details of <IP_ADDRESS>:9502 "
time="2018-02-23T13:02:38Z" level=info msg="Update peer details of <IP_ADDRESS>:9502 "

<snip.. accepting connection from initiator>
time="2018-02-23T13:05:13Z" level=info msg="<IP_ADDRESS>:3260"
time="2018-02-23T13:05:13Z" level=info msg="Accepting ..."
time="2018-02-23T13:05:13Z" level=info msg="connection is connected from <IP_ADDRESS>:50324...\n"
time="2018-02-23T13:05:13Z" level=info msg="Listening ..."
time="2018-02-23T13:05:13Z" level=info msg="New Session initiator name:iqn.1993-08.org.debian:01:b93aa358deea,target name:iqn.2016-09.com.openebs.jiva:pvc-93202023-1896-11e8-b8a8-96000007f375,ISID:0x23d010000"
time="2018-02-23T13:05:19Z" level=error msg=EOF
time="2018-02-23T13:06:50Z" level=info msg="<IP_ADDRESS>:3260"
time="2018-02-23T13:06:50Z" level=info msg="Accepting ..."
time="2018-02-23T13:06:50Z" level=info msg="connection is connected from <IP_ADDRESS>:40700...\n"
time="2018-02-23T13:06:50Z" level=info msg="Listening ..."
time="2018-02-23T13:06:50Z" level=warning msg="unexpected connection state: full feature"
time="2018-02-23T13:06:50Z" level=error msg=EOF
time="2018-02-23T13:06:50Z" level=info msg="<IP_ADDRESS>:3260"
time="2018-02-23T13:06:50Z" level=info msg="Accepting ..."
time="2018-02-23T13:06:50Z" level=info msg="connection is connected from <IP_ADDRESS>:40702...\n"
time="2018-02-23T13:06:50Z" level=info msg="Listening ..."
time="2018-02-23T13:06:51Z" level=info msg="New Session initiator name:iqn.1993-08.org.debian:01:b93aa358deea,target name:iqn.2016-09.com.openebs.jiva:pvc-93202023-1896-11e8-b8a8-96000007f375,ISID:0x23d020000"
time="2018-02-23T13:06:51Z" level=error msg="non support"
time="2018-02-23T13:06:51Z" level=warning msg="check condition"
time="2018-02-23T13:06:52Z" level=warning msg="check condition"
time="2018-02-23T13:06:52Z" level=warning msg="check condition"

==== END logs for container pvc-93202023-1896-11e8-b8a8-96000007f375-ctrl-con of pod

The above openebs controller logs shows that connection was made from IQN - New Session initiator name:iqn.1993-08.org.debian:01:b93aa358deea. Started to check from "worker-02" where the volume should be mounted.

Logged into the worker-02 node via ssh.
cat /etc/iscsi/initiatorname.iscsi
## DO NOT EDIT OR REMOVE THIS FILE!
## If you remove this file, the iSCSI daemon will not start.
## If you change the InitiatorName, existing access control lists
## may reject this initiator.  The InitiatorName must be unique
## for each iSCSI initiator.  Do NOT duplicate iSCSI InitiatorNames.
InitiatorName=iqn.1993-08.org.debian:01:b93aa358deea

_ However, the above doesn't always help, since most of the machines come with same IQN by default _
Looked at the kubelet and system (kernel/iscsid) logs (in this case, they were all in syslog - since kubelet was running as service on the host). The following entries show the connection establishment and failure to mount.
Feb 23 14:06:50 worker-02 kubelet[6072]: I0223 14:06:50.637206    6072 operation_generator.go:1111] Controller attach succeeded for volume "pvc-9320
2023-1896-11e8-b8a8-96000007f375" (UniqueName: "kubernetes.io/iscsi/<IP_ADDRESS>:3260:iqn.2016-09.com.openebs.jiva:pvc-93202023-1896-11e8-b8a8-96000007f375:0") pod "w
ordpress-55cbcdd99b-nrh5v" (UID: "8b9c9987-1899-11e8-b8a8-96000007f375") device path: ""

Feb 23 14:06:50 worker-02 kubelet[6072]: I0223 14:06:50.736320    6072 operation_generator.go:446] MountVolume.WaitForAttach entering for volume "pv
c-93202023-1896-11e8-b8a8-96000007f375" (UniqueName: "kubernetes.io/iscsi/<IP_ADDRESS>:3260:iqn.2016-09.com.openebs.jiva:pvc-93202023-1896-11e8-b8a8-96000007f375:0")
pod "wordpress-55cbcdd99b-nrh5v" (UID: "8b9c9987-1899-11e8-b8a8-96000007f375") DevicePath ""

Feb 23 14:06:50 worker-02 kubelet[6072]: E0223 14:06:50.744831    6072 iscsi_util.go:235] iscsi: failed to rescan session with error: iscsiadm: No session found.
Feb 23 14:06:50 worker-02 kubelet[6072]:  (exit status 21)
Feb 23 14:06:51 worker-02 kernel: [349862.236890] scsi host4: iSCSI Initiator over TCP/IP
Feb 23 14:06:51 worker-02 kernel: [349862.755929] scsi 4:0:0:0: Direct-Access     CLOUDBYT OPENEBS          0.2  PQ: 0 ANSI: 5
Feb 23 14:06:51 worker-02 kernel: [349862.758792] sd 4:0:0:0: Attached scsi generic sg2 type 0
Feb 23 14:06:51 worker-02 kernel: [349862.759072] sd 4:0:0:0: [sdb] 4194304 512-byte logical blocks: (2.15 GB/2.00 GiB)
Feb 23 14:06:51 worker-02 kernel: [349862.759075] sd 4:0:0:0: [sdb] 4096-byte physical blocks
Feb 23 14:06:51 worker-02 kernel: [349862.759834] sd 4:0:0:0: [sdb] Write Protect is off
Feb 23 14:06:51 worker-02 kernel: [349862.759836] sd 4:0:0:0: [sdb] Mode Sense: 03 00 10 08
Feb 23 14:06:51 worker-02 kernel: [349862.760109] sd 4:0:0:0: [sdb] No Caching mode page found
Feb 23 14:06:51 worker-02 kernel: [349862.763321] sd 4:0:0:0: [sdb] Assuming drive cache: write through
Feb 23 14:06:51 worker-02 kernel: [349862.883977] sd 4:0:0:0: [sdb] Attached SCSI disk

Feb 23 14:06:51 worker-02 iscsid: Connection2:0 to [target: iqn.2016-09.com.openebs.jiva:pvc-93202023-1896-11e8-b8a8-96000007f375, portal: 10.102.15
6.74,3260] through [iface: default] is operational now

Feb 23 14:06:54 worker-02 kubelet[6072]: E0223 14:06:54.962815    6072 iscsi_util.go:338] iscsi: failed to mount iscsi volume /dev/disk/by-path/ip-1
<IP_ADDRESS>:3260-iscsi-iqn.2016-09.com.openebs.jiva:pvc-93202023-1896-11e8-b8a8-96000007f375-lun-0 [ext4] to /var/lib/kubelet/plugins/kubernetes.io/iscsi/iface-defaul
t/<IP_ADDRESS>:3260-iqn.2016-09.com.openebs.jiva:pvc-93202023-1896-11e8-b8a8-96000007f375-lun-0, error 'fsck' found errors on device /dev/disk/by-path/ip-<IP_ADDRESS>
4:3260-iscsi-iqn.2016-09.com.openebs.jiva:pvc-93202023-1896-11e8-b8a8-96000007f375-lun-0 but could not correct them: fsck from util-linux 2.27.1

Feb 23 14:06:54 worker-02 kubelet[6072]: /dev/sdb: Superblock has an invalid journal (inode 8).
Feb 23 14:06:54 worker-02 kubelet[6072]: CLEARED.
Feb 23 14:06:54 worker-02 kubelet[6072]: *** ext3 journal has been deleted - filesystem is now ext2 only ***
Feb 23 14:06:54 worker-02 kubelet[6072]: /dev/sdb: One or more block group descriptor checksums are invalid.  FIXED.
Feb 23 14:06:54 worker-02 kubelet[6072]: /dev/sdb: Group descriptor 0 checksum is 0x0000, should be 0x9444.
Feb 23 14:06:54 worker-02 kubelet[6072]: /dev/sdb: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
Feb 23 14:06:54 worker-02 kubelet[6072]: #011(i.e., without -a or -p options)
Feb 23 14:06:54 worker-02 kubelet[6072]: .

Feb 23 14:06:54 worker-02 kubelet[6072]: E0223 14:06:54.965441    6072 nestedpendingoperations.go:263] Operation for "\"kubernetes.io/iscsi/10.102.1
56.74:3260:iqn.2016-09.com.openebs.jiva:pvc-93202023-1896-11e8-b8a8-96000007f375:0\"" failed. No retries permitted until 2018-02-23 14:06:55.465364682 +0100 CET m=+349
788.010169186 (durationBeforeRetry 500ms). Error: "MountVolume.WaitForAttach failed for volume \"pvc-93202023-1896-11e8-b8a8-96000007f375\" (UniqueName: \"kubernetes.i
o/iscsi/<IP_ADDRESS>:3260:iqn.2016-09.com.openebs.jiva:pvc-93202023-1896-11e8-b8a8-96000007f375:0\") pod \"wordpress-55cbcdd99b-nrh5v\" (UID: \"8b9c9987-1899-11e8-b8a
8-96000007f375\") : 'fsck' found errors on device /dev/disk/by-path/ip-<IP_ADDRESS>:3260-iscsi-iqn.2016-09.com.openebs.jiva:pvc-93202023-1896-11e8-b8a8-96000007f375-l
un-0 but could not correct them: fsck from util-linux 2.27.1\n/dev/sdb: Superblock has an invalid journal (inode 8).\nCLEARED.\n*** ext3 journal has been deleted - fil
esystem is now ext2 only ***\n\n/dev/sdb: One or more block group descriptor checksums are invalid.  FIXED.\n/dev/sdb: Group descriptor 0 checksum is 0x0000, should be
 0x9444.  \n\n/dev/sdb: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.\n\t(i.e., without -a or -p options)\n."

Feb 23 14:06:55 worker-02 kernel: [349866.797554] sd 4:0:0:0: lun280922523394096 has a LUN larger than allowed by the host adapter

Feb 23 14:06:55 worker-02 kubelet[6072]: E0223 14:06:55.663460    6072 iscsi_util.go:338] iscsi: failed to mount iscsi volume /dev/disk/by-path/ip-<IP_ADDRESS>:3260-iscsi-iqn.2016-09.com.openebs.jiva:pvc-93202023-1896-11e8-b8a8-96000007f375-lun-0 [ext4] to /var/lib/kubelet/plugins/kubernetes.io/iscsi/iface-default/<IP_ADDRESS>:3260-iqn.2016-09.com.openebs.jiva:pvc-93202023-1896-11e8-b8a8-96000007f375-lun-0, error 'fsck' found errors on device /dev/disk/by-path/ip-<IP_ADDRESS>:3260-iscsi-iqn.2016-09.com.openebs.jiva:pvc-93202023-1896-11e8-b8a8-96000007f375-lun-0 but could not correct them: fsck from util-linux 2.27.1

Feb 23 14:06:55 worker-02 kubelet[6072]: /dev/sdb: One or more block group descriptor checksums are invalid.  FIXED.
Feb 23 14:06:55 worker-02 kubelet[6072]: /dev/sdb: Group descriptor 0 checksum is 0x0000, should be 0x9444.
Feb 23 14:06:55 worker-02 kubelet[6072]: /dev/sdb: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
Feb 23 14:06:55 worker-02 kubelet[6072]: #011(i.e., without -a or -p options)
Feb 23 14:06:55 worker-02 kubelet[6072]: .

Work Around: To get the volume back online, ran fsck /dev/sdb on the host. Post this the volume became accessible and application started.

I've had this same issue happen for me twice; the first time I figured it was due to disk pressure and using sparse images, but this time around it happened without any disk pressure, and indeed, even without any input from me as far as I can tell.  One of the three pods of my elasticsearch cluster decided to die, and when it has come back up, the node can't mount the pvc filesystem.
The fact that this happens when a pod is in an otherwise healthy state is disquieting.  Deleting pods is a valid balancing strategy that should not affect the filesystem's consistency, especially if the mounting is handled at the host level.

Is there a way that I can ask a replica to become the master, in case it might work where the current master doesn't?

I use CentOS7 for my kubernetes nodes.  The fsck.ext4 that is provided in CentOS7 doesn't support the same features that openebs is using when it provisions the filesystem (is that openebs, or kubernetes, doing that?) -- as a result, I cannot fix this disk without adding third-party software to my node.

I also use Rancher to administer my kubernetes rollout, so I suppose the mkfs for the filesytem could actually be inside of rancher rather than at the CentOS7 OS level.

Fortunately this is a clustered service, so I can delete the pvc, recreate the pod and get back to work, but this would otherwise be a pretty serious problem in production.
One item in the below output that is interesting is that I've asked for the CAS to be provisioned as xfs, but they're being made as ext4 apparently.  I'm not sure if that's an issue with rancher or openebs.
pvc:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"4dd1d524-ca93-11e8-afd9-6a03d5095334","leaseDurationSeconds":15,"acquireTime":"2018-10-08T02:51:44Z","renewTime":"2018-10-08T02:51:46Z","leaderTransitions":0}'
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: openebs.io/provisioner-iscsi
  creationTimestamp: null
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app: eskeim
    release: eskeim
  name: eskeim-data-eskeim-0
  selfLink: /api/v1/namespaces/monitoring/persistentvolumeclaims/eskeim-data-eskeim-0
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: openebs-1repl
  volumeName: monitoring-eskeim-data-eskeim-0-3882014528
status: {}

storage class:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    cas.openebs.io/config: |
      - name: ReplicaCount
        value: "1"
      - name: StoragePool
        value: default
  creationTimestamp: null
  name: openebs-1repl
  selfLink: /apis/storage.k8s.io/v1/storageclasses/openebs-1repl
parameters:
  openebs.io/fstype: xfs
provisioner: openebs.io/provisioner-iscsi
reclaimPolicy: Delete
volumeBindingMode: Immediate

storage pool:
apiVersion: openebs.io/v1alpha1
kind: StoragePool
metadata:
  generation: 1
  labels:
    openebs.io/version: 0.7.0
  name: default
  namespace: ""
  resourceVersion: ""
  selfLink: /apis/openebs.io/v1alpha1/storagepools/default
  uid: ""
spec:
  path: /var/openebs

I am having this issue occurring consistently in my deployments.  It seems to happen when the storage fails to honor an eviction request during a kubectl drain scenario.  The problem is that whatever is provisioning the disk is using a newer version of ext4 that is not supported by CentOS 7.
I tried jumping into the replica pod to scan the filesystem on /openebs/volume-head-000.img, but the fsck in that container is ALSO too old to recognize metadata_csum.
What provisions the filesystem when the pvc requests it?  It has completely ignored my request to use xfs and provisions everything as ext4.

I am facing the same problems. The Pod event states to run fsck manually, but the elasticsearch Pod does not start at all, and I see no other way than entering a running container to run a fsck on that device. Whenever I define a new deployment (e.g., plain busybox) and the PV to be used, it has to be mounted, but the mount fails because of the faulty FS.

@damlub - which OS are you using?
@PeterGrace - The FSType was being ignored in 0.7.0. Fixed it in 0.7.1, the FSType needs to be mentioned under cas.openebs.io/config as shown below:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    cas.openebs.io/config: |
      - name: ReplicaCount
        value: "1"
      - name: StoragePool
        value: default
      - name: StoragePool
        value: xfs
  creationTimestamp: null
  name: openebs-1repl
  selfLink: /apis/storage.k8s.io/v1/storageclasses/openebs-1repl
provisioner: openebs.io/provisioner-iscsi
reclaimPolicy: Delete
volumeBindingMode: Immediate

@kmova My systems are CentOS 7.5
So far I figured out that the containers of openEBS use a newer version of ext4 - especially the e2fsck - than CentOS. Not sure if this is related.
If it would help, I can go with Ubuntu 16.04 or 18.04, too.

Thanks @damlub - Could you try with Ubuntu 16.04, while we check on the CentOS 7.5?

@damlub i deployed openebs on CentOS 7.0 successfully, no issues so far. I'm also trying to setup CentOS 7.5 using vagrant but i'm having some issues with it, will keep you posted about the progress.I have used following Vagrantfile to brought up kubernetes cluster.

@damlub i am having issues in bringing up setup (CentOS 7.5) using vagrant or kops. I would like to discuss about the same in details, please join us at openebs-community.slack.com and would you like to provide your handle if joined already.

Need help in reproducing this issue. Keeping the issue open as couple of users have hit it.


Fixed the issue that no space in string converted from tensor

Thanks, can you also change this to make --remove-bpe=sentencepiece work:
https://github.com/pytorch/fairseq/blob/f296824f4013dc28b471c47d7779547460afc7f0/fairseq/data/data_utils.py#L220
to:
sentence = sentence.replace(' ', '').replace('\u2581', ' ').strip()

Changed, thanks


govie-editor specs added

Adding the govie-editor platform to the gltf project explorer
Govie Editor is highly leveraging the GLTF 2.0 format. Converting various asset formats to glb and using it within the viewer (based on ThreeJS).

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Aruba Central Streaming API Client Failed due to Bad Request

Hi All,
We have realized that since early this week, the "wsclient_public.py" python client for the Aruba Central streaming API is no longer working due to a bad request:
  File "/dev_py/aruba/lib/python3.9/site-packages/websocket/_handshake.py", line 150, in _get_resp_headers
    raise WebSocketBadStatusException("Handshake status {status} {message} -+-+- {headers} -+-+- {body}".format(status=status, message=status_message, headers=resp_headers, body=response_body), status, status_message, resp_headers, response_body)
websocket._exceptions.WebSocketBadStatusException: Handshake status 400 Bad Request -+-+- {'date': 'Fri, 15 Sep 2023 08:40:34 GMT', 'content-type': 'text/plain; charset=utf-8', 'content-length': '19', 'connection': 'keep-alive', 'x-content-type-options': 'nosniff', 'strict-transport-security': 'max-age=31536000; includeSubDomains', 'x-frame-options': 'SAMEORIGIN', 'x-request-start': 't=1694767234.499', 'x-xss-protection': '1; mode=block'} -+-+- b'Unsubscribed Topic\n'

We haven't introduced a change on the client side and it is being executed as follows:
python3 wsclient_public.py --hostname <aruba-central-instance>.central.arubanetworks.com --jsoninput input.json --decode_data
We've also ensured to use the last version available and the issue persist.
Please, can you help us to fix the client?
Thank you very much,
Best regards,
Karl.

Dear Team,
Please, can you help us with this?
Taking a look on the last PRs we don't see major changes on the client, can it be linked with a change on the Streaming API?
Regards,
Karl.

Hi All,
Just to say that the same error is returned by the simple client:
(aruba_central_streamingapi)$ python3 simple_app.py
--- request header ---
GET /streaming/api HTTP/1.1
Upgrade: websocket
Host: XYZ.central.arubanetworks.com
Origin: https://XYZ.central.arubanetworks.com
Sec-WebSocket-Key: YYY
Sec-WebSocket-Version: 13
Connection: Upgrade
UserName<EMAIL_ADDRESS>Authorization: XXXXXX
Topic: monitoring

-----------------------
--- response header ---
HTTP/1.1 400 Bad Request
Date: Tue, 26 Sep 2023 13:42:12 GMT
Content-Type: text/plain; charset=utf-8
Content-Length: 19
Connection: keep-alive
X-Content-Type-Options: nosniff
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Frame-Options: SAMEORIGIN
X-Request-Start: t=1695735732.357
X-XSS-Protection: 1; mode=block
-----------------------
Handshake status 400 Bad Request -+-+- {'date': 'Tue, 26 Sep 2023 13:42:12 GMT', 'content-type': 'text/plain; charset=utf-8', 'content-length': '19', 'connection': 'keep-alive', 'x-content-type-options': 'nosniff', 'strict-transport-security': 'max-age=31536000; includeSubDomains', 'x-frame-options': 'SAMEORIGIN', 'x-request-start': 't=1695735732.357', 'x-xss-protection': '1; mode=block'} -+-+- b'Unsubscribed Topic\n'
Handshake status 400 Bad Request -+-+- {'date': 'Tue, 26 Sep 2023 13:42:12 GMT', 'content-type': 'text/plain; charset=utf-8', 'content-length': '19', 'connection': 'keep-alive', 'x-content-type-options': 'nosniff', 'strict-transport-security': 'max-age=31536000; includeSubDomains', 'x-frame-options': 'SAMEORIGIN', 'x-request-start': 't=1695735732.357', 'x-xss-protection': '1; mode=block'} -+-+- b'Unsubscribed Topic\n' - goodbye

Please, can you help us to move forward?
It looks like a change on any of the sides(either this client or the remote Streaming API) that triggers such bad request.
thanks in advance,
regards,
Karl.

@KarthikSKumar98 , have you seen this behaviour?

@Karl90 Hi, I haven't seen this behavior before. I did update the requirements.txt file in the streaming-api-client folder. The PR is still open. It will get merged soon. Could you check out the requirements.txt file and see if that resolves the issue  - https://github.com/aruba/central-python-workflows/pull/36
Could you also let me know what topic you are trying to stream when you are getting the error ?

Hi @KarthikSKumar98 ,
Thanks for your inputs.
Indeed, I executed the code including your PR #36, but unfortunately the installation of the dependencies failed "as is", due to the version of greenlet:

When removing the specific version, the installation ends successfully.
FYI, this is the actual requirements.txt that I am using:
certifi==2023.7.22
charset-normalizer==3.2.0
gevent==23.9.1
greenlet
idna==3.4
protobuf==4.24.3
requests==2.31.0
six==1.16.0
urllib3==2.0.5
websocket-client==1.6.3
zope.event==5.0
zope.interface==6.0

When the code is executed, then the same issue is found:
/aruba_central_sac/central-python-workflows/streaming-api-client/lib/python3.9/site-packages/websocket/_handshake.py", line 150, in _get_resp_headers
    raise WebSocketBadStatusException("Handshake status {status} {message} -+-+- {headers} -+-+- {body}".format(status=status, message=status_message, headers=resp_headers, body=response_body), status, status_message, resp_headers, response_body)
websocket._exceptions.WebSocketBadStatusException: Handshake status 400 Bad Request -+-+- {'date': 'Wed, 27 Sep 2023 11:27:16 GMT', 'content-type': 'text/plain; charset=utf-8', 'content-length': '19', 'connection': 'keep-alive', 'x-content-type-options': 'nosniff', 'strict-transport-security': 'max-age=31536000; includeSubDomains', 'x-frame-options': 'SAMEORIGIN', 'x-request-start': 't=1695814036.112', 'x-xss-protection': '1; mode=block'} -+-+- b'Unsubscribed Topic\n'

This actually started to happen 2 weeks ago, previously the client was able to pull the data from any topic, in this case we tested with the 'monitoring' one - which is very verbose.
For the sake of transparency and isolate issues, all my tests are done with the latest version of the repo, the only change is the requirements.txt as seen previously, that is why I highly think that the issue relies on the remote streaming API or the source client, do you think that any of the packages can trigger the issue on the client?
Thanks for the help!
Karl.

Hi @KarthikSKumar98 ,
Sure, this is the EU instance that I'm using:

hostname: app-eucentral3.central.arubanetworks.com
topic: monitoring

BTW, I've been told  that you also need to include the hostname in the input file. I also tried that and issue persisted.
Best regards,
Karl.

Hi Karl! I believe this issue could be due to a Streaming API bug. Would it be possible for you to file a TAC case regarding this?
Once you filed it, please send me the TAC number on my email -<EMAIL_ADDRESS>Thanks!


Add dependabot support for git

📦 Published PR as canary version: 1.0.2-canary.5.eecc514.0

:sparkles: Test out this PR locally via:
npm install<EMAIL_ADDRESS># or
yarn add<EMAIL_ADDRESS>

:rocket: PR was released in v1.1.0 :rocket:

:rocket: PR was released in v1.1.0 :rocket:


Provide assertion for Java8 predicates

Summary
I want to simply assert the behavior of a predicate  without evaluating it first
Example
// given some Predicate

Predicate<String> startsWithA = s -> s.startsWith("a");

// instead of
assertThat(startsWithA.test("abc")).isTrue();

// I want to write
assertThat(startsWithA).isTrueFor("abc");

Java 8 specific ?

YES : create Pull Request from the master branch

Remark: looks like a minor change and I would be happy tp provide the PR. Just let me know what you think (or if this is already possible somehow and I missed it).

This is already implemented. See #658.
It should come out in the next release.

Cool, thanks. cloning this one then.


Error: Unexpected properties found on Skill

Hi, I get this error if I try to use either the pageOverlapLength or the maximumPagesToTake properties:
"details": [
{
"code": "InvalidSkillset",
"message": "One or more skills are invalid. Details: Unexpected properties found on Skill. Parameters: skillset"
}
I am using API version 2023-07-01-Preview.
Thanks!
Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: e2887052-e54c-116f-fa90-160d0d7d0307
Version Independent ID: 205719f9-19c1-74b7-3093-478160a799f2
Content: Built-in skills - Azure Cognitive Search
Content Source: articles/search/cognitive-search-predefined-skills.md
Service: cognitive-search
GitHub Login: @HeidiSteen
Microsoft Alias: heidist

Hi @asocorro, can you post your question to Stack Overflow with more information, such as your skillset definition?  I can only respond to issues with the docs specifically.  Here's the link to Stack Overflow: https://stackoverflow.com/questions/tagged/azure-cognitive-search
#please-close