[SPARK-21176] [Web UI] Limit number of selector threads for admin ui proxy servlets to 8

What changes were proposed in this pull request?
Please see also https://issues.apache.org/jira/browse/SPARK-21176
This change limits the number of selector threads that jetty creates to maximum 8 per proxy servlet (Jetty default is number of processors / 2).
The newHttpClient for Jettys ProxyServlet class is overwritten to avoid the Jetty defaults (which are designed for high-performance http servers).
Once https://github.com/eclipse/jetty.project/issues/1643 is available, the code could be cleaned up to avoid the method override.
I really need this on v2.1.1 - what is the best way for a backport automatic merge works fine)? Shall I create another PR?
How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
The patch was tested manually on a Spark cluster with a head node that has 88 processors using JMX to verify that the number of selector threads is now limited to 8 per proxy.
@gurvindersingh @zsxwing can you please review the change?

Can one of the admins verify this patch?

Test build #3815 has started for PR 18437 at commit 2d7d4f1.

Test build #3815 has finished for PR 18437 at commit 2d7d4f1.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

retest this please

retest this please

Test build #78942 has started for PR 18437 at commit c6198d2.

Test build #78942 has finished for PR 18437 at commit c6198d2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78942/
Test PASSed.

LGTM, merging to maser/2.2/2.1!

Thank you all for your support!


--speed not working?

I've tried using the --speed option to speed up my gif but so far it hasn't worked. Specifically, I've tried using --speed 2 to try to get the gif to be twice as fast.
How is this option meant to be used?
The full command I used was
gifify in.mov --resize 730:-1 --speed 2 -o out.gif

Hi, can you host your movie file somewhere I can download it

Sure, here's a link

Seem that this still isn’t resolved. Have the same problem. And looks like it is related to the following issue: --fps doesn't take the input fps into account #21

Yes @khalidl I need help on this because I could not figure out the fps and speed issue

@vvo Wish I could help in some way, but my dev chops are not up there. I’ve been trying ffmpeg with Gifsicle, which handles fps and speed correctly, but images have more compression.
I can get the speed up by lowering the frames, e.g. setting --fps 15 on a video which has 25 fps.
Not sure if this has something to do with it at all: gifify - regarding framerates

I have the exactly same problem, was able to fix it by moving the delay from Convert to gifsicle, added a pull request -> #37

Published 2.1.5 with @eek fix, awesome you can now slllloooooooowwnnnnnn doowww or sppppeeed!


Circular imports between probability, text and utils modules

I am running into circular import problems while preparing a PR to add some functionality to nltk/text.py - specifically, I am trying to write a new class TextFreqDist in nltk/probability.py, which extends FreqDist with some custom features, such as ignoring non-alphanumeric characters. I want to use nltk.text.Text in nltk/probability.py, but this leads to an import error:
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-2-16ae48a68ad9> in <module>()
----> 1 import nltk
      2 from nltk.text import Text
      3 from nltk.probability import TextFreqDist
      4 import nltk.corpus

/Users/srm/Documents/sandeep/cst/dev/nltk/nltk/__init__.py in <module>()
    112 # Import top-level functionality into top-level namespace
    113
--> 114 from nltk.collocations import *
    115 from nltk.decorators import decorator, memoize
    116 from nltk.featstruct import *

/Users/srm/Documents/sandeep/cst/dev/nltk/nltk/collocations.py in <module>()
     35 from nltk.compat import iteritems
     36
---> 37 from nltk.probability import FreqDist
     38 from nltk.util import ngrams
 39 from nltk.metrics import ContingencyMeasures, BigramAssocMeasures, TrigramAssocMeasures

/Users/srm/Documents/sandeep/cst/dev/nltk/nltk/probability.py in <module>()
     48 from functools import reduce
     49 from nltk import compat
---> 50 from nltk.text import Text
     51 from nltk.compat import Counter
     52

/Users/srm/Documents/sandeep/cst/dev/nltk/nltk/text.py in <module>()
     22 import re
     23
---> 24 from nltk.probability import FreqDist, LidstoneProbDist
     25 from nltk.probability import ConditionalFreqDist as CFD
     26 from nltk.util import tokenwrap, LazyConcatenation

ImportError: cannot import name 'FreqDist'

Would it be possible to restructure the package __init__.py to avoid these issues?

I can deal this specific issue by avoid importing nltl.text.Text in nltk/probability.py, but I was wondering whether anyone else has had such issues.

Yes, the solution to this problem is to have more specific imports. Sorry I don't know of any way to organise __init__.py to avoid it.


Stop measuring alloc counts on 5.0

Motivation:
5.0 is several years old now, and while we still support it there is
beginning to be some conflict with performance optimizations for newer
Swift versions. We should stop counting the allocation counts on this
version now: we no longer care as much about regressions.
Modifications:

Stop setting alloc counter limits on 5.0
Remove the script that tries to add them back.

Result:
No longer measure allocs on 5.0

Don't we need to not run the tests in CI for 5.0?

Oh, 5.0 has just got green, apparently not.

@Lukasa to George's point: should we also edit the docker file to skip the perf tests? something like -f tests_0[012356789] (note the missing 04 which IIRC are the perf tests)?

@Lukasa to George's point: should we also edit the docker file to skip the perf tests? something like IntegrationTests/run-tests.sh -f tests_0[012356789] (note the missing 04 which IIRC are the perf tests)?

@weissi further question: should the tests fail if there are no limits, or is that by design?

Dockerfile edited

@weissi further question: should the tests fail if there are no limits, or is that by design?

that's by design. We can't expect everybody to have limits configured (we emit warnings with limit suggestions though). But we can expect everybody to be able to run the tests.

@Lukasa
13:29:58 FAILURE: assertion '95001' <= '95000' failed

in 5.3. This is a result of a too-tight limit and slight instability. In the 5.3 file, mind upping that limit from 95000 to  95050 ?


Reschedule MP4.0 delay announcement

#MicroProfileRelease 4.0 is delayed due to the @MicroProfileIO Working Group Requirement. #MicroProfilers 💛 working on the APIs should continue their work towards Milestone 1. Read more HERE: https://bit.ly/36NV21n
#OpenSource #OSS

All set

Thank YOU @rstjames!
So, the pushed was also deleted?

fixing issues with "surprise" deletions on media pushed https://github.com/eclipse/microprofile-marketing/issues/305


[sfm] Create a metric reconstruction.

The scale of the reconstruction is currently arbitrary. Develop a solution allowing to provide one or many measures to the reconstruction algorithm used to export a "correctly scaled" reconstruction.
This issue is critical for the evaluation against any ground truth as they are not properly scaled. Therefore, the error on the translation term is irrelevant, yet usable for a comparison between different algorithms as a relative measure.

Note that there is a solution based on GCP rigid registration that is already available.
Creating and developing a registration API with various input would be logical for OpenMVG v1.0
Origin, relative measurements... axis constraint... we can imagine many... or use CCtag know sizes...


Incorrect result for $Call type with function that maps to object

Flow version: 0.109.0
Consider the following example:
type Call = $Call<<T>(T) => {|prop: T|}, string>;
({prop: 10}: Call); // <--- No error?

Expected behavior
The type Call would be equivalent to {|prop: string|} and the cast would fail with
({prop: 10}: Call); // <--- No error?
           ^ Cannot cast object literal to `Call` because number [1] is incompatible with string [2] in property `prop`.

Actual behavior
No error. Flow adds number to the type of prop:
$ flow type-at-pos --evaluate-type-destructors types.js 4 6
type Call = {|prop: (number | string)|}

Example at TryFlow

This now errors with inference_mode=lti on


New version: PencilFFTs v0.2.0

Registering package: PencilFFTs
Repository: https://github.com/jipolanco/PencilFFTs.jl
Created by: @jipolanco
Version: v0.2.0
Commit: bdd3c593f339bb752df28bf4068d9da7206b4442
Reviewed by: @jipolanco
Reference: https://github.com/jipolanco/PencilFFTs.jl/commit/bdd3c593f339bb752df28bf4068d9da7206b4442#commitcomment-36695175

Automerge fails because the package requires MPI.
[noblock]


updated functorch imports for PyTorch2 frontend

Hi,
I am looking into using the PT frontend for Alpa. The current code works for PT1.12 but breaks for PT1.13 and PT2. To get the tests in torch_frontend/* to pass for PT2, I had to update some of the functorch imports under alpa/torch/__init__.py. The tests pass but let me know if there is some additional testing that needs to be done to update the PT frontend.

Can you please fix the CI and update the branch?

@ZYHowell, I ran the linting script and rebased on the main branch. I think it still needs your approval to kick off the CI tests.

@ZYHowell, the runtime.test_device_mesh.DeviceMesh_ResourceAwareness is failing but I am not sure why. When I ran it locally (G5 ec2 instance with 4 GPUs), it failed the first time but when I ran it again, it seems to pass. Not sure what is going on here, seems unrelated to the PT frontend changes I made?

That seems like some stability issue of our CI. I reran the test and now it shows some torch-related error. Seems like this is because the new version api is not compatible with old versions. Can you please update your code to make it compatible? The anticipated code may be like:
if torch.__version__ < xxx:
  from functorch._C import xxx
else:
  from torch._C._functorch import xxx

Thanks @ZYHowell! I have added the version specific imports.


[Lesson Page] - Rendering issues

Why is there a rendering issue? Watch the attached video. First you get a messy and ugly flicker then the page loads.
https://js5.c0d3.com/assetExercise/renderingIssues.mp4

Fixed: https://github.com/garageScript/c0d3-app/pull/135


Pointing out that default alerts are configured to have mail sent to tenantadmins group

I wonder if there is an opportunity to point out in tis article that default alerts are automatically sent to Tenantadmins and who that implies (Global Admins).
Have had feedback from customers that this isn't really clear and has caused some confusion. Many  global admin roles do not have email and default alerts can be missed.
Is this documented anywhere else if not in this article?

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 9f2879ba-e58e-cd47-f488-10d153c31418
Version Independent ID: 391a2c05-d63d-e84e-7c7d-80c3b3518419
Content: Microsoft 365 alert policies - Microsoft Purview (compliance)
Content Source: microsoft-365/compliance/alert-policies.md
Service: o365-seccomp
GitHub Login: @robmazz
Microsoft Alias: robmazz

@dkotars Thank you for your feedback. We have made changes to the document via PR#https://github.com/MicrosoftDocs/microsoft-365-docs/pull/11962
Once the author approves, the changes will be live.
Thanks
Sri


Server responded with NO ['System Error (Failure)']

Hello jay0lee,
I'm having an issue with using gyb. Whenever I backup email from a users account it works successfully for some time. It will eventually be unable to continue. I tried deleting the backup folder and start it over but it always stops at around the same point. I tried service account and logging into the account to back up and neither one worked successfully. It seems like it stops after 10k-13k emails.
Here's the log:
Using backup folder<EMAIL_ADDRESS>Using folder [Gmail]/All Mail
GYB needs to examine 16304 messages
GYB already has a backup of 0 messages
GYB needs to backup 16304 messages
backed up 10500 of 16304 messages
Server responded with NO ['System Error (Failure)'], will retry in 2.0 secondbacked up 11200 of 16304 messages
Server responded with NO ['System Error (Failure)'], will retry in 2.0 seconds
Server responded with NO ['System Error (Failure)'], will retry in 4.0 seconds
Server responded with NO ['System Error (Failure)'], will retry in 8.0 seconds
Server responded with NO ['System Error (Failure)'], will retry in 16.0 seconds
Server responded with NO ['System Error (Failure)'], will retry in 32.0 seconds
Server responded with NO ['System Error (Failure)'], will retry in 64.0 seconds
Server responded with NO ['System Error (Failure)'], will retry in 128.0 seconds
Error: failed to retrieve messages.
NO ['System Error (Failure)']

Hello jay0lee,
I have the same problem! :(

So glad it's not just me!  I only noticed it yesterday after probably a month of not using it.

The new version (0.41) looks to have fixed the issues. Also good to hear GYB is moving to Gmail API! :smile:

Hello Jpt62089!
excuse me for delay! I'm using the script several months ago. This error only happened to me once. I will try the version 0.41.
Sorry for my English.
Regards


Unable to save to png format under Windows 10 using node

Despite many attempts, I cannot seem to save charts in "png" format (really any format, but that's what I want) under Anaconda w/Windows 10.  I have no issue on a Mac.
The last part of the error is always:
JSONDecodeError: Expecting value: line 2 column 1 (char 2)
A png file is created, but it appears to contain a command rather than its output.
# Test Case
import altair as alt
from altair_saver import save as alt_save
import numpy as np
import pandas as pd
# make simple data frame
test_data = pd.DataFrame.from_dict({ 'x': np.arange(0,10), 'y':np.arange(0,10)})
# make a chart with Altair
test_data_chart = alt.Chart(test_data).mark_line().encode(x=alt.X('x'),y=alt.Y('y'))
# display it (renders fine in notebook, using Chrome)
test_data_chart

# Test Case 1:
test_data_chart.save('testoutput.png')

---------------------------------------------------------------------------
JSONDecodeError                           Traceback (most recent call last)
<ipython-input-29-2049dff4a23c> in <module>
      1 # Test Case 1:
----> 2 test_data_chart.save('testoutput.png')

C:\ProgramData\Anaconda3\envs\myenvname\lib\site-packages\altair\vegalite\v4\api.py in save(self, fp, format, override_data_transformer, scale_factor, vegalite_version, vega_version, vegaembed_version, **kwargs)
    474         if override_data_transformer:
    475             with data_transformers.disable_max_rows():
--> 476                 result = save(**kwds)
    477         else:
    478             result = save(**kwds)

C:\ProgramData\Anaconda3\envs\myenvname\lib\site-packages\altair\utils\save.py in save(chart, fp, vega_version, vegaembed_version, format, mode, vegalite_version, embed_options, json_kwds, webdriver, scale_factor, **kwargs)
    119             webdriver=webdriver,
    120             scale_factor=scale_factor,
--> 121             **kwargs,
    122         )
    123         if format == "png":

C:\ProgramData\Anaconda3\myenvname\lib\site-packages\altair\utils\mimebundle.py in spec_to_mimebundle(spec, format, mode, vega_version, vegaembed_version, vegalite_version, **kwargs)
     58                 "see http://github.com/altair-viz/altair_saver/".format(fmt=format)
     59             )
---> 60         return altair_saver.render(spec, format, mode=mode, **kwargs)
     61     if format == "html":
     62         html = spec_to_html(

C:\ProgramData\Anaconda3\myenvname\lib\site-packages\altair_saver\_core.py in render(chart, fmts, mode, embed_options, method, **kwargs)
    255         Saver = _select_saver(method, mode=mode, fmt=fmt)
    256         saver = Saver(spec, mode=mode, embed_options=embed_options, **kwargs)
--> 257         mimebundle.update(saver.mimebundle(fmt))
    258
    259     return mimebundle

C:\ProgramData\Anaconda3\myenvname\lib\site-packages\altair_saver\savers\_saver.py in mimebundle(self, fmts)
     88                 vegalite_version=self._package_versions["vega-lite"],
     89             )
---> 90             bundle[mimetype] = self._serialize(fmt, "mimebundle")
     91         return bundle
     92

C:\ProgramData\Anaconda3\myenvname\lib\site-packages\altair_saver\savers\_node.py in _serialize(self, fmt, content_type)
    112
    113         if self._mode == "vega-lite":
--> 114             spec = self._vl2vg(spec)
    115
    116         if fmt == "vega":

C:\ProgramData\Anaconda3\myenvname\lib\site-packages\altair_saver\savers\_node.py in _vl2vg(self, spec)
     66             [vl2vg], input=vl_json, stderr_filter=self._stderr_filter
     67         )
---> 68         return json.loads(vg_json)
     69
     70     def _vg2png(self, spec: JSONDict) -> bytes:

C:\ProgramData\Anaconda3\myenvname\lib\json\__init__.py in loads(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
    346             parse_int is None and parse_float is None and
    347             parse_constant is None and object_pairs_hook is None and not kw):
--> 348         return _default_decoder.decode(s)
    349     if cls is None:
    350         cls = JSONDecoder

C:\ProgramData\Anaconda3\envs\myenvname\lib\json\decoder.py in decode(self, s, _w)
    335
    336         """
--> 337         obj, end = self.raw_decode(s, idx=_w(s, 0).end())
    338         end = _w(s, end).end()
    339         if end != len(s):

C:\ProgramData\Anaconda3\envs\myenvname\lib\json\decoder.py in raw_decode(self, s, idx)
    353             obj, end = self.scan_once(s, idx)
    354         except StopIteration as err:
--> 355             raise JSONDecodeError("Expecting value", s, err.value) from None
    356         return obj, end

JSONDecodeError: Expecting value: line 2 column 1 (char 2)

If I try specifying mode="vega", it succeeds and produces a file, but it just contains a command not an PNG file:
# Try vega
test_data_chart.save('testoutput.png',mode='vega')

# testout.png contains this text (not a png file):
C:\>node C:/ProgramData/Anaconda3/envs/myenvname\Library\share\vega-cli\node_modules\vega-cli\bin\vg2png ""

FWIW:  Python 3.7.10 | packaged by conda-forge | (default, Feb 19 2021, 15:37:01) [MSC v.1916 64 bit (AMD64)] on win32

This package has not been tested with Windows (I don't have access to any windows machine on which to test). Any help would be appreciated!

Happy to help if I can.

I was able to solve a very similar issue by installing the required packages globally
npm install -g vega-cli vega-lite canvas
which you have probably already done
When I installed vega itself my issues went away
npm install -g vega

@TravisNeils solution worked for me on Win10 as well. Is this caused by a dependency not being installed correctly on Windows?

I'm having the same issue saving Altair charts to any format from within VSCode-Python (VSCode v1.58.2) running both a Python script as well as a Jupyter-Notebook opened in VSCode. The error message is the same.
JSONDecodeError: Expecting value: line 2 column 1 (char 2)
I'm familiar with Node. Could you provide pointers on how to save Altair charts using node? I understand the npm install commands should be run from a node.js command prompt. How do I proceed from there to save my Altair charts?

Since Altair 5.2, the functionality of Altair Saver is now available in Altair via the vl-convert package. Most of the functionality has been available since 5.0, and the main addition in 5.2 was PDF export. See the docs on how to save charts for more details
We are going to archive this repo, so I'm closing all the open issues and PRs before doing so. Try out the new options for saving charts mentioned above and if you run into issues, please open an issue directly in the altair or vl-convert repo.


rustc: fix building emulated x86_64-darwin with jemalloc on aarch64-darwin

Description of changes

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
22.11 Release Notes (or backporting 22.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[ ] Fits CONTRIBUTING.md.

Should this be one or two commits? e.g. rustc: re-enable building with jemalloc on x86_64-darwin or something.

Two commits to show the difference between old and new code.

Two commits to show the difference between old and new code.

Assuming you mean "old" as in code that has previously been in the repo versus the new vaddr changes: fair, but part of me thinks it unnecessarily clutters up the commit history (why add code that isn't used in the first commit), though I'm probably just being too nitpicky.

unnecessarily clutters up the commit history

If it was a leaf package I'd agree.

@vcunat Can we include this in the new staging-next?

Hydra probably won't even get to build rustc for darwin for a couple days, so if you "finish" this PR roughly within the time, it sounds perfectly fine to me.

/cc PR #193494

Cool, I'll leave it open a bit longer but if there aren't any further comments I'll merge in 12 hours.

If it was a leaf package I'd agree.

By "leaf package", do you mean a non-dependncy, or something else? Not familiar with that terminology.
Also: did you mean to swap the commit ordering on the latest push? (It was originally fix then re-enable IIRC, but now it's the reverse. If it was intentional, that's fine (why though?), just noting it in case it was a mistake.)

Other than just clarifying what I asked in my previous comment (if you don't mind doing so, I really don't mean to be a bother), LGTM.


Modify rename function use tips

When using the rename function, not only can use the old name, you can also use the container ID, so I think the original use of the prompt is not accurate
Signed-off-by: zhouhao<EMAIL_ADDRESS>

changes look good to me. There are some other docs also need be updated such as https://github.com/docker/docker/blob/master/man/docker-rename.1.md
there may be more.

OK，I will change it

LGTM

LGTM


nxp: boards: Add device specific pinctrl data to boards

Add pinctrl-0 nodes to various nxp boards.  The set of boards selected here is based on which boards utilize the nxp,kinetis-uart

@Kludentwo & @henrikbrixandersen would appreciate any help in convert the other kinetis boards similar to what I've done here.

@Kludentwo & @henrikbrixandersen would appreciate any help in convert the other kinetis boards similar to what I've done here.

I'll start working on this tomorrow.

I'll start working on this tomorrow.

Thanks, I've got issue #34145 to track the progress.

Build issue is unrelated.


Minor changes to ocean recipes & diagnostics for gmd-2019-291

I've had to make a few minor changes to a couple of the ocean recipes for the  ESMValTool v2.0 Part 2 paper. It would be nice to push these changes through to the main branch before the paper is accepted for final publication.
Some of the changes include:

additional fields in the recipe_ocean_scalar_fields.yml
improvements to some of the plotting tools.
bug fixes in the diagnostic.

More details and a PR will follow.
Discussions paper: https://www.geosci-model-dev-discuss.net/gmd-2019-291/

Just to summarise (mostly for my own notes), the diagnostics/ocean plots made the figures:

Figure 21. The volume weighted global mean temperature anomaly in several CMIP5 models,
Figure 22. The Atlantic Meridional Overturn Circulation (AMOC) in several CMIP5 models
Figure 23. The Antarctic circumpolar current calculated through Drake Passage for a range of CMIP5
Figure 24. The global total air to sea flux of CO2 for a range of CMIP5 models
Figure 25. The global total integrated primary production from phytoplankton for a range of CMIP5
Figure 38. The surface dissolved nitrate concentration in the CMIP5
Figure 39. The surface dissolved nitrate concentration in the CMIP5 HadGEM2-ES model
Figure 40. The global area-weighted average depth profile of the dissolved nitrate concentration in the CMIP5

These figures are produced by:

recipe_ocean_scalar_fields.yml: figures 21.
recipe_ocean_amoc.yml: figures 22, 23
recipe_ocean_scalar_bgc.yml (new in PR #1621): figures 24, 25
recipe_ocean_bgc.yml: figures 38, 39, 40.


Refactor scoped_model to be more scoped_modelly

Based on my current knowledge of scoped_model I refactored this todo app based on PR #8.
Changes:

AppState and StateContainer are now one thing -> TodoListModel, wrapping the application.
Detail and Edit will be started with the Todo id instead of the complete Todo object, loading it afterwards from the TodoListModel.
Todo is now immutable. Use .copy() to change the content.
Instead of passing all lambdas into ExtraActionsButton and FilterButton they now communicate directly with TodoListModel. This reduces the logic in HomeScreenState#build.
HomeScreen doesn't wrap the Scaffold in ScopedModelDescendant. This should improve the performance. The HomeScreen logic contains now nothing more than conducting the tabs and action items. All logic regarding the todo list items is now deeper in the Widgets.
The HomeScreen was triggering the loading of data from the repository. This is now the responsibility of the TodoListModel when a new listener gets added (see addListener). This could be improved by checking the listener count.

Amazing, thanks so much! Lemme give it a review :)

Really appreciate the work! Would love to chat about a few points, as they're things I've faced / wondered about when developing these apps. Interested in your thoughts as well :)

Combine AppState and StateContainer -- I tried this out as well, but was on the fence as to which approach was better. Good to have a second opinion, let's keep em together!
What do you think about the advantages / disadvantages of making Todo immutable? In some architectures (Redux) it fits the pattern, but for scoped_model and regular StatefulWidgets, it felt like it made the app a bit more complex. I felt like those architectures worked better with mutable values. Why? Once Todo is immutable, you need to pass an id to the Detail and Edit screens instead of the Todo directly. Then, you also need to create a function to look up Todos by id. On the Delete screen, this will cause an interesting problem -- after the Todo is deleted, the function will return null and you fall back to new Todo('');, which means the UI shows a blank Todo as the screen animates away. While it's not a huge detail, I see it as an interesting case because it shows where Reactive, Immutable Models need some care when interacting with Flutter. If Todos were mutable, you'd be able to pass the instances directly to the Edit and Detail screens, and wouldn't run into the issue with animations, for example. What are your thoughts?

Thanks again! One final, unrelated question:
Do you think I should move this repo to a new Github org? Would that make you more comfortable contributing? (I'll add your name to the Readme for sure)

Immutable Todo
Previously, to change the check state, the container does the actual change (container.updateTodo(todo, complete: !todo.complete);). The container first changes the data, then notifies the listeners and sends the changes to the repository.
Two things could go wrong:

Wrong usage for updates: One could call todo.complete = !todo.complete without calling the container. This would work in some cases when build gets called due to another change but it bypasses possible validation logic in the container. An immutable Todo forces anyone to use the container.
Missing subscription, implicit subscriber: With a reference to the mutable object it's not required to use a ScopedModelDescendant to get updates. Any setState updates the UI and could show it correctly (by accident without explicit subscription). The ScopedModel, without real subscribers, could for example stop polling changes from remote because it thinks nobody is interested in the data.

Detail view and edit could be a named routes:
When working with ids instead of PODOs to pass content around named routes can be used. Currently creating a new todo and editing an existing one requires to different ways to push the same AddEditScreen.
// add Todo, pushes AddEditScreen with todo == null
Navigator.pushNamed(context, ArchSampleRoutes.addTodo);

// edit Todo
Navigator.of(context).push(
  new MaterialPageRoute(
    builder: (context) {
      return new AddEditScreen(
        todo: todo,
        key: ArchSampleKeys.editTodoScreen,
      );
    },
  ),
);

Both cases should use Navigator.pushNamed, as well as the detail view. It helps later on with deep linking.
Deleting the todo before the exit animation ends
This problem was quite hard and unexpected. I'm not happy with my current solution. I should show an empty container or a centered "Not found/Deleted" text if the Todo was deleted instead of creating a new empty Todo. This would also cover the case someone starts this route with an invalid Todo id.
The correct solution would be to keep the Todo in the model, marked as deleted, and wait some time before actually deleting it (after the snackbar disappeard). Maybe in my next PR.

Currently I don't care where this repo is located. In the long term I see it here analog to android-architecture

Hey, sorry for the delay. Busy one! Very interesting points, thanks so much for sharing those insights...
Immutable Values
Hrm, tricky one indeed -- I definitely see your points. Making it immutable indeed forces the user to call up to the parent function, which will ensure the right call is made and that subscribers
What would you think about turning the Todo class into a scoped_model as well? Then the Detail page and Edit page could use ScopedModelDescendant<Todo>?
The user could then use mutable values (which call notify on the set methods), and the Screens would be updated properly? It would also solve the null on deletion problem....
Might just give that a whirl to see what it's like.
Deep Linking
Good call -- I've thought about making handling a /detail/:id route a requirement of the app, because of the importance of deeplinks.
If the Todo as Model idea from above works, I could see a path toward supporting this functionality by keeping the getTodoById function.
Deleting the todo before the exit animation ends
Yah, I was surprised by this problem as well :D Tough one to solve in some frameworks...
** Repo location **
Cool, I'll keep it here for now...
** Thanks again! Really appreciate everything!!! **

Hey hey :)
As a thought experiment, I changed the Todo into a TodoModel to see if that might feel like a cleaner API. I think some things are better, others are worse. Would love to hear your opinion!

TodoItem, AddEditScreen, and DetailsScreen no longer accept a Todo or todoId. They are now wrapped in a ScopedModel<Todo>, and must use a ScopedModelDescendant<Todo> in the build method to get the Todo the Widgets require.
The Todo is once again mutable, but replaced by get and set methods. The set methods set the value and then perform a notifyListeners() so the Widgets are rebuilt.
The todo_list listens for changes on all Todos. If a change is made to a Todo, the change will be stored in localstorage.
For funzies, wrote a named Detail route just to be sure that would work.

Benefits:

Both TodoListModel and TodoModel have mutable interfaces (consistency)
You can now do a simple todo.complete = !todo.complete and your Widget will update!
Fixes the problem with Todo screen being blank upon Deletion from the Detail page

Drawbacks:

Still nothing preventing you from just passing a todo or todoId and forgetting to use a subscriber...
Not sure if this would be perceived as easier to understand, or is still prone to the mistakes you mention (forgetting to subscribe)

https://github.com/brianegan/flutter_architecture_samples/commit/c5ad8a5fa66def865e14371684b40b79c5b05879

Interesting, I think I understand your intention although I don't agree with making Todo a  TodoModel.
Let's get back to what a Model is:
/// Base class for classes that provide data via [InheritedWidget]s.
abstract class Model {

A Model provides data but it is not the data itself. By changing the immutable PODO Todo to a TodoModel we again lose layers. I strongly belief Entities are very valuable in every architecture and shouldn't be mixed with higher layers such as UseCases or Controllers.
This combination already introduced an odd case:
The ListView listens to TodoListModel and the items are wrapped in ScopedModel<TodoModel> also listening for changes. A single item will never rerender alone. The itemBuilder will be called already for every change.
What I really like about you change is that you end up with one Model per Screen as described here:

Properly scope your inherited widgets
Inherited widgets are placed at the root of a widget tree. That, in effect, becomes their scope. In our team, we found that the power to declare context anywhere in the widget tree is a bit too much. So, we restrict our context widgets to accept only Scaffold (or its derivatives) as a child. That way, we ensure the most granular context can be at the page level ending up with two types of scope:

App-scoped widgets such as MediaQuery. These are accessible by any widget on any page in your app since they sit at the root of your app widget tree.
Page-scoped widgets such as the MyInheritedWidget in the example above.

When I started with refactoring, I also created a model for each screen and wrapping screens in that model before pushing a Route. I worked but I ended up with tons of duplicated code because the functionality of all screens is so similar. Instead of extracting logic in reusable components (UseCases) I promoted the TodoListModel to be an App-scoped Model, accessible by everyone.
If there would be a different functionality such as inviting other users to my list, I would create a Model for that screen which requires logic the rest of the app won't ever use.
Instead of a global TodoListModel I could think about this:

keep the immutable Todo and use it to show data on the screen.
create a TodoModel with CRUD methods changing a single Todo and sync it directly with the repository. (Id in constructor, no access to other Todos)
Use TodoListModel to sort the list and remove update and delete. Right now the TodoDetail page could change the list ordering.
Create a StatsCounterModel calculating the stats by getting data from the repository

This would create some code duplication but will be a better guidance how scoped_model can be used in larger applications.

Guten Morgen :) Thanks again for taking the time to work through this with me. Really appreciate it!
Overall, I think the changes you propose in the end make a lot of sense. Could we dig just a bit deeper on one point for clarity?
"A Model provides data but it is not the data itself. By changing the immutable PODO Todo to a TodoModel we again lose layers."
I think agree on this point, but I believe I misunderstood your original change set a bit, and therefore extended your approach in the wrong way. Originally, I was trying to follow a layered pattern you describe and create by having two classes: AppState, which was a PODO that holds the logic and a and a separate Model called StateContainer that interacted with the PODO. In your PR you combined the two into a TodoListModel.
Could you describe the reasoning behind combining the two a bit more? The reason I thought you might have done it was to remove code duplication (similar to the code duplication you talk about at the end of your comment).


Add partial support for HTMLElement.dataset

As brought up by issue https://github.com/tmpvar/jsdom/issues/961.
This PR supports getting and setting existing data-* attributes. From a brief conversation on IRC it seems like implementing support for adding new data-* attributes would be nearly impossible until Object.observe is available. Not sure if a partial feature is merge-able, but submitting the PR anyway to open the conversation.
Supporting HTMLElement.dataset also requires an implementation of DOMStringMap, which is included here. I was not sure where to put that implementation, so just let me know where the most natural place is for it.

Yep without Proxy there is no way to intercept access to arbitrary properties and Proxy is still not available in nodejs (v4.2.0). However I need it to unit-test CommonJS modules of in-browser app. So here a workaround which works for data-attributes already set it HTML, but of course ignore any new added as properties of dataset object. The polyfill:
 /**
     * Convert data-attr into key
     * data-foo-bar -> fooBar
     * @param {String} val
     * @returns {String}
     */
var _attrToDataKey = function( val ){
      var out = val.substr( 5 );
      return out.split( "-" ).map(function( part, inx ){
        if ( !inx ) {
          return part;
        }
        return part.charAt( 0 ).toUpperCase() + part.substr( 1 );
      }).join( "" );
    },
    /**
     * Produce dataset object emulating behavior of el.dataset
     * @param {Element} el
     * @returns {Object}
     */
    _getNodeDataAttrs = function( el ){
      var i = 0,
          atts = el.attributes,
          len = atts.length,
          map = [],
          proxy = Object.create({}),
          attr,
          datakey;
        for ( ; i < len; i++ ){
          attr = atts[ i ].nodeName;
          if ( attr.indexOf( "data-" ) === 0 ) {
            datakey = _attrToDataKey( attr );
            map[ datakey ] = atts[ i ].nodeValue;
            Object.defineProperty( proxy, datakey, {
              get: function() {
                return map[ datakey ];
              },
              set: function ( val ) {
                el.setAttribute( attr, val );
              }
            });
          }
        }
        return proxy;
    };

Object.defineProperty( global.window.Element.prototype, "dataset", {
  get: function() {
    return _getNodeDataAttrs( this );
  }
});

Just include it in the callback when the DOM is created. Usage example:
jsdom.env({
      html: html,
      done: function ( err, window ) {
        global.window = window;
        require( "./polyfill-jsdom" ) );

        // in the HTML <body data-bind="old value">
        var el = window.document.body;
        console.log( el.dataset.bind ); // old value
        el.dataset.bind = "new value";
        console.log( el.dataset.bind ); // new value, <body data-bind="new value">
        console.log( el.getAttribute( "data-bind" ) ); // new value

      }
    });

@dsheiko  According to your method of errors occurred.
"before all" hook:
     TypeError: Cannot redefine property: dataset

@jaywcjlove sorry it was buggy, now is claimed fixed (see the changes in my previous comment)
tested with Node v4.2.0 and<EMAIL_ADDRESS>

@dsheiko global.window.Element is not this prototype method.I have tested in Node v4.2.0 and<EMAIL_ADDRESS>

That's just weird. Pls, check on your side, when you receive window object from jsdom (as a parameter of done callback) there must be window.Element interface defined

Please download the GIST https://gist.github.com/dsheiko/37e4673cce20d5896510
unpack the files to an arbitrary directory
run npm i
run node test.js

On Linux (Ubuntu) it works fine. I will check later on MacOS and Windows, but supposed to be no difference.

@dsheiko Thank you, By your method,I solved the problem.


How to run the cli when I don't have a package/module set up?

I have a python setup where I've got some relatively simple scripts that interact with a fairly complex database that I've wrapped with SQLAlchemy.  Here's the structure:
working dir with script1.py, script2.py
|--- db/
|    |--- __init__.py (empty)
|    |--- models.py

No matter what I've tried to pass to paracelsus on the command line, I can't get it to load my base class.  Any ideas on what I can do to use this tool?

What file contains the base model class, and what is it named?

My original post had some bad formatting, I updated it to make the structure clearer.
To answer your question, my models.py contains my Base class:
class Base(DeclarativeBase):
    pass

What's the exact command you ran and the output?

(venv) justine@mbp python % paracelsus graph db.models:Base
Traceback (most recent call last):

  File "/Users/justine/development/project-code/python/venv/bin/paracelsus", line 8, in <module>
    sys.exit(app())
             ^^^^^

  File "/Users/justine/development/project-code/python/venv/lib/python3.12/site-packages/paracelsus/cli.py", line 94, in graph
    get_graph_string(

  File "/Users/justine/development/project-code/python/venv/lib/python3.12/site-packages/paracelsus/cli.py", line 44, in get_graph_string
    base_module = importlib.import_module(module_path)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File<EMAIL_ADDRESS>line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import

  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load

  File "<frozen importlib._bootstrap>", line 1310, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed

  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import

  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load

  File "<frozen importlib._bootstrap>", line 1324, in _find_and_load_unlocked

ModuleNotFoundError: No module named 'db'

(venv) justine@KRY-MB-AE-034 python % ls db
__init__.py	__pycache__	models.py

Thanks for reporting!
This could be resolved with the python-dir option, like so:

paracelsus graph db.models:Base --python-dir=.

That being said I think it makes sense to add the current working directory there automatically, so pull request #6 added that small change in. If you upgrade to the latest release your command should work without any changes.

Excellent, thanks so much.  I can verify 0.3.0 works for me!


obs-vkcapture does not work with 32 bit games

Describe the bug
I just did a fresh install of bazzite, so everything should be as default as it gets. The only package I've layered is mullvad-vpn. I set up my Steam library using the preinstalled native Steam, and moved my old Bottles (flatpak) games into new Lutris (native) prefixes, then installed OBS, the VK capture plugin, and the flatpak vulkan layer through the terminal. I did NOT install OBS from the first boot portal, so if that does anything special, that might be my whole problem.
After adding obs-gamecapture %command% into the launch options of a couple of my games, nothing seemed to be working.
After pulling my hair out over it and troubleshooting for several hours, including testing with both vkcube and glxgears (both captured fine), I realized that apart from one game, everything I was testing was very old, so I tried loading a more recent game, and it captured fine.
I also tested installing Bottles as a flatpak and running one of my old games through that instead. It captured properly. Out of curiosity, I rebased to testing just to see if there's been an update that fixes whatever's going on here, but nothing changed.
That made me think that for some reason, I didn't have the 32 bit version of the native vkcapture package installed, but rpm -qa | grep obs returns the following:
obs-vkcapture-common-0.0.git.354.aa3c43a0-1.fc41.noarch
obs-vkcapture-vulkan-layer-common-0.0.git.354.aa3c43a0-1.fc41.x86_64
libobs_vkcapture-0.0.git.354.aa3c43a0-1.fc41.x86_64
libobs_glcapture-0.0.git.354.aa3c43a0-1.fc41.x86_64
libobs_vkcapture-0.0.git.354.aa3c43a0-1.fc41.i686
libobs_glcapture-0.0.git.354.aa3c43a0-1.fc41.i686

The 64 bit files are in /usr/lib64 and what I assume to be the 32 bit files are in /usr/lib
So, in short:

OBS installed as flatpak, including necessary flatpak plugin and vulkan layer
64 bit games running through native Steam or native Lutris capture as expected
32 bit games running through native Steam or native Lutris do NOT capture
32 bit games running through flatpak Bottles DO still capture
For whatever reason, the 32 bit version of native vkcapture is not being loaded by flatpak OBS

What did you expect to happen?
I expect to be able to enter obs-gamecapture %command% in the launch options of any game and have capture functionality working, regardless of whether the game is 64 or 32 bit, and regardless of vulkan or opengl.
Output of rpm-ostree status
State: idle
Deployments:
● ostree-image-signed:docker://ghcr.io/ublue-os/bazzite:stable
                   Digest: sha256:777e4de18ea2dd200e214723a0832cefe01cf10cb9830b05665b95f88ab8760d
                  Version: 41.20241216 (2024-12-16T05:02:15Z)
          LayeredPackages: mullvad-vpn
            LocalPackages: lact-0.6.0-0.x86_64

  ostree-image-signed:docker://ghcr.io/ublue-os/bazzite:testing
                   Digest: sha256:30dd06d2d4e64592505b3c5bfe9c5c1f71911aa50e0b1d978f0fa5792b368e25
                  Version: testing-41.20241225.1 (2024-12-25T06:10:57Z)
          LayeredPackages: mullvad-vpn
            LocalPackages: lact-0.6.0-0.x86_64

Hardware
No response
Extra information or context
No response

Will be fixed in the next build

Rebased to testing again and confirmed, updating from 354 to 355 fixed it. Thank you.


Indicate that the implicit grant has a 5-group limit

The groups claim is not only restricted to 200 groups in JWTs, it also has a 5-group limit if the token is issued through the implicit flow.  Please update the docs to indicate that tokens requested via the implicit flow will only have a "hasgroups":true claim if the user is in more than 5 groups.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: c5a6779a-58f1-73fa-2b24-5f85deeea52b
Version Independent ID: 481c093b-6db3-c4e5-6644-b1a3a4497a30
Content: Configure group claims for applications with Azure Active Directory
Content Source: articles/active-directory/hybrid/how-to-connect-fed-group-claims.md
Service: active-directory
Sub-service: hybrid
GitHub Login: @billmath
Microsoft Alias: billmath

I have created an internal work item for this.  I need to verify this with engineering.
#please-close


chore: bdd-spring-1573152110 to 0.0.1

chore: Promote bdd-spring-1573152110 to version 0.0.1

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approvers:
If they are not already assigned, you can assign the PR to them by writing /assign  in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


Added a readthedocs.yml config file

Our ReadTheDocs builds have been getting stuck in a "Cloning" state indefinitely (instead of building through to completion). See:
https://readthedocs.org/projects/bigchaindb/builds/
When I searched to see if others were having the same problem, I found that there's a new readthedocs.yml config file in beta and that might be able to help.
I made a readthedocs.yml file. I think the only change it makes is to switch from the default of using Python 2 to using Python 3. I don't know if that will help or not. Locally, I always build the docs using Python 3 (because I use the same virtualenv as for developing BigchainDB).

[Current coverage][cc-pull] is 89.87%

Merging [#452][cc-pull] into [master][cc-base-branch] will not change coverage

@@             master       #452   diff @@
==========================================
  Files            18         18
  Lines          1274       1274
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
  Hits           1145       1145
  Misses          129        129
  Partials          0          0

[][cc-pull]

Powered by Codecov. Last updated by [c7aa2b1...694bf9f][cc-compare]
[cc-base-branch]: https://codecov.io/gh/bigchaindb/bigchaindb/branch/master?src=pr
[cc-compare]: https://codecov.io/gh/bigchaindb/bigchaindb/compare/c7aa2b198444c08768c3c1070d5965f275ce1088...694bf9f83a7df30e1517e8d26c170b1b595a0fbe?src=pr
[cc-pull]: https://codecov.io/gh/bigchaindb/bigchaindb/pull/452?src=pr

When I merged this pull request (which added a readthedocs.yml config file), ReadTheDocs got unstuck and finished building the docs through to completion. I don't consider that a definitive proof that this trick worked, but it's promising!


Evaluate response for indexing request if the remote store interaction fails

Is your feature request related to a problem? Please describe.
We should evaluate if the response being thrown at the time of indexing during a remote store outage shares only the required information.
Describe the solution you'd like
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.

@ashking94 please add details around the proposed solution

Sure @gulgulni. So, lets assume we are using S3 as the underlying remote store. So, if S3 was to throw a 500 or 503, then the associated exception that the SDK throws corresponding with these status codes would also be shown in the response message of the bulk or index api.


Analytics: Top Lender

As an active User from bookkeeper red, I would like to benefit from all the data that is generated during the lending processes. Therefore, I would like to have an analysis of who borrows the most items from me / a group from me.

Dependencies
Acceptance Criteria

[ ]  It's possible to select my personal_group or any group, which I am a member of.
[ ] The table will update after a selection.
[ ] The table shows all lenders from me/group, sorted by frequency of lending.
[ ] Only a limited number of items (e.g. 7) should be displayed at the same time (therefore the integration of a scroll view).

Linked with #303


!i Error: Could not fetch item information!

I just set pho_inspect_url to inspect.pricempire.com API but still get error and fetching item information not working by link
Server logs
[SM] Value of cvar "pho_inspect_url": "https://inspect.pricempire.com"
I still get error
[PHOSKIN] Fetching item information...
[PHOSKIN] Error: Could not fetch item information!

Please provide Server Logs and not your chat logs please.

I assume you have read the "Important" section in the readme?
There Ive said that the cvar has to be set before starting the plugin. Did you stick to it.

Yes, i'm not sure what execution sequence of files, but i have sm_cvar pho_inspect_url "https://inspect.pricempire.com" in my server.cfg file, after setting it, i restarted server and still get error.
Which one log do you want to see? /csgo/addons/sourcemod/logs errors file and log file do not have any issues, i can provide any log file you request

Please use normal cvars.
Like in the game itself.
Just put pho_inspect_url "https://inspect.pricempire.com" into it

I've tested it with normal pho_inspect_url and nothing changed

Then please provide your server console output.

L000_000_000_000_27015_202305071939_000.log
L000_000_000_000_27015_202305071939_001.log
L000_000_000_000_27015_202305071940_000.log

Changing it in the "main.sp" file and re-compiling worked for me!
somehow just adding it into the server.cfg wasn't enough even tough the console says it changed correct.

Changing it in the "main.sp" file and re-compiling worked for me! somehow just adding it into the server.cfg wasn't enough even tough the console says it changed correct.

Very interesting... thank you for the info. I don't have time at the moment testing things myself.
Would you mind testing it with the potential fix I’ve pushed?
Use the artifact from the job please:
https://github.com/araynimax/phoskin/actions/runs/4942994543

@pr0stre1 Sorry for the late reply.
Currently, I unfortunately have no time to test things myself. I hope that the problem had to do with the readout of the CVAR and should be fixed with my latest commit. (If someone confirms that it will work now, I will push a new release) You can test it with artifact of the latest build job (Read my previous comment).

No problem. I have tested your fix, now problem solved and all works fine, thank you

I'm glad to hear that. Have fun :)

New release published - issue closed

Having same issue currently, running v1.2.2, https://csgo.padow.ru


lookup of field_to_match conditional assignment issue

Hiya I am not sure if this is a bug or more likely that my approach is not correct.
Can you please investigate this issue.
I created a variable for a byte_match_statement_rule which uses 'single_header' for the field_to_match.
I am intending to scan the 'user-agent' header for a particular string.
terraform plan fails with the error shown below:
`
[jthomas@UKRHEL-MSR-01 exampleTwillo]$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

Error: Unsupported attribute
on .terraform/modules/example_wafv2.wafv2/rules.tf line 176, in resource "aws_wafv2_web_acl" "default":
176:                     name = single_header.value.name
|----------------
| single_header.value is 1
This value does not have any attributes.
`

Expected Behavior
A clear and concise description of what you expected to happen.
Steps to Reproduce
Steps to reproduce the behavior:

Define variable with the following code:
`variable "byte_match_statement_rule_400" {
type = list
default = [
{
name     = "byte_match_tillowHeaderUserAgent-400"
action   = "allow"
priority = 400
statement = {
positional_constraint = "CONTAINS_WORD"
search_string         = "myHeader"
text_transformation = [
{
priority = 30
type     = "COMPRESS_WHITE_SPACE"
}
]
 field_to_match = {
   single_header = { data = { name = "user-agent" } }
 }

}
visibility_config = {
cloudwatch_metrics_enabled = false
sampled_requests_enabled   = false
metric_name                = "rule-400-metric"
}
}
]
}`

Run 'terraform plan'

See error
`Error: Unsupported attribute

on .terraform/modules/example_wafv2.wafv2/rules.tf line 176, in resource "aws_wafv2_web_acl" "default":
176:                     name = single_header.value.name
|----------------
| single_header.value is 1
This value does not have any attributes.`
Screenshots
byte_match_statement variable:

modified dynamic single_header lookup from the local copy of rules.tf:

Environment (please complete the following information):
OS: RHEL 7.9,
Terraform v0.13.6,
provider registry.terraform.io/hashicorp/aws v3.37.0
Additional Context
Add any other context about the problem here.
I was able to get the 'terraform plan' command to complete successfully by making the following changes:

Edit the the local copy of the rules.tf file { '.terraform/modules/example_wafv2.wafv2/rules.tf' } from the downloaded source module and remove the '!= null' conditional assignment found on line 173
for_each = lookup(field_to_match.value, "single_header", null) != null ? [1] : []
so that it now becomes:
for_each = lookup(field_to_match.value, "single_header", null)
and then run the 'terraform apply' command which will then complete successfully.

I would also like to ask if the field_to_match structure is correct:
       field_to_match = {           single_header = { data = { name = "user-agent" } }         }
I would have thought that the following structure was expected:
        field_to_match = {           single_header = { name = "user-agent" }         }
but then you would have to modify the name assignment on line 176 of the dynamic single_header lookup in the same rules.tf file {removing the '.name'} as follows:
`
dynamic "single_header" {
for_each = lookup(field_to_match.value, "single_header", null)
              content {
                name = single_header.value
              }
            }

`
Thanks for your help.
Jay.

I would be happy to modify the code and raise a pull request if required, just let me know.

This is also the case for the dynamic config block for regex rules ("regex_pattern_set_reference_statement")
I will shortly raise a PR of my own to address the issue.

@JayTeeVF can you test this issue against https://github.com/cloudposse/terraform-aws-waf/releases/tag/0.0.4 and let me know if you run into the same issue? I believe @dannyibishev fixed this.

It's still broken on the latest.  I have opened PR https://github.com/cloudposse/terraform-aws-waf/pull/19 to address it.


Add bi-directional bindings support

This PR does the following:

Adds bi-directional support for bindings with enforced validation the operation field
Updates proto service
Updates existing unit tests
Adds new test cases
Updates e2e test

Closes #1116

@youngbupark we need to merge this first and then run e2e, right?

Please run e2e test once this pr is merged.

@youngbupark we need to merge this first and then run e2e, right?

Yes.

/ok-to-test

@yaron2 bindings e2e test is failling..
    TestBindings: bindings_test.go:115:
        	Error Trace:	bindings_test.go:115
        	Error:      	Not equal:
        	            	expected: []string{"1", "2", "3", "4", "5", "6", "7", "8", "9", "10"}
        	            	actual  : []string(nil)

        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1,13 +1,2 @@
        	            	-([]string) (len=10) {
        	            	- (string) (len=1) "1",
        	            	- (string) (len=1) "2",
        	            	- (string) (len=1) "3",
        	            	- (string) (len=1) "4",
        	            	- (string) (len=1) "5",
        	            	- (string) (len=1) "6",
        	            	- (string) (len=1) "7",
        	            	- (string) (len=1) "8",
        	            	- (string) (len=1) "9",
        	            	- (string) (len=2) "10"
        	            	-}
        	            	+([]string) <nil>


enums in modules can't be typed

Problem 1
module Foo { enum Bar { ...

eventually produces something like
(function (Foo) {
    /** @typedef {number} */
    var Bar;

But you apparently can't (?) put a @typedef within a scope like that, it has to be top level (?).
Problem 2
declare module Foo { enum Bar { ...

after sickle we produce the TypeScript code
declare module Foo {
  [...]
/** @type {Bar} */
(<any>Bar).BAZ =  1;
}

and you can't have statements in a declared module.  I think the workaround is to skip them in this case but I don't really know what the original code is trying to do...

Is it high priority to support "module"? We outlaw it in the style guide,
so it'd only come in through third party code...

(To answer Martin's question publicly, we encounter a module in a google-internal app.)

I worked around this in the Google-internal app by moving the relevant code to a d.ts file (which we don't Sickle-process), but this bug stands.

Moved part 2 to issue #74.

I fixed enums to not use a TS enum.


Dev

The AuthenticationInfo parameter needs to be an 'array' of strings, otherwise using it fails with Convert property 'AuthenticationInfo' value from type 'STRING' to type 'INSTANCE' failed. See http://serverfault.com/questions/788733/how-to-use-authenticationinfo-in-xwebapplication-dsc-resource/788921#788921

This change is

Hi @hahndorf, I'm your friendly neighborhood Microsoft Pull Request Bot (You can call me MSBOT). Thanks for your contribution!
You've already signed the contribution license agreement. Thanks!
The agreement was validated by Microsoft and real humans are currently evaluating your PR.
TTYL, MSBOT;

So with my changes the tests fail, but in a real life example using AuthenticationInfo only works after my changes. Are the tests wrong? Do I do something wrong?

It looks like you only updated the .mof file and not the .psm1 file

Did you check out the errors that came up for the build?

I tried to fix someone's problem on serverfault.com but it turned out my fix was completely incorrect. So I will delete this pull request.


Disable logging of exceptions in production environment

Currently, every exception will be logged in the logfile which is unnecessary. Unfortunately, there is no possibility to disable it.
This method should not be called on HttpKernel or Dingo specific exceptions to prevent a fast growing log file with unnecessary entries.
An example of a thrown and logged exception:
production.ERROR: exception 'Symfony\Component\HttpKernel\Exception\UnauthorizedHttpException' with message ...
Stack trace:
....

No I think it's more than that. The default kernel has the ability to not report certain exceptions, which I think HttpException is a part of. So, by default, none of those exceptions are being reported. I'll need to mimic this functionality as best I can, as errors still need to be reported, just not all of them.

Any update on this? Also needed to disable logging entirely on production for now.

A fix for this will be pushed in the next update.

Awesome thanks Jason!

Perfect, thanks Jason :+1:


Sparse rule for gather

Hello,
Are there any plans to implement the sparse rule for gather? I'd like to do something like the following MWE:
import jax.experimental.sparse as sparse
import jax.numpy as jnp

A = jnp.eye(2)
A = sparse.bcoo_fromdense(A)
A0 = A[0]

which raises NotImplementedError: sparse rule for gather is not implemented.

We'd like to implement this, but gather is a quite complicated primitive so it will take a lot of work.

I see, that makes sense. Could you please point me to relevant resources about implementing gather? I'm interested in potentially taking a crack at it in my spare time.
In the meantime, perhaps you can advise on the specific use case I have and suggest improvements that don't involve gather? I'm the author of a small library, Connex, that I'm working on as a personal project. The primary functionality is to turn any DAG into a trainable neural network. For maximal parallelism, the forward pass involves using vmap to "fire" neurons within topological batches (neurons are topologically sorted and then batched/grouped by topological equivalence, i.e. for any two neurons in a topological batch, there does not exist a path between them in either direction).
However, since vmap is designed to work only with arrays of the same shape (i.e. not "ragged"), and neurons within a topological batch may have different numbers of input connections, I use masking in the following way:
The parameters are stored in a parameter matrix P, where P[i, j] is neuron i's weight for neuron j, and P[i, -1] is neuron i's bias. At the start of each forward pass, P[:, :-1] is multiplied by a transposed adjacency matrix to zero out parameters where there does not exist a connection.
Then, jnp.dot is vmap-ed only in the first argument, taking the dot product of the respective rows of P with a 1D value array that's updated after each topological batch is fired (and activations and such are applied). The forward pass is implemented at https://github.com/leonard-gleyzer/connex/blob/main/connex/network.py#L138.
This approach scales unfavorably (quadratically in terms of memory) with respect to the number of neurons, regardless of which parameters are actually used. As the network grows larger, training becomes noticeably slower (especially on a CPU), e.g. comparing training an MLP in Equinox with the equivalent MLP in Connex.
Do you have any suggestions to increase efficiency/performance?
Thank you!

Could you please point me to relevant resources about implementing gather?

There aren't really any resources on this... this is experimental code with not much documentation coverage at the moment. The relevant sparse translation rules can be found here: https://github.com/google/jax/blob/main/jax/experimental/sparse/transform.py. There are several examples of adding support for primitives there; for example here's how lax.reduce_sum_p is implemented: https://github.com/google/jax/blob/5f7018a62ed0fe3e5f6898d99c71399f688d1abb/jax/experimental/sparse/transform.py#L593-L604
To implement sparse gather would look similar... however as I mentioned gather is a very complicated primitive, because it supports all modes of numpy indexing (i.e. scalars, slices, boolean masks, and index lists). You can read about the semantics of gather here: https://www.tensorflow.org/xla/operation_semantics#gather, and you can see the Python wrapper of this API here: https://github.com/google/jax/blob/55dcbec5b5a00d5ba2f2c29e7bb3c2f24fc540b6/jax/_src/lax/slicing.py#L280-L286
To implement the sparse rule for gather would require creating a function that accepts all those parameters and then computes the appropriate outputs given a BCOO data structure.

That said, if you want only a particular kind of indexing (from your description it sounds like all you're concerned about is mat[:, :-1]?) Then you may be able to write your own function that does just that specific transform by operating on the underlying data, indices, and associated metadata.

Thanks for the info!
I tried writing my own custom sparse function, but that actually made it significantly slower than the masking approach. It seems that the data is not sparse enough to benefit from sparse representation. Is there anything else you can recommend, maybe a smarter way of masking that generates more efficient XLA code?

After trying around 5 or 6 different things, I've come up with a solution where both JAX and I are happy! The idea is to use the topological ordering of the neurons as the internal ordering for the weights and neuron values, since this order minimizes the distance between the minimum and maximum array indices needed to process a topological batch in parallel using vmap. That means that each topological batch weight matrix has shape (len(topo_batch), max_idx - min_idx + 1) rather than (len(topo_batch), num_neurons), where min_idx and max_idx are the minimum and maximum indices, respectively, of the neuron values absolutely necessary to process that individual topo_batch in parallel. This is a drastic performance improvement. In the case where the network has a clean layer structure, such as an MLP, this method results in zero wasted memory (the weight matrices are 100% filled), yet remains just as parallel! This fixes the memory (and CPU training time) blowup, allowing for much greater scalability.
For completeness (and in case anyone comes across a similar issue in the future), here are the __init__ and __call__ methods (using Equinox Modules) originally:
    def __init__(
        self,
        num_neurons: int,
        adjacency_dict: Mapping[int, Sequence[int]],
        input_neurons: Sequence[int],
        output_neurons: Sequence[int],
        hidden_activation: Callable = jnn.silu,
        output_activation_elem: Callable = _identity,
        output_activation_group: Callable = _identity,
        dropout_p: Union[float, Sequence[float]] = 0.,
        *,
        key: Optional[jr.PRNGKey] = None,
        parameter_matrix: Optional[Array] = None,
        **kwargs,
    ):
        """**Arguments**:

        - `num_neurons`: The number of neurons in the network.
        - `adjacency_dict`: A dictionary that maps a neuron id to the ids of its
            outputs. Neurons must be ordered from `0` to `num_neurons - 1`. Neurons
            with no outgoing connections do not need to be included.
        - `input_neurons`: A sequence of `int` indicating the ids of the
            input neurons. The order here matters, as the input data will be
            passed into the input neurons in the order passed in here.
        - `output_neurons`: A sequence of `int` indicating the ids of the
            output neurons. The order here matters, as the output data will be
            read from the output neurons in the order passed in here.
        - `hidden_activation`: The activation function applied element-wise to the
            hidden (i.e. non-input, non-output) neurons. It can itself be a
            trainable `equinox.Module`.
        - `output_activation_elem`: The activation function applied element-wise to
            the  output neurons. It can itself be a trainable `equinox.Module`.
        - `output_activation_group`: The activation function applied to the output
            neurons as a whole after applying `output_activation_elem` element-wise,
            e.g. `jax.nn.softmax`. It can itself be a trainable `equinox.Module`.
        - `dropout_p`: Dropout probability. If a single `float`, the same dropout
            probability will be applied to all hidden neurons. If a `Sequence[float]`,
            the sequence must have length `num_neurons`, where `dropout_p[i]` is the
            dropout probability for neuron `i`. Note that this allows dropout to be
            applied to input and output neurons as well.
        - `key`: The `jax.random.PRNGKey` used for parameter initialization and dropout.
            Optional, keyword-only argument. Defaults to `jax.random.PRNGKey(0)`.
        - `parameter_matrix`: A `jnp.array` of shape `(num_neurons, num_neurons + 1)`
            where entry `[i, j]` is neuron `i`'s weight for neuron `j`, and entry
            `[i, num_neurons]` is neuron `i`'s bias. Optional, keyword-only argument --
            used primarily for plasticity functionality.
        """
        super().__init__(**kwargs)
        input_neurons = jnp.array(input_neurons, dtype=int)
        output_neurons = jnp.array(output_neurons, dtype=int)
        self._check_input(num_neurons, adjacency_dict, input_neurons, output_neurons)
        self.adjacency_dict = adjacency_dict
        adjacency_matrix = _adjacency_dict_to_matrix(num_neurons, adjacency_dict)
        self.topo_batches = self._topological_batching(adjacency_dict, num_neurons)
        self.adjacency_matrix = adjacency_matrix
        self.input_neurons = input_neurons
        self.output_neurons = output_neurons
        self.num_neurons = num_neurons

        # Activations may themselves be `eqx.Module`s, so we do this to ensure
        # that both `Module` and non-`Module` activations work with the same
        # input shape. The copies (hidden_activation_ and output_activation_elem_)
        # are for plasticity functionality.
        self.hidden_activation = hidden_activation \
            if isinstance(hidden_activation, Module) else vmap(hidden_activation)
        self.output_activation_elem = output_activation_elem \
            if isinstance(output_activation_elem, Module) else vmap(output_activation_elem)
        self.hidden_activation_ = hidden_activation
        self.output_activation_elem_ = output_activation_elem
        self.output_activation_group = output_activation_group

        # Set the random key. We use eqxe.StateIndex here so that the key
        # can be updated after every forward pass. This ensures that the
        # random values generated for determining dropout application
        # are different for each forward pass.
        self.key = eqxe.StateIndex()
        key = key if key is not None else jr.PRNGKey(0)
        eqxe.set_state(self.key, key)

        # Set dropout probabilities. We use eqxe.StateIndex here so that
        # dropout probabilities can later be modified, if desired.
        self.dropout_p = eqxe.StateIndex()
        self.set_dropout_p(dropout_p)

        # Set parameters.
        if parameter_matrix is None:
            # Initialize parameters as being drawn iid ~ N(0, 0.01).
            parameter_matrix = jr.normal(
                key, (self.num_neurons, self.num_neurons + 1)
            ) * 0.1
        shape = parameter_matrix.shape
        assert len(shape) == 2
        assert shape[0] == self.num_neurons
        assert shape[1] == self.num_neurons + 1
        self.parameter_matrix = parameter_matrix

    @filter_jit
    def __call__(self, x: Array) -> Array:
        """The forward pass of the network. Neurons are "fired" in topological batch
        order (see Section 2.2 of https://arxiv.org/pdf/2101.07965.pdf), with `jax.vmap`
        vectorization used within each topological batch.

        **Arguments**:

        - `x`: The input array to the network for the forward pass. The individual
            values will be written to the input neurons in the order passed in during
            initialization.

        **Returns**:

        The result array from the forward pass. The order of the array elements will be
        the order of the output neurons passed in during initialization.
        """
        # Neuron value array, updated as neurons are fired.
        values = jnp.ones((self.num_neurons + 1,))

        # Set input values.
        values = values.at[self.input_neurons].set(x)

        # Zero out network weights where there are no connections.
        parameters = self.parameter_matrix.at[:, :-1].multiply(self.adjacency_matrix.T)

        # Dropout.
        key = eqxe.get_state(self.key, jr.PRNGKey(0))

        dropout_p = self.get_dropout_p()
        rand = jr.uniform(key, dropout_p.shape, minval=0, maxval=1)
        apply_dropout = jnp.less_equal(rand, dropout_p)

        _, new_key = jr.split(key)
        eqxe.set_state(self.key, new_key)

        # Forward pass in topological batch order.
        for tb in self.topo_batches:
            # Affine transformation, wx + b.
            affine = vmap(jnp.dot, in_axes=[0, None])(parameters[tb], values)

            # Add a dimension.
            affine = jnp.expand_dims(affine, 1)

            # Apply activations/dropout.
            output_values = vmap(self._fire_neuron)(tb, affine, values[tb], apply_dropout[tb])

            # Set new values.
            values = values.at[tb].set(output_values)

        # Return values pertaining to output neurons, with the group-wise
        # output activation applied.
        return self.output_activation_group(values[self.output_neurons])

And here are the updated methods, with detailed comments explaining each step (not pushed to the main branch yet, as I need to test it more thoroughly):
    def __init__(
        self,
        num_neurons: int,
        adjacency_dict: Mapping[int, Sequence[int]],
        input_neurons: Sequence[int],
        output_neurons: Sequence[int],
        hidden_activation: Callable = jnn.silu,
        output_activation_elem: Callable = _identity,
        output_activation_group: Callable = _identity,
        dropout_p: Union[float, Sequence[float]] = 0.,
        *,
        key: Optional[jr.PRNGKey] = None,
        **kwargs,
    ):
        """**Arguments**:

        - `num_neurons`: The number of neurons in the network.
        - `adjacency_dict`: A dictionary that maps a neuron id to the ids of its
            outputs. Neurons must be ordered from `0` to `num_neurons - 1`. Neurons
            with no outgoing connections do not need to be included.
        - `input_neurons`: A sequence of `int` indicating the ids of the
            input neurons. The order here matters, as the input data will be
            passed into the input neurons in the order passed in here.
        - `output_neurons`: A sequence of `int` indicating the ids of the
            output neurons. The order here matters, as the output data will be
            read from the output neurons in the order passed in here.
        - `hidden_activation`: The activation function applied element-wise to the
            hidden (i.e. non-input, non-output) neurons. It can itself be a
            trainable `equinox.Module`.
        - `output_activation_elem`: The activation function applied element-wise to
            the  output neurons. It can itself be a trainable `equinox.Module`.
        - `output_activation_group`: The activation function applied to the output
            neurons as a whole after applying `output_activation_elem` element-wise,
            e.g. `jax.nn.softmax`. It can itself be a trainable `equinox.Module`.
        - `dropout_p`: Dropout probability. If a single `float`, the same dropout
            probability will be applied to all hidden neurons. If a `Sequence[float]`,
            the sequence must have length `num_neurons`, where `dropout_p[i]` is the
            dropout probability for neuron `i`. Note that this allows dropout to be
            applied to input and output neurons as well.
        - `key`: The `jax.random.PRNGKey` used for parameter initialization and dropout.
            Optional, keyword-only argument. Defaults to `jax.random.PRNGKey(0)`.
        """
        super().__init__(**kwargs)
        for n in range(num_neurons):
            if n not in adjacency_dict:
                adjacency_dict[n] = []
        adjacency_dict_inv = _invert_dict(adjacency_dict)
        self._check_input(
            num_neurons, adjacency_dict, adjacency_dict_inv, input_neurons, output_neurons
        )
        self.adjacency_dict = adjacency_dict
        self.adjacency_dict_inv = adjacency_dict_inv
        self.input_neurons = jnp.array(input_neurons, dtype=int)
        self.output_neurons = jnp.array(output_neurons, dtype=int)
        self.num_neurons = num_neurons

        # Activations may themselves be `eqx.Module`s, so we do this to ensure
        # that both `Module` and non-`Module` activations work with the same
        # input shape. The copies (hidden_activation_ and output_activation_elem_)
        # are for plasticity functionality.
        self.hidden_activation = hidden_activation \
            if isinstance(hidden_activation, Module) else vmap(hidden_activation)
        self.output_activation_elem = output_activation_elem \
            if isinstance(output_activation_elem, Module) else vmap(output_activation_elem)
        self.hidden_activation_ = hidden_activation
        self.output_activation_elem_ = output_activation_elem
        self.output_activation_group = output_activation_group

        # Set dropout probabilities. We use `eqxe.StateIndex`` here so that
        # dropout probabilities can later be modified, if desired.
        self.dropout_p = eqxe.StateIndex()
        self.set_dropout_p(dropout_p)

        # Get topological information.
        topo_batches = self._topological_batching()
        topo_sort = topo_batches[0]
        for tb in topo_batches[1:]:
            topo_sort = np.append(topo_sort, tb)
        # Maps a neuron id to its position in the topological sort.
        self.topo_sort_inv = jnp.array(np.argsort(topo_sort), dtype=int)
        self.topo_batches = [jnp.array(tb, dtype=int) for tb in topo_batches[1:]]
        num_topo_batches = len(self.topo_batches)

        # Here, `mins[i]` (`maxs[i]`) is the index representing the minimum
        # (maximum) position -- with respect to the topological order -- of
        # those neurons strictly necessary to process `self.topo_batches[i]`
        # from the previous topological batch. If `i == 0`, the previous
        # topological batch is the input neurons.
        mins = np.array([])
        maxs = np.array([])
        for tb in self.topo_batches:
            input_locs = [jnp.array(adjacency_dict_inv[int(i)], dtype=int) for i in tb]
            input_locs_topo = [self.topo_sort_inv[loc] for loc in input_locs]
            mins_ = np.array([np.amin(locs) for locs in input_locs_topo])
            maxs_ = np.array([np.amax(locs) for locs in input_locs_topo])
            mins = np.append(mins, np.amin(mins_))
            maxs = np.append(maxs, np.amax(maxs_))

        # Set the random key. We use eqxe.StateIndex here so that the key
        # can be updated after every forward pass. This ensures that the
        # random values generated for determining dropout application
        # are different for each forward pass.
        self.key = eqxe.StateIndex()
        key = key if key is not None else jr.PRNGKey(0)
        *wkeys, bkey, dkey = jr.split(key, num_topo_batches + 2)
        eqxe.set_state(self.key, dkey)

        # Set the network parameters. Here, `self.bias` is a `jnp.array` of shape
        # `(num_neurons - num_input_neurons,)`, where `self.bias[i]` is the bias of neuron
        # `i + num_input_neurons`. `self.weights`, on the other hand, is a list of 2D `jnp.array`s,
        # where `self.weights[i]` includes the weights used by the neurons in `self.topo_batches[i]`.
        # More specifically, `self.weights[i][j, k]` is the weight of the connection from the neuron
        # with topological index `k + mins[i]`, i.e. neuron `topo_sort[k + mins[i]]`, to neuron
        # `self.topo_batches[i][j]`. The weights are stored this way in order to use minimal memory
        # while allowing for maximal `vmap` parallelism during the forward pass, since the minimum
        # and maximum neurons needed to process a topological batch in parallel will be closest
        # together when in topological order.
        weight_lengths = np.array(maxs - mins, dtype=int) + 1
        self.weights = [
            jr.normal(
                wkeys[i], (jnp.size(self.topo_batches[i]), weight_lengths[i])
            ) * 0.1
            for i in range(num_topo_batches)
        ]
        self.bias = jr.normal(
            bkey, (num_neurons - jnp.size(self.input_neurons),)
        ) * 0.1

        # Here, `self.masks` is a list of 2D binary `jnp.array` with identical structure
        # to `self.weights`. These are multiplied by the weights during the forward pass
        # to mask out weights for connections that are not present in the actual network.
        masks = []
        for (tb, weights, min_) in zip(self.topo_batches, self.weights, mins):
            mask = np.zeros_like(weights)
            for (i, neuron) in enumerate(tb):
                inputs = jnp.array(adjacency_dict_inv[int(neuron)], dtype=int)
                inputs_topo = self.topo_sort_inv[inputs] - int(min_)
                mask[i, inputs_topo] = 1
            masks.append(jnp.array(mask, dtype=int))
        self.masks = masks

        # Here, `self.idxs[i]` includes the indices -- in topological order -- of the
        # neurons needed to process `self.topo_batches[i]`. This is done for the same
        # memory/parallelism reason as the structure of `self.weights`.
        self.idxs = [
            jnp.arange(mins[i], maxs[i] + 1, dtype=int) for i in range(num_topo_batches)
        ]

    @filter_jit
    def __call__(
        self, x: Array, *, key: Optional[jr.PRNGKey] = None,
    ) -> Array:
        """The forward pass of the network. Neurons are "fired" in topological batch
        order (see Section 2.2 of https://arxiv.org/pdf/2101.07965.pdf), with `jax.vmap`
        vectorization used within each topological batch.

        **Arguments**:

        - `x`: The input array to the network for the forward pass. The individual
            values will be written to the input neurons in the order passed in during
            initialization.
        - `key`: A `jax.random.PRNGKey` used for dropout. Optional, keyword-only argument.
            If `None`, a key will be generated by getting the key in `self.key` (which is
            an `eqxe.StateIndex`) and then splitting and updating the key for the next
            forward pass.

        **Returns**:

        The result array from the forward pass. The order of the array elements will be
        the order of the output neurons passed in during initialization.
        """
        # Neuron value array, updated as neurons are "fired".
        values = jnp.zeros((self.num_neurons,))

        # Dropout.
        if key is None:
            key = eqxe.get_state(self.key, jr.PRNGKey(0))
            _, new_key = jr.split(key)
            eqxe.set_state(self.key, new_key)
        dropout_p = self.get_dropout_p()
        rand = jr.uniform(key, dropout_p.shape, minval=0, maxval=1)
        dropout_keep = jnp.greater(rand, dropout_p)

        # Set input values.
        input_neurons_topo = self.topo_sort_inv[self.input_neurons]
        values = values.at[input_neurons_topo].set(
            x * dropout_keep[self.input_neurons]
        )

        # Function to apply the activation for a single neuron.
        def _apply_activation(id: int, affine: jnp.array) -> float:
            affine = jnp.expand_dims(affine, 0)
            return jnp.squeeze(lax.cond(
                jnp.isin(id, self.output_neurons),
                lambda: self.output_activation_elem(affine),
                lambda: self.hidden_activation(affine)
            ))

        # Forward pass in topological batch order.
        for (tb, weights, mask, idx) in zip(self.topo_batches, self.weights, self.masks, self.idxs):
            # Affine transformation, wx + b.
            affine = vmap(jnp.dot, in_axes=[0, None])(weights * mask, values[idx]) + \
                self.bias[tb + jnp.size(self.input_neurons)]
            # Apply activations/dropout.
            output_values = vmap(_apply_activation)(tb, affine) * dropout_keep[tb]
            # Set new values.
            values = values.at[self.topo_sort_inv[tb]].set(output_values)

        # Return values pertaining to output neurons, with the group-wise
        # output activation applied.
        output_neurons_topo = self.topo_sort_inv[self.output_neurons]
        return self.output_activation_group(values[output_neurons_topo])

Added in #13155


Update sbt to 1.9.9

About this PR
📦 Updates org.scala-sbt:sbt from 1.9.8 to 1.9.9
📜 GitHub Release Notes - Version Diff
Usage
✅ Please merge!
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

⚙ Adjust future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "org.scala-sbt", artifactId = "sbt" } ]

Or, add this to slow down future updates of this dependency:
dependencyOverrides = [{
  pullRequests = { frequency = "30 days" },
  dependency = { groupId = "org.scala-sbt", artifactId = "sbt" }
}]

labels: library-update, early-semver-patch, semver-spec-patch, version-scheme:early-semver, commit-count:1

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 56.84%. Comparing base (0d539bd) to head (f4b0032).

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #447   +/-   ##
=======================================
  Coverage   56.84%   56.84%
=======================================
  Files          35       35
  Lines        1752     1752
  Branches      233      233
=======================================
  Hits          996      996
  Misses        756      756

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Add name() method to UserDefinedLogicalNode

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
(This section helps Arrow developers understand the context and why for this feature, in addition to  the what)
ExtensionPlanner convert extension UserDefinedLogicalNode to physical plan. But currently UserDefinedLogicalNode doesn't have a method to recognize which extension plan it is. So we have to write something like this:
        if let Some(node) = node.as_any().downcast_ref::<SeriesNormalize>() {
            Ok(Some(node.to_execution_plan(physical_inputs[0].clone())))
        } else if let Some(node) = node.as_any().downcast_ref::<InstantManipulate>() {
            Ok(Some(node.to_execution_plan(physical_inputs[0].clone())))
        } else if let Some(node) = node.as_any().downcast_ref::<RangeManipulate>() {
            Ok(Some(node.to_execution_plan(physical_inputs[0].clone())))
        } else if let Some(node) = node.as_any().downcast_ref::<SeriesDivide>() {
            Ok(Some(node.to_execution_plan(physical_inputs[0].clone())))
        } else if let Some(node) = node.as_any().downcast_ref::<EmptyMetric>() {
            Ok(Some(node.to_execution_plan()))
        } else {
            Ok(None)
        }

Describe the solution you'd like
A clear and concise description of what you want to happen.
Add a new method to UserDefinedLogicalNode that allows it to state its name, like
pub trait UserDefinedLogicalNode {
  fn name(&self) -> &str;
}

Then the long if-else chain can be organized into match. It should be more clear.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.

Another example from iox: https://github.com/influxdata/influxdb_iox/blob/f3267f992ae338973ea71df169b91015c6dc1473/iox_query/src/exec/context.rs#L94


Unable to adjust decal alpha with MSDF texture when alpha is specified in 'main color/alpha'

Issue
I'm attempting to improve the quality of a texture by using an MSDF texture in 'main color 2nd' while lowering the resolution of the existing texture. However, if an alpha has already been specified in 'main color/alpha', I can't adjust the alpha of that part when using a decal with the MSDF texture, so I can't achieve the desired effect.
Solution
To be able to adjust the alpha of a decal when using an MSDF texture, I need to modify the code. For example, when applying the decal, I can ignore the alpha value specified in 'main color/alpha' and instead use the alpha value specified in the MSDF texture. This way, I can achieve the desired effect when using the decal.

I added a function to overwrite with the transparency of the main color 2nd and 3rd. This is available in the next version.


dividing, math command

Proposal:
Proposal written by Maxi
Is it a proposal for the website or for the bot?: bot
What is the title of your proposal?: dividing, math command
Explain your proposal more accurately (It's best to give as much information as possible, so that we can implement the proposal better): In ?math there are 3 kinds of questions +, ´-´ and *, maybe we can add / (:) to the questions. [Example: 6/3 = 2 or 6:3 = 2]
Why should we add this feature?: This enhancement makes the function more complex.
ReportID: 156

Test


[RadzenProfileMenu] Open or Close callbacks

Is your feature request related to a problem? Please describe.
RadzenProfileMenu like other components, has a Click handler, but i would need to be notified when the menu dropdown gets opened or gets closed, not just clicked on. Because when it is open the user can close it by just clicking anywhere else on the page, and in this case RadzenProfileMenu won't fire a Click.
Describe the solution you'd like
Provide the possibility to set a callback for Open and Close, like for Click
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Or alternatively a Radzen method i can poll at any time to ask if a RadzenProfileMenu is open or close.  ( I may e.g. handle click events on the whole page, and ask Radzen if the RadzenProfileMenu is open or close at that moment ).
But this latter solution is dirty, and at this point i can already solve it via a Jquery click handler , which queries the CSS selector matching with the RadzenProfileMenu and check if its display  or its ChildContent is none ( assuming the markup for RadzenProfileMenu  will never change, although it may change with future updates of Radzen ).

We are not planning to add such feature to ProfileMenu component. You can use Popup component instead where such events are available.


Standardization 7

Follow-up to #1131 #1161 #1164 #1282 #1326 #1375 #1417
(package was introduced during those modifications)
Closes #1119

@bors-libra r+

:pushpin: Commit 3dd55be has been approved by bmwill

:hourglass: Testing commit 3dd55be98d8165f5679384271628367803388d49 with merge 012105267ad999453907f4e705eb4bcea9620eda...

:sunny: Test successful - checks-circle_commit_workflow
Approved by: bmwill
Pushing 012105267ad999453907f4e705eb4bcea9620eda to master...


Bug Report online status and balance script v2_funding_rate_arb.py

Describe the bug
I'm testing the script v2_funding_rate_arb.py
https://github.com/hummingbot/hummingbot/blob/master/scripts/v2_funding_rate_arb.py
Installing a bot from Docker compose
. The online status works fine until the first order is created, after which neither the status nor the balance works.
Connectors: binance_perpetual and hyperliquid_perpetual
Error:
19:40:07 async_utils Unhandled error in background task:u
nsupported operand type(s)for *'decimal.Decimal'and 'NoneT
ype'(See log file for stack trace dump)

Steps to reproduce

Release version
1.28.0
Type of installation
Source
Attach required files
No response

Hello,
I'm also working with the v2_funding_rate_arb.py script and have come across an issue with balance display in binance_perpetual.
Specifically:

The USDT balance in my Binance USDⓈ-M Futures account is correctly displayed in binance_perpetual.

However, balances of other cryptocurrencies like TRX and WIF in my COIN-M Futures account show as 0 in binance_perpetual.

I'm wondering if you've experienced this issue where binance_perpetual only displays USDT balance but not other cryptocurrencies.
Could you advise which specific account I should deposit these other cryptocurrencies (TRX, WIF, etc.) to make them visible in binance_perpetual?
Any insights would be greatly appreciated. Thank you!

Hello,
I'm also working with the v2_funding_rate_arb.py script and have come across an issue with balance display in binance_perpetual.
Specifically:

The USDT balance in my Binance USDⓈ-M Futures account is correctly displayed in binance_perpetual.
However, balances of other cryptocurrencies like TRX and WIF in my COIN-M Futures account show as 0 in binance_perpetual.

I'm wondering if you've experienced this issue where binance_perpetual only displays USDT balance but not other cryptocurrencies.
Could you advise which specific account I should deposit these other cryptocurrencies (TRX, WIF, etc.) to make them visible in binance_perpetual?
Any insights would be greatly appreciated. Thank you!

Hello. There are also problems with the balance display Hyperliquid. Try changing the settings in the bot agent configuration to display the balance.


Revert the changes

Tested this on a vm, seems to work fine. But having troubles restoring the system to default after execution. Any ideas?

Never mind, figured it out:
rd "\\?\C:\Windows "


docs: update function docs on deriv

This updates docs to resolve #9929 .  I tried to match the level of details listed in other sections.
cc @sed-i
Signed-off-by: mtfoley<EMAIL_ADDRESS>

The fuzzing and linting checks appear to be stalled out on this one. It's just docs, so I don't know that they're needed.

Thanks for clarifying the docs. deriv is clearer now.

Based in the blog post linked in #9929 it sounds like rate is (v_f - v_i) / (t_f - t_i). This is very different from what the following phrase from the published docs conveys: "rate(v range-vector) calculates the per-second average rate of increase". Basically rate completely ignores all the datapoints between the first and the last, correct? If so then it might be worthwhile to clarify that in the docs for rate.
I think it makes sense to add a "see also: deriv" to rate and a "see also: rate" to deriv.

@mtfoley Sorry for letting this fall through the cracks. You should have gotten a review long ago.
@sed-i I guess it cannot harm to clarify the rate/increase/delta documentation about how it is different from deriv/predict_linear, but let's do that in a separate PR.

@beorn7 updated per requested changes.

@sed-i perhaps you could explain in more detail what you are (still) missing? (Could also be addressed in a separate PR, maybe bi @sed-i themself?)


Handling of Path Parameters

We currently do not let the user describe their path parameters.  The path parameters should be displayed in the Path Form and editable there.  Sadly the OpenAPI spec requires the path params to be defined at each of the operations, rather than on the path itself.  For this reason, the editor should allow the user to edit them on the Path Form, but the changes would need to be applied to all operations.
Changes needed:

Display table of path params on the Path Form (only if the Path has at least one operation)
Path Params are editable on the Path Form
Changes made to path params are applied to all operations in path
New operations must inherit path params from existing Operation(s)
First newly created operation for Path should automatically create required path parameters based on the path format

Now I'm thinking that the Path Params should appear on both the individual Operation form AND on the Path Item form.  If the value is changed on the Operation form, then it applies only to that one parameter.  If it is changed on the Path Item form, then ALL operations are affected.


chore(LpSpace): golf, generalize

drop mostly unused section variables;
generalize MeasureTheory.lintegral_indicator_const
and MeasureTheory.snorm_indicator_const
to a NullMeasurableSet;
golf a proof.

bors r+