properly jump the address gap in CleanUp

Starting with 0.15, the addition operator will no longer implicitly jump the gap in the address space. Instead, we can use Step::forward.
Fixes #468

We should probably create a patch release for this, right?

We should probably create a patch release for this, right?

I agree

We should probably create a patch release for this, right?

Hi there, so when will the plan be to release this patch?

If everything goes well, I'm planning on creating a PR for a new release after #472 is merged.


[WIP] Fix for ESXi Template

这个 PR 实现什么功能/修复什么问题:

是否需要 backport 到之前的 release 分支:

Codecov Report

Merging #8232 into master will decrease coverage by 0.03%.
The diff coverage is n/a.

@@            Coverage Diff            @@
##           master   #8232      +/-   ##
=========================================
- Coverage    6.21%   6.18%   -0.04%
=========================================
  Files         759     150     -609
  Lines      117256   20950   -96306
=========================================
- Hits         7289    1296    -5993
+ Misses     109308   19505   -89803
+ Partials      659     149     -510

Flag
Coverage Δ

#aFlag
6.18% <ø> (-0.04%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

pkg/appsrv/send.go

pkg/util/s3auth/interface.go

pkg/keystone/models/identityquota.go

pkg/multicloud/esxi/resourcepool.go

pkg/compute/models/loadbalanceracls.go

pkg/util/fileutils2/fileutils.go

pkg/multicloud/esxi/devtools.go

pkg/hostman/system_service/host_deployer.go

pkg/util/fileutils2/seeker.go

pkg/compute/guestdrivers/aws.go

... and 597 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4e901bd...0ddbcdc. Read the comment docs.

/cc @swordqiu @zexi


License

Neat.  Shouldn't you license this GPL, to respect pandoc's GPL license?

To be honest, I haven't really paid attention to license. My fork of pandoc is not included in this repo and my modifications are publicly available (I'm not building a different version of this for http://markup.rocks) so I don't think this app is bound by pandoc's license. Please feel free to correct me though.

Oh, I just realized you're the author of pandoc, thanks for the amazing software! Please let me know if you have any concerns about this.

From https://github.com/jgm/pandoc/blob/master/COPYING

These requirements apply to the modified work as a whole.  If
identifiable sections of that work are not derived from the Program,
and can be reasonably considered independent and separate works in
themselves, then this License, and its terms, do not apply to those
sections when you distribute them as separate works.  But when you
distribute the same sections as part of a whole which is a work based
on the Program, the distribution of the whole must be on the terms of
this License, whose permissions for other licensees extend to the
entire whole, and thus to each and every part regardless of who wrote it.

So I think that technically, you can distribute the code in this repository under any license you choose.  However, the derivative work created when you combine this code with your modified pandoc code to produce a JavaScript program is covered by the terms of the GPL and should conform to it.  So it might be clearer if you just licensed all of this as GPL.  Otherwise there's a risk people may not realize that the compiled result is GPL.  But I think it's up to you.  (Also: I'm not a lawyer.) At the very least you might want to note in the README that pandoc is GPL, so the compiled JS code will be too.

If your code is pretty general purpose and could be used with any X-to-Y markup processor, then maybe that's good reason to leave it with a liberal license.  (In fact, I might be tempted to use the UI code with commonmark.js!) But I think a note in the README about the derived JS code being subject to GPL is a good idea.

@jgm I'm definitely thinking about adding other JS based backends that increase the performance of some paths like Markdown -> HTML. My reasoning was this is mostly glue code and an interface that ties various other tools together in a non-widespread way and people should be free to grab any useful snippet they find in my (poor) Haskell code to make their browser app work as there is a definite shortage of documentation regarding libraries like Reflex.
I realize the fact that I'm also hosting this on a (albeit non-profit) website and delivering the compiler output that includes Pandoc instead of using it in a server-side service puts me on a tough spot. I think this matter is unrelated to the repo's license as even if it was GPL this wouldn't have anything to do with the file hosted at http://markup.rocks/static/all.js which might just as well generated by `cat pandoc.exe >> all.js```.
I've updated the Readme. Please let me know if this is clear enough for you.

+++ Ozan Sener [May 08 15 14:13 ]:

[1]@jgm I'm definitely thinking about adding other JS based backends
that increase the performance of some paths like Markdown -> HTML.

The problem here is that no available JS based parser will parse Markdown the way pandoc does.  (In fact, no two will agree with each other---leaving aside commonmark-compliant implementations.)

I've updated the [4]Readme. Please let me know if this is clear enough
for you.

Sure, this seems fine (except for a typo, "it's" for "its").

I agree, this is why I was not eager to do that but it might be useful for someone using this tool to preview Markdown documents instead of converting them.
Thanks for your understanding and insight about the license.


Support Envoy retry headers

To minimise 503s during a rolling update the Virtual Service could append the following Envoy headers:
  http:
  - appendHeaders:
      x-envoy-max-retries: "10"
      x-envoy-retry-on: gateway-error,connect-failure,refused-stream

The Canary service spec should allow headers to be appended.

Istio 1.1 will fully support this retry policy with http.route.retries.retryOn but will probably take ages before it makes it's way into the Istio go client made by knative.


Comment crud

Also, I have found an interesting comment in documentation : Dodanie komentarza przez wolontariusza powoduje dodanie nowej aktualności w zakładce „Newsy”. Wyświetlona zostaje data dodania komentarza, kto dodał nowy komentarz (imię i nazwisko wolontariusza) oraz skrócona treść komentarza. It seems that creating comments would have to be also followed by creating a news about comment. (Although I'm not sure we should probably ask SA).

I think, that in this context word "News" is only responsible for the animals and comments about them. "News" (in the old sense) refers to the information provided by TOZ, for example that they need some toys for dogs. But it can be asked SA if they want all new comments in "News" (in the old sense) section, because I understand it as I wrote it, but you can really have right.
And I will ask SA if admins can edit all comments.

It turned out that "epic" 104 is connected to 105, so for now I'm closing this because I have to add missing things.


python37Packages.inflect: 4.1.0 -> 5.0.2

Automatic update generated by nixpkgs-update tools. This update was made based on information from passthru.updateScript.
meta.description for python37Packages.inflect is: ""
meta.homepage for python37Packages.inflect is: ""
meta.changelog for python37Packages.inflect is: ""
Updates performed

Ran passthru.UpdateScript

To inspect upstream changes
Impact

Checks done (click to expand)

built on NixOS

Warning: a test defined in passthru.tests did not pass

0 of 0 passed binary check by having a zero exit code.

0 of 0 passed binary check by having the new version present in output.

found 5.0.2 with grep in /nix/store/h0vq3bp69qrrnzxvhwajkss42s7hrgaw-python3.7-inflect-5.0.2

directory tree listing: https://gist.github.com/fa8efc8293f1ddd1d28c1d8eef440c56

du listing: https://gist.github.com/2d94cf8d41f1f256b34706f8981538fa

Rebuild report (if merged into master) (click to expand)

43 total rebuild path(s)

11 package rebuild(s)

11 x86_64-linux rebuild(s)
11 i686-linux rebuild(s)
10 x86_64-darwin rebuild(s)
11 aarch64-linux rebuild(s)

First fifty rebuilds by attrpath
errbot
python37Packages.diff_cover
python37Packages.inflect
python37Packages.irc
python37Packages.jaraco_itertools
python37Packages.jinja2_pluralize
python38Packages.diff_cover
python38Packages.inflect
python38Packages.irc
python38Packages.jaraco_itertools
python38Packages.jinja2_pluralize

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/h0vq3bp69qrrnzxvhwajkss42s7hrgaw-python3.7-inflect-5.0.2 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A python37Packages.inflect https://github.com/r-ryantm/nixpkgs/archive/d4663ee0d3e7d502fc520c32e879db6a6bdae202.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/h0vq3bp69qrrnzxvhwajkss42s7hrgaw-python3.7-inflect-5.0.2
ls -la /nix/store/h0vq3bp69qrrnzxvhwajkss42s7hrgaw-python3.7-inflect-5.0.2/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review 1

11 packages built:

errbot
python37Packages.diff_cover
python37Packages.inflect
python37Packages.irc
python37Packages.jaraco_itertools
python37Packages.jinja2_pluralize
python38Packages.diff_cover
python38Packages.inflect
python38Packages.irc
python38Packages.jaraco_itertools
python38Packages.jinja2_pluralize

Maintainer pings

Can someone rebase this branch on master to fix the eval error?

@GrahamcOfBorg eval


Add support for YAML config

YAML configs are better suited for humans than JSON. Also YAML support comments. In this pull request we first try to read feeds.yaml, and if that fails fallback to feeds.json.
I'm sorry if the code in PR looks very silly (it's my first lines of code in Go).

Merged, thanks for your contribution.
FYI however, I do not plan to maintain maildir-feed anymore (except accepting external pull requests), since https://github.com/sloonz/ua does the job better. I think I should put a notice in the README file of this project.


Add support for multitenancy

At the moment OpenRefine is running under one user.
Proposed solution
It would be useful to have support for multitenancy.
Alternatives considered
Perhaps there's already a plugin that provides this information.

I would be interested to know what @MPParsley has in mind with multitenancy - this could mean a lot of different things! Could you expand on the use case you are thinking about?

My request is indeed to have support for multiple users:
At the moment the application is running under one Linux user. Access to the application can be limited by adding a simple basic authentication but anyone who has access can access the same workspace where everything is shared. Having a way to break this up for each user (or group of users) would add value.
Setting up a server per user does not seem like a viable option.
How this is done (multiple tenants vs multiple Linux users) is not really my concern as long as a (group of) user(s) can manage their own data.

OK, regardless the exact use case, the Server side will need user configuration of several parameters and options on implementation choices given to administrators (http1/2, ssl, virtual hosts, etc.)  I won't go into details, but you can read where some of which will be handled by Jetty and can be configured, https://www.eclipse.org/jetty/documentation/jetty-11/operations-guide/index.html#og-sessions  and then beyond the Server side is our Client side to handle those individual user sessions, avoid conflicts, and a whole complexity of tenant operations allowed or not or made configurable as explained some in #727

Thanks @MPParsley for expanding on your request, very much appreciated. I agree with you running one server per user would not add much to the current situation.

Over the last week, I have been thinking about our overall architecture decisions from the past and the basis of the use cases.  With this issue, the downplay we had of scaling issues becomes apparent especially where an OpenRefine service might be containerized and even clustered over K8S for multitenancy.  Our existing Servlet choice is not the best here since it has an imperative and blocking I/O thread model.  A better choice for the future is likely to be a reactive execution model.  Think Kotlin co-routines or Project Loom.
One of choices I had envisioned for a future stack was moving to Spring and WebFlux but there are problems with its focus not being entirely event-driven.  Quarkus on the other hand with Mutiny would make our lives and even extension developer lives easier. In particular Mutiny uses Reactive Streams and can enforce back pressure (imagine 4-5 users working on a single very large dataset where someone is doing a transform on 1 column, and the others might collaboratively be working through clustering and merging values at the same time in other columns) We could more easily map our dependent operations across Reactive Streams.
The other thing about Quarkus is the strong Camel support additionally for lots of integrations that extension authors might want to do.

I played some more this week with Quarkus and I do see its strengths but some weaknesses directly for our needs on this multi-tenancy issue directly which I mentioned in a mailing list thread here
In summary, Quarkus could be left to others that might want to adapt to a fully containerized execution environment for OpenRefine, but it's not something that we need directly for multi-tenancy I would say.  Jetty and Spring WebFlux (along with some of the Spring ecosystem like Spring Boot and Spring Security) could sustain us.  The bonus is that a Spring Boot application, if we evolve OpenRefine to that in a 4.0x branch, can be later turned into a native image with Quarkus and GraalVM if someone in the community wants to do that later.


key based access

Hi
Great project!
Key based aithentication don't seem to work. If I manually try to access my Aruba switch it works fine.
I tried to troubleshoot the code and all looks good, you have any hint on how I can troubleshoot it further?
Thanks
Felix

sorry it do seem to work - is related to group permissions it looks like

Thanks for testing.


book list component styling

Book-label: font-weight bold, size 1rem
Book/version timestamp (last updated/created by): font-weight normal, size .875rem

Fixed


[KYUUBI #3101] [Subtask][#3100] Build the content for extension points documentation

Why are the changes needed?

Build the content for extension points documentation, pre-work for #3100

the extensions are divided into 2: server side and engine side extensions. (Do we have client side extension support?)
the server side authentication page is cross-referenced by the security section, see 1 in the picture.
the engine side ones are grouped by different compute frameworks.
connector is one type of extension, so we cross-reference the connector pages directly, see 2 & 3 in the picture.

How was this patch tested?

[ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

[x] Add screenshots for manual tests if appropriate

[x] Run test locally before make a pull request

cc @ulysses-you @turboFei @deadwind4, can you help review this?

I see connectors is a part of extension, is it better to mark connectors as top level since it's quite important for lake house user

hi @ulysses-you both connectors and security are top-level ones currently.

thanks @ulysses-you, merged to master


Adding a Cactus Example of Running a Workflow with Autoscaling

Cactus: https://github.com/ComparativeGenomicsToolkit/cactus, a reference-free whole-genome multiple alignment program.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Jenkins skip this test.

Added in #1956.


Demo continues in an infinite loop

When executing:
roslaunch dpgo_ros dpgo_demo.launch local_initialization_method:=Odometry

from Section A first demo, the program continues in an infinite loop. Is this behavior intended? Looping appears to occur after optimized trajectories are logged to ../catkin_ws/src/dpgo_ros/logs/agent#/ for # = {0,1,2,3,4}

Hi @adthoms, yes, this is the intended behavior of this demo: after each optimization finishes, the program will wait a while and then do optimization again. This is motivated to reflect what often happens in practice, where we need to run optimization periodically on the latest pose graph (but for the demo, the pose graph is not evolving because we are using a fixed dataset)


Enhance: Dynamically add namespace

在运行时监听新的配置表，新监听配置表的配置可根据优先级更新至SpringValue中

Codecov Report

Merging #2047 into master will decrease coverage by 0.16%.
The diff coverage is 25%.

@@             Coverage Diff              @@
##             master    #2047      +/-   ##
============================================
- Coverage     50.25%   50.08%   -0.17%
- Complexity     1981     1984       +3
============================================
  Files           400      402       +2
  Lines         12349    12402      +53
  Branches       1275     1281       +6
============================================
+ Hits           6206     6212       +6
- Misses         5686     5731      +45
- Partials        457      459       +2

Impacted Files
Coverage Δ
Complexity Δ

...llo/spring/config/ConfigPropertySourceFactory.java
66.66% <0%> (-33.34%)
3 <0> (ø)

...apollo/spring/boot/ApolloDynamicConfiguration.java
100% <100%> (ø)
2 <2> (?)

...ork/apollo/spring/config/ConfigPropertySource.java
73.33% <20%> (-26.67%)
5 <0> (-1)

...llo/spring/config/DynamicDefaultConfigManager.java
21.87% <21.87%> (ø)
3 <3> (?)

...pring/property/AutoUpdateConfigChangeListener.java
79.1% <28.57%> (-13.76%)
17 <2> (ø)

...mework/apollo/portal/component/PortalSettings.java
65.07% <0%> (-4.77%)
5% <0%> (ø)

.../apollo/internals/RemoteConfigLongPollService.java
79.14% <0%> (-1.23%)
27% <0%> (-1%)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 21e76b3...715f671. Read the comment docs.

Coverage decreased (-0.2%) to 53.79% when pulling 715f6717f173e0fdef65880dfdc52abedb6e9086 on yangdihangN:dynamic-namespace into 21e76b3d43f455c1e28a9d6626601362cf4997ae on ctripcorp:master.

Is there a typical case to add namespace dynamically to Spring?

This feature can be used to degrade the service when the traffic is too large.

Thanks for the explanation, is it a case to dynamically add namespace to override normal configurations?
However, for the case you mentioned(circuit breaker), why don't we use a flag to switch the logic? I think switching the logic might be easier to understand.

If we use hystrix, we can use apollo to delivery HystrixCommand. In the normal namespace, we can use a relatively loose fuse strategy. Otherwise, we can use a strict fuse strategy.
This feature can change the configuration of an instance at runtime without affecting other instances., I think it makes apollo more flexible and safe in use。😄


Support SciPy1.0+

This PR supports old and new SciPy. Fix #680 .

Accept sparse matrix in numpy_cupy_allclose and numpy_cupy_array_equal.
Relax slicing restrictions
Fix test to use new style test functions

jenkins, test this please.

Jenkins CI test (for commit f60849a16771bd3a8bfd8273475ae9c2af1bf04a, target branch master) failed with status FAILURE.
(For contributors, please wait until the reviewer confirms the details of the error.)

LGTM

Jenkins CI test (for commit f60849a16771bd3a8bfd8273475ae9c2af1bf04a, target branch master) succeeded!


Garbled uft8 string from location captions.

I am getting garbled utf8 string returned by $instance->caption
This happens when I try to retrieve media-captions from a location page:
$ilocation = $instagram->getLocationById($locationid);
//[...]
if (count($ilocation[1]) > 0) {
			foreach ($ilocation[1] as $loc_media) {
                              $imedia_caption = ($loc_media->caption);
                              //Example: #LujánDeCuyo is returned as luj㢮decuyo
			}
		}

I just want to save captions from media on first page for locations - not scrape all media from location pages. Am I bypassing some uft8 encoding function here?
[edit]
Maybe I should say that I modified public function getLocationById to include an array of Medias of the first page (based on the function getLocationMediasById:
        //[...]
        $jsonResponse = json_decode($response->raw_body, true);
	$nodes = $jsonResponse ['location']['media']['nodes'];
        foreach ($nodes as $mediaArray) {
            $medias[] = Media::fromTagPage($mediaArray);
        }
	return array(Location::makeLocation($jsonResponse['location']), $medias);

My fault: I used strtolower on all characters before writing contents to the file, which messed with special unicode chars! The solution was to use mb_strtolower. Apologies!


[Enhancement]: Implement frontend environment variable handling with import-meta-env

Current Implementation and Limitations
In order to achieve build-once-run-anywhere (part of #280) building docker containers on GitHub which can be run anywhere, we want to be able to adjust the already-built frontend environment variables.
Associated Application Components
Client
Proposed Solution
This package offers the features to do so.
https://iendeavor.github.io/import-meta-env/
Estimated Difficulty
Low
Priority
High

Example of the runtime transformation tool in a docker runtime container:
https://github.com/iendeavor/import-meta-env/tree/main/packages/examples/docker-starter-example


{godot 3.0} what i faced while using it!

https://m.facebook.com/groups/377650389038228?view=permalink&id=1011791488957445

Please, fix your title and message to something more descriptive.
You can paste screenshots on GitHub by dropping them on message window.
I also suggest changing your ways of writing descriptions of your problems if your want to be heard.
Godot 3.0 is in pre-alpha stage, so bugs are expected.
You can help us by creating valid bug reports here :)

ok for sure as it's my first time of using github like this


Not working with Vaadin 23.0.0?

Hello,
looks like the component is not working with Vaadin 23.0.0.
I'm getting this error message:
WARNING in InjectManifest has been called multiple times, perhaps due to running webpack in --watch mode. The precache manifest generated after the first call may be inaccurate! Please see https://github.com/GoogleChrome/workbox/issues/1790 for more information.

ERROR in<EMAIL_ADDRESS>Module not found: Error: Can't resolve '@vaadin/vaadin-element-mixin' in 'xxxxxxxxxxxx/node_modules/@vaadin-component-factory/vcf-autosuggest/src' [build-status] 1 error and 1 warning were reported.

Thank you

Problem is solved with 1.2.0. Also, the component "feels faster" than with previous versions.


Use request.host for apps hosted behind AWS ELB

I recently inherited a legacy application which uses unobtrusive_flash and one of the first tasks I was given was to host it on AWS. It's generally a good practice to not expose your EC2 instances publicly, so I decided to use AWS ELB (service which provides you with load balancers). And everything worked except the component which relied on unobtrusive_flash!
The app is available under similarly looking URL: myapp-1989.eu-west-1.elb.amazonaws.com as well as behind some domain (CNAME pointing to load balancer's address). I traced back the AJAX requests while accessing the app in both ways and I noticed that cookies are restricted to .amazonaws.com domain and the cookie extraction function didn't see any flash cookie.
The observations are based on a quantitive analysis of a few different instances of AWS ELB and EC2 instances accessed directly via their IP addresses where the problem doesn't occur.
Let me know what you think!

I ran tests locally, but Travis says:
The repository at leonid-shevtsov/unobtrusive_flash was not found.

+1


Suggestions [Enhancement]

unbreakable anvil (give command)
make anything non-craftable

I don't know @gvlfm78's opinion on idea 1, but making anything unctaftable is not the job of OCM. It does not it in any way. gvlfm gets the finally say in all of this, of course, but I really feel the idea 2 has nothing to do with the goals of OCM.

@rayzr522 I just thought that since there is an option to disable shield crafting why not add a black list crafting section.
I thought of another feature, what about /ping command?  I think that would go great with this plugin. Since a players ping is important aspect when is comes to pvp
/ping

The ability to disable crafting of shields is so you can simulate older versions which didn't have shields, it doesn't make sense in other cases.
As for /ping, even though it's important for combat, it's still not the place of the plugin responsible for managing game mechanics to provide general-use commands.

I thought I had already added the ability to disable crafting for any item so that if someone wants to disable any new items that get introduced that they believe will affect pvp they can.
I am not sure this is the correct plugin for unbreakable anvils.


feat: pull contract ABI from explorer

What I did
Pulls contract ABI from explorer
fixes: https://github.com/ApeWorX/ape-etherscan/issues/1
fixes: https://github.com/ApeWorX/ape-etherscan/issues/7
fixes: https://github.com/ApeWorX/ape-etherscan/issues/8
How I did it
How to verify it
ape console --network :mainnet:infura
networks.ethereum.mainnet.explorer.get_contract_abi("0xBC4CA0EdA7647A8aB7C2061c2E118A18a936f13D")

Checklist

[ ] Passes all linting checks (pre-commit and CI jobs)
[ ] New test cases have been added and are passing
[ ] Documentation has been updated
[ ] PR title follows Conventional Commit standard (will be automatically included in the changelog)
[x] Fix #1
[ ] Fix #7
[ ] Fix #8

Run flake8 .
./ape_etherscan/explorer.py:2:1: F401 'ape.types.ContractType' imported but unused
./ape_etherscan/explorer.py:12:1: E731 do not assign a lambda expression, use a def

If you run the pre-commit hook, it will do all the linting at once. Also, it updates the isort managed lists in the pyproject.yaml file, which is helpful for future PRs.

If we anticipate making more HTTP calls to Etherscan, we should consider making a client.py with a class EtherscanClient that manages the requests related stuff. If we use a requests.Session object, we can add the user agent, API key, base path, etc to the session and it will automatically be set on each request and the individual client methods only need to pass in the arguments and the ending of the URL path

need to fix #7 and #8 to pass checks

Also a test would be helpful! Like one with a mocked response schema from Etherscan to show that we can create a contract class with the expected ABI and name.
Let me know, I can help with this or provide examples!

i'm still doing linting, i swear i added precommit tho 😿

(apeworx) sabotage@DESKTOP-E28BJ09:~/ape/ape-etherscan$ precommit
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = '/usr/bin/python3'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = '/usr/bin/python3'
  sys.base_prefix = '/usr'
  sys.base_exec_prefix = '/usr'
  sys.executable = '/usr/bin/python3'
  sys.prefix = '/usr'
  sys.exec_prefix = '/usr'
  sys.path = [
    '/usr/lib/python38.zip',
    '/usr/lib/python3.8',
    '/usr/lib/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007fe1f5cf8740 (most recent call first):
<no Python frame>

passing checks 🙌

To fix the mypy error, you have to add this:
[tool.mypy]
exclude = "build/"

to the pyproject.toml
See the other plugins as examples! I had to do this everywhere

You may have to do some other things to get the dependencies up to date for this project.
Namely:

Remove the exclude section from black in the pyproject.toml
add the arguments -p no:ape_test to the pytest section

Basically, just comb through a more-updated plugin's pyproject.toml and setup.py and see what is different and add those updates, especially if CI/CD is being weird.

Make sure to change this https://github.com/ApeWorX/ape-etherscan/blob/main/.github/workflows/test.yaml#L51
to use pythons 3.7, 3.8, and 3.9 instead

sorry you had to do so much updating random stuff on this PR! There were a lot of updates on linting dependencies, all the plugins had to go through this, kind of a reason we are thinking mono-repo may be a good idea.

sorry you had to do so much updating random stuff on this PR! There were a lot of updates on linting dependencies, all the plugins had to go through this, kind of a reason we are thinking mono-repo may be a good idea.

totally fine! I'm learning as I go :D
ape_etherscan/explorer.py:48: error: Unexpected keyword argument "abi" for "ContractType"
ape_etherscan/explorer.py:48: error: Unexpected keyword argument "contractName" for "ContractType"
Found 2 errors in 1 file (checked 5 source files)

I think i need to do typeignore on line 48?

I've got a test but it doesn't work :P

Might want to rename the abi.json to like get_contract_response.json

@unparalleled-js this is passing test now 🥇


Add long instructions about how to set the profiling configuration options! :)

Ideally this should probably go in some other documentation somewhere,
and we should link to that from here, so that these descriptions can be
used in other places besides just this specific endpoints package

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-3.7%) to 85.897%

Totals

Change from base Build<PHONE_NUMBER>:
-3.7%

Covered Lines:
67

Relevant Lines:
78

💛  - Coveralls

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-3.7%) to 85.897%

Totals

Change from base Build<PHONE_NUMBER>:
-3.7%

Covered Lines:
67

Relevant Lines:
78

💛  - Coveralls

@Sacha0 i've added you as a reviewer, too, even though i'm about to merge this. Please take a look if you get time, and if you have any changes we can always make them in a follow-up PR :)


There is a rendering problem（whitish） when importing 3D roads

CARLA version: 0.9.14
Platform/OS: Windows
When I imported a pavement Fbx file with elevation changes through source code on Windows, it showed unusual rendering results as shown on the right side of the figure below.
And when I import a horizontal road surface with no elevation changes, his rendering looks normal as shown on the left.

I have tried themake importcommand and complete manual operation to no avail.
And it is still not solved after packaging.
When I imported the Fbx exported by RoadRunner into other Unreal projects instead of CarlaUE4, I found that the rendering was normal, as shown in the figure below. Therefore, the initial judgment is related to the lighting environment or project settings of CarlaUE4.
If anyone has a solution to this problem that can be shared I would be very grateful。

I am sharing a similar rendering problem when exporting from RoadRunner. The ground looks darker in those areas.
CARLA version: 0.9.14
Platform/OS: Ubuntu 22.02


candidate files to remove

It seems like run.py and draft-interface.py were the first attempts to make a command line interface before we switched to Snakemake. Should these be removed from main?

I agree that we're not using any of this code and are unlikely to use it directly.  It could still serve as a useful reference for a future front end way to interact with the pipeline.  Even with the Snakemake workflow, we may want to have a user-friendly wrapper that configures and launches the workflow.
Should we move these to src/prototype, src/mockup, or something similar?

Making a subdirectory within src/ is a good idea. We can resolve this issue when we resolve Issue #7.


Multi-arch build: amd64, arm

With the release of Rancher 2.0, development on v1.6 is only limited to critical bug fixes and security patches


Error with digest-md5 auth

I could not connect to jabber.ru server with digest-md5 with the following snippet:
var XMPP = require('stanza.io');
var client = XMPP.createClient({
    jid: jid,
    password: password,
    wsURL: 'ws://jabber.ru/ws',
    transports: ['websocket'],
    sasl: 'digest-md5'
});
client.connect();

It throws an error:
Uncaught TypeError: must start with number, buffer, array or string
fromObject @ client.js:225
Buffer @ client.js:195
(anonymous function) @ client.js:8471
Client.getCredentials @ client.js:6097
(anonymous function) @ client.js:8465
WildEmitter.emit @ client.js:35256
(anonymous function) @ client.js:5887
WildEmitter.emit @ client.js:35256
(anonymous function) @ client.js:6108
WildEmitter.emit @ client.js:35268
(anonymous function) @ client.js:9301
WildEmitter.emit @ client.js:35256
WSConnection.connect.self.conn.onmessage @ client.js:9331

(I'm using the latest Chrome browser.)
I've traced it down to this js-sasl-digest-md5 line.
The server sends two challenges, the second contains rspauth. This line does this._completed = true;, after that the response function returns undefined, and that line tries to convert that undefined to a Buffer (and it does not succeed).
I'm not getting what's wrong here and how could I fix that error. Can anyone help? I am ready to test any proposed solution and contribute.

Minor patch has been published that should resolve this.
https://github.com/otalk/stanza.io/commit/20416da8d932d4aa45619e81cd6305e9710fb014

You guys are awesome! I'll check that tomorrow, thank you.

I've verified, everything works flawlessly now.


global: user initialization fix

Coverage decreased (-0.5%) to 99.528% when pulling 6c245fce11f39c98e9996e0229af7f12367c486c on lnielsen:userinit into 97f8846d7f851a62a755cc3bed31f57dbafb2be5 on inveniosoftware:master.


Footer of the Website for ASME

Write code to create footer of the webpage.
The footer should have ASME logo, basic contact details (email, contact person ) and links to different social media platforms, copyright notice, privacy policies.
Some points to remember

Use Roboto font only.
Refer ASME VIT website for color palette.
Create the page such that the html file is named as footer.html, CSS file is named as footer.css and other files should be named in similar manner.

I am Arti Tambade, TY IT
Can you please assign this issue to me?

@ArtiHere  You have already been assigned one issue. You can work only on one issue at a time . Once you complete it you may work on this issue if no one else comments on it.

@ArtiHere you can work on this issue, submit your solution in the ASME-Cummins folder by 3rd Apr'23, 11:59 pm

@ArtiHere
Have you Completed the issues?
Please create pull request.


Hiring & Looking Modal

What Functionality Are You Suggesting?
When a user runs the /hl command below the embed which is currently displayed should be two buttons:

Hire Someone
Offer A Service

Clicking on one of these buttons should trigger the hiring/looking flow. A modal should be presented to the user with the following:

A short text input of "Job Title" which is required with a maximum character length of 246
A long text input of "Job Description" which is required with a maximum character length of 4000
A short text input of "Payment" which is required with a maximum character length of 50

Once submitted, an embed should be posted in the #hiring-or-looking channel. With:

A title of "Hiring: " or "Offering: " depending on the button clicked
A body containing their input for "Job Description"
An inline field of "Payment" with their input for "Payment"
An inline field of "Posted By" with a mention of the user who triggered the flow

Examples:

Why Will This Benefit The Community?
Should hopefully reduce the amount of moderation required for the channel and remove a lot of the spammy looking posts.

Closing as we now use forums for Hiring/Looking posts rather than a text channel.


Place Nginx in front of Elasticsearch

Need to implement for security purposes

Basic configurations done, access is through port 8080


[Feature] allow using json file for theme tokenColors

currently we can only use tmTheme file as a template for the tokenColors config, however it would be much easier if we can also use a json file, so multi themes extension can extend the syntax definition without the need for a build tool.

We support tmTheme files, but also have our own, JSON base, *.color-theme.json format. That format is recommended as it  allows to set workbench colors as well as semantic colors.
Sorry if I misunderstood your request.  Can you explain in more detail what you are doing?

Sorry if I misunderstood your request. Can you explain in more detail what you are doing?

sure, lets say u have 2 themes with different colors but with identical tokenColors.
according to the docs we can only use tmTheme file to extend from.
so i was asking if its possible to also allow json files to be included/extended as well.

So it's a request for the yeoman generator to also accept JSON?

mostly yes, or if possible is it possible to bypass the yeoman step and get the vsce command to do that on compiling time ?

vsce ist the command for publishing

No sure if that's helpful, the color-theme.json also support to include another theme definition.
https://github.com/microsoft/vscode/blob/f1d7f7f6ce3d0a343bf03bea198aeedefa6cfaf2/extensions/theme-defaults/themes/dark_plus.json#L4-L4
So you can have one theme file just defining the token colors and other file (with different colors) include that one,

thank you so much, thats exactly what i was looking for, where do you find those information ?
thanx again. 🏆


support for non-windows platforms

Hi,
i just tired to install the env on a linux machine and after some fiddling and wondering i noticed that the dependencies are windows specific (some of the packages are only available on windows).
Could you share steps to use this software on other platforms? If no other platform is supported it would also help to write this in the beginning of the installation steps to avoid having people spend hours fruitlessly debugging that process.
Thank you!

Thank you for the suggestion. I apologize for the trouble that you went through trying to get the software to work on a Linux machine. We are working on improving compatibility across different devices, and I will reach out as soon as we have ensured compatibility with Linux.


dns_cache: tracking ttl but not yet using it

Fixing the dns cache to not fake resolution
tracking TTL and staleness
flushing ttl and last resolution time to disk

This shouldn't have a functional change (apart from what we cache to disk, which isn't yet used) but I have  mild preference for keeping the refactor separate from resolution + lifetime changes.
Risk Level: medium (refactoring code)
Testing: updated unit tests, integration test
Docs Changes: n/a
Release Notes: n/a

/wait on ci

Matt can you pick up with snow out?

@snowp is still in today. @snowp if it's just a final pass can you review today? Otherwise I can take it over.


[R] Improve error handling in the dplyr NSE code

Describe the enhancement requested
Issue filed retrospectively for #41576.
Component(s)
R

Closed by https://github.com/apache/arrow/pull/41576


Import an existing JSON file

Hi,
A killing feature would be to import a JSON file and that it fills the db with it.
If this feature is not planned, I will do it with a python script that reads it and call the api, but I was wondering if it was planned?
Thank you again for the project

Hi @0xecute,
This feature is indeed planned, however we can't give you any release date yet.
In the meantime, your Python script calling the API could be a great idea


Recording mode needs to be enabled twice

In dev.almond.stanford.edu, I need to click twice the checkbox to enable recording mode.
That is, I click the checkbox once, and it immediately unchecks itself. I click it again, and it sticks.
Looking at the log, it seems the "start" request completed successfully, but the "status" request fired immediately after still returns "off". This might be a race condition where we fire the "status" request too early. It might be exacerbated by different requests hitting different frontend processes.

Fixed by #916


Fixed project search field

Motivation and context

Fixed #2900
How has this been tested?

Manually
Checklist

[x] I submit my changes into the develop branch
[x] I have added description of my changes into CHANGELOG file
- [ ] I have updated the documentation accordingly
- [ ] I have added tests to cover my changes
[x] I have linked related issues (read github docs)
[x] I have increased versions of npm packages if it is necessary (cvat-canvas,
cvat-core, cvat-data and cvat-ui)

License

[x] I submit my code changes under the same MIT License that covers the project.
Feel free to contact the maintainers if that's a concern.
[x] I have updated the license header for each file (see an example below)

# Copyright (C) 2021 Intel Corporation
#
# SPDX-License-Identifier: MIT

Coverage remained the same at 72.119% when pulling 7aeb2dea64998e956f1fc43488c1731355ac4d76 on dk/fix-issue-2900 into 9b62b71ab13f03258d698b4128669aab4c54fde5 on develop.

Hi @dvkruchinin
This case (described in related issue) probably could be covered by tests


graphql filter 'function' does not work on testnet

{
transactionBlocks(
filter: {function:"0x5f9d5e9596375888ff2b5d5a4dbffbb4c3a1a0489da87a69743b5cf2c2ad70e0"}
) {
nodes {
sender {
address
}
gasInput {
gasPrice
gasBudget
}
}
}
}

What do you mean by does not work?

I think what @enoloo may have meant is that they expected this query to return some results because package 0x5f9d5e9596375888ff2b5d5a4dbffbb4c3a1a0489da87a69743b5cf2c2ad70e0 indeed has some transactions calling it according to the explorer but our graphql server is returning empty results.:
{
  "data": {
    "transactionBlocks": {
      "nodes": []
    }
  }
}

yes, and how to do it?

I follow now. Thanks for raising this. We're reviewing internally and will circle back when we identify the issue.

I follow now. Thanks for raising this. We're reviewing internally and will circle back when we identify the issue.

Bug has fixed ?

Bug has been found and fixed in the code but we have not yet released the version that fixed this issue. I will report back when it's available.

Bug has been found and fixed in the code but we have not yet released the version that fixed this issue. I will report back when it's available.

Thx


Cookie proxying and auto redirect configuration #128

HttpHandlerOptions are added to ReRoute configuration and passed down to HttpClientHttpRequester as Request properties.
NB! Properties in FileHttpHandlerOptions are set to true as default not to ruin already existing applications

@aringlot Thank you for the contribution! I will review this ASAP!

The PR is very good thank you! The only problem is that AppVeyor cannot build it for some reason. I will retry the build and then investigate!

@aringlot I will merge this when the build pipeline is working again.

@TomPallister thank you for your response. I hope, the solution for build issue will be found soon.

@TomPallister
Hi,
I found out that you found a solution for build issue a couple of hours ago.
To fix this issue in my PR I've merged develop to my local branch.
The good news - acceptance tests passed locally. Here is a screen shot of it: http://prntscr.com/gv5amx.
The bad news - I have no clue why check is green and acceptance tests are red on build machine.

@aringlot I fixed the build earlier today!

@aringlot thank you very much for your contribution!

@TomPallister you are welcome!
Just curious. When do you plan next release ? :)

@aringlot I have just pushed the release but there is yet another problem with the cake script for releasing to nuget so I need to look into this next ;(

@aringlot Ive manually pushed a package with your changes
Install-Package Ocelot -Version 1.4.11
Should work...

@TomPallister thank you !
Unfortunately, I have not enough knowledge to help you with the cake script(
Btw, the only thing I have forgotten is to ask you if I should prepare an example for configuration documentation.

@aringlot yes please prepare documentation and I can add it to the wiki.

Use HttpHandlerOptions in ReRoute configuration to set up HttpHandler behavior:

AllowAutoRedirect is a value that indicates whether the request should follow redirection responses.
Set it true if the request should automatically follow redirection responses from the Downstream resource; otherwise false. The default value is true.
UseCookieContainer is a value that indicates whether the handler uses the CookieContainer property to store server cookies and uses these cookies when sending requests.
The default value is true.

Here is an example configuration:

{
"DownstreamPathTemplate": "/posts",
"DownstreamScheme": "https",
"DownstreamHost": "jsonplaceholder.typicode.com",
"DownstreamPort": 443,
"UpstreamPathTemplate": "/posts",
"UpstreamHttpMethod": [ "Get" ],
"QoSOptions": {
"ExceptionsAllowedBeforeBreaking": 3,
"DurationOfBreak": 10,
"TimeoutValue": 5000
},
"HttpHandlerOptions": {
"AllowAutoRedirect" : false,
"UseCookieContainer": false
}
}

@aringlot added to wiki..thanks!


Disabled Buttons when sending a request and passed missed parameter in cartUpdate

@manisaiprasad i can merge once you make these changes, Chaitanya is waiting on this update


Changed CredentialsVerifier contract.

What this PR changes/adds
Changed CredentialsVerifier to take a DidDocument as an input.
Why it does that
DidDocument is a good input to verify credentials, as the way to get the credentials (identityHub service) is provided in the DidDocument.
For example, an IdentityHubCredentialVerifier would extract the identityHub URL from the did and query IdentityHub to get the verifiable credentials.
Checklist

[ ] added appropriate tests?
[ ] performed checkstyle check locally?
[ ] added/updated copyright headers?
[ ] documented public classes/methods?
[ ] added/updated relevant documentation?
[ ] added relevant details to the changelog? (skip with label no-changelog)
[ ] formatted title correctly? (take a look at the CONTRIBUTING and styleguide for details)

Codecov Report

Merging #323 (1810788) into main (bd320ed) will increase coverage by 0.49%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main     #323      +/-   ##
==========================================
+ Coverage   67.31%   67.80%   +0.49%
==========================================
  Files         771      784      +13
  Lines       16599    17011     +412
  Branches     1050     1068      +18
==========================================
+ Hits        11173    11534     +361
- Misses       4945     4989      +44
- Partials      481      488       +7

Impacted Files
Coverage Δ

...nnector/identity/DecentralizedIdentityService.java
82.05% <100.00%> (-0.45%)
:arrow_down:

.../dataspaceconnector/spi/result/AbstractResult.java
72.72% <0.00%> (-7.28%)
:arrow_down:

...gotiation/observe/ContractNegotiationListener.java
0.00% <0.00%> (ø)

...ctnegotiation/ContractNegotiationApiExtension.java
100.00% <0.00%> (ø)

...tractnegotiation/ContractNegotiationRequested.java
100.00% <0.00%> (ø)

...contractnegotiation/ContractNegotiationFailed.java
100.00% <0.00%> (ø)

...tractnegotiation/ContractNegotiationInitiated.java
100.00% <0.00%> (ø)

...vent/transferprocess/TransferProcessInitiated.java
100.00% <0.00%> (ø)

...act/listener/ContractNegotiationEventListener.java
8.69% <0.00%> (ø)

...ntractnegotiation/ContractNegotiationApproved.java
100.00% <0.00%> (ø)

... and 20 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bd320ed...1810788. Read the comment docs.

fine for me, other than a few minor checkstyle issues (https://github.com/agera-edc/DataSpaceConnector/runs/7358772029?check_suite_focus=true)


Scroll Primitives

Overview
Scroll introduces / modifies the transaction and transaction receipt primitives. We should implement these primitives in reth or a new primitives crate.
Transaction types
Scroll supports the following transaction types:

Type 0: Legacy Transactions
Type 1: EIP-2930
Type 2: EIP-1559
Type 126: L1MessageTx (EIP-2718 typed transaction)

L1 Message Transaction (typed transaction)
L1 initiated transactions use the following transaction type associated with an EIP-2718 type of 0x7E. There are no fees payed for this transaction type on L2 as the fees are already payed for when the transaction is submitted on L1. The encoding follows the standard EIP-2718 encoding
struct L1MessageTX {
    queue_index: u64,
    gas: u64,
    to: Address,
    value: U256,
    data: Bytes,
    sender: Address,
}

Receipt
An additional field is added to the TransactionReceipt. This field includes the l1 fee. See reference here.
Account
The account field has two additional fields code_size and poseiden_code_hash. Also the account hashing scheme is different. In the first instance I propose we add these additional fields behind a feature flag but in the future we should provide an abstraction such as providing the Account object in NodePrimites.
Design
Currently the Transaction type (and associated types, e.g. TxType, PooledTransactionsElement) and Receipt are hardcoded. Modifications to these types are currently achieved via feature flags as seen by the optimism feature flag. In the first instance we should implement the scroll changes behind a feature flag but we should consult with the reth team to see if we can provide the necessary abstractions to allow for configurability of these types.

Edited to add "Type 1" to the supported transaction types.

Relevant reth tracking issue: https://github.com/paradigmxyz/reth/issues/11239

Overview
The account structure used in Scroll contains two extra fields: code_size and poseidon_code_hash. code_size needs to be present for the execution and both need to be present for the computation of the state root, see following link for the hashing scheme.
Propositions

Abstract the Account. Ongoing work is being done upstream for storage abstraction which would be required.
Keep the account type but compute the code size and the poseidon code hash at runtime. This might cause some overhead for the poseidon hashing of the bytecode. All this can be hidden behind the StateRoot GAT of the StateCommitment.
Add a feature cfg(feature = "scroll") for now, modifying the Account from Reth. This is the easiest option imo. This also currently limits the amount of change we need to get in the code, and helps us get a better feeling about how to better abstract it.

Details

Account Abstraction:

After execution of the payload, the HashedPostState is populated with the accounts on which changes have occurred. These accounts would need to be abstracted and implement a trait which would allow defining the hashing of the account’s address and the encoding of the account.
Providers modifications:

Modifying the AccountReader and the StateChangeWriter, while adding a GAT of Account would allow to read and write the abstracted account from/to the database.
Use a FullAccountFetcher trait in order to populate an abstracted account right before hash computation. This fetcher could be defined to retrieve missing information from the database (Poseidon code hash, code size,…) and limit the amount of additional abstraction we need to add/propagate in the code.

Runtime addition: this runtime addition can be implemented in the HashedPostState and the StateRoot themselves. Changes more limited compare to the abstraction solution.
Feature gating: modify the type itself and define the way to convert a BundleAccount from revm into an Account with the additional fields. Changes seem limited as well.

For solution 2 and 3, needs to be verified if we can have a way to retrieve the Poseidon code hash from the AccountInfo structure from revm.

Below I will specify the changes that could be implemented in reth and revm to support the different account structure of scroll.
Reth Account Abstraction
First we will introduce an Account trait with basic functionality and implement it on our concrete ScrollAccount type.
pub trait AccountT {
    const fn has_bytecode(&self) -> bool;
    fn is_empty(&self) -> bool;
    fn get_bytecode_hash(&self) -> B256;
    fn get_nonce(&self) -> u64;
    fn get_balance(&self) -> U256;
}

pub struct ScrollAccount {
    nonce: u64,
    balance: U256,
    bytecode_hash: Option<B256>,
    poseidon_bytecode_hash: Option<B256>,
    code_size: Option<usize>,
}

impl AccountT for ScrollAccount {
    const fn has_bytecode(&self) -> bool {
        self.bytecode_hash.is_some()
    }

    fn is_empty(&self) -> bool {
        self.nonce == 0 &&
            self.balance.is_zero() &&
            self.bytecode_hash.map_or(true, |hash| hash == KECCAK_EMPTY)
    }

    fn get_bytecode_hash(&self) -> B256 {
        self.bytecode_hash.unwrap_or(KECCAK_EMPTY)
    }

    fn get_nonce(&self) -> u64 {
        self.nonce
    }

    fn get_balance(&self) -> U256 {
        self.balance
    }
}

We would then provide the concrete ScrollAccount type in scroll NodePrimitives as follows:
pub trait NodePrimitives {
    type Account: AccountT;
}

impl NodePrimitives for ScrollPrimitives {
    type Account = ScrollAccount;
}

Revm AccountInfo abstraction
In revm we would need to introduce a trait for AccountInfo. This would look something like this:
pub trait AccountInfoT {
    fn new(balance: U256, nonce: u64, code_hash: B256, code: Bytecode) -> Self;
    fn balance(&self) -> U256;
    fn nonce(&self) -> u64;
    fn code(&self) -> Option<&Bytecode>;
    fn code_hash(&self) -> B256;
    fn set_code(&mut self, code: Bytecode);
    fn set_balance(&mut self, balance: U256);
    fn set_nonce(&mut self, nonce: u64);
    fn copy_without_code(&self) -> Self;
    fn is_empty(&self) -> bool;
    fn exists(&self) -> bool;
    fn has_no_code_and_nonce(&self) -> bool;
    fn is_empty_code_hash(&self) -> bool;
    fn take_bytecode(&mut self) -> Option<Bytecode>;
    fn from_balance(balance: U256) -> Self;
    fn from_bytecode(bytecode: Bytecode) -> Self;
}

pub struct ScrollAccountInfo {
    balance: U256,
    nonce: u64,
    code_hash: B256,
    code: Option<Bytecode>,
    poseidon_code_hash: B256,
    code_size: usize,
}

impl AccountInfoT for ScrollAccountInfo {
    ...
}

We would then provide the concrete Account type by adding a new associated type in EvmWirign and specifying the ScrollAccountInfo type when configuring the evm.
pub trait EvmWiring: Sized {
    /// External context type
    type ExternalContext: Sized;

    /// Chain context type.
    type ChainContext: Sized + Default + Debug;

    /// Database type.
    type Database: Database;

    /// Account type.
    type Account: AccountT;

    /// The type that contains all block information.
    type Block: Block;

    /// The type that contains all transaction information.
    type Transaction: Transaction;

    /// The type that enumerates the chain's hardforks.
    type Hardfork: HardforkTrait;

    /// Halt reason type.
    type HaltReason: HaltReasonTrait;
}

This is a very invasive change and impacts many different data structures within revm so I think this is a no-go.
ScrollStateProviderDatabase and BundleState context
This solution would mitigate any changes needed in revm and would keep the additional account fields encapsulated in reth.
The idea would be to introduce a new ScrollStateProviderDatabase which is a modified version of StateProviderDatabase. This stateful object would include a cache for scroll code fields which is code_hash -> (code_size, poseidon_code_size) as seen below:
pub struct ScrollStateProviderDatabase<DB>{
    scroll_code: HashMap<B256, (usize, B256)>,
    pub db: DB
}

For the implementation of DatabaseRef on ScrollStateProviderDatabase we would cache the scroll code fields before returning the AccountInfo as follows:
impl<DB: EvmStateProvider> DatabaseRef for ScrollStateProviderDatabase<DB> {
    type Error = <Self as Database>::Error;

    /// Retrieves basic account information for a given address.
    ///
    /// Returns `Ok` with `Some(AccountInfo)` if the account exists,
    /// `None` if it doesn't, or an error if encountered.
    fn basic_ref(&self, address: Address) -> Result<Option<AccountInfo>, Self::Error> {
        let account = self.basic_account(address)?;
        let (account_info, scroll_code) = account.into_parts();
        self.scroll_code.insert(account.info.code_hash, scroll_code);
        Ok(account_info)
    }

We then look to introduce an additional generic context field on BundleState, hopefully upstream should support this as the surface area shouldn't be too large.
pub struct BundleState<C> {
    /// Account state.
    pub state: HashMap<Address, BundleAccount>,
    /// All created contracts in this block.
    pub contracts: HashMap<B256, Bytecode>,
    /// Additional context associated with the bundle state.
    pub context: C,
    /// Changes to revert.
    ///
    /// Note: Inside vector is *not* sorted by address.
    /// But it is unique by address.
    pub reverts: Reverts,
    /// The size of the plain state in the bundle state.
    pub state_size: usize,
    /// The size of reverts in the bundle state.
    pub reverts_size: usize,
}

Then on State we can implement a trait TakeScrollCode which will take the scroll code map from the database and store it in BundleState.
pub trait MergeScrollCode {
    fn merge_scroll_code(&mut self, code_hash: B256);
}

impl<DB> MergeScrollCode for State<DB> {
    fn merge_scroll_code(&mut self, code_hash: B256) {
        let scroll_code = self.db.scroll_code.take();
        self.bundle_state.context = scroll_code;
    }
}

This will mean that all code size and poseiden code hashes that were read from the database are available in BundleState. For newly deployed contract code we will have the bytecode available in the BundleState such that we can compute the poseiden hash and code size at runtime. Therefore when we convert from BundleState into HashedPostState which contains a mapping of ScrollAccount's we will have all of the code information available in the environment to populate those additional fields.

I agree with the second solution. Just two nits:

I think we need Database and not DatabaseRef implemented, here is the trait bound on the BlockExecutorProvider:

    /// Creates a new executor for single block execution.
    ///
    /// This is used to execute a single block and get the changed state.
    fn executor<DB>(&self, db: DB) -> Self::Executor<DB>
    where
        DB: Database<Error: Into<ProviderError> + Display>;

While waiting for the update to the BundleState to be merged upstream, I propose introducing our own ScrollBundleState:

pub struct ScrollBundleState {
    context: HashMap<B256, (usize, B256)>,
    bundle: BundleState
}

which we can re-export via the reth_evm crate:
// Convenience re-exports.
pub use revm::{self, *};
#[cfg(feature = "scroll")]
pub use revm_scroll::ScrollBundleState as BundleState

I agree with the second solution. Just two nits:

I think we need Database and not DatabaseRef implemented, required by BlockExecutorProvider for live sync block execution.

Agreed, the DatabaseRef trait wouldn't work as it takes an immutable &self reference but we need to modify the code cache.

While waiting for the update to the BundleState to be merged upstream, I propose introducing our own ScrollBundleState:

pub struct ScrollBundleState {
    context: HashMap<B256, (usize, B256)>,
    bundle: BundleState
}

Looks reasonable to me. The only question I have is do we want to unpack fields from BundleState such that they are at the root level of ScrollBundleState. I haven't looked at the code but I wonder if there are places in the code where public fields of BundleState are accessed directly.
I would propose that we open an issue in upstream revm to suggest the introduction of this new generic context field to get feedback. Would you be able to do that please @greged93?

After some research on the areas of the codebase affected by the above changes, I came across the below related to persistence:

During both live sync and pipelining, state is persisted into the database using the StateChangeWriter.
Methods write_state_reverts and write_state_changes from the trait receive PlainStateReverts and StateChangeset respectively. These structures don't have access to Account but rather AccountInfo from revm.
Method write_hashed_state receives a HashedPostStateSorted which does use the Account.
This means we wouldn't currently be able to maintain correct state in database for the code_size and the poseidon_code_hash fields when writing reverts or changes, while it is possible for the hashed state.

The below details a proposed approach to abstract the BundleState completely, in order to avoid issues with the StateChangeWriter:
Add a ExecutionPrimitives GAT into the NodePrimitives:
pub trait NodePrimitives {
    /// Block primitive.
    type Block;
    /// Node execution primitives
    type ExecutionPrimitives: NodeExecutionPrimitives;
}

We then bound the ExecutionPrimitives GAT with NodeExecutionPrimitives, which we use to define an additional execution context and a BundleState.
pub trait NodeExecutionPrimitives {
    /// Additional context extracted during execution
    type Context;
    /// Bundle state of values changed during execution
    type BundleState: From<(revm::db::BundleState, NodeExecutionPrimitives::Context)> + BundleStateT;
}

pub trait BundleStateT {
    type Reverts;
    type StateChangeset;
    ...
}

As an example, for Scroll, we would implement the following:
type KeccakHash = B256;
type PoseidonHash = B256;
type CodeSize = usize;
type ScrollExecutionContext = HashMap<KeccakHash, (CodeSize, PoseidonHash)>;

pub struct ScrollBundleState {
    context: ScrollExecutionContext,
    ...,
}

impl From<(revm::db::BundleState, ScrollExecutionContext)> for ScrollBundleState {
    fn from(value: (BundleState, ScrollExecutionContext)) -> Self {
        Self {
            context: value.1,
            ...,
        }
    }
}

pub struct ScrollExecutionPrimitives;

impl NodeExecutionPrimitives for ScrollExecutionPrimitives {
    type Context = ScrollExecutionContext;
    type BundleState = ScrollBundleState;
}

Using the ScrollStateProviderDatabase, we can then retrieve this context and convert the revm::BundleState to the NodeExecutionPrimitives::BundleState.
We will start implementing the above when time allows it and pushing it to upstream, but in the meantime will progress by defining our own ScrollBundleState and re-exporting it via a feature flag in reth_revm.
The ScrollBundleState would use a ScrollAccountInfo in place of the AccountInfo from revm, as well as our own ScrollPlainStateReverts and ScrollStateChangeset, which allows us to avoid the afore mentioned issue.

This looks good to me, just a few points. Do we need to introduce NodeExecutionPrimitives in NodePrimitives or can the concrete BundleState type be inferred by what is returned from the Executor? Maybe we just need to add the BundleState generic on ExecutionOutcome?
For the ScrollBundleState do we need to store the context field or can we just parse it during the conversion From<(revm::db::BundleState, ScrollExecutionContext)> for ScrollBundleState such that we construct ScrollAccountInfo in the BundleAccount's within ScrollBundleState?

This looks good to me, just a few points. Do we need to introduce NodeExecutionPrimitives in NodePrimitives or can the concrete BundleState type be inferred by what is returned from the Executor? Maybe we just need to add the BundleState generic on ExecutionOutcome?

After some discussion, the following route has been chosen:
Take advantage of the Output GAT on Executor and BatchExecutor by extracting them one level higher in the BlockExecutorProvider:
pub trait BlockExecutorProvider: Send + Sync + Clone + Unpin + 'static {
    type ExecutorOutput;
    type BatchExecutorOutput;

    type Executor<DB: Database<Error: Into<ProviderError> + Display>>: for<'a> Executor<
        DB,
        Input<'a> = BlockExecutionInput<'a, BlockWithSenders>,
        Output = Self::ExecutorOutput,
        Error = BlockExecutionError,
    >;
    type BatchExecutor<DB: Database<Error: Into<ProviderError> + Display>>: for<'a> BatchExecutor<
        DB,
        Input<'a> = BlockExecutionInput<'a, BlockWithSenders>,
        Output = BatchExecutorOutput,
        Error = BlockExecutionError,
    >;
    ...

This removes the need for the NodeExecutionPrimitives and allows us to define in our BlockExecutorProvider the structure returned after execution.
We will need to bound the ExecutorOutput and the BatchExecutorOutput by an additional trait ExecutionOutcome which would provide some fetching functionality. A fitting candidate would be the ExecutionDataProvider, which would need to be slightly updated:
pub trait ExecutionDataProvider {
    type Reverts;
    type StateChangeset;

    /// Return the execution outcome.
    fn execution_outcome(&self) -> &ExecutionOutcome;
    /// Return block hash by block number of pending or canonical chain.
    fn block_hash(&self, block_number: BlockNumber) -> Option<BlockHash>;
}

This looks good to me, just a few points. Do we need to introduce NodeExecutionPrimitives in NodePrimitives or can the concrete BundleState type be inferred by what is returned from the Executor? Maybe we just need to add the BundleState generic on ExecutionOutcome?

After some discussion, the following route has been chosen:
Take advantage of the Output GAT on Executor and BatchExecutor by extracting them one level higher in the BlockExecutorProvider:
pub trait BlockExecutorProvider: Send + Sync + Clone + Unpin + 'static {
    type ExecutorOutput;
    type BatchExecutorOutput;

    type Executor<DB: Database<Error: Into<ProviderError> + Display>>: for<'a> Executor<
        DB,
        Input<'a> = BlockExecutionInput<'a, BlockWithSenders>,
        Output = Self::ExecutorOutput,
        Error = BlockExecutionError,
    >;
    type BatchExecutor<DB: Database<Error: Into<ProviderError> + Display>>: for<'a> BatchExecutor<
        DB,
        Input<'a> = BlockExecutionInput<'a, BlockWithSenders>,
        Output = BatchExecutorOutput,
        Error = BlockExecutionError,
    >;
    ...

This removes the need for the NodeExecutionPrimitives and allows us to define in our BlockExecutorProvider the structure returned after execution.
We will need to bound the ExecutorOutput and the BatchExecutorOutput by an additional trait ExecutionOutcome which would provide some fetching functionality. A fitting candidate would be the ExecutionDataProvider, which would need to be slightly updated:
pub trait ExecutionDataProvider {
    type StateChanges;
    type Reverts;
    type PlainStateChanges;

    /// Return the reverts from execution.
    fn reverts(&self) -> &Self::Reverts;
    /// Returns the complete state changes from execution.
    fn state_changes(&self) -> &Self::StateChanges;
    /// Returns a reduced view of changes for database commitment.
   fn plain_state_changes(&self) -> &Self::PlainStateChanges;
}

@frisitano I think we can close this

@frisitano I think we can close this

agreed.
closed by #39, #33, #41, #42


NPD is lack of k8s cluster level e2e test coverage

We should add k8s cluster level e2e test coverage for NPD, so that situations like #227 could be prevented. There were past effort #43 and #86, but those never got completed. It is time to revisit.

Seems that presubmit e2e test needs to work with test-infra.

NPD presubmit and CI jobs are now available:

presubmit: https://k8s-testgrid.appspot.com/presubmits-node-problem-detector
CI: https://k8s-testgrid.appspot.com/sig-node-node-problem-detector

Will cherry pick the following PRs into k8s 1.11, 1.12, and 1.13 branches:

https://github.com/kubernetes/kubernetes/pull/73288
https://github.com/kubernetes/kubernetes/pull/74922 (test fix)
https://github.com/kubernetes/kubernetes/pull/75063 (test fix)

https://github.com/kubernetes/kubernetes/pull/73288 will be available in Kubernetes v1.11.10+, v1.12.8+, and v1.13.6+.


faostat: AssertionError in food balances

During ETL of faostat_fbsc, an assertion fails on this line:

@lucasrodes and I looked into this at some point.
This is @lucasrodes:

Although the bulk data comes from walden, the additional meta data (tables meta_item, meta_area, etc.) do not come from Walden. Instead they are added in the meadow step calling an FAO API.
This is very not-ideal, instead these tables should be also added in the walden step. So we have a complete frozen picture
However, we did not do this in Walden because a small restructuring of Walden is required and we did not prioritize this. Small refactoring = allowing Walden to have several source data files (currently it only has one).

The FAO updated their dataset at some point, and so some definitions between the data file sourced from walden and the data sourced from the FAO API diverged.

Ah, the lack of multi-file support strikes! https://github.com/owid/walden/issues/5

Thanks for raising it!
I can look into this later this week and fix it for the time being. However, I expect this error to pop up again as FAO updates the metadata files.

The current workaround is to build the ETL without fao data .venv/bin/etl --exclude=fao

@Marigold Just making a note that we should chat together about how the ETL runs, and about ways to get it running more reliably


Add live reload for static sites

Fixes #653. This PR adds support for wrangler preview --watch to work with static assets.
There is still a quirks I want to work out in the future: is the 2000 ms cooldown period enough to account for large static site generator builds that exceed that time? (I think it should be, we can wait to hear otherwise...)

Fixes #653

QA-ing this; the output confuses me, esp the duplicate "ignoring" line, and the "Success" message, i'm not sure what has succeeded in each case.
$ wrangler-dev preview --watch
🌀  Creating namespace for Workers Site "__hello-react-workers_sites_assets_preview"
💁  ignoring directory node_modules
✨  Success
💁  ignoring directory node_modules
✨  Success
✨  Built successfully, built project size is 4 KiB.

@ashleymichal ugh something in master is responsible for these double printouts, they did not happen until I merged master in. I'm investigating.


SO provides new column for ncumul_hosp - scraper doesn't pick up on it

they added ncumul_hosp but the scraper doesn't pick up on it

On it.


Show KOOL_VERSION on build_artifacts script

It would be nice to print out the KOOL_VERSION variable value before start building the artifacts for a new release, just for double-checking on a release process.

Hi 🖖  @danielsuguimoto  I would like to contribute 👍  where I have to add this print out?

@marti1125 I guess at the end of the script. What do you think about it, @fabriciojs ?

@marti1125 @danielsuguimoto
I guess we could do even more -

Right after the check for existing KOOL_VERSION, around line 13 (https://github.com/kool-dev/kool/blob/master/build_artifacts.sh#L13), we could add a confirmation since this is a manual process at the moment - You are going to build all artfacts for version **X.X.X**. Continue? (y/N) then have the user input y or yes in order to continue. (still could be automated at some point with yes |)

Then again at the end of the script, we could add as suggested a final message reinstating the version: Finished building all artfacts for version **X.X.X**.

Sounds a good plan for me

Hi @fabriciojs I created a new PR https://github.com/kool-dev/kool/pull/165 please check 😄

PR merged onto master.


Updating for consistency with API changes to Notes API

Coverage increased (+0.2%) to 52.026% when pulling 4aeb24ded053cd81bc4238f3b5919932bdac351c on update-notes into 91d98a0fbb0097bde7c82bdf9e13af711aa216a7 on master.


Crontabs in docker/alpine

There doesn't seem to be any elegant solutions to this, so i'll leave it completely manual until docker support is more mature.

Have you checked this? https://stackoverflow.com/questions/36453787/failed-to-edit-crontab-linux-alpine ?

Related to it, I found out this, unfortunately I can't test it out since I don't know how to create docker images myself with these changes.

yes, that's what i don't find elegant, i don't even have an entrypoint right now

a separate cron container seems to work, but pushing multiple containers/compose seems to be at a very early stage

a separate cron container seems to work, but pushing multiple containers/compose seems to be at a very early stage

Just set-up the multiple containers with docker compose, will check it out

I forgot to give feedback, it is working fine.

@KurtzPT So cron is working when using the dockerhub image?

@KurtzPT So cron is working when using the dockerhub image?

@danyal I didn't use the dockerhub image, I build it myself with docker compose, so I don't know. But according to @MrBrax on the front page:

Docker hub doesn't seem to fully support docker-compose apps, so the cron stuff won't work.

@danyal I didn't use the dockerhub image, I build it myself with docker compose, so I don't know. But according to @MrBrax on the front page:

Docker hub doesn't seem to fully support docker-compose apps, so the cron stuff won't work.

@KurtzPT Ah ok. I'm a Docker novice and was wondering if you could give me some info on how to build manually? I can get this working via the hub image but not build. Where do I need to run the build command and what files need to be in that dir besides the docker-compose? Do I pull the git repo first?

@KurtzPT Ah ok. I'm a Docker novice and was wondering if you could give me some info on how to build manually? I can get this working via the hub image but not build. Where do I need to run the build command and what files need to be in that dir besides the docker-compose? Do I pull the git repo first?


Default compression level of 1 should be 9

Is there a good reason to use the lowest compression level by default instead of the highest?

Are you aware that the decompression speed changes much less than the compression speed? I took some random Python, JavaScript, C, and Java files and concatenated them into a big 71 MB file:
% ll random-source-code.txt
-rw-r--r-- 1 mg mg 71M Jan 21 17:23 random-source-code.txt

I then compressed it with gzip -1 and uncompressed it:
% time gzip -1 random-source-code.txt
gzip -1 random-source-code.txt  1.33s user 0.04s system 99% cpu 1.372 total
% ll random-source-code.txt.gz
-rw-r--r-- 1 mg mg 17M Jan 21 17:23 random-source-code.txt.gz
% time gunzip random-source-code.txt.gz
gunzip random-source-code.txt.gz  0.60s user 0.09s system 99% cpu 0.695 total

I did this a couple of times, the timing was very stable (+/-0.01 sec).
With gzip -9 the numbers look like this:
% time gzip -9 random-source-code.txt
gzip -9 random-source-code.txt  8.74s user 0.03s system 99% cpu 8.778 total
% ll random-source-code.txt.gz
-rw-r--r-- 1 mg mg 14M Jan 21 17:23 random-source-code.txt.gz
% time gunzip random-source-code.txt.gz
gunzip random-source-code.txt.gz  0.52s user 0.08s system 98% cpu 0.612 total

What we see here is that using gzip -9 takes significantly longer, but produces a 18% smaller file in this example. Uncompressing the smaller file produced by gzip -9 is a little faster than uncompressing the larger file made by gzip -1.
I tried again with just some JavaScript files and got these numbers for a 1.3 MB file:
% ll source.js
-rw-r--r-- 1 mg mg 1.3M Jan 21 17:35 source.js
% time gzip -1 source.js
gzip -1 source.js  0.04s user 0.00s system 96% cpu 0.037 total
% ll source.js.gz
-rw-r--r-- 1 mg mg 399K Jan 21 17:35 source.js.gz
% time gunzip source.js.gz
gunzip source.js.gz  0.02s user 0.00s system 89% cpu 0.022 total
% time gzip -9 source.js
gzip -9 source.js  0.10s user 0.00s system 97% cpu 0.098 total
% ll source.js.gz
-rw-r--r-- 1 mg mg 332K Jan 21 17:35 source.js.gz
% time gunzip source.js.gz
gunzip source.js.gz  0.01s user 0.01s system 95% cpu 0.021 total

We see again that the uncompress time is virtually unchanged, if not a little better.

@mgeisler gzip seems to have a decompression speed that does not vary with compression level but this is not the case with zip for example.
On top of that, if you really care about 18% of your archive size, you should first look at your code and assets, you'll be able to gain much much more by removing unused code and libraries and optimizing images. If so, you're in the very small part who care about.
Additionally, node is mostly used to develop and package web apps. The archive is uploaded to the server and unziped once, unlike the build and zipping step that could be done more often locally, in test/dev environment. So gaining 7s of zipping is much more important than gaining 18% size for most users.

again agree with @felicienfrancois here. you can always set the level via option if you feel differently

The scenario I was thinking of was generating a compressed (and minified) version of my JavaScript, a file that would be generated once and downloaded by many clients. That's why I was interested in spending a little more time upfront compressing the file so that the clients may save a bit of time when downloading it.
If that's not the scenario you have in mind, then your considerations may of course change :-) It's fine as it is now -- I've already set the config option :-)


Added example in docs to create keys with expires

Closes #159

Cool! Could you please change ex to EX since the latter is used in the official documentation.

Sure! Done  :)

:laughing:


fix: large session expiration consideration

Related Issues
Fixes https://github.com/descope/etc/issues/4292
Description
when session expiration is above 24.8 days - setTimeout will fire immediately - resulting a loop of refresh where eventually it will result 429
see more details in https://developer.mozilla.org/en-US/docs/Web/API/setTimeout#maximum_delay_value
the solution here is to limit upper session with that upper bound
here is where session expiration is 30 days

👍👍


openssl-qat-engine: update qatlib and QAT engine versions

Signed-off-by: Mikko Ylinen<EMAIL_ADDRESS>

Codecov Report

Merging #1083 (6bd372d) into main (3c948cc) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1083   +/-   ##
=======================================
  Coverage   53.01%   53.01%
=======================================
  Files          40       40
  Lines        4350     4350
=======================================
  Hits         2306     2306
  Misses       1917     1917
  Partials      127      127

:mega: Codecov can now indicate which changes are the most critical in Pull Requests. Learn more


Feature request: link iAPS carb input to an existing food database

Is it possible to enable a feature that will allow the meal input on iAPS (carbs, fat, protein) to be linked to existing online databases such as my fitness pal or carbs and cals etc?
The thought is that it would be possible to scan barcodes (for sandwiches, snacks etc) and the info populated into the meal entry tab on iAPS.

It appears carbs and Cals now writes to Apple health so it would be a quick win if iAPS could read carbs etc from there.


for Mac

There is no option to choose a Snap Camera in Google Meet or any other platform on Mac how can I use these filters for video calls? Please help me I have a MacBook Air  M1

How can I try new lenses on Snapchat?

I have the same problem

Same problem

me too

same

There is no option to choose a Snap Camera in Google Meet or any other platform on Mac how can I use these filters for video calls? Please help me I have a MacBook Air M1

Have you tried this solution? It works for MacOS Sonoma 14.1 and above.

Did you try deleting and reinstalling?I have no problem using snap camera
on my macbook air M1 with its built in camera. Also, go into your browser
settings and make sure snap camera permissions are on for video (go to
privacy settings). Sometimes I can’t open it from the docked version (which
I delete) and need to open it from within apps folder.
On Wed, Dec 11, 2024 at 11:43 PM alô mundo! @.***>
wrote:

There is no option to choose a Snap Camera in Google Meet or any other
platform on Mac how can I use these filters for video calls? Please help me
I have a MacBook Air M1
Have you tried this solution https://support.apple.com/en-us/108387? It
works for MacOS Sonoma 14.1 and above.
—
Reply to this email directly, view it on GitHub
https://github.com/jaku/SnapCameraPreservation/issues/103#issuecomment-2537803065,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/BLM7B4XSF6DNHGZJXUGPDS32FEIAZAVCNFSM6AAAAABOJO5PF6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMZXHAYDGMBWGU
.
You are receiving this because you commented.Message ID:
@.***>


Bump versions

$subject

PR builder started
Link: https://github.com/wso2/product-is/actions/runs/5473985898

PR builder completed
Link: https://github.com/wso2/product-is/actions/runs/5473985898
Status: failure

PR builder started
Link: https://github.com/wso2/product-is/actions/runs/5475500904

PR builder completed
Link: https://github.com/wso2/product-is/actions/runs/5475500904
Status: success


[BugFix, Featuree] Vmap randomness in losses

Description
RuntimeError running off-policy examples with dropout.
RuntimeError: vmap: called random operation while in randomness error mode. Please either use the 'same' or 'different' randomness flags on vmap or perform the randomness operation out of vmap

Propose setting randomness flag to randomness="different" and not same as this would be the natural case running over the networks sequentially. Also, we want to have different randomness values in for example the ensemble Q-Networks.
Motivation and Context
Why is this change required? What problem does it solve?
If it fixes an open issue, please link to the issue here.
You can use the syntax close #15213 if this solves the issue #15213

[ ] I have raised an issue to propose this change (required for new features and bug fixes)

Types of changes
What types of changes does your code introduce? Remove all that do not apply:

[X] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds core functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] Documentation (update in the documentation)
[ ] Example (update in the folder of examples)

Checklist
Go over all the following points, and put an x in all the boxes that apply.
If you are unsure about any of these, don't hesitate to ask. We are here to help!

[X] I have read the CONTRIBUTION guide (required)
[ ] My change requires a change to the documentation.
[ ] I have updated the tests accordingly (required for a bug fix or a new feature).
[ ] I have updated the documentation accordingly.

Maybe we could make this optional? I'm assuming that if this isn't the default for vmap it is because it comes with a drawback, we should investigate

I found this on functorch:
The flag can only be passed to vmap and can take on 3 values, “error,” “different,” or “same,” defaulting to error. Under “error” mode, any call to a random function will produce an error asking the user to use one of the other two flags based on their use case.

We could make it an additional flag for the objective being error as default like vmap_randomness="error". But not sure, it's only important when using the objective with networks that utilize dropout.

Looks good to me, I will adapt one objective class and then we can recheck if its what we expect.
I wonder if at some point we can put together all those general functions to a higher level objective class lets say OffpolicyObjective from which the specific objectives inherit. Such a higher level class would handle the vmap and also the make_value_estimator etc. Because right now the objective classes get bigger and bigger probably hard to read and understand for people who just want to see the td3/sac loss calculation.

Turns out that self.modules does not include modules like Dropout. I updated the function accordingly using actor and qvalue modules directly like:
    @property
    def vmap_randomness(self):
        if self._vmap_randomness is None:
            # look for nn.Dropout modules
            dropouts = (torch.nn.Dropout, torch.nn.Dropout2d, torch.nn.Dropout3d)
            for a, q in zip(
                self.actor_network.modules(), self.qvalue_network.modules()
            ):
                if isinstance(a, dropouts) or isinstance(q, dropouts):
                    self._vmap_randomness = "different"
                    break
            else:
                self._vmap_randomness = "error"

        return self._vmap_randomness

let me know what you think :)

OffpolicyObjective

In general we can think about it but if there's a clear use case for it, not just some vmap compatibility. Plus I'm working on consistent dropout for RL, which works also on the online setting...

    @property
    def vmap_randomness(self):
        if self._vmap_randomness is None:
            # look for nn.Dropout modules
            dropouts = (torch.nn.Dropout, torch.nn.Dropout2d, torch.nn.Dropout3d)
            for a, q in zip(
                self.actor_network.modules(), self.qvalue_network.modules()
            ):
                if isinstance(a, dropouts) or isinstance(q, dropouts):
                    self._vmap_randomness = "different"
                    break
            else:
                self._vmap_randomness = "error"

        return self._vmap_randomness

Ah right bc we don't want the modules to appear in the modules list actually!
The problem with this is that it breaks as soon as you name your actor_network something like actor.
Also why do we consider that actor and qvalue networks have the same number of modules? I don't think we should use zip, we should iterate over one and then the other. Something like
do_break = False
for val in self.__dict__.values():
    if isinstance(val, nn.Module):
        for module in val.modules():
             if isinstance(module, RANDOM_MODULE_LIST): # not only nn.Dropout is random, could be something else
                 self._vmap_randomness = "different"
                 do_break = True
                 break
    if do_break:
        # double break
        break
else:
    self._vmap_randomness = "error"

(one guy claims that we should break nested loops with exception but I'd rather be dead in a ditch than doing that)

Also why do we consider that actor and qvalue networks have the same number of modules? I don't think we should use zip, we should iterate over one and then the other.

You are right, we shouldn't expect them to be the same. I'll update it!

Some tests would be helpful!

Any Idea regarding how we could test it effectively? Adding dropout as an option to the mock actors/critics in the tests for the objectives is an option but this would mean adding it for all of the objectives independently again. Not sure if ideal...

@BY571
You can adapt this:
from torchrl.objectives import LossModule
import torch
from torchrl.objectives.utils import _vmap_func
from tensordict.nn import TensorDictModule as Mod
from torch import nn
from tensordict import TensorDict

def test_loss_vmap_random():
    class MyLoss(LossModule):
        def __init__(self):
            super().__init__()
            mod = Mod(nn.Dropout(0.1), in_keys=["obs"], out_keys=["action"])
            self.convert_to_functional(mod, "mod", expand_dim=4)
            self.vmap_mod = _vmap_func(self.mod, (None, 0))

        def forward(self, td):
            out = self.vmap_mod(td, self.mod_params)
            return {"loss": out["action"].mean()}
    loss_mod = MyLoss()
    td = TensorDict({"obs": torch.randn(3, 4)}, [3])
    loss_mod(td)
test_loss_vmap_random()

Please clean up the code before putting it in ;)


regexp: character class not working in set

Package: ucode
Version: 2024.06.18~4dd98370-r1
Depends: libc, libucode20230711
Platform: x86/64-generic SNAPSHOT
When I try to use the digits class \d in a character set, it doesn't match, but if I expand it manually, it does:
$ ucode -p 'version_code="r24012-d8dd03c46f"; match(version_code, /^r(\d+)-[\da-f]+$/i)'

$ ucode -p 'version_code="r24012-d8dd03c46f"; match(version_code, /^r(\d+)-[0-9a-f]+$/i)'
[ "r24012-d8dd03c46f", "24012" ]

Clearly the class works, as it matches in the capture group, but fails when added to a set.

Yes, this is a limitation of POSIX extended regular expressions. Outside character classes, ucode translates some well known macros manually into sets, see here: https://github.com/jow-/ucode/blob/master/lexer.c#L292
But inside bracket expressions this is not really possible by doing mere text substitutions since it is impossible to express negations this way, e.g. [xy\d] could be substituted with [xy[:digit:]] but [xy\D] cannot be substituted with [xy^[:digit:]] as it would match a literal ^ and the numbers 0-9 instead of everything except 0-9.
Anyhow, the tl;dr version is that macros such as \d, \D, \w etc. are specific to Perl compatible regular expressions while ucode, for size reasons, utilizes POSIX extended regular expressions as supported by the underlying libc.
In your particular case, consider using /^r(\d+)-[[:xdigit:]]+$/ instead.

Got it, thanks!


[Feat-Req]: Improve Hydration Mismatch Handling

Use https://github.com/danielroe/vue-bind-once instead of onMounted Hook.
Vue-Bind-Once allows SSR for unique IDs like in https://github.com/fantasyflip/nuxtwind/pull/38

fixed in #71


[RTM] use nibabel to merge nifti files

Fixes #221

Without this PR beast processing will crash on most runs. Merging.


Fixed CDI API version mismatch by updating Java EE 6 to Java EE 7

Fixed conflicts for weld. Weld required version CDI 1.1 or better. CDI 1.0 API detected due to Java EE6

This change is

Thanks for the contribution @dipak-pawar! Can you also adjust guides in other languages?

getting_started_de.textile
getting_started_el_gr.textile
getting_started_es.textile
getting_started_fr.textile
getting_started_it.textile
getting_started_ja.textile
getting_started_nl.textile
getting_started_pl.textile
getting_started_pt.textile

@bartoszmajsak  Adjusted guides in other languages as well.

There are few other minor things which would be great to bring together with this PR.
Forge commands, such as
project-add-dependencies org.jboss.spec:jboss-javaee-6.0:1.0.0.Final:provided:pom

still points to the wrong version. Sample generated POM which we paste as the one Forge generated should be adjusted accordingly.
Also, the warning about Java EE 6 API should be entirely removed, as we are not using 7.0 version, which I believe doesn't have this problem anymore. Exactly this section:

The org.jboss.spec:jboss-javaee-6.0 artifact is comprised of only standard Java EE 6 APIs. We strongly recommend that you do not use the Java EE API artifact with coordinates javax:javaee-api. The latter artifact contains classes with stripped method bodies, which will cause your application to throw obscure Absent Code errors if present on the classpath at runtime (even when running tests). Read this FAQ for more details.

In addition, we could bump JUnit version from 4.8.2 to latest 4.12.

@bartoszmajsak Updated PR with all suggested changes. Please have a look into it.

Great! Thanks a lot!

Upstream @2c9c3a8. Thanks!


Set EnvironmentVariables from options

I would like to set some environment variables from the host ASP.NET application. I found out that JavaScriptService supports is: https://github.com/aspnet/JavaScriptServices/blob/fb19ee8923b5ef7ebb125bc6e5a1a182cfadfe88/src/Microsoft.AspNetCore.NodeServices/Configuration/NodeServicesOptions.cs#L90
Do you accept a PR that allows to set additional environment variable from NodeRenderEngineOptions?
Great project btw!

I'd be happy to accept a pull request!


Question : How to receive messages once connected to a server and subscribed to a topic

I am using Vue 3 and I managed to connect to a server and to subscribe to a topic, but i can't figure out how to display the messages..
Any help would be appreciated.
This is my code so far...
`<script lang='ts' setup>

    import { ref, watch, onMounted, onUnmounted} from "vue";
    import { useMQTT } from 'mqtt-vue-hook'
    import { toast } from 'vue3-toastify';
    import 'vue3-toastify/dist/index.css';

    const mqttHook = useMQTT()

    const protocol = "wss";
    const host = "xxxxxxxxxxxxxxxxx";
    const port = 443;

    mqttHook.connect(`${protocol}://${host}:${port}`, {
        clean: false,
        keepalive: 60,
        clientId: `mqtt_client_${Math.random().toString(16).substring(2, 10)}`,
        connectTimeout: 4000,
    })

    onMounted(() => {

        const mqttHook = useMQTT()
        // mqttHook.subscribe([...topic], qos, opts?, callback?, clientID?)
        // mqttHook.unSubscribe(topic, opts?, callback?, clientID?)
        // '+' == /.+/
        // '#' == /[A-Za-z0-9/]/
        mqttHook.subscribe(

            ['/v3/gre-application-1@ttn/devices/23-v6-02-20/#'],
            0,
            undefined,
            () => {
                console.log('subscribed!')
            }
        );

        // mqttHook.registerEvent(topic, callback function, vm = string, clientID?)
        // mqttHook.unRegisterEvent(topic, vm)
        mqttHook.registerEvent(
            '/v3/gre-application-1@ttn/devices/23-v6-02-20/#',
            (topic: string, message: string) => {
                console.log(`topic=${topic} message=${message.toString()}`);
            },
            'string_key',
        );

        mqttHook.registerEvent(
            'on-connect', // mqtt status: on-connect, on-reconnect, on-disconnect, on-connect-fail
            (topic: string, message: string) => {
                console.log('mqtt connected')
            },
            'string_key',
        );
    });

    onUnmounted(() => {

        mqttHook.unSubscribe(

            '/v3/gre-application-1@ttn/devices/23-v6-02-20/#',
            undefined,
            () => {
                console.log('unSubscribed!')
            }
        );
    })`

Hi @pierregremaud,
Thank you for your question. Your code is correct. Have you seen the "mqtt connected" log? Additionally, if you want to see the "topic=${topic} message=${message.toString()}", you need to publish a message to the topic "/v3/gre-application-1@ttn/devices/23-v6-02-20/#".
You can also refer to my other repository at "https://github.com/tommy44458/light-vue3-starter" to learn how to control the topic event using "mqtt-vue-hook".

Hi,
Thank you for your quick reply. I managed to receive the messages after a few tries. Everything is working now...
Any idea how I could manage the connection globally in the main.ts fine (using app.use(...)) and then do only subscribe and unsubscribe in the vue component ?

Hi @pierregremaud,
Congratulations! You can connect to the broker and subscribe to the topic from anywhere. For example, you can call mqttHook.connect() in main.ts and mqttHook.subscribe() or mqttHook.registerEvent() in a Vue component or another .ts file.


sorted_list.txt ==> sorted_list.dat

In README.md
In "usage": typo: ./lookup-service.py sorted_list.txt, it should be (I think) ./lookup-service.py sorted_list.dat

@asher-stern
Did you get this working?
When I use the dat file I get the following issue UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd8 in position 2: invalid continuation byte


Handled case for missing '\n' char in .bashrc

If '\n' char is missing at the end in .bashrc,
the previous command to add PATH will append onto
the last command in the .bashrc file

Yes, correct.


Add mod/modpacks support

btw if you're moving file structure and refactoring, modpack support should be added in a follow-up pr