Bug/check imds version 57

Description
aws.sh defaults to IMDSv1 instead of checking.  Any instances using IMDSv2 only (set to "required") fail when calling this module for things like getting the Region.
Documentation
Updated README and checked all examples.  Updated any reference to IMDS versions.
TODOs
Please ensure all of these TODOs are completed before asking for a review.

[x] Ensure the branch is named correctly with the issue number. e.g: feature/new-vpc-endpoints-955 or bug/missing-count-param-434.
[x] Update the docs.
[x] Keep the changes backward compatible where possible.
[ ] Run the pre-commit checks successfully.
[ ] Run the relevant tests successfully.

Related Issues
Issue #57
https://github.com/gruntwork-io/terraform-aws-monitoring/issues/340
https://github.com/gruntwork-io/terraform-aws-service-catalog/pull/1834
https://github.com/gruntwork-io/terraform-aws-openvpn/pull/144
And probably a lot more.

Ready for pre-commit checks and/or review.

@brikis98 would appreciate your feedback on this.  Seems like it's time to make IMDSv2 the default.  This PR retains the option to use IMDSv1, but also checks what is available on the instance.

The "if" statement actually checks for version 1, and if it is available it defaults to that.  If not it sets the default to version 2.
If the curl command is "refused" with an error code of 7, then it sets the default to 0 which should cause the script to fail.
Could reverse this and check for v2 by trying to get the token.  If that fails, check the v1.  If both fail, set to "0".

Changed to check for IMDSv2 first and if available, use it.  Fall back to IMDSv1 if v2 is not available.  If neither is available (i.e. the endpoint is disabled or blocked) set default version to 0.

Running local test now with
docker-compose up --exit-code-from bats_ubuntu2004_imds_v2 bats_ubuntu2004_imds_v2

Not sure what this is on about:
bash-commons-bats_ubuntu2004_imds_v2-1  | #     stream.read().decode(),
bash-commons-bats_ubuntu2004_imds_v2-1  | #   File "/usr/local/lib/python3.8/dist-packages/werkzeug/wsgi.py", line 577, in readall
bash-commons-bats_ubuntu2004_imds_v2-1  | #     data = self.read(1024 * 64)
bash-commons-bats_ubuntu2004_imds_v2-1  | #   File "/usr/local/lib/python3.8/dist-packages/werkzeug/wsgi.py", line 558, in readinto
bash-commons-bats_ubuntu2004_imds_v2-1  | #     b[:out_size] = data
bash-commons-bats_ubuntu2004_imds_v2-1  | # TypeError: can assign only bytes, buffers, or iterables of ints in range(0, 256)
bash-commons-bats_ubuntu2004_imds_v2-1  | # <IP_ADDRESS> - - [18/Jun/2024 18:20:45] "POST / HTTP/1.1" 500 -
bash-commons-bats_ubuntu2004_imds_v2-1  | #
bash-commons-bats_ubuntu2004_imds_v2-1  | # Unable to parse response (syntax error: line 1, column 0), invalid XML received. Further retries may succeed:

And in this bug branch:
bash-commons-bats_ubuntu2004_imds_v2-1  | #     return parse_func(stream, mimetype, content_length, options)
bash-commons-bats_ubuntu2004_imds_v2-1  | #   File "/usr/local/lib/python3.8/dist-packages/werkzeug/formparser.py", line 286, in _parse_urlencoded
bash-commons-bats_ubuntu2004_imds_v2-1  | #     stream.read().decode(),
bash-commons-bats_ubuntu2004_imds_v2-1  | #   File "/usr/local/lib/python3.8/dist-packages/werkzeug/wsgi.py", line 577, in readall
bash-commons-bats_ubuntu2004_imds_v2-1  | #     data = self.read(1024 * 64)
bash-commons-bats_ubuntu2004_imds_v2-1  | #   File "/usr/local/lib/python3.8/dist-packages/werkzeug/wsgi.py", line 558, in readinto
bash-commons-bats_ubuntu2004_imds_v2-1  | #     b[:out_size] = data
bash-commons-bats_ubuntu2004_imds_v2-1  | # TypeError: can assign only bytes, buffers, or iterables of ints in range(0, 256)
bash-commons-bats_ubuntu2004_imds_v2-1  | # expected: bar
bash-commons-bats_ubuntu2004_imds_v2-1  | # actual:
bash-commons-bats_ubuntu2004_imds_v2-1  | # Unable to parse response (syntax error: line 1, column 0), invalid XML received. Further retries may succeed:
bash-commons-bats_ubuntu2004_imds_v2-1  | # b'<!doctype html>\n\n<title>500 Internal Server Error</title>\n
Internal Server Error
\n
The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.
\n'
bash-commons-bats_ubuntu2004_imds_v2-1  | not ok 60 aws_describe_asg empty
bash-commons-bats_ubuntu2004_imds_v2-1  | # (from function `flunk' in file test/test-helper.bash, line 10,
bash-commons-bats_ubuntu2004_imds_v2-1  | #  from function `assert_success' in file test/test-helper.bash, line 21,
bash-commons-bats_ubuntu2004_imds_v2-1  | #  in test file test/aws-cli.bats, line 73)
bash-commons-bats_ubuntu2004_imds_v2-1  | #   `assert_success' failed```

Oh... maybe this...
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 67 configure_imdsv2_ttl_with_maximum_ttl
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 68 configure_imdsv2_ttl_with_excessively_high_ttl
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 69 aws_get_instance_private_ip
bash-commons-bats_ubuntu2004_imds_v2-1  | IMDS endpoint connection refused.
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 70 aws_get_instance_public_ip
bash-commons-bats_ubuntu2004_imds_v2-1  | IMDS endpoint connection refused.
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 71 aws_get_instance_private_hostname
bash-commons-bats_ubuntu2004_imds_v2-1  | IMDS endpoint connection refused.

This happens as a result of curl returning this error code:
curl: (7) Failed connect to <IP_ADDRESS>:80; Connection refused
In my environment, this means the user does not have permissions to access the endpoint.  But it works with Root.

So, I'm not sure how related these two things are, but under the master branch the docker-compose exits with an error code of 1 just the same as my branch.
bash-commons-bats_ubuntu2004_imds_v2-1 exited with code 1
Aborting on container exit...
[+] Running 1/0
 ⠿ Container bash-commons-bats_ubuntu2004_imds_v2-1  Stopped

When I run it on my branch, it fails there as well.  It is clear that the IMDSv2 checks are failing though.  I'm not sure that is related at all to the error code 1.
From check 69-108:
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 69 aws_get_instance_private_ip
bash-commons-bats_ubuntu2004_imds_v2-1  | Check for IMDSv2 failed. IMDS endpoint connection refused.
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 70 aws_get_instance_public_ip
bash-commons-bats_ubuntu2004_imds_v2-1  | Check for IMDSv2 failed. IMDS endpoint connection refused.
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 71 aws_get_instance_private_hostname
bash-commons-bats_ubuntu2004_imds_v2-1  | Check for IMDSv2 failed. IMDS endpoint connection refused.
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 72 aws_get_instance_public_hostname
bash-commons-bats_ubuntu2004_imds_v2-1  | Check for IMDSv2 failed. IMDS endpoint connection refused.
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 73 aws_get_instance_id
bash-commons-bats_ubuntu2004_imds_v2-1  | Check for IMDSv2 failed. IMDS endpoint connection refused.
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 74 aws_get_instance_region
bash-commons-bats_ubuntu2004_imds_v2-1  | Check for IMDSv2 failed. IMDS endpoint connection refused.
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 75 aws_get_ec2_instance_availability_zone
bash-commons-bats_ubuntu2004_imds_v2-1  | Check for IMDSv2 failed. IMDS endpoint connection refused.
bash-commons-bats_ubuntu2004_imds_v2-1  | Check for IMDSv2 failed. IMDS endpoint connection refused.
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 76 aws_wrapper_get_asg_name ASG tag only
bash-commons-bats_ubuntu2004_imds_v2-1  | Check for IMDSv2 failed. IMDS endpoint connection refused.
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 77 aws_wrapper_get_asg_name ASG tag and other tags
bash-commons-bats_ubuntu2004_imds_v2-1  | Check for IMDSv2 failed. IMDS endpoint connection refused.
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 78 aws_wrapper_get_asg_name no tags
bash-commons-bats_ubuntu2004_imds_v2-1  | Check for IMDSv2 failed. IMDS endpoint connection refused.
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 79 aws_wrapper_get_asg_name no ASG tags
bash-commons-bats_ubuntu2004_imds_v2-1  | Check for IMDSv2 failed. IMDS endpoint connection refused.
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 80 aws_wrapper_get_instance_tag no tags
bash-commons-bats_ubuntu2004_imds_v2-1  | Check for IMDSv2 failed. IMDS endpoint connection refused.
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 81 aws_wrapper_get_instance_tag no matching tag
bash-commons-bats_ubuntu2004_imds_v2-1  | Check for IMDSv2 failed. IMDS endpoint connection refused.
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 82 aws_wrapper_get_instance_tag matching tag
bash-commons-bats_ubuntu2004_imds_v2-1  | Check for IMDSv2 failed. IMDS endpoint connection refused.
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 83 aws_wrapper_wait_for_instance_tags one tag
bash-commons-bats_ubuntu2004_imds_v2-1  | Check for IMDSv2 failed. IMDS endpoint connection refused.
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 84 aws_wrapper_wait_for_instance_tags multiple tags
bash-commons-bats_ubuntu2004_imds_v2-1  | Check for IMDSv2 failed. IMDS endpoint connection refused.
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 85 aws_wrapper_wait_for_instance_tags no tags
bash-commons-bats_ubuntu2004_imds_v2-1  | Check for IMDSv2 failed. IMDS endpoint connection refused.
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 86 aws_wrapper_get_asg_size ASG exists
bash-commons-bats_ubuntu2004_imds_v2-1  | Check for IMDSv2 failed. IMDS endpoint connection refused.
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 87 aws_wrapper_get_asg_size ASG does not exist
bash-commons-bats_ubuntu2004_imds_v2-1  | Check for IMDSv2 failed. IMDS endpoint connection refused.
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 88 aws_wrapper_wait_for_instances_in_asg ASG size 1
bash-commons-bats_ubuntu2004_imds_v2-1  | Check for IMDSv2 failed. IMDS endpoint connection refused.
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 89 aws_wrapper_wait_for_instances_in_asg ASG size 3
bash-commons-bats_ubuntu2004_imds_v2-1  | Check for IMDSv2 failed. IMDS endpoint connection refused.
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 90 aws_wrapper_wait_for_instances_in_asg ASG size 2, 3 available
bash-commons-bats_ubuntu2004_imds_v2-1  | Check for IMDSv2 failed. IMDS endpoint connection refused.
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 91 aws_wrapper_wait_for_instances_in_asg ASG size 3, only 2 available
bash-commons-bats_ubuntu2004_imds_v2-1  | Check for IMDSv2 failed. IMDS endpoint connection refused.
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 92 aws_wrapper_wait_for_instances_in_asg ASG size not available
bash-commons-bats_ubuntu2004_imds_v2-1  | Check for IMDSv2 failed. IMDS endpoint connection refused.
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 93 aws_wrapper_get_ips_in_asg empty ASG
bash-commons-bats_ubuntu2004_imds_v2-1  | Check for IMDSv2 failed. IMDS endpoint connection refused.
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 94 aws_wrapper_get_ips_in_asg ASG size 1, public IPs
bash-commons-bats_ubuntu2004_imds_v2-1  | Check for IMDSv2 failed. IMDS endpoint connection refused.
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 95 aws_wrapper_get_ips_in_asg ASG size 3, public IPs
bash-commons-bats_ubuntu2004_imds_v2-1  | Check for IMDSv2 failed. IMDS endpoint connection refused.
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 96 aws_wrapper_get_ips_in_asg ASG size 3, private IPs
bash-commons-bats_ubuntu2004_imds_v2-1  | Check for IMDSv2 failed. IMDS endpoint connection refused.
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 97 aws_wrapper_get_hostnames_in_asg empty ASG
bash-commons-bats_ubuntu2004_imds_v2-1  | Check for IMDSv2 failed. IMDS endpoint connection refused.
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 98 aws_wrapper_get_hostnames_in_asg ASG size 1, public IPs
bash-commons-bats_ubuntu2004_imds_v2-1  | Check for IMDSv2 failed. IMDS endpoint connection refused.
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 99 aws_wrapper_get_hostnames_in_asg ASG size 3, public IPs
bash-commons-bats_ubuntu2004_imds_v2-1  | Check for IMDSv2 failed. IMDS endpoint connection refused.
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 100 aws_wrapper_get_hostnames_in_asg ASG size 3, private IPs
bash-commons-bats_ubuntu2004_imds_v2-1  | Check for IMDSv2 failed. IMDS endpoint connection refused.
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 101 aws_wrapper_get_hostname public
bash-commons-bats_ubuntu2004_imds_v2-1  | Check for IMDSv2 failed. IMDS endpoint connection refused.
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 102 aws_wrapper_get_hostname private
bash-commons-bats_ubuntu2004_imds_v2-1  | Check for IMDSv2 failed. IMDS endpoint connection refused.
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 103 aws_wrapper_get_asg_rally_point by instance id, private
bash-commons-bats_ubuntu2004_imds_v2-1  | Check for IMDSv2 failed. IMDS endpoint connection refused.
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 104 aws_wrapper_get_asg_rally_point by instance id, public
bash-commons-bats_ubuntu2004_imds_v2-1  | Check for IMDSv2 failed. IMDS endpoint connection refused.
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 105 aws_wrapper_get_asg_rally_point by launch time, private
bash-commons-bats_ubuntu2004_imds_v2-1  | Check for IMDSv2 failed. IMDS endpoint connection refused.
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 106 aws_wrapper_get_asg_rally_point by launch time, public
bash-commons-bats_ubuntu2004_imds_v2-1  | Check for IMDSv2 failed. IMDS endpoint connection refused.
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 107 aws_wrapper_get_asg_rally_point empty asg, private
bash-commons-bats_ubuntu2004_imds_v2-1  | Check for IMDSv2 failed. IMDS endpoint connection refused.
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 108 aws_wrapper_get_asg_rally_point empty asg, public

Which appear to all be related to the aws.sh if statement.  I might try adding in a function and specifically calling that at the right point in the script.
It's weird that the checks show as "ok" though.  Like check # 74, ok 74 aws_get_instance_region which I assume is pulling from IMDS.

It appears to no different than the master branch exit error code.

Interesting, it looks like the test itself may be buggy. Confirmed that some tests are failing on main as well. Let me take a closer look and see if we can identify why the tests are failing unrelated to your change.

IMDSv1 check result is =  0
bash-commons-bats_ubuntu2004_imds_v2-1  | IMDSv2 check result is =  0
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 66 configure_imdsv2_ttl_with_minimal_ttl
bash-commons-bats_ubuntu2004_imds_v2-1  | IMDSv1 check result is =  0
bash-commons-bats_ubuntu2004_imds_v2-1  | IMDSv2 check result is =  0
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 67 configure_imdsv2_ttl_with_maximum_ttl
bash-commons-bats_ubuntu2004_imds_v2-1  | IMDSv1 check result is =  0
bash-commons-bats_ubuntu2004_imds_v2-1  | IMDSv2 check result is =  0
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 68 configure_imdsv2_ttl_with_excessively_high_ttl
bash-commons-bats_ubuntu2004_imds_v2-1  | IMDSv1 check result is =  0
bash-commons-bats_ubuntu2004_imds_v2-1  | IMDSv2 check result is =  0
bash-commons-bats_ubuntu2004_imds_v2-1  | IMDSv1 check result is =  0
bash-commons-bats_ubuntu2004_imds_v2-1  | IMDSv2 check result is =  0
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 69 aws_get_instance_private_ip
bash-commons-bats_ubuntu2004_imds_v2-1  | IMDSv1 check result is =  7
bash-commons-bats_ubuntu2004_imds_v2-1  | IMDSv2 check result is =  7
bash-commons-bats_ubuntu2004_imds_v2-1  | Check for IMDSv2 failed. IMDS endpoint connection refused.
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 70 aws_get_instance_public_ip
bash-commons-bats_ubuntu2004_imds_v2-1  | IMDSv1 check result is =  7
bash-commons-bats_ubuntu2004_imds_v2-1  | IMDSv2 check result is =  7
bash-commons-bats_ubuntu2004_imds_v2-1  | Check for IMDSv2 failed. IMDS endpoint connection refused.
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 71 aws_get_instance_private_hostname
bash-commons-bats_ubuntu2004_imds_v2-1  | IMDSv1 check result is =  7
bash-commons-bats_ubuntu2004_imds_v2-1  | IMDSv2 check result is =  7
bash-commons-bats_ubuntu2004_imds_v2-1  | Check for IMDSv2 failed. IMDS endpoint connection refused.
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 72 aws_get_instance_public_hostname

Looks like some test pass and show both versions passing, but then others show failed responses ("7")

Maybe the aws-wrapper.sh

After a bit of digging I think I see the issue. It looks like the older cfn-lint and pyrsistent pinned versions are causing an issue. Removing the pins against the master branch allows tests to run successfully:
-RUN pip3 install flask moto moto[server] cfn-lint==0.35.1 pyrsistent==0.16.0
+RUN pip3 install flask moto moto[server] cfn-lint pyrsistent

If you want to make this change in your branch, you should be able to test successfully again. Go ahead and make this change to unblock this PR, then I'll open an issue to track this and work towards a more appropriate fix.

I removed those version numbers from the two Docker files (18.04 and 20.04).  Didn't see that in any other Dockerfiles.
I think the problem may be in the mock metadata server.  I don't think it is built to handle IMDSv2 calls (i.e. PUT requests).  It's located in the test/aws-helps.bash file.  In a function start_ec2_metadata_mock.
I suspect these functions are not working either.

create_mock_instance_with_tags
create_mock_asg

The mock ASG is using Launch Configurations (which are depreciated now I belive) and needs to switch to Launch Templates.  This is on line 141
aws autoscaling create-launch-configuration --launch-configuration-name "$asg_name" --image-id ami-fake --instance-type t3.micro

So, I wanted to see the http response and the curl error code.  I put these checks into the modules/bash-commons/src/aws.sh file:
imdsv1_check=$(curl -s -o /dev/null ${metadata_endpoint}; echo $?)
echo "IMDSv1 curl check result is = " ${imdsv1_check}
imdsv2_check=$(curl -s -o /dev/null -X PUT ${imdsv2_token_endpoint} -H "X-aws-ec2-metadata-token-ttl-seconds: 10"; echo $?)
echo "IMDSv2 curl check result is = " ${imdsv2_check}

imdsv1_http_check=$(curl -s -o /dev/null -w "%{http_code}\n" ${metadata_endpoint})
echo "IMDSv1 http check result is = " ${imdsv1_http_check}
imdsv2_http_check=$(curl -s -o /dev/null -w "%{http_code}\n" -X PUT ${imdsv2_token_endpoint} -H "X-aws-ec2-metadata-token-ttl-seconds: 10")
echo "IMDSv2 http check result is = " ${imdsv2_http_check}

on check 57 I get:
bash-commons-bats_ubuntu2004_imds_v2-1  | IMDSv1 curl check result is =  0
bash-commons-bats_ubuntu2004_imds_v2-1  | IMDSv2 curl check result is =  0
bash-commons-bats_ubuntu2004_imds_v2-1  | IMDSv1 http check result is =  404
bash-commons-bats_ubuntu2004_imds_v2-1  | IMDSv2 http check result is =  404
bash-commons-bats_ubuntu2004_imds_v2-1  | IMDSv1 curl check result is =  0
bash-commons-bats_ubuntu2004_imds_v2-1  | IMDSv2 curl check result is =  0
bash-commons-bats_ubuntu2004_imds_v2-1  | IMDSv1 http check result is =  404
bash-commons-bats_ubuntu2004_imds_v2-1  | IMDSv2 http check result is =  404
bash-commons-bats_ubuntu2004_imds_v2-1  | not ok 57 aws_get_instance_tags empty

On 66, I get:
bash-commons-bats_ubuntu2004_imds_v2-1  | IMDSv1 curl check result is =  0
bash-commons-bats_ubuntu2004_imds_v2-1  | IMDSv2 curl check result is =  0
bash-commons-bats_ubuntu2004_imds_v2-1  | IMDSv1 http check result is =  429
bash-commons-bats_ubuntu2004_imds_v2-1  | IMDSv2 http check result is =  429
bash-commons-bats_ubuntu2004_imds_v2-1  | ok 66 configure_imdsv2_ttl_with_minimal_ttl

For check 76:
bash-commons-bats_ubuntu2004_imds_v2-1  | IMDSv1 curl check result is =  7
bash-commons-bats_ubuntu2004_imds_v2-1  | IMDSv2 curl check result is =  7
bash-commons-bats_ubuntu2004_imds_v2-1  | IMDSv1 curl check result is =  7
bash-commons-bats_ubuntu2004_imds_v2-1  | IMDSv2 curl check result is =  7
bash-commons-bats_ubuntu2004_imds_v2-1  | IMDSv1 curl check result is =  7
bash-commons-bats_ubuntu2004_imds_v2-1  | IMDSv2 curl check result is =  7
bash-commons-bats_ubuntu2004_imds_v2-1  | IMDSv1 curl check result is =  7
bash-commons-bats_ubuntu2004_imds_v2-1  | IMDSv2 curl check result is =  7
bash-commons-bats_ubuntu2004_imds_v2-1  | IMDSv1 curl check result is =  7
bash-commons-bats_ubuntu2004_imds_v2-1  | IMDSv2 curl check result is =  7
bash-commons-bats_ubuntu2004_imds_v2-1  | IMDSv1 curl check result is =  7
bash-commons-bats_ubuntu2004_imds_v2-1  | IMDSv2 curl check result is =  7
bash-commons-bats_ubuntu2004_imds_v2-1  | IMDSv1 curl check result is =  7
bash-commons-bats_ubuntu2004_imds_v2-1  | IMDSv2 curl check result is =  7
bash-commons-bats_ubuntu2004_imds_v2-1  | not ok 76 setup_file failed
bash-commons-bats_ubuntu2004_imds_v2-1  | # (from function `source' in file modules/bash-commons/src/aws.sh, line 33,
bash-commons-bats_ubuntu2004_imds_v2-1  | #  from function `source' in file test/../modules/bash-commons/src/aws-wrapper.sh, line 9,
bash-commons-bats_ubuntu2004_imds_v2-1  | #  from function `source' in file test/aws-wrapper.bats, line 3,
bash-commons-bats_ubuntu2004_imds_v2-1  | #  from function `bats_run_setup_file' in file /usr/local/libexec/bats-core/bats-exec-file, line 97,
bash-commons-bats_ubuntu2004_imds_v2-1  | #  from function `main' in test file /usr/local/libexec/bats-core/bats-exec-file, line 379)
bash-commons-bats_ubuntu2004_imds_v2-1  | #   `bats_run_setup_file' failed with status 7

And then it skips to 109 which gets an ok along with all the rest.

@justin-octo thank you for all of the additional research on this. I agree with your findings and this appears to be due to how the test framework is designed (not actually exposing a real endpoint). I've just completed some testing against a live EC2 instance and confirmed your updated script is working as expected with no errors.
To get around the test limitations and to ensure users can still override the default behavior, I believe the best course of action is to wrap your new logic in a simple if block that only executes if the GRUNTWORK_BASH_COMMONS_IMDS_VERSION env var is not set. Example:
if [[ -z "$GRUNTWORK_BASH_COMMONS_IMDS_VERSION" ]]; then
    to_token_or_not_to_token=$(sudo curl -s -o /dev/null -X PUT ${imdsv2_token_endpoint} -H "X-aws-ec2-metadata-token-ttl-seconds: 10"; echo $?)
    if [ ${to_token_or_not_to_token} -eq 0 ]; then
      default_instance_metadata_version="2"
    elif [ ${to_token_or_not_to_token} -eq 7 ]; then
      echo "Check for IMDSv2 failed. IMDS endpoint connection refused."
      default_instance_metadata_version="0"
    else
      finish_code=$(sudo curl -s -o /dev/null $metadata_endpoint; echo $?)
      if [ ${finish_code} -eq 0 ]; then
        default_instance_metadata_version="1"
      elif [ ${finish_code} -eq 7 ]; then
        echo "Check for IMDSv1 and v2 failed. IMDS endpoint connection refused."
        default_instance_metadata_version="0"
      else
        echo "IMDS endpoint connection failed for an unknown reason with error code: ${finish_code}"
        default_instance_metadata_version="0"
      fi
    fi
fi

This will prevent the curl script from running within the docker tests (since we specifically set GRUNTWORK_BASH_COMMONS_IMDS_VERSION based on the test case) and will ensure users can properly override the value as-needed. If the value is not set, your logic will run and execute properly.

I think that's a smart solution.  I updated the branch with the change.

Fantastic, this is looking great! I'm going to re-run the tests now.

I can't see the details of what failed, but it looks like the bats_v2 all passed.  Just the v1 failed.  Which might just be due to the tests being outdated?

Everything passed except the Ubuntu 16 and 18 tests, which need to be removed. I'm going to approve + merge this shortly.

This has been released in v0.2.0.


Temporary fix for youtube embeddables

see https://github.com/oscarotero/Embed/issues/417

fyi; tests pass for 7.4, fail for 8.0

stale


Improve eslint UX

Can't eslint just autofix issues with --fix? It's quite jarring otherwise.

https://github.com/MoOx/eslint-loader#fix-default-false


Disable selection

Hello,
Is it possible to disable user click on date ? For instance with setSelectionMode(eventsCalendar.NO_SELECTION)
It would be very great
Thanks

Not possible right now!


Issue with pulling docker/singularity image

Running into an issue when processing the test data set when pulling a singularity image with a singularity profile and an additional slurm profile:
`ERROR ~ Error executing process > 'SCATACPIPE:INPUT_CHECK_FASTQ:SAMPLESHEET_CHECK_FASTQ (1)'
Caused by:
Failed to pull singularity image
command: singularity pull  --name hukai916-miniconda3_bio-0.1.img.pulling.1729537270774 docker://hukai916/miniconda3_bio:0.1 > /dev/null
status : 255
hint   : Try and increase singularity.pullTimeout in the config (current is "10h")
message:
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
INFO:    Fetching OCI image...
INFO:    Extracting OCI image...
FATAL:   While making image from oci registry: error fetching image to cache: while building SIF from layers: packer failed to pack: while unpacking tmpfs: error unpacking rootfs: unpack entry: opt/conda/pkgs/certifi-2021.5.30-py38h578d9bd_0/lib/python3.8/site-packages/certifi-2021.5.30-py3.8.egg-info/PKG-INFO: link: unpriv.link: unpriv.wrap target: no such file or directory
-- Check '.nextflow.log' file for details
`
Also fails when pulling image manually including with docker.

Hi @plim10,
What command did you use when you manually pull the image, and what error? Ty,

Hi Hukai,
I first tried to directly manually pull the image in the HPC with:
singularity pull docker://hukai916/miniconda3_bio:0.1
And then I tried using docker locally and building in the HPC with 'singularity build':
docker pull hukai916/miniconda3_bio:0.1
docker save hukai916/miniconda3_bio:0.1 -o miniconda3_bio.tar

Then copied onto the cluster and tried:
singularity build miniconda3_bio.sif docker-archive://miniconda3_bio.tar
Both methods failed to load the image and give me this error when extracting the OCI image.
INFO:    Starting build...
INFO:    Fetching OCI image...
INFO:    Extracting OCI image...
FATAL:   While performing build: packer failed to pack: while unpacking tmpfs: error unpacking rootfs: unpack entry: opt/conda/pkgs/certifi-2021.5.30-py38h578d9bd_0/lib/python3.8/site-packages/certifi-2021.5.30-py3.8.egg-info/PKG-INFO: link: unpriv.link: unpriv.wrap target: no such file or directory

Thank you so much for your quick reply!

Singularity can pull directly from Docker images hosted on DockerHub, so, you typically don't need to pull docker into tar and build singularity using the tar. What error did you encounter when running singularity pull docker://hukai916/miniconda3_bio:0.1 on HPC?

On HPC I get this error:
INFO:    Converting OCI blobs to SIF format INFO:    Starting build... INFO:    Fetching OCI image... 3.5MiB / 3.5MiB [============================================================] 100 % 0.0 b/s 0s 38.6MiB / 38.6MiB [==========================================================] 100 % 0.0 b/s 0s 1.1MiB / 1.1MiB [============================================================] 100 % 0.0 b/s 0s 50.5MiB / 50.5MiB [==========================================================] 100 % 0.0 b/s 0s 44.3MiB / 44.3MiB [==========================================================] 100 % 0.0 b/s 0s 10.0MiB / 10.0MiB [==========================================================] 100 % 0.0 b/s 0s 192.7MiB / 192.7MiB [========================================================] 100 % 0.0 b/s 0s 43.7MiB / 43.7MiB [==========================================================] 100 % 0.0 b/s 0s 76.0MiB / 76.0MiB [==========================================================] 100 % 0.0 b/s 0s 21.6MiB / 21.6MiB [==========================================================] 100 % 0.0 b/s 0s INFO:    Extracting OCI image... FATAL:   While making image from oci registry: error fetching image to cache: while building SIF from layers: packer failed to pack: while unpacking tmpfs: error unpacking rootfs: unpack entry: opt/conda/pkgs/certifi-2021.5.30-py38h578d9bd_0/lib/python3.8/site-packages/certifi-2021.5.30-py3.8.egg-info/PKG-INFO: link: unpriv.link: unpriv.wrap target: no such file or directory
On my local computer the docker pull works:
0.1: Pulling from hukai916/miniconda3_bio Digest: sha256:e89fbba537fb2d84da4f1dd661b9b85bebf1980c0f7ea1bfb808876b93783b0e Status: Image is up to date for hukai916/miniconda3_bio:0.1 docker.io/hukai916/miniconda3_bio:0.1

I found a similar discussion here: https://github.com/sylabs/singularity/discussions/2675
Can you try singularity pull docker://quay.io/benlangmead/recount-rs5:1.0.6 and see if you can replicate the same error? If so, it is likely an issue in Singularity. What is your current Singularity version? It may help to ask your HPC admin to install the latest Singularity.

Trying singularity pull docker://quay.io/benlangmead/recount-rs5:1.0.6 succeeds in creating the sif file so it looks like it's not an issue with singularity.
INFO:    Converting OCI blobs to SIF format INFO:    Starting build... INFO:    Fetching OCI image... 13.1MiB / 13.1MiB [=======================================================] 100 % 37.0 MiB/s 0s 1.5MiB / 1.5MiB [=========================================================] 100 % 37.0 MiB/s 0s 42.3MiB / 42.3MiB [=======================================================] 100 % 37.0 MiB/s 0s 33.4MiB / 33.4MiB [=======================================================] 100 % 37.0 MiB/s 0s 2.2MiB / 2.2MiB [=========================================================] 100 % 37.0 MiB/s 0s 192.7MiB / 192.7MiB [=====================================================] 100 % 37.0 MiB/s 0s 98.1MiB / 98.1MiB [=======================================================] 100 % 37.0 MiB/s 0s 33.4MiB / 33.4MiB [=======================================================] 100 % 37.0 MiB/s 0s 11.0MiB / 11.0MiB [=======================================================] 100 % 37.0 MiB/s 0s 89.0MiB / 89.0MiB [=======================================================] 100 % 37.0 MiB/s 0s 152.2MiB / 152.2MiB [=====================================================] 100 % 37.0 MiB/s 0s 19.7MiB / 19.7MiB [=======================================================] 100 % 37.0 MiB/s 0s 70.3MiB / 70.3MiB [=======================================================] 100 % 37.0 MiB/s 0s 130.9KiB / 130.9KiB [=====================================================] 100 % 37.0 MiB/s 0s 1.6MiB / 1.6MiB [=========================================================] 100 % 37.0 MiB/s 0s INFO:    Extracting OCI image... 2024/10/22 11:27:03  warn rootless{usr/local/man} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers" INFO:    Inserting Singularity configuration... INFO:    Creating SIF file...
We are using singularity version 4.1.1

The last resort I can think of is to try building the images with a newer base: when we developed scATACpipe, the image base was Ubuntu:16.04, which might be too low for some systems. I have just built an image with a newer base, can you try
singularity pull docker://hukai916/miniconda3_biox:0.1

If the above works on your HPC, I can try to rebuild all images, which may take me a few days. Lmk,

Hi, yes this works! It would be great if the images were rebuilt then with a newer base. Thank you!

I have just re-built images with newer bases. To apply the changes, you can try:
cd scATACpipe
git checkout dev
git pull

Hope this can solve the problem. Pls report any glitches.

Thanks for the image rebuild.
Just tried it again after the git pull and unfortunately it's now giving me this similar error:
ERROR ~ Error executing process > 'SCATACPIPE:INPUT_CHECK_FASTQ:SAMPLESHEET_CHECK_FASTQ (1)'
Caused by:
Failed to pull singularity image
command: singularity pull  --name hukai916-miniconda3_v24.7_amd64_bio-0.1.img.pulling.1729788608773 docker://hukai916/miniconda3_v24.7_amd64_bio:0.1 > /dev/null
status : 255
hint   : Try and increase singularity.pullTimeout in the config (current is "10h")
message:
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
INFO:    Fetching OCI image...
INFO:    Extracting OCI image...
FATAL:   While making image from oci registry: error fetching image to cache: while building SIF from layers: packer failed to pack: while unpacking tmpfs: error unpacking rootfs: unpack entry: opt/conda/pkgs/regex-2024.9.11-py312h5eee18b_0/lib/python3.12/site-packages/regex-2024.9.11.dist-info/INSTALLER: link: unpriv.link: unpriv.wrap target: no such file or directory
-- Check '.nextflow.log' file for details

This is bizarre, try to manually pull the image from your HPC:
singularity pull docker://hukai916/miniconda3_v24.7_amd64_bio:0.1

If the above pops out the same error, I suggest to consult with your HPC admin for further debugging.

I'm getting the same error when I try to pull it manually. Perhaps there are some permissions that require admin privileges. Thanks for your help anyhow!

I'm getting the same error when I try to pull it manually. It also gives me a new warning while extracting the OCI image:
INFO:    Extracting OCI image...
2024/10/25 14:40:46  warn xattr{var/cache/apt/archives/partial} ignoring ENOTSUP on setxattr "user.root          lesscontainers"
2024/10/25 14:40:46  warn xattr{/lab/tambora_li/Patrick/singularity_cache/tmp/build-temp-2346428033/roo          tfs/var/cache/apt/archives/partial} destination filesystem does not support xattrs, further warnings wi          ll be suppressed
FATAL:   While making image from oci registry: error fetching image to cache: while building SIF from layers: packer failed to pack: while unpacking tmpfs: error unpacking rootfs: unpack entry: opt/conda/pkgs/regex-2024.9.11-py312h5eee18b_0/lib/python3.12/site-packages/regex-2024.9.11.dist-info/INSTALLER: link: unpriv.link: unpriv.wrap target: no such file or directory

Perhaps there are some permissions that require admin privileges. Thanks for your help anyhow!

I ran singularity pull docker://hukai916/miniconda3_v24.7_amd64_bio:0.1 on my HPC and it worked!


debug mode error

i build this plugin template in debug mode and got this error

but nothing error in release mode , i don't know why.

You also need to build x64dbg in debug mode for this to work.
On Mon, 14 Dec 2020 at 03:20, Truman<EMAIL_ADDRESS>wrote:

i build this plugin template in debug mode and got this error
[image: image]
https://user-images.githubusercontent.com/22972430/102033218-cb2df280-3df5-11eb-9f12-b0efd7ebba4c.png
but nothing error in release mode , i don't know why.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/x64dbg/QtPlugin/issues/1, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AASYFGJ6IFGPNXKC676MEMDSUVY6VANCNFSM4U2ADKCA
.


Firecapture 2.7.14a PKGBUILD

Updated the PKGBUILD to pull from the hosted .deb package. Added symlinks and jre divert.
Future releases should only need replacement of the .deb, and update of the $pkgver and the .debs sha512sum in the PKGBUILD.

Lowered the priority of the FireCapture udev rules so they are ignored if already present at a higher priority, which they are for the default installed rules, ie. they will be ignored unless the user has uninstalled KStars.
Tweaked the FireCapture.ini file to reference /home/astronaut instead of /home/pi

gonna pack it up in few minutes @LuckyEddie47

Great, I believe you can close Issue #4 on the back of that.

cgobat
I deliberately didn't go that route as I felt that working around a deliberately constructed security block is unethical. Having originally hosted the aarch64.deb myself, I experienced first hand the shear volume of bot downloads (mostly from Russia). It was enough that I had to pull the download as the traffic was going to cost me a significant amount on top of my standard hosting. I therefore completely understand and sympathise with Torsten's decision to add a challenge/response on the firecapture.de downloads. Please remember that, although not open source, FireCapture is free software provided by an individual as an altruistic undertaking - Torsten bares all costs (£$€ and his time) and retains ownership and all rights.
Please, if possible, remove your PKGBUILD from public view. I really don't want to tar Torsten's view of AstroArch or myself.

Okay—"unethical" seems like a bit of a stretch to me, but I've removed it.

cgobat - Thank-you. I did not mean to imply that your creating that solution was an unethical action, rather it was an explanation of my feelings that guided my approach. Peace, and clear skies.


RFC: Inform the user if the bootloader EEPROM is out of date

It's normally desirable to use the latest bootloader when testing the
latest firmware releases. rpi-eeprom-update is specific to Raspbian but
so rpi-update can't easily do the EEPROM update itself without
duplicating all the functionality. Instead, just indicate that the
bootloader is out of date unless FREEZE_VERSION has been specified.
N.B. The MIN_VERSION is set to the Sep-10 release which will be promoted
to stable today.

Updated to use "command -v" to check for vcgencmd  before calling it.
Added a check to verify that the bootloader_config command is supported, if not, the firmware is too old

Don’t mind either way but parsing cpuinfo appears to be more complex. The OTP is also the source of truth for the board revision

But OTP is an implementation detail we don't publish. /proc/cpuinfo is the documented way of identifying processor type.

Processor type or board revision ? Both are potentially significant when selecting bootloader compatibility. Happy to use cpuinfo but the reason I used otp_dump is that the output is less likely to change than cpuinfo

cpuinfo is the right place to look. It will also work with force_board_rev.
/proc/cpuinfo is unlikely to change format, as we use this for hello_pi demos, omxplayer, kodi, vlc, ffmpeg etc.

ok, force_board_rev is a good reason to use cpuinfo


Fix typos

This PR fixes some typos: tf.constant, tf.placeholder.

Can one of the admins verify this patch?

I signed it!


Scrolling is buggy

Scrolling is buggy in this lib. Please have a look at it.
Sometimes when I Scroll over widget list the scrolling doesn't work. I have to move my hands on it 2 or 3 times to scroll over it.

Noted, I'll check it out

Caught it!

Caught it!

I guess you figured it out , i am waiting for the update . Thanks

Ohh just saw the update, Thanks Budy

Thanks for reporting!


Compiler crash redefining local variable in Proc

This code gives BUG: trying to assign String <- Int32:
https://carc.in/#/r/2pdc
macro mac(arg)
  ->{local = 1; {{ arg }} }
end

arg = nil
mac(arg)
local = "hello"

->{local}

This is not preventing me from doing anything practical, but carc.in says to report it.

Reduced:
arg = nil
->{
  local = 1
  arg
}
local = "hello"

->{local}


folly: 2<IP_ADDRESS> -> 2<IP_ADDRESS>

Semi-automatic update. These checks were done:

built on NixOS
Warning: no binary found that responded to help or version flags. (This warning appears even if the package isn't expected to have binaries.)
found 2<IP_ADDRESS> with grep in /nix/store/a1f6l0wcn1cw5v7ycmpdd21kmka3bxpg-folly-2<IP_ADDRESS>
found 2<IP_ADDRESS> in filename of file in /nix/store/a1f6l0wcn1cw5v7ycmpdd21kmka3bxpg-folly-2<IP_ADDRESS>

cc @abbradar

@GrahamcOfBorg eval

Failure on x86_64-darwin (full log)
Partial log (click to expand)
Package ‘folly-2<IP_ADDRESS>’ in /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/pkgs/development/libraries/folly/default.nix:28 is not supported on ‘x86_64-darwin’, refusing to evaluate.

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowBroken = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowBroken = true; }
to ~/.config/nixpkgs/config.nix.

Failure on aarch64-linux (full log)
Partial log (click to expand)
Package ‘folly-2<IP_ADDRESS>’ in /var/lib/gc-of-borg/nix-test-rs-23/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-23/pkgs/development/libraries/folly/default.nix:28 is not supported on ‘aarch64-linux’, refusing to evaluate.

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowBroken = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowBroken = true; }
to ~/.config/nixpkgs/config.nix.

Success on x86_64-linux (full log)
Partial log (click to expand)
shrinking RPATHs of ELF executables and libraries in /nix/store/a1f6l0wcn1cw5v7ycmpdd21kmka3bxpg-folly-2<IP_ADDRESS>
shrinking /nix/store/a1f6l0wcn1cw5v7ycmpdd21kmka3bxpg-folly-2<IP_ADDRESS>/lib/libfolly.so.57.0.0
shrinking /nix/store/a1f6l0wcn1cw5v7ycmpdd21kmka3bxpg-folly-2<IP_ADDRESS>/lib/libfollybenchmark.so.57.0.0
shrinking /nix/store/a1f6l0wcn1cw5v7ycmpdd21kmka3bxpg-folly-2<IP_ADDRESS>/lib/libfollyinit.so.57.0.0
shrinking /nix/store/a1f6l0wcn1cw5v7ycmpdd21kmka3bxpg-folly-2<IP_ADDRESS>/lib/libfollylogging.so.57.0.0
strip is /nix/store/b0zlxla7dmy1iwc3g459rjznx59797xy-binutils-2.28.1/bin/strip
stripping (with command strip and flags -S) in /nix/store/a1f6l0wcn1cw5v7ycmpdd21kmka3bxpg-folly-2<IP_ADDRESS>/lib
patching script interpreter paths in /nix/store/a1f6l0wcn1cw5v7ycmpdd21kmka3bxpg-folly-2<IP_ADDRESS>
checking for references to /tmp/nix-build-folly-2<IP_ADDRESS>.drv-0 in /nix/store/a1f6l0wcn1cw5v7ycmpdd21kmka3bxpg-folly-2<IP_ADDRESS>...
/nix/store/a1f6l0wcn1cw5v7ycmpdd21kmka3bxpg-folly-2<IP_ADDRESS>

@ryantm Thanks!


update examples

The examples no longer use CocoaPods Rome and have no external build dependencies. They can be build using Swift Package Manager via make examples from the root and then ran as ./Examples/hello etc.

ohh i see.


Question: Source maps and debugging support from JDK?

Hi!
Is debugging and source map features only available when using node or is there some way to do any of this using JDK?
Source maps would be useful to us because we're transpiling and bundling from typescript, and of course when an error occurs, the referenced line is currently the bundled one instead of the inlined source map reference.
Thanks!

This question on stackoverflow.com may cover what you are looking for.

Thanks. It seemed promising, but I'm getting Could not find option with name inspect..
final Context ctx =
        Context.newBuilder("js")
            .allowAllAccess(true)
            .option("js.strict", "true")
            .option("inspect", "localhost:4444")
            .build();

OK. I got the debugger working, great! Thanks.
However when an error is thrown from within the script do not add any hinting towards where in the typescript this relates to.
Would there be a way to enhance this?

@pintomau I had this question a while ago. I've ended up by reformating the exception textual content on my logger, replacing the character indexes by, the line and column indexes so later I can plugin the source maps or like you did, just rely on the debugger:
https://github.com/reactiverse/es4x/blob/develop/es4x/src/main/java/io/reactiverse/es4x/jul/ANSIFormatter.java#L181-L275

Thanks for the hints, @pmlopes . Will look into it.
Also noticed VertxFileSystem that seems to allow loading files from within the JAR, as claimed in https://github.com/reactiverse/es4x/blob/98bc57a0210940946652cf15129300a5f9f48322/docs/advanced/commonjs.md#specific-differences. Interesting, as we eventually would also like to be able to load node_modules from within the jar.


Is there any best practice of how to handle queries?

For example, how to get an artist by email. Extend the UserRepository with Observable getUserByEmail(String email); ?

Yep, that can work.

But is it a good way? Because in this case we put query logic in the data layer. Isn't it so, that the domain layer and data layer can communicating only using CRUD repository?

There isn't any better layer for data fetching query logic than the data layer. If the email is something the user needs to be able to enter in the presentation layer, and that will end up fetching the User object in the data layer, then what you've suggested makes perfect sense.

I also don't have any better idea. But at the same time i don't like the idea of extending the UserRepository interface with the methods like getUserByEmail(String email). Why? Suppose the situation, if as the next we want to query the user by his name. Will we again extend the UserRepository with the method getUserByName(String name)? And by surname, age etc... What about combinations (something like getUserBySurnameAndAgeLessThan(String name, int age))?

Generally anything that requires user input for data (such as a form, etc.) in the presentation layer in order to obtain data for the current application state from the network/repository in the data layer matches what you are describing, and is the valid and correct way to implement it.
The only time you should question this pattern is if those fields (such as email, name, age that you mentioned in your examples) are not fetched from user input in the presentation layer.

Of course, i mean obtaining the specific data from the data layer and and show it in the presentation layer.
May be my problem is, that i understand the Repository pattern not in the way meant in this architecture. Repository is
public interface Repository<T>
{
	Observable<T> getById(int id);
	Observable<List<T>> getAll();
	bool add(T entity);
	bool remove(T entity);
}

In this case change of the underlying DAL is a simple task. But if we extend the interface implementation with query methods not more so simple, because we have to implement all query methods ...
public interface UserRepository extends Repository<User> {
Observable<List<User>> getUsersByAge(int age);
Observable<List<User>> getUsersByMail(String mail);
Observable<List<User>>gettUsersByMailAndAgeLessThan(String mail, int age);
... //more query methods
}

As a not optimal solution would be to implement the search according to some criteria in the use case object. But this is for very big price: every time we have to load all users from the data base.

If things like getUsersByAge are not something that would make up a UseCase directly then they do not belong in the Repository interface. They belong in the factory pattern implementation as a data fetching implementation detail in the data layer.
If things like getUsersByAge are something that would make up a UseCase directly then they belong in the Repository interface in the domain layer.

Makes sense, thank you!


[Unity] Update legalizer for variance

Now the legalization method for relax.variance is:
var(x) = mean((x - mean(x)) ** 2)

This PR changes the legalization method into
var(x) = mean(x * x) - mean(x) ** 2

for better data locality (which means better performance)

there might be some numerical stability issues here, so would be good to put this on hold a bit and come and visit later.


Why finetuning a backbone required for S3DIS but not for ScanNetv2 ?

Dear all,
Could you please educate me on - why finetuning an HAIS backbone required for training in case of S3DIS dataset but the same is not required for the ScanNetv2 dataset ?(reference : README file)
Is there any parameter like variance in the dataset or is it because HAIS (the backbone network) was originally trained on ScanNetv2 ? It would be great to learn this because I need to train the network for my custom dataset.
Thank you in advance !

+1

The backbone for ScanNet was added later (on 25-Nov-2022) https://github.com/thangvubk/SoftGroup/commit/33ca8c75f18d571dbff6e1696209e54c4b1cdcfc#diff-cb8af471fdaddd5a14d55e28f55c7527d679cfaa4e693f22268cf498cda42002
Hence, closing the issue.


"Applying multiple functions at once" example from the docs broken

Using agg with a list of functions as parameter on a groupby object doesn't work as described:
df = DataFrame({'A' : ['foo', 'bar', 'foo', 'bar', 'foo', 'bar', 'foo', 'foo'],
                'B' : ['one', 'one', 'two', 'three', 'two', 'two', 'one', 'three'],
                'C' : randn(8), 'D' : randn(8)})
grouped = df.groupby('A')
grouped.agg([np.sum, np.mean, np.std])

leads to this (with 0.6.1 stable and master):

AssertionError                            Traceback (most recent call last)
/home/ilja/env/ in ()
----> 1 grouped.agg([np.sum, np.mean, np.std])
/usr/lib/python2.7/site-packages/pandas/core/groupby.pyc in agg(self, func, *args, **kwargs)
295         See docstring for aggregate
296         """
--> 297         return self.aggregate(func, *args, **kwargs)
298
299     def _get_names(self):
/usr/lib/python2.7/site-packages/pandas/core/groupby.pyc in aggregate(self, arg, *args, **kwargs)
954                 return self._python_agg_general(arg, *args, **kwargs)
955             else:
--> 956                 result = self._aggregate_generic(arg, *args, **kwargs)
957
958         if not self.as_index:
/usr/lib/python2.7/site-packages/pandas/core/groupby.pyc in _aggregate_generic(self, func, *args, **kwargs)
982                     result[name] = func(data, *args, **kwargs)
983             except Exception:
--> 984                 return self._aggregate_item_by_item(func, *args, **kwargs)
985         else:
986             for name in self.indices:
/usr/lib/python2.7/site-packages/pandas/core/groupby.pyc in _aggregate_item_by_item(self, func, *args, **kwargs)
1027             result_columns = result_columns.drop(cannot_agg)
1028
-> 1029         return DataFrame(result, columns=result_columns)
1030
1031     def _wrap_aggregated_output(self, output, mask):
/usr/lib/python2.7/site-packages/pandas/core/frame.pyc in init(self, data, index, columns, dtype, copy)
191             mgr = self._init_mgr(data, index, columns, dtype=dtype, copy=copy)
192         elif isinstance(data, dict):
--> 193             mgr = self._init_dict(data, index, columns, dtype=dtype)
194         elif isinstance(data, ma.MaskedArray):
195             mask = ma.getmaskarray(data)
/usr/lib/python2.7/site-packages/pandas/core/frame.pyc in _init_dict(self, data, index, columns, dtype)
260
261         # consolidate for now
--> 262         mgr = BlockManager(blocks, axes)
263         return mgr.consolidate()
264
/usr/lib/python2.7/site-packages/pandas/core/internals.pyc in init(self, blocks, axes, do_integrity_check)
274
275         if do_integrity_check:
--> 276             self._verify_integrity()
277
278     @property
/usr/lib/python2.7/site-packages/pandas/core/internals.pyc in _verify_integrity(self)
353         mgr_shape = self.shape
354         for block in self.blocks:
--> 355             assert(block.values.shape[1:] == mgr_shape[1:])
356         tot_items = sum(len(x.items) for x in self.blocks)
357         assert(len(self.items) == tot_items)
AssertionError:

Sorry, this is an upcoming 0.7 (dev) feature.
As the docs are misleading, one to-do is to version the documentation to sync with the pandas version.

ah, thanks.

On 0.6.1 i get the same error, but on master it works fine.
You can also see in the documentation (which is on master): http://pandas.sourceforge.net/groupby.html?highlight=agg#applying-multiple-functions-at-once

How do I get the master revision to test things against it before reporting issues like this? I'm on a corporate network behind a proxy that doesn't allow checking out with git directly, and the .tar.gz I've downloaded just 30 minutes ago did throw this error. Pip shows its version as 0.7.0.dev-. Is it still behind the head?

When I download the tarball it's signed v0.6-xxx.  We should get that updated.

Actually nevermind, it should be the head. Strange.  Can you just verify with import pandas; pandas._ _ version _ _? I'm confident it must still be < 0.7

Yes, it says "0.6.1".

Try, from the downloaded source directory, "python setup.py install"
Or to use pip, perhaps run "python setup.py develop" and then "pip install
-e /path/to/pandas/source"
On Wed, Jan 18, 2012 at 9:32 AM, elpres <
<EMAIL_ADDRESS>
wrote:

Yes, it says "0.6.1".

Reply to this email directly or view it on GitHub:
https://github.com/wesm/pandas/issues/644#issuecomment-3546416

Ok, I've figured out what I was doing wrong. Thanks for your help.


bump falafel to 2.0.0

Hello,
This PR updates the falafel dependency. falafel 2.0.0 now uses acorn 3 that parse js as ES6 by default. This solves issues like benbria/aliasify#43

Coverage remained the same at 91.892% when pulling be936f4480767d73fae46a7b2303bc1a26fc61f8 on flozz:update-dep-falafel into 64ca9bfa60b8242a4d1c9dbef603f003f46e6ccd on benbria:master.

Would it be possible to have a new version out with this pr merged in ?


Error: require() of ES Module

I'm using react-datetimerange-picker which has react-clock as  a sub-dependency. When I try to run my NextJs app I get the following error:
Error: require() of ES Module /Users/dean/dev/oyoun-project/node_modules/react-clock/node_modules/get-user-locale/dist/esm/index.js from /Users/dean/dev/oyoun-project/node_modules/react-clock/dist/cjs/shared/hourFormatter.js not supported. Instead change the require of index.js in /Users/dean/dev/oyoun-project/node_modules/react-clock/dist/cjs/shared/hourFormatter.js to a dynamic import() which is available in all CommonJS modules.
I have in my package.json the type: module. I assume this is causing the error, which says I cannot use the require statement.
Have you encountered this before?

I have the same error. I am also using Next JS. I tried looking into the transpilePackages option for next config but that didn't help.

That's quite strange I must say, as react-clock ships CJS and ESM versions, CJS requiring - surprise - CJS, ESM - importing ESM. I have a suspicion #158 might fix that, or at least move the error a dependency "up", resolving the issue between react-clock and get-user-locale, but I'm not sure.

Mmmm, I think I got it. Working for me in Next.js.
You'll need to bump get-user-locale package to verify. For example, in Yarn:
yarn up get-user-locale -R
Alternatively, remove react-clock and add again.

Wow, that did the trick! I removed react-clock, ran rm -rf node_modules, and then added it back in. Thanks so much for the speedy help here @wojtekmaj !

Gosh CJS/ESM double packages are a pain :joy:

I have the same error. I am also using Next JS. I tried looking into the transpilePackages option for next config but that didn't help. I am just using react-clock. Next version: 13.2.3

Since creating the issue I found a fix for my situation. Firstly, I was using Next 12.x, so I upgraded to Next 13.2.3. This allowed me to use transpilePackages.
To get it working then, I added this block to my next.config.mjs:
transpilePackages: [         "react-clock",         "@wojtekmaj/react-datetimerange-picker",         "get-user-locale",     ],     experimental: { esmExternals: "loose" },
Since then it's working.


Update ip_warmup.md

Description of the change: Grammar error - line 69
Reason for the change: Grammar should insert YOU
Link to original source:

Closes #

Hello @kdcr2d2,Thanks again for the PR! We appreciate your contribution and look forward to                                              continued collaboration. Thanks!                                             Team SendGrid DX


Revert "Enforce authentication on api/status route by default (#943)"

Reverts https://github.com/opensearch-project/security-dashboards-plugin/pull/943. The change was planned to be merged after 2.0.0-rc1.

Codecov Report

Merging #950 (44b73bf) into main (350e605) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #950   +/-   ##
=======================================
  Coverage   72.14%   72.14%
=======================================
  Files          87       87
  Lines        1906     1906
  Branches      242      242
=======================================
  Hits         1375     1375
  Misses        477      477
  Partials       54       54

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 350e605...44b73bf. Read the comment docs.


Fix crash when printing leaked nodes

Fixes #39937
I couldn't find any reason why object is nullptr, because code looks fine.
This is maybe not the best solution, but allows to get info about leaked nodes which is very helpful to find regression and leaks.
CC @reduz since this code was added in https://github.com/godotengine/godot/commit/867d073b98344b848c96012418912a7e72841a31

I'm not sure that's enough. That could skip some objects.
I've been trying to come up with the algorithm that would iterate through the used object slots (by the way, that would apply both for ObjectDB::cleanup() and ObjectDB::debug_objects()).
After playing a bit with creating and destroying objects in different orders, I've reached this state in the object_slots array:

(With slot_count = 7,  slot_max = 16.)
After considering that case, the only solution that comes right now to my mind is the following, but I suspect this is not leveraging the design of this data structure for doing it more optimally:
uint32_t count = 0;
for (uint32_t i = 0, count = slot_count; i < slot_max && count != 0; i++) {
	if (object_slots[i].validator) {
		// Do whatever with object_slots[i].object

		count--;
	}
}

I'd still like to know @reduz's opinion about this implementation, because I still think there is a non-brute-force approach.
Anyway, I guess this could get merged because it already addresses the issue and an optimization can be done later.

Looks ok to me

Thanks!


ghc-8.10.1: rc1 -> final

Motivation for this change
https://mail.haskell.org/pipermail/ghc-devs/2020-March/018729.html
https://mail.haskell.org/pipermail/ghc-devs/2020-March/018730.html
Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[ ] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[ ] Fits CONTRIBUTING.md.

@GrahamcOfBorg build haskell.packages.ghc8101

To be checked: are these diffs still necessary?
$ diff -u pkgs/development/compilers/ghc/8.8.3.nix pkgs/development/compilers/ghc/8.10.1.nix
--- pkgs/development/compilers/ghc/8.8.3.nix	2020-03-02 15:30:57.684259857 +0100
+++ pkgs/development/compilers/ghc/8.10.1.nix	2020-03-24 22:04:40.116642419 +0100
@@ -88,12 +88,12 @@

 in
 stdenv.mkDerivation (rec {
-  version = "8.8.3";
+  version = "8.10.1";
   name = "${targetPrefix}ghc-${version}";

   src = fetchurl {
-    url = "https://downloads.haskell.org/ghc/${version}/ghc-${version}-src.tar.xz";
-    sha256 = "128g932i3wix6ic03v04nh5755vyjiidzri9iybwad72yfmc1p70";
+    url = "https://downloads.haskell.org/ghc/8.10.1/ghc-${version}-src.tar.xz";
+    sha256 = "1xgdl6ig5jzli3bg054vfryfkg0y6wggf68g66c32sr67bw0ffsf";
   };

   enableParallelBuilding = true;
@@ -149,21 +149,15 @@
   # TODO(@Ericson2314): Always pass "--target" and always prefix.
   configurePlatforms = [ "build" "host" ]
     ++ stdenv.lib.optional (targetPlatform != hostPlatform) "target";
-
   # `--with` flags for libraries needed for RTS linker
   configureFlags = [
     "--datadir=$doc/share/doc/ghc"
     "--with-curses-includes=${ncurses.dev}/include" "--with-curses-libraries=${ncurses.out}/lib"
-  ] ++ stdenv.lib.optionals (libffi != null) [
-    "--with-system-libffi"
-    "--with-ffi-includes=${targetPackages.libffi.dev}/include"
-    "--with-ffi-libraries=${targetPackages.libffi.out}/lib"
+  ] ++ stdenv.lib.optionals (libffi != null) ["--with-system-libffi" "--with-ffi-includes=${targetPackages.libffi.dev}/include" "--with-ffi-libraries=${targetPackages.libffi.out}/lib"
   ] ++ stdenv.lib.optional (targetPlatform == hostPlatform && !enableIntegerSimple) [
-    "--with-gmp-includes=${targetPackages.gmp.dev}/include"
-    "--with-gmp-libraries=${targetPackages.gmp.out}/lib"
+    "--with-gmp-includes=${targetPackages.gmp.dev}/include" "--with-gmp-libraries=${targetPackages.gmp.out}/lib"
   ] ++ stdenv.lib.optional (targetPlatform == hostPlatform && hostPlatform.libc != "glibc" && !targetPlatform.isWindows) [
-    "--with-iconv-includes=${libiconv}/include"
-    "--with-iconv-libraries=${libiconv}/lib"
+    "--with-iconv-includes=${libiconv}/include" "--with-iconv-libraries=${libiconv}/lib"
   ] ++ stdenv.lib.optionals (targetPlatform != hostPlatform) [
     "--enable-bootstrap-with-devel-snapshot"
   ] ++ stdenv.lib.optionals useLdGold [

@GrahamcOfBorg build haskell.compiler.ghc8101

LGTM, thanks for always being on top of these updates!


Fix sorting of filenames with alphanumeric prefixes.

Enhance filename sorting by extracting numeric parts

Added a new function _extract_numeric to extract numeric values from filenames containing alphanumeric characters.
Updated the sorting key function to utilize _extract_numeric, improving handling of filenames with both numeric prefixes and postfixes (e.g., 'DSCF5858', 'IMG1234').
Ensured compatibility with filenames where numeric values are embedded within strings.

My solution here is to allow passing a list into it:
https://github.com/facebookresearch/segment-anything-2/pull/214


[Internal] Performance : Improve Region Contacted information Logic

We are continuously addressing and improving the SDK, if possible, make sure the problem persist in the latest SDK version.

Describe the bug
It is non functional issue. As region contacted collection logic needs to be optimized.
Avoid reinitialization of this Hashset everytime, it has impact on client telemetry also as this information is getting fetched in each request.
https://github.com/Azure/azure-cosmos-dotnet-v3/blob/b73ed615e03aed0e7adec5ec309ad3eecc2d293a/Microsoft.Azure.Cosmos/src/Diagnostics/CosmosTraceDiagnostics.cs#L49

[ ] Perf benchmark to measure the impact
[ ] Optimization for primary flows

Benchmarked on the regular perf tool with three runs average, here is how they stack up

Measure
baseline
snapshot
%overead

P50 Rps
11035
10878
1.42

P95 Rps
10792
10615
1.64

P99 Rps
10699
10441
2.41

Of course its not a long running a perf bed. And the allocations might show-up there on higher percentile.

p99 is 2.5% percent overhead. I think its considerable overhead.


Separate type for unaggregated network attestations

As a complement to
https://github.com/ethereum/consensus-specs/pull/3787, this PR introduces a SingleAttestation type used for network propagation only.
In Electra, the on-chain attestation format introduced in EIP-7549 presents several difficulties - not only are the new fields to be interpreted differently during network processing and onchain which adds complexity in clients, they also introduce inefficiency both in hash computation and bandwidth.
The new type puts the validator and committee indices directly in the attestation type, this simplifying processing and increasing security.

placing the validator index directly in the attestation allows verifying the signature without computing a shuffling - this closes a loophole where clients either must drop attestations or risk being overwhelmed by shuffling computations during attestation verification
the simpler "structure" of the attestation saves several hash calls during processing (a single-item List has significant hashing overhead compared to a field)
we save a few bytes here and there - we can also put stricter bounds on message size on the attestation topic because SingleAttestation is now fixed-size
the ambiguity of interpreting the attestation_bits list indices which became contextual under EIP-7549 is removed

Because this change only affects the network encoding (and not block contents), the implementation impact on clients should be minimal.

Attestation signature verification in the existing protocol implicitly runs committee association check. Without this check and with EIP-7549 a committee_index can be tampered and a subnet can be flooded with attestations from attesters that doesn’t belong to the subnet’s committee. Before EIP-7549 this wouldn’t be a problem as signing incorrect committee_index would be quite expensive attack.
The committee association check requires shuffling computations. Given the EIP-7549 can we avoid this check anyhow? @arnetheduck

The committee association check requires shuffling computations.

The committee_index still needs to be checked but crucially, with this change we can do so after signature verification meaning that only validators can create new shufflings.
Since the shuffling order is needed to tell whether the validator should vote in the particular slot, I don't see that it can be avoided without more extensive surgery to the protocol.

The proposed change prevents (1) but doesn’t help with (2). Is this correct?

the shuffling is computed from the signed fields (target / block_root depending on design), so this is harmless to the extent that this shuffling will be signed by a validator (and therefore likely cached)

add the explicit check

broadly, the phase0 conditions still apply since they reference index which we replace in this PR.

If this change is introduced I believe we should also make a change to submitPoolAttestationsV2
PR: https://github.com/ethereum/beacon-APIs/pull/472

Echoing @eserilev concerns, we may force on ourselves another decent churn of type changes across the codebase. While I see the benefits of this change I believe the upside of EIP-7549 is much more significant and impactful towards Ethereum participants than this change. The unified type of Attestation is annoying yes, but our codebases are already implemented to handle it.

of it being "minimal"

minimal here means that it does not affect the consensus spec significantly since the parts outside of the state transition are much less strictly specified leaving clients more leeway to implement this as they wish - in fact, one possible implementation is to simply translate to Attestation in the gossip handler which indeed would be a "minimal" implementation that requires no further changes.
Also notable is that the comparison is done not to the work-in-progress spec but rather the deneb spec: seen through this lens, this PR (together with 3787) makes the total change of going from deneb to electra much smaller (since the changes much less invasive), even if in-between there existed a high-churn version in the development version of electra.
If anything, the churn should really be attributed EIP-7549 which changed more than it needed (had it introduced just the on-chain change, it would have had much smaller impact) and with #3787 we're just reverting some of that and simplifying further).
Strictly, the REST API changes are not necessary and should be judged on their own merit (since the full Attestation object is a superset).

If anything, the churn should really be attributed EIP-7549 which changed more than it needed (had it introduced just the on-chain change, it would have had much smaller impact)

The core change introduce by EIP-7549 is removing the committee_index from the signing data which unlocked the opportunity to pack attestations more tightly. The form of the on chain aggregate isn’t a source of the vast majority of the engineering complexity around this EIP. Having a separate type for on chain aggregate is cleaner, but isn’t crucial IMO, tho client devs might disagree

Wouldn't you want to update the validator spec here as well, in particular how validator constructs the attestation should use SingleAttestation?

Agreed: 1c529a8

Given https://github.com/ethereum/consensus-specs/pull/3900#issuecomment-2351908558
there is an impact on Web3Signer API to introduce SINGLE_ATTESTATION signing request type

Given https://github.com/ethereum/consensus-specs/pull/3900#issuecomment-2351908558
there is an impact on Web3Signer API to introduce SINGLE_ATTESTATION signing request type

this is not 100% necessary - we can also translate back and forth - Attestation, SingleAttestation etc are just envelopes - one reason we kept AttestationData intact with the zeroed the same was to avoid changing web3signer and other third-party tooling that just blindly signs what it is given.
Notably, web3signer is trusted so the rationale about shufflings doesn't apply and since this is the producer, the performance argument is less important as well (since you don't have to do it thousands of times per slot).
All in all, I have no strong opinion either way.

@arnetheduck
I nearly immediately deleted the comment because double checking the Web3Signer API i saw only AttestationData is sent. So I'm ok with leaving everything untouched there.

Looks like the decision on this PR has not been made yet and it's a trade-off between the benefit and the implementation complexity.
If we decide to do SingleAttestation, why don't we use this opportunity to do separate types for all stages of attestations.

One type for attestation subnets - SingleAttestation (this PR)
Another type for aggregated attestations - AggregatedAttestation (we can use something similar to phase0 Attestation but with committe index moved outside)

class AggregatedAttestation(Container):
    aggregation_bits: Bitlist[MAX_VALIDATORS_PER_COMMITTEE]
    data: AttestationData
    signature: BLSSignature
    committee_index: CommitteeIndex

Another type for attestations to be put in the block - BlockAttestation (use EIP-7549 Attestation)

I don't know if this has a performance improvement or not, but this will make the spec look cleaner.

@ppopth that's more or less the outcome of applying this PR together with https://github.com/ethereum/consensus-specs/pull/3787

applying this PR together with https://github.com/ethereum/consensus-specs/pull/3787

I think the idea is to be more explicit about aggregated attestations, or would those just use the electra modifiedAttestation container?

I've spent some time on this and the related https://github.com/ethereum/consensus-specs/pull/3787
My current impression is that together they add complexity in code and some impact on API (including event stream).
At the current stage I'm not in favour of considering for inclusion.
I see the impact of having SingleAttestation and we could consider including only this change, but still API (including event stream) make me doubting about cost\benefits as of right now.

API (including event stream).

I would probably not change the event stream for the same reason as we left index in AttestationData - this affects a greater amount of downstream tooling and I agree this would probably not be worth it. In part, this is why the PR focuses on the network format of the gossip channel which is critical to security in a way that APIs aren't (because it's exposed to the wider p2p network).
The event stream encoding can trivially be generated from SingleAttestation.

We started implementing this in Prysm and found that implementing this is not so trivial. It would be better to postpone this until the next fork and implement it along with #3787.

Verification + Slashability checks need to be implemented for this new type. These are sensitive codepaths and will now need to be either significantly refactored or duplicated to handle this new type.

What kind of changes are required here? Slashing protection only operates on the data (type AttestationData) which is unaffected by this change.

What kind of changes are required here? Slashing protection only operates on the data (type AttestationData) which is unaffected by this change.

Signature verification is one example of a check that doesn't operation on AttestationData
My comments were more directed at how Lighthouse has implemented these codepaths. Though there are some checks that only operate on AttestationData we are, in many cases, passing the full Attestation object to these methods. Introducing this new type to those codepaths will require a significant refactor.
In the interim we've decided to simply translate SingleAttestation to Attestation when received over gossip. This keeps the rest of the downstream code unchanged while being fully spec compliant for any upcoming devnets. In parallel we'll be working on a larger refactor.

placing the validator index directly in the attestation allows verifying the signature without computing a shuffling - this closes a loophole where clients either must drop attestations or risk being overwhelmed by shuffling computations during attestation verification

In normal network conditions, clients will either already have the shuffling cached and have to cache it for later processing of aggregate attestations.
If under a deliberate DOS attack, a malicious peer can still trigger wasteful shuffling computations by forging aggregated attestations

If under a deliberate DOS attack, a malicious peer can still trigger wasteful shuffling computations by forging aggregated attestations

The aggregate topic uses an envelope that is already signed by a validator - it does not offer the opportunity of DoS by a non-validator as the attestation topic does.

The https://github.com/ethereum/consensus-specs/pull/3900#issuecomment-2315736339 uses an envelope that is already signed by a validator - it does not offer the opportunity of DoS by a non-validator as the attestation topic does (see previous discussion).

I see this as a strong point in favor of this PR. With this PR only validators can attack nodes to compute wasteful epoch transitions. So we'll have attributability in case of an attack.

I think this should not be named "SingleAttestation". This should be "Attestation" while the other is "AggregatedAttestion". When people see "Attestation", they more likely think of an attestaion (a single one) rather an aggregated one.

Would things be even more clear if we had a SingleAttestation and an AggregatedAttestation (and potentially an OnChainAttestation as per https://github.com/ethereum/consensus-specs/pull/3787), and just let go of the generic Attestation?

I'm happy to pick any name that gets consensus ;)
I agree that things would be better if #3787 was also merged and if the aggregated attestation was renamed, but not to the point that I want this PR to be delayed by it as the severity of the issue they fix is different.
In Nimbus, it's trivial to implement #3787 simply by changing a constant - we're planning on having a separate OnchainAttestation type anyway because they appear in different contexts and are conceptually different even if the spec shoehorns them into one type. If you want to experiment with the complexity of implementing it, this is one way to go and then we can decide on 3787 separately from this PR.
Notably though, any renames can also be done after the substance of this PR has been merged.

+1 for detaching this from eventual renamings. Let's merge as is.

In Nimbus, it's trivial to implement https://github.com/ethereum/consensus-specs/pull/3787 simply by changing a constant - we're planning on having a separate OnchainAttestation type anyway because they appear in different contexts and are conceptually different even if the spec shoehorns them into one type. If you want to experiment with the complexity of implementing it, this is one way to go and then we can decide on 3787 separately from this PR.

This is Lodestar's experience as well. If we decide to merge #3900, might as well merge #3787 too since it's not that hard to implement.


🐛 [firebase_app_check] pub: 0.1.2, xcode: 14.3, ios: 16.4 run error

Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel stable, 3.7.8, on macOS 13.3 22E252 darwin-arm64, locale
    zh-Hans-CN)
[✓] Android toolchain - develop for Android devices (Android SDK version 33.0.2)                           [✓] Xcode - develop for iOS and macOS (Xcode 14.3)
[✓] Chrome - develop for the web
[✓] Android Studio (version 2022.1)
[✓] IntelliJ IDEA Ultimate Edition (version 2023.1)
[✓] IntelliJ IDEA Ultimate Edition (version 2023.1)
[✓] IntelliJ IDEA Ultimate Edition (version 2022.3.3)
[✓] VS Code (version 1.76.2)
[!] Proxy Configuration
    ! NO_PROXY does not contain ::1
[✓] Connected device (4 available)
[✓] HTTP Host Availability

! Doctor found issues in 1 category.

same issue.
I changed it to 'alloc' and using it.

Hi @mdddj, is this error only on Xcode 14.3? or did you get this error before the upgrade?

Yes, it didn't appear until xcode was upgraded.

Same issue here

same issue. I changed it to 'alloc' and using it.

Will this "stick" for the compile process? Or does compiling ignore a manual change like that?

Same issue here since upgrading xcode:
Failed to build iOS app
Could not build the precompiled application for the device.
Semantic Issue (Xcode): 'new' is unavailable

Happened to me after I upgraded Xcode.
I updated my iPhone and Mac as well but no results.

My current workaround is to switch to older version of Xcode.
And here is the way:
https://stackoverflow.com/questions/14756026/how-to-downgrade-xcode-to-previous-version

Same issue here - I can confirm this occurred after updating to the latest Xcode version 14.3 (14E222b).
Note: I am also using iOS 16.4

same issue.
I changed it to 'alloc' and using it.

This tmp workaround will get your app to compile while we wait for an update

I attempted to reproduce this locally but I'm running into an issue with cocoapods. Similar to https://github.com/CocoaPods/CocoaPods/issues/11808#issuecomment-1481244508
Labeling for further investigation from the team.
cc @Lyokone
Launching lib/main.dart on Nexus in debug mode...
Automatically signing iOS for device deployment using specified development team in Xcode project: X8NNQ9CYL2
Running pod install...
Running Xcode build...
Xcode build done.                                            6.1s
Failed to build iOS app
Could not build the precompiled application for the device.
Error (Xcode): File not found: /Applications/Xcode-14.3.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/arc/libarclite_iphoneos.a

Error (Xcode): Linker command failed with exit code 1 (use -v to see invocation)

Error launching application on Nexus.

I attempted to reproduce this locally but I'm running into an issue with cocoapods. Similar to CocoaPods/CocoaPods#11808 (comment)
Labeling for further investigation from the team.
cc @Lyokone
Launching lib/main.dart on Nexus in debug mode...
Automatically signing iOS for device deployment using specified development team in Xcode project: X8NNQ9CYL2
Running pod install...
Running Xcode build...
Xcode build done.                                            6.1s
Failed to build iOS app
Could not build the precompiled application for the device.
Error (Xcode): File not found: /Applications/Xcode-14.3.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/arc/libarclite_iphoneos.a

Error (Xcode): Linker command failed with exit code 1 (use -v to see invocation)

Error launching application on Nexus.

This has to do with https://github.com/flutter/flutter/issues/123903 . Solution is to upgrade min deployment target for all Pods to 11.0 (or greater) by adding the following to your Podfile:
installer.generated_projects.each do |project|
    project.targets.each do |target|
        target.build_configurations.each do |config|
            config.build_settings['IPHONEOS_DEPLOYMENT_TARGET'] = '11.0'
         end
    end
  end

The issue is, once you do this and get past the CocoaPods error, if you depend on firebase_app_check, you get this error.

In the meantime, I'm personally just downgrading to XCode 14.2 until Flutter + CocoaPods update to not require these workarounds. Easier than trying to play whack-a-mole w/ different libraries :)

Every time there's an update from Xcode we are forced to download it (because we can't run the app on our device), it's at least 7GB large, it's slow, AND it causes problems every time. Apple should be more careful about doing this.
A temporary solution is to replace those new keywords to alloc.

After working around the issue with level-db, I can reproduce the issue.
Changing new to alloc at the lines in the error allowed the app to run.
Launching lib/main.dart on Nexus in debug mode...
Automatically signing iOS for device deployment using specified development team in Xcode project: X8NNQ9CYL2
Running Xcode build...
Xcode build done.                                           164.4s
Failed to build iOS app
Could not build the precompiled application for the device.
Semantic Issue (Xcode): 'new' is unavailable
/Users/nexus/.pub-cache/hosted/pub.dev/firebase_app_check-0.1.2/ios/Classes/FLTAppCheckProvider.m:18:55

Semantic Issue (Xcode): 'new' is unavailable
/Users/nexus/.pub-cache/hosted/pub.dev/firebase_app_check-0.1.2/ios/Classes/FLTAppCheckProvider.m:22:53

Semantic Issue (Xcode): 'new' is unavailable
/Users/nexus/.pub-cache/hosted/pub.dev/firebase_app_check-0.1.2/ios/Classes/FLTAppCheckProvider.m:30:57

Error launching application on Nexus.

Flutter issued a new release supposed to fix this: https://groups.google.com/g/flutter-announce/c/9qZL8GOVNb4
Can you check if it also fixes the issue with firebase_app_check?

Flutter issued a new release supposed to fix this: https://groups.google.com/g/flutter-announce/c/9qZL8GOVNb4 Can you check if it also fixes the issue with firebase_app_check?

After updating to version 3.3.10, there were no noticeable changes.

Flutter issued a new release supposed to fix this: https://groups.google.com/g/flutter-announce/c/9qZL8GOVNb4 Can you check if it also fixes the issue with firebase_app_check?

Sadly the Flutter 3.10 upgrade didn't fix this app firebase_app_check

it still happens with Flutter 3.7.10

Same issue

when can we expect an update?


feat(api): Scene classification webhook

What does this pull request do? Explain your changes. (required)
Implements https://github.com/livepeer/livepeer-com/issues/450 and https://github.com/livepeer/livepeer-com/issues/451

Specific updates (required)

-

How did you test each of these updates (required)

Does this pull request close any open issues?

Screenshots (optional):

Checklist:

[ ] I have read the CONTRIBUTING document.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[ ] I have added tests to cover my changes.

Changed base to be on top of https://github.com/livepeer/livepeer-com/pull/453 so I can use some of the new abstractions already, specifically the webhook cannon.

Merged into https://github.com/livepeer/livepeer-com/pull/466


[demo] unreadable button on navigation

Describe the bug
Button on the right, leading to stoplight website is unreadable:

To Reproduce
Visit https://elements-demo.stoplight.io/?spec=https://raw.githubusercontent.com/github/rest-api-description/main/descriptions/ghes-3.0/ghes-3.0.json#/operations/search/code
Expected behavior
a) looks cool
b) is readable

New insight - appears only on safari.
@philsturgeon does this change priority?

Yeah, bumped this to longterm.

Not reproducible anymore.


Update instances.yml

I have added my Instance to the yml file.
Please let me know if anything else is needed.  I am not a developer, just a regular guy.

👍


[BUG] VoteTeam server freeze

When votes get only 8 or 9 players voted the server sometimes freezes and needs to be restarted manually. No crash logs, so it freezes due to GetWinner() function?
Sometimes when we get 8 or 9 votes from total 10 players the server freezes, sometimes it is okay
More info:
We are 10 people everything okay, VoteTeam appers and only 8 or 9 players votes, then the server freezes sometimes and need to be restarted manually.
Sorry for my bad english and the bad explanation.

Bump!
Tested a hundred times with BOTS and everything is okay and no freeze.
The problem still remains with online players/people.
Any ideas from what it can be? The HideMenu loop? or GetWinner() function?
When the countdown ends and when not everyone voted, usually at 8 or 9 votes from 10 players, the server freezes and need to be restarted manually.

@axl303
Add-me on steam: https://steamcommunity.com/id/_SmileY/

Here is the whole comment at what I did:
https://pastebin.com/raw/WtdB1MR1
The video is in discord I don't know how to share it.
That's the only problem (critical) that needs to be fixed in this project. I will do whatever I can to provide information, please check ur Discord, so we can communicate easily and fix this thing.

I can confirm that last Task System fixes are working perfect and the crash doesnt exist. https://github.com/SmileYzn/RePugMod/actions/runs/5383573167


Making .mention-bot valid JSON

so that it will work :)
see https://github.com/studentinsights/studentinsights/pull/1045#issuecomment-329992968

selfie


Independent delta not working in .then

It seems like independent deltas don't work in .then() functions.
I am trying to create a jump animation using this code:
const circle = new mojs.Shape({
  top: 40,
  left: 40,
  y: {
    0: 100,
    easing: "cubic.out"
  },
  x: {
    0: 20,
    easing: "linear.none"
  },
  radius: 40,
  duration: 250,
}).then({
  y: {
    100: 0,
    easing: "cubic.in"
  },
  x: {
    20: 40,
  }
});

When adding any of the independent delta properties specified in https://mojs.github.io/api/modules/html/#independent-deltas, the .then() is ignored altogether.
Am I using it improperly or is there any fix for this?

@strukteon Could you create a Codepen for us to test? But I think the problem here is the 0. If I remember correctly, if the key is a zero or negative values, you need to define it as a string instead of a number.
So "0" : "100"

Weirdly enough, the value 0 works when not specifying any easing parameters. Also, some properties (like scale) can be used with a separate easing function.
Here's the codepen: https://codepen.io/strukteon-the-vuer/pen/jOYeBex

Hi @strukteon,
Good catch, I confirm it's a bug!
You can workaround using a "global" easing define at the root of then, like this:
const scale = new mojs.Shape({
  top: 200,
  left: 500,
  scale: {
    0: 3,
    easing: "elastic.out"
  },
  x: {
    0: 200
  },
  radius: 40,
  duration: 1500
}).then({
  easing: "elastic.out", // x later will use this easing
  scale: {
    3: 1,
    easing: "cubic.in" // scale will properly use this easing
  },
   x: {
    200: 0
    // specifying an easing here won't work at all
  }
});

For the moment, I am not seeing any other workaround for this case, or you may need to build your animation in a different way.

@xavierfoucrier Alright, thanks! 😃


[TF:TRT] Fix Grappler params to enable layout optimizer for the C++ API

This PR sets the VirtualCluster argument while calling the Grappler optimization pass for TF-TRT. Setting the cluster arg is necessary to enable layout optimizer (otherwise layout optimizer returns without changing the graph). This change only affects conversions initiated by the C++ API of TF-TRT #52012.
Tagging @bixia1 for review.

@tfeher Can you please check @bixia1's comments and keep us posted ? Thanks!

Added a mutex because we can only provision one cluster per process.

@tfeher  Can you please address Ubuntu Sanity errors? Thanks!

The logs says "ERROR: An error occurred during the fetch of repository 'rules_proto':" This is not related to my changes.

The changes are already included in master, in commit https://github.com/tensorflow/tensorflow/commit/1d5124678decc63d3b64f2598b91a61169748c3e. Closing the PR.


[TEST] Analyze master-head pipeline test result (build 282)

What's the test to develop? Please describe
Analyze master-head pipeline test result (build 282)
test_ha.test_autosalvage_with_data_locality_enabled

Issue [BUG] test_autosalvage_with_data_locality_enabled failed in v1.3.1-rc2

test_basic.test_allow_volume_creation_with_degraded_availability_restore

Issue [BUG] test_allow_volume_creation_with_degraded_availability_restore failed at master-head pipeline

test_basic.test_backup_volume_restore_with_access_mode[s3-rwx-rwo]

PR bug: fix to create rwx volume with PV & PVC deployment #1147

test_csi_snapshotter.test_csi_snapshot_with_invalid_param

Teardown fail on pipeline, can not reproduce on local

test_ha.test_auto_remount_with_subpath

PR https://github.com/longhorn/longhorn-tests/pull/1159

test_scheduling.test_replica_auto_balance_node_best_effort

Flaky, fixing

test_settings.test_setting_concurrent_rebuild_limit

Flaky, fixing

Describe the items of the test development (DoD, definition of done) you'd like

[ ] Analyze failed item, create related issue or fix test script

Additional context
N/A

Hi @chriscchien
If you have tasks about test_ha.test_auto_remount_with_subpath, test_scheduling.test_replica_auto_balance_node_best_effort & test_settings.test_setting_concurrent_rebuild_limit, could
you also tag me in the ticket? I need to reference that.

Currently master-head pipeline there few failed test case, distro AMD64 and ARM64 both only have 2 failed case, 1 was known issue test_autosalvage_with_data_locality_enabled and another 1 was flaky test case, close this ticket


Add row filtering

We can do it the postgrest way though that doesn't allow for OR conditions; just AND. We could also use something like ?where=... though then we'd have to sanitise it.

cc @jmealo any thoughts on API structure for row filtering? Primarily need this to allow updating/deleting rows (for which the postgrest way is fine), but later we'll add a way to filter rows via the UI.

For now we have basic equality-based row filtering (?age=30&location=Philadelphia or more commonly ?id=3). This will do for now until we add a search feature or really dive into the usability of the API.


Weather station as location

Is it possible to use a specific weather station (e.g. KMOSTCHA6) as a location in the config?

Yes try prepending it with pws: ie pws:KMOSTCHA6

pws:KMOSTCHA6 works as I'd hoped, thank you.

"pws:KMOSTCHA6" is used as "Location" in the Config.JSON file instead of city, zip code, GPS coordinates (which also work to select a specific weather station as the station's Lat and Long is listed at the WeatherUnderground website). Any type or form of location identification that WeatherUnderground recognizes can be used.

and when using pws:xxxx as the location so i still use my api key or do i use my pws key?  Neither are working for me.  So now im wondering if the change in the wunderground api is the culprit?


remove var_dump

Hello,
I can't find a good reason for this var_dump to be here, and it makes my build outputs dirty :)
thanks for your work!

Thanks :)


Streaming quality

What stream quality is selected when streaming?

The bitrate_preset option sets the preferred bitrate. Unfortunately, I haven't gotten around to document it (#36) yet.

I just pushed 66daa39f08b0, which documents the available configuration options, including bitrate_preset. Thanks for asking!

Thank you!


Invalid option error

Hey
Not really sure if I'm the one doing something wrong here, but I've tried a few things to see if I can fix it, with no luck.

This happens alot, and I'm not really sure why. He got read/write access, running as admin, and it seem to be a x/y issue. Could it be a multi monitor issue? I'm running a 5 screen setup with 3 horizontal screens, and 2 above. Could this maybe screw with it?
Running beta .8, and newest autohotkey version .37.1

It seems to be having problems with getting the correct screen coordinates to display help-tooltips when hovering over ui-elements in the settings menu.
Could you right-click the script's tray icon and open Window Spy? Make sure Follow Mouse is un-checked, then open the script's settings menu again and click into it so it's focused.
I need to know if the script is misreading window coordinates and, if so, what the coordinates are.

First attempt, general tab.

Clicking the Cheat-sheets

Seem to trigger when I hoover over the enable checkbox

Wonder if this is what your looking for. This is the same spot; with each in focus.

Mouse same place with script focused (it auto focuses when error trigger).

or are the coords relative to anchor?

Nevermind the coords relative question. I see it now. When it focuses the window changes, so it makes sense. Screen is always the same

Thanks for the screenshots. I can see a help-tooltip on the first screenshot of the general tab. Does the error-message appear only when switching tabs and hovering stuff, or does it also appear when hovering over buttons/checkboxes in the general tab?

General tab and betrayal-info seem to work fine. All hoovers work.
Cheat-sheets
Clone-frames
Hotkeys (only the bottom checkbox trigger error (misc keys))
Item-info
Mapping tracker
Screen-checks
search-strings
updater
^ All above failed on all hoovers. I assume its when he tries to generate a tooltip. I find it abit weird though, since the updater tab, I screenshoted that yesterday when I first posted, and it worked. Not now for some reason. I even reloaded the script between each tab test

Can you send me a screenshot of your display arrangement?

I'll try and replicate it on my end. I can't see anything out of the ordinary in the Window Spy coordinates.

technically could disable all other displays just to see if that changes anything, just to save you some trouble. Gimme a minute

Still the same with only one display. Ironically betrayal info checkbox hoover failed this time xD

Just tested something: If you check Follow Mouse in the Window Spy and repeatedly click any tab in the settings menu, the ahk_id in the top-most box should change its value with every click but everything else should stay the same (some values in the mouse position box may flash briefly, but that doesn't matter). Could you check if it behaves the same on your end?

Something else entirely:

Just noticed how your Window Spy looks different between screenshots. Scaling seems to be bricked or something. May not be significant, but I'm grasping at straws here. The fact that the error pops up inconsistently between tabs could also point to some AutoHotkey bug. You could try a clean reinstall.

Thats because I tried to drag it out I think to see the entire error message in the all text box. The spy thingie broke alittlebit when I started resizing it

can uninstall and try anyway though just to check

I couldn't get this weird issue out of my head, so here's something you can try. It's a modified version of the script that shows the screen-coordinates. If you hold the shift-key and hover over checkboxes/buttons, a text-box will open:

Lailloken UI.zip

Enabled cheatsheet, and wh en I hoovered over hotkey settings :

Are you using any software/hardware solution for your monitor setup, or do they all just go straight into your graphics card? The script cannot reliably get the position of ui-elements that are being hovered over.

straight into gpu. Nothing other than windows handle it

But this seem so far to be a only me problem though (?), so maybe it isn't worth spending time on.

I'll release an update tomorrow morning that uses the cursor-position for tooltips as a fallback if the normal coordinates cannot be retrieved correctly.

Since these tooltips are also used outside the settings menu, more testing needs to be done. Use this modified version for now:
Lailloken UI.zip
The easiest way to test them outside the settings menu is enabling the Lab Overlay in the Minor QoL Tools tab, then entering Aspirants' Plaza and holding TAB. An overlay with an icon in the top-left corner will open, and that icon has a tooltip when hovered over.

Hoovers in the lab overlay works fine

Please note that this new file broke the F1 key in clone-frame's

Tried to save. That might be my fault though, since I fiddled pretty hard with the frame

Something seems fundamentally broken. The modified file doesn't handle anything with hotkeys, so it's strange that the F-keys are bricked. But generally this looks like there are problems with screen/mouse coordinates yet again, with F3 only moving on one axis.
The error-message after saving is due to the script not being able to refresh/rebuild the settings window correctly. It normally creates a second window on top of the first one, then destroys the first one in order to prevent the window from causing a flash. In your case, it can't find the first one anymore and throws an error.
Did you end up reinstalling AutoHotkey two days ago, when I suggested it?

I did uninstall, then rebooting and installed again. And nothing is modified in windows except murdering the telemetry services windows is shipped with these days.
On a sidenote, I did notice something different now though. I reloaded the script, and suddenly F1 works, but F3 doesn't work vertically now. Only horizontally. There is something very weird going on here. Positive note is that I cant manage to get an error no matter how hard I try.

I see he changes the X/Y coordinates in the window when I press F3, but he doesn't move down for some reason


Test pipeline

Description
Changes proposed in this pull request:

...
...
...

Related issue(s)

/test all


Warning in HA log

WARNING (MainThread) [homeassistant.helpers.frame] Detected that custom integration 'energy_id' uses async_config_entry_first_refresh, which is only supported when entry state is ConfigEntryState.SETUP_IN_PROGRESS, but it is in state ConfigEntryState.LOADED, This will stop working in Home Assistant 2025.11 at custom_components/energy_id/sensor.py, line 72: await record_peak_power_coordinator.async_config_entry_first_refresh(), please create a bug report at https://github.com/tijsverkoyen/HomeAssistant-EnergyID/issues

This is fixed in https://github.com/tijsverkoyen/HomeAssistant-EnergyID/releases/tag/v1.3.3