aws s3 support

hi
at first, thanks for this awesome package.
I'm working on a web app project that uses Amazon S3 for keeping files.
now, I want to create a preview for each file in S3.
do you have any ideas on how can I do this? creating preview without download each files.

Hello and thank you for using our package.
Using S3 stored files without downloading them seems to be feasible by leveraging AWS Lambda which can execute Python code (thus use preview-generator). So if it is applicable for your use-case I'd suggest to go this way.
If not using AWS Lambda I'm afraid I do not know of a way to generate previews without downloading the files, either directly with the S3 API or with a Fuse filesystem to access S3 files as in a normal filesystem.
If you have any other questions/issues or ideas of improvement for preview-generator, don't hesitate to ask or propose them.

@grignards thanks for your answer.
unfortunately, I can't use AWS Lamba, our provider just can deliver S3 service.
do you have any idea about creating a preview with streaming? because I use multipart upload (chunks) and chunk size is 5MB and at the first, chunks save in buffer (ram) then upload them.

In its current state preview-generator cannot generate previews of streaming content as it relies on several packages/programs which do not necessarily provide a way to read streams.
I understand that uploaded files are going through your web application. If this is the case you could temporarily save the stream contents on disk, generate the preview then discard the temporary file.
Also do you buffer all the chunks in RAM? If yes a possibility would be to use a tmpfs as a store for both S3 upload and preview generation.

no, at the moment just get each chunk from the client web app then (after decoding from base64) pass to the s3 upload chunk.
i haven't all chunks together in ram.

In its current state preview-generator cannot generate previews of streaming content as it relies on several packages/programs which do not necessarily provide a way to read streams.
I understand that uploaded files are going through your web application. If this is the case you could temporarily save the stream contents on disk, generate the preview then discard the temporary file.
Also do you buffer all the chunks in RAM? If yes a possibility would be to use a tmpfs as a store for both S3 upload and preview generation.

@grignards  I had testd
ffmpeg -i http://<IP_ADDRESS>:9000/test/test.mp4 -ss 00:00:10 -vframes 1 -f image2 "image.jpg"

ffmped ,  it will work fine, as long as the url of this video can be accessed anonymously.
so, if we just support video streaming , not all files , are we possible to complete?

As ffmpeg supports reading over http using preview generator classically is possible:
from preview_generator.manager import PreviewManager
m = PreviewManager("/tmp/")
preview_path = m.get_jpeg_preview("https://test-videos.co.uk/vids/bigbuckbunny/mp4/h264/1080/Big_Buck_Bunny_1080_10s_30MB.mp4")

works just fine.
By default (page=-1) the preview will be done with the image at 2% of the video's duration so in worst case (if ffmpeg does not support skipping to frame) the downloaded part should be around 2% of the total video size.


How should I traverse the graph that is built using hidden layer

I would like to traverse the graph generated by hidden layer to investigate the model(any I choose) for architectural changes. How can I do that?

I am also trying to traverse the graph. Have you found any tools or approaches yet?


(BKR-40) Fix UTF8 buffers for SSH and SCP

Prior to this change, attempting to send UTF8 commands through
SSH, or attempting to copy files with UTF8 filenames could fail.
This was particularly easy to trigger by attempting to execute
commands that were 128 bytes or longer.

monkey patch net-ssh gem to allow UTF-8 strings >= 128 bytes
The buffer @content is often built as a UTF-8 string, until the
point at which it appends data that cannot be encoded as a UTF-8
sequence.
One case occurs when the call to write_string is made to append a
string that exceeds 127 bytes in length.  The SSH2 format says
that strings must be length prefixed, and when the value [128]
has pack("N*") called against it, the resultant 4 byte network
order representation does not have a valid UTF-8 equivalent,
resulting in an ASCII-8BIT / BINARY string.
[127].pack('N*').encode('utf-8')
=> "\u0000\u0000\u0000\u007F"
[128].pack('N*').encode('utf-8')
Encoding::UndefinedConversionError: "\x80" from ASCII-8BIT to UTF-8
Ruby has a subtle behavior where appending a BINARY string to
an existing UTF-8 string is allowed and the resultant string
changes encoding to BINARY.  However, once this has happened,
the string can no longer have UTF-8 encoded strings appended as
Ruby will raise an Encoding:CompatibilityError
Appending BINARY to UTF-8 always creates BINARY:
"foo".encode('utf-8') << [128].pack('N*')
=> "foo\x00\x00\x00\x80"
Appending UTF-8 representable strings to existing strings:
Ruby 2.1.7 keeps the string as its default UTF-8
"foo" << [127].pack('N*')
=> "foo\u0000\u0000\u0000\u007F"
Ruby 1.9.3 keeps UTF-8 strings as UTF-8
"foo".encode('utf-8') << [127].pack('N*')
=> "foo\u0000\u0000\u0000\u007F"
Ruby 1.9.3 defaults to US-ASCII which changes it to BINARY
pry(main)> "foo" << [127].pack('N*')
=> "foo\x00\x00\x00\x7F"
The simple solution is to call force_encoding on UTF-8 strings
prior to appending them to @content, given it's always OK to
append ASCII-8BIT / BINARY strings to existing strings, but
appending UTF-8 to BINARY raises errors.
"\x80".force_encoding('ASCII-8BIT') << "\u16A0"
Encoding::CompatibilityError: incompatible character encodings: ASCII-8BIT and UTF-8
force_encoding in this case, will simply translate a valid UTF-8
string to its BINARY equivalent
"\u16A0".force_encoding('BINARY')
=> "\xE1\x9A\xA0"
Correct conversion per http://www.fileformat.info/info/unicode/char/16a0/index.htm

Refer to this link for build results (access rights to CI server needed):
http://jenkins-beaker.delivery.puppetlabs.net//job/qe_beaker_btc-intn/2658/

@Iristyle it looks like the tests are failing yard doc generation

last comment still current

last comment still current

@kevpl / @tvpartytonight -
Updated with a silly escape of the @content in the comments as \@content.  Also added @private, even though we're not passing --no-private to YARD doc generation from what I can see.

Refer to this link for build results (access rights to CI server needed):
http://jenkins-beaker.delivery.puppetlabs.net//job/qe_beaker_btc-intn/2730/

FYI - I have opened 4 PRs against net-ssh itself to address this problem -

4.x alpha branch - https://github.com/net-ssh/net-ssh/pull/394
3.1 branch - https://github.com/net-ssh/net-ssh/pull/397
2.10 branch - https://github.com/net-ssh/net-ssh/pull/396
2.9 branch - https://github.com/net-ssh/net-ssh/pull/395

Refer to this link for build results (access rights to CI server needed):
http://jenkins-beaker.delivery.puppetlabs.net//job/qe_beaker_btc-intn/2735/

last comment is current

last comment is current.

last comment is current. If this hasn't moved forward in a week, we'll close this due to inactivity.

Refer to this link for build results (access rights to CI server needed):
http://jenkins-beaker.delivery.puppetlabs.net//job/qe_beaker_btc-intn/2948/

Still failing spec tests:

Failures:

  1) Net::SSH::Buffer can build up a buffer properly with a UTF8 string of 128 bytes or longer
     Failure/Error: expect { described_class.from(:string, ("\u16A0" * 128)) }.to_not raise_error

       expected no Exception, got #<TypeError: no implicit conversion of false into String> with backtrace:
         # ./lib/beaker/ssh_connection.rb:45:in `block in write'
         # ./lib/beaker/ssh_connection.rb:45:in `each'
         # ./lib/beaker/ssh_connection.rb:45:in `write'
         # ./vendor/bundle/ruby/2.2.0/gems/net-ssh-2.9.4/lib/net/ssh/buffer.rb:322:in `block in write_string'
         # ./vendor/bundle/ruby/2.2.0/gems/net-ssh-2.9.4/lib/net/ssh/buffer.rb:319:in `each'
         # ./vendor/bundle/ruby/2.2.0/gems/net-ssh-2.9.4/lib/net/ssh/buffer.rb:319:in `write_string'
         # ./vendor/bundle/ruby/2.2.0/gems/net-ssh-2.9.4/lib/net/ssh/buffer.rb:55:in `block in from'
         # ./vendor/bundle/ruby/2.2.0/gems/net-ssh-2.9.4/lib/net/ssh/buffer.rb:47:in `step'
         # ./vendor/bundle/ruby/2.2.0/gems/net-ssh-2.9.4/lib/net/ssh/buffer.rb:47:in `from'
         # ./spec/beaker/ssh_connection_spec.rb:14:in `block (3 levels) in <module:SSH>'
         # ./spec/beaker/ssh_connection_spec.rb:14:in `block (2 levels) in <module:SSH>'
     # ./spec/beaker/ssh_connection_spec.rb:14:in `block (2 levels) in <module:SSH>'

Finished in 24.16 seconds (files took 3.27 seconds to load)
1707 examples, 1 failure

last comment is still current

I haven't updated this yet, because my patch + some extra code landed in the upstream net-ssh project at https://github.com/net-ssh/net-ssh/pull/407 that would make this monkey patch unnecessary (which is preferred).
Unfortunately, the patch landed on the master branch of net-ssh, which only targets the 4.0 (currently alpha series).  I opened up another PR at https://github.com/net-ssh/net-ssh/pull/434 to backport those commits to the 3.x branch, so that Beaker should automatically get the updated gem when it's next published (assuming it includes that fix)

Of note that one way to trigger this bug easily is to modify the test at https://github.com/puppetlabs/puppet/blob/master/acceptance/tests/resource/scheduled_task/should_modify.rb and change:
name = "pl#{rand(999999).to_i}"

To include some UTF-8 characters in the name (or anywhere else in the string)
name = "plᚠᛇᚻ#{rand(999999).to_i}"

If the task has the correct name, then the bug has been fixed.  If it does not, the bug is still present.

@Iristyle if we're going to wait on determining if you're going to do work on this PR to make it ready based on the outcome of your https://github.com/net-ssh/net-ssh/pull/434 PR, is there a reason to keep this PR open in the meantime? It seems to me that that open PR should be enough to track this effort, and that keeping this open is redundant.
If there's no comment made about this in a week, I'll close this PR for now.


Feature Request - Smart Locks connected through Amazon Key - E.g. Schlage Encode WiFi

Hi
Thanks for the development on this addon, I'm loving the new features.
I'm wondering if wifi locks connected through amazon key are exposed by the script and whether these could be integrated into home assistant.
At the moment there is no way to get such locks into home assistant unlike zwave or zigbee locks.
I've attached a screenshot of the lock in my ring app and the log output showing discovered devices.
Thanks
Daniel

2021-03-01T05:47:27.462Z ring-mqtt Found new location Porirua with id
2021-03-01T05:47:30.080Z ring-mqtt   New device of type: hub.redsky
2021-03-01T05:47:30.085Z ring-mqtt   New device of type: security-panel
2021-03-01T05:47:30.088Z ring-mqtt   New device of type: sensor.contact
2021-03-01T05:47:30.091Z ring-mqtt   New device of type: security-keypad
2021-03-01T05:47:30.094Z ring-mqtt   New device of type: range-extender.zwave
2021-03-01T05:47:30.101Z ring-mqtt   New device of type: sensor.motion
2021-03-01T05:47:30.102Z ring-mqtt   New device of type: sensor.contact
2021-03-01T05:47:30.105Z ring-mqtt   New device of type: alarm.smoke
2021-03-01T05:47:30.108Z ring-mqtt   New device of type: alarm.co
2021-03-01T05:47:30.108Z ring-mqtt   New device of type: sensor.contact
2021-03-01T05:47:30.109Z ring-mqtt   New device of type: sensor.contact
2021-03-01T05:47:30.110Z ring-mqtt   New device of type: sensor.motion
2021-03-01T05:47:30.111Z ring-mqtt   New device of type: alarm.smoke
2021-03-01T05:47:30.111Z ring-mqtt   New device of type: alarm.co
2021-03-01T05:47:30.112Z ring-mqtt   New device of type: alarm.smoke
2021-03-01T05:47:30.113Z ring-mqtt   New device of type: alarm.co
2021-03-01T05:47:30.113Z ring-mqtt   New device of type: sensor.contact
2021-03-01T05:47:30.114Z ring-mqtt   New device of type: sensor.contact
2021-03-01T05:47:30.115Z ring-mqtt   New device of type: alarm.smoke
2021-03-01T05:47:30.115Z ring-mqtt   New device of type: alarm.co
2021-03-01T05:47:30.116Z ring-mqtt   New device of type: alarm.smoke
2021-03-01T05:47:30.117Z ring-mqtt   New device of type: alarm.co
2021-03-01T05:47:30.119Z ring-mqtt   New device of type: doorbell_v4
2021-03-01T05:47:30.120Z ring-mqtt   New device of type: hp_cam_v1
2021-03-01T05:47:30.120Z ring-mqtt   New device of type: hp_cam_v1
2021-03-01T05:47:30.121Z ring-mqtt   Unsupported device of type: access-code.vault
2021-03-01T05:47:30.122Z ring-mqtt   Unsupported device of type: access-code
2021-03-01T05:47:30.122Z ring-mqtt   Unsupported device of type: adapter.zigbee
2021-03-01T05:47:30.123Z ring-mqtt   Unsupported device of type: adapter.zwave
2021-03-01T05:47:30.123Z ring-mqtt --------------------------------------------------------------------------------
2021-03-01T05:47:30.123Z ring-mqtt Ring location/device data updated, sleeping for 5 seconds.
2021-03-01T05:47:31.081Z ring-mqtt Account has access to set volume on base station, enabling volume control`

Unfortunately these locks are not supported by ring-client-api and so can't be easily integrated into this project.  I know there was some research done by the maintainer of ring-client-api but it was determined that these Amazon Key locks are supported by the Ring app communicating directly with the Amazon Key APIs rather than through some backend server integration.  Because of this it was determined to impossible to add at this time to the ring-client-api project and which means it's also out of script for this project.  You can read this post for more details.
I'm not sure if anyone else has managed to get this to work other than some creative integrations via Alexa but but if it's ever supported by ring-client-api, or perhaps via some other API, then it could be considered.


Huge framerate jumps, jittery tracking

Hello! I've managed to get everything working for the most part using a Kinect for Azure which tracks my head.
When I play in the editor, things seem to run pretty smoothly, but when I try running in a new editor window, I seem to get some erratic behavior. The Eye Position jumps forwards and backwards over and over again and the framerate is all over the place. Here's a video. Any idea what might be causing this issue or how I could diagnose it? I've tested the Kinect project without the Off-Axis plugin and the framerate and tracking are steady. This happens rarely when I play in editor, but always if I play in a new editor window.
Any help would be greatly appreciated. Thank you!

Hi, sorry for the late reply - for some reason, github didn't send me a notification. Does the plugin work without Azure Kinect? If not, there's an issue with updating the eye position.
Afaik, the value in the top-left is the eye position and it doesn't change at all. I think you might have updated the wrong value in the blueprints/or code.

Hey, no worries. I appreciate you getting back to me. The plugin does not have any of the jitter issues without the Kinect. Which is the correct value to be updating in the blueprints? Thank you!


UserAgent is handled globally by WKWebview Engine

There is a breaking change coming  for<EMAIL_ADDRESS>which impacts how the UserAgent is handled.
Since this plugin depends on CDVUserAgentUtil it fails to build with the upcoming version of cordova-ios.

:tada: This issue has been resolved in version 4.2.0 :tada:
The release is available on:

GitHub release
npm package (@latest dist-tag)

Your semantic-release bot :package::rocket:


POI 5.0.0 release

useful for https://github.com/OpenRefine/OpenRefine/issues/2601
Release notes - https://www.apache.org/dyn/closer.lua/poi/release/RELEASE-NOTES.txt
Change log - https://poi.apache.org/changes.html#5.0.0

Thanks for the contribution. It would be useful to include a link to the release notes as well as a description of how much testing you've done.
Given that this is a major version upgrade for POI, we probably want to be relatively conservative in doing the upgrade and not rush to do it immediately before a little cooling off period to see how stable the release ends up being.

Thanks for the contribution. It would be useful to include a link to the release notes as well as a description of how much testing you've done.
Given that this is a major version upgrade for POI, we probably want to be relatively conservative in doing the upgrade and not rush to do it immediately before a little cooling off period to see how stable the release ends up being.

@pjfanning thank you so much for proactively proposing the update! And for your work on POI in general. We just need to decide how this should fit in our release schedule but this should definitely make it to the users eventually.

@pjfanning thank you so much for proactively proposing the update! And for your work on POI in general. We just need to decide how this should fit in our release schedule but this should definitely make it to the users eventually.

Looks like there's already one fix targeted for a 5.0.1 in April. We probably want to continue to monitor this.
https://poi.apache.org/changes.html#5.0.1

This fix looks important: https://github.com/apache/poi/pull/221
I don't know exactly which bug it fixes but by the looks of it it seems to be something we might rely on.
I propose we wait for 5.0.1. The update to that version should be proposed automatically by dependabot so we can close this PR. Or if @pjfanning wants to increment their commit count in OpenRefine, manual PRs are totally welcome too. Thank you again!

https://github.com/apache/poi/pull/221 affects Powerpoint tables

next POI release is likely to be v6.0.0


Unify group for deploy commands

At this point this is out of date. Closing in favor of https://github.com/Microsoft/vscode-azureappservice/pull/737


Included requirements files should be relative to parent, not cwd

. λ rm -rf tmp
. λ mkdir tmp
. λ cd tmp
./tmp λ printf 'mypy' > a.in
./tmp λ printf '-r a.in\nruff' > b.in
./tmp λ cd ..
. λ pip-compile --no-header --no-annotate tmp/b.in
WARNING: --strip-extras is becoming the default in version 8.0.0. To silence this warning, either use --strip-extras to opt into the new default or use --no-strip-extras to retain the existing behavior.
mypy==1.8.0
mypy-extensions==1.0.0
ruff==0.2.1
typing-extensions==4.9.0
. λ uv pip compile --no-header --no-annotate tmp/b.in
error: Error parsing included file in `tmp/b.in` at position 0
  Caused by: failed to open file `./a.in`
  Caused by: No such file or directory (os error 2)

I believe this is the same as https://github.com/astral-sh/uv/issues/1367.

Gonna look at it now.

Thanks! I'll close this as a duplicate

Do you know if pip has the same treatment for local packages, or is this just for requirements / constraints files?

Ahh, I see, it's just files (not directories): https://github.com/pypa/pip/issues/8765

Looks like pip is inconsistent, in that in the following case it is relative to cwd:
λ ls tmp
a.in                         pypyp-1.1.0-py3-none-any.whl
λ cat tmp/a.in
pypyp-1.1.0-py3-none-any.whl
λ pip install -r tmp/a.in
WARNING: Requirement 'pypyp-1.1.0-py3-none-any.whl' looks like a filename, but the file does not exist
Processing ./pypyp-1.1.0-py3-none-any.whl (from -r tmp/a.in (line 1))
ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: '/Users/shantanu/devoai/openai/pypyp-1.1.0-py3-none-any.whl'
λ cd tmp
./tmp master λ pip install -r a.in
Processing ./pypyp-1.1.0-py3-none-any.whl (from -r a.in (line 1))
pypyp is already installed with the same version as the provided wheel. Use --force-reinstall to force an installation of the wheel.

I don't care about^ case at all

Okay yeah, makes sense.


Saved search - Default name is not displayed correctly for multiline system message thread

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Version Number: 9.0.36-0
Reproducible in staging?: Y
Reproducible in production?: Y
Email or phone of affected tester (no customers<EMAIL_ADDRESS>Issue reported by: Applause Internal Team
Action Performed:

Go to staging.new.expensify.com
Go to DM.
Submit an expense with a multiline description.
Go to transaction thread.
Click Description.
Edit the description while maintaining the multiline format and save it.
Right click on the system message > Reply in thread.
Send a reply in thread.
Go to Search > Chats.
Click Filters > Select "In".
Select the thread in Step 8 > Save.
Click Save search.

Expected Result:
The title of the saved search should only have one line.
Actual Result:
The title of the saved search has multiple lines and it is not displayed correctly.
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[ ] Android: Native
[x] Android: mWeb Chrome
[ ] iOS: Native
[ ] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[x] MacOS: Desktop

Screenshots/Videos
https://github.com/user-attachments/assets/ada2fbd2-fe7c-442a-a7f1-080bbdbbaa1b

View all open jobs on GitHub

We think that this bug might be related to #wave-control

@abekkala  FYI I haven't added the External label as I wasn't 100% sure about this issue. Please take a look and add the label if you agree it's a bug and can be handled by external contributors

Proposal
Please re-state the problem that we are trying to solve in this issue.
Saved search - Default name is not displayed correctly for multiline system message thread
What is the root cause of that problem?
The title text is not wrapped correctly.
What changes do you think we should make in order to solve the problem?
Add titleStyle: styles.textNoWrap, here
https://github.com/Expensify/App/blob/14b99ca0a12e9686818bc3e937f091199de69750/src/pages/Search/SearchTypeMenu.tsx#L110

Proposal
Please re-state the problem that we are trying to solve in this issue.
Saved search - Default name is not displayed correctly for multiline system message thread
What is the root cause of that problem?
The title text is not wrapped correctly.
What changes do you think we should make in order to solve the problem?
add numberOfLinesTitle: 2, in
https://github.com/Expensify/App/blob/14b99ca0a12e9686818bc3e937f091199de69750/src/pages/Search/SearchTypeMenu.tsx#L108-L117
What alternative solutions did you explore? (Optional)
add noWrap style
titleStyle: styles.textNoWrap,

Proposal
Please re-state the problem that we are trying to solve in this issue.

The title of the saved search has multiple lines and it is not displayed correctly.

What is the root cause of that problem?

We return the raw report name in this:
https://github.com/Expensify/App/blob/7bd9e53edfd2508dd4584e1db97fbf7a69f8a646/src/libs/SearchUtils.ts#L715

so the saved search name can contain multiple lines.
What changes do you think we should make in order to solve the problem?

We can fix the issue by fixing the Text style, but with that approach, we need to apply the fix on all platforms. So I suggest that we can format this:

https://github.com/Expensify/App/blob/7bd9e53edfd2508dd4584e1db97fbf7a69f8a646/src/libs/SearchUtils.ts#L715
so it can be:
        return ReportUtils.formatReportLastMessageText(ReportUtils.getReportName(reports?.[`${ONYXKEYS.COLLECTION.REPORT}${filter}`]));

like we did in this:
https://github.com/Expensify/App/blob/7bd9e53edfd2508dd4584e1db97fbf7a69f8a646/src/components/DisplayNames/DisplayNamesWithTooltip.tsx#L59
What alternative solutions did you explore? (Optional)

@paultsimura What do you think about my solution, which just needs to use ReportUtils.formatReportLastMessageText without any style changes? Thank you.

@paultsimura Please note we need to fix the search header page with that solution as well:

It should only be 1 line and then truncated but why is it not occupying more horizontal space? There's a lot more space over on the right that it could be using.

Ah nice, that's much better

PR merged

PAYMENT SUMMARY FOR OCT 08

Fix: @Krishna2323 [$250, if no regressions] OFFER
PR Review: @paultsimura [$250, if no regressions] OFFER
Please complete checklist below

BugZero Checklist: The PR fixing this issue has been merged! The following checklist (instructions) will need to be completed before the issue can be closed:

[ ]  [@paultsimura] The PR that introduced the bug has been identified. Link to the PR:
[ ]  [@paultsimura] The offending PR has been commented on, pointing out the bug it caused and why, so the author and reviewers can learn from the mistake. Link to comment:
[ ]  [@paultsimura] A discussion in #expensify-bugs has been started about whether any other steps should be taken (e.g. updating the PR review checklist) in order to catch this type of bug sooner.
[ ]  [@paultsimura] Determine if we should create a regression test for this bug.
[ ]  [@paultsimura] If we decide to create a regression test for the bug, please propose the regression test steps to ensure the same bug will not reach production again.
[ ]  [@abekkala] Link the GH issue for creating/updating the regression test once above steps have been agreed upon:

@Krishna2323 @paultsimura payments sent and contracts ended - thank you! 🎉


Avoid versioning npm locks

Thanks!

Why the locks were removed? The pakage-lock.json should always be committed. More information.
Quoting:

It is highly recommended you commit the generated package lock to source control: this will allow anyone else on your team, your deployments, your CI/continuous integration, and anyone else who runs npm install in your package source to get the exact same dependency tree that you were developing on.

I was looking, all the lockfiles, even for other package managers are been removed. You are experiencing any issues with the lockfiles?

Hi @thiamsantos,
lock files are removed for this project for node but also for crystal and ruby.
More, I'd say that lock files are recommended for production-ready apps or library, which is not the case here.
Since this tool is creating environments on-the-fly, keeping lock files are irrelevant since the environment and host could differs (language version, for example).
Regards,


Task Quick Pick

https://dev.azure.com/monacotools/Monaco/_build/results?buildId=178883&view=logs&j=4801dce2-64f3-53d6-b366-d49a1977c639&t=525606ab-8bc1-59bd-2d29-e98f94fa71fe&l=109
1) VSCode Smoke Tests (Electron)
       Task
         Task Quick Pick
           "after each" hook for "hide property - undefined":
     Error: Timeout: get elements '#terminal .terminal-wrapper' after 20 seconds.
      at Code.poll (D:\a\_work\1\s\test\automation\src\code.ts:256:11)
      at Code.waitForElements (D:\a\_work\1\s\test\automation\src\code.ts:205:10)
      at Terminal.runCommand (D:\a\_work\1\s\test\automation\src\terminal.ts:96:5)
      at Context.<anonymous> (src\areas\task\task-quick-pick.test.ts:23:4)

Flaking on this which is a little strange, kill all isn't killing all?
https://github.com/microsoft/vscode/blob/0cf530cd3cc29d1695dd60a67082e413698990f1/test/automation/src/terminal.ts#L95-L97
No trace seemed to have got uploaded.


Does the client need clear sight to the domain controllers?

I have followed the guide, and it works well on my AADJ client as long as they have connectivity to the domain controllers. But when accessing RDS servers through the Remote Desktop Gateway from the internet I get a message that NLA cannot be performed because the domain controllers cannot be contacted (when I try with username/password it works). If I then connect a VPN client it works with cert / PIN code. Is this behavior by design, or is there something wrong in my RDS Gateway setup? I was hoping to get passwordless login to RemoteApp from anywhere. It can of course be worked around with "Always-on-VPN".

Thanks @janeide75 for taking the time to share this with the Windows documentation team. I’m sorry that you’re having problems, and I want to make sure it gets to the right people that can help. There are a couple of options available to provide the best level of support for this issue:

Post to Windows on Q&A. These are moderated by community members who are very experienced, knowledgeable, and helpful.
Use Microsoft Support to search for the issue, or open a case with professional support.
Contact your Microsoft Unified Support team, such as a Customer Engineer or Customer Success Account Manager.

I'm closing this issue as there's nothing actionable for docs at this time. Also, I don't think we have any content under this Windows client repo for Remote Desktop Gateway. If you have more information about which specific article, that might help us determine if there's any documentation revisions necessary.


split dns with systemd-resolved + tailscale stop working after a deploy

Describe the bug
I have a tunnel to work using tailscale.
I'm using systemd-resolved to resolve work-related domains over the tailscale tunnel.
When I redeploy with nixos-rebuild switch my work-related domains doesn't resolve anymore.
I'm thinking maybe tailscale replaces my dns settings.
I think I tried tailscale up -accept-dns false without success.
EDIT: my "main dns config" seems to still work since https://www.cloudflare.com/ssl/encrypted-sni/ tells me that I'm still using dnssec with tls.
To Reproduce
Steps to reproduce the behavior:

...
...
...

Expected behavior
Screenshots
Additional context
Notify maintainers

@danderson @mbaillie
Metadata
nix-pre-commit-hooks: hooks up to date

system: "x86_64-linux"
host os: Linux 5.7.10-zen1, NixOS, 21.03pre-git (Okapi)
multi-user?: yes
sandbox: yes
version: nix-env (Nix) 2.3.8
channels(root): "nixos-21.03pre251123.b839d4a8557"
channels(bbigras): "home-manager"
nixpkgs: /run/current-system/nixpkgs

Maintainer information:
# a list of nixpkgs attributes affected by the problem
attribute:
# a list of nixos modules affected by the problem
module:

Can you extend the description with some more bits of your configuration, so this can be reproduced?

I think this has to do with a race condition where Tailscale starts after systemd-resolved, which makes Tailscale DNS unreachable by the time systemd-resolved wanted to use it as default, but instead fall back to whatever DNS your system may have. In your case, however, there might have been none.
A solution would be to just restart systemd-resolved whenever Tailscale starts, but that seems hacky. After all, it should have worked out-of-the-box, but with more complex usage like I have, maybe using that hack is reasonable enough. See this for more info.

@danderson could you take a look into this?
I'm not entirely sure what tailscale is doing to configure itself in resolved setups (and to ensure it stays configured the way it is, both after networkd restarts and tailscale restarts).
If it's only editing /etc/hosts, it won't work, as resolved brings its own NSS module.

This was likely fixed in either 1.22 or 1.24: we used to not detect systemd-resolved restarting (which clears all its DNS config), but some months ago we added the logic to monitor dbus and reprogram resolved after a restart. Anecdotally, my NixOS desktops and servers don't have any Tailscale DNS issues across deploys at this point.
That fix also addressed a more subtle race condition during bootup, where tailscaled could sometimes start up prior to resolved being registered on the system bus. When that happened, tailscaled would switch to direct /etc/resolv.conf management, which broke a bunch of things. The change to monitor the bus for resolved also fixed that, by having tailscaled figure out that resolved should be in charge, and waiting for it to appear on the system bus.
IOW, unless this is still reproducible, I think this can be closed as fixed.

And just for the record: unstable has tailscale 1.24.0, 21.11 has 1.22.2. Both of those have the resolved management bugfix, so all currently supported versions of NixOS should be happy.

Thanks for confirming. Let's close this then, we can still reopen if it still happens to be an issue.


Does not generate preview on FB

As title says...
Why that issue??

-> Question is still pendent, but maby it's too complex to find a solution for that issue here?
Thank you very much in advance for your feedbacks.
With kind regards,
Jan

-> https://github.com/Athlon1600/php-proxy-app/issues/199

Problem solved by not-using your shitware...

-> https://www.php.de/forum/webentwicklung/php-einsteiger/1587913-warum-kann-dieses-reverse-proxy-php-skript-keine-vorschau-bei-fb-posts-generieren

So don't use it - never ever: https://www.php.de/forum/webentwicklung/php-einsteiger/1587913-warum-kann-dieses-reverse-proxy-php-skript-keine-vorschau-bei-fb-posts-generieren/page2
With kind regards,
Jan

...


how to build 1.0.1?

 Authored by kalsolio
Oct 25, 2017

how to build bin files.

 Authored by jandrassy
Aug 28, 2017

This repository is WiFi Link. The instructions are for the factory firmware. Two very different firmwares.

community documentation

 Authored by jandrassy
Aug 28, 2017

This repository is WiFi Link. The instructions are for the factory firmware. Two very different firmwares.

community documentation


Have gx-go rw install missing packages

Developers may want to change their dependencies (e.g. in the package.json file) and when they perform a gx-go rw will see their imports rewritten. However, unless they also run gx install their code will not be functional. It may be worth have gx-go rw call gx install.

I think this would be better solved by a shell alias/script that you define yourself, that does a gx install before gx-go rw, instead changing the way gx-go rw works internally.


Read/configuration signals in StandardReadable don't have to be signals

When making a StandardReadable you can provide a set of signals to be read by default e.g.
class Sensor(StandardReadable):
    """A demo sensor that produces a scalar value based on X and Y Movers"""

    def __init__(self, prefix: str, name="") -> None:
        # Define some signals
        self.value = epics_signal_r(float, prefix + "Value")
        self.mode = epics_signal_rw(EnergyMode, prefix + "Mode")
        # Set name and signals for read() and read_configuration()
        self.set_readable_signals(
            read=[self.value],
            config=[self.mode],
        )
        super().__init__(name=name)

Now the value signal will be read and the output included in sensor.read() and ditto for the mode signal in sensor.read_configuration(). set_readable_signals takes a list of SignalRs.
However, if you make sub-devices and pass them into set_readable_signals, it works fine, the output of sub_device.read() is included in sensor.read() as well. The only difference is that mypy complains because the subdevices are not signals. Example:
class DoubleSensor(StandardReadable):
    def __init__(self, prefix: str, name: str = "", sensor_count: int = 3) -> None:
        self.sensor_a = Sensor(prefix + "SENSOR_A:")
        self.sensor_b = Sensor(prefix + "SENSOR_B:")

        # Makes read() produce the values of all sensors
        self.set_readable_signals(
            read=[self.sensor_a, self.sensor_b],  # Mypy complains here!
        )
        super().__init__(name)

# Functionally equivalent to the class above
class DoubleSensor(StandardReadable):
    def __init__(self, prefix: str, name: str = "", sensor_count: int = 3) -> None:
        self.sensor_a = Sensor(prefix + "SENSOR_A:")
        self.sensor_b = Sensor(prefix + "SENSOR_B:")

        # Makes read() produce the values of all sensors
        self.set_readable_signals(
            read=[self.sensor_a.value, self.sensor_b.value],  # Mypy is happy here, but it is extra boilerplate and requires knowledge of the sensor class; you have to know that .value is a read signal on the sensor.
        )
        super().__init__(name)

We do not want this behaviour to continue because allowing something potentially useful at runtime that mypy disallows will just lead to # type: ignores all over the place. There are two options:

Check the arguments of set_readable_signals at runtime and raise if they are not signals
Change the type annotation of set_readable_signals

From discussion with @coretl, 1. is easier. 2. is more complete but requires work. We do not currently have a suitable type for the type signature of set_readable_signals, it can technically take any device with async def read(...): but there is not protocol for that. We would have to write and maintain one. Another option would be to take a union of SingalR and StandardReadable which would get us most of the way.
We have decided not to do 2. unless concrete use cases can be identified. Does anybody have a use case for trees of nested devices that need to be auto-read?

I think the current behaviour is fine. My only issues with your example is what do to if someone adds a new signal to Sensor that they want to be read. My solution would instead be to have:
class DoubleSensor(StandardReadable):
    def __init__(self, prefix: str, name: str = "", sensor_count: int = 3) -> None:
        self.sensor_a = Sensor(prefix + "SENSOR_A:")
        self.sensor_b = Sensor(prefix + "SENSOR_B:")

        # Makes read() produce the values of all sensors
        self.set_readable_signals(
            read=[*self.sensor_a._read_signals, *self.sensor_b._read_signals],
        )
        super().__init__(name)

Then it just always picks up the read_signals of the children and means we're not doing anything particularly magic.

@DominicOram I think that's a good solution actually, although I'd want read_signals to become a public property

@DominicOram I think that's a good solution actually, although I'd want read_signals to become a public property

Credit to @dperl-dls actually. Yes, I agree it should become public for this. We would probably want the equivalent for config too?

@callumforrester sold me on the idea of supporting StandardDetector as well as SignalR here, and now it looks like we're not doing that?

We should also consider how we would do #44 here. Especially how we mark signals to be read as hinted...

I thought parent.read() already called child.read()?

Worth considering also PVI case then, may need to re-calculate readable_signals after connect

Ok, after discussion with @callumforrester I think the usecase is compound devices like a set of slits. We want a way to opt into the behaviour that we got with ophyd where we aggregate all the signals our children have chosen to read. With that in mind I propose that we make self.set_readable take things that we can call read, describe, read_configuration, describe_configuration and hints on, and provide some utility functions like config_signals or uncached_signals or hinted_signals to wrap with the right verbs:
class Motor(StandardReadable):
    def __init__(self, prefix, name=None):
        self.setpoint = epics_signal_rw(prefix + ".VAL")
        self.readback = epics_signal_rw(prefix + ".RBV")
        self.units = epics_signal_rw(prefix + ".EGU")
        self.velocity = epics_signal_rw(prefix + ".VELO")
        self.set_readable(
            *config_signals(self.units, self.velocity),
            *hinted_signals(self.readback)
        )
        super().__init__(name)

class Slits(StandardReadable):
    def __init__(self, prefix, name=None):
        self.x_gap = Motor(prefix + ":X:GAP")
        self.y_gap = Motor(prefix + ":Y:GAP")
        self.x_centre = Motor(prefix + ":X:CENTRE")
        self.y_centre = Motor(prefix + ":Y:CENTRE")
        self.set_readable(*self.children())
        super().__init__(name)

Could even use the power of the walrus...
class Motor(StandardReadable):
    def __init__(self, prefix, name=None):
        self.setpoint = epics_signal_rw(prefix + ".VAL")
        self.set_readable(
            *config_signals(
                self.units := epics_signal_rw(prefix + ".EGU"),
                self.velocity := epics_signal_rw(prefix + ".VELO"),
            ),
            *hinted_signals(
                self.readback := epics_signal_rw(prefix + ".RBV"),
            ),
        )
        super().__init__(name)

class Slits(StandardReadable):
    def __init__(self, prefix, name=None):
        self.set_readable(
            self.x_gap := Motor(prefix + ":X:GAP"),
            self.y_gap := Motor(prefix + ":Y:GAP"),
            self.x_centre := Motor(prefix + ":X:CENTRE"),
            self.y_centre := Motor(prefix + ":Y:CENTRE"),
        )
        super().__init__(name)

For now, I think we will just make the read signals public: https://github.com/bluesky/ophyd-async/issues/196

Last idea before I forget:
class Motor(StandardReadable):
    def __init__(self, prefix, name=None):
        self.setpoint = epics_signal_rw(prefix + ".VAL")
        with self.add_readables(ConfigSignal):
            self.units = epics_signal_rw(prefix + ".EGU")
            self.velocity = epics_signal_rw(prefix + ".VELO")
        with self.add_readables(HintedSignal):
            self.readback = epics_signal_rw(prefix + ".RBV")
        super().__init__(name)

class Slits(StandardReadable):
    def __init__(self, prefix, name=None):
        with self.add_readables():
            self.x_gap = Motor(prefix + ":X:GAP")
            self.y_gap = Motor(prefix + ":Y:GAP")
            self.x_centre = Motor(prefix + ":X:CENTRE")
            self.y_centre = Motor(prefix + ":Y:CENTRE")
        super().__init__(name)

Show of hands, should we:

:tada: use with self.add_readables(SomeWrapperClass): as per https://github.com/bluesky/ophyd-async/issues/192#issuecomment-2051090558
:eyes: use self.set_readable(*some_wrapper_function) as per https://github.com/bluesky/ophyd-async/issues/192#issuecomment-2049796039
:rocket: use self.set_readable_signals(read=..., config=...) with public read_signals and config_signals as per https://github.com/bluesky/ophyd-async/issues/192#issue-2237360654
Please vote with the emojis below

I think 🚀  is the simplest thing to do to solve the original issue of "I want to easily pass Devices to set readables". I'm not against 🎉 but it seems to be a complete re-write of how we set signals that goes beyond the original issue

Would it be feasible to add a method that returns lists of readable/config signals in a device that can then be used instead of the device itself? That way this functionality can still fairly easily exist without needing significant refactoring to satisfy mypy.
For the record I can think of cases where something like this would be desirable. For example a device with 4 duplicate channels that offer several data collecting signals each. I'd make a class that represents each channel and the overall device would instantiate 4 - and then I'd like to be able to easily get all relevant signals.

The only problem with :rocket: is that we still need to find a solution that satisfies #44 . This means we will either need to write a def hints() function that pulls together the hints of the child functions, or change the underlying datastructures to be dictionaries instead. For completeness here are the 2 examples:
:rocket:
class Motor(StandardReadable):
    def __init__(self, prefix, name=None):
        self.setpoint = epics_signal_rw(prefix + ".VAL")
        self.units = epics_signal_rw(prefix + ".EGU")
        self.velocity = epics_signal_rw(prefix + ".VELO")
        self.readback = epics_signal_rw(prefix + ".RBV")
        self.set_readable_signals(
            read=[self.readback],
            config=[self.units, self.velocity],
        )
        super().__init__(name)

    @property
    def hints(self):
        return {"fields": [self.readback.name]}

class Slits(StandardReadable):
    def __init__(self, prefix, name=None):
        self.x_gap = Motor(prefix + ":X:GAP")
        self.y_gap = Motor(prefix + ":Y:GAP")
        self.x_centre = Motor(prefix + ":X:CENTRE")
        self.y_centre = Motor(prefix + ":Y:CENTRE")
        self.set_readable_signals(
            read=[*self.x_gap.read_signals, *self.y_gap.read_signals, *self.x_centre.read_signals, *self.y_centre.read_signals],
            config=[*self.x_gap.config_signals, *self.y_gap.config_signals, *self.x_centre.config_signals, *self.y_centre.config_signals],
        )
        super().__init__(name)

    @property
    def hints(self):
        return {**self.x_gap.hints, **self.y_gap.hints, **self.x_centre.hints, **self.y_centre.hints}

:tada:
class Motor(StandardReadable):
    def __init__(self, prefix, name=None):
        self.setpoint = epics_signal_rw(prefix + ".VAL")
        with self.add_readables(ConfigSignal):
            self.units = epics_signal_rw(prefix + ".EGU")
            self.velocity = epics_signal_rw(prefix + ".VELO")
        with self.add_readables(HintedSignal):
            self.readback = epics_signal_rw(prefix + ".RBV")
        super().__init__(name)

class Slits(StandardReadable):
    def __init__(self, prefix, name=None):
        with self.add_readables():
            self.x_gap = Motor(prefix + ":X:GAP")
            self.y_gap = Motor(prefix + ":Y:GAP")
            self.x_centre = Motor(prefix + ":X:CENTRE")
            self.y_centre = Motor(prefix + ":Y:CENTRE")
        super().__init__(name)

Personally I think :tada: is worth the extra effort now, if it satisfies all our use cases. The edge conditions of "I want to pick and choose between signals of child devices rather than reading the whole device" could be added with some additional functions on the self.add_readables() context manager...
Thoughts?

I'm happy with :tada: being done but maybe for clarity of what we're doing we could make a new issue that specifically says what we're doing and why as it's getting tangential to this?

Closed in favour of #223


DOC-6405 standardize sys priv language - Rich's PRs

Addresses: DOC-6405

Standardized language around v22.2's RBAC feature: system-level privileges.

This is just for Rich's PRs -- those organically his and those inherited from others (See https://github.com/cockroachdb/docs/pull/15826 for Kathryn's side of things). Essentially:

MODIFYCLUSTERSETTING
GRANT
REVOKE
DROP OWNED BY

Standardized linking to new central resource.

Staging (selected):
v22.2.md | drop-owned-by.md | grant.md | revoke.md | set-cluster-setting.md

Hi @rmloveland  - here's a language update for your PRs that deal with v22.2's new RBAC feature to standardize on language from my https://github.com/cockroachdb/docs/pull/15718 (Rafi-approved!). Please let me know if you'd like to see any changes, or feel free to ping Rafi directly on any point you think needs engineering once-over. Thanks so much!!

Thanks Rich!


Typing annotation incompatible with python 3.8 in v0.1.8

The typing annotation in contrib/dadapt_adamw.py --> betas: tuple[float, float] = (0.9, 0.999) is incompatible with python 3.8, giving a type error: TypeError: 'type' object is not subscriptable.
However, python 3.8 is accepted if you refer to the pyproject.toml file.
Is the pyproject.toml file wrong and should only accept python 3.9 and above (as Jax does) or should the typing be adapted to be compatible with python 3.8?

Yes, this fails for me also.

Thanks for catching this! #713 will fix that. Also we shall drop support of Python 3.8.
JAX requires also python 3.9 or higher https://github.com/google/jax/blob/main/setup.py.

closing since we now dropped support for 3.8

Apologies if this is the wrong thread, but is there any documentation on using optax on cloud TPUs? TPUs still ship with 3.8 IIRC.

seems to me colab on TPUs uses 3.10

other than that, we support the same versions as JAX, so I would guess that existing guides for running JAX on cloud TPUs should apply to optax? https://cloud.google.com/tpu/docs/run-calculation-jax

Apologies if this is the wrong thread, but is there any documentation on using optax on cloud TPUs? TPUs still ship with 3.8 IIRC.

Seconded. I just ran into this issue because the default python version is 3.8 (I followed this tutorial: https://cloud.google.com/tpu/docs/tutorials/LLM/single-device-pax)

thanks for the report, I've filled in a bug for that webpage asking them to upgrade the python version


LimeSDR Mini RPI3,  LimeUtil --info,  LimeUtil --find

Using latest master LimeSuite.
Only the PCIEXillybus connection is  available.
I need FT601 for the LimeMini - or not ?
Bug or is there a cmake switch ?

LimeUtil --find  -> no device detected

LimeUtil --info
######################################################
LimeSuite information summary
######################################################
Version information:
Library version:	v18.06.0-gf1e54444
Build timestamp:	2018-06-11
Interface version:	v2018.6.0
Binary interface:	18.06-1
System resources:
Installation root:	/usr/local
User home directory:	/home/pi
App data directory:	/home/pi/.local/share/LimeSuite
Config directory:	/home/pi/.limesuite
Image search paths:
- /home/pi/.local/share/LimeSuite/images
- /usr/local/share/LimeSuite/images
Supported connections:

PCIEXillybus

Did you install the udev rules?

Yes i did:
cd LimeSuite/udev-rules
chmod +x *sh
sudo ./install.sh
lsusb
Bus 001 Device 014: ID 0403:601f Future Technology Devices International, Ltd
Info in /etc/udev/rules.d/64-limesuite.rules:
SUBSYSTEM=="usb", ATTR{idVendor}=="0403", ATTR{idProduct}=="601f", SYMLINK+="stream-%k", TAG+="uaccess"
This is my Hardware:
LimeSDR Mini v1.1
Firmware: 5
Hardware: 0
Protocol: 1
Gateway: 1.24
Gateway rev.: 1,24
Gateway Target: LimSDR-Mini
The LimeSDR-Mini works on Windows and Ubuntu 16.04
TNX

Could this the problem:
On the RPI3 (from latest repo) limeutil --info shows:
Binary interface:	18.06-1
On the Ubuntu 16.04  limeutil --info shows:
Binary interface:	18.04-1
The last update for LimeSDR-Mini on Ubuntu was:
LimeUtil --update
Connected to [LimeSDR Mini [USB 2.0] 1D3AD534B68430]
[100%] 303104/303104 Bytes programming: completed (/usr/share/LimeSuite/images/18.04/LimeSDR-Mini_HW_1.1_r1.24.rpd)
Programming update complete!
TNX

I found the solution :-)
Raspberry PI3:
sudo apt-get install libusb-1.0-0-dev
LimeUtil --info
######################################################
LimeSuite information summary
######################################################
Version information:
Library version:	v18.06.0-gf1e54444
Build timestamp:	2018-06-12
Interface version:	v2018.6.0
Binary interface:	18.06-1
System resources:
Installation root:	/usr/local
User home directory:	/home/pi
App data directory:	/home/pi/.local/share/LimeSuite
Config directory:	/home/pi/.limesuite
Image search paths:
- /home/pi/.local/share/LimeSuite/images
- /usr/local/share/LimeSuite/images
Supported connections:

FT601
FX3
PCIEXillybus

LimeUtil --find

[LimeSDR Mini, media=USB 2.0, module=FT601, addr=24607:1027, serial=1D3AD534B68430]
[LimeSDR-USB, media=USB 2.0, module=FX3, addr=1d50:6108, serial=0009062000C43419]

i hope this help other folks.
TNX

Can i use RTL-SDR
[ 8790.700908] rtl2832_sdr rtl2832_sdr.0.auto: Registered as swradio0
[ 8790.700921] rtl2832_sdr rtl2832_sdr.0.auto: Realtek RTL2832 SDR attached
[ 8790.700932] rtl2832_sdr rtl2832_sdr.0.auto: SDR API is still slightly experimental and functionality changes may follow
[ 8790.712909] Registered IR keymap rc-astrometa-t2hybrid
[ 8790.725165] IR NEC protocol handler initialized
In place of LimeSDR?
I am trying, but LimeUtil is unable to detect it.
Thank you.

@ShitalBorganve No. Lime Suite only supports LimeSDR/LimeNET hardware.

could you please suggest what other software stack i can use with RTL2832 SDR/HackRF one to achieve this?

Achieve what? This is an issue on the repo for a driver for a totally different SDR. Not an application.
I'd suggest asking in the RTL and HackRF communities.

okay


Finalize the HPWH Template

Work with Edward Louie to make any remaining needed changes to the HPWH template.
I know Edward would like more add to the report.
For reference, here is a PDF HPWH report that Edward generated using different software. This is what I used at my starting point when I wrote the HPWH template.
Heat Pump Water Heater Installation QA Final Filled Example.pdf

This is not the final  version. There will be more iterations with Edward.


[BUG] Unable to get tiktok session ID?

Cant get tiktok session ID so how can I use this without it?

Not a bug; reopen with correct slug.

Slug?

Sorry - Issue Title.


Center Suites Display

On the landing page (/suites page for production), adjusted the suites display to center within the available space leftover from adding in a banner.

Codecov Report

Merging #166 (acd948e) into main (cbcfae8) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #166   +/-   ##
=======================================
  Coverage   93.84%   93.84%
=======================================
  Files         110      110
  Lines        2780     2780
=======================================
  Hits         2609     2609
  Misses        171      171

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cbcfae8...acd948e. Read the comment docs.

How this change looks on dev.
Before:

After:


Disable DIS publishing of all CMSSWTask

All CMSSWTasks will publish to DIS, even though they might not be CMS4 jobs. We don't want to confuse querying later on down the line, since the only thing that distinguishes the output will be the "cms3tag" parameter. Put in a parameter that disables publishing for the CMSSWTask object.

Maybe set a kwarg in CMSSWTask (defaulting to False) to handle publishing. Then if user decides to make central samples, they can toggle it to True
publish_to_dis = False,

Above action taken. Solved in https://github.com/aminnj/ProjectMetis/commit/157a60966a61c0bf23c8290c17694965e950add0.


Editor lag with big canvas polygon buffer

Godot version:
v3.3.rc8
OS/device including version:
Macbook Pro 2019 (details)
Issue description:
The whole Godot editor is extremely laggy and unresponsive on version v3.3 RC but works fine on v3.2.3 (stable)
Steps to reproduce:
Create new project
Change rendering/limits/buffers/canvas_polygon_buffer_size_kb
and rendering/limits/buffers/canvas_polygon_index_buffer_size_kb
to something big like 32*1024
Minimal reproduction project:
GodotIssue.zip

By 3.2 (stable) do you mean 3.2.3?

Yes, I did mean 3.2.3 :)

FWIW, I'm unable to reproduce this with rc8 on Arch Linux. Tried both with GLES2 and GLES3. @kristijandraca which of these are you using and does it make a difference?

Originally I noticed on GLES3 but I tried now with GLES2 and its the same thing.
Here is quick demo of Main scene from reproduction project running 3.3 RC8
https://user-images.githubusercontent.com/53337160/114444130-cc6a0300-9bce-11eb-9645-5c1e17bfc695.mov
And here is same project running on 3.2.3
https://user-images.githubusercontent.com/53337160/114444194-ddb30f80-9bce-11eb-9077-8ecee2cdd23c.mov

Unfortunately we've had all kinds of problems with performance on various Macs in different builds for a long time, and made many attempts to pin them down in conjunction with users, but realistically it is complete guesswork as none of the render guys seem to use / have access to Macs, and there are too many variables. We would welcome interest to investigate this from any contributor that does use Macs.
It could be due to the Mac drivers not liking some way we use OpenGL, or frame rate limiting in the editor, or any number of reasons.
You could try switching off batching to see if you get any better performance. Batching doesn't use the canvas_polygon buffers as far as I can remember.
One long term option may be to use ANGLE to translate OpenGL into Metal. We haven't used it so far because apparently it is a pain to compile. Also in Godot 4.0 there may be MoltenVK to convert Vulkan to Metal. Apple support for Metal is much better than OpenGL.

I tried switching off
rendering/batching/options/use_batching_in_editor
rendering/batching/options/use_batching
restart Godot and still problem persist.

You may have some joy trying out 3.2.4 beta 2:
https://github.com/godotengine/godot/issues/43038#issuecomment-722336615
We had these extra settings for buffer orphaning available for some of the betas, but unfortunately I don't think many people tried them.

Ok, so switching off
rendering->options->api_usage_legacy->orphan_buffers
solves the issue for me

Ok, so switching off
rendering->options->api_usage_legacy->orphan_buffers
solves the issue for me
on 3.2.4 beta 2

Yes as I suspected the orphaning / flags are having effects on machines with dodgy drivers. See here for another example:
https://github.com/godotengine/godot/issues/47824#issuecomment-818184804
This is somewhat of a difficult problem, in that OpenGL isn't absolutely clear about the best way to dynamically use buffers. As such it seems that different vendors / driver versions have their own interpretation, and what works best on some hardware may not be what works best on others. I'm not sure whether to blame OpenGL or the driver guys, but it is a difficult problem to solve on our side.
Ideally we'd like to have a certain 'correct' usage and apply it to all platforms. We can certainly have a distinction between mobile and desktop, with mobile seeming to work better with orphaning. My desktop doesn't make a lot of difference if orphaning is used or not, and that seems the case with a lot of desktop. However macs seem to drop a lot of performance when their preferred method is not used, and this seems to vary on different Macs, possibly even between OS versions.
The problem is not so much using the editor, as it is feasible (but not ideal) to have a developer change these options, but the end users, game players, who will not want to change these options and will expect it to work.
So maybe we will have to work towards a hardware database as @Calinou has suggested in the past.
To start with it maybe an idea for us to reintroduce the orphan / flag options in the project settings in some way. And also it may be wise to add command line options to choose these for the engine, so that worst case an end user can choose them.
At this stage (with the RCs) we have to tread with extreme caution changing the defaults for different platforms. Although 1 person may shout about low performance, if we change the default on that platform, we may get 1 happy person, and 1000 unhappy people because the old default worked better on theirs.

@lawnjelly I can confirm way better performance with RC9

It's been two years and I think the performance of Godot 3.x is decent on macOS, so I think we can close this as fixed.


fix duplicate SQL parsing and add docs

[x] Added or updated tests
[x] Documented user facing changes

Changes
The SQL Parser calls Parser::parse in the constructor so I removed the duplicate call.
I also added some docs about Laravel 9 attributes and PostgreSQL dumps.
Breaking changes
N/A

Hi,
Thank you for this!
But maybe having a separate new features documentation in docs/ folder would be better. To not clutter the README. Then we can reference it from README. What do you think?

@canvural, something I noted in the docs is that the sql parser can take a very long time parsing Postgres dumps (over 7 minutes for a <1MB file... only to return no tables). This is going to become a problem because the dumps are parsed automatically. I could point squashedMigrationsPath to a non-existent/empty directory but it would be more useful to just have an option disabling the parsing.

Great work has been done on master. Please consider rebasing.

Your commits are made with a non-GitHub email address.
If you want you can use your email address registered here on GitHub.

Your commits are made with a non-GitHub email address. If you want you can use your email address registered here on GitHub.

Ah good catch, how do I update the commits?

Thank you!


pack ringbuf for importScripts (and get rid of mlworkerscripts.js)

mlworkerscripts was hacked into existence for PAdenot's class Ringbuffer to be loaded with importScripts into the WorkerGlobalSpace.
That forced other issues such as having a bunch of code all dynamically loaded with geval, including:
MLSABOutputTransducer
createOutputChannel
input
output
outputSABs
inputSABs

Resolution was to standardise RingBuffer across the solution, import it directly from ringbuf.js (it has named export for the class RingBuffer) an assign it to a variable in the WorkerGlobalScope.
This still supports dynamic code eval in the worker.


0.8.2 calls backward on '_GeneratorContextManager'

🐛 Bug
0.8.2 calls backward on '_GeneratorContextManager' and crashes training.
0.8.1 works correctly. my training_step returns {'loss':loss, 'log':{'learn_rate':self.lr}}
Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
    fn(i, *args)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/trainer/distrib_data_parallel.py", line 538, in ddp_train
    self.run_pretrain_routine(model)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 1100, in run_pretrain_routine
    self.train()
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/trainer/training_loop.py", line 370, in train
    self.run_training_epoch()
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/trainer/training_loop.py", line 452, in run_training_epoch
    batch_output = self.run_training_batch(batch, batch_idx)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/trainer/training_loop.py", line 630, in run_training_batch
    self.hiddens
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/trainer/training_loop.py", line 804, in optimizer_closure
    model_ref.backward(self, closure_loss, optimizer, opt_idx)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/core/hooks.py", line 189, in backward
    loss.backward()
AttributeError: '_GeneratorContextManager' object has no attribute 'backward'

Expected behavior
backward is called on the loss and training runs correctly

did you override optimizer step?
could you try master? we just pushed a fix to a typo we had

Can confirm this happens on 0.8.3

ok. Can you post a colab example that replicates this?

@Anjum48 @s-rog
colab please

@williamFalcon my optimizer step was untouched, I can't run more testing atm but I'll get to it as soon as I can

@williamFalcon Hi I also encountered this, with normal Adam optimizer. I don't have a colab to recreate this atm but from what I saw earlier, this can be recreated with any setting as long as the Trainer is set to precision=16 when using Apex. Under this condition, the following lines from training_loop.py and hooks.py will run:
if self.precision == 16 and not self.on_tpu      closure_loss = model_ref.amp_scale_loss(closure_loss, optimizer, opt_idx)
scaled_loss = amp.scale_loss(unscaled_loss, optimizer)
will cause the closure_loss be a _GeneratorContextManager object. Which then cannot have a backward() method.
It seems under the current design, pytorch lighting's scale_loss function can only be used as a context?

@williamFalcon Here's a colab example (my first time using colab so let me know if you have issues seeing it) https://colab.research.google.com/drive/1G08jVDpx-T-5HE2c89RLJdq4u67mM2-o?usp=sharing
I suspect the issue lies with AMP as suggested above

ummm. I think this is an apex issue. I can't replicate it with 16-bit native.

@aeryen min share a minimal example to reproduce?

hi sorry for the delay: https://colab.research.google.com/drive/1rjaRRwgBTm4CKPfe9po_WSxnKqY4jDRv?usp=sharing

@aeryen , @Anjum48 ,@s-rog this is fixed on master. Give it a try?

yes, the master version works for me now. Thanks!

@williamFalcon can confirm as well! and sorry couldn't be more helpful earlier

Hi @williamFalcon thanks for the quick fix. I just upgraded but am now seeing a different error:
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
CUDA_VISIBLE_DEVICES: [0,1]
Using APEX 16bit precision.
initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/2
Loaded pretrained weights for efficientnet-b0
/home/anjum/PycharmProjects/kaggle/siim_isic_melanoma_classification/train.py:140: DtypeWarning: Columns (5) have mixed types.Specify dtype option on import or set low_memory=False.
  train_single_fold(args)
Using APEX 16bit precision.
initializing ddp: GLOBAL_RANK: 1, MEMBER: 2/2
----------------------------------------------------------------------------------------------------
distributed_backend=ddp
All DDP processes registered. Starting ddp with 2 processes
----------------------------------------------------------------------------------------------------
Selected optimization level O1:  Insert automatic casts around Pytorch functions and Tensor methods.

Defaults for this optimization level are:
enabled                : True
opt_level              : O1
cast_model_type        : None
patch_torch_functions  : True
keep_batchnorm_fp32    : None
master_weights         : None
loss_scale             : dynamic
Processing user overrides (additional kwargs that are not None)...
After processing overrides, optimization options are:
enabled                : True
opt_level              : O1
cast_model_type        : None
patch_torch_functions  : True
keep_batchnorm_fp32    : None
master_weights         : None
loss_scale             : dynamic

  | Name      | Type             | Params
-----------------------------------------------
0 | critereon | CrossEntropyLoss | 0
1 | net       | EfficientNet     | 4 M
Validation sanity check:  50%|███████████████████████▌                       | 1/2 [00:00<00:00,  1.01it/s]Traceback (most recent call last):
  File "/home/anjum/PycharmProjects/kaggle/siim_isic_melanoma_classification/train.py", line 140, in <module>
    train_single_fold(args)
  File "/home/anjum/PycharmProjects/kaggle/siim_isic_melanoma_classification/train.py", line 64, in train_single_fold
    trainer.fit(model)
  File "/home/anjum/anaconda3/envs/kaggle/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 957, in fit
    self.ddp_train(task, model)
  File "/home/anjum/anaconda3/envs/kaggle/lib/python3.8/site-packages/pytorch_lightning/trainer/distrib_data_parallel.py", line 538, in ddp_train
    self.run_pretrain_routine(model)
  File "/home/anjum/anaconda3/envs/kaggle/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1141, in run_pretrain_routine
    eval_results = self._evaluate(model,
  File "/home/anjum/anaconda3/envs/kaggle/lib/python3.8/site-packages/pytorch_lightning/trainer/evaluation_loop.py", line 346, in _evaluate
    self.reduce_eval_ddp(eval_results)
  File "/home/anjum/anaconda3/envs/kaggle/lib/python3.8/site-packages/pytorch_lightning/trainer/evaluation_loop.py", line 363, in reduce_eval_ddp
    self.reduce_eval_ddp(v)
  File "/home/anjum/anaconda3/envs/kaggle/lib/python3.8/site-packages/pytorch_lightning/trainer/evaluation_loop.py", line 365, in reduce_eval_ddp
    dist.all_reduce(v, op=dist.reduce_op.SUM)
  File "/home/anjum/anaconda3/envs/kaggle/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 898, in all_reduce
    work = _default_pg.allreduce([tensor], opts)
RuntimeError: Tensors must be CUDA and dense
Traceback (most recent call last):
  File "train.py", line 140, in <module>
    train_single_fold(args)
  File "train.py", line 64, in train_single_fold
    trainer.fit(model)
  File "/home/anjum/anaconda3/envs/kaggle/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 973, in fit
    self.spawn_ddp_children(model)
  File "/home/anjum/anaconda3/envs/kaggle/lib/python3.8/site-packages/pytorch_lightning/trainer/distrib_data_parallel.py", line 449, in spawn_ddp_children
    self.ddp_train(local_rank, model, is_master=True)
  File "/home/anjum/anaconda3/envs/kaggle/lib/python3.8/site-packages/pytorch_lightning/trainer/distrib_data_parallel.py", line 538, in ddp_train
    self.run_pretrain_routine(model)
  File "/home/anjum/anaconda3/envs/kaggle/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1141, in run_pretrain_routine
    eval_results = self._evaluate(model,
  File "/home/anjum/anaconda3/envs/kaggle/lib/python3.8/site-packages/pytorch_lightning/trainer/evaluation_loop.py", line 346, in _evaluate
    self.reduce_eval_ddp(eval_results)
  File "/home/anjum/anaconda3/envs/kaggle/lib/python3.8/site-packages/pytorch_lightning/trainer/evaluation_loop.py", line 363, in reduce_eval_ddp
    self.reduce_eval_ddp(v)
  File "/home/anjum/anaconda3/envs/kaggle/lib/python3.8/site-packages/pytorch_lightning/trainer/evaluation_loop.py", line 365, in reduce_eval_ddp
    dist.all_reduce(v, op=dist.reduce_op.SUM)
  File "/home/anjum/anaconda3/envs/kaggle/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 898, in all_reduce
    work = _default_pg.allreduce([tensor], opts)
RuntimeError: Tensors must be CUDA and dense

I'm not manually assigning tensors to a device (i.e. PL should be assigning all tensors as CUDA tensors) and I am not using sparse tensors (at least not that I am aware of).

I thought that mayber the error above is caused by my unbalanced dataset meaning that the target in the validation sanity check will be all zeros (i.e. sparse-like). I set num_sanity_val_steps=0 and am now seeing this error:
Loaded pretrained weights for efficientnet-b0
train.py:141: DtypeWarning: Columns (5) have mixed types.Specify dtype option on import or set low_memory=False.
  train_single_fold(args)
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
CUDA_VISIBLE_DEVICES: [0,1]
Using APEX 16bit precision.
initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/2
Loaded pretrained weights for efficientnet-b0
/home/anjum/PycharmProjects/kaggle/siim_isic_melanoma_classification/train.py:141: DtypeWarning: Columns (5) have mixed types.Specify dtype option on import or set low_memory=False.
  train_single_fold(args)
Using APEX 16bit precision.
initializing ddp: GLOBAL_RANK: 1, MEMBER: 2/2
----------------------------------------------------------------------------------------------------
distributed_backend=ddp
All DDP processes registered. Starting ddp with 2 processes
----------------------------------------------------------------------------------------------------
Selected optimization level O1:  Insert automatic casts around Pytorch functions and Tensor methods.

Defaults for this optimization level are:
enabled                : True
opt_level              : O1
cast_model_type        : None
patch_torch_functions  : True
keep_batchnorm_fp32    : None
master_weights         : None
loss_scale             : dynamic
Processing user overrides (additional kwargs that are not None)...
After processing overrides, optimization options are:
enabled                : True
opt_level              : O1
cast_model_type        : None
patch_torch_functions  : True
keep_batchnorm_fp32    : None
master_weights         : None
loss_scale             : dynamic

  | Name      | Type             | Params
-----------------------------------------------
0 | critereon | CrossEntropyLoss | 0
1 | net       | EfficientNet     | 4 M
Epoch 1:   0%|                                                                     | 0/226 [00:00<?, ?it/s]Traceback (most recent call last):
  File "/home/anjum/PycharmProjects/kaggle/siim_isic_melanoma_classification/train.py", line 141, in <module>
    train_single_fold(args)
  File "/home/anjum/PycharmProjects/kaggle/siim_isic_melanoma_classification/train.py", line 65, in train_single_fold
    trainer.fit(model)
  File "/home/anjum/anaconda3/envs/kaggle/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 957, in fit
    self.ddp_train(task, model)
  File "/home/anjum/anaconda3/envs/kaggle/lib/python3.8/site-packages/pytorch_lightning/trainer/distrib_data_parallel.py", line 538, in ddp_train
    self.run_pretrain_routine(model)
  File "/home/anjum/anaconda3/envs/kaggle/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1158, in run_pretrain_routine
    self.train()
  File "/home/anjum/anaconda3/envs/kaggle/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 370, in train
    self.run_training_epoch()
  File "/home/anjum/anaconda3/envs/kaggle/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 452, in run_training_epoch
    batch_output = self.run_training_batch(batch, batch_idx)
  File "/home/anjum/anaconda3/envs/kaggle/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 657, in run_training_batch
    grad_norm_dic = self.run_batch_backward_pass(split_batch, batch_idx, opt_idx, optimizer)
  File "/home/anjum/anaconda3/envs/kaggle/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 709, in run_batch_backward_pass
    self.call_optimizer_step(optimizer, opt_idx, batch_idx, split_batch)
  File "/home/anjum/anaconda3/envs/kaggle/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 746, in call_optimizer_step
    model.optimizer_step(self.current_epoch, batch_idx, optimizer, opt_idx, lambda_closure,
TypeError: optimizer_step() got an unexpected keyword argument 'using_native_amp'
Traceback (most recent call last):
  File "train.py", line 141, in <module>
    train_single_fold(args)
  File "train.py", line 65, in train_single_fold
    trainer.fit(model)
  File "/home/anjum/anaconda3/envs/kaggle/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 973, in fit
    self.spawn_ddp_children(model)
  File "/home/anjum/anaconda3/envs/kaggle/lib/python3.8/site-packages/pytorch_lightning/trainer/distrib_data_parallel.py", line 449, in spawn_ddp_children
    self.ddp_train(local_rank, model, is_master=True)
  File "/home/anjum/anaconda3/envs/kaggle/lib/python3.8/site-packages/pytorch_lightning/trainer/distrib_data_parallel.py", line 538, in ddp_train
    self.run_pretrain_routine(model)
  File "/home/anjum/anaconda3/envs/kaggle/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1158, in run_pretrain_routine
    self.train()
  File "/home/anjum/anaconda3/envs/kaggle/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 370, in train
    self.run_training_epoch()
  File "/home/anjum/anaconda3/envs/kaggle/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 452, in run_training_epoch
    batch_output = self.run_training_batch(batch, batch_idx)
  File "/home/anjum/anaconda3/envs/kaggle/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 657, in run_training_batch
    grad_norm_dic = self.run_batch_backward_pass(split_batch, batch_idx, opt_idx, optimizer)
  File "/home/anjum/anaconda3/envs/kaggle/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 709, in run_batch_backward_pass
    self.call_optimizer_step(optimizer, opt_idx, batch_idx, split_batch)
  File "/home/anjum/anaconda3/envs/kaggle/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 746, in call_optimizer_step
    model.optimizer_step(self.current_epoch, batch_idx, optimizer, opt_idx, lambda_closure,
TypeError: optimizer_step() got an unexpected keyword argument 'using_native_amp'
Exception ignored in: <function tqdm.__del__ at 0x7f394ab74310>
Traceback (most recent call last):
  File "/home/anjum/anaconda3/envs/kaggle/lib/python3.8/site-packages/tqdm/std.py", line 1077, in __del__
  File "/home/anjum/anaconda3/envs/kaggle/lib/python3.8/site-packages/tqdm/std.py", line 1284, in close
  File "/home/anjum/anaconda3/envs/kaggle/lib/python3.8/site-packages/tqdm/std.py", line 1461, in display
  File "/home/anjum/anaconda3/envs/kaggle/lib/python3.8/site-packages/tqdm/std.py", line 1080, in __repr__
  File "/home/anjum/anaconda3/envs/kaggle/lib/python3.8/site-packages/tqdm/std.py", line 1424, in format_dict
TypeError: cannot unpack non-iterable NoneType object

The kwarg using_native_amp was causing my code to crash since I had my optimizer_step set up like the bottom example here to enable warm up: https://pytorch-lightning.readthedocs.io/en/latest/optimizers.html?highlight=warm up#step-optimizers-at-arbitrary-intervals
So I guess there are two minor issues here:

Validation sanity check is mistaking my unbalanced dataset to look like a sparse tensor (I think)
using_native_amp in optmizer_step breaks the example in the documentation

These might be different issues to the Apex issue to feel free to transfer to a different issue

@Anjum48 mind send a new issue?


Change bounds for WALSegmentSize flag value

This allows the flag sets WALSegmentSize to be negative, which allows the user to disable the WAL. I saw that in tsdb/db.go:60 there is a comment stating that setting the segment size to anything less than zero will actually disable the WAL, but in the main file, the flag is restricted to a value between 10 and 256 MB. I wanted to expose this setting through the flag because the WAL can use a significant amount of RAM and for some users I think it may be worth disabling the WAL to avoid the OOM crash cycle as described in this comment: https://github.com/prometheus/prometheus/issues/5560#issuecomment-537210222

I don't think this is something we should expose, in fact with the current
code I don't think we need this flag at all.
On Tue 22 Oct 2019, 01:50 Afnan Enayet<EMAIL_ADDRESS>wrote:

This allows the flag that controls WALSegmentSize to be negative, which
gives the user the option to disable the WAL. I saw that in tsdb/db.go:60
there is a comment stating that setting the segment size to anything less
than zero will actually disable the WAL, but in the main file, the flag is
restricted to a value between 10 and 256 MB. I wanted to expose this
setting through the flag because the WAL can use a significant amount of
RAM and for some users I think it may be worth disabling the WAL to avoid
the OOM crash cycle as described in this comment: #5560 (comment)
https://github.com/prometheus/prometheus/issues/5560#issuecomment-537210222
You can view, comment on, or merge this pull request online at:
https://github.com/prometheus/prometheus/pull/6195
Commit Summary

Change bounds for WALSegmentSize flag value

File Changes

M cmd/prometheus/main.go
https://github.com/prometheus/prometheus/pull/6195/files#diff-0 (6)

Patch Links:

https://github.com/prometheus/prometheus/pull/6195.patch
https://github.com/prometheus/prometheus/pull/6195.diff

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/prometheus/prometheus/pull/6195?email_source=notifications&email_token=ABWJG5QEKPT7G2CXJQODDFDQPY53LA5CNFSM4JDIPCGKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HTLN2PQ,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABWJG5QU5QQ2VF5WBUEXMM3QPY53LANCNFSM4JDIPCGA
.

I apologize because I think I may have been unclear and framed this as an addition rather than a bugfix (not trying to add a new flag or anything).
We can see here that the WALSegmentSize value already accepts negative numbers to disable the WAL, and I just wanted to update main so it accurately reflects the WAL settings since it is already documented in the WAL options and it seems like the command line flag doesn't accurately reflect that.
https://github.com/prometheus/prometheus/blob/5f1be2cf455409d3577ac9ddb43144ec2d39c90b/tsdb/db.go#L59-L62

It's not sane to run the Prometheus binary without a WAL, and if you're OOMing at startup you were going to OOM anyway.

Gotcha, makes sense. Closing


User watching list pagination not docummented

I was trying to write some artist discovery logic, only to find that the watching list only returns the first page of the list.
Ideally there should be a route that either gets a certain page or every page.

Found the offending code, it seems the API does in fact a page parameter, but this is not documented anywhere. So I'm changing the issue name to reflect this.
So I'm
# GET /user/{name}/watching.json
    # GET /user/{name}/watching.xml
    # GET /user/{name}/watchers.json
    # GET /user/{name}/watchers.xml
    get %r{/user/#{USER_REGEX}/(watching|watchers)\.(json|xml)} do |name, mode, type|
      set_content_type(type)
      page = params[:page] =~ /^[0-9]+$/ ? params[:page] : 1
      is_watchers = mode == 'watchers'
      cache("watching:#{name}.#{type}.#{mode}.#{page}") do
        case type
        when 'json'
          JSON.pretty_generate @fa.budlist(name, page, is_watchers)
        when 'xml'
          @fa.budlist(name, page, is_watchers).to_xml(root: 'users', skip_types: true)
        end
      end
    end

Oh, it's documented under watching but not if you want the rest of the user data, nevermind then

Glad you got it figured out :)


Feature: beta transaction details page

@shelegdmitriy FYI, CI and preview deployment failed

@shelegdmitriy This page does not show the follow-up receipts, and also it says that the receipt has failed:
https://frontend-pr-940-67ml.onrender.com/beta/transactions/37LoeUEmL6P4MGkgXJdqm8jAUGDq1DAz5XFaLHYzECKa
This is the expected view: https://frontend-pr-940-67ml.onrender.com/transactions/37LoeUEmL6P4MGkgXJdqm8jAUGDq1DAz5XFaLHYzECKa


Provide more context in case of Error bumping dependencies

It could be helpful if there as a file and line number in an error messages such as: "Error bumping dependencies: Dependabot couldn't parse the Property not found: karaf.version found at Property not found: karaf.version." which I am getting from that little red triangle on my https://app.dependabot.com/accounts/vorburger/repos for https://github.com/vorburger/infrautils.
That error in this case is most likely due to #503, so this is more of a general suggestion, than something I need to figure out that particular problem.

Sounds like it's definitely going to be #503 related, but I agree the error message could be better here - I'll take a look.

OK, I've updated the error message so it will read:
Dependabot can't evaluate your Java dependency files.

As a result, Dependabot couldn't check whether any of your dependencies are out-of-date.

The error Dependabot encountered was:

Property not found: karaf.version

Hopefully that's a little clearer! At the moment it will stop the update process entirely, but an alternative would be for Dependabot to just ignore dependencies that use a property version that it can't find. Would that be better?
(I've stopped short of including line numbers because I think grepping for the property name is probably the easiest way to find it.)

an alternative would be for Dependabot to just ignore dependencies that
use a property version that it can't find. Would that be better?

actually I think that would be great! Best still show a warning though, not just completely silently ignore?

Done! You'll still get an error if all of your dependencies rely on properties that Dependabot can't reach, but won't do otherwise.
Unfortunately it's not easy for Dependabot to show a warning in the partial case without it being kind of annoying (it will create a GitHub issue for you each time), so I've left that out for now. I'll have a think about how it could be done better, but in the case where Dependabot can't find a property there's no way that it would be able to update that property, so I think the new behaviour might be what's desired anyway.
Thanks for your feedback on this one - much nicer solution now 😄

Cool - it works: https://github.com/vorburger/infrautils/pulls - thank you!


fixed #49 issues for different functions

functions that are changed: digifiltersim, step_response, ramp_response, parabolic_response

This looks excellent! thank you so much! :heart:

You're most welcome!


Adding cocoapods support

Hi, can we add cocoapods support? If I create pull request, will You publish it to spec repo?

@agruchala  Hi! Yeap =)
Thx for ur future contributing!👍🏻

Cool, I’ll do the required changes today, You can publish then to podspec
repo :)
W dniu pt., 12.05.2023 o 13:06 Alexander Kraev @.***>
napisał(a):

@agruchala https://github.com/agruchala Hi! Yeap =)
Thx for ur future contributing!👍🏻
—
Reply to this email directly, view it on GitHub
https://github.com/c-villain/SwipeActions/issues/17#issuecomment-1545569853,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABLQNXMAXMMFYCCFOUCAFJTXFYKR3ANCNFSM6AAAAAAX7LYVJE
.
You are receiving this because you were mentioned.Message ID:
@.***>

#18 is Waiting :)

Hello:) Do You need help with that?

✅ done


Stop when we find the block

Depends

https://github.com/marigold-dev/deku/pull/644

Problem
Same but with a return type and early stopping. We said we were OK with using Base, right? If not, we can always use an exception.
Feel free to just say "no" and close.
Testing
 echo $(wget -q http://localhost:4440/block-by-level --post-data '{"level":4115}' -O -)

Now that I think of it, you will probably find the return type annoying to parse with bash.

Closed after discussion on Slack (not that useful).


experiment

Signed-off-by: Hiroshi Muraoka<EMAIL_ADDRESS>

This is created to do experiment of git-neco.


add support for connexion

please add support for connexion as discussed on https://forums.aws.amazon.com/message.jspa?messageID=890026

For developer reference:
If we attempt to patch the internal Flask App used by the framework, the methods added to the flask app's before_request() call are never actually called. The after_request() methods are called, however, so this is problematic. A segment is never generated by the middleware, but a call to end it occurs.
A thing to look at is how the internal flask app is being utilized by Connexion, and seeing if there is a way possible to invoke a method call before a request is handled.

+1


bug: Physical Back button exits app instead of going back in the nav

Type: bug
Ionic Version: 2.x
Platform: android 5.1 webview
Steps to reproduce issue:

ionic start myApp sidemenu --v2 --ts
ionic run android --device --livereload
Tap Menu Icon
Select "Lists"
Tap a list item to push a new ListPage onto the nav stack
Tap your device's physical Back button

Results:
The app exits and you are brought to the Android launch screen
Expected Results:
The ListPage would be popped off of the nav stack just like if you had tapped the software back button at the top left of the app.

Thank you for the issue, we have this on our roadmap to finish around beta.4. There is an issue for it here: https://github.com/driftyco/ionic/issues/5419
Please take a look at our roadmap for other features and fixes:
https://docs.google.com/document/d/1Qlc5X2eJyOB0izkFlH7KJ5BmMi0MeXUZRHJHt3hS6Wo/edit?usp=sharing


Possible bug in certificate validation?

Hi and thank you for this component.
I think there is an inverted condition inside the file Connection\Protocols\Protocol.php, function defaultSocketOptions().
private function defaultSocketOptions() {
	return $this->getCertValidation()
		? [
			'ssl' => [
				'verify_peer_name' => false,
				'verify_peer'      => false,
			]
		]
		: [];
}

if $this->getCertValidation() returns true, defaultSocketOptions() returns options to NOT validate the certificate. I had to put 'cert_validation' to true in my config to disable the validation. So I think it should be this way:
private function defaultSocketOptions() {
	return $this->getCertValidation()
		? [] : [
			'ssl' => [
				'verify_peer_name' => false,
				'verify_peer'      => false,
			]
		];
}

Hope it helps :)
Edit: this is inside php-imap package, maybe I should have opened this issue there

Hi @mynamespace ,
thanks for your report. It seems indeed strange.. but I think I know the reason - there is a step missing:
$stream = stream_socket_client($socket, $errno, $errstr, $timeout,
            STREAM_CLIENT_CONNECT,
            stream_context_create($this->defaultSocketOptions())
);

This is the only part where the method is called and where the socket options should be updated with the actual account setting based on the currently not provided information if the certs should be validated or not.
Protocol::defaultSocketOptions() should not care if the certs get validated but actually care about the used transport method.
I'm closing this issue here. Please feel free to reopen it under webklex/php-imap. However I have it on my todo-list - will be fixed within the next few days :)
Thanks again for your report.
Best regards,


feat(logs): support LogGroup as a Stream

closes #28

Uh, first off, I'd like to apologize for completely missing this. I know it's been a few years but I was not in the right headspace to work on Punchcard around then.
Sorry for disregarding your hard work!


Socket Options Integrated Into Ember

FS2 Just Integrated in Support for various socket options into the fs2 Socket code.
https://github.com/functional-streams-for-scala/fs2/pull/1549
So when released, we should continue this support into ember to offer the ability to configure the client and server with these changes.

I would like to give this one a go.

I've pushed a draft of how I imagine it would be done when the new fs2 version would be released. Would love some feedback if things aren't looking ok.


Include link to stencil-apexcharts

Hi.
I've been playing around with creating a StencilJS wrapper for ApexCharts and I have something that resembles the React example completed.
It's still a WIP but I'd love it if you can take a quick look so when the ts fixes (48b8b9d550b6d871a40d7251153558f39dbde952 and 6c7e9a1d768405e94b5fa068e30150bac67981fd)  are released I can push it to NPM.
If you have any feedback please reply here - I'd love to hear it.

Sure, I will gladly put a link in the core-apexcharts README for this wrapper.
Good work there.

Updated README


Support for has*CauseInstance* variants of isInstanceOfAny, ...

Feature summary
Provide isInstanceOfAny, isNotInstanceOfAny variants for hasCauseInstanceOf, hasCauseExactlyInstanceOf, hasRootCauseInstanceOf, hasRootCauseExactlyInstanceOf ... throwable asserts.
Example
Before
assertThat(result.getResolvedException())
                .isInstanceOfAny(ConstraintViolationException.class, MethodArgumentNotValidException.class)

After
assertThat(result.getResolvedException())
                .hasCauseExactlyInstanceOfAny(ConstraintViolationException.class, MethodArgumentNotValidException.class)

Have you considered the cause() navigation method?
assertThat(result.getResolvedException())
  .cause()
  .isInstanceOfAny(ConstraintViolationException.class, MethodArgumentNotValidException.class);

Thanks @scordio - you've helped me catch an understanding gap here. What I had use to test my case was assertThat(...).isInstanceOfAny(...). I didn't realise cause() only verified the immediate cause for the exception.
The solution you've provided should work for anyone willing to do the full instance check on the exception causes & root causes.
Given that, I still think having similar set of interfaces for cause and rootCause will provide a richer experience in using AssertJ.
P.S: Linking to AssertJ throwable assertion docs for anyone winding up here: https://assertj.github.io/doc/#assertj-core-throwable-cause-and-root-cause-assertions

rootCause() is also available, I updated the previous comment.
 Is there a use case you can't achieve with cause or rootCause?

Is there a use case you can't achieve with cause or rootCause?

No, both cause & rootCause should solve for the all the usecases.

The ticket can be closed if this is the stance.

On a tangent, are there APIs which tests if at least one cause ( from the throwable given, to the root cause ) matches the given test.
Ex:
// a = new T1();
// b = new T2(a);
// c = new T3(b);
// d = new T4(c);
assertThat(d).hasAnyCauseExactlyInstanceOf(T4.class);
assertThat(d).hasAnyCauseExactlyInstanceOf(T3.class);
assertThat(d).hasAnyCauseExactlyInstanceOf(T2.class);
assertThat(d).hasAnyCauseExactlyInstanceOf(T1.class);

If this is something you're willing to explore, we can create a different ticket for this.

If I understand it correctly, you're looking for something to verify the type of a cause, which might be direct or indirect, up to the root cause.
As of now, we don't have any assertion to achieve that.
Out of curiosity, are you facing a concrete use case where you cannot predict at which level the cause to be verified is?

This is not something that happens during unit testing, but during integration testing, where we want to check if certain business exception has been raised, which could be intercepted by other layers ( framework or otherwise ).

Isn't where rootCause() would help to verify the business exception which was originally raised?

What if we provide a cause(Class<?>), which would recursively search for a cause of the given type and allow further assertions on that?
I imagine a usage like the following:
assertThat(result.getResolvedException())
  .cause(MyBusinessException.class)
  .hasMessage("...") // further assertions

I'm not entirely satisfied with my proposal because the recursive behavior might be too hidden, but it might be a direction worth refining.

I feel a navigation method will be helpful, but it should be something other than cause & rootCause since they denote immediate cause & root cause.
Maybe forCause, forExactCause navigation methods. Where forCause will navigate to the first match even with inheritance relationship and forExactCause will navigate to the first exact match. Both of them will give assertion failure if no cause matched them.

I feel a navigation method will be helpful, but it should be something other than cause & rootCause since they denote immediate cause & root cause.

I agree. I wouldn't go with a for prefix but rather try to find a suitable adjective that could be used together with cause, perhaps .intermediateCause() or something similar... naming is hard
Thanks for the input, we'll discuss it internally and get back to you.

What are you thoughts on the names atCause and atExactCause?

We tend to name navigation methods after getters but without the "get" part, i.e., cause() from Throwable::getCause().
Using an "at" preposition might be less intuitive and usually adjectives fit better.


v1.9 backports 2021-11-26

#17865 -- Fixes for IPsec and endpoint routes (@kkourt)
#17916 -- bugtool: fix data race occurring when running commands (@rolinh)

Once this PR is merged, you can update the PR labels via:
$ for pr in 17865 17916; do contrib/backporting/set-labels.py $pr done 1.9; done

/test-backport-1.9

/test-gke

GKE tests are in fact broken on v1.9 at the moment (https://github.com/cilium/cilium/pull/17797 for context and WIP fix). All other tests are passing, I'll go ahead and merge the PR.


v0.2.2 breaks SMBFilesystem

hi, we are using upath to access an smb share. Unfortunately v0.2.2 broke the functionality.
It appears that the url components are somehow not passed on correctly.
E.g. UPath("smb://server.com/share/folder/file.csv")
will result in

ValueError: Failed to connect to 'server.comshare:445': [Errno 8] nodename nor servname provided, or not known

Edit: I would be happy and try to contribute a basic SMB implementation, is there any guide or example I can follow?

I appear I can fix this behaviour by implementing a very simple subclass that returns an absolute path:
class SMBPath(UPath):

    @property
    def path(self):
        return "/" + super().path

The docs from ffspec mention that:

Note: if using this with the open or open_files, with full URLs, there is no way to tell if a path is relative, so all paths are assumed to be absolute.

Which sounds like interpreting every path as absolute should be the default behaviour. Should I maybe file a bug with fsspec instead?

The fact that it worked with a previous version of UPath suggests that it's a UPath bug which got introduced during the refactor. Ideally we would add a SMBPath implementation to UPath to be able to give more guarantees for all the path methods.
I'm on holiday until the end of May and will then try to find some time to work on setting this up. We would need to implement an SMBPath subclass and the test fixtures to run the base test suite against it.

Edit: I would be happy and try to contribute a basic SMB implementation, is there any guide or example I can follow?

Sorry, just saw your comment. Thank you for offering to contribute! You would have to create tests in upath/tests/implementations/test_smb.py which requires you to write a fixture that sets up the smb filesystem via a docker container. There is code in the fsspec/filesystem_spec repository that does this. You would then put your minimal implementation in upath/implementations/smb.py and add an entry pointing at your new class into upath/registry.py
This should allow you to run the test suite against your class and make adjustments for various edge cases.
Cheers,
Andreas 😊

Hi, we have the same problem with an sftp-based filesystem:
path = UPath("ssh://sat01-utv.smhi.se/home/user1/Documents/hello.txt")
print(path.path)

'home/user1/Documents/hello.txt'


typescriptOptions System.config required?

When I don't add a typescriptOptions to System.config I get the following error.
System.typescriptOptions is undefined

Adding a empty object (see below) is enough, but couldn't this be made optionally?
System.config({
  'typescriptOptions': {}
});

Thanks for raising this, it's a bug and I'll put out a fix shortly.

fixed in 2.0.2


Feature request: Add integration tests

Use case
Some releases of AWS Lambda Powertools have run into runtime regression. So without integration tests, customers of this library would have to run their own integration tests
Solution/User Experience
AWS Lambda Powertools would include a series of integration tests for the supported runtimes.
See how this was resolved in the typescript version https://github.com/awslabs/aws-lambda-powertools-typescript/issues/418
Or how AWS CDK runs full suite of integration tests for every release and PR
Alternative solutions
Each usage of AWS Lambda Powertools should also include integration tests. Especially for features that are mocked locally like idempotency.
Acknowledgment

[X] This feature request meets Lambda Powertools Tenets
[X] Should this be considered in other Lambda Powertools languages? i.e. Java, TypeScript

Closing as @mploski (maintainer) is already working on that as part of our operational readiness review.

It's there a tracking issue for this ? Michael brewer (contributor & customer)


Consider adding Julia community registry

Since Julia 1.0, public packages are registered in https://github.com/JuliaRegistries /General, by creating some files under LETTER/PACKAGENAME. For instance, the package NLPModels.jl is registered because there are files under https://github.com/JuliaRegistries/General/blob/master/N/NLPModels/.
All packages need an OSI-approved license to be registered in the default registry.
One check can be to verify that Package.toml exists in that folder.

Hi Abel, my initial thoughts are that we should stick to a similar mechanism as most of the other tests we have now (i.e. regex on the README) for the time being. If we do go through with the refactor into plugin architecture (#334), I anticipate that the implementation of any given test will be hidden away in a separate libray so those libraries can do whatever they think is useful for their use case.