Add data to local Inplace Upgrade w/ IAMv2p1

Overview
Currently, https://a2-iamv2p1-local-inplace-upgrade-acceptance.cd.chef.co/ and https://a2-iamv2p1-local-inplace-upgrade-dev.cd.chef.co/ do not have any node, event, or compliance data. We want them to have data like the other iam environments so we can do better ux reviews and demos.

closing card, will connect with releng.


The diff coverage is n/a.

@@            Coverage Diff             @@
##           master      #45      +/-   ##
==========================================
+ Coverage    91.7%   91.89%   +0.19%
==========================================
  Files          13       13
  Lines        1000      962      -38
==========================================
- Hits          917      884      -33
+ Misses         83       78       -5

Impacted Files
Coverage Δ

src/mutation/apply.jl
81.81% <ø> (+27.27%)
:arrow_up:

src/NaiveNASlib.jl
100% <ø> (ø)
:arrow_up:

src/mutation/vertex.jl
92.77% <ø> (-0.42%)
:arrow_down:

src/mutation/op.jl
85.71% <ø> (-3.42%)
:arrow_down:

src/mutation/size.jl
90.94% <0%> (-2.59%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a57fcdd...579bcc5. Read the comment docs.


Is there a way to get the sink from the 1.16.5 version

Hey,
This is the only mod i know that provied huge amounts of water for mekanism cables, is there a way to get the sink from 1.16.5 in 1.7.10, because the sink in 1.7.10 doesnt work like that other one

1.7.10 is no longer supported


Linker Warning

Seems some additional linker sections are missing from bl808:
/home/fish/bl808/bin/lib/gcc/riscv64-unknown-elf/10.2.0/../../../../riscv64-unknown-elf/bin/ld:/home/fish/bl808/bl_mcu_sdk/bsp/board/bl808dk/mcu/bl808_flash_m0.ld:289: warning: memory region `ram_code' not declared
/home/fish/bl808/bin/lib/gcc/riscv64-unknown-elf/10.2.0/../../../../riscv64-unknown-elf/bin/ld:/home/fish/bl808/bl_mcu_sdk/bsp/board/bl808dk/mcu/bl808_flash_m0.ld:310: warning: memory region `ram_wifi' not declared

just ignore.

To avoid the issue of duplication, i have removed it.

Thanks!


Dependency issue

I noticed there was a side discussion on dependencies in this issue, so apologies if this is a dupe.
I seem to be experiencing a dependency conflict with the latest alpha2, but I can't quit seem to figure out why this is happening. I've excluded jackson-core and jackson-databind, as well as aws-java-sdk from all the relevant dependencies in my own project. Using boot.
Exception in thread "pool-5-thread-1" java.lang.NoSuchMethodError: com.fasterxml.jackson.core.JsonParser.setCurrentValue(Ljava/lang/Object;)V
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:243)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:136)
        at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3564)
        at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2533)
        at com.amazonaws.internal.config.InternalConfig.loadfrom(InternalConfig.java:236)
        at com.amazonaws.internal.config.InternalConfig.load(InternalConfig.java:249)
        at com.amazonaws.internal.config.InternalConfig$Factory.<clinit>(InternalConfig.java:304)
        at com.amazonaws.util.VersionInfoUtils.userAgent(VersionInfoUtils.java:141)
        at com.amazonaws.util.VersionInfoUtils.initializeUserAgent(VersionInfoUtils.java:136)
        at com.amazonaws.util.VersionInfoUtils.getUserAgent(VersionInfoUtils.java:97)
        at com.amazonaws.ClientConfiguration.<clinit>(ClientConfiguration.java:60)
        at org.springframework.build.aws.maven.S3Utils.getClientConfiguration(S3Utils.java:45)
        at org.springframework.build.aws.maven.PrivateS3Wagon.connectToRepository(PrivateS3Wagon.java:92)
        at org.springframework.build.aws.maven.AbstractWagon.connect(AbstractWagon.java:143)
        at org.sonatype.aether.connector.wagon.WagonRepositoryConnector.connectWagon(WagonRepositoryConnector.java:345)
        at org.sonatype.aether.connector.wagon.WagonRepositoryConnector.pollWagon(WagonRepositoryConnector.java:385)
        at org.sonatype.aether.connector.wagon.WagonRepositoryConnector$GetTask.run(WagonRepositoryConnector.java:571)
        at org.sonatype.aether.util.concurrency.RunnableErrorForwarder$1.run(RunnableErrorForwarder.java:60)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

Any ideas? Is there anything I can do on my end to solve this?

Thanks @danielcompton for the response. Is there a conventional way to check the dependencies in an uberjar?
I'm using boot 2.5.5. Looking through the source it seems it is pulling in cheshire 5.3.1, which in turn is pulling in jackson 2.3.1.

Thanks @danielcompton for the response. Is there a conventional way to check the dependencies in an uberjar?

Not really sure apart from opening the JAR up, don't think there is.

I'm using boot 2.5.5. Looking through the source it seems it is pulling in cheshire 5.3.1, which in turn is pulling in jackson 2.3.1.

 It looks like there's a newer version of Cheshire out which has a more recent Jackson version (although it's newer than the version that we bundle, so not sure what that would do).

On a perhaps related note, we had issues with both earlier and later versions of Jackson outside of 2.5.5. There's some context from this PR where we worked through similar sounding "No such method" exceptions as you've experienced.

@zalky Quick note:  I managed to "fix" this error by compiling a version of boot where I updated the version of Cheshire to 5.7.0.  However, I am now experiencing #40.

Quick update, I managed to get everything working by changing the dependency in boot/aether from org.apache.maven.wagon/wagon-http "2.9" to "2.12".

Yup, this issue is blocking me from upgrading to 1.3.0.

Should be fixed with #59 and boot-clj/boot#679


nxstyle include check is too strict

As a rule, all #include pre-processor commands must appear in the "Included Files" section:  That section header must be in the file and all of the #include commands must follow.
However, there are exceptions.  For example:
$ tools/nxstyle.exe arch/arm/include/xmc4/irq.h
arch/arm/include/xmc4/irq.h:81:3: warning: #include outside of 'Included Files' section
arch/arm/include/xmc4/irq.h:83:3: warning: #include outside of 'Included Files' section
arch/arm/include/xmc4/irq.h:85:3: warning: #include outside of 'Included Files' section

This kind of problem is the case with many architectures. I have been ignoring this error in the past, but it is harder to ignore with this automated style checker.
And in this case, the #include cannot be moved because it depends on a #define that must precede it. And you cannot move the #define or your will get a similar error. I think that there is no alternative but you use good engineering judgement and accept the change as is.
This kind of logic is very common in architecure specific code:
Set up some definitions, then
include some architecture specific code

You will see this a lot in arch/xyz/include and arch/src/xyz/hardware directories.  This usually occurs int he "Pre-processor Definitions" section, and not in the "Included Files" section.
Ideally, then all #include's should be in the "Included Files" section. But there are exceptions where it makes sense to relax this requirement.  So example iss one and there are several others.
I don't know of any way to programmatically "relax" the requirement other than inspecting the nxstyle complaint and then making a good, informed engineering decision.  One possibility to to assure that "Included Files" was encountered sometime prior to the #include command.  That would allow the #include to occur in any section after the "Included Files" section but would have the odd side-effect of requiring that an "Included Files" section is present even when there are no other files to include.  That would, however, be rare.

i had to make a change to similar construct: https://github.com/apache/incubator-nuttx/commit/63395b295f9820b4570e7327111f05a9683793e7

And you cannot move the #define or your will get a similar error.

really? in my case, i moved #define and it worked.
how about allowing an optional "Conditionally Included Files" section?

I've worked around this in a few files by putting multiple "Included Files" section headers. That will eliminate the warning buf is probably not the right way to go.


dhcpcd: add extraFlags option

Motivation for this change
Allow passing of additional command line flags to the dhcpcd daemon.
Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[X] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[x] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Fits CONTRIBUTING.md.

@GrahamcOfBorg test networking.scripted.dhcpSimple

No attempt on aarch64-linux (full log)
The following builds were skipped because they don't evaluate on aarch64-linux: tests.networking.scripted.dhcpSimple
Partial log (click to expand)
Cannot nix-instantiate `tests.networking.scripted.dhcpSimple' because:
error: while evaluating anonymous function at /var/lib/gc-of-borg/nix-test-rs-23/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-23/lib/attrsets.nix:386:24, called from /var/lib/gc-of-borg/nix-test-rs-23/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-23/lib/attrsets.nix:335:15:
while evaluating 'handleTest' at /var/lib/gc-of-borg/nix-test-rs-23/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-23/nixos/tests/all-tests.nix:17:22, called from /var/lib/gc-of-borg/nix-test-rs-23/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-23/nixos/tests/all-tests.nix:142:25:
while evaluating 'discoverTests' at /var/lib/gc-of-borg/nix-test-rs-23/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-23/nixos/tests/all-tests.nix:13:19, called from /var/lib/gc-of-borg/nix-test-rs-23/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-23/nixos/tests/all-tests.nix:18:5:
access to path '/nix/store/chlcl98b2ardsq30kmqqb7dqj4dm76cj-grahamc-aarch64-community-23' is forbidden in restricted mode

@GrahamcOfBorg test nixosTests.networking.scripted.dhcpSimple

No attempt on aarch64-linux (full log)
The following builds were skipped because they don't evaluate on aarch64-linux: tests.nixosTests.networking.scripted.dhcpSimple
Partial log (click to expand)
Cannot nix-instantiate `tests.nixosTests.networking.scripted.dhcpSimple' because:
error: attribute 'nixosTests' in selection path 'tests.nixosTests.networking.scripted.dhcpSimple' not found

No attempt on x86_64-linux (full log)
The following builds were skipped because they don't evaluate on x86_64-linux: tests.nixosTests.networking.scripted.dhcpSimple
Partial log (click to expand)
Cannot nix-instantiate `tests.nixosTests.networking.scripted.dhcpSimple' because:
error: attribute 'nixosTests' in selection path 'tests.nixosTests.networking.scripted.dhcpSimple' not found

No attempt on x86_64-linux (full log)
The following builds were skipped because they don't evaluate on x86_64-linux: tests.networking.scripted.dhcpSimple
Partial log (click to expand)
Cannot nix-instantiate `tests.networking.scripted.dhcpSimple' because:
error: while evaluating anonymous function at /var/lib/gc-of-borg/.nix-test-rs/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/rbvermaa-spot/lib/attrsets.nix:386:24, called from /var/lib/gc-of-borg/.nix-test-rs/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/rbvermaa-spot/lib/attrsets.nix:335:15:
while evaluating 'handleTest' at /var/lib/gc-of-borg/.nix-test-rs/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/rbvermaa-spot/nixos/tests/all-tests.nix:17:22, called from /var/lib/gc-of-borg/.nix-test-rs/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/rbvermaa-spot/nixos/tests/all-tests.nix:142:25:
while evaluating 'discoverTests' at /var/lib/gc-of-borg/.nix-test-rs/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/rbvermaa-spot/nixos/tests/all-tests.nix:13:19, called from /var/lib/gc-of-borg/.nix-test-rs/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/rbvermaa-spot/nixos/tests/all-tests.nix:18:5:
access to path '/nix/store/ckq607zh70k858h7bv4v2lm1czr6sxa7-rbvermaa-spot' is forbidden in restricted mode

@GrahamcOfBorg test networking.scripted.dhcpSimple

No attempt on x86_64-linux (full log)
The following builds were skipped because they don't evaluate on x86_64-linux: tests.networking.scripted.dhcpSimple
Partial log (click to expand)
Cannot nix-instantiate `tests.networking.scripted.dhcpSimple' because:
error: while evaluating anonymous function at /var/lib/gc-of-borg/.nix-test-rs/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/rbvermaa-spot/lib/attrsets.nix:386:24, called from /var/lib/gc-of-borg/.nix-test-rs/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/rbvermaa-spot/lib/attrsets.nix:335:15:
while evaluating 'handleTest' at /var/lib/gc-of-borg/.nix-test-rs/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/rbvermaa-spot/nixos/tests/all-tests.nix:17:22, called from /var/lib/gc-of-borg/.nix-test-rs/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/rbvermaa-spot/nixos/tests/all-tests.nix:142:25:
while evaluating 'discoverTests' at /var/lib/gc-of-borg/.nix-test-rs/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/rbvermaa-spot/nixos/tests/all-tests.nix:13:19, called from /var/lib/gc-of-borg/.nix-test-rs/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/rbvermaa-spot/nixos/tests/all-tests.nix:18:5:
access to path '/nix/store/5zni90v17w3scmxjdsrjs3pd7wdljp8z-rbvermaa-spot' is forbidden in restricted mode

No attempt on aarch64-linux (full log)
The following builds were skipped because they don't evaluate on aarch64-linux: tests.networking.scripted.dhcpSimple
Partial log (click to expand)
Cannot nix-instantiate `tests.networking.scripted.dhcpSimple' because:
error: while evaluating anonymous function at /var/lib/gc-of-borg/nix-test-rs-5/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-5/lib/attrsets.nix:386:24, called from /var/lib/gc-of-borg/nix-test-rs-5/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-5/lib/attrsets.nix:335:15:
while evaluating 'handleTest' at /var/lib/gc-of-borg/nix-test-rs-5/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-5/nixos/tests/all-tests.nix:17:22, called from /var/lib/gc-of-borg/nix-test-rs-5/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-5/nixos/tests/all-tests.nix:142:25:
while evaluating 'discoverTests' at /var/lib/gc-of-borg/nix-test-rs-5/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-5/nixos/tests/all-tests.nix:13:19, called from /var/lib/gc-of-borg/nix-test-rs-5/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-5/nixos/tests/all-tests.nix:18:5:
access to path '/nix/store/hw9vj1ynh3lfhav6s61gz6vx00gbqmli-grahamc-aarch64-community-5' is forbidden in restricted mode


Fix db upgrade script b4456560d4f3

closes https://github.com/airbnb/caravel/issues/369

Coverage remained the same at 72.471% when pulling 9270ad3160aa1d955337a335622f7afe48f0668e on fix_db_upgrade into 3f0171b77be43adbce17a98b0e109e1935a6d96f on master.


Fix going back from card provider in Company cards

Explanation of Change
We missed correct go back logic from card provider for any selected bank (except Other)
Fixed Issues
$ https://github.com/Expensify/App/issues/53507
PROPOSAL:
Tests
Test case 1:

Enable company card in workspace
Click Add card
Choose any bank (except Other) and click Next
Choose any feed type and click Next
Click the back arrow in the Who's your provider page
Verify that user goes back to feed type

Test case 2:

Enable company card in workspace
Click Add card
Choose Other bank and click Next
Click the back arrow in the Who's your provider page
Verify that user goes back to select bank screen

[x] Verify that no errors appear in the JS console

Offline tests
No changes
QA Steps
Test case 1:

Enable company card in workspace
Click Add card
Choose any bank (except Other) and click Next
Choose any feed type and click Next
Click the back arrow in the Who's your provider page
Verify that user goes back to feed type

Test case 2:

Enable company card in workspace
Click Add card
Choose Other bank and click Next
Click the back arrow in the Who's your provider page
Verify that user goes back to select bank screen

[x] Verify that no errors appear in the JS console

PR Author Checklist

[x] I linked the correct issue in the ### Fixed Issues section above
[x] I wrote clear testing steps that cover the changes made in this PR

[x] I added steps for local testing in the Tests section
[x] I added steps for the expected offline behavior in the Offline steps section
[x] I added steps for Staging and/or Production testing in the QA steps section
[x] I added steps to cover failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[x] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
[x] I tested this PR with a High Traffic account against the staging or production API to ensure there are no regressions (e.g. long loading states that impact usability).

[x] I included screenshots or videos for tests on all platforms
[x] I ran the tests on all platforms & verified they passed on:

[x] Android: Native
[x] Android: mWeb Chrome
[x] iOS: Native
[x] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[x] MacOS: Desktop

[x] I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
[x] I followed proper code patterns (see Reviewing the code)

[x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick)
[x] I verified that comments were added to code that is not self explanatory
[x] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[x] I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method

[x] If any non-english text was added/modified, I used JaimeGPT to get English > Spanish translation. I then posted it in #expensify-open-source and it was approved by an internal Expensify engineer. Link to Slack message:

[x] I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
[x] I verified any copy / text that was added to the app is grammatically correct in English. It adheres to proper capitalization guidelines (note: only the first word of header/labels should be capitalized), and is either coming verbatim from figma or has been approved by marketing (in order to get marketing approval, ask the Bug Zero team member to add the Waiting for copy label to the issue)
[x] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[x] I verified the JSDocs style guidelines (in STYLE.md) were followed

[x] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[x] I followed the guidelines as stated in the Review Guidelines
[x] I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar are working as expected)
[x] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[x] I verified any variables that can be defined as constants (ie. in CONST.ts or at the top of the file that uses the constant) are defined as such
[x] I verified that if a function's arguments changed that all usages have also been updated correctly
[x] If any new file was added I verified that:

[x] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory

[x] If a new CSS style is added I verified that:

[x] A similar style doesn't already exist
[x] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(theme.componentBG))

[x] If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
[x] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[x] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[x] If the PR modifies a component or page that can be accessed by a direct deeplink, I verified that the code functions as expected when the deeplink is used - from a logged in and logged out account.
[x] If the PR modifies the UI (e.g. new buttons, new UI components, changing the padding/spacing/sizing, moving components, etc) or modifies the form input styles:

[x] I verified that all the inputs inside a form are aligned with each other.
[x] I added Design label and/or tagged @Expensify/design so the design team can review the changes.

[x] If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
[x] I added unit tests for any new feature or bug fix in this PR to help automatically prevent regressions in this user flow.
[x] If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.

Screenshots/Videos

Android: Native
https://github.com/user-attachments/assets/eb58ba88-6977-438e-8c35-29822e0d2fcf

Android: mWeb Chrome
https://github.com/user-attachments/assets/83c7bb59-be8e-4d21-860f-00119953dcc4

iOS: Native
https://github.com/user-attachments/assets/fe37a58e-7312-44b2-b8e7-9148a4de1a22

iOS: mWeb Safari
https://github.com/user-attachments/assets/f7fe8f5e-e524-4afa-a330-6242f854c3e1

MacOS: Chrome / Safari
https://github.com/user-attachments/assets/227b0b5a-d775-4b32-9065-87e6b538fbe8

MacOS: Desktop
https://github.com/user-attachments/assets/40930de8-557d-4611-a8db-75f1b4f8a9a9

Confirmed both cases working now!
https://github.com/user-attachments/assets/e8449cda-612f-46a5-9869-41bb7b0aea97


Causes very high system load

The adapter causes very high system load.
System:
Proxmox host (3 node cluster), i7-12650H, 16 cores, 32 GB RAM, storage via NFS from a QNAP
iobroker 4 Cores, 16 GB RM
The only VM on that Proxmox host
————
After installing and using the adapter, my iobroker incl. the Proxmox host almost “died”.
CPU-temp of the host went up to above 70 degrees celsius
Top gives back:
Espresence up to 23% system load (average approx. 19%)
is-controller 75% (min. 64%) system load
————
After stopping Espresence..
CPU-temp going down to approx. 45-50 degrees celcius
Top:
js-controller max. 31% system load (average 23%)

Sorry
Used adapter v0.4.2
Node v18.19.0
Npm v10.2.3
Controller v5.0.17
Admin v6.13.15

Are you using the internal MQTT server? How many esp32? I have about 1% system load with 2 esp32.
Could also be because no filters are used, then there is a lot of data to a lot of devices. But I don't know how to reduce this.

Used an sep. MQTT server (iobroker).
3 ESPs
Filters are set at the ESPs - so only known devices are transferred to MQTT

I can do a sep. test with the internal MQTT server. Just let me know.

I'll add a summary of the state updates every 15 seconds. Maybe this will reduce the system load. If that doesn't bring any improvement. Let's try the same thing directly behind the mqtt client. But that will have to wait until next weekend.
When we have the problem under control, I'll make the delay configurable.

In the next version I will try to reduce the system load by collecting the mqtt messages for 5 seconds.
Release in the next few minutes

Let me know if the current version reduces the system load.

Seems to be much better.
Trying with 1 ESP and system load is very good.
I‘ll keep an eye on it and using more ESPs during the day

Increased up to 3 ESPs and 2 Apple devices …. still working good


Is it possible to save directly to S3 bucket?

When separating a PDF, I'm currently saving to the local file system and then uploading to S3.  Is this the best way to go performance wise, or is it possible to directly upload to S3?
I read the following thread, but couldn't get it to work:
https://github.com/libvips/ruby-vips/issues/27
pseudo code:
pages.each_with_index do |page, index|
    page.write_to_file(@page_path)
    upload_page = s3.bucket('mybucket').object("myfile.png")
    upload_page.upload_file("#{upload_path}.png")
end

I guess you want to make a PNG for each page, is that right? You can use write_to_buffer to make a string containing the PNG.
PNG write is very slow, so that will dominate. Probably nothing else you do will make much difference.
The quickest would be:
require 'vips'

# n: is the number of pages to load, -1 means all pages in tall, thin image
image = Vips::Image.pdfload ARGV[0], n: -1

# we can get the number of pages and the height of each page from the metadata
n_pages = image.get 'n-pages'
page_height = image.get 'page-height'

# loop down the image cutting it into an array of separate pages
(0 ... n_pages).each do |page_number|
    page = image.crop(0, page_number * page_height, image.width, page_height)
    puts "rendering page #{page_number} ..."
    png_string = page.write_to_buffer ".png"
    # now send the string containing the formatted PNG image off to S3
end

That'll only work if every page has the same pixel dimensions. If you can have mixed page sizes, you'll need to do:
require 'vips'

image = Vips::Image.pdfload ARGV[0]
n_pages = image.get 'n-pages'

(0 ... n_pages).each do |page_number|
    puts "rendering page #{page_number} ..."
    page = Vips::Image.pdfload ARGV[0], page: page_number
    png_string = page.write_to_buffer ".png"
    # now send the string containing the formagtted PNG image off to S3
end

Which will be slightly slower.

Issue 27 is for two ruby-vips versions ago, it's all changed a lot since then heh.
That version was all in C, then we broke the API for vips8 and redid it with gobject-introspection, and now we've redone it all again for plain ffi and libvips' own introspection system, but with the same API as v2.

You where right about png being slow. Jpg sped things up quite a bit.  Thanks for all the great info, I’ll test it all out this weekend!

libvips 8.9 has a new thing for this: true streaming.
Here's a post introducing it, with Ruby examples:
https://libvips.github.io/libvips/2019/12/11/What's-new-in-8.9.html
You need git master libvips and git master ruby-vips. It should all be released in the next few days.


Add a compartment initalisation framework

Sometimes there is code in a compartment that you want to run once at startup - for example in the configuration broker example each parser needs to register a callback with the broker (to avoid hard coding the list of parers into to the broker).
Running a thread in each compartment to do this is wasteful and since threads aren't expected to exit means they are left hanging around.   Doing it from a single thread means exposing a compartment-call that's not otherwise need.
Following the pattern of compartment_error_handler() with something like compartment_init() would work ?

I agree that something in this space would be nice.  There are a lot of problems that are similar to the ones with shared-library global initialisers.
To start with, this is what we have now:

C++ static initialisers work, so you can lazily initialise variables the first time they enter scope.
We build on this to run global constructors, so it's easy to have code that runs all constructors the first time that a compartment entry point is invoked.
Compartments that are thread entry points can run construction code early.
We can build barriers with futexes so that threads can block until some initialisation is run.

If we have a generic mechanism, a few problems arise immediately:

What order do the constructors run in?  If I have two compartments, A and B, how do I express the idea that B's constructors depend on A's having run and so must happen first?  What should happen in more complex cases where there's circular dependency (ideally, we'd catch that at link time somehow).
What thread do constructors run on and how do we ensure that it is large enough (both stack and trusted stack) for all constructor logic?
What happens if one of the constructors doesn't terminate?  Does this block the boot?  Do we also require another thread as a watchdog that can handle partial startup?
What happens if one compartment's initialisation faults and leaves it in an undefined state?  Related to the first one, what happens if B has constructors that depend on A being initialised, but A faults during construction.

I don't think we can solve this in the general case (happy to be wrong!), but I would welcome suggestions for how we can at least provide some general infrastructure for the common cases.
Currently, the simplest pattern is:

Designate one thread to run constructors and have it run them on startup.
Block all other threads behind a barrier.
Add a policy that says that the initialisation entry points can be called only from the startup compartment.

The thread that runs the cleanup needs one trusted stack frame to handle the startup bits before it enters its main entry point.  It can then define the initialisation order and what to do if any fail.  If you want to handle infinite looping, a second thread can do a timed wait on the barrier and handle errors if the timeout is triggered.  This requires a lot of bespoke code, so is not ideal.


updated PYTHON_INSTALL_DIR generation

Replaces #636.
@ros/ros_team Since this significantly affects the core I would like to have some feedback. Not only but especially on the following thought:

It is a significant change. But it has been a long standing issue blocking some platforms. I would target Kinetic only for this patch.
Invoking Python will be an overhead which we avoided before. I just don't see a better way to make this work consistently across Linux, OS X, Windows, Gentoo / Arch.....
I tried to maintain the semantic of other variables like SETUPTOOLS_ARG_EXTRA.

Hi sorry for the very, late reaction. I tried this on Slackware 14.2 but it still does not seem to be working - on 64 bit OS packages are installed in both lib and lib64 - as mentioned in https://github.com/ros-infrastructure/catkin_pkg/issues/153. Just wanted to make sure I am doing it correctly - I substituted my ~/ros_catkin_ws/src/catkin with the contents of this branch, removed any installed packages to ensure clean build and invoked ./src/catkin/bin/catkin_make_isolated --install -DCMAKE_BUILD_TYPE=Release. Or should I pass other command line arguments too?

I will set the milestone to "untargeted" for now until @wjwwood will have a chance to continue implementing the approach he described above.

@nikonikolov I think that you could instead test the contents of https://github.com/ros/catkin/pull/832 to see if it would work. I'm not sure what I'm doing there would be necessary for your fix or not.

Hey, what's the status of this issue/is it going to be resolved?

I have not had time to implement the fix I described. I don't think I'll have time to do it in the near future.

See: https://github.com/ros/catkin/pull/831#discussion_r84548941 and the following comment for what needs to be done still.

Closing due to inactivity.


Integration Test - Release - 2019/03/21 02:24:06

Auto create for integration test. IssueCommandFlow

@PRMergerTest2 : Thanks for your contribution! The author, @v-caxian, has been notified to review your proposed change.


fix: handle legacy conversation data format and check image in data

Description

The current vision data processing expects in the OAI format and with images. This PR allows passing text-only dataset to training vision models and converts the old sharegpt-like format datasets.
The caveat is that, it still does not allow mixing text-only and text+image data points per batch yet. We would need to patch upstream for this.
Motivation and Context

Could not pass normal sharegpt dataset.
How has this been tested?

Not yet after refactor.
Screenshots (if appropriate)
Types of changes

Social Handles (Optional)

@hjc-puro

would love to have a unit test for this in place.


docs(README): add rustaceanvim

rustaceanvim now has a neotest adapter :smile:

Awesome, and sweet plugin!


Environment variables do not take effect for wsl --exec

Bug description
Currently Nix environment variables do not take effect for wsl --exec. This problem affects VSCode Remote WSL extension badly because by default it uses wsl --exec to start the code server, which requires nix-ld and NIX_LD environment variable to execute properly.
To Reproduce
Steps to reproduce the behavior:

Install last NixOS-WSL release https://github.com/nix-community/NixOS-WSL/releases/tag/22.05-5c211b47
Run wsl --exec nix --version from PowerShell

<3>WSL (1769) ERROR: CreateProcessEntryCommon:570: execvpe nix failed 2
<3>WSL (1769) ERROR: CreateProcessEntryCommon:579: Create process not expected to return

Expected behavior
Show Nix version, which should be 2.8.1
Logs
PS C:\Users\atry> wsl --exec nix
<3>WSL (1769) ERROR: CreateProcessEntryCommon:570: execvpe nix failed 2
<3>WSL (1769) ERROR: CreateProcessEntryCommon:579: Create process not expected to return

PS C:\Users\atry> wsl --exec env
HOSTTYPE=x86_64
LANG=en_US.UTF-8
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/Program Files/ImageMagick-7.1.0-Q16-HDRI:/mnt/c/Python311/Scripts/:/mnt/c/Python311/:/mnt/c/Windows/system32:/mnt/c/Windows:/mnt/c/Windows/System32/Wbem:/mnt/c/Windows/System32/WindowsPowerShell/v1.0/:/mnt/c/Windows/System32/OpenSSH/:/mnt/c/ProgramData/chocolatey/bin:/mnt/c/Program Files/Microsoft VS Code/bin:/mnt/c/Program Files/NVIDIA Corporation/NVIDIA NvDLISR:/mnt/c/Program Files (x86)/NVIDIA Corporation/PhysX/Common:/mnt/c/Program Files/Git/cmd:/mnt/c/Program Files/dotnet/:/mnt/c/Program Files/GitHub CLI/:/mnt/c/Program Files (x86)/sbt/bin:/mnt/c/ProgramData/nvm:/mnt/c/Program Files/nodejs:/mnt/c/tools/gsudo/Current:/mnt/c/Program Files (x86)/Nodist/bin:/mnt/c/Program Files/Java/jdk1.8.0_211/bin:/mnt/c/Program Files/OpenJDK/jdk-19.0.2/bin:/mnt/c/ProgramData/chocolatey/lib/mpv.install/tools:/mnt/c/Users/atry/AppData/Local/Microsoft/WindowsApps:/mnt/c/ProgramData/chocolatey/lib/psutils/tools/psutils-master/:/mnt/c/Users/atry/AppData/Roaming/Python/Scripts
TERM=xterm-256color
XDG_RUNTIME_DIR=/mnt/wslg/runtime-dir
DISPLAY=:0
WAYLAND_DISPLAY=wayland-0
PULSE_SERVER=/mnt/wslg/PulseServer
WSL2_GUI_APPS_ENABLED=1
WSLENV=WT_SESSION::WT_PROFILE_ID
WT_SESSION=4d87b811-c89e-4982-a071-2f47852fd7eb
WT_PROFILE_ID={61c54bbd-c2c6-5271-96e7-009a87ff44bf}
WSL_INTEROP=/run/WSL/1799_interop
NAME=DESKTOP-SICQMTD
HOME=/root
USER=root
LOGNAME=root
SHELL=env
WSL_DISTRO_NAME=NixOS

The workaround for using nix-ld with vscode server is to change remote.WSL2.connectionMethod setting. I added approach the to the wiki page
https://nixos.wiki/wiki/Visual_Studio_Code#:~:text=should "just work".-,Remote WSL,-Similar to SSH
Overall speaking, nix-ld is a more general approach than patching for development environment with package managers other than Nix, because these package managers could download binaries only compatible with FHS.

Does nix-ld remove the need for patching the server? If so, I should give it a shot

Yes

Does https://github.com/msteen/nixos-vscode-server work in WSL? On the project's main page it says it is not working quite well...
For anyone who want a quick (and impure) way to work with vscode, take a look at my solution: https://github.com/sonowz/vscode-remote-wsl-nixos

I switched to nix-ld and it seems to work far better than https://github.com/msteen/nixos-vscode-server. I was using that before and it worked as well, but the connection would sometimes fail after a vscode update. That doesn't happen with nix-ld

When I tried using nix-ld, the websocket connection wouldn't work anymore. Are you using ssh remote or wsl? Also are you connecting through localhost or wsl.exe?

I use the WSL remote with "remote.WSL2.connectionMethod": "wsl2VMAddress"

FYI, as of the 1.77 release of vscode setting remote.WSL2.connectionMethod is ignored and so setting this as a workaround no longer works.
I have raised an issue with vscode for this: https://github.com/microsoft/vscode-remote-release/issues/8305


Error running .\vcpkg integrate install after build

Describe the bug
vcpkg installs from bootstrap fine, but calling .\vcpkg integrate install (or any other vcpkg command) causes an error:
Program 'vcpkg.exe' failed to run: The system cannot find the file specifiedAt line:1 char:1
+ .\vcpkg.exe integrate install

Environment

OS: Windows 10
Compiler: Microsoft (R) C/C++ Optimizing Compiler Version 19.25.28614 for x86

To Reproduce
Steps to reproduce the behavior:

.\bootstrap-vcpkg.bat (installation proceeds fine):

[... enumeration of .cpp files to compile]
  Generating code
  Previous IPDB not found, fall back to full compilation.
  All 17630 functions were compiled because no usable IPDB/IOBJ from previous compilation was found.
  Finished generating code
  vcpkg.vcxproj -> C:\Users\Celso\vcpkg\toolsrc\msbuild.x86.release\vcpkg.exe
  vcpkgmetricsuploader.cpp
  Generating code
  Previous IPDB not found, fall back to full compilation.
  All 6758 functions were compiled because no usable IPDB/IOBJ from previous compilation was found.
  Finished generating code
  vcpkgmetricsuploader.vcxproj -> C:\Users\Me\vcpkg\toolsrc\msbuild.x86.release\vcpkgmetricsuploader.exe
Building vcpkg.exe... done.

Run .\vcpkg integrate install

Program 'vcpkg.exe' failed to run: The system cannot find the file specifiedAt line:1 char:1
+ .\vcpkg integrate install
+ ~~~~~~~~~~~~~~~~~~~~~~~~~.
At line:1 char:1
+ .\vcpkg integrate install
+ ~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ResourceUnavailable: (:) [], ApplicationFailedException
    + FullyQualifiedErrorId : NativeCommandFailed

Additional context
I've tried running these commands from both Developer Command Prompt for VS 2019 and Windows PowerShell. Both instances have been run with Administrator rights (as per the docs). I've tried pulling as well, but everything is up to date.
EDIT:
Once I run .\vcpkg integrate install, the vcpkg.exe disappears from the directory, as if deleted. I have also tried using Developer Command Prompt for VS 2017, to no avail.

@celsomilne, sorry for my late reply.
It seems that vcpkg.exe removed, it might due to false alert in virus detection, could you check that your anti-virus software isn't blocking VCPKG?

@celsomilne, please reopen this issue if it still be a problem for you.


Fix typo in triangle and use floats

is_isosceles was spelt wrongly.
Canonical data now uses floats for sides, so changing
to match that.

The canonical data implies the language has functions which overload numerics, taking ints or floats. I don't think that's appropriate for this exercise.


Outage: Getting a 400 from API on launch endpoints!

body:
getaddrinfo ENOTFOUND mongo

Same here!
Appreciate the help.

Same here on launches!

v4 Endpoints been down for quite a few hours already.
Appreciate the help on restoring them!

Should be resolved now, the VM ran out of storage space at the worst possible time.
Apologies everyone!

Thank you! Really appreciate it.


GEOPM_SHMKEY environment variable feature should be removed

estimate 1

This feature is confusing at best, and may cause failures in the attach process.  We have resolved issues around stale keys that this feature was originally a work around for.

@cmcantalupo Do we remove the GEOPM_SHMKEY environment variable only, or do we also remove the --geopm-shmkey and all features pertaining to shared memory keys?

--geopm-shmkey key
Specify a prefix to be used with all the shared memory keys
generated by the GEOPM runtime for communication with the
application.  This prefix is not used for the endpoint.
This is useful for avoiding collisions with keys
that were not properly cleaned up.  The default key prefix
is "geopm-shm".  A shared memory key must have no
occurrences of the '/' character.  The base key is used as
the prefix for each shared memory region used by the
runtime.  If the keys are left behind, a simple command to
clean up after an aborted job is:
$ test -n "$GEOPM_SHMKEY" && rm -f /dev/shm${GEOPM_SHMKEY}* || rm -f /dev/shm/geopm-shm*

This option is used by the launcher to set the
GEOPM_SHMKEY environment variable.  The command line
option will override any value currently set in the
environment

This option is used by the launcher to set the GEOPM_SHMKEY environment variable. The command line option will override any value currently set in the environment

If we want to remove the GEOPM_SHMKEY environment variable, then it stands that we also want to remove the --geopm-shmkey command line option?


from moving from 3.x to 4.x set data source not working

if (!$scope.gridOptions.datasource) { // must attach datasource only once
        $scope.gridOptions.datasource = $scope.dataSource;
        console.log($scope.gridOptions)
        $scope.gridOptions.api.onNewDatasource();
      }

i'm getting an error
build.min.js:25996 TypeError: Cannot read property 'onNewDatasource' of undefined
    at init_datasource (build.min.js:131911)
    at new SearchResultProjectsCtrl (build.min.js:131633)
    at Object.invoke (build.min.js:17364)
    at extend.instance (build.min.js:22615)

to note that i did not need the  $scope.gridOptions.api.onNewDatasource(); when i was on 3.0.3
just updated now and went through documentation and found this change.
without the .api.onNewDatasource(); the issue was that nothing was displayed on the screen

onNewDatasource() is not used in the new version. please check the docs.

In the Datasource section of the docs it says this is possible way to add datasource.
// way 1 - set the the datasource using the API
gridOptions.api.setDatasource(myDataSource);

// way 2 - set the the datasource directly and inform the grid a new datasource is there
gridOptions.datasource = myNewDatasource;
gridOptions.api.onNewDatasource();

sorry for this. docs are out of date. i've fixed and they will be updated with the next release.

Could you please share the solution with us?

use:
// way 1 - set the the datasource using the API
gridOptions.api.setDatasource(myDataSource);


Add server path to serve frontend

We lost a valuable production debugging tool with bde4d925f21fed2ddf87e9164e6c1ea7d3073d38.
Indeed, to determine if a problem was specific to a given server, or to put out a canari for a sensitive patch, I often deploy only with a single server (eg. make deploy-s0) and hit that server's frontend (eg. https://vps71670.vps.ovh.ca/try.html).
Because of the bde4d92 change, this is no longer possible, as /try.html no longer exists and / gets redirected to shields.io.
Indeed, infosite is not / in production (otherwise we would not be redirected to shields.io when hitting img.shields.io, which we want).
@paulmelnikow What do you think we can do?

Yea, that makes a lot of sense. I'll give it some thought, and either push a fix or leave a PR open for you to test.

I opened #1222 which I think should fix this. It creates a dev build of the frontend, which is like the old try.html. The badge URIs do not have hostnames nor a maxAge parameter. The deploy tasks will push this to the servers. I tested the output of make website, npm run build, and npm run build:production and they all seemed right. However I couldn't actually run the deploy tasks.
It also addresses a request you made offline, which is to enable local testing with production parity – i.e. with maxAge. To do that, run npm run build:production instead of npm run build. npm run build:production no longer puts https://img.shields.io in the generated frontend, so the badges will be generated by the local server.


python312Packages.pipenv-poetry-migrate: 0.5.7 -> 0.5.8

Automatic update generated by nixpkgs-update tools. This update was made based on information from passthru.updateScript.
meta.description for python312Packages.pipenv-poetry-migrate is: This is simple migration script, migrate pipenv to poetry
meta.homepage for python312Packages.pipenv-poetry-migrate is: https://github.com/yhino/pipenv-poetry-migrate
meta.changelog for python312Packages.pipenv-poetry-migrate is: https://github.com/yhino/pipenv-poetry-migrate/blob/v0.5.8/CHANGELOG.md
Updates performed

Ran passthru.UpdateScript

To inspect upstream changes
Impact
Checks done

built on NixOS
The tests defined in passthru.tests, if any, passed
found 0.5.8 with grep in /nix/store/b35byngbqx1djzkrjqwn1kz05q6iyb38-python3.12-pipenv-poetry-migrate-0.5.8
found 0.5.8 in filename of file in /nix/store/b35byngbqx1djzkrjqwn1kz05q6iyb38-python3.12-pipenv-poetry-migrate-0.5.8

Rebuild report (if merged into master) (click to expand)

3 total rebuild path(s)

3 package rebuild(s)

First fifty rebuilds by attrpath

python311Packages.pipenv-poetry-migrate
python312Packages.pipenv-poetry-migrate

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/b35byngbqx1djzkrjqwn1kz05q6iyb38-python3.12-pipenv-poetry-migrate-0.5.8 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A python312Packages.pipenv-poetry-migrate https://github.com/r-ryantm/nixpkgs/archive/679ad72d20fc5ec0be846cd06806eb4fc636cebd.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/679ad72d20fc5ec0be846cd06806eb4fc636cebd#python312Packages.pipenv-poetry-migrate

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/b35byngbqx1djzkrjqwn1kz05q6iyb38-python3.12-pipenv-poetry-migrate-0.5.8
ls -la /nix/store/b35byngbqx1djzkrjqwn1kz05q6iyb38-python3.12-pipenv-poetry-migrate-0.5.8/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

4 packages built:

python311Packages.pipenv-poetry-migrate
python311Packages.pipenv-poetry-migrate.dist
python312Packages.pipenv-poetry-migrate
python312Packages.pipenv-poetry-migrate.dist

Maintainer pings
cc @gador for testing.

Result of nixpkgs-review pr 328827 run on aarch64-darwin 1

4 packages built:

python311Packages.pipenv-poetry-migrate
python311Packages.pipenv-poetry-migrate.dist
python312Packages.pipenv-poetry-migrate
python312Packages.pipenv-poetry-migrate.dist

Result of nixpkgs-review pr 328827 run on x86_64-linux 1

4 packages built:

python311Packages.pipenv-poetry-migrate
python311Packages.pipenv-poetry-migrate.dist
python312Packages.pipenv-poetry-migrate
python312Packages.pipenv-poetry-migrate.dist


fix(Header + About + Newsletter + Footer): modified Footer layout and…

… modified responsive layout
for other components

:tada: This PR is included in version 1.0.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


how webhook works

hi all, reading the documentation I can't understand how the webhook works. Once the payment process has been done, I need to redirect the user to a page, but from here on it's not clear to me how I can take the data and extrapolate it. I have already set up the webhook link from the panel. thank you

Dear PeraDev7
If you just want the existing data of a transaction on the success page, I recommend you to pass the transaction ID to the success page and read from there the transaction:
// e.g. Success URL: www.your-shop.com/order/successful?transaction_id=5
$transactionId = htmlspecialchars($_GET['transaction_id']);
$transaction = $apiClient->getTransactionService()->read($spaceId, $transaction->getId());
// with e.g. $transaction->getBillingAddress()->getFamilyName(); you can get then the data out of it

But it can take longer till the transition between authorize to fulfill is done (for example in deferred mode for credit cards).
This information will be transmitted via Webhooks to your shop system.
The URL you define for your Webhooks needs to have a script behind, that is fetching each incoming request (Webhook) from the wallee portal.
Workaround to get transaction updates by Webhooks:

First, the client uses a URL with the destination address where he wants to receive the webhooks (to test you can use Webhook.site): https://app-wallee.com/s/your-space/webhook/url/list

Then the client creates a listener with the type "Transaction" and with the statuses at which the webhooks should be sent: https://app-wallee.com/s/your-space/webhook/listener/list

At the defined target URL of the customer, the customer can intercept the webhooks. There is the entity ID (transaction ID included).

To get the transaction status and other transaction information, the customer must call the following method with the entity and space ID: https://app-wallee.com/api/client/transaction-service--read

Does this answer your questions?
Best regards

Dear PeraDev7
You can do it this way with TransactionPending as a step after the create() and before calling the payment page url:
$transaction = $this->client->getTransactionService()->create($this->spaceId, $transactionPayload);

  $transactionPending = new \Wallee\Sdk\Model\TransactionPending([
      "id" => $transaction->getId(), // id of created transaction
      "version" => 1
  ]);

$transactionPending->setSuccessUrl("https://yourshop.com/order/success.php?transaction_id="
. $transaction->getId());
$transactionPending->setFailedUrl("https://yourshop.com/order/failed.php?transaction_id="
. $transaction->getId());

$this->client->getTransactionService()->update($this->spaceId, $transactionPending); // update created transaction

$redirectionUrl =
$this->client->getTransactionPaymentPageService()->paymentPageUrl($this->spaceId, $transaction->getId());

Best regards

Thank you!


Fix heading

Pull request description
Fix heading not rendering correctly.
Separately the sentence doesn’t seem right, like something was accidentally deleted, but I’m not sure what it was. I can tweak that too if needed if you let me know what it is.
PR meta checklist

[x] Pull request is targeted at main branch for code
or targeted at stable branch for documentation that is live on bunit.dev.
[ ] Pull request is linked to all related issues, if any.
[ ] I have read the CONTRIBUTING.md document.

Code PR specific checklist

[ ] My code follows the code style of this project and AspNetCore coding guidelines.
[ ] My change requires a change to the documentation.

[ ] I have updated the documentation accordingly.

[ ] I have updated the appropriate sub section in the CHANGELOG.md.
[ ] I have added, updated or removed tests to according to my changes.

[ ] All tests passed.

I still need to fix this properly. I realise now the text is part of the heading, but it's the heading itself is broken for some reason. I'll need to do that when I'm back at my laptop with a decent Markdown editor.

Hey @martincostello
Thanks for pointing that out. The simple fix is the wrong whitespace between ## and the title. The reason is: Me :D
On mac if you use Command+Space it does something like a whitespace, but it isn't printable via Markdown.
Anyway - the fix belongs to you ;)

Thanks - that would have taken me ages to puzzle out 😆

LGTM and big thanks @martincostello !
As soon as the CI pipeline says: 👍 I will merge it.

Thanks - that would have taken me ages to puzzle out 😆

Imagine the first time I stumbled across this "feature" :D

@egil Disabling the "ghost" workflows didn't help, unfortunately.
By the power of Greyskull: Can you force-merge?


Pick up modifying user from controller method (e.g. "current_user")

Is it possible, or are there any plans to make it possible, to pick up the modifying user from the controller (if there is one), like acts_as_audited does? Seems fairly easy to implement. If not, maybe I'll have a go...

You mean wherever you create/update/destroy a tracked record, you explicitly pass in the user with the attributes?

Yes, we just assign that first, thing.updated_by = current_user followed by thing.save! and thing has a   track_history :modifier_field => :updated_by

Hmm yeah, it's annoying to have to add the every time. Well I am working on something better now - basically just copying what acts_as_audited does. The problem is Rails sweepers only work with ActiveRecord for some reason (not sure why they don't use ActiveModel), but it's not hard to make something similar. I'll update in here if/when we have something working on our fork.

Closing this in favour of issue #19.

#19 merged and release 0.1.5


Segfault with one.vn.reserve method. Potential DoS.

Vulnerability Details
Sometimes calling one.vn.reserve method causes segfault.
To Reproduce
Choose any network with free IP
NETWORK_ID=0 # any existing network id
REQUEST="$(cat <<EOT
<?xml version="1.0"?>
<methodCall>
    <methodName>one.vn.reserve</methodName>
    <params>
        <param><value>$(cat /var/lib/one/.one/one_auth)</value></param>
        <param><value><i4>$NETWORK_ID</i4></value></param>
        <param><value><string>SIZE="10000" NAME="test-netwok-$[RANDOM%100000]"</string></value></param>
    </params>
</methodCall>
EOT
)"

for i in $(seq 1 100); do
  curl -sS -k "${ONE_XMLRPC:-http://localhost:2633/RPC2}" --data "$REQUEST" || break
done

after few more attempts, oned dies with segfault:
[5725434.956988] oned[18051]: segfault at 1c0 ip 00007ffff5ed62b0 sp 00007ffff7fd1768 error 4 in libstdc++.so.6.0.25[7ffff5db1000+179000]

stack trace using gdb:
<F28>#0  0x00007ffff61e0970 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::empty() const () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#1  0x000000000057f578 in VirtualNetworkPool::set_vlan_id (this=0xb4ce50, vn=0x0) at src/vnm/VirtualNetworkPool.cc:409
#2  0x000000000057e2fb in VirtualNetworkPool::allocate (this=0xb4ce50, uid=0, gid=0, uname="oneadmin", gname="oneadmin", umask=127, pvid=0, vn_template=0x7fff986a2460, oid=0x7ffff7f4e918, cluster_ids=std::set with 2 elements = {...},
    error_str="") at src/vnm/VirtualNetworkPool.cc:154
#3  0x00000000004b9964 in VirtualNetworkReserve::request_execute (this=0xb678e0, paramList=..., att=...) at src/rm/RequestManagerVirtualNetwork.cc:375
#4  0x00000000004e8f77 in Request::execute (this=0xb678e0, _paramList=..., _callInfoP=0x7ffff7f4f660, _retval=0x7ffff7f4f400) at src/rm/Request.cc:422
#5  0x0000000000649263 in xmlrpc_c::c_executeMethod (envP=0x7ffff7f4f520, paramArrayP=0x7fff98758520, methodPtr=0xb678e0, callInfoPtr=0x7ffff7f4f660) at registry.cpp:281
#6  0x000000000064d0e9 in callNamedMethod (resultPP=0x7ffff7f4f550, callInfoP=0x7ffff7f4f660, paramArrayP=0x7fff98758520, methodP=<optimized out>, envP=0x7ffff7f4f520) at registry.c:307
#7  xmlrpc_dispatchCall (envP=envP@entry=0x7ffff7f4f520, registryP=registryP@entry=0xb631f0, methodName=0x7fff984ad790 "one.vn.reserve", paramArrayP=0x7fff98758520, callInfoP=callInfoP@entry=0x7ffff7f4f660,
    resultPP=resultPP@entry=0x7ffff7f4f550) at registry.c:337
#8  0x000000000064d22c in xmlrpc_registry_process_call2 (envP=0x7ffff7f4f5e0, registryP=0xb631f0,
    callXml=0x7fff987081f0 "<?xml version=\"1.0\"?> \n<methodCall>\n    <methodName>one.vn.reserve</methodName>\n    <params>\n        <param><value>oneadmin:redacted</value></param>\n        <param><value><i4>0</i4></value"..., callXmlLen=330, callInfo=0x7ffff7f4f660, responseXmlPP=0x7ffff7f4f5d8) at registry.c:426
#9  0x0000000000649db3 in xmlrpc_c::registry::processCall (this=0xb62c98,
    callXml="<?xml version=\"1.0\"?> \n<methodCall>\n    <methodName>one.vn.reserve</methodName>\n    <params>\n        <param><value>oneadmin:redacted</value></param>\n        <param><value><i4>0</i4></value"...,
    callInfoP=0x7ffff7f4f660, responseXmlP=0x7ffff7f4f710) at registry.cpp:524
#10 0x0000000000647bbd in xmlrpc_c::serverAbyss_impl::processCall (this=0x7fffbc0009c0,
    call="<?xml version=\"1.0\"?> \n<methodCall>\n    <methodName>one.vn.reserve</methodName>\n    <params>\n        <param><value>oneadmin:redacted</value></param>\n        <param><value><i4>0</i4></value"...,
    abyssSessionP=0x7ffff7f4f860, responseP=0x7ffff7f4f710) at server_abyss.cpp:789
#11 0x0000000000646d4c in xmlrpc_c::processXmlrpcCall (envP=0x7ffff7f4f7c0, arg=0x7fffbc0009c0,
    callXml=0x7fff984354a0 "<?xml version=\"1.0\"?> \n<methodCall>\n    <methodName>one.vn.reserve</methodName>\n    <params>\n        <param><value>oneadmin:redacted</value></param>\n        <param><value><i4>0</i4></value"..., callXmlLen=330, abyssSessionP=0x7ffff7f4f860, responseXmlPP=0x7ffff7f4f7b8) at server_abyss.cpp:381
#12 0x000000000064c82c in processCall (trace=<optimized out>, accessControl=..., wantChunk=false, xmlProcessorArg=0x7fffbc0009c0,
    xmlProcessor=0x646c9d <xmlrpc_c::processXmlrpcCall(xmlrpc_env*, void*, char const*, size_t, TSession*, xmlrpc_mem_block**)>, contentSize=<optimized out>, abyssSessionP=0x7ffff7f4f860) at abyss_handler.c:392
#13 handleXmlRpcCallReq (requestInfoP=<optimized out>, accessControl=..., wantChunk=<optimized out>, xmlProcessorArg=0x7fffbc0009c0,
    xmlProcessor=0x646c9d <xmlrpc_c::processXmlrpcCall(xmlrpc_env*, void*, char const*, size_t, TSession*, xmlrpc_mem_block**)>, abyssSessionP=0x7ffff7f4f860) at abyss_handler.c:509
#14 xmlrpc_handleIfXmlrpcReq (handlerArg=<optimized out>, abyssSessionP=0x7ffff7f4f860, handledP=<optimized out>) at abyss_handler.c:573
#15 0x00000000006545e0 in runUserHandler (srvP=0x7fffbc000a60, sessionP=0x7ffff7f4f860) at server.c:631
#16 processRequestFromClient (connectionP=connectionP@entry=0x7fffbc000d70, lastReqOnConn=lastReqOnConn@entry=false, timeout=<optimized out>, keepAliveP=keepAliveP@entry=0x7ffff7f4f9ac) at server.c:724
#17 0x000000000065484d in serverFunc (userHandle=0x7fffbc000d70) at server.c:796
#18 0x0000000000659da7 in connJob (userHandle=0x7fffbc000d70) at conn.c:39
#19 0x0000000000659627 in execute (arg=0x7fffbc001de0) at thread_pthread.c:59
#20 0x00007ffff644a6ba in start_thread (arg=0x7ffff7f50400) at pthread_create.c:333
#21 0x00007ffff58df41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Expected behavior
Don't die. Handle all the errors.
Details

Affected Component: Core
Hypervisor: KVM
Version: 5.6.1 (tested on xenial and bionic)

Additional context
Finaly, after restart, this reservation network will be created with 0 leases, but during removing it is showing an error and disappears:
[one.vn.delete] Cannot delete virtual network. Network ID is not part of the cluster set.

Progress Status

[ ] Branch created
[ ] Code committed to development branch
[ ] Testing - QA
[ ] Documentation
[ ] Release notes - resolved issues, compatibility, known issues
[ ] Code committed to upstream release/hotfix branches
[ ] Documentation committed to upstream release/hotfix branches

Problem occurs only with Percona Xtradb Cluster
sqlite and mysql with tokudb engine is wotking without this problem

It may be an issue with Percona Xtradb Cluster, but oned should behave nicer. Thanks for the backtrace, we have identified the corner case that may lead to this. (It is actually assuming a DB behavior that is not follow by Percona, go figure...) We are fixing it.
BTW, I've updated the subject of the issue to better describe the issue.
Thanks.


Subgraph syncing "reading file existence" error

Another interesting error for @sduchesneau
First time seeing it, seems to recover from it, so probably no big deal.

Text:
An error occurred while streaming blocks: status: Internal, message: "reading file existence: Get \"https://storage.googleapis.com/storage/v1/b/dfuseio-global-blocks-uscentral/o/eth-mainnet%2Fv4%2F0016883700.dbin.zst?alt=json&prettyPrint=false&projection=full&userProject=dfuseio-global\": stream error: stream ID 2939; INTERNAL_ERROR; received from peer", details: [], metadata: MetadataMap { headers: {} }, component: SubstreamsBlockStream

It is a transient error, it can be ignored.
It is not good that the details of the error bubble up to the user (it leaks infrastructure details..), so I will change that. But if the storage starts to fail, it is the wanted behavior that it fails the request (and not silently retry..)

Closing as this is the correct behavior when the storage backend gives an error and will be fixed automatically when the storage works again.
The other errors (old cursor...) are discussed thoroughly here https://github.com/streamingfast/substreams/issues/196


Fixed #61

Resolves an issue, where the locale.url parameter would always show the taxonomy slug in the default locale.
Fixes: #61
@SahAssar Could you have a quick look and merge it, if you're OK with the fix?

LGTM :+1:


update: migrate everyone from linuxbrew-core to homebrew-core

[ ] Have you followed the guidelines in our Contributing document?
[ ] Have you checked to ensure there aren't other open Pull Requests for the same change?
[ ] Have you added an explanation of what your changes do and why you'd like us to include them?
[ ] Have you written new tests for your changes? Here's an example.
[ ] Have you successfully run brew style with your changes locally?
[ ] Have you successfully run brew typecheck with your changes locally?
[ ] Have you successfully run brew tests with your changes locally?

@iMichka Agreed. I'd like GITHUB_ACTIONS to land sooner rather than later, though, because it's a large proportion of our usage.

Instead of forcing a migration onto GitHub Actions, we really should update the default install remote in Homebrew/install and allow GitHub to ship new images that use this remote (since new image versions are clean built). That means migration will not be required at all from GitHub Actions.

We don't have control over when GitHub does that. We can ask but it's unlikely that they will before this is in a new tag. You'll also only end up on this commit if HOMEBREW_DEVELOPER and/or homebrew.devcmdrun are set.

The docker image used by the CI is built by this dockerfile (https://github.com/Homebrew/brew/blob/master/Dockerfile), right? It does not use install.sh, so not sure how this is going to change the images we ship.

Anything that doesn't reference Homebrew/linuxbrew-core specifically but does homebrew-core will change after this PR. This means these images will be updated but not for the tagged ones.

Hi, I'm very glad to see this migration to let users have unified settings for Homebrew on both macOS and Linux. May I ask is there any migration plan for formula bottles (HOMEBREW_BOTTLE_DOMAIN)?
https://github.com/Homebrew/brew/blob/ff2985c6aa4a22cc8773e7ed385875ca6e1b99a7/Library/Homebrew/env_config.rb#L41-L52
https://github.com/Homebrew/brew/blob/ff2985c6aa4a22cc8773e7ed385875ca6e1b99a7/Library/Homebrew/env_config.rb#L87-L92

Tested and the migration is not done anymore:
$ docker run -it homebrew/ubuntu16.04
$ cd /home/linuxbrew/.linuxbrew/Homebrew
$ git remote add iMichka https://github.com/iMichka/brew
$ git fetch iMichka
$ git checkout migrate2
$ brew update
HOMEBREW_CORE_GIT_REMOTE set: using https://github.com/Homebrew/linuxbrew-core for Homebrew/core Git remote.
remote: Enumerating objects: 42040, done.
remote: Counting objects: 100% (19560/19560), done.
remote: Compressing objects: 100% (2501/2501), done.
remote: Total 42040 (delta 17245), reused 19369 (delta 17059), pack-reused 22480
Receiving objects: 100% (42040/42040), 14.15 MiB | 1.53 MiB/s, done.
Resolving deltas: 100% (32291/32291), completed with 1831 local objects.
From https://github.com/Homebrew/linuxbrew-core
   70d37801899..5be975b7335  master     -> origin/master
...
Migrating formulae from linuxbrew-core to homebrew-core
...

The remote stays on linuxbrew-core.

Ah I just noticed you said this broke the syntax job before? Sorry, wasn't following very closely.

Tested on my side, it worked. Thanks @MikeMcQuaid for all the cleanup! I feel free now :)


Regular Expression Fails to Match Newline-Starting Action Inputs

System Info
LangChain version: 0.0.189
OS: Ubuntu 22.04.2
Python version: 3.10.7
Who can help?
@vowelparrot
Information

[ ] The official example notebooks/scripts
[X] My own modified scripts

Related Components

[ ] LLMs/Chat Models
[ ] Embedding Models
[ ] Prompts / Prompt Templates / Prompt Selectors
[ ] Output Parsers
[ ] Document Loaders
[ ] Vector Stores / Retrievers
[ ] Memory
[X] Agents / Agent Executors
[X] Tools / Toolkits
[ ] Chains
[ ] Callbacks/Tracing
[ ] Async

Reproduction
Reproducing this issue can be challenging as it hinges on the action input generated by the LLM. Nevertheless, the issue occurs when using an agent of type CONVERSATIONAL_REACT_DESCRIPTION. The agent's output parser attempts to extract both the tool's name that the agent needs to utilize for its task and the input to that tool based on the LLM prediction.
In order to accomplish this, the agent's output parser employs the following regular expression on the LLM prediction: Action: (.*?)[\n]*Action Input: (.*). This expression divides the LLM prediction into two components: i) the tool that needs to be utilized (Action) and ii) the tool's input (Action Input).
However, the action input generated by an LLM is an arbitrary string. Hence, if the action input initiates with a newline character, the regular expression's search operation fails to find a match, resulting in the action input defaulting to an empty string. This issue stems from the fact that the . character in the regular expression doesn't match newline characters by default. Consequently, (.*) stops at the end of the line it started on.
This behavior aligns with the default functionality of the Python re library. To rectify this corner case, we must include the re.S flag in the re.search() method. This will allow the . special character to match any character, including a newline.

re.S
re.DOTALL
Make the '.' special character match any character at all, including a newline; without this flag, '.' will match anything except a newline. Corresponds to the inline flag (?s).

Expected behavior
The re.search() method correctly matches the action input section of the LLM prediction and sets it to the relevant string, even if it starts with the new line character.
I am prepared to open a new Pull Request (PR) to address this issue. The change involves modifying the following line:
From:

match = re.search(regex, text)

To:

match = re.search(regex, text, re.S)

Hi - I am running into this as well. Any news on a fix/


Use HTML5 source as download URL

Description
Allows fallback videos to be used for download, when the main video URL is not downloadable (e.g., HLS .m3u8).
JIRA
OSPR-2019
Allows the streaming video URL to differ from the download URL.  Iterates through the all_sources HTML5 video URLs to locate a downloadable video URL, if the streaming video URL is not downloadable.
Notes
Functionality depends on API changes in https://github.com/edx/edx-platform/pull/16590, but the app continues to work even if this change is not deployed yet.
Builds on https://github.com/edx/edx-app-ios/pull/1014, which differentiates between streamable and downloadable video URLs.
We'd like to get both this and https://github.com/edx/edx-app-ios/pull/1014 merged before the next edx-app-ios release.
How to test this PR
Ensure that the changes from https://github.com/edx/edx-platform/pull/16590 are running in your LMS, and import this video_test_course.tar.gz, which contains permutations of all the various video streaming/download scenarios.
Alternately, run against this sandbox server, using:
API_HOST_URL: 'https://pr16590.sandbox.opencraft.hosting'
OAUTH_CLIENT_ID: '14cdc120d61349b7eeba'

Run the iOS app.
Login as a student to the test course.
Visit each of the course subsections in turn, noting the expected Streaming and Download behaviours for each unit below.
note Once you've downloaded a video, that video will be used in the player, regardless of what the original streaming video URL is.  So to test the permutations below, try streaming first, then download.

YouTube:

Standard: YouTube

Streaming: YouTube only; will not play on iOS app, but will play in browser.
Download: none

Standard w Fallback:

Streaming: 0.05 mp4 (same as MP4 - 1), different from YouTube video
Download: yes, same as stream

HLS live stream:

Streaming: YouTube only; will not play on iOS app, but will play in browser.
Download: none

HLS w fallback:

Streaming: 0.05 mp4 (same as MP4 - 1)
Note The main video is configured to use the same YouTube HLS used in the previous unit, but USING_VIDEO_PIPELINE: false causes the fallback URL to be streamed instead.
Download: Same as stream.

MP4

MP4 - 1

Streaming: 0.05 mp4
Download: yes, same as stream

MP4 - 1 w fallback

Streaming: 0.05 mp4
Download: yes, same as stream
Note Fallback video is configured to be the 0:27 mp4 used by MP4 - 2, but because the main video
can be downloaded, the fallback video is ignored.

MP4 - 2

Streaming: 0:27 mp4
Download: yes, same as stream

MP4 - 2 w fallback

Streaming: 0:27 mp4
Download: yes, same as stream
note Fallback video is configured to be the 0:05 mp4 used by MP4 - 1, but because the main video
can be downloaded, the fallback video is ignored.

HLS .m3u8

HLS (10 min):

Streaming: 10:35 video
Download: none

HLS (10 min) w fallback:

Streaming: 10:35 video
Download: 0:27 mp4, same as MP4 - 2

Vimeo

Streaming: 8:22 Cloudera m3mu
Download: none

Vimeo w fallback

Streaming: 8:22 Cloudera m3mu
Download: 8:22 Cloudera mp4

Reviewers

[x] @bradenmacdonald
[ ] edX TBD

Thanks for the pull request, @pomegranited! It looks like you're a member of a company that does contract work for edX. If you're doing this work as part of a paid contract with edX, you should talk to edX about who will review this pull request. If this work is not part of a paid contract with edX, then you should ensure that there is an OSPR issue to track this work in JIRA, so that we don't lose track of your pull request.
Create an OSPR issue for this pull request.

Thanks for the pull request, @pomegranited! I've created OSPR-2019 to keep track of it in JIRA. JIRA is a place for product owners to prioritize feature reviews by the engineering development teams.
Feel free to add as much of the following information to the ticket:

supporting documentation
edx-code email threads
timeline information ("this must be merged by XX date", and why that is)
partner information ("this is a course on edx.org")
any other information that can help Product understand the context for the PR

All technical communication about the code itself will still be done via the GitHub pull request interface. As a reminder, our process documentation is here.
If you like, you can add yourself to the AUTHORS      file for this repo, though that isn't required.     Please see the CONTRIBUTING     file for more information.

@saeedbashir cf your reply from https://github.com/edx/edx-app-ios/pull/1014:
I was unable to complete the conversion to using AVFoundation to allow HLS downloads.
AVFoundation uses AVAssets, which obscure the downloaded file locations, and so our existing video resource management logic and the video player would need to be re-worked.
So instead, I offer this change, which allows clients who are not using the video processing service (VAL) to provide downloadable fallback video URLs in their courses.

@edx/mobile-product Please take a look.
@edx/ios-devs This will need engineering approval, thanks.

@nedbat We have some other high priority tasks on our plates. Will get back to it soon.

Thanks for the update @saeedbashir !

@pomegranited The download icon are showing with videos on section level which are not downloadable.

@salman2013

The download icon are showing with videos on section level which are not downloadable.
  Are these pipeline videos?  What are the primary and secondary URLs?

Section Level

SubSection Level

The download Progress View

@pomegranited you can see in the attached screenshots on section level how the download icon are showing for videos which are not in supported formats.
For videos encodings edX is using the Videos pipeline and only supporting  mobile_high and mobile_low videos encoding for mobile.
For the open edX community we are providing support for fallback video encoding with flag  "USING_VIDEO_PIPELINE". If the "USING_VIDEO_PIPELINE" is disabled only then fallback can be considered. It would be great if you can check behaviour of your PR by enabling "USING_VIDEO_PIPELINE", because on edX we are using this flag and not supporting fallback encoding anymore.
Now there are two things which are happening

First is by enabling the pipeline it would not support the .mp4 and fallback extension so if the video url is of .m3u8 extension then the code should support that which is not working.
Second is when we enable the pipeline flag the download icons are showing at section level which should not because the videos format are not supported.

What are the primary and secondary URLs?

Can you please elaborate it?

Thank you for your review, @salman2013 !  I don't have any experience with the video pipeline, so please see aef2651 and let me know if I've misinterpreted.

What are the primary and secondary URLs?
Can you please elaborate it?

I did not know if you were using my test course, but your screenshots answered my question.  Thank you!

@pomegranited Please remove the conflicts. Thanks

@salman2013 I've rebase on master to resolve the conflict, and addressed your suggestion above.  Is it ok for me to squash?  Any more changes required?
Thank you!

I've rebase on master to resolve the conflict, and addressed your suggestion above. Is it ok for me to squash? Any more changes required?

@pomegranited I have add few more comments to make some changes. Thanks

Thanks @salman2013 !  Is it ok for me to squash now?

@saeedbashir Please take a look into PR as it looks fine to me. Thanks

Thanks @salman2013 !
FYI @saeedbashir , I've squashed the commits from c79c747 into 6585855, so this is ready to merge whenever you approve.

@pomegranited Sure it will also help in reviewing. What I observer, HLS videos are not supported if USING_VIDEO_PIPELINE flag is true. We want to support streamable HLS videos in all cases.

Thanks @saeedbashir -- I thought I fixed these issues in a previous commit (see https://github.com/edx/edx-app-ios/pull/1033#issuecomment-350722529).
Is it still not working correctly?  Do you mind to send me a course export so I can test on my devstack with the course and videos you're using?

Yes, unfortunately, it still has some issues. I was using the sandbox that is mentioned in PR description.
This is what I got after setting the value of USING_VIDEO_PIPELINE to true.

@saeedbashir Ah ok.. let's see if I understand correctly:
To find a downloadable video URL:

First, check the supported encodings returned by the Course Blocks API.
Second, check the streamable videoURL.
Finally, check the all_sources list, as returned by the Course Blocks API.

So then, the only thing that the USING_VIDEO_PIPELINE setting does is change whether "fallback" is in the a supported encodings list, which happens only when USING_VIDEO_PIPELINE: False.
Does this sound correct?

Yup and in case of USING_VIDEO_PIPELINE: true we will be doing the same but only for streamable videos. Even if some videos have both streamable and download URLs in  streamable videoURL or all_sources then we will be considering only streamable URL.

@pomegranited You don't need to squash because we use git squash while merging PR into master.

@saeedbashir

Yup and in case of USING_VIDEO_PIPELINE: true we will be doing the same but only for streamable videos. Even if some videos have both streamable and download URLs in streamable videoURL or all_sources then we will be considering only streamable URL.

Ok cool..
How do I determine which video URLs are "streamable"?
Is this handled by not including the "fallback" encoding in the supported encodings list?
Or is it handled by the URL extension check we're doing here with VIDEO_URL_EXTENSION_OPTIONShttps://github.com/edx/edx-app-ios/pull/1033/files#diff-be681ce662f3fff24b6cc423333815e2R24) and ONLINE_ONLY_VIDEO_URL_EXTENSIONS

We need to include fallback encoding here only for the iteration.

How do I determine which video URLs are "streamable"?

Any URL which has the extension .m3u8 is a streamable URL.

@saeedbashir

Any URL which has the extension .m3u8 is a streamable URL.

This change doesn't actually add support for downloading .m3m8 videos.
Because .m3m8 URLs are really playlists, not videos themselves, you need a special handler to allow them to be downloaded.  This change was deemed to be too difficult for us this round, because it required converting to using the AVFoundation library to provide the video downloads as AVAssets, and to provide the streaming player.
That's why we introduced this additional ONLINE_ONLY_VIDEO_URL_EXTENSIONS , which indicates which URLs can only be streamed when online, and cannot be downloaded.
So instead, we're allowing course authors to provide alternate, downloadable video URLs when they're using .m3m8 videos for streaming, so that at least there's an option for mobile users to download the course videos.

Sorry I wasn't clear. By streamable it was meant 'ONLINE PLAYABLE' videos. Let me make it easy for you with an example.
Let's consider video 'HLS (10 min) w fallback' it has URLs with .mp4 and .m3u8
A user will be able to play and download the video if USING_VIDEO_PIPELINE: false, but with USING_VIDEO_PIPELINE: true user will only be able to play it online.
Do let me know if it is still confusing?

@saeedbashir Isn't that what you're seeing in the screenshot you sent above?  There's no "download" link shown for 'HLS (10 min) w fallback', because the fallback .mp4 URL is allowed when the app checks for downloadable videos.  But the HLS video can be streamed, correct?

@pomegranited This is how it should behave after enabling/disabling USING_VIDEO_PIPELINE

Thanks @saeedbashir !  I can see where I was going wrong now.  Thank you for your patience -- plan to have an update for you in the next 24h.

@saeedbashir bb9b3fd fixes the issues with the streaming behaviour for USING_VIDEO_PIPELINE: true that you noted in https://github.com/edx/edx-app-ios/pull/1033#issuecomment-354066987.  I've also:

added tests for the expected isSupportedVideo and isDownloadableVideo behaviours for the permutations affected here.
reduced the diff for OEXVideoSummary initWithDictionary by moving the downloadURL property to a method instead of storing a variable.
added screenshots to the PR description to show the effect of both USING_VIDEO_PIPELINE options.

@pomegranited All the scenarios work perfectly fine. But, I like the previous approach, populating the downloadURL in OEXVideoSummary initWithDictionary, because with the introduction of getter for downloadURL looping and all the required checks being performed every time downloadURL being accessed and also it introduces a new instance variable allSources.
If the code readability is concerned here then downloadURL populating logic can be moved to a method that can take allSources or any other required data as params. What do you think?

@pomegranited One more point just for the sake of knowledge, the PR is specifically for the handling of HLS Videos then why the title "Use HTML5 source as download URL" was given?


add WXRotation class and ASDF schema

Hello @CagtayFabry! Thanks for opening this PR.

In the file weldx/asdf/tags/weldx/core/transformations/rotation.py:

Line 42:12: F632 use ==/!= to compare constant literals (str, bytes, int, float, tuple)
Line 59:39: F541 f-string is missing placeholders

Do see the Hitchhiker's guide to code style


Custom themes

Hey hey!
Great module!
Will there be any API for adding custom themes?

Not yet, but I am open to it!

Will not happen.


JSON objects output invalid JSON on to_s

Not a bug per se, but it seems like a questionable design decision IMO to have JSON objects print like a hash instead of as a string of valid JSON on conversion to a string.
For example,
json = JSON.parse("{\"name\": \"David\"}")
json.to_s
# output: "{\"name\" => \"David\"}"

If it weren't for the use of an arrow => instead of a colon :, the above would be valid JSON, and I could rely on implicit conversion, as opposed to being required to use #.to_json on a JSON object. In the case that I don't have access to the JSON object or, more likely, that it's inconvenient to handle JSON objects separately, it's necessary to use json.to_s.gsub("=>", ":") everywhere.
I'm wondering if this is intentional and it's been discussed, or if it simply stems from a minor implementation detail. At any rate, is this is something that could be considered for a change? I would imagine not many people if any are currently relying on the current format, and any (like myself) that are already converting to valid JSON through a substitution would be unaffected.
As a side note, however, the use of an arrow here seems perfectly fine:
json = JSON.parse("{\"name\": \"David\"}")
json
# output: {"name" => "David"}

because it's simply describing the object. Of course, if any change is made, it wouldn't hurt to apply the change here for consistency as well.

The method intended for JSON serialization is #to_json, not #to_s, see docs for more detailed explanation.

You're getting a String representation of JSON::Any, which is represented as a Hash.  If you want the actual JSON string you should use .to_json.

@Sija I understand that when I'm converting from a hash or an array, but I don't see why an actual JSON object can't just treat this as default behavior.

You're not converting from a Hash or Array.  You're parsing a JSON string into another data type.  I'm not sure I see the problem?

Actually I think I see your point.  Is there a reason to not just do .to_json as the default to_s behavior?  Seems like that would make sense...

Yeah, I think my mention of a Hash or Array was beside the point, and I hadn't really considered that to_json really just is parsing an object to a String. But if to_s is just a string representation of an object, it seems convenient (not to mention logical) that the string representation of a JSON object be valid JSON itself.

I guess it would make some sense for JSON::Any#to_s to actually show a JSON representation, i.e. delegate to raw#to_json instead of raw.to_s.
But JSON::Any is intended as a kind of thin layer around the raw value it wraps and can even be used interchangebly in some situations (comparison, hashing). I'm not sure that's the best idea, but long as it works like that, it should have the same string representation as the wrapped raw value.
This also depends on the data type of the wrapped value. The difference between #to_s and #to_json on a Hash is very small. On Array Array and most scalar types it's usually pretty much the same. But on a String, it makes a huge difference because #to_json quotes the value. So #to_json is similar to String#inspect, not String#to_s.

Yes, I think the misconception here is that there's any such think as a "JSON Object type" in  Crystal, which there is not.

I see, that's a fair argument, although I do feel that the difference between a proper JSON object type and a raw value in a "JSON wrapper" seems pretty small.
With that said, I've looked a little more closely into my use case, and, quite awkwardly, it seems like my issue isn't with JSON::Any#to_s at all... 🤦 . Instead, it's with the returned value itself, i.e. the result of evaluating json at the end of my initial examples.
I have a method which accepts a value of several possible types, specifically the union of all primitives, arrays consisting of a primitive, and, now, JSON::Any. At some point, this method relies on the value passed in conjunction with the value's type. Previously, this worked because evaluating an Int results in an Int, as does evaluating a Float (result in a Float), and so on for strings and even arrays. In fact, for all cases but a hash, evaluating a value wrapped in JSON::Any will result in a JSON string.
However, when I evaluate an instance of JSON::Any that wraps a hash, the output is not JSON, solely because of the use of =>. Whether or not it was coincidental in the first place, it's inconsistent with all other cases, and it feels odd to me that JSON.parse(object) would output valid JSON for everything but a Hash object, simply because of an underlying implementation detail.
Also, I apologize for the abrupt turn in topic... Please let me know if this warrants a separate issue, a renaming, or anything of the like.

On second thought, without looking at the actual source for how JSON::Any wraps values, I would guess that on evaluation, a JSON::Any instance simply returns its wrapped value... in which case changing this behavior would be less trivial. Under that understanding, I can see that the convenience of guaranteeing that JSON.parse(object) output valid JSON might not be worth the work.

I'm not sure I can quite follow you. Do you have some minimal self contained example to illustrate what you're talking about?

Hmm, unfortunately not at the moment, see my edit above. I'm going to try a different approach to my issue, thanks for the help.

When you say evaluate, what do you mean? A thing doesn't evaluate. Are you using Crystal play to try this out? Putting json evaluates to something, and json.to_s evaluates to something else doesn't make sense. json doesn't evaluate to anything, unless you are using irc or crystal play which do evaluate the objects by calling some method, presumably inspect.
So could you explain where are you seeing this behavior?

Yeah @asterite , I've been using icr to test these out. I see, I'll have to read up on what inspect does, then.

To try and clear up some of the confusion I've caused, my motivation for this is trying to make a quick and dirty patch to the DB library so that it works with JSON types, which the underlying driver library for Postgres, pg, supports. I've been able to get as far as having the library pass my wrapped value to pg, but when the value is inspected(?) right before it's passed to my Postgres database, it results in a string of invalid JSON thanks to the => characters, which chokes Postgres.

Could you share some code? If you need to pass a JSON string to postgres, you would call to_json on the object. But of course, without code we don't know what you are trying to do.

If you need to pass a JSON string to postgres, you would call to_json on the object.

Only if you want to store it as a string! PostgreSQL has a JSON datatype: https://github.com/will/crystal-pg/blob/cafee448dc09ef487bd3941febce8796830f1bd7/src/pg/decoder.cr#L379-L405

Yep! That's exactly it, I'm trying to take advantage of PostgreSQL's JSON type, but when I pass in a hash value wrapped in JSON::Any, at some point along the line something happens so that the final parameter passed to the database (which I'm using put statements to check, in case that affects things) is something like {"a" => 3} instead of JSON.
I don't have one now, but I'll try to create as simple a self-contained example as I can and update this comment shortly.

Maybe report a bug in crystal-pg? It might be a bug of that shard.

is something like {"a" => 3} instead of JSON.

Make sure to use p, p!, pp, pp! for debug printing. Using puts this could be both, {"a" => 3} or "{\"a\" => 3)"!

Hmm, I'm not sure it is. I'm currently using the Granite ORM library, and it's through that that I'm using DB and (through that) crystal-pg. To try and patch in support for my use case, I've been making tweaks all around - also the reason I haven't provided actual code.
At this moment, I think what I'm most confident about is that changing the default behavior of JSON::Any#inspect (or is it #to_s? still a little confused here) would fix my use case. Since it seemed to me like a reasonable default to have #to_s and #to_json be equivalent anyway, I opened this issue.
@jhass Oh interesting, I wasn't aware of that. Thanks, that seems really good to know.


First-pass ledger hashes in replayer checkpoint files

The replayer checks that every new snarked ledger hash it sees is some first-pass ledger hash. If we run the replayer from a checkpoint file, a new snarked ledger hash might be equal to a first-pass ledger hash that was produced before the checkpoint.
Solution: save the pending first-pass ledger hashes to the checkpoint file, and populate the set of hashes from the saved hashes when the replayer starts. Also save the last snarked ledger hash, which is used to know whether to search the first-pass hashes.
The set of ledger hashes are indexed by order of occurrence. That indexing allows flushing older hashes when we find the first-pass hash that matches a snarked ledger hash. The ledger hashes are saved as a list whose order is given by the indexes. When starting the replayer, the set of hashes uses new indexes created when iterating over that list.
The first_pass_ledger_hashes and last_snarked_ledger_hash fields are optional in the input file, so compatibility with old checkpoint files is maintained. Confirmed that fact by running the replayer on an old checkpoint file.
Closes #13732.

!ci-build-me

!ci-build-me

!ci-build-me

!ci-build-me

!approved-for-mainnet


Workaround for missing pause event in Youtube on Chrome

This is a fix for issue #458 where the pause event is not fired in Youtube HTML on Chrome, meaning that player initialization does not complete.

Hm, interesting.
How do we know for sure that the settimeout fires after it is paused? I also wonder what happened with youtube and chrome to cause this breakage.

The pauseVideo() call does not change the state to paused, so we can not checked whether the video is actually paused after the timeout fires. One possibility could be to check if time changes or not, adding further delay. So I just assumed 0.2 seconds is safe enough.
As for what happened to Youtube (nothing probably changed on Chrome as issue is consistent across versions), I have no idea and did not find anything. But the issue has been here for more than a week I think.

Yeah, it does seem like something changed on YouTube's end, which doesn't surprise me at all. It sounds like a bug on YouTube's end.
What happens if you delay this call by 0-200 ms https://github.com/pculture/popcorn-js/blob/master/wrappers/youtube/popcorn.HTMLYouTubeVideoElement.js#L248

Delaying the call to pause does not fix the missing pause event.
Actually my fix was really wrong (the parameter sent to onFirstPause was actually not sent) but was still working because the pause call is apparently synchronous, unlike the call to seek, which the onFirstPause function makes sure completes before proceeding. So I updated the fix, completely taking out waiting for that event. It works well on Chrome and Firefox.

I'm not super comfortable with accepting this yet, until we have a better idea why this broke.
That might give us more insight, and lead us to a cleaner fix.
Right now it is actually just circumventing the pause event completely.

This is not necessary anymore as youtube seems to have fixed the missing event.