ext_proc - sampling support

Title: ext_proc - sampling support
Description:
We need the ability to sample ext_proc calls. A simple sample_rate field (0.0 to 1.0) within ext_proc filter that activates the filter and makes the external call only when necessary. This becomes especially useful when we finish implementing the async support, allowing users to selectively sample a fraction of traffic for observation.
Once sampled, the stream should be active for the entire request, honoring all the processing modes, etc.
cc @tyxia @htuch

Yeah, preference is this is outside of ext_proc and part of the match model.

/assign @tyxia

I am a little bit confused on the comments.
The original issue description is for adding sampling support for ext_proc to makes the external call only when necessary. To implement that, we can simply toss a coin in the ext_proc filter and make the call based on the coin toss result.
@kyessenov @htuch could you please elaborate the idea to implement it in the unified matcher?

/assign @yanjunxiang-google

I think @kyessenov is suggesting we just have the coin flip in the matcher, which will be more generic, as there are other legit use cases for sampling.

I think it can be understood as: Perform the sampling at the higher level and centralized place. This will benefit all delegated filters (like ext_proc, ext_authz, etc)  as they are invoked based on sampling result

If we want to do sampling in xDS matcher,  a few questions are:

Where to store the sampling ratio config?
Where to execute the coin toss based on the configured sampling ratio


Request failed

I keep getting request failed with status code 403

Fixed, please pull the repo again!


Prefer structured communication

As long as the charpter is growing and the number of people is increasing, we will have less and less oportunities to sync the whole charpter.
Due to that, whenever you feel the need to start a discussion or anything similar, prefer structured communications like Github Issues instead Slack. If you belive you need the acknowledgement of the whole chapter about something you have to say or do, on behalf or for the charpter, consider opening a Issue.
It will make far more easy to newcomers to reach this topics and issues that may, in sometime, be their issues.
It also improves a lot our documentation about our own development as a reference software engineer team.
Use the repositories Issues tab to create issues related to that project and further a task related to that issue and then a PR related to the task is a good cycle of improvement.

I agree! Keeping a history of our discussions it's also a good way to remember why we decided or not to do something.

Agreed 100%, it's also easier to center everything in one place and keep things organized.
We can also have more styleguides and documents, other than backend and frontend (e.g. development styleguide, code review styleguide, etc)

warming up my neurons here to make a PR creating the development styleguide o/


fix(avm): correctly generate public inputs in verifier

PIL must be compiled with: https://github.com/AztecProtocol/powdr/pull/69/
Now the __is_public suffix is also not needed.

#7018  👈
master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @fcarreiro and the rest of your teammates on  Graphite

Benchmark results
Metrics with a significant change:

app_circuit_witness_generation_time_in_ms (SchnorrAccount:spend_private_authwit): 104 (+36%)
avm_simulation_time_ms (Token:mint_public): 68.0 (-87%)
avm_simulation_time_ms (Token:assert_minter_and_mint): 220 (+214%)

Detailed results
All benchmarks are run on txs on the Benchmarking contract on the repository. Each tx consists of a batch call  to create_note and increment_balance, which guarantees that each tx has a private call, a nested private call, a public call, and a nested public call, as well as an emitted private note, an unencrypted log, and public storage read and write.
This benchmark source data is available in JSON format on S3 here.
Proof generation
Each column represents the number of threads used in proof generation.

Metric
1 threads
4 threads
16 threads
32 threads
64 threads

proof_construction_time_sha256_ms
5,728
1,546
696
745
777 (+1%)

proof_construction_time_sha256_30_ms
11,459
3,068
1,375
1,427 (-1%)
1,458 (+1%)

proof_construction_time_sha256_100_ms
43,911
11,843 (+1%)
5,432
5,408 (-1%)
5,368

proof_construction_time_poseidon_hash_ms
78.0
34.0
34.0
58.0 (+2%)
88.0 (-1%)

proof_construction_time_poseidon_hash_30_ms
1,518
417 (-1%)
201 (+1%)
229 (+2%)
269 (+1%)

proof_construction_time_poseidon_hash_100_ms
5,741
1,561
725
781 (-1%)
800 (+1%)

L2 block published to L1
Each column represents the number of txs on an L2 block published to L1.

Metric
4 txs
8 txs
16 txs

l1_rollup_calldata_size_in_bytes
1,412
1,412
1,412

l1_rollup_calldata_gas
9,452
9,458
9,440

l1_rollup_execution_gas
610,273
610,279
610,261

l2_block_processing_time_in_ms
773 (+1%)
1,448 (+3%)
2,760 (+2%)

l2_block_building_time_in_ms
23,618 (+1%)
48,964 (+2%)
92,897 (+1%)

l2_block_rollup_simulation_time_in_ms
23,508 (+1%)
48,578 (+2%)
92,479 (+1%)

l2_block_public_tx_process_time_in_ms
20,188 (+1%)
44,882 (+2%)
88,998 (+1%)

L2 chain processing
Each column represents the number of blocks on the L2 chain where each block has 8 txs.

Metric
3 blocks
5 blocks

node_history_sync_time_in_ms
6,684 (+3%)
9,547 (+3%)

node_database_size_in_bytes
10,526,800
14,393,424

pxe_database_size_in_bytes
9,242
15,127

Circuits stats
Stats on running time and I/O sizes collected for every kernel circuit run across all benchmarks.

Function
time_ms

GasToken:_increase_public_balance
73.3 (+7%)

GasToken:set_portal
17.4 (-6%)

Token:constructor
96.0 (+3%)

FPC:constructor
61.3 (+1%)

GasToken:mint_public
54.2 (+5%)

Token:mint_public
:warning: 68.0 (-87%)

Token:assert_minter_and_mint
:warning: 220 (+214%)

SchnorrAccount:approve_public_authwit
32.6 (+1%)

FPC:prepare_fee
74.4

Token:transfer_public
46.9 (+10%)

FPC:pay_refund
66.9 (-5%)

Benchmarking:increment_balance
2,570 (+2%)

Token:_increase_public_balance
57.3 (+1%)

FPC:pay_refund_with_shielded_rebate
57.5 (+7%)

Public DB Access
Time to access various public DBs.

Function
time_ms

get-nullifier-index
0.179

Tree insertion stats
The duration to insert a fixed batch of leaves into each tree type.

Metric
1 leaves
16 leaves
64 leaves
128 leaves
256 leaves
512 leaves
1024 leaves

batch_insert_into_append_only_tree_16_depth_ms
10.6 (+2%)
17.2 (+2%)
N/A
N/A
N/A
N/A
N/A

batch_insert_into_append_only_tree_16_depth_hash_count
16.7
31.8
N/A
N/A
N/A
N/A
N/A

batch_insert_into_append_only_tree_16_depth_hash_ms
0.614 (+2%)
0.527 (+2%)
N/A
N/A
N/A
N/A
N/A

batch_insert_into_append_only_tree_32_depth_ms
N/A
N/A
49.7 (+3%)
77.6 (+2%)
137 (+4%)
253 (+3%)
482 (+1%)

batch_insert_into_append_only_tree_32_depth_hash_count
N/A
N/A
95.9
159
288
543
1,055

batch_insert_into_append_only_tree_32_depth_hash_ms
N/A
N/A
0.507 (+3%)
0.479 (+2%)
0.468 (+3%)
0.459 (+3%)
0.450 (+2%)

batch_insert_into_indexed_tree_20_depth_ms
N/A
N/A
61.1 (+3%)
115 (+3%)
185 (+1%)
364 (+3%)
710 (+2%)

batch_insert_into_indexed_tree_20_depth_hash_count
N/A
N/A
109
208
355
691
1,363

batch_insert_into_indexed_tree_20_depth_hash_ms
N/A
N/A
0.517 (+3%)
0.514 (+2%)
0.488
0.493 (+3%)
0.487 (+1%)

batch_insert_into_indexed_tree_40_depth_ms
N/A
N/A
76.5 (+3%)
N/A
N/A
N/A
N/A

batch_insert_into_indexed_tree_40_depth_hash_count
N/A
N/A
135
N/A
N/A
N/A
N/A

batch_insert_into_indexed_tree_40_depth_hash_ms
N/A
N/A
0.534 (+3%)
N/A
N/A
N/A
N/A

Miscellaneous
Transaction sizes based on how many contract classes are registered in the tx.

Metric
0 registered classes
1 registered classes

tx_size_in_bytes
83,854
676,749

Transaction size based on fee payment method
| Metric |  |
| - |  |

Merge activity

Jun 11, 4:47 PM EDT: @fcarreiro merged this pull request with Graphite.


Windows Installer Should Include Common Pre-requisites

Feature Request
Feature Information:
The Windows AMP installer should install the pre-requisites used by many game servers. This includes the C++ Redistributables and DirectX. The Custom install option should allow you to ignore installing these. Many other software installers include the web installers of these and should allow the AMP setup file to remain the same size by mirroring their method.
All C++ x86/64 versions here:
https://learn.microsoft.com/en-US/cpp/windows/latest-supported-vc-redist?view=msvc-170
DirectX here:
https://www.microsoft.com/en-us/download/details.aspx?id=35
I confirm:

[x] that I have searched for an existing feature request matching the description.

I actually tried this before and had a really hard time getting the merge modules to work correctly with WIX, but I'll see what I can do.

At the very least, maybe it could direct people with links to those sites. Anything would be better than them coming to Discord after a server fails to start. :)

CI/nightly has this (although you'd have to get a nightly version of the installer) - ampinstmgr --nobanner DownloadCPPRuntime is available on Windows.


Expose ORPort and DirPort in Dockerfile

Thanks for providing this image! I'm using this container as a Tor relay, so these two ports are important for me.

Thanks for merging! Can you push the changes so that the updated image goes live?

Alexander, sorry about delay with response. Yes, I'm revert this changes.

Dockerfile is not allowed this style inline comments, so this build failed (this is my fault - don't check carefully).

This project docker-tor-simple is simple as possible (this main idea), but powerful, to implement complex scenarios.

In your case, I strongly recommend that you use the --expose and --publish arguments for docker run command to throw the needed ports. Dockerfile modification is not required. I will add the relevant information to the documentation in the near future.

Dockerfile is not allowed this style inline comments, so this build failed (this is my fault - don't check carefully).
This project docker-tor-simple is simple as possible (this main idea), but powerful, to implement complex scenarios.

How about adding a single EXPOSE 9001 9030 9050 9051 instead? I thought it would be nice to document each port inline, but now I see that this might (uselessly) create extra layers.

In your case, I strongly recommend that you use the --expose and --publish arguments for docker run command to throw the needed ports. Dockerfile modification is not required. I will add the relevant information to the documentation in the near future.

I'm using --publish right now for my Tor relay, but I'd prefer to have the containers intrinstic documentation reflect this. Also I'd like to try this image with rkt which makes it very hard to use containers that don't properly declare the ports they expose.
What is your stance on having EXPOSE 9001 9030 9050 9051 in the Dockerfile?


Unable to join the zipline chat room!!

Unable to join the zipline chat room!!

Do you mean https://gitter.im/FreeCodeCamp/HelpZiplines? It works fine for me.

@manojansh thanks for posting. Any steps to reproduce? Any errors?

The HelpZiplines Chat Room is deprecated and will be losing Camperbot support soon.
The new help room for Front End Projects is HelpFrontEnd.

Closing as not reproducible. If you continue having trouble with Gitter rooms you should contact their support team as we don't maintain that service.


API Key Still in Static Map Component

Just a heads up, there is still a Bing Maps API key in the Static Maps Component. If you haven't already, deactivate it :)

Thanks for the awesome toolset!

You guys can probably close this one because I'm getting a white screen now, I imagine they have deactivated it.


use import pattern that supports windows external drives

closes #901

This commit broke generation for me as well


Add default ranges to date range

Add default preset date ranges such as last week, last month etc. Make the options configurable.

Merged into develop.


Remote Sorting and Custom Filter renderer

Any time I click inside custom component for filtering (react-select), table fires onTableChange with type:"sort"
https://codesandbox.io/embed/7kky87yv2x

@rchaser sorry for lately reply, I will track this issue and see if we can fix this issue, thanks!

@rchaser BTW, is it possible to avoid the click event on custom filter bubble up?  because in react-bootstrap-table2, we will listen the click event on header cell and for all the default filter component, we will call the stopPropagation to stop event bubble up.

@rchaser sorry for lately reply, Is it possible to avoid the click event on custom filter bubble up? because in react-bootstrap-table2, we will listen the click event on header cell and for all the default filter component, we will call the stopPropagation to stop event bubble up.

I achieved the same by having an onClick event on my rendered component and calling stopPropagation when the event is triggered. Could be wrapped in a div

👍


Crash when a lot of data needs to be proxied

Problem Describing
Can't alloc os_mbuf when a lot of data needs to be proxied in BLE-MESH, so om is null to cause crash(assert).
static int proxy_send(uint16_t conn_handle, const void *data, uint16_t len) {
    ···
    BT_DBG("%u bytes: %s", len, bt_hex(data, len));
    om = ble_hs_mbuf_from_flat(data, len);
    assert(om);
    err = ble_gatts_notify_custom(conn_handle, svc_handles.proxy_data_out_h, om);
    notify_complete();
    ···
}

Related
These are also handled with assert: pb_gatt_srv.c#L410-L411 and proxy_srv.c#L876-L877. We should reconsider how to deal with it.

@michal-narajowski Please help look into this issue. Thx very much.


Update image ghcr.io/zoriya/kyoo_migrations to ed601f8

This PR contains the following updates:

Package
Update
Change

ghcr.io/zoriya/kyoo_migrations
digest
119ec7d -> ed601f8

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
🔕 Ignore: Close this PR and you won't be reminded about this update again.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

Edited/Blocked Notification
Renovate will not automatically rebase this PR, because it does not recognize the last commit author and assumes somebody else may have edited the PR.
You can manually request rebase by checking the rebase/retry box above.
⚠ Warning: custom changes will be lost.


Adding custom domain words and abbreviations to vocab.txt

Hi i am working on adding a few terms from my domain to the vocab.txt. I am working with the multi language cased pre-trained model 'multi_cased_L-12_H-768_A-12' . However i am unsure of what words should be added. So my question is should the new words added to the [unused X] lines be words not found in the english dictionary and be words unique to my domain, or should i be adding all frequently used words, even if they are common english words, but are not in the vocab.txt file. Additionally should abbreviations be added such as "LOL", and should the lowercase version 'lol', be also added since i am using a cased pre-trained model.

Have you found a way


Fix multiple select filter

After using multipleSelect filter with dot-notation (e.g. categories.id), option elements don't have selected prop.

In the template views/filter/multipleselect.blade.php, request($name, []) is used to get current value, but it will always be [] when dot-notation is used. I fixed that with the use of $value passed from controller instead of request($name). I would appreciate it if you could confirm this.

Thanks！


Initialize pruning point depth store -- minor bug fix

Initialize the pruning point depth store for the rare case where the pruning point is the first sink during initial IBD (virtual processor calls the depth store for each new sink)

Thanks @michaelsutton
So this fixes https://github.com/kaspanet/rusty-kaspa/issues/455 ?

Thanks @michaelsutton So this fixes #455 ?

Yes it should


Upgrade plugin posthtml-expressions

          The issue with the dependency of plugin posthtml-expressions has been resolved, try to update your plugin.

https://github.com/posthtml/posthtml-expressions/issues/151
https://github.com/posthtml/posthtml-expressions/releases/tag/v1.11.1
Originally posted by @Ulyanov-programmer in https://github.com/thewebartisan7/posthtml-components/issues/33#issuecomment-1575062552

This is upgraded here https://github.com/posthtml/posthtml-components/commit/0c79060ffd770883a7529d6d4b7960eadd834bec


WIP esp-idf cmake build

This is very work-in-progress work to compile the esp-idf using cmake.
Currently, it compiles (tested for esp32 on windows) but doesn't run when it has been flashed.
Please give feedback and/or tell me if you find something wrong.

This is a great start, thank you very much for this contribution!
I'll review the patch thoroughly, but let me make some general remarks first:

Ideally, we should be moving most of your  sdk_build_support to cargo-pio. In the meantime, cargo-pio is no longer just for PlatformIO. For example, this, this, this, this and this are generally usable utilities even outside the context of PlatformIO. My idea for cargo-pio was to later rename it to something like  embedded-build, where cmake-driven and PlatformIO-driven builds can co-exist in separate modules, The cargo pio subcommand, will stay, just like cargo pio-link, but the latter will be likely renamed to cargo linkproxy as it has nothing to do with PlatformIO. Your cmake CodeModel utilities seem generic enough and usable even outside of the context of building the ESP-IDF SDK, so those would be a great addition to cargo-pio (i.e. embedded-build).

The outcome of the above is that all your Bindgen-specific work in this PR can take advantage of the bindgen utilities in cargo-pio. Ditto for deriving the MCU based on the Rust target. Ditto for the python executable detection and running (and btw I don't mind merging your cmd! macros work in cargo-pio as well)

We should not be removing the PlatformIO based build. It should stay, as another option

As for why the build does not run - you are probably not applying these patches to ESP-IDF. All of these that don't start with _master are necessary for the ESP-IDF version you are using, and some of them (the missing atomics one starting with master_) will be necessary even for ESP-IDF master.

Thank you for the response!

Ideally, we should be moving most of your sdk_build_support to cargo-pio. In the meantime, cargo-pio is no longer just for PlatformIO.

I agree. I only looked at cargo-pio sparsely and didn't think it was to be more general purpose.

Your cmake CodeModel utilities for example, seem generic enough and usable even outside of the context of building the ESP-IDF SDK, so those would be a great addition to cargo-pio

These are just some utilities that I threw together to get a working example. My intention was to contribute them to the cmake crate (maybe under a feature) at some point. But I agree there should be a more general purpose library for compiling embedded stuff.

The outcome of the above is that all your Bindgen-specific work in this PR can take advantage of the bindgen utilities in cargo-pio. Ditto for deriving the MCU based on the Rust target. Ditto for the python executable detection and running (and btw I don't mind merging your cmd! macros work in cargo-pio as well)

That's good to know, will try to do this. And this first try was just to get an initial working example as fast as possible, without considering all the things (like all OS/esp-idf version/... related things).

We should not be removing the PlatformIO based build. It should stay, as another option

That was also my intention (should have stated that more clearly).

As for why the build does not run - you are probably not applying these patches to ESP-IDF. All of these that don't start with master_ are necessary for the ESP-IDF version you are using, and some of them (the missing atomics one starting with master_) will be necessary even for ESP-IDF master.

I've also considered this (need to try it, though I doubt it will work), but I've dismissed it because I thought if this was the problem the linker would've complained. I think this problem has more to do with the way I'm linking the esp-idf. I currently combine all *.a libraries into a single library that gets linked with the rust program and all other linker flags. I think this approach was too naive, though I'll be doing more debugging right now.

I've also considered this (need to try it, though I doubt it will work), but I've dismissed it because I thought if this was the problem the linker would've complained. I think this problem has more to do with the way I'm linking the esp-idf. I currently combine all *.a libraries into a single library that gets linked with the rust program and all other linker flags. I think this approach was too naive, though I'll be doing more debugging right now.

Most of these patches are bugfixes ant not really pollyfils.
But you are probably right - it might be the way you are assembling the final static lib, as link order matters a lot with ESP-IDF. For example, they use link order to overwrite some symbols from newlib. What I do is really to take the "native" link arguments from PIO, and pass them - exactly in the same order and exactly as they are - to the linker.

Regarding link command line length on Windows: strange, this is what I actually receive on Windows. Aside from being weird, this is actually taking care of the command line length.
As for some link args being relative, that's annoying and I am also fixing it in a bit of a hacky way.

Okay, so I've figured some things out.
As I've told you already, I had problems with the link command line length and that is because cmake generates 362 arguments whereas SCons (the build-system platformio uses) generates only 140. This is because the esp-idf has approximately a million circular references and cmake is fine with just using the same library multiple times in the command line while SCons uses --start-group/--end-group.
But I'm really not sure how the esp-idf even links with platformio, because some -u flags towards the end don't even have a corresponding library that defines the symbol (undefined by -u).
But anyhow, to fix this for cmake I'll just have to do more processing of the linker arguments, to also incorporate --start-group/--end-group (or -(/-) for short).

As I've told you already, I had problems with the link command line length and that is because cmake generates 362 arguments whereas SCons (the build-system platformio uses) generates only 140. This is because the esp-idf has approximately a million circular references and cmake is fine with just using the same library multiple times in the command line while SCons uses --start-group/--end-group.

What is still a mystery for me is how come cmake itself does not have a problem with these 362 link parameters, given that - in the end - it has to call the native linker and pass these 362 link parameters to it? Are we missing something there?

But anyhow, to fix this for cmake I'll just have to do more processing of the linker arguments, to also incorporate --start-group/--end-group (or -(/-) for short). If we incorporate --start-group/--end-group we'll end up with a lot less arguments, but still not as few as platformio (if we do it right).

Yes, we can try this way, but we should bear in mind that any "post-processing" on top of the cmake-generated link arguments we do would be fragile as in that it will be manually implemented effort which makes a lot of ESP-IDF-specific assumptions around what these parameters are. And it might break with a subsequent next release of ESP-IDF. What PlatformIO does is exactly such a workaround - it is interrogating the cmake build system about its linker flags, and then manually pushing these into the Scons build system. At places, it does assumptions (related to linker scripts) and instead of taking those from native cmake, it is just assuming those reside at certain locations inside the ESP-IDF directory tree. This is by the way the reason why I cannot compile ESP-IDF master with PlatformIO ATM - because the PlatformIO python code assumes there are some linker scripts which reside at certain locations in the ESP-IDF and is blindly - without checking if these really exist - putting references to them on the linker command line. Well, these do exist in ESP-IDF V4.3, but no longer exist in ESP-IDF master. Guess what happens.
So the above goes to say - can't we explore other paths to solve the "too long list of arguments" problem? For example, if GCC (&LD) do support passing the arguments to them in a file, we can build a temp file, shuffle all cmake-generated link arguments there and then just pass the temp file to the linker?
And also as per above - can we get to the root cause of why we are having the "to long list of arguments" problem, and cmake itself - when calling the native linker - doesn't?

So the above goes to say - can't we explore other paths to solve the "too long list of arguments" problem? For example, if GCC (&LD) do support passing the arguments to them in a file, we can build a temp file, shuffle all cmake-generated link arguments there and then just pass the temp file to the linker?

That doesn't work either because even if I pass all the arguments for the esp-idf in a response file, gcc then just calls collect2 with all argument files expanded. And on windows there is a strict limit of 32767 characters for the arguments, which is overshot by a mile (50502). The reason this doesn't happen with only cmake  is that, like already mentioned, it uses relative paths. Converting all paths to absolute ones inflates the 13448 characters command line to 50502 minus the arguments rust adds (which is about 100 (9238 chars)).

So the above goes to say - can't we explore other paths to solve the "too long list of arguments" problem? For example, if GCC (&LD) do support passing the arguments to them in a file, we can build a temp file, shuffle all cmake-generated link arguments there and then just pass the temp file to the linker?

That doesn't work either because even if I pass all the arguments for the esp-idf in a response file, gcc then just calls collect2 with all argument files expanded. And on windows there is a strict limit of 32767 characters for the arguments (a limit of CreateProcess), which is overshot by a mile (50502). The reason this doesn't happen with only cmake is that, like already mentioned, it uses relative paths. Converting all paths to absolute ones inflates the 13448 characters command line to 50502 minus the arguments rust adds (which is about 100 (9238 chars)).

Got it, great analysis.
I'm starting to think of bigger hacks. First of all, you anyway need to pass all the linker args from esp-idf-sys up to the binary crate itself. This is because cargo:link-arg does not really work transitively (that is: anymore; it used to, for a short period of time due to a regression, but they closed this loophole). I assume you are doing that with a mechanism similar to mine. If so, prior to emitting the linker args with cargo:link-arg (or whatever the name was), can't we symlink the relative-path linker arguments into the out directory of the binary crate? Or even copy them there? I think the out directory is the "current" directory during the build.

@igrr ^^^ Any bright ideas?

As the commit says, I've implemented a workaround that simply copies the cmake build output to the workspace root. This solution isn't great but it is the simplest right now. Making a symlink instead of copying requires elevated permissions on windows.

Well we can at least try this path. You'll need a linker proxy similar to mine. It will just do a chdir() to a directory passed as a special parameter, then run the GCC linker, and then chdir() back.

I don't like that we have to use an extra tool just to work around this, but if we did this it would require us (at least on windows) to parse the response file that cargo gives the linker which contains its command line. We need to do this to handle quoted arguments and paths with spaces. The current carg-pio linker wrapper doesn't do this right now unless I'm mistaken. So the quick fix right now is just to copy the esp-idf directory from the cmake build output to the workspace dir, and adding its name to .gitignore.
But I think the best solution forward is to make a linker wrapper which handles the above. Even if a new build of gcc would use response files internally, we would still have to sanatize all relative paths with absolute ones, which can also go wrong if we get a linker arg that we don't expect. Or we'd have to parse the linker args entirely which is a lot more work.
That's just my reasoning, please tell me what you think.

As the commit says, I've implemented a workaround that simply copies the cmake build output to the workspace root. This solution isn't great but it is the simplest right now. Making a symlink instead of copying requires elevated permissions on windows.

I'll look into that later.

Well we can at least try this path. You'll need a linker proxy similar to mine. It will just do a chdir() to a directory passed as a special parameter, then run the GCC linker, and then chdir() back.

I don't like that we have to use an extra tool just to work around this, but if we did this it would require us (at least on windows) to parse the response file that cargo gives the linker which contains its command line. We need to do this to handle quoted arguments and paths with spaces. The current carg-pio linker wrapper doesn't do this right now unless I'm mistaken. So the quick fix right now is just to copy the esp-idf directory from the cmake build output to the workspace dir, and adding its name to .gitignore.

I didn't do it because it is still a bit of a hack, and moreover - I don't think ESP-IDF supports paths with whitespaces on it anyway. But yeah, more diligent processing of that file would be necessary. Not a rocket science, though.

But I think the best solution forward is to make a linker wrapper which handles the above. Even if a new build of gcc would use response files internally, we would still have to sanatize all relative paths with absolute ones, which can also go wrong if we get a linker arg that we don't expect. Or we'd have to parse the linker args entirely which is a lot more work.

In the previous paragraph you say that you kind of don't like the idea of a linker proxy just to workaround the long file paths. Here you say that the best solution forward is to make a linker wrapper. I'm a bit confused. For me, these things (linker proxy & linker wrapper) are one and the same thing. You agree?

That's just my reasoning, please tell me what you think.

Well, I definitely prefer a linker wrapper = linker proxy over spilling out stuff outside of the out directories, which is something Cargo explicitly advises against. It is still a hack, but probably the better hack.
One little additional benefit of having a linker proxy is that you don't have to have the GCC linker (= Espressif toolchain) on your path as you can pass the GCC tollchain to the proxy as an argument. You might say "big deal", but it actually is a win, if you consider that we might be downloading the toolchains during the esp-idf-sys crate build. The linker proxy solves this chicken-and-egg problem too.

Also, it compiles and runs fine now. I don't know about the patches though. How should these be applied knowing that using cmake one can specify every version? For which esp-idf version are the non-master patches?

I'm surprised that it works for you without you applying the thread-local patch at least. Without that one, it should break right after "joining the threads" in the demo binary crate.
The non-master patches are for V4.3.

In the previous paragraph you say that you kind of don't like the idea of a linker proxy just to workaround the long file paths. Here you say that the best solution forward is to make a linker wrapper. I'm a bit confused. For me, these things (linker proxy & linker wrapper) are one and the same thing. You agree?

Yes they're the same. I don't like this solution as it requires a tool that the user has to install beforehand. But it's the best solution because we don't have to sanitize the linker args if/when gcc is updated (or I guess rebuilt), if that doesn't happen it's the only solution. But that doesn't mean I have to like it ^^.

In the previous paragraph you say that you kind of don't like the idea of a linker proxy just to workaround the long file paths. Here you say that the best solution forward is to make a linker wrapper. I'm a bit confused. For me, these things (linker proxy & linker wrapper) are one and the same thing. You agree?

Yes they're the same. I don't like this solution as it requires a tool that the user has to install beforehand. But it's the best solution because we don't have to sanitize the linker args if/when gcc is updated (or I guess rebuilt), if that doesn't happen it's the only solution. But that doesn't mean I have to like it ^^.
Btw. for me the difference between this linker proxy tool and cmake (I guess that's not even needed, we could install this in the build script too) or python is that the latter are really common if you do anything with programming while the former is not.

Except that our linker proxy is a cargo subcommand. This means two things:

It is the most natural thing to install by a Rust developer
It is very easy to install and update. What can be simpler than cargo install cargo-pio? For both install and update? No $PATH changes, no sudo - nothing. I've yet to see somebody complaining that installing cargo-pio is failing on him.

In the previous paragraph you say that you kind of don't like the idea of a linker proxy just to workaround the long file paths. Here you say that the best solution forward is to make a linker wrapper. I'm a bit confused. For me, these things (linker proxy & linker wrapper) are one and the same thing. You agree?

Yes they're the same. I don't like this solution as it requires a tool that the user has to install beforehand. But it's the best solution because we don't have to sanitize the linker args if/when gcc is updated (or I guess rebuilt), if that doesn't happen it's the only solution. But that doesn't mean I have to like it ^^.
Btw. for me the difference between this linker proxy tool and cmake (I guess that's not even needed, we could install this in the build script too) or python is that the latter are really common if you do anything with programming while the former is not.

Except that our linker proxy is a cargo subcommand. This means two things:

It is the most natural thing to install by a Rust developer
It is very easy to install and update. What can be simpler than cargo install cargo-pio? For both install and update? No $PATH changes, no sudo - nothing. I've yet to see somebody complaining that installing cargo-pio is failing on him.

That is also true.

@N3xed Sorry for the late notice: there is a meeting this Tuesday of rust-on-esp contributors and some of the Espressif folks. Given that having a cargo-first build story with "native" ESP-IDF is important, you might want to participate?
If so, can you contact me privately on the esp-rs chat. I need an email address of yours to forward you the invite.
In any case, I'll update the agenda for the meeting with the progress done here by you.

@ivmarkov
Here's my proposal on how to proceed.
(1)
The cargo-pio library is currently pretty specific to platformio, but I like your idea of creating an embedded-build library.
So we would need to refactor the current cargo-pio code so that it can be used for different build environments. After that is done, move the bulk of this logic into the embedded-build library (and reuse anything that's there already).
(2)
Rename cargo-pio-link into cargo-linkproxy like you also already suggested, and add another argument for specifying the working directory. To parse the arguments that cargo potentially gives us in a response file we can just port the seperate_arguments command from cmake (ParseWindowsCommandLine for windows, there's this kwsysSystem__ParseUnixCommand for unix but not sure how easy it is to port).
(3)
I'd also like to create a tool that allows build scripts of dependencies to register commands for the end user. This allows similar behavior to the current cargo-pio (ex. cargo pio espidf menuconfig). This would very much simplify using the cmake version. This can then be used for flashing (ex. cargo bcmd espidf flash) or for creating a filesystem binary (maybe cargo bcmd generate-spiffs <path> <out-file>) and so on. This tool could live in the embedded-build or a separate repository.
Tell me if you agree to (1) and (2), I'll start with that then. I'd also like to know what you think about (3) or alternatives.

I definitely agree with (1) and (2). (3) seems reasonable too - we could use it for flashing, filesystem creation, and why not for monitoring (idf.py ... monitor ...) too? (The ESP-IDF monitor should be capable of decoding addresses into elf symbols. Not a rocket science honestly, can be done with pure Rust too, but why not having the ESP-IDF option too anyway.)

@ivmarkov @N3xed I've requested help with too long path on Windows and gcc flags from our team. They'll look into this issue.

Hi folks! @ivmarkov @N3xed

how is Espressif compiling their GCC-based toolchains for Windows? With --with-gnu-ld or not?

Not. Let me build a toolchain with that option (you need win64 for esp32, right?). Would it help?

Hi folks! @ivmarkov @N3xed

how is Espressif compiling their GCC-based toolchains for Windows? With --with-gnu-ld or not?

Not. Let me build a toolchain with that option (you need win64 for esp32, right?). Would it help?

As far as I can see, without this, response files don't really do anything to alleviate the command-line length limitation on windows. As I've described in a comment above:

So the above goes to say - can't we explore other paths to solve the "too long list of arguments" problem? For example, if GCC (&LD) do support passing the arguments to them in a file, we can build a temp file, shuffle all cmake-generated link arguments there and then just pass the temp file to the linker?

That doesn't work either because even if I pass all the arguments for the esp-idf in a response file, gcc then just calls collect2 with all argument files expanded. And on windows there is a strict limit of 32767 characters for the arguments (a limit of CreateProcess), which is overshot by a mile (50502). The reason this doesn't happen with only cmake is that, like already mentioned, it uses relative paths. Converting all paths to absolute ones inflates the 13448 characters command line to 50502 minus the arguments rust adds (which is about 100 (9238 chars)).

So this option should really always be used on windows (assuming ld is a gnu linker).

@N3xed thanks! Do you have some fast test to check the using of response files,  and to check that it helps us? If not, I'll try to check on my own soon.
--with-gnu-ld build:
https://dl.espressif.com/dl/toolchains/preview/withgnuld/xtensa-esp32-elf-gcc8_4_0-esp-2021r1-7-gec835ae-win64.zip
https://dl.espressif.com/dl/toolchains/preview/withgnuld/xtensa-esp32-elf-gcc8_4_0-esp-2021r1-7-gec835ae-linux-amd64.tar.gz

@antmak Thank you for the builds. I don't have a quit check currently, but will test if this solves the problem with the command-line length as soon as I make some progress with this PR.

Okay, I've updated the build script alongside my changes of embuild.
The command-line length is  pretty close to the limit (31190 chars):
- All cfg values from kconfig amount to about 8409 chars
- the linker args from cmake are about 19414 chars
- and the args purely from rust about 3367.
Now the linker invocation isn't the problem anymore but the rustc invocation from cargo.
We should probably switch to passing the linker args in a file and/or setting only select configuration options from the kconfig file.

Okay, I've updated the build script alongside my changes of embuild.
The command-line length is pretty close to the limit (31190 chars):

All cfg values from kconfig amount to about 8409 chars
the linker args from cmake are about 19414 chars
and the args purely from rust about 3367.

Now the linker invocation isn't the problem anymore but the rustc invocation from cargo.
We should probably switch to passing the linker args in a file and/or setting only select configuration options from the kconfig file.

Indeed. Let's have the first iteration of the native build working, and then we can take one of these paths.
By the way, for me the native build fails (after all the issues I've mentioned so far are workarounded) with the following error:
  = note: Running ldproxy
          Error: Linker /Users/imarkov/projects/esp32/rust-esp32-std-hello/.sdk/tools/riscv32-esp-elf/<IP_ADDRESS>_64eb9ff-8.4.0/riscv32-esp-elf/bin/riscv32-esp-elf-gcc failed: exit status: 1
          STDERR OUTPUT:
          /Users/imarkov/projects/esp32/rust-esp32-std-hello/.sdk/tools/riscv32-esp-elf/<IP_ADDRESS>_64eb9ff-8.4.0/riscv32-esp-elf/bin/../lib/gcc/riscv32-esp-elf/8.4.0/../../../../riscv32-esp-elf/bin/ld: /Users/imarkov/projects/esp32/rust-esp32-std-hello/.sdk/tools/riscv32-esp-elf/<IP_ADDRESS>_64eb9ff-8.4.0/riscv32-esp-elf/bin/../lib/gcc/riscv32-esp-elf/8.4.0/../../../../riscv32-esp-elf/lib/rv32im/ilp32/libc.a(lib_a-svfprintf.o): in function `_svfprintf_r':
          /builds/idf/crosstool-NG/.build/HOST-x86_64-apple-darwin12/riscv32-esp-elf/src/newlib/newlib/libc/stdio/vfprintf.c:1217: undefined reference to `__trunctfdf2'
          /Users/imarkov/projects/esp32/rust-esp32-std-hello/.sdk/tools/riscv32-esp-elf/<IP_ADDRESS>_64eb9ff-8.4.0/riscv32-esp-elf/bin/../lib/gcc/riscv32-esp-elf/8.4.0/../../../../riscv32-esp-elf/bin/ld: /Users/imarkov/projects/esp32/rust-esp32-std-hello/.sdk/tools/riscv32-esp-elf/<IP_ADDRESS>_64eb9ff-8.4.0/riscv32-esp-elf/bin/../lib/gcc/riscv32-esp-elf/8.4.0/../../../../riscv32-esp-elf/lib/rv32im/ilp32/libc.a(lib_a-vfprintf.o): in function `_vfprintf_r':
          /builds/idf/crosstool-NG/.build/HOST-x86_64-apple-darwin12/riscv32-esp-elf/src/newlib/newlib/libc/stdio/vfprintf.c:1192: undefined reference to `__trunctfdf2'
          collect2: error: ld returned 1 exit status

By the way, for me the native build fails (after all the issues I've mentioned so far are workarounded) with the following error:
  = note: Running ldproxy
          Error: Linker /Users/imarkov/projects/esp32/rust-esp32-std-hello/.sdk/tools/riscv32-esp-elf/<IP_ADDRESS>_64eb9ff-8.4.0/riscv32-esp-elf/bin/riscv32-esp-elf-gcc failed: exit status: 1
          STDERR OUTPUT:
          /Users/imarkov/projects/esp32/rust-esp32-std-hello/.sdk/tools/riscv32-esp-elf/<IP_ADDRESS>_64eb9ff-8.4.0/riscv32-esp-elf/bin/../lib/gcc/riscv32-esp-elf/8.4.0/../../../../riscv32-esp-elf/bin/ld: /Users/imarkov/projects/esp32/rust-esp32-std-hello/.sdk/tools/riscv32-esp-elf/<IP_ADDRESS>_64eb9ff-8.4.0/riscv32-esp-elf/bin/../lib/gcc/riscv32-esp-elf/8.4.0/../../../../riscv32-esp-elf/lib/rv32im/ilp32/libc.a(lib_a-svfprintf.o): in function `_svfprintf_r':
          /builds/idf/crosstool-NG/.build/HOST-x86_64-apple-darwin12/riscv32-esp-elf/src/newlib/newlib/libc/stdio/vfprintf.c:1217: undefined reference to `__trunctfdf2'
          /Users/imarkov/projects/esp32/rust-esp32-std-hello/.sdk/tools/riscv32-esp-elf/<IP_ADDRESS>_64eb9ff-8.4.0/riscv32-esp-elf/bin/../lib/gcc/riscv32-esp-elf/8.4.0/../../../../riscv32-esp-elf/bin/ld: /Users/imarkov/projects/esp32/rust-esp32-std-hello/.sdk/tools/riscv32-esp-elf/<IP_ADDRESS>_64eb9ff-8.4.0/riscv32-esp-elf/bin/../lib/gcc/riscv32-esp-elf/8.4.0/../../../../riscv32-esp-elf/lib/rv32im/ilp32/libc.a(lib_a-vfprintf.o): in function `_vfprintf_r':
          /builds/idf/crosstool-NG/.build/HOST-x86_64-apple-darwin12/riscv32-esp-elf/src/newlib/newlib/libc/stdio/vfprintf.c:1192: undefined reference to `__trunctfdf2'
          collect2: error: ld returned 1 exit status

That's weird. I can't test it as windows spits out file name or extension is too long again, but I'm guessing it because of a missing -lgcc after -lc in the linker args.
Note: I've tested only esp32 up till now.

By the way, for me the native build fails (after all the issues I've mentioned so far are workarounded) with the following error:
  = note: Running ldproxy
          Error: Linker /Users/imarkov/projects/esp32/rust-esp32-std-hello/.sdk/tools/riscv32-esp-elf/<IP_ADDRESS>_64eb9ff-8.4.0/riscv32-esp-elf/bin/riscv32-esp-elf-gcc failed: exit status: 1
          STDERR OUTPUT:
          /Users/imarkov/projects/esp32/rust-esp32-std-hello/.sdk/tools/riscv32-esp-elf/<IP_ADDRESS>_64eb9ff-8.4.0/riscv32-esp-elf/bin/../lib/gcc/riscv32-esp-elf/8.4.0/../../../../riscv32-esp-elf/bin/ld: /Users/imarkov/projects/esp32/rust-esp32-std-hello/.sdk/tools/riscv32-esp-elf/<IP_ADDRESS>_64eb9ff-8.4.0/riscv32-esp-elf/bin/../lib/gcc/riscv32-esp-elf/8.4.0/../../../../riscv32-esp-elf/lib/rv32im/ilp32/libc.a(lib_a-svfprintf.o): in function `_svfprintf_r':
          /builds/idf/crosstool-NG/.build/HOST-x86_64-apple-darwin12/riscv32-esp-elf/src/newlib/newlib/libc/stdio/vfprintf.c:1217: undefined reference to `__trunctfdf2'
          /Users/imarkov/projects/esp32/rust-esp32-std-hello/.sdk/tools/riscv32-esp-elf/<IP_ADDRESS>_64eb9ff-8.4.0/riscv32-esp-elf/bin/../lib/gcc/riscv32-esp-elf/8.4.0/../../../../riscv32-esp-elf/bin/ld: /Users/imarkov/projects/esp32/rust-esp32-std-hello/.sdk/tools/riscv32-esp-elf/<IP_ADDRESS>_64eb9ff-8.4.0/riscv32-esp-elf/bin/../lib/gcc/riscv32-esp-elf/8.4.0/../../../../riscv32-esp-elf/lib/rv32im/ilp32/libc.a(lib_a-vfprintf.o): in function `_vfprintf_r':
          /builds/idf/crosstool-NG/.build/HOST-x86_64-apple-darwin12/riscv32-esp-elf/src/newlib/newlib/libc/stdio/vfprintf.c:1192: undefined reference to `__trunctfdf2'
          collect2: error: ld returned 1 exit status

That's weird. I can't test it as windows spits out file name or extension is too long again, but I'm guessing it because of a missing -lgcc after -lc in the linker args.
Note: I've tested only esp32 up till now as I don't have the other chips on hand.

Are you preserving the order of all linker args that you get from cmake when sending these down to ldproxy? This is really really crucial. If you don't, then we'll be faced with issues like this. The thing is, the linking of ESP-IDF is a bit weird and very sensitive to correct linking order, as they override some symbols from -lc. That's why (a) I had to remove the -lc and -lm that are placed by the Rust libc crate itself - in ldproxy - as it is really breaking the link order; fortunately this won't be needed with the new libc used by Rust 1.56 and (b) I'm super careful to preserve the linkage order.

I'll try if we have the same issue with the ESP32.

Are you preserving the order of all linker args that you get from cmake when sending these down to ldproxy?

Well yeah that was it is doing, I'm getting these arguments from cmake. And my guess is that they are missing a final -lgcc to resolve link-dependencies in -lc which comes before it.

Okay, so my guess was partially correct. But it actually happens because rustc unconditionally passes -nodefaultlibs to the linker which causes these unresolved references.
There is a workaround:
Add rustflags = ["-C", "default-linker-libraries"] to the [target.riscv32imc-esp-espidf] section.
Not sure how to proceed with this though.

Okay, so my guess was partially correct. But it actually happens because rustc unconditionally passes -nodefaultlibs to the linker.
There is a workaround:
Add rustflags = ["-C", "default-linker-libraries"] to the [target.riscv32imc-esp-espidf] section.
Not sure what better way there is to prevent that.
Maybe we could emit something for ldproxy which removes the -nodefaultlibs arg only if it is not specified in the LinkArgs we get from cmake.

Argh, we have been looking at the same problem. See https://github.com/ivmarkov/embuild/issues/16

Argh, we have been looking at the same problem. See ivmarkov/embuild#16

Not to worry, I didn't spend much time on this; you did go much more in-depth than me, nice work.

I actually don't buy this:

I don't mind it so much for platformio, because if it needs platformio I can guess that the build script needs to install platformio (or just the needed toolchains) for it to work

If I paraphrase the above for the native build, it would sound just as convincing:
"I don't mind it so much for the Espressif idf.py build framework, because if it needs idf.py and the ESP-IDF SDK, I can guess that the build script needs to install the Espressif idf.py build framework and the ESP-IDF SDK for it to work.

That's not really true, don't you have to install idf.py and the toolchains used for the esp yourself?
I see where you're coming from though.

That is to say, whatever we decide to be the default, it should be for both build systems. To summarize, perhaps 4 options:

Option 1 - install and use the default global locations (i.e. ~/.platformio and ~/.espressif)

Pros:

Does not pollute the binary crate with stuff that should be .gitignore-d
Downloaded toolchains etc. can be re-used across builds of different binary crates
Already downloaded toolchains etc. will just be used

Cons:

Patching ESP-IDF by the build itself is not so ideal as this is kinda build-specific
Messes up with the global user configuration of either PlatformIO or Espressif idf.py

Option 2 - install and use an embuild-specific global location (i.e. ~/.embuild/platformio and ~/.embuild/espressif)

Pros:

Does not pollute the binary crate with stuff that should be .gitignore-d
Downloaded toolchains etc. can be re-used across builds of different binary crates
Already downloaded toolchains etc. will just be used
Does not mess up with the user configuration

Cons:

Patching ESP-IDF by the build itself is not so ideal as this is kinda build-specific
Creates something outside the binary crate directory

Option 3 - install and use an embuild-specific location within the root of the binary crate (i.e. .embuild/platformio and .embuild/espressif)

Pros:

Does not mess up with the user configuration
No global state
Patching ESP-IDF OK as it remains build specific

Cons:

Pollutes the binary crate with stuff that should be .gitignore-d
Still does not follow the cargo policy that everything should be in the $OUT_DIR
Downloaded toolchains etc. cannot be re-used across builds of different binary crates

Option 4 - install and use an embuild-specific location within the OUT_DIR of the binary crate (i.e. $OUT_DIR/embuild/platformio and $OUT_DIR/embuild/espressif)

Pros:

Does not mess up with the user configuration
No global state
Patching ESP-IDF OK as it remains build specific
Does not pollute the binary crate with stuff that should be .gitignore-d
Does not follow the cargo policy that everything should be int he $OUT_DIR
cargo clean means everything is removed, including ESP-IDf patches. Really clean state

Cons:

cargo clean means everything will be re-downloaded again
Downloaded toolchains etc. cannot be re-used across builds of different binary crates

Preferences for a default?
Mines are:

Option 2
Option 3 (if global state is something you guys feel uneasy)

In any case, having a flag/switch/env var/whatever so that the build framework is operating in Option 1 mode is a prerequisite IMO.

I don't like option 4 as it implies that every time esp-idf-sys is rebuilt all tools and the esp-idf repo have to be redownloaded.
And like you point out in the cons of option 1, we should probably not interfere with any installation the user did themselves, though in the case of platformio I don't think that is as much of a problem because really only the patches and scons scripts could interfere which I also think are generally useful.
If both build variants (i.e. pio and native) have to be consistent with the way they are installing the tools this would imply options 2 and 3.
But then it doesn't really make sense that this build script installs tools under a .embuild folder instead, the logic for installing these tools should be moved to the embuild library itself. But I'm not sure that we'd want install logic inside embuild  specific to the esp-idf.
Maybe the way forward would be to roll our own install logic (currently this leverages idf_tools.py) and install the tools this way. That wouldn't be specific to the esp-idf anymore and would be a good addition to embuild.
Leaving this issue aside for now though, I propose that we leave the installed tools under a folder in the workspace dir for now (there is already an environment variable to change the default directory). But we should definitely pick a better name than .sdk. I propose .espressif.
Other than that I think we should merge this like it is for right now and mention in the README/docs that this is an experimental feature for now.

If both build variants (i.e. pio and native) have to be consistent with the way they are installing the tools this would imply options 2 and 3.
But then it doesn't really make sense that this build script installs tools under a .embuild folder instead, the logic for installing these tools should be moved to the embuild library itself. But I'm not sure that we'd want install logic inside embuild specific to the esp-idf.

Actually, why not? As I mentioned in another comment, ESP-IDF (and Espressif) are big enough to deserve their own little "espidf" module in embuild? I don't consider this necessarily "dirty" given that it will nicely built upon the other modules of embuild.
But this can be a next step.

Maybe the way forward would be to roll our own install logic (currently this leverages idf_tools.py) and install the tools this way. That wouldn't be specific to the esp-idf anymore and would be a good addition to embuild.

I think that would mean extra maintenance burden for us. Why bother? And then - as per above, IMO no big deal to have something ESP-IDF specific in embuild, in its own module.

Leaving this issue aside for now though, I propose that we leave the installed tools under a folder in the workspace dir for now (there is already an environment variable to change the default directory). But we should definitely pick a better name than .sdk. I propose .espressif.

OK I can somehow live with that. However you really really don't like .embuild/espressif? With this name I can shuffle my platformio in .embuild/platformio as well...
So folks will just know of .embuild and won't be bothered which one is the default (we will silently switch from pio to ... how shall we name this? idf? in the next weeks but all embuild artefacts wiould still be in the .embuild folder.).

Other than that I think we should merge this like it is and mention in the README/docs that this is an experimental feature for now (and of course setting the default to pio, since it is experimental).

Agreed.

I think that would mean extra maintenance burden for us. Why bother? And then - as per above, IMO no big deal to have something ESP-IDF specific in embuild, in its own module.

I mean it is a maintenance burden for us either way if we use idf_tools.py or our own solution. We can still leverage the tools.json in the esp-idf which specifies which tools are where and what version, but without depending on a python script which also needs some workarounds (like the msystem one for windows).
Also, we'd need to add some logic in this regard anyway if we'd want to remove the requirement that the user has to install cmake themselves, which this currently requires as most cmake versions installed by idf_tools.py are too old even one version back (v4.2) and even the current version's (v4.3) cmake doesn't support the toolchains cmake-file-api object.

And then - as per above, IMO no big deal to have something ESP-IDF specific in embuild, in its own module.

Yeah, but having some code that depends on some python script inside the esp-idf which wasn't even documented before v4.2?

With this name I can shuffle my platformio in .embuild/platformio as well...

Do you mean you want to change the default of the platformio variant to <workspace dir>/.embuild/platformio? I think we should let the default of .platformio be or maybe event change it to ~/.embuild/platformio not to inside the workspace.
I picked .espressif as installing the tools doesn't really come from embuild but this build script so having them under .embuild/espressif would be misleading, wouldn't it? And we can change this location later as this is an experimental feature (i.e. unstable).

Also, we should probably release a new embuild version before this gets merged, so that I can use the new version in this PR.

Also, we should probably release a new embuild version before this gets merged, so that I can use the new version in this PR.

Done. Also created https://github.com/ivmarkov/embuild/issues/17


"Crawling File System" of packager insanely slow

I'm seeing it take upwards of 1.5 mins on a very basic project. From poking around the packager, it appears to be crawling every directory of my project root (all files under .git, node_modules, Pods, etc).
I'm not seeing a good way of configuring the ignoreFilePath callback (nor its blacklistRE).
Am I doing something wrong w/ the configuration/layout of my project?

@nevir When you run watchman -v from console, what version is reported?  This is usually due to watchman being out of date. I, for example, am using version 3.3.0.
https://facebook.github.io/watchman/docs/install.html for instructions on installing newer watchman releases.

Aha! No global instance of watchman installed, at all. After installing it, the crawl time is down to about a second.  Much better, thanks!

How about windows 10, there's no watchman on windows 10.

Same here. On windows 10 the file system crawling is awfully long. Is there a way to improve that ?


Champion search resets when any player locks in

While searching for a champion to get rune pages for (before locking in), the search is cleared as well as any results whenever any player locks in. Only happens when auto select is enabled.

Yeah... auto-select should never "clear" the champion selected....


Lightbulb is only visible when you can see the first column

This means you cannot see that code actions are available when authoring relatively long lines on a narrow monitor.

Tricky... The current idea of the lightbulb is to utilise the often occurring leading whitespace of a line. That doesn't work with horizontal scrolling. We have ideas to also show the lightbulb in the hover which might mitigate this.

/cc @stevencl

Yes, just commented on the original issue here: https://github.com/Microsoft/vscode/issues/25174#issuecomment-331472857.
Perhaps we should close this issue and keep the discussion going in the original?

Yeah, agreed. Closing as dupe of #25174


Support for methods

One thing that would be nice is to have the possibility to define methods that can afterwards be called using your library. A use case for that is that developers can define methods than can be called via GUI in a game editor, for example functions like "Reset" for a component.
I tried to use the existing mechanics but it crashes when trying to call the method:
Struct Test
{
void test(int parameter) { std::cout << "Called test with parameter: " << parameter << std::endl;
}:
namespace meta
{
template <>
inline auto registerMembers()
{
return members(member("test", &Test::test));
}
}
Test testObject;
meta::setMemberValue(testObject, "test", 12);
I think ideally it would work similar as members, e.g. :
namespace meta
{
template <>
inline auto registerMembers()
{
return members(method("test", &Test::test),member("x", &Test::getX));
}
}
Test testObject;
meta::callMethod(testObject, "test", 12);
What do you think ?

Oh, I just found out it is working already with members, I just used the wrong template parameter :-)
This is the working version:
Struct Test
{
void test(int parameter) { std::cout << "Called test with parameter: " << parameter << std::endl;
}:
namespace meta
{
template <>
inline auto registerMembers()
{
return members(member("test", &Test::test));
}
}
Test testObject;
meta::setMemberValue(testObject, "test", 12);

Yup, it works. Same for the getters. :D

Is there any trick to make it work with methods without parameters or with multiple parameters ?(Multiple parameters aren't necessary in my case but methods without parameters would be nice)

Sure, that’s fine.


 Can you update Colorbox so it works with SVG files?

None of my SVG images are working.

Then provide a demonstration.  SVG works fine in my testing.

My guess is that Colorbox's regex can't match the src you are using for your images.  Colorbox expects src to images to end with a known image file extension, for example: https://upload.wikimedia.org/wikipedia/commons/0/02/SVG_logo.svg
But you don't have to change your src, just set Colorbox's photo property to true to by-pass the regex matching: http://www.jacklmoore.com/colorbox/#setting-photo


DagsHub Logger: Fix unsupported metric formats for MLflow, Add example notebook

MLflow doesn't support some of the formats YOLO-NAS uses, so we either reformat them or raise an expiration that this format is not supported and will not be logged. This way, if YOLO-NAS uses a metric we didn't encounter during our tests and is not supported by MLflow, the training process will not stop, and the user will get an informative error message about it.

Hey @ofrimasad, I fixed the issue you mentioned, can this be merged in now?
Thanks,
Dean

@BloodAxe we reviewed and fixed your comments on the unsupported symbols. Please let me know if there's anything else, or if we can go ahead and merge this in. Thanks in advance! 🙏

Fixed both comments. Is it possible to merge now?

Hi @ofrimasad @BloodAxe
We've fixed all comments and ran a few tests to check the "happy path" works and other edge cases.
Is it possible to merge now?

Hey @BloodAxe @ofrimasad
Did you have a chance to review the changes? We want to promote the integration on our channels but waiting for the fixes to be merged.


Interest in joining the Tidier organization?

Hi @jdiaz97,
I’m one of the owners of the Tidier organization (https://github.com/TidierOrg), where we are trying to bring the comforts of R’s tidyverse to Julia.
So far, we have built 100% Julia implementations of large chunks of dplyr, tidyr, ggplot2, stringr, lubridate, and forcats.
If you’re interested in joining Tidier and/or bringing along this package (TidierVest?), would love to have you. If not, no worries and feel free to close this issue.

Sounds good, I'm in.

How do we proceed?

Awesome, I just added you as a member of TidierOrg on GitHub.
Can you transfer the repository to https://github.com.TidierOrg and rename it to "TidierVest.jl" (just for consistency w/ other package names)?
You should still have direct control over commits to it.
I can help with renaming it and also on the package registry.

Here are instructions on how to transfer: https://docs.github.com/en/repositories/creating-and-managing-repositories/transferring-a-repository

(Alternatively, if you'd like to keep this repo here, I can also clone this repo and host a cloned version on TidierOrg, with the idea being that further updates would get pushed there.)
Totally up to you, and again no pressure if you want to leave it here.

Let me know if still interested in either moving or cloning this repo within the Tidier org. Happy to help if still interested.

Hello! Sorry for the delay
I decided to clone and I'll maintin ofc
https://github.com/TidierOrg/TidierVest.jl
Unsure if it's better to "import code" or just push all the code.

All good, and welcome!
Feel free to set it up as a new repo (so it gets us own UUID) and push your code there.


Fix the Responsivenes

Fix the Responsiveness of the Humburger Icon in the team page

I want to work on this under JWOC

I want to work on this under JWOC

I guess you are already working on 2 issues. Finish them asap then i'll assign you.

I want to work on this under JWOC

I guess you are already working on 2 issues. Finish them asap then i'll assign you.

I have made a PR for one of them just now

@Tess-Vanta @Shruti025  First Come First serve :)


The data root directory (represented by the datarootdir Makefile var) of the dependencies is currently /usr/local/share

The correct location for datarootdir should be /usr/share. This must be set for configure, cmake and meson.

Solved.


our accomplishment section made responsive

Closes
Closes #1628

Description

Screenshots
https://github.com/OSCode-Community/OSCodeCommunitySite/assets/119532943/b498bd36-7288-4688-a296-ceda686d4362

Note to reviewers

Checklist:

[x] I have linked the PR to the correct issue.
[x] I have read the Contribution Guidelines
[x] I have built and tested the changes, and they do not break or show any errors.

@Divesh2003 can you acknowledge the PR and merge it .. our accomplishment section made responsive


feat: Added Hash to DynSolType and StructProp

I saw the open issue #402 and decided to implement it
Solution
Simply added hash to the structs and tests to make sure nothing broke
PR Checklist

[x] Added Tests - Yes
[?] Added Documentation - Not sure if there was a need
[] Breaking changes - No

Closed while I search for an alternative to STD default hasher for the tests

Closed while I search for an alternative to STD default hasher for the tests


Why the wait_processing_interval key not exists in deliver action

as we know, iTC processes the build after uploading a ipa file, the processing interval is different, may be 30 seconds, 1 minute or even 1week.
pilot action has the functionality that upload binary ipa, and then submit it, so pilot provide the wait_processing_interval key to specify the time interval to call submit api, default to 30s.
deliver action also has the functionality to upload & submit, but it does not provide a akin key, so why ?

Feel free to create a new Issue to discuss a proposed change. 👍


Request body is preserved across 'next' calls #147

added test case to confirm issue is still there
adapted patch by @thomasjm https://github.com/scotty-web/scotty/pull/206/files to handle max body size parameter

@RyanGlScott Hi, github suggested you as a reviewer :) the original bug (#147 ) is fixed, but CI fails due to an unrelated test: an exception raised by a too large request body is uncaught, which I find very puzzling since I didn't modify exception behaviour in this PR. Thanks for all pointers.


I want to add JavaScript animation resources

Is your feature request related to a problem? Please describe.
Users often encounter challenges finding comprehensive and up-to-date resources for JavaScript animations. These resources could include libraries, tutorials, frameworks, or best practices. The lack of centralized, curated, and comprehensive information can frustrate individuals looking to create engaging JavaScript animations.
Describe the solution you'd like
I would like to propose adding a curated list or a dedicated section within our repository that focuses specifically on JavaScript animation resources. This section could include links to libraries, tutorials, articles, and frameworks that assist developers in creating animations using JavaScript. Could further aid users in finding the most relevant resources for their needs.
Describe alternatives you've considered
One alternative might be creating a separate website solely dedicated to JavaScript animation resources. However, integrating this within our existing repository would centralize the information for users already engaging with our project, providing added value without needing separate navigation.
Additional context
Adding this feature could significantly enhance the usability and value of our repository, catering to the needs of developers seeking guidance or resources on implementing JavaScript animations. This enhancement aligns intending to provide comprehensive resources and support within our community.

Is this still available?
I want to take it up. I can add libraries, tutorials and frameworks, and other supporting resources for JS animations.

Is this still available? I want to take it up. I can add libraries, tutorials and frameworks, and other supporting resources for JS animations.

Yes, it is.  I've assigned it to you.

noted.


Remove Q::Statement::Block

The Qtype is not pulling its own weight. I have written a fair amount of idiomaic 007 code at this point, and I haven't ever felt a need to just put in a block statement somewhere.
The typical use case otherwise is to willingly isolate small scopes from each other, preventing local variables from bleeding too far, etc. This could happen for example when writing isolated unit tests. But, meh. I don't see a big need.
If someone really wanted to do a once-only block after Q::Statement::Block has bit the dust, perhaps the simplest way would be:
if True {
    # ...
}

or
for ^1 {
    # ...
}

That latter one is even nicely self-documenting, in a way. "Do it once."

Closing this one; don't know what I was thinking.
We use statement blocks quite naturally and fruitfully in the test suite already. See, for example, https://github.com/masak/007/blob/6216fff8d28ef6dec43bd23efe5dd77098c5b56f/t/features/custom-ops.t#L533-L539 — I wouldn't want to re-write that as any of the above forms just to get a nested block.


#3 Adding caching library.

Codecov Report

:exclamation: No coverage uploaded for pull request base (main@45dbf3c). Click here to learn what that means.
The diff coverage is n/a.

@@           Coverage Diff            @@
##             main        #5   +/-   ##
========================================
  Coverage        ?   100.00%
========================================
  Files           ?         1
  Lines           ?       292
  Branches        ?        35
========================================
  Hits            ?       292
  Misses          ?         0
  Partials        ?         0

Flag
Coverage Δ

cache
100.00% <0.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.


Iphone IOS10 Safari array map function not work in react render

React Easy State version: 7.0.0-alpha.1
Platform: Iphone IOS10 Safari
Describe the bug
in ios10 safari ,
use store wrap a array,
in react render function ,use this array's map function not render item
This is the right situation

This is the problematic situation

This is the source code
const datas = store({ list: [{ a: 1 }, { a: 2 }] });

function App() {
  const items = [];
  datas.list.forEach(item => {
    items.push(
{item.a}
);
  });
  return (

Hello CodeSandbox

      {datas.list.map(item => {
        return
{item.a}
;
      })}

===============

      {items}

  );
}

export default view(App);

For tougher bugs
To Reproduce
https://codesandbox.io/s/react-easy-state-alpha-42dgc?file=/src/App.js

I added the following code
console.log(items, [...datas.list.map(item=>
{item.a}
)])

Safari IOS10 print

Chrome print

The error should have occurred in this package @nx-js/observer-util
in proxyHandlers.js file remove construct handler IOS10 is OK

I add some log and found that it will return an empty array first，and then add the array elements on chrome, but not on Safari
Screenshot of chrome is：

This is the first time I have checked the source code of this package, it is not clear why this happens

Is caused by this https://github.com/nx-js/observer-util/issues/48


[9.x] Use new CommonMark 2.2 API

We've fixed the build on 8.x so please rebase and mark as ready when tests pass. Thanks!


Remove login_se_chat()

This broke after the new login rolled out... and turns out, we don't need it! I think this should work, but I'm not certain. Might want to make sure Travis passes and if not, revert this commit.
r? @Manishearth

Sure it's not just picking up on cookies on your machine or something? (Haven't tested it myself)

I don't think so... but I'm not sure.

I've tested it on my chatbot and it appears to work.


Fix typo in migrate.md

@alexbuckgit : Thanks for your contribution! The author(s) have been notified to review your proposed change.

#sign-off

@alexbuckgit: I'm sorry - only the author of this article, @deepakonics, can sign off on your changes. But we do have an exception process - if you are on the Microsoft content or product team for this product area, you can ask the PR review team to review and merge it by sending mail to the techdocprs alias.

Looks like the change is merged, thank you!


Fixed #26582 - prettier admin display for list values

Opting for the simple solution outlined in https://code.djangoproject.com/ticket/26582
Also has the nice side-effect of correctly rendering value types returned by fields Django doesn't know about - for instance, a custom field that returns datetime.date but is not a DateField subclass.

Thanks for the patch, however, tests are also required.

Added tests for formatting lists.


Text and General background don't match.

VSCode Version: Code 1.7.1 (02611b40b24c9df2726ad8b33f5ef5f67ac30b44, 2016-11-03T13:46:00.226Z)
OS Version: Darwin x64 16.1.0
Extensions:

Extension
Author
Version

python
donjayamanne
0.3.23

restructuredtext
lextudio
0.0.17

preview-vscode
searKing
1.1.1

rst-vscode
tht13
1.3.0

The 'grays' don't match in VS Code.  The background near text is #232323 in OSX, but #1E1E1E on my Windows machine.  The 'general' background is #1E1E1E on both.
When scrolling, you can see the widths of the gray blocks changing as text of varying widths gets virtualized in and out.
Steps to Reproduce:

Open a new file
alt 1. Open terminal

Dupe of #12473


Allow unicode characters in usernames

Some foreign characters (non-english) which some people might want to use in their usernames are currently coming back as invalid.
Clearly we don't want to allow all characters for security reasons but we should allow the most common foreign ones.

Is there a good blacklist to use?
It seems like we shouldn't allow zero-width characters and control characters, at least.

I would like to see unicode support too. You need to do a couple of things.
Normalization, see MediaWiki for a widely used implementation of type C: http://www.unicode.org/reports/tr15/
For the security aspect, see here: http://www.unicode.org/reports/tr39/
I'm sure there are implementations of these you could grab from somewhere.


ERROR  [nuxt] Could not compile template i18n.options.mjs.

Environment
Working directory: /home/meerzulee/Workspace/Jumbo/aptocms-web-nuxt                                                    10:37:28 AM
Nuxt project info: (copied to clipboard)                                                                               10:37:28 AM

Operating System: Linux
Node Version:     v18.17.1
Nuxt Version:     3.7.3
CLI Version:      3.8.4
Nitro Version:    2.6.3
Package Manager<EMAIL_ADDRESS>Builder:          -
User Config:      devtools, css, components, modules, googleFonts, i18n, directus, imports, pinia
Runtime Modules<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Build Modules:    -

Reproduction
Here is my package.json:
"devDependencies": {
"@iconify-json/mdi": "^1.1.54",
"@nuxt/devtools": "0.8.3",
"@nuxtjs/i18n": "^8.0.0-rc.4",
"@nuxtjs/google-fonts": "^3.0.2",
"@types/node": "^20.6.3",
"@unocss/nuxt": "^0.56.0",
"@unocss/preset-icons": "^0.56.0",
"@unocss/preset-wind": "^0.56.0",
"@unocss/transformer-directives": "^0.56.0",
"nuxt": "^3.7.3",
"nuxt-directus": "^5.5.0",
"vue-tsc": "^1.8.11"
},
"dependencies": {
"@pinia/nuxt": "^0.4.11",
"@unocss/reset": "^0.56.0"
}
And nuxt.config  i18n section:
i18n: {
vueI18n: "./i18n.config.ts",
compilation: {
strictMessage: false,
},
trailingSlash: true,
strategy: "prefix",
defaultLocale: "ru",
lazy: true,
langDir: "locales",
locales: [
{ code: "ru", file: "ru.js" },
{ code: "en", file: "en.js" },
{ code: "kg", file: "kg.js" },
],
},
run "yarn" or "yarn dev"
Describe the bug
After installing     "@nuxtjs/i18n": "^8.0.0-rc.4", "yarn" and "yarn dev" throws and error:
ERROR  [nuxt] Could not compile template i18n.options.mjs.
Inside .nuxt folder I noticed there is no i18n.options.mjs. file has been generated.
Additional context
No response
Logs
No response

It's difficult to determine the cause of the error without a minimal reproduction, could you provide one? You can use this starter to set it up.
I'm not sure what locale files look like, but have you tried adding experimental: { jsTsFormatResource: true } to the i18n configuration?

I guess this error occurs only on specific conditions which I am not sure how to reproduce.
I have created a new project and then added the i18n module, everything works well!


function Faker.Date.date_of_birth/1 is undefined

I don't know what is happening but I can't use the all the Faker.Date methods.
iex(5)> Faker.Date.date_of_birth(10..19)
** (UndefinedFunctionError) function Faker.Date.date_of_birth/1 is undefined (module Faker.Date is not available)
    Faker.Date.date_of_birth(10..19)

does anyone have some idea why?

What version of Faker are you using?
What version of Elixir are you using?
date_of_birth is a very recent addition and is only enabled for Elixir >=1.3 IIRC

@tobyhinloopen Hello, my versions are:
# Elixir
➜  ~ elixir --version
Elixir 1.4.1

# Version
:faker, "~> 0.7"

I think you are using hex version of faker. This Date module part of unreleased version: https://github.com/igas/faker/blob/master/CHANGELOG.md
You can change you dependency to get latest version from github if you want.

hummmmmm @igas make sense. Thank you and my apologies. I'm so dumb. xD

@adrianotadao no worries, it's my fault, I have a plans for this version and don't have much time right now =(
@tobyhinloopen thank you. For all help in recent issues and PRs.


Fix "IDHandler.get_prop is not a function" error when calling some methods on WebRTCDataChannel

When calling WebRTCDataChannel.is_negotiated() and a few other methods on WebRTCDataChannel, I get this error when using the HTML5 export:
IDHandler.get_prop is not a function

It looks like this method is really on GodotRTCDataChannel!
With the changes in this PR, I'm able to call those methods.
This should cherry-pick to Godot 3.x

Cherry-picked for 3.4.

Cherry-picked for 3.3.3.


Creating Pages Instead Of Posts

I would like to create "pages" as opposed to "posts" that are displayed on the homepage. Is there an effective way to accomplish this?
My goal is to have pertinent pages in the header (about us, contact, etc).

+1 for this.  I'd like static pages that I can place in the main menu, but do not display in 'recent posts'.  Also being able to sticky them to the homepage would be nice.

Have a look at this issue's comment which explains in more detail how this is solved in the upcoming version of this theme.


get_path_to_repo_root() failing after release of v2.1.2

Describe the bug
Calls of the function get_path_to_repo_root() failing only on v2.1.2, if i pin to v2.1.1.
To Reproduce
Run this action with inputs:
...
jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Check terragrunt HCL
        uses<EMAIL_ADDRESS>        with:
          tf_version: "1.7.5"
          tg_version: "0.55.18"
          tg_dir: "deployment"
          tg_command: "run-all validate"
        env:
          TARGET: "dev"

Expected behavior
Validate passes (does locally, and does if version pinned to 2.1.1).
Nice to have

[x] Terminal output
Fail on 2.1.2:

2024-05-11 15:29:42 Starting Terragrunt Action
2024-05-11 [15](https://github.com/oliverlambson/env.infrastructure/actions/runs/9044698053/job/24853626016?pr=5#step:4:16):29:42 Installing Terraform version 1.7.5
cloning https://github.com/asdf-community/asdf-hashicorp.git
mise plugin:terraform ✓ https://github.com/asdf-community/asdf-hashicorp.git#f[16](https://github.com/oliverlambson/env.infrastructure/actions/runs/9044698053/job/24853626016?pr=5#step:4:17)02b6
installing
~/.local/share/mise/plugins/terraform/bin/install
Downloading terraform version 1.7.5 from https://releases.hashicorp.com/terraform/1.7.5/terraform_1.7.5_linux_amd64.zip
Skipping verifying signatures and checksums either because gpg is not installed or explicitly skipped with ASDF_HASHICORP_SKIP_VERIFY
Cleaning terraform previous binaries
Creating terraform bin directory
Extracting terraform archive
mise<EMAIL_ADDRESS>✓ installed
mise ~/.config/mise/config.toml tools<EMAIL_ADDRESS>2024-05-11 15:29:43 Installing Terragrunt version 0.55.[18](https://github.com/oliverlambson/env.infrastructure/actions/runs/9044698053/job/24853626016?pr=5#step:4:19)
cloning https://github.com/ohmer/asdf-terragrunt.git
mise plugin:terragrunt ✓ https://github.com/ohmer/asdf-terragrunt.git#4a6651a
installing
~/.local/share/mise/plugins/terragrunt/bin/install
Downloading terragrunt from https://github.com/gruntwork-io/terragrunt/releases/download/v0.55.18/terragrunt_linux_amd64
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 71.8M  100 71.8M    0     0   161M      0 --:--:-- --:--:-- --:--:--  225M
mise<EMAIL_ADDRESS>✓ installed
mise ~/.config/mise/config.toml tools<EMAIL_ADDRESS>time=[20](https://github.com/oliverlambson/env.infrastructure/actions/runs/9044698053/job/24853626016?pr=5#step:4:21)24-05-11T15:29:44Z level=error msg=Error: Error in function call

 prefix=[/github/workspace/deployment/platform/vpn]
time=2024-05-11T15:29:44Z level=error msg=  on /github/workspace/deployment/platform/vpn/terragrunt.hcl line 9, in dependency "vpc":
 prefix=[/github/workspace/deployment/platform/vpn]
time=20[24](https://github.com/oliverlambson/env.infrastructure/actions/runs/9044698053/job/24853626016?pr=5#step:4:25)-05-11T15:29:44Z level=error msg=   9:   config_path = "${get_path_to_repo_root()}/deployment/platform/vpc"
 prefix=[/github/workspace/deployment/platform/vpn]
time=2024-05-11T15:29:44Z level=error msg=
 prefix=[/github/workspace/deployment/platform/vpn]
time=2024-05-11T15:29:44Z level=error msg=Call to function "get_path_to_repo_root" failed:
[/github/workspace/deployment/platform/vpn] exit status 1[28](https://github.com/oliverlambson/env.infrastructure/actions/runs/9044698053/job/24853626016?pr=5#step:4:29).

 prefix=[/github/workspace/deployment/platform/vpn]
time=2024-05-11T15:[29](https://github.com/oliverlambson/env.infrastructure/actions/runs/9044698053/job/24853626016?pr=5#step:4:30):44Z level=error msg=Error: Unsuitable value type

... (continues with same error in other places)

Pass on 2.1.1:
2024-05-11 16:10:29 Starting Terragrunt Action
2024-05-11 16:10:29 Installing Terraform version 1.7.5
cloning https://github.com/asdf-community/asdf-hashicorp.git
mise plugin:terraform ✓ https://github.com/asdf-community/asdf-hashicorp.git#f[16](https://github.com/oliverlambson/env.infrastructure/actions/runs/9044698053/job/24853626016?pr=5#step:4:17)02b6
installing
~/.local/share/mise/plugins/terraform/bin/install
Downloading terraform version 1.7.5 from https://releases.hashicorp.com/terraform/1.7.5/terraform_1.7.5_linux_amd64.zip
Skipping verifying signatures and checksums either because gpg is not installed or explicitly skipped with ASDF_HASHICORP_SKIP_VERIFY
Cleaning terraform previous binaries
Creating terraform bin directory
Extracting terraform archive
mise<EMAIL_ADDRESS>✓ installed
mise ~/.config/mise/config.toml tools<EMAIL_ADDRESS>2024-05-11 16:10:30 Installing Terragrunt version 0.55.[18](https://github.com/oliverlambson/env.infrastructure/actions/runs/9044698053/job/24853626016?pr=5#step:4:19)
cloning https://github.com/ohmer/asdf-terragrunt.git
mise plugin:terragrunt ✓ https://github.com/ohmer/asdf-terragrunt.git#4a6651a
installing
~/.local/share/mise/plugins/terragrunt/bin/install
Downloading terragrunt from https://github.com/gruntwork-io/terragrunt/releases/download/v0.55.18/terragrunt_linux_amd64
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

100 71.8M  100 71.8M    0     0  69.2M      0  0:00:01  0:00:01 --:--:-- 69.2M
mise<EMAIL_ADDRESS>✓ installed
mise ~/.config/mise/config.toml tools<EMAIL_ADDRESS>time=[20](https://github.com/oliverlambson/env.infrastructure/actions/runs/9044698053/job/24853626016?pr=5#step:4:21)24-05-11T16:10:32Z level=info msg=The stack at /github/workspace/deployment will be processed in the following order for command validate:
Group 1
- Module ...
- Module ...

Group 2
- Module /github/workspace/deployment/platform/vpn
- ...

...
time=2024-05-11T16:10:33Z level=info msg=Skipping terragrunt module /github/workspace/deployment/platform/vpn/terragrunt.hcl due to skip = true. prefix=[/github/workspace/deployment/platform/vpn]
Success! The configuration is valid.

... (continues)

Versions

Terragrunt Action version: 2.1.2
Environment details (Terragrunt version, Terraform version, etc.):

Additional context
I think it's this change: https://github.com/gruntwork-io/terragrunt-action/compare/v2.1.1...v2.1.2#diff-e4d34fc353097073f2f6ebbdc1d89cbdac9f0edf5354442c2f884839a2e8afb3R104

Having the same issue with get_repo_root()function as well.
Hoping this can be sorted quickly as its currently broken our workflows properly 🫠

Me too

We are also experiencing this issue. We've confirmed that this behavior is new in v2.1.2. When we pin to v2.1.1, we do not encounter the Call to function "get_repo_root" failed: error.

Thank you @denis256 🙌

Included permissions fix in https://github.com/gruntwork-io/terragrunt-action/releases/tag/v2.1.3


ONNX export from Java

I came across this repo from the DJL repo, and you might be interested in the ONNX export functionality we built in Tribuo's last release (v4.2.0). We have a separate module (which only depends on protobuf) that makes writing ONNX models from Java significantly easier, it provides similar helper functionality to the Python ONNX module. We don't have full coverage of the space of ONNX operators yet, but we'd be happy to work with you to expand that set.
The Maven artifact is here, the javadoc is here, and our repo is on GitHub.
You can see examples of how we build ONNX models in the rest of Tribuo (e.g. exporting a FactorizationMachine).

Thanks for the tip! I'm looking forward to taking a close look.
By the way, do you have a MuZero implementation on Tribuo? ... I am asking because I want to export models trained in DJL from a small MuZero sample implementation to run in a browser using ONNX Runtime Web.

We don't have a MuZero implementation, Tribuo's focus is more on being scitkit-learn in Java than being a general purpose deep learning system. We do classification & regression rather than reinforcement learning, and haven't spent too much time thinking about what an RL system would look like in Java.

Thanx for the insight to Tribuo.


CAP-218:  Drop the hibernate3-maven-plugin in favor of providing dire…

…ct support for database schema drop+create
https://issues.jasig.org/browse/CAP-218
The Calendar portlet implements a 'db-init' Maven goal that drops & re-creates the database schema.  This capability is integrated into the uP4 Maven build via uportal-portlets-overlay.
The current implementation is based on the hibernate3-maven-plugin.  This solution is problematic because (1) it's not well integrated into the rest of the app's features (so it's always been very brittle), and (2) it doesn't port well to the Gradle-based build in uP5.
We need to  replace it with something we provide directly that is a first-class part of the Spring app.

Jasig/AnnouncementsPortlet/pull/54

Updated with the latest from master and re-tested locally.


When performing load testing with the number of concurrent transactions more than 15 on each chain, the relay is not working as expected.

Describe the bug
A clear and concise description of what the bug is.
To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.
Additional context
Add any other context about the problem here.

no longer required.


Define initial metrics to export to Prometheus as MVP

Prometheus MVP Metrics

The Sovrin Network name being monitored
Should be able to get this from the pool being connected to

Node alias name

Detect when a node is inaccessible and produce standard output for that situation.

Should generate a timeout when trying to pull validator_info from inaccessible nodes.

Detect any nodes that are accessible but that are "unreachable" to some or all of the other Indy nodes.

That indicates that the internal port to the node is not accessible, even though the public port is accessible.

"Reachable_nodes": [
            [
              "Node1",
              0
            ],
            [
              "Node3",
              null
            ],
            [
              "Node4",
              null
            ]
          ],
          "Unreachable_nodes": [
            [
              "Node2",
              null
            ]
          ],
          "Reachable_nodes_count": 3,
          "Unreachable_nodes_count": 1,

The number of transaction per Indy ledger, especially the domain ledger.

"transaction-count": {
              "ledger": 21,
              "pool": 4,
              "config": 0,
              "audit": 1042
            },

The average read and write times for the node.

"throughput": {
              "0": 0.0017547843
            },
            "master throughput": 0.0017547843,
            "total requests": 16,
            "avg backup throughput": null,
            "master throughput ratio": null,
            "average-per-second": {
              "read-transactions": 0.0338584473,
              "write-transactions": 0.0001539895
            },

The average throughput time for the node.

"throughput": {
              "0": 0.0017547843
            },
            "master throughput": 0.0017547843,
            "total requests": 16,
            "avg backup throughput": null,
            "master throughput ratio": null,
            "average-per-second": {
              "read-transactions": 0.0338584473,
              "write-transactions": 0.0001539895
            },

The uptime of the node (time is last restart).

    "transaction-count": {
              "ledger": 21,
              "pool": 4,
              "config": 0,
              "audit": 1042
            },
            "uptime": 103903
          },

The time since last freshness check (should be less than 5 minutes).

          "Freshness_status": {
            "1": {
              "Last_updated_time": "2020-07-06 23:55:07+00:00",
              "Has_write_consensus": true
            },
            "0": {
              "Last_updated_time": "2020-07-06 23:57:33+00:00",
              "Has_write_consensus": true
            },
            "2": {
              "Last_updated_time": "2020-07-06 23:57:33+00:00",
              "Has_write_consensus": true
            }
          }

Node IP address information

"Node_info": {
          "Name": "Node4",
          "Mode": "participating",
          "Client_port": 9708,
          "Client_ip": "<IP_ADDRESS>",
          "Client_protocol": "tcp",
          "Node_port": 9707,
          "Node_ip": "<IP_ADDRESS>",

Total nodes in pool information

"Pool_info": {
          "Read_only": false,
          "Total_nodes_count": 4,

Since Prometheus will only gather numeric metrics, there are some things to consider when modeling the metrics.

The Sovrin Network name being monitored

Should be a label attached to all metrics.

Node alias name

Should also be a label, for each node we get (with the node being the top-level objects in the duct we are currently fetching).

Detect when a node is inaccessible and produce standard output for that situation.

This would happen outside of the exporter, either in Prometheus through Altermanager, or in Grafana.

The number of transaction per Indy ledger, especially the domain ledger.

Should work as Gauge transactions_total with a label per ledger.

The average read and write times for the node.

Here I wonder how the values are measured. Ideally, we could just record the total requests in a Gauge and let Prometheus infer the other metrics. Else having histograms for throughput might be fine, we just have to be careful with regards to statistically wrong double aggregations.

The uptime of the node (time is last restart).

Clearly a gauge with a label per node.

The time since last freshness check (should be less than 5 minutes).

Diff against time of the and record as Gauge?

Node IP address information

This could be a label, same as the node name.

Total nodes in pool information

Gauge with pool name as label.

One question regarding freshness status:
When I have a test network with 4 nodes, I get 3 freshness values, as you have posted above:
          "Freshness_status": {
            "1": {
              "Last_updated_time": "2020-07-06 23:55:07+00:00",
              "Has_write_consensus": true
            },
            "0": {
              "Last_updated_time": "2020-07-06 23:57:33+00:00",
              "Has_write_consensus": true
            },
            "2": {
              "Last_updated_time": "2020-07-06 23:57:33+00:00",
              "Has_write_consensus": true
            }
          }

What does these numbers as keys (0,1,2) represent and how should we interpret them?