Documentation of properties could be improved

There are a couple of issues in the documentation of properties. First, the following example is provided:

meta::reflect<my_type>("reflected", std::make_pair("tooltip", "message"));

In this case, the first element of the pair, which is used to identify the property, will decay to a const char* and then stored in a meta::any object. Subsequently, properties are found applying operator == on meta::any instances, which is implemented by comparing the value of the pointers. This is a problem, because string literals potentially all have unique addresses, so the property cannot be portably and reliably be retrieved at later time.
The retrieval example is also problematic:

meta::prop prop = meta::resolve<my_type>().prop("tooltip");

since this doesn't compile on at least one compiler (Visual Studio 2019). The problem is that in this case the literal string does not decay to a pointer and apparently the meta::any constructor is not able to accept a const char (&)[8].
These two issues could be solved by using std::string as the type of the property key. However, the use case described in the documentation seems legit and useful, so if that's the intent, we should fix the code instead so that it works as described. There may be multiple ways to achieve this, for example, we could add the following non-template overload of the meta::type::prop method:
       const auto* curr = internal::find_if<&internal::type_node::prop>([key](auto* candidate) {
          auto ptr = candidate->key().try_cast<const char*>();
          return ptr && strcmp(*ptr, key) == 0;
       }, node);

       return curr ? curr->clazz() : meta::prop{};
    }```

As stated in the README, this is a standalone version of the meta subdir from EnTT. The doc is similar too and this is something that would work with EnTT but that cannot work with this library. It's due to the fact that here the identifiers are actual strings and the library hashes them under the hood while in the original version the identifiers are literally numbers (usually hashed strings).
Probably the best bet is to use the model of EnTT for this standalone version as well. strcmp isn't the fastest thing ever and definitely something to avoid using massively in production. Am I wrong?
The drawback is that the name won't be available anymore because we cannot get the original string from its hash. However one can still re-add it as a property when needed.

Of course, using strcmp is not the fastest approach but at least it's guaranteed to be correct and reliable, while using a hash always exposes you to the risk of a hash collision. If the user wants to use hashes for best performance, he/she could just declare a custom literal operator "_hash" and write "hello"_hash instead of just "hello". OTOH, if you used hash internally, the user would have no way to get back to strcmp, if desired. The hash approach is is correct in EnTT since meta is not directly exposed to the user. However, when designing a standalone library, my rule of thumb is "between two approaches, one suboptimal but correct and one optimal with caveats, the default should be the correct one". That said, I agree with you that using strings for id is not the best things for performances and would use a different kind of ids in production.

I'd use integers for the identifiers to be honest. They are pretty handy: you can use hashed strings ofc, but also runtime or compile-time generated ids as well as identifiers obtained through a pre-processing step. Moreover, they are definitely by far faster than comparing strings.
In this case I don't consider hash collisions a problem though, neither for the types nor for the properties. We have a target space that is much larger than what you need for this kind of tool usually. Moreover, strings here are user defined, not something random that we receive from an external source on which we have no control. Long story short: I never hit a collision but I'd have no problems to flip a character in this case! :-)
That said, the idea of setting the name of the type from its id hasn't revealed a good choice.
First of all, most of the time I don't care of the name and therefore I'm just wasting a pointer for literally nothing. Moreover, the library allows you to set the name as a property in case you need it, so an alternative exists. Finally, ids and names aren't always the same, there are cases in which I use numeric ids that have nothing to do with actual names (when I use meta through EnTT, because it decouples the two things).
So, to sum up, rather than updating the doc to reflect the current behavior, I'd change the latter to make it closer to that of EnTT, that is more correct from my point of view.
What do you think about?

I agree that integers are better for the task of identifying properties, but I believe enumerators are even better. They behave like integers but they automatically also acts as a sort of "namespace", as enumerators from different enumerations do not clash even if they have the same numerical value. If I'm not mistaken you use enums in the test suite, which is good.
I have to trust you if you say the EnTT approach is better and I see no problem if you decide to go that way. The second best alternative remains to just rewrite the docs to use enums instead of strings.

but I believe enumerators are even better

The only problem with enums here is that we don't know in advance what type the user is going to use. On the other side, reflect<T>(??) need an actual type for ??.
By the way, both old fashioned enums and enum classes can be converted to/from integers. Therefore to use integers in place of strings means that we can use also enums for the identifier for our meta types other than an hashed strings and the others.
Does it make sense?

I lost you. Allow me to rephrase. I was referring to the following idiom, similar to the one that meta already uses in the test suite:
enum class properties {
    nice_property = 0,
    another_property = 1
};
int main()
{
    meta::reflect<MyType>("MyType", std::make_pair(properties::nice_property, 42)) /*...*/;
    auto prop = meta::resolve<MyType>().prop(properties::nice_property);
    assert(prop.value() == 42);
}

I could have used int instead of the property enum, std::make_pair(0, 42) to set the property and .prop(0) to retrieve it, but using an enum makes the code self-describing and avoids collisions with a potential different set of properties. What is the problem you see with this approach?

Oh, sorry, I thought we were speaking of the names for the meta type, like:
meta::reflect<MyType>(MyEnum::MyValue)

Instead of:
meta::reflect<MyType>(hash("MyName"))

Enums already work like a charm with properties afaik. :+1:

So, to sum up, the idea is to use integers (hashed strings or whatever, it's up to the user to decide it) for meta types' names, as it happens in EnTT. On the other side, properties already work and I should only update the doc to add a note about using strings with them. Right?

Enums already work like a charm with properties afaik. 👍

Yes, they do!

So, to sum up, the idea is to use integers (hashed strings or whatever, it's up to the user to decide it) for meta types' names, as it happens in EnTT. On the other side, properties already work and I should only update the doc to add a note about using strings with them. Right?

That's a good plan. I like it.

I've pushed the version that uses numeric identifiers on the branch wip.
I've still to review the documentation, but the code is there in its final version.
Any comment?

I checked the wip branch and it seems good to me.


Catalog pagination breaks sort_order? [v5.5]

Reported by Art:

I've been cleaning up our reservations site so students can more easily navigate it. One issue I'm having is getting certain equipment models to appear with the rest of their categories on the same page. For example I have a category called "Cables". Every model I have tagged as Cable appears on the first page of our site except one VGA cable which stubbornly stays on the second page. This is confusing for the students because they don't think to keep looking for other types of cables. How can I change this?

It gets weirder - when I'm set to view 50 items per page, 8 of the 9 cables appear on the first page, but when I'm set to view 30 items per page only 7 of them do. Similarly, the next category down has 8/8 on page 1 for 50 items/page, but only 5/8 for 30 items/page. Even stranger, the categories don't obey the sort order on the second page, i.e. missing models from the top category on page 1 are not at the top of page 2.
This is pretty critical, and somewhat complex - I couldn't figure out the issue by scanning through the relevant code (prepare_catalog_index_vars, relevant views). This is top priority for the next release.

Never mind, all of the categories had a sort order set to zero so I'm guessing there were some issues with the sorting in the query. I manually went in and assigned sort orders, and it was pretty painful... I'm going to repurpose this issue for making sort order assignment drag-and-drop (similar to #250), although it will probably wait for the hypothetical JS rewrite.


Removed unused libraries

Codecov Report

Merging #18 into master will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master      #18   +/-   ##
=========================================
  Coverage     41.73%   41.73%
  Complexity       93       93
=========================================
  Files            40       40
  Lines           611      611
  Branches         36       36
=========================================
  Hits            255      255
  Misses          344      344
  Partials         12       12

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9572d76...48a932b. Read the comment docs.


Fix upstream dependent test failure in Laravel 11.23 and later

Summary
Fixed test failures in Laravel 11.23 and later. The cause is the following changes on the laravel/framework side:

laravel/framework#52729
laravel/framework#52762

I addressed the issue by changing the Laravel method that we were testing.
The following points were taken into consideration when selecting the target method:

It must be a method of the Eloquent Builder class, just like before the fix.
The content does not change between the CI options prefer-oldest and prefer-stable, and is unlikely to change in the future.
It must be a method that accepts multiple parameters.
Some of these parameters must have default values.

By satisfying these conditions, we can perform tests equivalent to with(), which was the target of tests before the fix. upsert met the conditions, so I selected it.
Note
This pull request, #1594, and #1591 can all be merged independently.
IMO, the following order is best:

Merging 1594 should make all of this pull request's tests pass.
If we merge this pull request, all of 1591's tests should pass.
1591 contains a breaking change. Merge it last.

Type of change

[x] Misc. change (internal, infrastructure, maintenance, etc.)

Checklist

[x] Existing tests have been adapted and/or new tests have been added

It seems to work now :)
Guess we'll just stick with the latest version for now.


refactor: updating field conversion code without pointer hack

We currently use a pointer hack for functions like calc_num_bn254_frs and convert_from_bn254_frs. This PR aims to clean these up using traits and template metaprogramming.
Also closes https://github.com/AztecProtocol/barretenberg/issues/846, by just sending and receiving an std::array instead of an AllValues object.
Please read contributing guidelines and remove this line.

Benchmark results
No metrics with a significant change found.

Detailed results
All benchmarks are run on txs on the Benchmarking contract on the repository. Each tx consists of a batch call  to create_note and increment_balance, which guarantees that each tx has a private call, a nested private call, a public call, and a nested public call, as well as an emitted private note, an unencrypted log, and public storage read and write.
This benchmark source data is available in JSON format on S3 here.
Values are compared against data from master at commit ce51d206 and shown if the difference exceeds 1%.
L2 block published to L1
Each column represents the number of txs on an L2 block published to L1.

Metric
8 txs
32 txs
128 txs

l1_rollup_calldata_size_in_bytes
45,700
180,580
720,100

l1_rollup_calldata_gas
222,880
870,964
3,466,300

l1_rollup_execution_gas
316,071
989,412
3,686,678

l2_block_processing_time_in_ms
1,167
4,384
17,382

note_successful_decrypting_time_in_ms
320 (-2%)
1,004 (+1%)
3,762

note_trial_decrypting_time_in_ms
31.6 (-15%)
37.5 (-70%)
149 (-4%)

l2_block_building_time_in_ms
17,928 (-1%)
70,980
285,036

l2_block_rollup_simulation_time_in_ms
13,183 (-1%)
52,155
210,039

l2_block_public_tx_process_time_in_ms
4,716
18,745
74,730

L2 chain processing
Each column represents the number of blocks on the L2 chain where each block has 16 txs.

Metric
5 blocks
10 blocks

node_history_sync_time_in_ms
13,756
26,317 (-1%)

note_history_successful_decrypting_time_in_ms
2,346
4,633 (-1%)

note_history_trial_decrypting_time_in_ms
89.9 (-5%)
231 (+2%)

node_database_size_in_bytes
17,346,640
33,046,608

pxe_database_size_in_bytes
29,923
59,478

Circuits stats
Stats on running time and I/O sizes collected for every circuit run across all benchmarks.

Circuit
circuit_simulation_time_in_ms
circuit_input_size_in_bytes
circuit_output_size_in_bytes

private-kernel-init
309
44,528
25,825

private-kernel-ordering
196 (+1%)
44,145
16,225

base-rollup
1,419
128,654
933

root-rollup
82.7
4,192
729

private-kernel-inner
439
71,331
25,825

public-kernel-private-input
242
32,287
25,825

public-kernel-non-first-iteration
238
32,329
25,825

merge-rollup
7.86 (-1%)
2,712
933

Tree insertion stats
The duration to insert a fixed batch of leaves into each tree type.

Metric
1 leaves
2 leaves
8 leaves
16 leaves
32 leaves
128 leaves
64 leaves
512 leaves
1024 leaves
2048 leaves
8192 leaves

batch_insert_into_append_only_tree_16_depth_ms
10.3
10.6 (-6%)
12.6 (-1%)
17.0 (+1%)
22.9
65.0 (+1%)
N/A
N/A
N/A
N/A
N/A

batch_insert_into_append_only_tree_16_depth_hash_count
16.9
17.5
23.0
31.6
47.0
143
N/A
N/A
N/A
N/A
N/A

batch_insert_into_append_only_tree_16_depth_hash_ms
0.596
0.594 (-6%)
0.538 (-1%)
0.527 (+1%)
0.479
0.447 (+1%)
N/A
N/A
N/A
N/A
N/A

batch_insert_into_append_only_tree_32_depth_ms
N/A
N/A
N/A
N/A
N/A
74.9 (-1%)
47.7
240
460
906
3,590

batch_insert_into_append_only_tree_32_depth_hash_count
N/A
N/A
N/A
N/A
N/A
159
96.0
543
1,055
2,079
8,223

batch_insert_into_append_only_tree_32_depth_hash_ms
N/A
N/A
N/A
N/A
N/A
0.463 (-1%)
0.489
0.438
0.431
0.430
0.431

batch_insert_into_indexed_tree_20_depth_ms
N/A
N/A
N/A
N/A
N/A
103
57.0
359 (+1%)
686
1,380 (+1%)
5,455

batch_insert_into_indexed_tree_20_depth_hash_count
N/A
N/A
N/A
N/A
N/A
197
104
691
1,363
2,707
10,771

batch_insert_into_indexed_tree_20_depth_hash_ms
N/A
N/A
N/A
N/A
N/A
0.492
0.499 (+1%)
0.485 (+2%)
0.473
0.478
0.473

batch_insert_into_indexed_tree_40_depth_ms
N/A
N/A
N/A
56.6
N/A
N/A
N/A
N/A
N/A
N/A
N/A

batch_insert_into_indexed_tree_40_depth_hash_count
N/A
N/A
N/A
94.1
N/A
N/A
N/A
N/A
N/A
N/A
N/A

batch_insert_into_indexed_tree_40_depth_hash_ms
N/A
N/A
N/A
0.579
N/A
N/A
N/A
N/A
N/A
N/A
N/A

Miscellaneous
Transaction sizes based on how many contracts are deployed in the tx.

Metric
0 deployed contracts
1 deployed contracts

tx_size_in_bytes
16,859
40,311

Transaction processing duration by data writes.

Metric
0 new commitments
1 new commitments

tx_pxe_processing_time_ms
554
1,313

Metric
0 public data writes
1 public data writes

tx_sequencer_processing_time_ms
0.513 (-1%)
579


Clicking pocket containers while viewing it

Someone reported this, but I still need to check it:
clicking on an pocket container when its inventory is open
will close it and delete the containers inventory
on PER_LOCATION and PER_LOCATION_PER_PLAYER

I reproduced it


[Android] Incorrect border coloration with sufficiently high border radius

Description
It seems Android has trouble displaying the correct border colors when borderRadius exceeds a certain value.
The use case here is having an easily reusable style or component which can be circular regardless of the size; most easily achieved by setting a high border radius. Given that this works as intended on iOS and web, I assume it is a supported use case. If border radius is not intended to exceed a certain calculable value it should be noted in the docs somewhere but I was unable to find any relevant information. Alternatively, It might make sense to clamp the value in the RN core code but I'll leave that to the maintainers to decide.
Based on my testing this issue only impacts Android as you can see in the screenshots from the snack I've linked below.
Web:

iOS:

Android:

I've seen some other recent issues around border radius on Android but none of them seem to be directly related to this particular behavior based on my understanding so apologies if this is a duplicate, obviously feel free to close/merge in that case.
React Native Version
0.72.7
Output of npx react-native info
System:
OS: macOS 13.6
CPU: (10) arm64 Apple M1 Max
Memory: 2.82 GB / 64.00 GB
Shell:
version: "5.9"
path: /bin/zsh
Binaries:
Node:
version: 18.16.0
path: ~/.nvm/versions/node/v18.16.0/bin/node
Yarn:
version: 1.22.19
path: ~/.nvm/versions/node/v18.16.0/bin/yarn
npm:
version: 9.5.1
path: ~/.nvm/versions/node/v18.16.0/bin/npm
Watchman:
version: 2<IP_ADDRESS>
path: /opt/homebrew/bin/watchman
Managers:
CocoaPods:
version: 1.14.2
path: /opt/homebrew/opt/ruby/bin/pod
SDKs:
iOS SDK:
Platforms:
- DriverKit 23.0
- iOS 17.0
- macOS 14.0
- tvOS 17.0
- watchOS 10.0
Android SDK: Not Found
IDEs:
Android Studio: 2022.2 AI-222.4459.24.2221.9971841
Xcode:
version: 15.0.1/15A507
path: /usr/bin/xcodebuild
Languages:
Java:
version: 17.0.2
path: /usr/bin/javac
Ruby:
version: 3.2.2
path: /opt/homebrew/opt/ruby/bin/ruby
npmPackages:
"@react-native-community/cli": Not Found
react:
installed: 18.2.0
wanted: 18.2.0
react-native:
installed: 0.72.7
wanted: 0.72.7
react-native-macos: Not Found
npmGlobalPackages:
"react-native": Not Found
Android:
hermesEnabled: true
newArchEnabled: false
iOS:
hermesEnabled: true
newArchEnabled: false
Steps to reproduce

Create a <View />.
Give the view different colors for each of borderTopColor, borderBottomColor, borderRightColor, and borderLeftColor
Set the view's height and width to any number, n
Set the view's borderRadius to any number greater than n/2
Observe the coloration of the border. Optionally, you can change the borderRadius value to see how the colors change with different values.

Snack, screenshot, or link to a repository
https://snack.expo.dev/gl3ZjJQi1

⚠	Newer Version of React Native is Available!
ℹ	You are on a supported minor version, but it looks like there's a newer patch available - 0.72.7. Please upgrade to the highest patch for your minor or latest and verify if the issue persists (alternatively, create a new project and repro the issue in it). If it does not repro, please let us know so we can close out this issue. This helps us ensure we are looking at issues that still exist in the most recent releases.

Verified that this impacts 0.72.7 as well.


Fixed #25457 -- Join createsuperuser validation errors with newline instead of comma

Solution suggested by timgraham.
Sample error message after applying this patch:
$ ./manage.py createsuperuser
Username (leave blank to use 'uranusjr'):
Email address:
Password: admin
Password (again): admin
This password is too short. It must contain at least 8 characters.
This password is too common.
Password: ^C
Operation cancelled.

changepassword is also affected if you could update that as well, thanks.

merged in 7372cdebed19a9d8e5527823c1e6825156babf98, thanks!


Filter new endpoints, fixes #347

@krodyrobi This one needs a rebase and also the tests are failing.

Ok tests locally are passing but not on remote. Any ideas?

Nvm there was a test in another suite with side-effects. Still weird though that locally drone does not run tests in the same order as the remote one :D.


Add code highlight

Title
Add code highlight
Summary
previous doc have no code highlight.
added it.
Details
Suggested Reviewers

This seems like a small (but important) contribution, so no Contribution License Agreement is required at this point. We will now review your pull request.
Thanks,
.NET Foundation Pull Request Bot

This seems like a small (but important) contribution, so no Contribution License Agreement is required at this point. We will now review your pull request.
Thanks,
.NET Foundation Pull Request Bot


Bugs in 1.11.2

I have a couple bugs I noticed
The toggles key doesn't seem to work as I'm trying to set the sprint and sneak keys, but nothing happens. Also I can't use the menu to spawn animals, right click on things, or use items: i.e. bows, swords, ,fishing rod, etc. (If that's not in, could you add that function in?)

I'm having some troubles too. Most part of the mod doesn't seems to work. And when I try to use it with another mod, for example, opening the Lexica Botania book of Botania, it simply crash.

I haven't tried using it with mods but it doesn't seem to work

Hey @GirafiStudios, this issue still persists! Is it ever going to be fixed in 1.11.2?


On iOS (possibly also Android) the response result is not mapped into the response body

Therefore we never see Declined or any of the Judopay payment response.
This was on preauth and is also visible in the sample app.

Thanks @scotthooker for pointing that out.
We need to update the RN mapping of our native response. Will release a fix for this asap.

Thanks @scotthooker for pointing that out.
We need to update the RN mapping of our native response. Will release a fix for this asap.


docs: local replace package dependency

Description:
local replace package dependency docs
Is this change user facing?
YES
References (if applicable):
This is part of the Forked cleanup package project

@leeederek I had to create the PR again because the previous one was accidentally closed


Publish the Linux version to Flathub

Is your feature request related to a problem? Please describe.
Having to use .tar.gz archives for running apps on immutable distros and non-Debian distros results in a subpar user experience. There are no app drawer entries and you have to manually navigate to your download every time you want to launch the app.
Describe the solution you'd like
If the app gets published to Flathub, it can be installed with just one click click on any distro, even immutable ones. The app drawer entries will automatically be added too. The app also has a chance to end up on the front page of GNOME Software, KDE Discover, Linux Mint's Software Manager, and other app stores in many cases.
Describe alternatives you've considered
Manually running the app inside the archive, but it does not result in a good user experience.
Additional context
Here are the Flatpak and Flathub docs.

Bifrost uses Conveyor to build apps, which doesn't currently support creating Flatpaks. It's on their radar, but not a priority.
It seems like it might be possible to create a Flatpak from a compiled binary, but the documentation really isn't clear to me on how to actually do this, and no one seems to have simple explanations on how Flatpak manifests work.


Ability to turn off no-cache

Taking Bruno's suggestion to raise tickets for enhancement requests :)
Would it be possible to have a flag that turns on the Docker build cache during the execution of buildDockerImages.sh ?
At the moment it is hardcoded as no-cache=true. This means that subsequent builds will be slower than they need to be (e.g. 10 minutes instead of 10 seconds). The cache feature in docker is usually turned on by default but the buildDockerImages.sh is overriding it for some reason. Not sure why you'd want to do this...
Alternatively to allowing the cache to be turned on, if you could simply have the cache turned on by default (i.e. remove the no-cache=true line) that would be great.
Thanks in advance!

Also should mention.... without having cache enabled, every CI server build will be new image (i.e. several Gigabytes in size) even if it doesn't need to be. This has the potential to fill up space quickly and even take down the CI server in a worst case scenario.

:+1 from me. I didn't realise this was even set.

The build shell script is a utility. Expert users are welcome to use native "docker build" call

Can you explain your reasoning behind using this option in the script? I don't understand why it's the default.

On 1 Feb 2016, at 7:03 PM, Bruno Borges<EMAIL_ADDRESS>wrote:
The build shell script is a utility. Expert users are welcome to use native "docker build" call
—
Reply to this email directly or view it on GitHub.

Initially because there were bugs with the Docker layer cache system.
But then to ensure the image will be always clean. Docker caches layers based on what the command looks like, not what it does internally. For example:
    RUN wlst.sh createDomain.py

The command above may generate files to represent a unique domain (or in the case of base image, a WLS installation). If cached, doesn't matter if the output is different every time the command is called, Docker will use the cached layer.
So yeah, I advocate for not using cache by default, unless you really know what you are doing.

Hi Bruno,
Will an Oracle customer still be supported / certified for WebLogic on Docker if they don't use the utility script which is documented in the Oracle whitepaper?
Thanks,
Craig

Absolutely

Phew wipes sweat from face

Based on the previous comment, I’m happy to go down the no utility script path although I would have liked to have stuck to that as it is what is in the Oracle Whitepaper and it could be perceived as deviating from the best wishes of Oracle even if that is not reality.
Regarding your comments on caching, I just wanted to clarify some things...
Firstly, this ticket was raised in the context of this file -> https://github.com/oracle/docker-images/blob/master/OracleWebLogic/dockerfiles/buildDockerImage.sh
Here we are dealing with an installation of binaries. The cache is always enabled by default in Docker for good reason. I think if it wasn't stable or was a feature used by "experts" it wouldn't be the default behaviour.
The problem with having no-cache=true set is that the execution will not be idempotent. For example, if you run the buildDockerImage.sh job for infrastructure once it will result in 3.752 GB of space used. If you run it again, it will result in another 3.752 GB of space used even though nothing has changed and there is no need to rebuild a reusable Docker image that already exists....
If we take a simple example, when you run the following it will do the build for the first time which could take several minutes
docker build .
 This helps the developer or operations person by not using up disk space unnecessarily with redundant images when it can safely and reliably reuse the ones that already exist.
Secondly, your example of createDomain.py relates to the sample Dockerfiles that create the domain which is not what I raised the ticket about. However, since we are on the topic I just wanted to mention some things in relation to your point.
In your samples, the createDomain.py gets passed in via the Docker build context when you execute this:
COPY container-scripts/* /u01/oracle/
What that means is that if you ever change the createDomain.py or any other files it will invalidate the build cache for all subsequent steps after they are copied into the container. Docker always does this even when you don’t have no-cache=true set. Docker is smart enough to know the input has changed and make sure that it turns caching off from that point. This is not an advanced feature this is just the way Docker works to save developers or operations people time and effort. If you have no-cache=true set as a blanket rule the developer and/or operations person may end up needing to run clean up scripts[1] often to get rid of all your redundant images that are serving no value.
[1] Like this one -> https://github.com/oracle/docker-images/blob/master/OracleWebLogic/samples/clean-up-docker.sh

Hi @craigbarrau,
About your concern on user space, the utility script combines --force-rm=true with --no-cache=true, so no layer will be stored after the build.
About your concern with RUN wlst.sh createDomain.py example, indeed that happens only on samples. But it is important to clarify that even though createDomain.py does not change at all, the output of that call generates random hashes internally, as part of the domain metadata.
To be safe, cache should be disabled for the base image (WLS + JDK installation), and must be disabled for building domains.

Thanks Bruno,
If I understand correctly, the cache "must be disabled for building domains" if you are building derivative images from the same WebLogic domain and want to ensure each domain has different password entropy. That's very important, I agree.
As for the WLS + JDK installation, to be honest I'm not really convinced but I'm happy to agree to disagree on this one because the drawbacks of the cache disablement can be easily and safely worked around. Thanks again for your prompt responses Bruno! :)

No worries. :-)


As a user, I want to see an online reference documentation of the library

Checked for duplicates
No - I haven't checked
🧑🔬 User Persona(s)
No response
💪 Motivation
...so that I can [why do you want to do this?]
📖 Additional Details
No response
Acceptance Criteria
Given
When I perform
Then I expect
⚙ Engineering Details
No response
🎉 I&T
No response

@collinss-jpl started that this morning. This is now ready for a PR shortly.


Abstracted methods?

Are there not abstracted methods for media like play(), pause() mute() seek() etc?

I think it is possible using ref to video, or?

The idea of the component is that methods are inferred from the props given to the player, to match React's declarative style. For example, changing the playing prop from true to false will cause pause() to be called internally. Setting the muted prop to true will mute the player. This logic can be seen in the Player component:
https://github.com/CookPete/react-player/blob/1f158d1b8bb353553ea5680ed9041db26055da53/src/Player.js#L28-L51
For seeking there is a seekTo method that is outlined in the readme.
If you really want to call the methods yourself, you could use ref and call getInternalPlayer(), which will let you call any method you like on the player instance, like so:
class App extends React.Component {
  ref = player => {
    this.player = player
  }
  render() {
    return (
      <div>
        <ReactPlayer
          ref={this.ref}
          url='http://clips.vorwaerts-gmbh.de/big_buck_bunny.mp4'
          playing
          controls
        />
        <button onClick={() => this.player.getInternalPlayer().play()}>Play</button>
        <button onClick={() => this.player.getInternalPlayer().pause()}>Pause</button>
      </div>
    );
  }
}

https://jsfiddle.net/p56tp1ex/1/
Generally I would advise against doing this, as you will become out of sync with any props passed to the player, and you will be more prone to bugs and getting mixed up with your player state.


ISPN-4772 Increase the lock acquisition timeout in core tests

https://issues.jboss.org/browse/ISPN-4772
Even when the test expects a failure, it usually also has some
operations that must succeed, so using a timeout of only 100ms
can lead to random failures.

Allrighy.. let's give this a try on TC :D

Integrated, thanks!


How do we deal with multiple versions of the same OKH in different languages

We have two OKH that represent the same version of a Serge Mask as defined by
https://makermask.org/masks/surge/ that are in two different languages Spanish and English Currently these are two distinct things and it would seem they need to be connected
Ref
https://github.com/helpfulengineering/library/blob/main/alpha/okh/okh-manifest-surge-english.yml
https://github.com/helpfulengineering/library/blob/main/alpha/okh/okh-manifest-surge-spanish.yml

Perhaps we could deal with this by having the underlying representation of the OKH be pointers to the parts and equipment, and abstract language out of it.
Then, both the Spanish and English, and all other languages as we bring them online, would have common words to point to the pointer.
For example:
Q25294 points to the object used to put in nails
"hammer" --> Q25294
"martillo" --> Q25294
Additionally, we could try using language models to encode the words as vectors as in word2vec


Added (very basic) functionality to handle special workspaces

Hi :)
Currently, windows residing in a special workspace will be treated by nwg-dock just like any other window, triggering focuswindow. This does not work of course.
What I did at the moment is simply to actually dispatch togglespecialworkspace. Now when we click on a client icon residing in a special workspace, the client will be moved in and out of the special workspace (it won't be focused), together with whatever else might be in the special workspace. This works well, I guess, for users that will only put one and only one client in a special workspace.
That said, I know that this is probably not ready for merging, but I'm still opening this pull request for discussion.

Ah damn, I totally forgot about the unnamed special workspace, I always give names :smiling_face_with_tear:
It should work now. We cannot use your code because there are also named special workspaces (for instance special:something).
Basically, if we want to toggle the special workspaces there are two syntaxes:

togglespecialworkspace for unnamed special workspaces (the ones you create by simply using movingtoworkspace special);
togglespecialworkspace something for named special workspaces (the ones you create by using movingotworkspace special:something)

Very well. When your PR is finished, it'll have to wait until I'm back home. Just started my holiday over the lake, w/ only an old laptop and no coding tools.

No worries, enjoy your vacation! The PR should be ok now, I'll mark it as ready for review

Since there are some other fixes waiting to be released, I'll merge your PR and add the changes I requested myself.

Thank you very much! I've been busy with moving these days and I totally missed your message


Support extraVolumes for Cribl workergroups helm charts

I would like to enable extraVolumes support in helm values similar to what we have for extraVolumeMounts. Any reason for not enabling it ?
I have a use case where I need to add extraVolumes to enable WLI in GKE. Let me know if it's good to work, I ll enable them. Thanks!

@Kavinraja-G by defining extraVolumeMounts in your values, you will also add the volume definition to the pod.
See: https://github.com/criblio/helm-charts/blob/master/helm-chart-sources/logstream-workergroup/templates/_pod.tpl#L120-L131
If this doesn't fix the issue, please let me know.

Nah, that doesn't help in my use-case. Check out Sample App in Anthos using WLI example it has custom volumes & volumeMounts that I need to configure.

Looks like you need extraSecretMounts... This should accomplish your use case based on the linked document:
env:
  AWS_ROLE_ARN: goat
  AWS_WEB_IDENTITY_TOKEN_FILE: /var/run/secrets/aws-iam-token/serviceaccount/token
  AWS_REGION: us-west-2
extraSecretMounts:
  - name: aws-iam-token
    mountPath: /var/run/secrets/aws-iam-token/serviceaccount
    readOnly: true
    projected:
      defaultMode: 420
      sources:
        - serviceAccountToken:
            audience: sts.amazonaws.com
            expirationSeconds: 86400
            path: token

This outputs the correct volumeMount and volume settings on the Pod definition.

@Kavinraja-G can you confirm this works for you?

Hey @Kavinraja-G, any luck with testing last week?


📖 Docs: using icons from public URLs

The icon shortcode docs on icons provide info on how to have your own icon file located in assets/icons.
What about if the icon is available at a link, can we document how to reference an icon by URL only?  Also, are non-SVG extensions accepted?
Here is an example URL: https://tukuz.com/wp-content/uploads/2019/05/university-of-pittsburgh-logo-vector.png

Also, to help noobs, it could be good to add a sentence explaining what an icon partial is to the icon partial docs.

Hello @jamesbraza - closing this one. Hugo provides the freedom for users to enhance their own deploys with custom icons. The theme offers a set of icons ready to use to make things easier. I doubt asking people to configure their icons via URL is a better/easier experience.

Okay gotchu.  The universities I attended or employers I work(ed) at, their icons are not in the blowfish supported list.
I didn't want to copy their icons to my website, I would prefer to just link their public URL.  However, I agree it's simpler for blowfish to support just one workflow, local files.  Thanks!

@jamesbraza depending on what you are trying to achieve there are ways to import directly a remote image - just not an icon. Also it is possible to add custom icons. Create a discussion thread if you have further questions. Happy to help you

Alright, I made https://github.com/nunocoracao/blowfish/discussions/888, thank you any help!


Add column 2 org profiles

🔗 Issue
#285
✍ Description
Added an about_us column on the organization_profiles table. Updated factories and seeds to reflect the migration. Added validation for about us on organization_profile model, and added validation test.
📷 Screenshots/Demos

https://github.com/all-contributors
please add @yuricarvalhop for code.
please add @edwinthinks for review.

https://github.com/all-contributors
please add @jadekstewart3 for code.
please add @edwinthinks  for review.

@all_contributors
please add @jadekstewart3 for code.
please add @edwinthinks for review.

@all-contributors
please add @jadekstewart3 for code.
please add @edwinthinks for review.


duplicate declaration of version ${project.version}

Running mvn clean install on latest of master:
[WARNING] Some problems were encountered while building the effective model for org.pac4j:pac4j-core:jar:1.8.0-SNAPSHOT
[WARNING] 'dependencyManagement.dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.pac4j:pac4j-http:jar -> duplicate declaration of version ${project.version} @ org.pac4j:pac4j:1.8.0-SNAPSHOT, /Users/Misagh/Workspace/GitWorkspace/pac4j/pom.xml, line 187, column 16
[WARNING]
[WARNING] Some problems were encountered while building the effective model for org.pac4j:pac4j-oauth:jar:1.8.0-SNAPSHOT
[WARNING] 'dependencyManagement.dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.pac4j:pac4j-http:jar -> duplicate declaration of version ${project.version} @ org.pac4j:pac4j:1.8.0-SNAPSHOT, /Users/Misagh/Workspace/GitWorkspace/pac4j/pom.xml, line 187, column 16
[WARNING]
[WARNING] Some problems were encountered while building the effective model for org.pac4j:pac4j-cas:jar:1.8.0-SNAPSHOT
[WARNING] 'dependencyManagement.dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.pac4j:pac4j-http:jar -> duplicate declaration of version ${project.version} @ org.pac4j:pac4j:1.8.0-SNAPSHOT, /Users/Misagh/Workspace/GitWorkspace/pac4j/pom.xml, line 187, column 16
[WARNING]
[WARNING] Some problems were encountered while building the effective model for org.pac4j:pac4j-openid:jar:1.8.0-SNAPSHOT
[WARNING] 'dependencyManagement.dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.pac4j:pac4j-http:jar -> duplicate declaration of version ${project.version} @ org.pac4j:pac4j:1.8.0-SNAPSHOT, /Users/Misagh/Workspace/GitWorkspace/pac4j/pom.xml, line 187, column 16
[WARNING]
[WARNING] Some problems were encountered while building the effective model for org.pac4j:pac4j-test-cas:war:1.8.0-SNAPSHOT
[WARNING] 'dependencyManagement.dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.pac4j:pac4j-http:jar -> duplicate declaration of version ${project.version} @ org.pac4j:pac4j:1.8.0-SNAPSHOT, /Users/Misagh/Workspace/GitWorkspace/pac4j/pom.xml, line 187, column 16
[WARNING]
[WARNING] Some problems were encountered while building the effective model for org.pac4j:pac4j-http:jar:1.8.0-SNAPSHOT
[WARNING] 'dependencyManagement.dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.pac4j:pac4j-http:jar -> duplicate declaration of version ${project.version} @ org.pac4j:pac4j:1.8.0-SNAPSHOT, /Users/Misagh/Workspace/GitWorkspace/pac4j/pom.xml, line 187, column 16
[WARNING]
[WARNING] Some problems were encountered while building the effective model for org.pac4j:pac4j-saml:jar:1.8.0-SNAPSHOT
[WARNING] 'dependencyManagement.dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.pac4j:pac4j-http:jar -> duplicate declaration of version ${project.version} @ org.pac4j:pac4j:1.8.0-SNAPSHOT, /Users/Misagh/Workspace/GitWorkspace/pac4j/pom.xml, line 187, column 16
[WARNING]
[WARNING] Some problems were encountered while building the effective model for org.pac4j:pac4j-gae:jar:1.8.0-SNAPSHOT
[WARNING] 'dependencyManagement.dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.pac4j:pac4j-http:jar -> duplicate declaration of version ${project.version} @ org.pac4j:pac4j:1.8.0-SNAPSHOT, /Users/Misagh/Workspace/GitWorkspace/pac4j/pom.xml, line 187, column 16
[WARNING]
[WARNING] Some problems were encountered while building the effective model for org.pac4j:pac4j-oidc:jar:1.8.0-SNAPSHOT
[WARNING] 'dependencyManagement.dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.pac4j:pac4j-http:jar -> duplicate declaration of version ${project.version} @ org.pac4j:pac4j:1.8.0-SNAPSHOT, /Users/Misagh/Workspace/GitWorkspace/pac4j/pom.xml, line 187, column 16
[WARNING]
[WARNING] Some problems were encountered while building the effective model for org.pac4j:pac4j-jwt:jar:1.8.0-SNAPSHOT
[WARNING] 'dependencyManagement.dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.pac4j:pac4j-http:jar -> duplicate declaration of version ${project.version} @ org.pac4j:pac4j:1.8.0-SNAPSHOT, /Users/Misagh/Workspace/GitWorkspace/pac4j/pom.xml, line 187, column 16
[WARNING]
[WARNING] Some problems were encountered while building the effective model for org.pac4j:pac4j-ldap:jar:1.8.0-SNAPSHOT
[WARNING] 'dependencyManagement.dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.pac4j:pac4j-http:jar -> duplicate declaration of version ${project.version} @ org.pac4j:pac4j:1.8.0-SNAPSHOT, /Users/Misagh/Workspace/GitWorkspace/pac4j/pom.xml, line 187, column 16
[WARNING]
[WARNING] Some problems were encountered while building the effective model for org.pac4j:pac4j:pom:1.8.0-SNAPSHOT
[WARNING] 'dependencyManagement.dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.pac4j:pac4j-http:jar -> duplicate declaration of version ${project.version} @ line 187, column 16
[WARNING]
[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
[WARNING]

Yes, I saw that yesterday. I planned to address it along another PR.

PR: https://github.com/pac4j/pac4j/pull/205


[BUG] ODFE References and out-of-date links

Describe the bug
There are still a couple of references to OpenDistro across the experience.

"Read about Get started with Anomaly detection" links to https://opendistro.github.io/for-elasticsearch-docs/docs/ad/ rather than https://docs-beta.opensearch.org/docs/ad/.
Sample detector "Monitor HTTP responses" includes ODFE language (ex. opendistro-sample-http-responses-detector & opendistro-sample-http-responses).
Sample detector "Monitor eCommerce orders" includes ODFE language (ex. opendistro-sample-ecommerce-detector & opendistro-sample-ecommerce).
Sample detector "Monitor host health" includes ODFE language (ex. opendistro-sample-host-health-detector & opendistro-sample-host-health).
"Looking to get more familiar with anomaly detection? Read the documentation or create a sample detector to get started." points to https://opendistro.github.io/for-elasticsearch-docs/docs/ad/ rather than https://docs-beta.opensearch.org/docs/ad/.

Other plugins installed
N/A
To Reproduce
Steps to reproduce the behavior:

Go to 'Anomaly detection'
Click on 'Sample detectors' or 'Create detector'
See error

Expected behavior
Links to correct documentation and no references to ODFE.
Screenshots
If applicable, add screenshots to help explain your problem.

Fixed in https://github.com/opensearch-project/anomaly-detection-dashboards-plugin/pull/24


Fix for chart tooltip flickering + Report for broken fallback method

Hi,
I don't really know if it's the most appropriate place to post this but since it's the project repo, I think it still worth posting this here.
So this issue is more to provide a fix to the chart tooltip' flickering issue that's kinda annoying.
After studying a bit how it's displayed, I created a small piece of CSS that seems to fix that issue for good.
As those charts are not displayed on IE, I think trying to make a fix for those browsers is pointless so I'll only provide the fix for the "other" browsers (even if this fix should also work on IE11 only (as IE10 and earlier doesn't support the pointer-events rule)).
.google-visualization-tooltip,
svg g path[stroke="#636363"],
svg g path[stroke="#636363"] + circle {
    pointer-events: none;
}

Here's a GIF showing the fix in action:

About the CSS code itself, I had to compose with the limitations of the lack of distinct names in the rendered SVG itself, but it should not disrupt anything in the page.
--
(Also this can be the occasion to point out the fact the image-based fallback rendering is broken and displays nothing, a look inside the "image" reports a 404 error)

Thanks heaps for looking into this! This is a really well-written report, and the gif was a good idea 😊
Google finally discontinued the old server-rendered charts API (which they announced would be shut down in 2019, then mysteriously it didn't go away for another 4 years, lol). I put off fixing it since I figured I'd have to find a different charts service or a library, and libraries I tried in the past kinda didn't look amazing. I did find quickchart.io, which looks great, so I just switched the prerendered charts to that. They should be live now.
I might switch the JS charts to quickchart too, because it just looks better overall in terms of being maintained, which I really don't trust Google with these days. I like the 3D pie and the legend with connecting lines + speech-bubble tooltips, but, well, it's Google, and this bug has existed in Google Charts at least since we started using it.


Advice: how to size-to-fit

Most reveal objects support a "size to fit" / "stretch", e.g. <img> tags.
I'd love to get my d3 iframe resized to fit the available space. Is this something that can be achieved? Directly embedding the <svg> instead of via an <iframe> seems to afford this ability, but of course that's not possible using this plugin.
Any advice? Thanks!

By default the embedded iframe takes the full with of its container. But if your d3 visualization is not reactive to window's size changes, then the d3 visualization inside the iframe, whatever the iframe size will be, will always have the same size (so if you actually resize your browser page too much you'll crop the d3 visualization inside).
You can achieve a "fit to available space" effect by making your d3 visualization fit the window's width (which will be the embedded iframe) and making it reactive to a resize.
See this block for an example of pattern that can lead to this effect (this is the raw view so you can resize the window, see the code at this page, in particular the window.addEventListener('resize', render); bit that redraw the chart when the window is resized).

To add a bit more to my previous comment, in the demo, the globe is reactive (opens this slide in a browser and resize the window), while the second slide (this slide) is not and the d3 visualization won't move and will be cropped when your resize.

Thanks for the info; I'll try making my d3 reactive! You see from my other bug though that I'm having other troubles setting the size :)

Thanks: that worked perfectly. Feel free to close this.


Chat control should support images

Description
The chat control of #1092 is really cool! It would be a good addition to support images in the messages of the chat. This would be especially useful since now LLMs start supporting image vision and image generation. I could create cool demos around this chat control.
Florian:
People are also using chat demos to visualize tables inside their chat. This could also be a nice addition.
Acceptance Criteria

[ ] Ensure new code is unit tested, and check code coverage is at least 90%.
[ ] Create related issue in taipy-doc for documentation and Release Notes.
[ ] Check if a new demo could be provided based on this, or if legacy demos could be benefit from it.
[ ] Ensure any change is well documented.

Code of Conduct

[X] I have checked the existing issues.
[ ] I am willing to work on this issue (optional)

Hello there I think I could work on this issue!

Great, I will assign it to you!

hello again! can you specify where the chat code exists?

hello again! can you specify where the chat code exists?

is it taipy\frontend\taipy-gui\src\components\Taipy\chat.tsx ?

Hey @FlorianJacta, could you please assign this issue to me, if no one is working on this?

@ioannastantzou is normally working on it. Could you confirm @ioannastantzou ?

@ioannastantzou is normally working on it. Could you confirm @ioannastantzou ?

It's ok if someone is working on this, but if not I'm in 👍

Hey @FlorianJacta @AlexandreSajus , can this issue be assigned to me?
Also, I would like to ask that in here, do we've to add a files icon on the left side of the message box (similar to what is present in chatgpt)?

What do you mean by a file icon?
Can this be assigned? @FredLL-Avaiga @FabienLelaquais

We could propose an upload button near the input area that opens a file upload window. The chat control should be able to display file images. (A download button close to the image can be considered)

Hi @jrobinAV, can you please assign me this issue ?

hey i can work on this please can u assign thi sto me

@yaten2302, I have assigned you this issue!
@sOnU1002 I have already assigned you to another issue. @mehulmathur16, there are other issues to take!

@FlorianJacta , thanks for assigning this issue 👍
I'll create a PR for this by Tuesday or Wednesday max!

Apologies for the huge delay on this issue. Actually, I was having a doubt that in this file - frontend/taipy-gui/src/components/Taipy/Chat.tsx, it's showing 2 problems
1.

I haven't changed anything in the codebase, is this issue arising because something is not installed? Like in the 1st one, it's showing that - not able to find module Grid2?
@FredLL-Avaiga @AlexandreSajus

I think your installed version of Mui is not up-to-date.
My advice would be to remove the node_modules directory and the package-lock.json file and reinstall everything from scratch

@FredLL-Avaiga , I re cloned the taipy repo and tried to install the dependencies, but now it's showing this error :/

You need to build taipy-gui before taipy
And then install taipy-gui in taipy with npm i in taipy

Thanks @FredLL-Avaiga , now it's working👍
Also, I wanted to ask about this comment, like for the chat to support, we need an icon to show the user to attach an image. Should I add an icon in the chat itself (as in chatGPT)?

Also, I wanted to ask that in this PR https://github.com/Avaiga/taipy/pull/1092, you took a screenshot of the chat element, could you please guide that if I want to test the element locally, how can I view them? (as you've shown in this PR)

Also, I wanted to ask that in this PR https://github.com/Avaiga/taipy/pull/1092, you took a screenshot of the chat element, could you please guide that if I want to test the element locally, how can I view them? (as you've shown in this PR)

You create a simple python script and run it locally
You can find simple examples in doc/examples/gui

Thanks @FredLL-Avaiga , now it's working👍
Also, I wanted to ask about this comment, like for the chat to support, we need an icon to show the user to attach an image. Should I add an icon in the chat itself (as in chatGPT)?

I'm not sure to understand your question.
If it is about adding an icon button to allow the user to choose an image then yes,I think it should go in the input field besides the send icon.
But we also want to support drag and drop and copy paste...

@FredLL-Avaiga , yes, I was asking about this only. I got it now 👍 I'll add an icon in the txt bar for adding images.
First, I'll create a draft PR for this and then if it looks good, then I'll also add support for drag and drop and copy paste.

@FredLL-Avaiga , I've a doubt, that when I'm running this command - py -m pip install taipy-location, it's showing this error:
ERROR: Could not install packages due to an OSError: [WinError 2] The system cannot find the file specified: 'C:\\Python312\\Scripts\\cftp.exe' -> 'C:\\Python312\\Scripts\\cftp.exe.deleteme

I tried searching about it on stackoverflow, found this solution but, still it's not wokring :/ - py -m pip install --user taipy-location.
Actually, my application is still using taipy main lib even if I've changed certain things in the taipy elements(locally).
Any suggestions regarding how can I fix this?

What is taipy-location ?

@FredLL-Avaiga ,the taipy location is: d:/taipy and the taipy-prac(in which the main.py file is present) is also in the d:/

If you don't mind, can I start working on this issue?

@Satoshi-Sh , thanks for the offer, but could you give me some time, currently working on to fix it :)

Hi @yaten2302 , I think several people can work on the same issue. I will wait for a response from the maintainers.

Yes, multiple people can work on the same issue and you can also collaborate together on it

@quest-bot embark

PR #2078


docs: Update troubleshooting section with Vagrant 1.8.7 curl issue

Automated message from Dropbox CLA bot
@christi3k, it looks like you've already signed the Dropbox CLA. Thanks!

Looks great, merged, thanks @christi3k!


Multi GPU NVIDIA AND AMD

is it possible to multi gpu mine with amd and nvidia
.\tokenminer.exe -G -S -t 2
doesnt work
also tried
.\tokenminer.exe -G -S -t 2 --opencl-devices 0 2
if i specify each opencl device they can mine individually my vega, or gtx 1070

No, sorry.  That is not  supported.


pgBackRest: configuration file contains invalid option 'include-path'

After successful pgBackRest install there is a warning about invalid option in /etc/pgbackrest/pgbackrest.conf:
sudo -u postgres pgbackrest --stanza=postgres-cluster --log-level-console=info check P00   WARN: configuration file contains invalid option 'include-path'
According to latest pgbackrest docs (https://pgbackrest.org/configuration.html, 7.24) there is no 'include-path' option - yet there is 'repo1-host-config-include-path' option.
So setting it to repo1-host-config-include-path = /etc/pgbackrest/conf.d makes the warning to disappear.
Could be fixed in: https://github.com/vitabaks/autobase/blob/e93fbe0c667c18a24edeeda3a264638f81df1277/automation/roles/pgbackrest/templates/pgbackrest.server.conf.j2#L7

Thank you for paying attention to this. In fact, this warning can be safely ignored, as it was implemented to support multiple sets of databases on a single dedicated PgBackRest server. I believe the repo1-host-config-include-path option wasn’t available at the time, so I used include-path instead.
Feel free to submit a PR.

Done https://github.com/vitabaks/autobase/pull/829


[SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs

What changes were proposed in this pull request?
This patch changes the tool tip string in Structured Streaming UI graphs to show batch ID (and timestamp as well) instead of only showing timestamp, which was a key for DStream but no longer a key for Structured Streaming.
This patch does some refactoring as there're some spots on confusion between js file for streaming and structured streaming.
Note that this patch doesn't actually change the x axis, as once we change it we should decouple the logic for graphs between streaming and structured streaming. It won't change UX meaningfully as in x axis we only show min and max which we still would like to know about "time" as well as batch ID.
Why are the changes needed?
In Structured Streaming, everything is aligned for "batch ID" where the UI is only showing timestamp - end users have to manually find and correlate batch ID and the timestamp which is clearly a huge pain.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Manually tested. Screenshots:

cc. @tdas @zsxwing @gaborgsomogyi @uncleGen @xuanyuanking

Test build #118886 has started for PR 27687 at commit 69eaa40.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23635/
Test PASSed.

LGTM except for some minor issues.

Test build #118886 has finished for PR 27687 at commit 69eaa40.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118886/
Test PASSed.

Thanks @sarutak for the review! I've addressed review comments. Please take a look again. Thanks!

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23638/
Test PASSed.

Test build #118889 has started for PR 27687 at commit 5c2534a.

LGTM. I'll wait for comment/approvement from other developers.

Test build #118889 has finished for PR 27687 at commit 5c2534a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118889/
Test PASSed.


GithubFileLoader only imports repositories with main branch

Checked other resources

[X] I added a very descriptive title to this issue.
[X] I searched the LangChain documentation with the integrated search.
[X] I used the GitHub search to find a similar question and didn't find it.
[X] I am sure that this is a bug in LangChain rather than my code.
[X] The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code
loader = GithubFileLoader(
repo="hotgluexyz/tap-netsuite",
access_token=gh_token,
github_api_url="https://api.github.com",
file_filter=lambda file_path: file_path.endswith(
".py"
), # retrieves all python files
)
documents = loader.load()
Error Message and Stack Trace (if applicable)

HTTPError                                 Traceback (most recent call last)
 in <cell line: 1>()
----> 1 documents = loader.load()
3 frames
/usr/local/lib/python3.10/dist-packages/requests/models.py in raise_for_status(self)
1019
1020         if http_error_msg:
-> 1021             raise HTTPError(http_error_msg, response=self)
1022
1023     def close(self):
HTTPError: 404 Client Error: Not Found for url: https://api.github.com/repos/hotgluexyz/tap-netsuite/git/trees/main?recursive=1
Description
I am trying to use langchain to load a github repository according to the documentation
System Info
langchain==0.1.14
langchain-community==0.0.31
langchain-core==0.1.38
langchain-openai==0.1.1
langchain-text-splitters==0.0.1
langcodes==3.3.0
langsmith==0.1.38

The branch parameter is set by default to "main" in the GithubFileLoader API. Change it to "master" (or whichever branch of your GitHub repo you want)

Thanks @sepiatone will fix it


Trouble getting ClassNLLCriterion to work

Hey,
First of all thanks for the great work on this, this project has been really helpful.
I have been struggling to get clnn to run StochasticGradient with ClassNLLCriterion.
I have been following this guide https://github.com/soumith/cvpr2015/blob/master/Deep Learning with Torch.ipynb
My first question is that regular nn's ClassNLLCriterion seems fine accepting 1D tensors, while clnn's needs 2D ones. I tried adjusting for this by adding a "net:add(nn.Reshape(10, 1))" as the last step of my neural network. Is this the correct approach?
Also, torch's ClassNLLCriterion accepts integers as the target, while, if I understand correctly, clnn's requires tensors with the correct label set to 1. I've converted the targets to be 1D vectors of 0's with a 1 on the correct label.
With these changes I can get it to run but I'm getting nonsensical error numbers when training (sometimes nan, sometimes impossibly high values).
Here's the full code. It works when not running through OpenCL (it gets a training error of 1.432 in about 60 seconds), so I think I did something wrong, but I can't figure out what.
require('nn')
require('cltorch')
require('clnn')

-- os.execute('wget -c https://s3.amazonaws.com/torch7/data/cifar10torchsmall.zip')
-- os.execute('unzip cifar10torchsmall.zip')

net = nn.Sequential()

net:add(nn.SpatialConvolutionMM(3, 6, 5, 5))
net:add(nn.SpatialMaxPooling(2,2,2,2))
net:add(nn.SpatialConvolutionMM(6, 16, 5, 5))
net:add(nn.SpatialMaxPooling(2,2,2,2))
net:add(nn.View(16*5*5))
net:add(nn.Linear(16*5*5, 120))
net:add(nn.Linear(120, 84))
net:add(nn.Linear(84, 10))
net:add(nn.LogSoftMax())
net:add(nn.Reshape(10, 1))

net = net:cl()

trainset = torch.load('cifar10-train.t7')
trainset.data = trainset.data:double()

testset = torch.load('cifar10-test.t7')
testset.data = testset.data:double()

mean = {} -- store the mean, to normalize the test set in the future
stdv  = {} -- store the standard-deviation for the future
for i=1,3 do -- over each image channel
    mean[i] = trainset.data[{ {}, {i}, {}, {}  }]:mean() -- mean estimation
    print('Channel ' .. i .. ', Mean: ' .. mean[i])
    trainset.data[{ {}, {i}, {}, {}  }]:add(-mean[i]) -- mean subtraction

    stdv[i] = trainset.data[{ {}, {i}, {}, {}  }]:std() -- std estimation
    print('Channel ' .. i .. ', Standard Deviation: ' .. stdv[i])
    trainset.data[{ {}, {i}, {}, {}  }]:div(stdv[i]) -- std scaling
end

for i=1,3 do -- over each image channel
    testset.data[{ {}, {i}, {}, {}  }]:add(-mean[i]) -- mean subtraction
    testset.data[{ {}, {i}, {}, {}  }]:div(stdv[i]) -- std scaling
end

trainset.data = trainset.data:cl()

setmetatable(trainset,
    {__index = function(t, i)
		return {t.data[i], t.label[i]}
	end}
);

function trainset:size()
    return self.data:size(1)
end

local labels = trainset.label

trainset.label = torch.Tensor(trainset.label:size(1), 10)

for i=1,trainset:size() do
	trainset.label[i] = torch.Tensor(10):fill(0)
	trainset.label[i][labels[i]] = 1
end

trainset.label = trainset.label:cl()

print(trainset)
--print(net:forward(trainset.data[4]))

criterion = nn.ClassNLLCriterion()
criterion = criterion:cl()

trainer = nn.StochasticGradient(net, criterion)
trainer.learningRate = 0.001
trainer.maxIteration = 2

trainer:train(trainset)

If the differences between clnn's ClassNLLCriterion and nn's version are unintentional I'd love to help consolidate the interfaces.

Hi PavleMiha,
So basically, normally one would tend to train in mini-batches, for various reasons, but for GPUs, it will be significantly faster.  Otherwise, the program spends all its time starting and stopping kernels, and the GPU doesnt have enough data to do what it does well, which is crunching tens of thousands of numbers in one go.
To create a mini-batch, you add one more dimension, as the first dimension to the tensor.  If you have a mini-batch with a batch-size of 128, then the first dimension of each tensor would be 128.  At the the output of your network, just before the criterion, the tensors would have dimension:
output: 128 x 10
target: 128   (ie a vector with 128 floats in, not just a single float)
Now, if you train with a mini-batch size of 1 ( which is possible, though slow), you would need the following tensor sizes:
output: 1 x 10
target: 1    (ie a vector with a single float in, rather than a single float)
So, the reshape you did was along the right lines, but the dimensions need to be swapped, like this:
net:add(nn.Reshape(1, 10))

For the target, the labels, they should be simply the correct class for each example, eg 5, or 10, but they also need to be reshaped into a mini-batch, in this case a mini-batch with a size of 1.  But the trainset.label is being indexed to give each 'mini-batch', ie:
setmetatable(trainset,
    {__index = function(t, i)
        return {t.data[i], t.label[i]}
    end}
);

So, we need to reshape as follows:
trainset.label = trainset.label:reshape(trainset.label:size(1), 1)

And then this runs ok for me.
Here is the test program, modified as above.  I added a parameter to it, so you can run it with -backend nn or -backend clnn:
require('nn')
require('cltorch')
require('clnn')

local cmd = torch.CmdLine()
cmd:option('-backend', 'nn')
local params = cmd:parse(arg)

local backend = params.backend
if backend ~= 'nn' and backend ~= 'clnn' then
    error('backend should be nn or clnn')
end

-- os.execute('wget -c https://s3.amazonaws.com/torch7/data/cifar10torchsmall.zip')
-- os.execute('unzip cifar10torchsmall.zip')

net = nn.Sequential()

net:add(nn.SpatialConvolutionMM(3, 6, 5, 5))
net:add(nn.SpatialMaxPooling(2,2,2,2))
net:add(nn.SpatialConvolutionMM(6, 16, 5, 5))
net:add(nn.SpatialMaxPooling(2,2,2,2))
net:add(nn.View(16*5*5))
net:add(nn.Linear(16*5*5, 120))
net:add(nn.Linear(120, 84))
net:add(nn.Linear(84, 10))
net:add(nn.LogSoftMax())
net:add(nn.Reshape(1, 10))

if params.backend == 'clnn' then
    net = net:cl()
end

trainset = torch.load('cifar10-train.t7')
trainset.data = trainset.data:double()

testset = torch.load('cifar10-test.t7')
testset.data = testset.data:double()

mean = {} -- store the mean, to normalize the test set in the future
stdv  = {} -- store the standard-deviation for the future
for i=1,3 do -- over each image channel
    mean[i] = trainset.data[{ {}, {i}, {}, {}  }]:mean() -- mean estimation
    print('Channel ' .. i .. ', Mean: ' .. mean[i])
    trainset.data[{ {}, {i}, {}, {}  }]:add(-mean[i]) -- mean subtraction

    stdv[i] = trainset.data[{ {}, {i}, {}, {}  }]:std() -- std estimation
    print('Channel ' .. i .. ', Standard Deviation: ' .. stdv[i])
    trainset.data[{ {}, {i}, {}, {}  }]:div(stdv[i]) -- std scaling
end

for i=1,3 do -- over each image channel
    testset.data[{ {}, {i}, {}, {}  }]:add(-mean[i]) -- mean subtraction
    testset.data[{ {}, {i}, {}, {}  }]:div(stdv[i]) -- std scaling
end

if params.backend == 'clnn' then
    trainset.data = trainset.data:cl()
end

setmetatable(trainset,
    {__index = function(t, i)
        return {t.data[i], t.label[i]}
    end}
);

function trainset:size()
--    return self.data:size(1)
  return 128
end

local labels = trainset.label

--trainset.label = torch.Tensor(1, trainset.label:size(1))
trainset.label = trainset.label:reshape(trainset.label:size(1), 1)

--for i=1,trainset:size() do
--    trainset.label[i] = labels[i]
--end

print('trainset.label:size()', trainset.label:size())
if params.backend == 'clnn' then
    trainset.label = trainset.label:cl()
end

print(trainset)
--print(net:forward(trainset.data[4]))

criterion = nn.ClassNLLCriterion()
if params.backend == 'clnn' then
    criterion = criterion:cl()
end

trainer = nn.StochasticGradient(net, criterion)
trainer.learningRate = 0.001
trainer.maxIteration = 2

trainer:train(trainset)

This runs for me, with both nn and clnn backends. Note that you will generally get faster training times if you can arrange a mini-batch size of around 128, or 256, rather than 1.  For example, in vgg, they use 256 http://www.robots.ox.ac.uk/~vgg/publications/2015/Simonyan15/simonyan15.pdf (third sentence, section 3.1) .  In alexnet he uses 128 http://www.cs.toronto.edu/~fritz/absps/imagenet.pdf ( first sentence of section 5) .

Hey Hugh,
Thank you so much for the help, batching to run on the GPU makes perfect sense.
Your code worked perfectly but I wanted to try to make it actually use batching so I changed some stuff so it accepts a batch size. I couldn't get trainer to accept batches (it might not do that at all) so I did it with optim, based on what I saw at https://github.com/torch/demos/blob/master/train-a-digit-classifier/train-on-mnist.lua
Here's the code, with batching, for reference: https://github.com/PavleMiha/torchdemos/blob/master/ir3.lua
The weird thing is that I'm finding that the nn and clnn variants are running at pretty much exactly the same speed, regardless of batch size, or the size of the convolutional layers.
Is this just a matter of the neural net and my machine (I'm on a 2011 iMac, so it's an AMD Radeon HD 6770M) or did I mess something up?

Hmmm, yes, I get similar results actually.  I think it's because the images are quite small.  I think if you try larger images, then you might see a larger difference.  Note that in imagenet and so on, most of the time goes into the first few layers, eg see the 'layerwise' results in https://github.com/soumith/convnet-benchmarks (cltorch isnt in layerwise, but can see that the earlier layers take a lot more time than the later ones).


Touch UI module update

The current state of Touch UI module prevents more advanced authoring tests.
The main problems are:

the module contains some redundant, project-specific approaches
it is lacking some important functionality like inline editing
some of the existing page objects are not easily extendable, which introduces problems when dealing with extendable platforms like Zen Garden

This epic gathers all tasks related to the Touch UI module update. Required changes are going to break the existing contract and will not be backward-compatible.

Branch containing whole update: 172-touch-ui-module-update

TouchUI module is EOL since 1.6.0 and will be replaced by a new API.


Turn off the verbose(debug) option for disable tests linter workflow

Disable tests linter workflow enables the verbose(debug) outpput, which is used for debug and would be better to turn off for the workflow. So it's easy to notice the error message.

Feel free to assign me

I noticed that @annaibm is working on the fix: https://github.com/annaibm/aqa-tests/commit/df233002f4c9b603344da9ead2c5fa2468f1043b


Transferred CAN frame latency issue

Do you have any issue with unstable sample rate of sent out can frames?

Hi there,
Could you be a little more specific about the issue please? What behavior have you encountered?
The XCU embedded system expects CAN comms at 200Hz (min). It's reasonably tolerant to a small amount of jitter, but will handback to manual mode if the can comms accumulate errors, or if doesn't receive a fresh frame within a small time tolerance.
If you're encountering issues here, try raising the frequency a small amount by increasing ROS_LOOP to 250 or so. You can keep the same control rate by also increasing CONTROL_LOOP to the same ratio, i.e 25.
I hope that helps,
Fionan

Closed due to inaction


Update create.rs

Used terminal to print output in project create command.
fix #6571

Proposed changes
Before :
println!("{:?}", project);
After:
opts.terminal
.stdout()
.plain(project.output()?)
.json(serde_json::to_string_pretty(&project).into_diagnostic()?)
.write_line()?;

Looks good, thanks @Ritish134! Could you ammend your commit message to "feat(rust): flatten json output for identity list command" (with no body)? Once that's ready, we'll merge it 👍

Hi @Ritish134, I'm closing this PR in favor of https://github.com/build-trust/ockam/pull/6584 since it was submitted before yours fixing the same issue. I'll add you as a co-author in that PR and mark this PR as accepted so you can get full credit for your contribution
Thanks 🙏


Firewall Rule Doesn't get applied when including pdnpipe

We are trying to include traffic limiters with the firewall rule through API.
When trying to include the pdnpipe  in the request body, we receive
{"status":"bad request","code":400,"return":4225,"message":"Firewall rule dnpipe and pdnpip queue and a virtual interface cannot be selected","data":[]}
The request body is
{'apply': True,  'descr': 'xxxxxx',  'dnpipe': 'up_pipe',  'dst': 'any',  'interface': 'lan',  'ipprotocol': 'inet',  'pdnpipe': 'down_pipe',  'protocol': 'any',  'src': '<firewall-alias>',  'statetype': 'keep state',  'type': 'pass'}
But when removing the pdnpipe, it worked except that we could not limit the download speed. But when updating the firewall from web interface and selecting the download pipe, it worked. And we also tried retrieving the firewall rule from the API after updating from the web-interface, it showed the same data (with pdnpipe being present). We tried updating the firewall rule with pdnpipe, even then it throws the same error.
PfSense-ver: 2.5.1
pfSense-pkg-API-ver: 1.2_2

Hey!
Thanks for reporting this issue. Could you include the configuration for the up_pipe and down_pipe limiters? That will help me locate any logic error there may be causing this. The API seems to think they are of incompatible types which is why that response is being given.
Thanks again!

The configuration for the up_pipe and down_pipe limters
data = { 	"name": <filter_name (up or down)>, 	"bandwidth": [ 		{ 			"bw": speed, 			"bwscale": "Mb" 		} 	], 	"description": description, 	"apply": true, 	"aqm": "gred", 	"sched": "wf2q+", }

Hi,
Also modifiing same problem, is possible only to put dnpipe value
pdnpipe have error also if called separately
curl -s -H "Content-Type: application/json" -d '{"client-id": "admin", "client-token": "password", "tracker": "1620915610", "pdnpipe": "d014"}' -X PUT http://pfwan.munetwork.it/api/v1/firewall/rule
{"status":"bad request","code":400,"return":4225,"message":"Firewall rule dnpipe and pdnpip queue and a virtual interface cannot be selected","data":[]}
thanks Nicola

Hi,
Also modifiing same problem, is possible only to put dnpipe value
pdnpipe have error also if called separately
curl -s -H "Content-Type: application/json" -d '{"client-id": "admin", "client-token": "password", "tracker": "1620915610", "pdnpipe": "d014"}' -X PUT http://pfwan.munetwork.it/api/v1/firewall/rule
{"status":"bad request","code":400,"return":4225,"message":"Firewall rule dnpipe and pdnpip queue and a virtual interface cannot be selected","data":[]}
thanks Nicola

Why closed the issue ?

Why closed the issue ?

I closed the issue by mistake... thanks for mentioning.


[HelmInstaller] Failed to download Helm from location https://storage.googleapis.com/kubernetes-helm

Question, Bug, or Feature?
Type: Bug
Enter Task Name: HelmInstaller
Environment

Server: Azure Pipelines

Agent - Hosted:

Issue Description
According this new announcement on June 10 2019, the URL to download the Helm Client has changed: https://helm.sh/blog/get-helm-sh/ resulting since June 20 2019 to an error while trying to install/download a specific Helm version:
Failed to download Helm from location https://storage.googleapis.com/kubernetes-helm

PR https://github.com/microsoft/azure-pipelines-tasks/pull/10729 to fix this

Closing this issue as the above PR is merged.


Woke rules are applied to wokeignore comments

wokeignore:rule=master-slave causes the slave rule to fail.  This can be worked around by using wokeignore:rule=master-slave,slave instead, but I don't think that should be necessary.

[x] Yes, I've searched similar issues on GitHub and didn't find any.
[x] Yes, I've included all information below (version, config, etc).

Please include the following information:
Version of woke
$ woke --version
# woke version 0.4.0

Config file
$ cat .woke.yml
# cat: .woke.yml: No such file or directory

Go environment
$ go version && go env
# go: command not found

Verbose output of running
$ woke --debug README.md
2020-11-10T12:54:08-05:00 DBG woke version 0.4.0 built from 038d4dd on 2020-10-29T02:30:44Z
2020-11-10T12:54:08-05:00 DBG tried default config file error="stat .woke.yaml: no such file or directory" cfg=.woke.yaml
2020-11-10T12:54:08-05:00 DBG tried default config file error="stat .woke.yml: no such file or directory" cfg=.woke.yml
2020-11-10T12:54:08-05:00 DBG rules enabled rules=["blackbox","blacklist","dummy","grandfathered","guys","man-hours","master-slave","sanity","slave","whitebox","whitelist"]
2020-11-10T12:54:08-05:00 DBG adding ignorefile file=.gitignore
2020-11-10T12:54:08-05:00 DBG skipping ignorefile error="open .wokeignore: no such file or directory" file=.wokeignore
2020-11-10T12:54:08-05:00 DBG finished compiling ignores durationMS=0.778501
2020-11-10T12:54:08-05:00 DBG created new printer printer=text
2020-11-10T12:54:08-05:00 DBG process files path=README.md type=parallel
2020-11-10T12:54:08-05:00 DBG ignoring via in-line file=README.md line=349 rule=master-slave
2020-11-10T12:54:08-05:00 DBG finished generateFileViolations durationMS=5.644912 file=README.md
README.md:349:114-119: `slave` may be insensitive, use `follower`, `replica`, `standby` instead (error)
master <!-- wokeignore:rule=master-slave -->
                                                                                ^
2020-11-10T12:54:08-05:00 DBG woke completed durationMS=8.229421

Nice find! I agree, you shouldn't need to ignore the rule that is triggered within the wokeignore comment. I have a PR up for a fix for this, which will remove the part of the line that contains wokeignore:rule=rulename before it's processed https://github.com/get-woke/woke/pull/43


Comparison with wasm2c + some ideas

Just performed a quick test to understand how it compares to wasm2c.
For this test I took a ~25 MB wasm (WASI) file, clang.wasm

w2c2 finishes in 5.1s, RAM usage: 193 Mb. Output: 1.3 GiB file (34.8 Mb zipped)
wasm2c finishes in 1m23s, RAM usage: 1432 Mb. Output: 301 MiB file (22 Mb zipped)

Clearly, speed and RAM usage are really good :tada: :rocket:
Wondering if file size can be reduced :scientist:
This also brings another question. C compilers are not very efficient when working with huge C files, so monsters like clang.wasm will take ages to compile. This is mentioned in my wasm2native tool: https://github.com/vshymanskyy/wasm2native#todo
One of the possible solutions is to split output into smaller chunks (which also enables parallel build).

I think I can add w2c2 as an alternative translator for wasm2native.
Now on Features comparison.
w2c2 is not yet handling some of important wasm proposals:

Sign-extension operations
Non-trapping Float-to-int Conversions
Multi-value

These are implemented in wasm2c and covered by opam-1.1.1 spec tests.
Other important proposals include:

Bulk memory operations
Exception handling
Reference types
Multiple memories
These (at the time of writing) are not available in wasm2c. Spec tests for these features are available in main branch.

Coremark 1.0 results
Intel(R) Core(TM) i5-10400 CPU @ 2.90GHz, single-thread
GCC 10.3.0, optimization level: -O3

Native: 33305.578684
wasm2c output: 27458.936027
w2c2 output:  27469.603167

Thank you for this analysis Volodymyr!
These are great points for the README, I will detail better which features are supported and also which are not, and the trade-offs / goals of the project itself.
Performance
I had no idea clang was itself available as a WebAssembly binary, amazing! This is a great  stress-test.
My main goal is porting new software to older machines, which have only tens of MHz and memory, ideally on the host itself (no cross-compiling). This is why I looked into streaming compilation (no IR), but also took a few short-cuts.
For now I have focused on getting the test suite to pass and some small programs to compile and run. So far I have not focused on any performance optimizations, other than employing a streaming-compilation approach. It is great to see that even large programs like clang can be translated and performance is already good.
I think there are multiple opportunities to improve the performance and resource footprint  further, by improving memory allocation, improving speed by parallelizing compilation, and especially for reducing size by removing the unnecessarily generated code:

Always remove parameter names from function prototypes, they are not needed
Behind a "compact" flag:

Remove spaces for indentation
Remove braces for blocks
Remove newlines for statement of a single op (e.g. jump op is if-statement + goto statement)
Remove unnecessary labels. Streaming compilation makes this harder

Great point about compiling large C files: This is one of the reasons I started this project. SwiftWasm generates very large WebAssembly binaries, which take long to compile to C and then very long to compile to a native binary.
I am currently working on parallel compilation, where the compiler is writing function implementations into several smaller files that can be compiled independently, and also performing this code generation concurrently. This will hopefully make compiling the resulting C code more manageable and also speed up the translation process from WebAssembly to C.
Features
For now I have focused on supporting the core specification 1.0 as outlined in https://www.w3.org/TR/wasm-core-1/.
I have not looked into new approved features, like the ones you listed, mostly because I was not sure if they are needed for compiling applications and libraries written in C, C++, Rust, Swift, etc. It seems that most compilers seem to target the "MVP" specfication.
As for the concrete proposals you listed:

Sign-extension operations and non-trapping float-to-int conversions: I think this should be "just" implementing the new opcodes
Multi-value: I had a quick look, it seems like it is going to complicate the compiler quite a bit
Bulk memory operations, exception handling, reference types, multiple memories: These features all seem to add new opcodes, new types, and extend the language quite a lot. They are going to be quite a bit of work to implement

Do you know if there are certain compilers targeting WebAssembly that leverage or even require these feature?
Coremark results
Performance should be very similar to wasm2c, as the generated code pretty much the same.
I have not looked into opportunities to generate more efficient code.
wasm2native integration
This looks like a great project! I was looking for a wasi implementation for wasm2c, awesome work!
Currently the API that w2c2 provides is slightly different than wasm2c, and it produces differently mangled names (mostly they are shorter, e.g. don't include type information). I think this difference could be worked around or even removed.

Some thoughts from me (great project btw!):

Sign-extension [..] non-trapping float-to-int [..] Multi-value [..] Bulk memory operations, exception handling, reference types, multiple memories [..] Do you know if there are certain compilers targeting WebAssembly that leverage or even require these feature?

Several of those features are implemented in LLVM and are optional in the Emscripten and Rust toolchains for example. But they are not required, so projects using them could just be recompiled for wasm 1.0.
Exceptions is maybe the hardest of those. You can compile C++ exceptions and longjmp with wasm 1.0 today, but then you have to use the Emscripten EH model which requires supporting some extra imports (emscripten's wasm2c layer emits them), but also it is probably slower than wasm EH.
Overall, my guess is that compiling wasm 1.0 to C is "good enough" for most things in the C/C++/Rust/Zig/etc. world today. Exceptions maybe raise the question of compiling to C++ instead, and wasm GC maybe suggests compiling to a GC language, but maybe those features would be out of scope of this project anyhow?

Coremark results: Performance should be very similar to wasm2c, as the generated code pretty much the same. I have not looked into opportunities to generate more efficient code.

I would guess nothing is needed in w2c2 or wasm2c for performance since the C compiler does the hard work anyhow. The big question is whether there are things that wasm->C translation can optimize that a C compiler can't, but it's hard for me to think of anything...

@kripken Thank you for answering my questions, that makes a lot of sense!

Exceptions maybe raise the question of compiling to C++ instead, and wasm GC maybe suggests compiling to a GC language, but maybe those features would be out of scope of this project anyhow?

I have no immediate plans for generating other code in other languages, but even though the C code generation (in c.c) is currently writing C directly, it would be possible to abstract it away, e.g. by calling code generating functions, for different target languages.

[...] the C compiler does the hard work anyhow. The big question is whether there are things that wasm->C translation can optimize that a C compiler can't, but it's hard for me to think of anything..

I mostly focused on streaming code generation and assumed that tools like binaryen's wasm-opt can take care of optimizations and w2c2 does not need to reinvent the wheel here. It is likely that such pre-C optimizations are actually useful, as older C compilers likely do not have as advanced optimizations as modern compilers, though I have not tested this yet.

@vshymanskyy I've added support for parallel compilation in #2. Going to work on reducing the output size now

With #3 merged, the output size is now reasonable

Great news. #2 along with #3 bring clang compilation time from 20 minutes to ~3 minutes (gcc, 12 threads, -O3).
I was not able to link it yet, will take look into this later.
Overall, this is a huge improvement :tada: :tada: :tada:

@vshymanskyy That's great to hear! Linking clang.wasm will require implementing quite a few more WASI functions (e.g. file I/O, FS operations), so far I had only implemented enough to get coremark.wasm to run.
I saw you started work on a WASI implementation for wasm2c in wasm2native, great work! It would be nice to leverage this existing work and not have to re-implement the WASI spec specifically for w2c2 again. First step would be to make the API of the generated code of w2c2 match that of wasm2c. I started looking into the differences and noticed that wasm2c includes the parameter types and return types in the mangled names of the imports and exports – is this really necessary?
I also wonder what the minimum requirements for uvwasi and especially libuv are (C standard, endianness, etc.)

WASI implementation improvements are work in progress in #4

I also wonder what the minimum requirements for uvwasi and especially libuv are (C standard, endianness, etc.)

I was able to add w2c2 as an alternative translator for wasm2native.
Re-mapping of symbols is easy:
    #define Z_fd_prestat_getZ_iii               fdX5FprestatX5Fget
    #define Z_fd_prestat_dir_nameZ_iiii         fdX5FprestatX5FdirX5Fname
    #define Z_environ_sizes_getZ_iii            environX5FsizesX5Fget
    #define Z_environ_getZ_iii                  environX5Fget

Along with customized IMPORT_IMPL* definitions.
@turbolent When generating multiple files with -j flag, I'm getting lots of "multiple definitions" of e_X5Fstart and e_memory (defined in decls.h, then included in each c file). Fixed it by making them extern and defining in my main.c.
After that, I was able to build clang.wasm into a working clang.elf. Beware, linking takes ages :grinning:

Ok, got it working. clang.wasm was compiled and linked (using Clang 12 + LLD) in parallel mode (-j 12) in just 3m34s.
Will send a PR with my changes.

@vshymanskyy Wow, nice! It didn't even occur to me to use the pre-processor to add compatibility, great idea 👍
Also, thank you for the bug report and also the fix!
Great to hear clang builds in reasonable time 🎉  Maybe it can be sped up with -O0? (see https://maskray.me/blog/2021-12-19-why-isnt-ld.lld-faster)

@turbolent do all/any of these platforms support CMake? Do you compile on the target, or do you cross-compile?
wasm2native should conceptually support Big-Endian systems, like Wasm3 does. It needs some debugging, as testing with QEMU showed there are issues (but looks like we're almost there).

I'd like to get rid of CMake dependency for wasm2native, but libuv only supports autotools or CMake officially.

@vshymanskyy Node.js builds libuv with gyp. I'm not sure if you can use it for inspiration, but here is the gyp file.

@vshymanskyy For Mac OS X 10.4/10.5 tigerbrew provides CMake 3.6, I'm not sure if that's enough.
As for big-endian support: wasm2c and w2c2 both use "negative memory" (see the end of https://skmp.dev/blog/negative-addressing-bswap/), so access in memory mem with size size at offset off is mem + size - off. Values are in native endianness, but that also means that e.g. data (e.g. string values) is stored in "reversed order" and e.g. needs a reverse before a syscall (e.g. write). I think the first step is to add a big-endian definition for https://github.com/vshymanskyy/wasm2native/blob/afa64bee90b3483e8747f3533906a7332c588a6b/src/wasi-main.c#L16 and then add reverse operations where data is read. I wonder which of the two options I can think off are more efficient:

Two reverses in the linear memory that is pointed to
One copy of the memory that is pointed to into a temporary allocation and a single reverse of it

@vshymanskyy I now have enough of a WASI implementation that should be able to run clang.wasm, e.g. the input file is checked to be existing, but then clang just exits. A simple Rust program exercising all the functionality works as expected, so I'm wondering what I'm missing.
Do you have instructions on how you compiled/created clang.wasm? I would like to build it with assertions enabled, so I can debug it further.

@turbolent thanks for explanations on Big-Endian, will check it soon.
Just checked my clang compilation. For this test I replaced clang.wasm here in my wasm3 self compilation experiment here: https://github.com/wasm3/wasm3-self-compiling/blob/ee61ccecdf30bee73f3c640764896da8f6ca439d/Makefile#L33
It looks working well.
I didn't push my changes to wasm2native yet. I'll let you know when it's ready.

Overall, it may be a good idea to move wasi implementation into a separate project. It's rather complicated, esp. if targeting multiple OS environments.

@turbolent just pushed changes to wasm2native. You should be able to:
git clone https://github.com/vshymanskyy/wasm2native.git
cd wasm2native
export CC="clang-12"
export LDFLAGS="-fuse-ld=lld"
./build.sh path/to/clang.wasm

To run the resulting clang.elf, you can replace the compilation command in wasm3-self-compiling.
But it should work as a standalone compiler as well.

@vshymanskyy Great to see you've added support for w2c2!
I've got clang.wasm working in #4 and documented the status for all WASI syscalls in the README. There are still a few missing and I'll add them as needed, contributions are very welcome! Agreed on moving the WASI implementation to a separate repository once it is in a more complete state.
When compiling clang.wasm I noticed that inits.c is still huge (16MB), so I'll look into splitting how to split it up further, as currently I can only compile it with -O0.

@vshymanskyy Great to see you've added support for w2c2!
I've got clang.wasm working in #4 and documented the status for all WASI syscalls in the README. There are still a few missing and I'll add them as needed, contributions are very welcome! Agreed on moving the WASI implementation to a separate repository once it is in a more complete state.
When compiling clang.wasm I noticed that inits.c is still huge (16MB), so I'll look into splitting how to split it up further, as currently I can only compile it with -O0.

Awesome! 🙌

Update: The WASI implementation now has support for big-endian machines and fd_readdir.
QEMU's user mode emulation is really useful for development, but also has some issues, e.g. when using it on a 64-bit host running 32-bit executables and performing filesystem operations.

With some efforts, it should be possible to wasm2c wasm2c.wasm > wasm2c.c 😱

@turbolent , wasm3 w2c2.wasm w2c2.wasm > w2c2.c works with #7 ! 🎉 🎉 🎉

@vshymanskyy Wow, awesome! This is actually going to be really useful on platforms with bad C compilers (e.g. the SGi MIPSpro compiler on IRIX) 👍

As mentioned above, I ran into linking errors for clang on Mac OS X 10.4 on PowerPC. I assumed this was due to large static arrays that are generated for the data segments. I found some platform-specific ways of embedding the data segments directly in the binary without having to generate large array literals in the source in #8. This is generally useful, however, I am still getting linker issues for the example above, probably due to the binary being too large and jump instructions overflowing?


Identifier Regex is too restrictive.

Bug Report 🐛
As originally reported by @DS-AdamMilazzo.
Regarding the regex for a valid Concerto field name in the JSON representation of a model:
^(?!null|true|false)(\p{Lu}|\p{Ll}|\p{Lt}|\p{Lm}|\p{Lo}|\p{Nl}|\$||\\u[0-9A-Fa-f]{4})(?:\p{Lu}|\p{Ll}|\p{Lt}|\p{Lm}|\p{Lo}|\p{Nl}|\$||\\u[0-9A-Fa-f]{4}|\p{Mn}|\p{Mc}|\p{Nd}|\p{Pc}|\u200C|\u200D)*$

Is it really necessary to avoid the names "null", "true", and "false"?  There's no ambiguity between "null" and null in JSON, anyway. The only trouble may be in the Concerto language, but this regex applies to the JSON format.

More broadly, I think that Concerto should enforce different formats in different contexts. First of all, I'll note that the Concerto metamodel (where these regexes are defined) is supposed to be for the JSON format and not the Concerto (.cto) format, which may well have different rules.

Field names and enum values - There should be no restrictions whatsoever in my opinion. Any legal JSON property name or string should be a legal field name or enum value in Concerto, except maybe empty strings if they're too hard to support.
Type names (concepts, etc) - I think these should be restricted. In fact, I would restrict them even more than you do above. I would not allow arbitrary characters via escaping. The usual ^[_A-Za-z][_0-9A-Za-z]*$ format should be fine. As for trying to exclude certain reserved words (e.g. "concept"), I think that's okay (if done correctly) but I also think it's unnecessary.
Namespace names - I think these should also be restricted more than you currently do. You currently allow namespace names like foo\u0020\bar. I'd limit them to dot-separated strings where each piece has to match ^[_0-9A-Za-z]+$ (or maybe ^[_A-Za-z][_0-9A-Za-z]*$) or something.

As for the Concerto (.cto) language, I think the \uXXXX escaping mechanism that you have today is okay (except where it doesn't fully work), but it'd be nicer if you could simply quote things, e.g. "my value" versus my\u0020value.

Regular Expressions
First, some definitions.
Unicode Categories
See https://en.wikipedia.org/wiki/Unicode_character_property#General_Category for an full explanation of Unicode Categories such as \p{Lu}.

Value
Category

\p{Lu}
Letter, uppercase

\p{Ll}
Letter, lowercase

\p{Lt}
Letter, titlecase

\p{Lm}
Letter, modifier

\p{Lo}
Letter, other

\p{Nl}
Number, letter

\p{Nd}
Number, decimal digit

\p{Mn}
Mark, non-spacing

\p{Mc}
Mark, spacing-combining

\p{Pc}
Punctuation, connector

\u200C
Zero-width non-joiner

\u200D
Zero-width joiner

Psuedo-JavaScript Identifier
From ECMAScript 2015, 11.6 Names and Keywords
https://262.ecma-international.org/6.0/#sec-names-and-keywords
/^(?!null|true|false)(\p{Lu}|\p{Ll}|\p{Lt}|\p{Lm}|\p{Lo}|\p{Nl}|\$|_|\\u[0-9A-Fa-f]{4})(?:\p{Lu}|\p{Ll}|\p{Lt}|\p{Lm}|\p{Lo}|\p{Nl}|\$||\\u[0-9A-Fa-f]{4}|\p{Mn}|\p{Mc}|\p{Nd}|\p{Pc}|\u200C|\u200D)*$/

Note 1. This expression does not exclude keywords such as while, for which are prohibited in ECMAScript 2015.

Note 2. Concerto does not use this expression directly because negative lookaheads permit dangerous expressions (ReDOS). Instead reserved words are prevented in the API.

JSON String
From ECMA-404, 2nd Edition. As used in String keys and values in JSON objects.
/"([^"\\\u0000\u001f]*|\\["\\bfnrt\/]|\\u[0-9a-f]{4})*"/i

JavaScript Identifiers, allowing keywords
This option does not prevent the use of language keywords (such as true, null, while etc.). This would force all code generation logic to escape/unescape identifiers for the relevant target language.
/^(\p{Lu}|\p{Ll}|\p{Lt}|\p{Lm}|\p{Lo}|\p{Nl}|\$|_|\\u[0-9A-Fa-f]{4})(?:\p{Lu}|\p{Ll}|\p{Lt}|\p{Lm}|\p{Lo}|\p{Nl}|\$|\\u[0-9A-Fa-f]{4}|\p{Mn}|\p{Mc}|\p{Nd}|\p{Pc}|\u200C|\u200D)*$/

Comparison
Definitions for cell values are defined above.

Identifier Type
CTO Pattern
AST Pattern†
Proposed CTO Pattern
Proposal is backwards compatible CTO

Property Names
Psuedo-JavaScript Identifier
JSON String
JavaScript Identifiers, allowing keywords
Yes

Enumeration Names‡
Psuedo-JavaScript Identifier
JSON String
JavaScript Identifiers, allowing keywords
Yes

Declaration Names(Concepts, Enums, Scalars)
Psuedo-JavaScript Identifier
JSON String
JavaScript Identifiers, allowing keywords
Yes

Namespaces Part*
Psuedo-JavaScript Identifier
JSON String
JavaScript Identifiers, allowing keywords
Yes

*All Namespaces are period-separated and terminated by an optional version expression, e.g. @1.2.3-alpha.4

†Although the Concerto metamodel defines these constraints on identifier fields in the AST, they are not enforced today.

‡Concerto doesn't support enumeration values (in addition to enumeration identifiers) and so supporting JSON Strings is difficult. In TypeScript enums have optional values which as JSON Strings, we could adopt the same approach. JSON Schema also doesn't distinguish between enumeration identifiers and values, however, it implicitly allows JSON Strings (and other primitive values) as enumeration options.

In summary, today, we only partially prevent the use of JavaScript keywords, this causes confusion to downstream users, but without full protection during code generation for TypeScript. I suggest that we relax the constraint to allow false, true and null as identifiers, and delegate escaping/unescaping to each code generation library in a language-specific way (our reserved keywords in Concerto, should not be the union of reserved words for all target languages).

This looks good @mttrbrts — I'd be in favour of enforcing the same restrictions on AST as in CTO, to ensure that we can't generate invalid CTO from AST in the CTO printer. We should add something to the Concerto docs to make these restrictions more visible.

As a second set of PRs perhaps we can add some utilities or conventions in how the code gen targets indicate their reserved words, and how we escape them?