fix(core): allow onUpdate access to other onUpdate results from earlier properties

Allow onUpdate to get access to previous entity values returned via onUpdate,
which can be useful for ensuring created and updated dates have the exact same time
or basing the value of one property off the value of another.
This already worked for onCreate, as onCreate directly updates the entity.
The onUpdate hook however stores its values in a map to ensure values are not updated
unless they actually need to be, which means onUpdate would always get property values
not accounting for prior onUpdate hooks.

I just realized, that as this no longer provides the whole entity, mutating the entity in an onUpdate is no longer possible which might be an issue? Be interested to hear your feedback on this, as I could just mutate the entity in my code, but then every flush call will make a change to the entity, even if no other property has changed, which onUpdate seems specifically designed around being able to only commit "onUpdate" changes when other non-auto update values change.

I am not interested in doing this, I don't see how it could work propertly. We can't modify the entity directly, as it would always produce queries even if nothing else would be updated. That is the whole point of using the pairs thingy. onUpdate on property level was never designed for this - its purpose is to allow property initializer for interface-only entities.
Use @BeforeUpdate hook instead.

@B4nan thanks for your quick response. Of course it is your call on what is merged in, but I feel from your response that you might have misunderstood what this pull request is doing, or I might not have expressed it clearly enough.
This pull request does not allow mutating the entity directly, and uses the pairs thingy so extra queries are not produced. In fact, this pull request means mutating the entity in an onUpdate hook will now actually not produce a query, where it currently would.
All I have done is pass through the results of onUpdate calls from other properties on the entity object provided to the onUpdate hook.
I feel my second comment might have confused the issue, as I'm advocating against mutating the entity directly, and would like my onUpdate hooks to be able to use values from other properties onUpdate hooks, but still only actually commit the results and run the query if other entities are actually changed.
If I'm mistaken on this, I apologize for the extra noise and thanks for an amazing project!

What I don't like is that you no longer have the entity instance in the argument. While it may sound like a feature to ignore any updates on it, for others it may easily be a breaking change.
As said above, the use case for onUpdate is to mimic property initializers, you should not need the entity reference for such. onUpdate was never meant to "replace BeforeUpdate hook".
What is the actual use case?

I agree that would be a breaking change, and was something I realized not long after I pushed this request.
The beforeUpdate hooks like exactly what I should be using. I saw it in the docs but for some reason though it was global, and not per entity.
One thing I'm struggling with though, is how to define this hook. I'm currently using entity schema with typescript and I can't seem to work out how to define the hook function and I have not been able to find a good example of it in the docs.
The use case I have is to add a microsecond epoch timestamp as a BIGINT, as well as a DATETIME of the same timestamp. If only javascript date objects support microseconds! haha

One example is this:
https://github.com/mikro-orm/mikro-orm/blob/master/tests/entities-js/Author3.js#L116
You define the methods on entity class as usual, and just add those method names to hooks object in the schema definition.

Oh ok. So it requires a concrete class? I currently only have an interface. I think it should be ok to change to use a class.

I don't remember if its implemented, I remember I was planning to allow also passing function handles instead of function names for this use case. Can't find it in the code now, so probably still missing.

@B4nan just made pull request #2895, is this what you were thinking?


docs: update asciicast

We have been forced to merge this PR without a status check because it’s not working properly.


python3Packages.poolsense: init at 0.0.8

Motivation for this change
Python module to access PoolSense device.
https://github.com/haemishkyd/poolsense
This is a Home Assistant dependency.
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[x] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

This is a semi-automatic executed nixpkgs-review which does not build all packages (e.g. lumo, tensorflow or pytorch)
If you find some bugs or got suggestions for further things to search or run please reach out to SuperSandro2000 on IRC.
Result of nixpkgs-review pr 109298 run on x86_64-darwin 1

3 packages built:

python37Packages.poolsense
python38Packages.poolsense
python39Packages.poolsense

This is a semi-automatic executed nixpkgs-review which does not build all packages (e.g. lumo, tensorflow or pytorch)
If you find some bugs or got suggestions for further things to search or run please reach out to SuperSandro2000 on IRC.
Result of nixpkgs-review pr 109298 run on x86_64-darwin 1

3 packages built:

python37Packages.poolsense
python38Packages.poolsense
python39Packages.poolsense

This is a semi-automatic executed nixpkgs-review which does not build all packages (e.g. lumo, tensorflow or pytorch)
If you find some bugs or got suggestions for further things to search or run please reach out to SuperSandro2000 on IRC.
Result of nixpkgs-review pr 109298 run on x86_64-linux 1

3 packages built:

python37Packages.poolsense
python38Packages.poolsense
python39Packages.poolsense

This is a semi-automatic executed nixpkgs-review which does not build all packages (e.g. lumo, tensorflow or pytorch)
If you find some bugs or got suggestions for further things to search or run please reach out to SuperSandro2000 on IRC.
Result of nixpkgs-review pr 109298 run on x86_64-linux 1

3 packages built:

python37Packages.poolsense
python38Packages.poolsense
python39Packages.poolsense


Need docs and samples on proactive messaging

Hi @apulliam , here's a link to our v4 topic on How to use proactive messaging. Let me know if that is missing information that you need.

We have a doc on the topic.


Remove references to out-of-date book and video.

LGTM.
What about this file?
https://github.com/stampit-org/stampit/blob/master/docs/pjabook-updated-examples.md
Should we leave it or remove too?


Video Classification / Activity-Recognition

I've been experimenting with this project on Jetson TX1 for some time and found it's a very good base start. I'm interested by doing video classification. I modified imagenet-console to receive to video ( from file or rtsp stream) do classify each frame using googlenet. This is working, but I would like to go further and been able to classify video (spatio-temporal data) has opposed to static image. Did someone tried to video classification using TensorRT? Is there example of Temporal-Segment-Network (TSN) model used on TensorRT.  I'm hoping to do classification on UCF101 dataset.
I found a UCF101 pre-trained networks ( image+optical flow) on this project:
https://github.com/yjxiong/temporal-segment-networks
Unfortunately TensorRT does not support Batch Normalization layer used in this network.
I found a note in TensorRT documentation:
"Batch Normalization can be implemented using the TensorRT Scale layer."
Read more at: http://docs.nvidia.com/deeplearning/sdk/tensorrt-user-guide/index.html#ixzz4ul8v2yAv
Follow us: @GPUComputing on Twitter | NVIDIA on Facebook
I'm note sure how to do the conversion from Batch Normalization to Scale layer. Any advice would be welcome on this subject.
Assuming I can find a way to convert Batch Normalization layer to Scale layer. Can I simply modify the prototxt file and keep using pre-trained network (caffemodel) without re-training?
Thank you.

Hi dmorin1,
I'm trying for some time now to modify the imagenet-console to work with a rtsp stream (from ip camera) but cannot let it work so far?
is it possible to share me your code or to give me some directions how to let it work?
thank you


h3 3.6.4 OSX builds failed

github action log, https://github.com/Homebrew/homebrew-core/runs/792748977

build failure log
[  8%] Linking C shared library lib/libh3.dylib
/usr/local/Cellar/cmake/3.17.3/bin/cmake -E cmake_link_script CMakeFiles/h3WithTestAllocators.dir/link.txt --verbose=1
/usr/local/Cellar/cmake/3.17.3/bin/cmake -E cmake_link_script CMakeFiles/h3.dir/link.txt --verbose=1
/usr/local/Homebrew/Library/Homebrew/shims/mac/super/clang -DNDEBUG -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk -dynamiclib -Wl,-headerpad_max_install_names -compatibility_version 1.0.0 -o lib/libh3.1.dylib -install_name @rpath/libh3.1.dylib CMakeFiles/h3.dir/src/h3lib/lib/algos.c.o CMakeFiles/h3.dir/src/h3lib/lib/coordijk.c.o CMakeFiles/h3.dir/src/h3lib/lib/bbox.c.o CMakeFiles/h3.dir/src/h3lib/lib/polygon.c.o CMakeFiles/h3.dir/src/h3lib/lib/h3Index.c.o CMakeFiles/h3.dir/src/h3lib/lib/vec2d.c.o CMakeFiles/h3.dir/src/h3lib/lib/vec3d.c.o CMakeFiles/h3.dir/src/h3lib/lib/linkedGeo.c.o CMakeFiles/h3.dir/src/h3lib/lib/localij.c.o CMakeFiles/h3.dir/src/h3lib/lib/geoCoord.c.o CMakeFiles/h3.dir/src/h3lib/lib/h3UniEdge.c.o CMakeFiles/h3.dir/src/h3lib/lib/mathExtensions.c.o CMakeFiles/h3.dir/src/h3lib/lib/vertexGraph.c.o CMakeFiles/h3.dir/src/h3lib/lib/faceijk.c.o CMakeFiles/h3.dir/src/h3lib/lib/baseCells.c.o  /usr/lib/libm.dylib
/usr/local/Homebrew/Library/Homebrew/shims/mac/super/clang -DNDEBUG -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk -dynamiclib -Wl,-headerpad_max_install_names -compatibility_version 1.0.0 -o lib/libh3WithTestAllocators.1.dylib -install_name @rpath/libh3WithTestAllocators.1.dylib CMakeFiles/h3WithTestAllocators.dir/src/h3lib/lib/algos.c.o CMakeFiles/h3WithTestAllocators.dir/src/h3lib/lib/coordijk.c.o CMakeFiles/h3WithTestAllocators.dir/src/h3lib/lib/bbox.c.o CMakeFiles/h3WithTestAllocators.dir/src/h3lib/lib/polygon.c.o CMakeFiles/h3WithTestAllocators.dir/src/h3lib/lib/h3Index.c.o CMakeFiles/h3WithTestAllocators.dir/src/h3lib/lib/vec2d.c.o CMakeFiles/h3WithTestAllocators.dir/src/h3lib/lib/vec3d.c.o CMakeFiles/h3WithTestAllocators.dir/src/h3lib/lib/linkedGeo.c.o CMakeFiles/h3WithTestAllocators.dir/src/h3lib/lib/localij.c.o CMakeFiles/h3WithTestAllocators.dir/src/h3lib/lib/geoCoord.c.o CMakeFiles/h3WithTestAllocators.dir/src/h3lib/lib/h3UniEdge.c.o CMakeFiles/h3WithTestAllocators.dir/src/h3lib/lib/mathExtensions.c.o CMakeFiles/h3WithTestAllocators.dir/src/h3lib/lib/vertexGraph.c.o CMakeFiles/h3WithTestAllocators.dir/src/h3lib/lib/faceijk.c.o CMakeFiles/h3WithTestAllocators.dir/src/h3lib/lib/baseCells.c.o  /usr/lib/libm.dylib
Undefined symbols for architecture x86_64:
  "_test_prefix_calloc", referenced from:
      _kRingDistances in algos.c.o
      __polyfillInternal in algos.c.o
      _compact in h3Index.c.o
      _addNewLinkedPolygon in linkedGeo.c.o
      _addNewLinkedLoop in linkedGeo.c.o
      _normalizeMultiPolygon in linkedGeo.c.o
      _initVertexGraph in vertexGraph.c.o
      ...
  "_test_prefix_free", referenced from:
      _kRingDistances in algos.c.o
      __polyfillInternal in algos.c.o
      _compact in h3Index.c.o
      _destroyLinkedGeoLoop in linkedGeo.c.o
      _destroyLinkedPolygon in linkedGeo.c.o
      _normalizeMultiPolygon in linkedGeo.c.o
      _destroyVertexGraph in vertexGraph.c.o
      ...
  "_test_prefix_malloc", referenced from:
      __polyfillInternal in algos.c.o
      _compact in h3Index.c.o
      _addLinkedCoord in linkedGeo.c.o
      _normalizeMultiPolygon in linkedGeo.c.o
      _addVertexNode in vertexGraph.c.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
/usr/local/Cellar/cmake/3.17.3/bin/cmake -E cmake_symlink_library lib/libh3.1.dylib lib/libh3.1.dylib lib/libh3.dylib
make[2]: *** [lib/libh3WithTestAllocators.1.dylib] Error 1
make[1]: *** [CMakeFiles/h3WithTestAllocators.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[  8%] Built target h3
make: *** [all] Error 2

PR reference, https://github.com/Homebrew/homebrew-core/pull/56633
cc @robsalasco

This is an older post but it looks like this may have to do with XCode defaulting to not looking up undefined symbols dynamically: https://glandium.org/blog/?p=2764 I was able to reproduce this problem with XCode 11.5 installed.
The undefined symbols are only used in tests to confirm that the library can built with undefined symbols for memory allocation that are then provided at runtime. If that functionality isn't being used, we could just disable the test.

@isaacbrodsky applied the patch in https://github.com/Homebrew/homebrew-core/pull/57084, working well for me. Thanks for the fix!
Closing the issue now.


Fix: PHP require()/require_once() and include()/include_once() wrong syntax

Explanation
In PHP we can use these commands, like this:
<?php
require 'file.ext';
require_once 'file.ext';

But we can also use it with the path between parentheses, like this:
require('file.ext');
require_once('file.ext');

However, the syntax highlight were only detecting the right scope of the command that doesn't use parentheses:

Fix
This pull request resolves it by adding the ( char after the require/require_once and include/include_once as a valid match to the  keyword.control.import.include.php scope.

Thanks for the PR! However our PHP syntax is actually just pulled from this upstream repo: https://github.com/atom/language-php/blob/master/grammars/php.cson could you open an issue and PR with them first, then I can pull the fix into VS Code? The fix looks good to me.

Done!
Pull request opened: atom/language-php#420
Thanks!


Switch default openai chatbot model to gpt-4o.

I've switched gpt-4o in place of gpt-4-turbo for the chatbot model. By default, gpt-4o will be loaded now.
Also, I noticed that in chat.py file, openai_model is used in session_state instread of openai_chatbot_model:
# Set a default model
if "openai_model" not in st.session_state:
    st.session_state["openai_model"] = "gpt-4o"

I guess this is a mistake, if someone can confirm this then i'll go ahead and add a commit to fix that as well.

yes that's an error. chatbot.py uses the key openai_chatbot_model to store the model info in session state and in db.json

PR is ready to be merged.


Add the iModel.js Docs build as yaml

This pull request replaces a classic build pipeline that builds the itwinjs documentation site artifacts from the various sources within this repository, and some other currently internal repositories.
The reason for the switch is to allow for easier structural build changes to the itwinjs-core repo without creating a mess of conditionals all over the classic release pipeline that makes maintenance difficult.

Note: Over time we will be moving the other sources of docs into the open (primarily the BIS related schema docs). The timeframe for that is currently to be determined.

The steps in order to get this PR completed:

[ ] Review the PR to ensure no steps are missing from the current pipeline into this one (@williamkbentley, @BaoTon  and @calebmshafer should review)

[ ] Correctly handle the .updated.json file

[ ] Determine strategy for updating the release pipeline to handle both this build and the 2.x and 1.x branches.
[x] Fix the commits that were committed with a bad email address that cannot sign the CLA (@naveedkhan8067)

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.3 out of 4 committers have signed the CLA.:white_check_mark: BaoTon:white_check_mark: calebmshafer:white_check_mark: naveedkhan8067:x: Naveed.KhanNaveed.Khan seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.

closing PR as this has been finished in a different PR.


Team data

#744
Converting HTML data to JS Object for card Injection. Will be used as a dependency for #747

@anushbhatia Can you open this one since the other one is opened by Sit late night and I need this to make that work!


can not test source logins

What steps will reproduce the problem?
hitting the test button on the credentials tab of a source
What is the expected behavior? What do you get instead?
I suppose I should get a "successful" instead of " unable to test"
How often does this problem occur?
everytime
What version of the program are you using? On what operating system?
5.2.3
Please provide any additional information below
beside that, the sources window width could be bigger, I see only half of the "Options" buttons :)

Depends on the source, but most the "unable to test" ones use URL logins (by passing the username and hashed password directly in the URL), and it's impossible to properly test if the login works or not with that system. One option would be to simply remove the "Test" button for those sources though.
Let's take Danbooru as an example, there's no real way to check. One could check premium accounts with a 3-tags or a "forbidden" tag search, but for normal accounts, it's not that easy (unless I missed something actually).
For Sankaku for example, it's the same system but they have different max page limit for logged in users, so Grabber checks by doing a search at page 50 and see if there's an error message. Another option could be trying to connect to the JSON API which is reserved to logged in users.
Which source exactly were you mentioning in this issue?

ah, thanks for the background information :)
the sources, that don't work, are zerochan, both rule34 sites, e621.net and chan.sankakucomplex
only gelbooru returns a success
so I suppose e621 won't work because of danbooru

Closing because now the "Test" button is not shown anymore for sources that do not have a configured way to test logins.


[vcpkg baseline] [embree2/embree3] Set cmake policy

Related to https://github.com/microsoft/vcpkg/pull/17978
Set cmake policy for fixing failures:
CMake Error in CMakeLists.txt:
  You have set CMAKE_BACKWARDS_COMPATIBILITY to a CMake version less than
  2.4.  This version of CMake only supports backwards compatibility with
  CMake 2.4 or later.  For compatibility with older versions please use any
  CMake 2.8.x release or lower.

Reference https://cmake.org/cmake/help/latest/policy/CMP0001.html#policy:CMP0001

Merge this changes to PR https://github.com/microsoft/vcpkg/pull/17983 since the baseline ports affected by each others.


README fixes

What

Fix callback URI. This one matches what's been hardcoded in src/auth/auth.js.
Add example required/optional environment variables.

How to review
Check the new changes work for running the server locally against a UAA in e.g. a dev env.
Who can review
Not @carolinegreen or me.

I've smashed together the env var commits.


Verify regenerating CSV doesn't result in changes

Ensure changes that cause CSV updates are commited with the CSV updates.
Relates-to: #395
Signed-off-by: Daniel Farrell<EMAIL_ADDRESS>

/cc @skeeey @tpantelis @sridhargaddam

/lgtm

/test golangci-lint


GH-1437 Check presence of working directory parameter in REPOSILITE_OPTS (Fix #1437)

c.f https://github.com/dzikoysk/reposilite/issues/1437

Thanks!


Invalid transformation of Pat.Var.name: Term.Name -> Term.Apply

I'm trying to transform this program such that the variable x becomes Client.__read("x"):
object Main
{
    def main(args: Array[String]) =
    {
        var y = x + 1
        y
    }
}

using the following transformation:
 case t@Term.Name(variable) => {
 println("found="+variable)
 Term.Apply(Term.Select(Term.Name("Client"), Term.Name("__read")), List[Term](Lit.String("x")))
}

But I get the following error, why is this not allowed? It is a perfectly fine transformation?

Invalid transformation of Pat.Var.name: Term.Name -> Term.Apply
From: Term.Name("x"), to: Term.Apply(Term.Select(Term.Name("Client"), Term.Name("__read")), List(Lit.String("x")))

Thanks for reporting!
@ import $ivy.`org.scalameta::contrib:2.0.0`, scala.meta._, contrib._
import $ivy.$                             , scala.meta._, contrib._

@ q"val x = y"
res1: Defn.Val = Defn.Val(List(), List(Pat.Var(Term.Name("x"))), None, Term.Name("y")

As you can see, Term.Name can be used both at definition site and call-site. You can guard against this situation so you only transform references (call-site) with
import scala.meta.internal._
...
case t @ Term.Name(variable) if t.isReference =>

We should probably expose .isDefinition/.isReference in the non-internal API, wdyt @xeno-by ?

👍


Passing argument of no-dram mode

So we wouldn't need to maintain two versions of each regression test for dram/no-dram mode.

Possibly related to #324? Duplicate?

yeah it is. Let's close this one.


Cargo publish path does not point to folder with .so files

Issue
Following your blog post at: Rust cross-platform... The Android part..., I run into the issue that cargo run --bin publish gives the error Error copying library file. The full output that that command gives:
warning: `/home/{user}/github/Rust-Cross-Platform-Development/rust-library/cryptor_jni/.cargo/config` is deprecated in favor of `config.toml`
note: if you need to support cargo 1.38 or earlier, you can symlink `config` to `config.toml`
   Compiling cryptor_jni v0.1.0 (/home/{user}/github/Rust-Cross-Platform-Development/rust-library/cryptor_jni)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.33s
     Running `target/debug/publish`
Message: Error copying library file

This error message comes from line 150 in publish.rs.
Probable cause
It seems that cryptor_jni is missing from the path used in publish.
The output paths:

cargo run --bin release path: Rust-Cross-Platform-Development/rust-library/cryptor_jni/target/armv7-linux-androideabi/release/libcryptor_jni.so
cargo run --bin publish path: Rust-Cross-Platform-Development/rust-library/target/armv7-linux-androideabi/release/libcryptor_jni.so

I executed the release and publish commands from: Rust-Cross-Platform-Development/rust-library/cryptor_jni
Release from rust-library
If I run the command rust-library$ cargo run --bin release (instead of in rust-library/cryptor_jni), it throws the error:
error: multiple workspace roots found in the same workspace:
  /home/{user}/github/Rust-Cross-Platform-Development/rust-library/cryptor_jni
  /home/{user}/github/Rust-Cross-Platform-Development/rust-library

Debugging
I modified that Err line to output crate_lib_file_path and android_lib_file_path:
return Err(format!("Error copying library file {} |\n{}", crate_lib_file_path, android_lib_file_path).into())

Which gives me:
Message: Error copying library file /home/{user}/github/Rust-Cross-Platform-Development/rust-library/target/armv7-linux-androideabi/release/libcryptor_jni.so |
/home/{user}/github/Rust-Cross-Platform-Development/android-sample/app/src/main/jniLibs/armeabi-v7a/libcryptor_jni.so

Comparing this to the output from cargo run --bin release:
warning: `/home/{user}/github/Rust-Cross-Platform-Development/rust-library/cryptor_jni/.cargo/config` is deprecated in favor of `config.toml`
note: if you need to support cargo 1.38 or earlier, you can symlink `config` to `config.toml`
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.02s
     Running `target/debug/release`
Message: Releasing Android Targets...Be patient... :)
Message: Building Android Target --> armv7-linux-androideabi
warning: `/home/{user}/github/Rust-Cross-Platform-Development/rust-library/cryptor_jni/.cargo/config` is deprecated in favor of `config.toml`
note: if you need to support cargo 1.38 or earlier, you can symlink `config` to `config.toml`
   Compiling cryptor_jni v0.1.0 (/home/{user}/github/Rust-Cross-Platform-Development/rust-library/cryptor_jni)
    Finished `release` profile [optimized] target(s) in 0.49s
...

Adding the following 2 lines:
    crate_lib_file_path.push_str(MAIN_SEPARATOR_STR);
    crate_lib_file_path.push_str("cryptor_jni");

To the function:
/// `$ rust-library/target/x86_64-linux-android/release/JNI_LIB_FILE_NAME`
///
fn crate_file_path_for_target(project_dir_path: &str, android_target: &str) -> String {
    let mut crate_lib_file_path = project_dir_path.to_owned();

    crate_lib_file_path.push_str(MAIN_SEPARATOR_STR);
    crate_lib_file_path.push_str("cryptor_jni");
    crate_lib_file_path.push_str(MAIN_SEPARATOR_STR);
    crate_lib_file_path.push_str("target");
    crate_lib_file_path.push_str(MAIN_SEPARATOR_STR);
    crate_lib_file_path.push_str(&android_target);
    crate_lib_file_path.push_str(MAIN_SEPARATOR_STR);
    crate_lib_file_path.push_str("release");
    crate_lib_file_path.push_str(MAIN_SEPARATOR_STR);
    crate_lib_file_path.push_str(JNI_LIB_FILE_NAME);

    crate_lib_file_path
}

Solved the issue for me and it was able to execute successfully.
Hardcoding "cryptor_jni" might not be the correct approach to solve it properly though?


EMI stealing focus even when turned off

I've started to have an issue using REI where if I right click on the search bar to clear it, I cannot type anything until I exit and enter again the inventory. I thought it was REI having focus problems, but it's instead directly depending on EMI.
If I turn on the EMI UI what I typed on the REI search bar, but wasn't shown, appears instead in EMI's own search bar.
I think the same might happen when pressing R or U while hovering on the right panel item list.
I need to have both REI and EMI for compatibility reasons, and I toggle them on/off with their respective shortcuts.
The problem EMI has is that it steals focus even when it's "off". What it should do, instead, is to disable its own UI-related shortcuts, if EMI UI is turned off. So the search bar and recipe shortcuts. It doesn't make sense to have the shortcuts active while the UI is switched off (outside of the shortcuts to toggle the UI itself, of course).

Oops. Turns out this was a 1 line mistake. Fixed locally, I'll close when it's released.

Should be resolved (or at least improved) in 0.6.4


refactor: 🎨 Refactor the players array into a map

Keep in mind this refactor also removes the UID array

Are we still planning to remove alivePlayers?

@mrjoshua520 some of my comments got resolved but I don't see any new commits am I missing something?

@mrjoshua520 some of my comments got resolved but I don't see any new commits am I missing something?

There are commits waiting to be pushed but I figured I wouldn't until I got a chance to fix some of the bugs that are still in the branch


RFC: Declare Maps Up-Front

This is a proposal to introduce a new syntax for declaring maps ahead of their use, outside of probes:
let @x[uint64] = stack_t;

BEGIN
{
  @x[123] = ustack();
}

I would like this functionality, but am not tied to this syntax in particular.
Use Case 1 - Declare key/value types
The types of map keys and values are currently inferred automatically from usage. This works well for simple scripts, but there are maintainability benefits to declaring the expected types of variables in larger scripts, both for understandability and ease of refactoring.
Example - Types not inferred from first usage
let @x[uint64] = int64;

PROBE1
{
  @x["123"] = 456; // with a map declaration this line produces a type error
}
PROBE2
{
  @x[123] = 456;
}
PROBE3
{
  @x[0] = 0;
}

In the middle of refactoring a large script, changing the type of the key in PROBE1 might have been missed and it would be useful to get an error on that line. Without a map declaration, PROBE2 and PROBE3 would trigger type errors as PROBE1 was seen first.
Example - Control exact types used
let @x[uint8] = int32;

BEGIN
{
  @x[123] = 456; // the literals would be inferred as int64 without the declaration
}

Use Case 2 - Configure Map Settings
The default capacity of a hash map is not large enough for many purposes. Increasing a map's capacity is currently only possible through a global config option that applies to all maps. This wastes memory by increasing the capacity of all maps, when other maps may be fine with a smaller capacity.
Example - Custom size for hash map
let @x[int64] = int64 : HashMap(1024);

BEGIN
{
  @x[123] = 456;
}

Use Case 3 - Create non-hash-maps
Allow users to declare maps backed by data structures other than a BPF Hash Map. For example, an array map might be used for efficiency when it is known that a fixed selection of keys is needed.
Example - Create an array map
let @x[uint32] = int64 : ArrayMap(128); // BPF specifies that array maps must have a "uint32" key

BEGIN
{
  @x[123] = 456;
}

Yeah, use cases seem totally reasonable. Exact syntax should hopefully leverage type annotation work done for user defined functions.

Agreed with @danobi, #2624 uses the $arg: type syntax for function arguments so we should somehow mirror that here (we could eventually add $var: type = ... syntax for local vars.

All makes sense to me. Maybe because I just worked on the config syntax, I'm considering the "floating" element of these type annotations e.g. will they be forced to be declared at the top? should they go after config = {?
Perhaps they should get wrapped in some other similar demarcation e.g.
types = {
   let @x[uint64] = int64;
}

User-defined command line args could also live here. Just something to think about as bpftrace has no (and doesn't want) header files and we want a way to keep things organized.

Taking a look at this now. The current (verbose) syntax I'm looking at is:
let global @x: int32;
let global @x: [int32 -> int64];
let global @x: [int32, int16 -> int64];

This would be extensible in future to (with certain conditions):
let global @y: int64 = ArrayMap;
let global @y = ArrayMap;

I considered making the global optional, but for now have left it global.
Placement rules are currently equivalent to probes. That is, you can place these let statements anywhere in the global scope. The semantic analyser rejects duplicate lets for the same map, and processes all maps before processing the rest of the program. This means that a map declared after a probe that uses it has full effect.
The let syntax could be further extensible to things like:
let t1 = [int32 -> int64];
let global @x: type;
let t2 = int32;
BEGIN { let y: t2; }

Feedback seems to be to drop the global, which doesn't create any more ambiguity so that's fine. I like the rest for a uniform "declare" something syntax but could be swayed too. Will put up a PR soon.

@JakeHillion Thanks for picking this up.
As discussed offline, I'm curious if dropping both the "global" and the "let" is possible to improve the conciseness of this new syntax e.g.
@x: int32;
@x: [(int32, int16) -> int64];

I think this could then be re-used for scratch variables inside probes when we want to add variable declarations e.g.
kprobe:do_nanosleep {
  $scratch: macaddr_t;
  if (condition) {
    $scratch = ...
  } else {
    $scratch = ...
  }
}

Restarting this discussion on map declarations as it seems to be of a higher priority now.
After the decision to add the let syntax: https://github.com/bpftrace/bpftrace/issues/3423
I propose this for basic map declarations:
Scalar maps (maps with no keys)
let @x: uint64; // no initialization
let @x: uint64 = 1; // with initialization

Maps with keys:
let @y: <string, uint64>; // no initialization
let @y: <string, uint64> = {{"hello", 1}, {"bye", 2}}; // with initialization

Don't want to use brackets here for the map type as those are reserved for arrays. Also as a first step we don't have to allow initialization with declaration for maps with keys.
Maps with multi-value keys:
let @z: <(string, int8), uint64>;

Something to note here is that, unlike scratch variables, declaring maps without attributes is somewhat useless. Unless we want to allow pre-populating a map with values (like in the example above) but you don't need a declaration for that - could just be some added sugar. The let keyword is also a tad suspect as maps don't suffer from block scoping so there is no case when a user would need to declare a map like let @y;.
As per some discussion here: https://github.com/bpftrace/bpftrace/issues/3360) we would add map attributes so users can define what kind of BPF map type they want to use e.g.
[type = LRUHashMap]
let @z: <(string, int8), uint64>;

Scalar maps (maps with no keys)
let @x: uint64; // no initialization
let @x: uint64 = 1; // with initialization

Looks good to me. Initialisation could be a fairly big extension but it seems reasonable to allow it.

[type = LRUHashMap]
let @z: <(string, int8), uint64>;

I'm thinking that the type of the map should appear directly in the type declaration rather than as an attribute - it is the "type" after all! Maybe like:
let @map: LRUHashMap<string, uint64>;

The syntax is starting to look pretty familiar 😅
Then my question is do we leave max_entries as an optional attribute for all map types, or should an array map require it to be specified? e.g.:
let @map: Array<int32, 32>; // 32 elements long

Scalar maps (maps with no keys)
let @x: uint64; // no initialization
let @x: uint64 = 1; // with initialization

Looks good to me. Initialisation could be a fairly big extension but it seems reasonable to allow it.

[type = LRUHashMap]
let @z: <(string, int8), uint64>;

I'm thinking that the type of the map should appear directly in the type declaration rather than as an attribute - it is the "type" after all! Maybe like:
let @map: LRUHashMap<string, uint64>;

I think this is the important question here. B/c if we decide this is the best path forward, then we need to take on map key/value type declarations along with underlying map type in the same go.
But if we decide the two are decoupled (ie. with a map attribute), then we can take on the two things independently. In which case I'd strongly believe we should attack underlying map type first. But that is contingent on what we think the long term solution is.

But if we decide the two are decoupled (ie. with a map attribute), then we can take on the two things independently. In which case I'd strongly believe we should attack underlying map type first. But that is contingent on what we think the long term solution is.

I don't have many concrete use-cases in mind right now so I may be missing some context but I am (as always) calling out for bpftrace simplifying user's lives as much as possible. That is, if the user wants to use a particular map type, we still shouldn't force him to specify full map signature including keys and value types, max entries, etc. So, from my perspective, these should be independent.
A counter-argument would be that if the user is expert enough to know which kind of map they want, they may as well fill the other information in. I sort of agree with that but still would prefer bpftrace to infer things where possible.

Then my question is do we leave max_entries as an optional attribute for all map types, or should an array map require it to be specified? e.g.:
let @map: Array<int32, 32>; // 32 elements long

Other maps have max_entries, too. We may then end up with something like:
let @map: PercpuHashMap<int64, int64, 32>

which starts to be a bit unintuitive and named args would help a lot here.
OTOH I agree with @ajor that map type (as in "hash", "array", etc.) is a part of the overall map type (as in "data type of the storage") and therefore shouldn't be set from a different syntax construct. What about allowing optional argument names in the angle brackets? All these would be valid, then:
let @map: PercpuHashMap<key=int64, value=int64, max_entries=32>;
let @map: PercpuHashMap<int64, int64, 32>;
let @map: <key=int64, value=int64, max_entries=32>;
let @map: <int64, int64>;

Some of them are a bit long but I believe terseness is no longer a point for this syntax. We could also simplify our lives a bit by requiring either all or none of the args to have names but that mostly depends on how hard this would be to parse.

I like the addition of the map "type" in front of the angle bracket but I agree with @viktormalik that it would be nice if users didn't have to specify the key/value types in order to set the map "type" so maybe this is ok syntax:
let @map: PercpuHashMap;
// or
let @map: PercpuHashMap<>;

This would potentially allow us to decouple the implementation but, at the same time, adding optional key/value types doesn't seem like it would be that much work or code.

let @map: PercpuHashMap<key=int64, value=int64, max_entries=32>;

Do we feel like including max_entries and other map attributes within the angle brackets scales? Especially with named arguments that declaration might start to get long and ugly. Whereas if we go with putting attributes above the declaration then we can have line breaks e.g.
[max_entries = 32]
[other_attr = "whatever"]
let @z: PercpuHashMap<int8, uint64>;

Also if we go with probe attributes at some point there will be continuity there with map attributes, which I like.

let @map: PercpuHashMap<key=int64, value=int64, max_entries=32>;

Do we feel like including max_entries and other map attributes within the angle brackets scales? Especially with named arguments that declaration might start to get long and ugly. Whereas if we go with putting attributes above the declaration then we can have line breaks e.g.
[max_entries = 32]
[other_attr = "whatever"]
let @z: PercpuHashMap<int8, uint64>;

Also if we go with probe attributes at some point there will be continuity there with map attributes, which I like.

Where do we draw the line between what's part of a map type and what's an "attribute"? In kernel, all the information (usually type, key, value, max_entries but some map types have other fields) is a part of the map type definition. Do we say that in bpftrace, just key, value, and type are a part of the type (i.e. in angle brackets) and everything else is an attribute? What if there are maps which require the other fields to be set and there's no "default" value?

RE max_entries: For hash maps, stacks, queues, etc., users generally won't care what the exact capacity is as long as big enough to hold the entries they want to insert. It's fine for the capacity to be bigger than needed.
For arrays, there's no difference between capacity and current size, so it is probably something that users should always set themselves. The capacity being bigger than needed will impact on user programs (other than memory usage): len(@myarray) will always be max_entries and for-loops will always loop that number of times.
That's why I think it'd be good to make max_entries a required field for arrays but keep it optional for other map types. This could be made to work with either syntax option though.

There's also flags to consider: https://github.com/torvalds/linux/blob/b1b46751671be5a426982f037a47ae05f37ff80b/tools/include/uapi/linux/bpf.h#L1357-L1412
I suspect it's unavoidable in the long run that kernel will add all sorts of fancy flags / fields for special maps. So having a good way to compose additional attributes is a must.

This has all been going into a sidebox, my bad! I was going to dig out my branch, but it seems the syntax has changed enough that it's not worth it - most of the change was in the parser, actually implementing it (without specifying a map implementation) was easy, and I still hadn't done the remaining hard bit of getting decent error messages.

Do we feel like including max_entries and other map attributes within the angle brackets scales? Especially with named arguments that declaration might start to get long and ugly. Whereas if we go with putting attributes above the declaration then we can have line breaks e.g.
[max_entries = 32]
[other_attr = "whatever"]
let @z: PercpuHashMap<int8, uint64>;

Also if we go with probe attributes at some point there will be continuity there with map attributes, which I like.

@jordalgo is there any reason we can't have the line breaks like:
let @z : PercpuHashMap<int8, uint64,
    max_entries = 64,
    other_attr = "whatever",
>;

Should all parse as whitespace right?
As an aside, and not to jump into another syntax debate, I think that's my preferred syntax:

Starts with let @z : - familiar if you've seen variable decls in a lot of languages, you're declaring something named @z. @z:  by itself takes me a lot of mental parsing and I'd need to read the rest of the statement to work out I was declaring something variable like.
Immediately tells you the type of the map if there is one, and throws the word Map at you if you weren't otherwise clear.
Key and Value are as expected in a lot of languages, the first two arguments. Extend to tuples for the multi-key version, which is also familiar.
Attributes like max_entries come last and are named (not positional). These are likely the least relevant part of the declaration, so last is good. Would be nice if these allowed trailing commas, but it isn't a dealbreaker.
Leaves the door open for = {}; initialisation, which could be a nice bit of sugar over filling them in in BEGIN manually.

The only thing I do wonder is whether we should reconsider using -> to denote a mapping from key(s) to value. IMO in most languages which use generics for maps the comma makes a lot of sense, but in BpfScript maps are a first class feature. Is a bit of special syntax to demonstrate the "mapping" (or lack thereof if it's not there) a nice to have? I kind of like it.
My 2c.

let @z : PercpuHashMap<int8, uint64,
    max_entries = 64,
    other_attr = "whatever",
>;

@JakeHillion I don't think there is a technical reason against this syntax but I do have other reasons why I'm not a fan 😄  :

It wouldn't align with probe attributes if we end up having those. I think coming up with a consistent "attribute" syntax for anything that needs them would help make scripts both easier to read and write. And I don't see us being able to re-use the above syntax for probes.
In general, I find it visually unappealing and hard to read. In my mind (and this speaks to @viktormalik's question above about where we draw the line between an attribute and type information) the above declaration has 2 categories of information (map attributes which include the map "type" and the key/value types) that are mixed into three kinds of syntax. And to be fair, the same is true for what I suggested earlier e.g.[max_entries = 32]
[other_attr = "whatever"]
let @z: PercpuHashMap<int8, uint64>;

Which is why I think we should NOT opt for adding the map type in front of the angle bracket (even though it's tempting). Let's keep attributes and key/value types distinct visually. This also gives the user the most flexibility in terms of declaring maps and utilizing any of these optional parts of the declaration (e.g. it feels weird to let them do let @map: PercpuHashMap<>). So, again, all of these would be legal: [type = LRUHashMap]
 [max_entries = 32]
 let @x <int, uint64>;

 [type = LRUHashMap]
 let @z: <int8, uint64>;

 [type = LRUHashMap]
 let @y;

 let @v: <int8, uint64>;

 let @u;

Note: I don't think I have a strong opinion yet on this:
[type = LRUHashMap]
[max_entries = 32]
let @x <(string, int8), uint64>;

vs.
[type = LRUHashMap;
 max_entries = 32;]
let @x <(string, int8), uint64>;

(maybe the former because it's easier to align the attributes visually).

the above declaration has 2 categories of information (map attributes which include the map "type" and the key/value types) that are mixed into three kinds of syntax.
[...]
Which is why I think we should NOT opt for adding the map type in front of the angle bracket (even though it's tempting). Let's keep attributes and key/value types distinct visually.

@jordalgo this is a good argument and I agree that we should try not to mix different syntax kinds to express semantically similar things.
With that, I'm still in doubt whether key/value types are so distinct from other map attributes to get a different syntax (<> vs named attributes in []).
My arguments:

Map key/value types are not necessarily decoupled from the map type. For instance for array maps, you can't choose the key type, it's always u32 so we'd only want the user to specify the value type. Other map types (e.g. bloom filter) do not even have keys, only values. So, changing the map type (via attributes) inherently influences the key/value definition.
Other map attributes are not pairwise independent either. As @ajor mentioned, for an array map, we'll probably want the user to always specify the map size (max_entries) so adding [type = ArrayMap] means that the user must give [max_entries = ...] as well.
I'm not even sure if we want to force users to always give both the key and value types at the declaration either. You may want to just define the value type and let bpftrace auto-deduce the key type for you.

So, I think the important question here is: what makes key/value types so special that they should have a different syntax from the rest of the map attributes? At the moment, I don't really see a reason to do that. I admit that it feels like key/value specify the map "type" so they should use the same syntax as scratch variable types but as seen above, the rest of the attributes are a part of the map "type" as well. So, I'd probably suggest to settle on a single syntax for all map attributes.
With the attributes syntax (which seems a bit preferred here), the above @jordalgo's examples would become:
 [type = LRUHashMap]
 [key = int; value = uint64]
 [max_entries = 32]
 let @x;

 [type = LRUHashMap]
 [key = int8; value = uint64]
 let @z;

 [type = LRUHashMap]
 let @y;

 [key = int8; value = uint64]
 let @v;

 let @u;

Also, these would give semantic analyser errors:
[type = ArrayMap]
[key = int8; value = int64]
[max_entries = 8]
let @x;  // cannot override key type for ArrayMap

[type = ArrayMap]
[value = int64]
let @x;  // ArrayMap requires the map_entries attribute

@viktormalik In light of those restrictions on the map key and value types, I agree with your suggestion of making everything an "attribute" with a single syntax. It's simple, it's readable, it easily allows every attribute to optional, AND this syntax will also work for probe attributes.
Small follow-up question: Should we restrict attributes to a single one per line (no semi-colon to divide e.g. [key = int; value = uint64]). I find it's easier to read with just one per line (might also be easier to parse) but maybe that's too restrictive. e.g.
[type = LRUHashMap]
[key = int]
[value = uint64]
[max_entries = 32]
let @x;

I find it too restrictive and too space-consuming. For instance, the typical use-case would be to define an array map and this:
[type = ArrayMap; value = uint64; max_entries = 8]
let @x;

IMHO reads just fine and doesn't consume much vertical space.

I don't really like mixing the type info into the attributes

it doesn't seem consistent with how let does types for normal vars
its a lot of additional typing to do
the info doesn't seem relevant to the type system
I don't see how you can make it work with automatic type detection. E.g. what if you only want to set max_entries on your array, do you then also have to specify

What if instead of types and attributes, maps have an initializer
let @: Bloomfilter<i32> = Bloomfilter(k=10)
let @: LRUHashMap<i8, i8> = LRUHashMap(max=10)

There might be some nice tricks to automatically determine the type, making the type signature itself optional

the info doesn't seem relevant to the type system

Yeah, I thought so too. This is why I had originally grouped these into two separate buckets: types for type system (key/value) and other "attributes" but given what @viktormalik and @ajor brought up, it feels like this separation isn't clean e.g. I think this syntax is very confusing: let @: Bloomfilter<i32> - is it the key type? is it the value type?

What if instead of types and attributes, maps have an initializer
let @: LRUHashMap<i8, i8> = LRUHashMap(max=10)

This doesn't look bad but I don't like that it closes the door on actual map initialization (e.g. if we wanted to insert keys and values at initialization, how would that look?) And to restate it's not consistent if/when we add probe attributes (https://github.com/bpftrace/bpftrace/issues/3360) - I think it would be really nice if we had one syntax for map and probe attributes.

the info doesn't seem relevant to the type system

Yeah, I thought so too. This is why I had originally grouped these into two separate buckets: types for type system (key/value) and other "attributes" but given what @viktormalik and @ajor brought up, it feels like this separation isn't clean e.g. I think this syntax is very confusing: let @: Bloomfilter<i32> - is it the key type? is it the value type?

A bloom filter doesn't have keys, right? The fact that everything is a map is just a bit confusing

What if instead of types and attributes, maps have an initializer
let @: LRUHashMap<i8, i8> = LRUHashMap(max=10)

This doesn't look bad but I don't like that it closes the door on actual map initialization (e.g. if we wanted to insert keys and values at initialization, how would that look?) And to restate it's not consistent if/when we add probe attributes (#3360) - I think it would be really nice if we had one syntax for map and probe attributes.

Hmm I thought map initialization would still happen in begin probes, if needed. Although I agree that having let @ = somefn() in the 'global' bit also looks a bit weird and does imply that let @ = [1,2] should also work.

I don't see how you can make it work with automatic type detection. E.g. what if you only want to set max_entries on your array, do you then also have to specify

Can you elaborate on this?

Say you have a program like:
[max_entries=8]
let @a

kprobe:bla {
   @a[tid] = 1
}

In this case you need to run some kind of type inference before you validate the attribute, as the attribute won't work on any type.
but in
[type=hash,max_entries=8]
let @a

kprobe:bla {
   @a[tid] = 1
}

You need the attribute to be able to do type checking.
My worries is that it creates a lot of interdependencies between passes (or it becomes the type and attribute pass). Or you need ordering where you have passes that 1. extract type info from attributes, 2. run the type inference, 3. full attribute validation

This doesn't look bad but I don't like that it closes the door on actual map initialization (e.g. if we wanted to insert keys and values at initialization, how would that look?) And to restate it's not consistent if/when we add probe attributes (#3360) - I think it would be really nice if we had one syntax for map and probe attributes.

I think that we should clarify one thing here. Map type (hash, array, ...) and max_entries are not attributes, they are a fundamental part of the map's type. It's the same as for "normal" compound types in programming languages - for instance in C, the array size is the part of the variable's type, not just some optional attribute. The fact that some of this information is optional in bpftrace (b/c it has a default value) doesn't make it decoupled from the underlying map type.
After giving it some thought, I'm starting to lean back towards using the map type in front of the angle bracket. Here's my reasoning:

the map type is the fundamental information which must be defined for every BPF map,
the map type specifies what other fields (attributes if you will) must be set on the map type - key/value type, max entries, additional, flags, etc.
therefore, the MapType<...> syntax actually makes sense - MapType defines the actual data type (i.e. the layout of the memory storage for this map), the rest in <...> just defines additional attributes of that type.

In fact, it would be quite analogous (both syntax- and semantics-wise) to C++ template types - in std::vector<int>, std::vector says how this is stored in memory (contiguous sequence of elements) and <int> says what's the size of each element.
So I'd say that we should try to squeeze all the information into the angle brackets, although I agree that it won't be too nice when there will be many fields set. OTOH, I don't expect many users to specify all kinds of fields, most of them will use default values for most.
Some of the above examples would be:
// These two are equivalent
let @: Bloomfilter<int32, 10>;
let @: Bloomfilter<value=int32, max_entries=10>;

// Same here
let @: LRUHashMap<int8, int8, 10>;
let @: LRUHashMap<key=int8, value=int8, max_entries=10>;

For the default map type (HashMap), you could omit the first part and just give key/value types pair (which is IMO a typical use-case) so the declaration would be in line with scratch variable declarations.
let @: <int8, int8>;

A bloom filter doesn't have keys, right? The fact that everything is a map is just a bit confusing

Yes, this is a great point. We call everything a "map" b/c kernel does the same but many of these just aren't maps. Bloom filter is one example but one could also argue the array map is not a map, it's an array (with standard indexation by an integer).
We may want to consider changing our docs to say that values starting with @ are "global values/variables". It would also prevent some confusion like why scalar globals (@x = 1) are referred to as maps or why bloom filter (once we support it) doesn't have keys.

I think that we should clarify one thing here. Map type (hash, array, ...) and max_entries are not attributes, they are a fundamental part of the map's type. It's the same as for "normal" compound types in programming languages - for instance in C, the array size is the part of the variable's type, not just some optional attribute. The fact that some of this information is optional in bpftrace (b/c it has a default value) doesn't make it decoupled from the underlying map type.

Hmm, I guess arrays are a bit of a special case as pre-allocated static size arrays are possible, which can be useful into for static checking. For a hash or bloom filter this info seems a lot less relevant to the type system. E.g. for a bloom filter you'd specify the number of items and the acceptable false positive rate, but thats not  relevant to the type of data you put in it, which is the bit the type checking cares about.

We may want to consider changing our docs to say that values starting with @ are "global values/variables". It would also prevent some confusion like why scalar globals (@x = 1) are referred to as maps or why bloom filter (once we support it) doesn't have keys.

+1

My worries is that it creates a lot of interdependencies between passes (or it becomes the type and attribute pass). Or you need ordering where you have passes that 1. extract type info from attributes, 2. run the type inference, 3. full attribute validation

I feel like we're kinda in that boat already with all the back type propagation stuff we need e.g. https://github.com/bpftrace/bpftrace/pull/3536 - I'm not saying that's a good reason but inferring the map "type" based on how it's used seems inline with the flexibility we already provide to users.

We may want to consider changing our docs to say that values starting with @ are "global values/variables".

I'm for this idea. It's always been the source of a lot of user confusion. And maybe we can update the error messages to stop calling things "scalar maps".
As far as this syntax goes:
// These two are equivalent
let @: Bloomfilter<int32, 10>;
let @: Bloomfilter<value=int32, max_entries=10>;

For me the only way to NOT make this super confusing to users is to force named arguments and not let anything be positional. There is no way I wouldn't have to look at the docs every time I saw this: let @: Bloomfilter<int32, 10>;. The mixing of types and values without being explicit makes my eyes water like I'm peeling an onion 😿
I think this restriction is ok though because other than users saying they want to set this global to be a Bloomfilter /LRUHashMap we haven't gotten any (IIRC) requests to be able to state the key/value types up front. So the majority of the time we'll probably only see let @: Bloomfilter; (we don't need the angle brackets) or let @: Bloomfilter<max_entries=10>;. Type inference for the keys (if they exist) and values should still work as normal.
Forcing named attributes also makes the multi-line definitions a little less hideous (though I still don't like it):
let @z : PercpuHashMap<
    key=int8,
    value=uint64,
    max_entries = 64,
    other_attr = "whatever",
>;

And to keep sounding like a broken record, the above syntax (whether or not we are considering them "attributes" or fundamental parts of the map's type) will be different from the syntax we provide for probe "attributes". I don't love forcing users to learn both even though their use is going to be minimal.

For me the only way to NOT make this super confusing to users is to force named arguments and not let anything be positional.

Sure, no problem with that on my side. As we already agreed, terseness is not a target for this syntax so requiring explicit names for attributes sounds good to me.

Forcing named attributes also makes the multi-line definitions a little less hideous (though I still don't like it):
let @z : PercpuHashMap<
    key=int8,
    value=uint64,
    max_entries = 64,
    other_attr = "whatever",
>;

I'm not saying it looks nice but it's not much worse than what we had with many attributes above. And, as you already mentioned, we don't expect users to set this many attributes often.

And to keep sounding like a broken record, the above syntax (whether or not we are considering them "attributes" or fundamental parts of the map's type) will be different from the syntax we provide for probe "attributes". I don't love forcing users to learn both even though their use is going to be minimal.

I don't think we have too many good options here. I feel like we all agree that at least the key/value types should be set the same way as scratch variable types which means that we're introducing a new syntax (<>) anyways. Therefore, I think that we should put everything related to the map type into the same place b/c in my opinion (and this also addresses @fbs comment above) the map type should not just be relevant to bpftrace type system but it should also be relevant to users to understand (and influence) how the map is stored in memory. And the number of entries is a part of that.
With that, I still see space for adding attributes to maps in future. For instance, we may have an attribute which influences how integers are printed for that particular map when using print (in decimal or hex format).

Ok, it feels like we're converging around this syntax (correct me if I'm wrong):
let @x: BloomFilter;

let @y: PercpuHashMap<max_entries = 64>;

let @z : PercpuHashMap<
    key=int8,
    value=uint64,
    max_entries = 64,
    other_attr = "whatever",
>;

And just to summarize a bit why:

It's consistent with scratch variable declarations (as per @JakeHillion and @fbs point above)
It still leaves the door open for global "attributes" (e.g. how to print or sort perhaps) that would be consistent with potential probe "attributes"
Everything to the right of the colon is part of the global variable's definition or "type" (as per @viktormalik 's point)
It leaves the door open for = {}; initialization
The syntax makes it immediately clear what kind of global variable it is: BloomFilter, HashMap, etc. and we can use that when validating the use of it in the script

Please continue commenting with concerns/suggestions. Just thought it was good to get a quick summary of where we, hopefully, are.

yes seems reasonable. A fairly common way of doing this.
Will the names be optional? I.e. is HashMap<int8, int8> equal to HashMap<key=int8, value=int8>?
I'm not yet convinced about having max_entries there. To me thats a probe attribute of sorts and not related to the type but do like that you dont have to slap on on a mixture of attributes and type hints.

Will the names be optional?

No. All names are required so this would not be valid: HashMap<int8, int8> or this let @: Bloomfilter<int32, 10>;.

I'm not yet convinced about having max_entries there.

I think @viktormalik made a good point that in C the number of elements is part of the type syntax e.g. int myArr[20].

I'm not yet convinced about having max_entries there.

I think @viktormalik made a good point that in C the number of elements is part of the type syntax e.g. int myArr[20].

To me thats a special case that as knowing the array size at compile time can help as you can allocate a fixed block of stack space (could also argue that int[] arr[8] would've make more sense for C). Knowing the amount of entries in a hashmap seems a lot less relevant and brings a lot of runtime info into the type system.
It also makes the type signatures very complex, a combination of attributes and types seems like a better fit to me (or more object like initialization)
If we do this:
[max_entries=1000]
let @map: Bloomfilter<int64>

The types and other attributes are clearly separated. It also keeps the type signature simple to type, don't have to add value=int64 , but that can be supported

High level
This feature is big enough of a carrot and wide enough of a design space to consider marking this as an unstable feature (https://github.com/bpftrace/bpftrace/issues/3521). We've had enough demand for this that users will go out of their way to deal with unstable-ness. We've seen this in practice with big strings recently.
Marking as unstable lets us collect feedback / gain experience and continue discussion. The bar for stabilizing this feature should be at least as high as an upfront design discussion.
Design thoughts
Based on brief discussion in https://github.com/bpftrace/bpftrace/issues/3360, my feeling was that attributes are a way to specialize behavior. So I was thinking about attributes more in the context of specializing maps to percpu, arraymap, lru, etc. Bloom filter is a good example but a borderline case. In theory existing map syntax can work with bloom filter (although in practice the bpf-side helpers are different, see https://docs.kernel.org/bpf/map_bloom_filter.html). Obviously we are not going to support arena, structops, prog_array, etc. with this syntax, right? For the latter, I expect we'd come up with a different API.
When I think of map attributes as a specialization, attributes make a lot of sense to me. Composable, optional, and explicit. It also fits in nicely with probe-level attributes. Less syntax is better - every additional construct is a new "weird" thing for user to learn. Better to reuse existing constructs where we can.
IOW, I don't see this as much of a typing decision. Bringing the scope of this design to the level of typing complicates things. And I don't see it as necessary.

In theory existing map syntax can work with bloom filter ... Obviously we are not going to support arena, structops, prog_array, etc. with this syntax, right?

@danobi  I think one of the things that came out of this discussion (and please feel free to disagree) is that the @ sigil is used to describe "globals" and not necessarily just maps, which has maybe caused the scope of this discussion to grow e.g. the discussion of bloom filter.
It does seem like we're having a hard time agreeing on where the line is between what is an "attribute" and part of the global's "type".
If we carry the idea further that @ describes a global then I think, to keep consistent with scratch variable type declarations (as well as future function arguments), we need to support some kind of "type" to the right of these globals e.g.
let @a: LRUHashMap;
let @b: BloomFilter;
let @c: Arena;

This feels to me like a good baseline. Now the question becomes, where do we put the other information about these globals.
It also seems, from the discussion above, that we're not considering NOT having (sorry for the double negative) the "attribute" syntax e.g. [myattr = whatever] as this will be used by probes as well. So that leaves us with this:
[myattr = whatever]
let @a: LRUHashMap;

So the question then becomes: where do we put the inner "type" information about this global? TBH, I'm not sure if I care at this point because I don't think we have a good use-case for users actually utilizing this feature. It would mostly be for further communication to the reader of a script as we can infer the inner type information as normal - and so far we haven't had much of a problem with type ambiguity.
My vote at the moment, though it seems to change daily, is to just allow users to specify the key and value types for map globals in the attribute syntax. This more-or-less allows us to put off the decision to add an additional angel bracket syntax. Even if we decide to add one later, we could always still support setting the "type" in the attribute syntax. I don't think it costs us anything. So...
[myattr = whatever]
[key = int64]
[value = string]
let @a: LRUHashMap;

Marking as unstable lets us collect feedback / gain experience and continue discussion. The bar for stabilizing this feature should be at least as high as an upfront design discussion.

Also totally good for marking this feature as unstable/experimental.

In theory existing map syntax can work with bloom filter ... Obviously we are not going to support arena, structops, prog_array, etc. with this syntax, right?

@danobi I think one of the things that came out of this discussion (and please feel free to disagree) is that the @ sigil is used to describe "globals" and not necessarily just maps, which has maybe caused the scope of this discussion to grow e.g. the discussion of bloom filter.

Hmm, I'd be careful there. @ is more than just scoping - it also enables kev/value semantics. ie @[key] = val. Then one could wonder why they can't do $scratch[key] = val.
Not saying it shouldn't be considered. But I recommend spinning that off into its own discussion.

It does seem like we're having a hard time agreeing on where the line is between what is an "attribute" and part of the global's "type".
If we carry the idea further that @ describes a global then I think, to keep consistent with scratch variable type declarations (as well as future function arguments), we need to support some kind of "type" to the right of these globals e.g.
let @a: LRUHashMap;
let @b: BloomFilter;
let @c: Arena;

This feels to me like a good baseline. Now the question becomes, where do we put the other information about these globals.

Thanks for laying this out. I had a good sleep on this and have some new thoughts. Looks like I didn't think enough and what I was proposing would just complicate things in the future.
Originally I was hoping we could get away with the minimal amount of changes. For completeness, here's what I had in mind:
[key_type = int64]
[value_type = int64]
[map_type = percpu]
@[$foo] = $bar;

So skip let stuff altogether but not rule it out for future.
But then I had the thought that it'll be important that users can pass a map to a user defined function. In that case, we need the map_type/key_type/value_type information in the type system in order for UDF to be generic across maps. This will be crucial for a bpftrace standard library as well as the C plugin mechanism. For example, we'll want to be able to do this:
fn top_n(@m: PercpuHash<i64, i64>, $n: int64) {
  // do stuff with @m
  ...
}

It's important to top_n that it knows @m is percpu as well as what the key/value types are. So that different maps can be passed into the same function. But it's not important it knows max_entries, numa affinity, etc. So basically what @fbs is proposing earlier.

It also seems, from the discussion above, that we're not considering NOT having (sorry for the double negative) the "attribute" syntax e.g. [myattr = whatever] as this will be used by probes as well. So that leaves us with this:
[myattr = whatever]
let @a: LRUHashMap;

So the question then becomes: where do we put the inner "type" information about this global? TBH, I'm not sure if I care at this point because I don't think we have a good use-case for users actually utilizing this feature. It would mostly be for further communication to the reader of a script as we can infer the inner type information as normal - and so far we haven't had much of a problem with type ambiguity.

Previously agreed. But see UDF example above.

. In that case, we need the map_type/key_type/value_type information in the type system in order for UDF to be generic across maps.

Very good point and sad I missed it myself lol.

So skip let stuff altogether but not rule it out for future.

Don't agree with this. We added let after a lot of discussion and though it seems a bit superfluous in this case, it's an indication of declaration be they scratch variables, globals, maps, whatever. I think it's inconsistent if we don't use let no matter what we're calling things defined with @.

One other consideration: if we implement map-in-map (https://github.com/bpftrace/bpftrace/issues/3553), we'll need a way to express inner map type as well. So having the key/value argument be present in the type name would ensure map-in-map can be captured in type system. For example:
[myattr = whatever]
let @m: Hash<i64, Hash<i64, i64>>;

. In that case, we need the map_type/key_type/value_type information in the type system in order for UDF to be generic across maps.

Very good point and sad I missed it myself lol.

Yeah, completely forgot about user-defined functions.

It's important to top_n that it knows @m is percpu as well as what the key/value types are. So that different maps can be passed into the same function.

Yes, this looks like a reasonable distinction point - whatever information is needed to be passed to functions must be a part of the map type, the rest can be attributes.

But it's not important it knows max_entries, numa affinity, etc.

Isn't it? I'm especially looking into ArrayMap here. I'm not sure, but I'd expect that the verifier checks bounds when accessing the map so it would fail on OOB access. Since we don't want to expose users to verifier errors, we'll probably need to make max_entries mandatory and therefore somehow pass it to UDF.

One other consideration: if we implement map-in-map (#3553), we'll need a way to express inner map type as well. So having the key/value argument be present in the type name would ensure map-in-map can be captured in type system. For example:
[myattr = whatever]
let @m: Hash<i64, Hash<i64, i64>>;

Hmm, can the outer and inner maps have different attributes (like max_entries)? If yes, then this syntax won't allow to express them.

Isn't it? I'm especially looking into ArrayMap here. I'm not sure, but I'd expect that the verifier checks bounds when accessing the map so it would fail on OOB access. Since we don't want to expose users to verifier errors, we'll probably need to make max_entries mandatory and therefore somehow pass it to UDF.

But array maps still use bpf_map_elem_lookup() API, so the bounds check is done there. Accessing out of bounds results in a zero-d out value.

Hmm, can the outer and inner maps have different attributes (like max_entries)? If yes, then this syntax won't allow to express them.

Yeah, good point. Unless we added inner_ prefix to attributes. Like inner_map_map_entries. Kinda ugly. Doesn't fully solve arbitrary map-in-map nesting if kernel ever supports that. No other ideas from me here.

Yeah, good point. Unless we added inner_ prefix to attributes. Like inner_map_map_entries. Kinda ugly. Doesn't fully solve arbitrary map-in-map nesting if kernel ever supports that. No other ideas from me here.

I also was thinking of some named attribute like this (I agree it's ugly). I also don't see UDFs needing to specify the max_entries for a map type signature e.g. fn(@mp: Hash<i64, Hash<i64, i64>>;): int64. Another possibility is to force map definitions to only be single level if you want/need to add attributes.
[myattr = whatever]
let type MyInnerMap = Hash<int64, int64>;

[myattr = whatever]
let @b: Hash<i64, MyInnerMap>;

But array maps still use bpf_map_elem_lookup() API, so the bounds check is done there. Accessing out of bounds results in a zero-d out value.

Right, forgot about that. Then it's ok and we don't need to pass the size to functions.

I also was thinking of some named attribute like this (I agree it's ugly). I also don't see UDFs needing to specify the max_entries for a map type signature e.g. fn(@mp: Hash<i64, Hash<i64, i64>>;): int64. Another possibility is to force map definitions to only be single level if you want/need to add attributes.
[myattr = whatever]
let type MyInnerMap = Hash<int64, int64>;

[myattr = whatever]
let @b: Hash<i64, MyInnerMap>;

This is a good idea. I don't think adding attributes would be used very often (especially for nested maps) and this provides a consistent way to specify them for arbitrary map nesting.
It looks like you've convinced me - as long as the map types are specified as Hash<int64, int64>, I'm ok with specifying the rest of the fields via attributes.

Ok folks, just to summarize from office hours and the above discussion. We're converging on this syntax. Additionally we will be putting this functionality behind a feature gate (https://github.com/bpftrace/bpftrace/issues/3521)
All of these below will be valid:
let @a;

let @b: Hash<u64, u64>;

[myattr = whatever]
let @c: Hash<u64, u64>;

And when we add user-defined types and maps in maps:
[myattr = whatever]
type MyInnerMap = Hash<int64, int64>;

[myattr = whatever]
let @b: Hash<i64, MyInnerMap>;

Additionally, we need to figure out what BPF map types are valid and what types can be used with what internal and external types.
Please thumbs up if you agree with this decision and thumbs down if you don't (but please add reasons why not in further comments below).

Quick maintainers roll call. If you disagree with any part of the above proposal, please comment soon as I plan to start development work in the next few weeks. @fbs @ajor @danobi @viktormalik

Declaration syntax aside, was there closure on where the statements are placed? Just as a datapoint, the idea that these lets exist at the global scope (alongside probes) was non-obvious to me, and I guess I'm just wondering why they couldn't be used in any probe (mirroring how maps are already "declared" implicitly). This would simplify one-liners (since the let would appear inline in the single probe), and you could just have a convention to declare all maps up front in the BEGIN probe for complex programs.
Then again, an argument in favor of having the statement at the top-level would be if there as also an intent to allow global variables of non-map types (which could be implemented using a suitable lift into a map for the set of scalar globals), which would fit naturally with this. I would probably try a global let $x as soon as I discovered the global-level map variables. It's worth noting that this does open another can of worms, as I also want to be able to declare probe-local maps (shared state just between invocations of the probe, but not within the global namespace) ;) Is there a sense for this roadmap?

was there closure on where the statements are placed

@amscanne Thanks for following up on this. In my mind, when using the let syntax you are declaring a scope that this variable is valid in. Since @ variables are not reset/recreated during every probe entry, utilizing the let syntax to define the their "type" and additional attributes should happen at the global level. This doesn't prevent users from just using globals as they have been - that being said maybe we should think about allowing globals to specify their "type" inside a probe for the one liner case (as you mentioned) e.g.
BEGIN { @a: LRUHash = 1; }

if there as also an intent to allow global variables of non-map types

We've talked about this case and that's why I'm careful to say "global" above and not map 😉

as I also want to be able to declare probe-local maps (shared state just between invocations of the probe, but not within the global namespace)

What is the use case for this? Why couldn't the script just not use/reference said global in other probes?

What is the use case for this? Why couldn't the script just not use/reference said global in other probes?

I think it's pretty easy to contrive scenarios where implicit globals lead to paper cuts. Any case where you think you're making a local change, but you're actually affecting something globally. Suppose that I had two probes, X and Y, and maybe they tracked some invocation time using @since and @last. Maybe I go in and "clean up" X, and change the map used to @last. Now I'm scratching my head wondering why X and Y are producing occasionally garbage results (they are both using the same map now, but I only changed X!). Having an explicit let @last in each probe would make it clear that it was only local to that specific probe, and changes to naming there don't affect any other probes implicitly.
I don't think this is urgent or anything, I'm just curious about the long-term trajectory. Having globals/locals be dependent on the scope, and maps/scalar values a function of the type (with both supported in both scopes) makes a ton of sense, and just that today you have only global maps, local scalars. Any implicit declarations would continue to default to those scopes for backwards compatibility, but as programs grow you encourage explicit declarations (presumably).

The issue I see with allowing persistent maps (or anything) local to only specific probes is perhaps in the declaration - it feels confusing that a data structure is persisting even though every time the program is called (like a function) you are re-declaring it with let, which says to me this variable is new/empty at this point forward. I'm not opposed to the concept of probe-specific persistent data structures, I just don't think the let syntax is suited for them. Maybe having file-local maps/globals when we start allowing imports of other bpftrace scripts is a good first start OR we could have an attribute that allows you to specify in which probes it can be accessed (though that feels too heavy weight).


fix donate link page

Current Pallets Project Donate Link is broken atm.
Clicking on it gives you "Page not found"

@davidism pinging as projects can miss donations :(

This change isn't correct, that link is supposed to redirect to the psf donation page. @jab can you look at netlify

Seems to be working again, maybe a temporary issue with CloudFlare (the redirect is configured there). https://palletsprojects.com/donate should go to a psfmember.org donation page.

👍🏽


problem with datetime_groupby_apply()

I called ProcessInf and encountered an error: _pickle.PicklingError: Could not pickle the task to send it to the workers.
It seems to be a problem of joblib and my joblib version is 1.1.0
https://github.com/microsoft/qlib/blob/84103c7d43eaa0ff74118a4d05884f659f0548eb/qlib/data/dataset/processor.py#L175
Any suggestion? Thanks!

Which script is being run?
What is the python environment: python scripts/collect_info.py all

Which script is being run? What is the python environment: python scripts/collect_info.py all

I wasn't running a script.  In the dataset initialization step, I use a processor named ProcessInf  to clean the inf in the dataset and the error happened.
Below please find the python environment:
Windows
AMD64
Windows-10-10.0.19041-SP0
10.0.19041
Python version: 3.7.1 (default, Dec 10 2018, 22:54:23) [MSC v.1915 64 bit (AMD64)]
Qlib version: <IP_ADDRESS>
numpy==1.19.4
pandas==1.3.1
scipy==1.7.1
requests==2.25.1
sacred==0.8.2
python-socketio==3.1.2
redis==3.5.3
python-redis-lock==3.7.0
schedule==0.6.0
cvxpy==1.0.21
hyperopt==0.1.1
fire==0.3.1
statsmodels==0.12.1
xlrd==2.0.1
plotly==4.12.0
matplotlib==3.1.3
tables==3.6.1
pyyaml==5.3.1
mlflow==1.20.2
tqdm==4.56.0
loguru==0.5.3
lightgbm==3.1.1
tornado==5.1.1
joblib==1.1.0
fire==0.3.1
ruamel.yaml==0.16.12

Which script is being run? What is the python environment: python scripts/collect_info.py all

I wasn't running a script. In the dataset initialization step, I use a processor named ProcessInf to clean the inf in the dataset and the error happened.
Below please find the python environment:
Windows AMD64 Windows-10-10.0.19041-SP0 10.0.19041
Python version: 3.7.1 (default, Dec 10 2018, 22:54:23) [MSC v.1915 64 bit (AMD64)]
Qlib version: <IP_ADDRESS> numpy==1.19.4 pandas==1.3.1 scipy==1.7.1 requests==2.25.1 sacred==0.8.2 python-socketio==3.1.2 redis==3.5.3 python-redis-lock==3.7.0 schedule==0.6.0 cvxpy==1.0.21 hyperopt==0.1.1 fire==0.3.1 statsmodels==0.12.1 xlrd==2.0.1 plotly==4.12.0 matplotlib==3.1.3 tables==3.6.1 pyyaml==5.3.1 mlflow==1.20.2 tqdm==4.56.0 loguru==0.5.3 lightgbm==3.1.1 tornado==5.1.1 joblib==1.1.0 fire==0.3.1 ruamel.yaml==0.16.12

How did you implement it and how should you reproduce this error?


Error when calling Mathf.Triangulate

Nuget: Love2dCS 11.0.42:
Using Windows 32-bit application, .net 4.8, when I call Mathf.Triangluate, I get the following:
Managed Debugging Assistant 'FatalExecutionEngineError'
  Message=Managed Debugging Assistant 'FatalExecutionEngineError' : 'The runtime has encountered a fatal error. The address of the error was at 0x746a86b5, on thread 0x119c. The error code is 0xc0000005. This error may be a bug in the CLR or in the unsafe or non-verifiable portions of user code. Common sources of this bug include user marshaling errors for COM-interop or PInvoke, which may corrupt the stack.'

Running as 64-bit application I get this:
System.AccessViolationException
  HResult=0x80004003
  Message=Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
  Source=<Cannot evaluate the exception source>
  StackTrace:
<Cannot evaluate the exception stack trace>

In case some context is needed, here where I am doing the call:
        public Polygon(float[] points)
        {
            _points = points;

            //put points into a vector structure as required by Triangulate
            var pointsAsVectors = new Vector2[points.Length/2];
            var j = 0;
            for (var i = 0; i < points.Length; i+= 2)
            {
                pointsAsVectors[j].X = points[i];
                pointsAsVectors[j].Y = points[i + 1];
                j++;
            }

            var triangles = Mathf.Triangulate(pointsAsVectors);
            foreach (var tri in triangles)
            {
                var trianglePoints = new float[6];
                trianglePoints[0] = tri.a.X;
                trianglePoints[1] = tri.a.Y;
                trianglePoints[2] = tri.b.X;
                trianglePoints[3] = tri.b.Y;
                trianglePoints[4] = tri.c.X;
                trianglePoints[5] = tri.c.Y;
                _triangles.Add(trianglePoints);
            }
        }

Thanks for your report!
I had reproduce your problem on my pc. i will fix that bug.

The bug on Mathf.Triangulate has fixed in version 11.0.43 .

Thanks for the super-fast response!  I got the new version and it works great!


Initial commit

closes #1

PR-Codex overview
This PR updates various files in the project, including the application.yaml, Maven and Codecov workflows, banner.txt, LICENSE, HubApplicationEntry, and other files. It also adds checkstyle suppression, updates the README.md, and adds MIT License to package-info.java and HubApplicationEntryTest.java.
Detailed summary

Updates email in application.yaml
Updates Java versions in Maven workflow
Adds Codecov token in Codecov workflow
Updates banner.txt with application title and Spring Boot version
Updates LICENSE.txt with new copyright holder name
Adds MIT License to package-info.java and HubApplicationEntryTest.java
Adds checkstyle suppression
Updates README.md with corrected formatting and tool descriptions
Updates HubApplicationEntry.java with MIT License and entry point method

The following files were skipped due to too many changes: pom.xml

✨ Ask PR-Codex anything about this PR by commenting with /codex {your question}

@rultor merge

@rultor merge

@l3r8yj OK, I'll try to merge now. You can check the progress of the merge here

@rultor merge

@l3r8yj Done! FYI, the full log is here (took me 1min)


Data Preparation Tool

A Stand-alone browser-based tool to help process data for the training set

Great job! I will review and begin to test now.

It doesn't need webserver, can run on Google Chrome locally by double clicking the HTML file. At moment, Chinese/Korean etc. the logogram based language are not supported, as the JS separator is simply using space. Haven't found a robust JavaScript tokenizer to support logogram language.

I just emailed you to your exeter.ac.uk mailbox, asked couple of questions. looking forward to you reply. Thanks,


Add the Chat Completions functionality to the OpenAI Client

Currently we only support operations for generating embeddings, we need to add the ability to get chat completions as well given a prompt and also some chat history

This will also need solid testing for mocking the http requests. Will also need to openAIClient to split off into separate files as it is already quite large

ideally we want them to be able to set additional config options on the client. That way any fancy stuff we do where they pass an impl of AsyncChatClient can be avoided


added feature to tilt the cover

add feature to set the tilt position for raffstore
small changes don't work properly
had to use time.sleep and a normal open/close + stop message.
while adjusting the tilt you should not request a new tilt position
I used a blind as device_class to get an other default symbol in HA

Thanks a lot for your contribute. Unfortunately, I'm working in parallel on something else and don't have the environment yet to test it properly. When this is available I will definitely merge it in.
Would it help of you to put it in between into a separated branch?

@chriseberlein, can you please check if it is still working. I need to change quite a lot to make the tests run through.
Thanks for the contribution! When we have made this straight I will merge it to main and make a release.

Hei! Your commit from the 9. 7 is working well. With the current feature-branch I get errors.


No Layout Files Found for "HTML"

Describe the bug
Since the release yesterday I have been getting the below error when attempting to serve locally:
WARN  found no layout file for "html" for kind "taxonomy": You should create a template file which matches Hugo Layouts Lookup Rules for this combination.
This warning happens for kinds of: "taxonomy, home, term, section, page.
WARN  found no layout file for "json" for kind "home": You should create a template file which matches Hugo Layouts Lookup Rules for this combination.
This warning happens only for the JSON kind of Home.
To Reproduce

hugo serve -D

Expected behavior
Website to run locally on localhost:1313
Screenshots

Desktop (please complete the following information):

OS: Mac
Browser: Chrome

Hugo & Blowfish versions

Version: Hugo v0.120.4

Hey @Lachlan-White is this still happening?

Hey @nunocoracao Yes it is, replicated on other devices and other users also

Can you share a link to a reproducible example for us to investigate?

I think I am getting this same error with a site I put together today. I switched from one machine to another and noticed on the second machine after I pulled down the repo from GitHub. I didn't seem to have the issue on my first machine oddly.
Repo here
PS C:\blah\blah\blah\greek-sweets> hugo server --noHTTPCache
Watching for changes in C:\blah\blah\blah\greek-sweets{archetypes,assets,content,static}
Watching for config changes in C:\blah\blah\blah\greek-sweets\config_default
Start building sites …
hugo v0.131.0-bfbee17932ff24009008aa94cdd75c0c41f59279+extended windows/amd64 BuildDate=2024-08-02T09:03:48Z VendorInfo=gohugoio
WARN  found no layout file for "html" for kind "home": You should create a template file which matches Hugo Layouts Lookup Rules for this combination.
WARN  found no layout file for "html" for kind "taxonomy": You should create a template file which matches Hugo Layouts Lookup Rules for this combination.
WARN  found no layout file for "html" for kind "page": You should create a template file which matches Hugo Layouts Lookup Rules for this combination.
WARN  found no layout file for "html" for kind "section": You should create a template file which matches Hugo Layouts Lookup Rules for this combination.
WARN  found no layout file for "html" for kind "term": You should create a template file which matches Hugo Layouts Lookup Rules for this combination.
WARN  found no layout file for "json" for kind "home": You should create a template file which matches Hugo Layouts Lookup Rules for this combination.


Style Transfer Colab Notebook error

The first part of the image-to-painting-nst.ipynb works fine since this is creating a painting
through brush strokes and then a video.
The second part is the style transfer code, and I am getting an error here:

Hi @metaphorz, thanks for your feedback. It seems your style image isn't loaded correctly. Did you check your arg.style_img_path before loading it?

You are right. I had a mistake in the style file spec.


FSA : GitRepository object fails to clone private github repo.

Hi,
We are facing issues when using FSA with ArgoCD. Upon creating ArgoCD application thru UI with a private github repo, the GitRepository instance shows error that its not able to clone the repo.
How to pass the github credentials in case of FSA when we select to "User Flux subsystem" and "auto-create Flux resources"?

@chanwit fyi.

Thanks @chanwit . closing this ticket for now.


Help: How to pre-render individual layers?

I have multiple layers, one intended for the foreground and one intended for the background. The player need to render between these layers. Is there a way to pre-render individual layers, and subsequently draw individual layers?
Would the simplest way be making new instances of LDtkLevel for each layer?

Just tried making each layer into a separate LDtkLevel, but that will not work since I don't have access to the Identifier property on the level, which is null if a simply make a new instance of LDtkLevel

https://github.com/IrishBruse/LDtkMonogame/blob/feature-ideas/LDtk/Renderer/LDtkRenderer.cs
I think there is a bug on the renderer you can customize it like this and the layers should be rendered separately for custom rendering you have to make a custom ldtk renderer use the one linked as a base.

Ahh I see! I went ahead and added functionality for pre-rendering layers and rendering those pre-rendered layers in the base LDtkRenderer in my own fork. I think it would be nice to have in the built in LDtkRenderer, but if you would like to keep the base one limited to just rendering levels as a whole, I will reimplement it in my own class :)

You dont need to make a fork just make a LDtkRenderer file the one apart of the library is just an example as its not possible for me to know what the user would need or if they want to create a custom effect on each layer like what you want to do. Nothing in the ldtk renderer file is internal

Its mentioned in the docs


[Omnivore] support tagging in quick capture

Extension
https://www.raycast.com/karolusd/omnivore
Description
When doing quick captures it would be amazing to be able to apply tags. e.g. quick capture [link] [tags]
The current workflow is tedious:

quick tag w raycast extension
open omnivore
find the new link
tag it
close omnivore

Who will benefit from this feature?
ME! :D And all omnivore power users.
Anything else?
No response

@geddski thanks for reporting, I will have a look on it.

awesome thank you!

@raycastbot close this issue


[inetstack] Do Not Drop Packets on Inflight Accept

Context
We currently drop packets that arrive before on an inflight accept. Instead, we should buffer some, until we get the ACK on the SYN+ACK. This may happen due to network latencies
Proposed Solution
Buffer some packets that arrive before the ACK on the SYNC+ACK.
Alternative Solutions
Dropping all the packets may hurt performance.

This has been implemented. See PR #1068


switch for multiple home manager versions

it should add a switch for selecting other HM versions

first steps.


[ENH] Refactored series_length and n_instances to n_timepoints and n_cases

Fixes #1153
I have refactored series_length and n_instances to n_timepoints and n_cases in convolution based classifiers
Please let me know if any more changes are required
PR checklist

For all contributions

[ ] I've added myself to the list of contributors. Alternatively, you can use the @all-contributors bot to do this for you.
[x] The PR title starts with either [ENH], [MNT], [DOC], [BUG], [REF], [DEP] or [GOV] indicating whether the PR topic is related to enhancement, maintenance, documentation, bugs, refactoring, deprecation or governance.

@all-contributors add @Raya679 for docs


Determinations not added to containers in "Edit Loan"task

Describe the bug
When I check a container in the "Edit Loan" task in order to add a new determination to it, the determination is not added to the collection objects within the container.
To Reproduce
Steps to reproduce the behavior:

When I go to "Edit Loans" task
Then I check a containerized object.
And I add a determination.
Then I click "Update Determination(s)"
The determinations are not added to Containerized objects.

Containerized object:

Expected determination:

Determination not changed on the containerized object (still has old determination, which is fine):

Expected behavior
Should be able to add determinations to containerized specimens in a loan
Environment (please identify where you experience this bug:
*** One of ***

Production
Chrome

Hey @mjy @jlpereira need this fixed please.


Multiple database support

Feature request
TypeORM supports multiple databases, but typeorm-model-generator only allows for one. This makes it unusable if you have cross-db foreign keys.
This is the error when trying to generate a model from a database that has a foreign key relation to another database:
Relation between tables field and source didn't found entity model source.
Here is the TypeORM reference for multiple connection support:
https://github.com/typeorm/typeorm/blob/master/docs/multiple-connections.md

@Kononnable thanks for this awesome library. please add this feature

Implemented and released in<EMAIL_ADDRESS>

@Kononnable  Thanks for your great work 👏. will try this now


Priority 1 accessibility issue: Duplicate ID value

What

The ID associated with the ‘Close notification’ button is duplicated. Having duplicated ID values can cause anomalous behaviour with assistive technology as they sometimes rely on IDs to utilise certain functionalities.
ID value(s):
id="icon-cross-title"
Solution:
Ensure that ID values are unique to avoid potential anomalous behaviour for users of assistive technology.
Where

URL: https://wellcomecollection.org/whats-on/today Page: What's on | Wellcome Collection
URL: https://wellcomecollection.org/access Page: Accessibility | Wellcome Collection
This issue occurs throughout the website.

unable to find a duplicate id on the page


Sort order no longer working

Expected Behavior
When I open my IDE it should show my previous requests in the order I had them in
Actual Behavior
They opened in a random order and I can't resort them.  The drag and drop as well as the sort button are broken
Reproduction Steps
No response
Is there an existing issue for this?

[X] I have searched the issue tracker for this problem.

Additional Information
Open the IDE and move around some requests/use the sorter and nothing happens
Insomnia Version
2023.5.3
What operating system are you using?
Windows
Operating System Version
windows
Installation method
windows downloader
Last Known Working Insomnia version
2023.4.0

I'm having that same issue. This might be related, the filter input stopped working as well

This is incredibly painful.  All levels of folders and endpoints have had their orders changed and cannot be reordered.
Version: Insomnia 2023.5.3
Build date: 8/16/2023
OS: Windows_NT x64 10.0.22631
Electron: 25.2.0
Node: 18.15.0
Node ABI: 116
V8: <IP_ADDRESS>-electron.0
Architecture: x64

Same here. Not only sort order but the Interface font is no longer applying...
All of my ordered API calls are now all.. over.. the place. Documentation missing parts...
Oh man... I am never turning auto update back on. NEVER. Like.. NEVER. EVER.
Someone please tell me what the last perfectly-fine-should-not-have-been-changed version is.

Same issue for me too.
@suchislife801: bit dramatic don't you think?

No. No I don't think at all. Not at all I don't think. Nope. Not.
I need the version number of the last version that worked.
What is the version number.

Duplicate of #6284

Dude, just read the bottom of the issue above - 2023.4.0

The link. Here is the link to the last working version.
2023.4.0


feat(reaction): add onComplete

Description
Add onComplete event handler for Reaction component.
Links

Added, sorry forgot to push last commit @maxinteger


Provide separate step for uploading all files, ref #453

Note: I'm not completely enamored with this solution because it basically side-steps the problem with a hack. However, I'm not really sure how else to fix the underlying problem because I think it lies with the jbuilder templating system and not with jQuery fileupload plugin. It also has a few hidden benefits, maybe? Uploads are actually occurring faster now, and there's a little more flexibility where the user can add files in multiple batches and only hit "Upload all" once.
This solution injects a new UI behavior where auto-uploading is disabled and a new button is provided to being uploading all the files after their records are individually added to the DOM. (see the circled button)
Although this does add an additional step to the workflow, it fixes the problem and also enables non-sequential, concurrent uploading for increased performance.

ping @cam156 @mtribone @olendorf

Well it may also give people the ability to cancel individual uploads in a "better" way, since I think before it was happening so fast on autoload that you had to be quick with the trigger.  So I'm guessing that a user can upload all OR individually upload items?


Adjusted usage of NSComparisonResult cases to Darwin API

This is a follow-up PR for #356 Changed API of comparison/sort/enumeration/qos options to match Darwin version in swift-corelibs-foundation

Thanks @tomquist! This should be good to go once the Foundation PR goes in 👍

I'll merge both.