bazel: 0.22.0 -> 0.23.2

Motivation for this change
Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[ ] NixOS
[x] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[x] Assured whether relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

there is already https://github.com/NixOS/nixpkgs/pull/58116

there is already #58116

Haha! I should have checked! I know @uri-canva ;)

Ok, I'm going to try again.

@marsam @Profpatsch
I'm looking at this line: https://github.com/NixOS/nixpkgs/pull/58147/files#diff-4746b81564a01e241a84d455575437e7R244
Would that usage of TMP cause sandbox builds to fail?
Locally, I get a different error to hydra. I get the following when running with sandbox = true:
mkdir: cannot create directory '/tmp/.bazel-501': Operation not permitted
When running with sandbox = false it appears to build on Darwin if I create a fake directory /blah with 777 chmod and change that line to use /blah
Hydra timing out after 600 seconds of silence, might be because the compilation takes longer than 10 minutes? I suspect that maybe hydra doesn't have the same TERM or something to see the fancy colored Bazel compilation progress. It takes at least 20-30 minutes to build.

I've hacked the derivation a bit and it's almost working now. Failing at the installCheckPhase.
In file included from examples/cpp/hello-world.cc:1:
./examples/cpp/hello-lib.h:4:10: fatal error: 'string' file not found
#include <string>
         ^~~~~~~~
1 error generated.
INFO: Elapsed time: 34.407s, Critical Path: 0.16s
INFO: 0 processes.
FAILED: Build did NOT complete successfully
//examples/java-native/src/test/java/com/example/myproject:hello      NO STATUS

I'll continue looking at this tomorrow. In the meantime, if anybody can review my changes and suggest better ways to do it, I'm happy to modify things within reason.

Note, my error above is with sandbox = false (I suspect it's a CPP or C toolchain issue).
If I build with sandbox = true I get failed with exit code 134 so I suspect I'm doing something that is banned in sandbox mode.

It might be because you move .bazelrc into bazel_src and then run the tests from outside it, so the tests don't pick up .bazelrc, which is necessary for the bazel build to pick up the right flags from nix.

It might be because you move .bazelrc into bazel_src and then run the tests from outside it, so the tests don't pick up .bazelrc, which is necessary for the bazel build to pick up the right flags from nix.

Good point. I'll check this. I thought that I was copying the .bazelrc along for the ride and then cd ./bazel_src would make it available. I'll try to debug.

Ok, this is now building successfully on my Mac with sandbox = false.
With sandbox = true I still get failed with exit code 134

Exit codes higher than 128 are usually from signals, 134-128 = 6 is SIGABRT. Do you know what command is aborting? Is it nix itself or one of the bazel commands?

Exit codes higher than 128 are usually from signals, 134-128 = 6 is SIGABRT. Do you know what command is aborting? Is it nix itself or one of the bazel commands?

Yes, I suspect the same thing.
Building Bazel from scratch
builder for '/nix/store/14gpkg2y5fg7my3kc5fpjm0qfcbm22vz-bazel-0.23.2.drv' failed with exit code 134
error: build of '/nix/store/14gpkg2y5fg7my3kc5fpjm0qfcbm22vz-bazel-0.23.2.drv' failed```

Patched my build to output tracing. I think this is the reason why I fail to build with sandbox = true.
...
++ test -z /nix/store/9qyzw01bnla43457iwp6l9r6fbp7gk4y-zulu1.8.0_121-<IP_ADDRESS>
++ JAVAC=/nix/store/9qyzw01bnla43457iwp6l9r6fbp7gk4y-zulu1.8.0_121-<IP_ADDRESS>/bin/javac
++ [[ -x /nix/store/9qyzw01bnla43457iwp6l9r6fbp7gk4y-zulu1.8.0_121-<IP_ADDRESS>/bin/javac ]]
+++ /nix/store/9qyzw01bnla43457iwp6l9r6fbp7gk4y-zulu1.8.0_121-<IP_ADDRESS>/bin/javac -version
++ JAVAC_VERSION='dyld: Library not loaded: /System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
  Referenced from: /nix/store/9qyzw01bnla43457iwp6l9r6fbp7gk4y-zulu1.8.0_121-<IP_ADDRESS>/bin/javac
  Reason: no suitable image found.  Did find:
        /System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa: file system sandbox blocked stat()
        /System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa: file system sandbox blocked stat()'
+ run_atexit_handlers 134
+ local exit_code=134
...

@GrahamcOfBorg build bazel

Darwin build is timing out.

There's headless versions of jdk11 and jre for Linux. If we could have headless versions of jdk8 for macOS, then the build wouldn't attempt to access Cocoa and it might work. Did something change recently with how sandboxing is enforced? I think system libraries on macOS were still being access in non-pure fashion until a couple of months ago.

@Profpatsch

Darwin build is timing out.

I see that. What would you suggest now? This is the same as others have seen trying to get a new version of Bazel over the line. How can we get more build output? On my own Mac, I see a lot more output.
Given the local sandbox failure loading Cocoa, I think it's possible that it never built successfully on Darwin with sandbox = true or the sandboxing mechanisms have changed recently.
I'll keep digging, but it's looking like this might have to be changed to a linux only pkg if sandboxed Darwin builds are a blocker.

Quick post to summarise the difference between failing sandbox builds on a Mac vs on Hydra.
Mac (Tracing disabled. The underlying failure is due to a javac call reaching outside the sandbox for Cocoa libraries):
fixing libtool script ./third_party/protobuf/3.6.1/ltmain.sh
fixing libtool script ./third_party/protobuf/3.6.1/third_party/googletest/googletest/build-aux/ltmain.sh
fixing libtool script ./third_party/protobuf/3.6.1/third_party/googletest/googlemock/build-aux/ltmain.sh
no configure script, doing nothing
preBuildPhase
building
Building Bazel from scratch
builder for '/nix/store/aqg0l592dmr8ny5xgiiv3k3vbgdxg0lz-bazel-0.23.2.drv' failed with exit code 134
error: build of '/nix/store/aqg0l592dmr8ny5xgiiv3k3vbgdxg0lz-bazel-0.23.2.drv' failed

Hydra (https://logs.nix.ci/?key=nixos/nixpkgs.58147&attempt_id=2a30e2bb-eabb-41f2-8786-a53b3be2509a):
fixing libtool script ./third_party/protobuf/3.6.1/ltmain.sh
fixing libtool script ./third_party/protobuf/3.6.1/third_party/googletest/googlemock/build-aux/ltmain.sh
fixing libtool script ./third_party/protobuf/3.6.1/third_party/googletest/googletest/build-aux/ltmain.sh
no configure script, doing nothing
preBuildPhase
building
building of '/nix/store/14gpkg2y5fg7my3kc5fpjm0qfcbm22vz-bazel-0.23.2.drv' timed out after 600 seconds of silence
error: build of '/nix/store/14gpkg2y5fg7my3kc5fpjm0qfcbm22vz-bazel-0.23.2.drv' failed

Note the missing "Building Bazel from scratch" output in Hydra. That line comes from ./bazel_src/compile.sh. Either we are losing logging somehow or buildPhase isn't even starting.
I'm going to try enable tracing and kick off a build, but I'm unsure if the bot will listen to me.

@GrahamcOfBorg build bazel

Just in case you're curious, even if I patch away the line in the build where it initially sets JAVAC_VERSION to hardcode "javac 1.8" it still fails with sandbox = true. This is failing with the core invocation of javac to compile the Bazel java classes.
/nix/store/9qyzw01bnla43457iwp6l9r6fbp7gk4y-zulu1.8.0_121-<IP_ADDRESS>/bin/javac -classpath third_party/truth8/truth-java8-extension-0.42.jar:third_party/javax_annotations/javax.annotation-api-1.3.2.jar:third_party/javax_annotations/javax.annotation-api-1.3.2-sources.jar
<SNIP ... />
-sourcepath src/java_tools/singlejar/java/com/google/devtools/build/zip:src/main/java:src/tools/xcode-common/java/com/google/devtools/build/xcode/common:src/tools/xcode-common/java/com/google/devtools/build/xcode/util:tools/java/runfiles:third_party/java/dd_plist/java:/private/var/folders/0f/v0lxnbcx4v94rbhtz37d6f2m0000gn/T/nix-build-bazel-0.23.2.drv-0/bazel_S1B9kq7r/src -d /private/var/folders/0f/v0lxnbcx4v94rbhtz37d6f2m0000gn/T/nix-build-bazel-0.23.2.drv-0/bazel_S1B9kq7r/classes -source 1.8 -target 1.8 -encoding UTF-8 @/private/var/folders/0f/v0lxnbcx4v94rbhtz37d6f2m0000gn/T/nix-build-bazel-0.23.2.drv-0/bazel_nuJxfgmI/param
dyld: Library not loaded: /System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
  Referenced from: /nix/store/9qyzw01bnla43457iwp6l9r6fbp7gk4y-zulu1.8.0_121-<IP_ADDRESS>/bin/javac
  Reason: no suitable image found.  Did find:
        /System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa: file system sandbox blocked stat()
        /System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa: file system sandbox blocked stat()
++ exit 0
+ run_atexit_handlers 0
+ local exit_code=0
+ local failed=no
+ for handler in ${ATEXIT_HANDLERS}

Apologies for all the spamming.
The above indicates to me that it is not possible to compile Java applications on Darwin with sandbox = true unless there is a way to prevent javac from reaching out for Cocoa. I hope that I'm mistaken. Does anybody know of any other Java applications that are building on Darwin with sandboxing enabled?
If not, I think this leaves Bazel 0.23.2 on Darwin dead in the water as an official Nixpkg unless a headless jdk comes along for Darwin.

0.24.0 was just released. @Profpatsch any ideas how we can debug this issue? It looks like the issues we get when building locally on macOS are completely different from building in hydra on macOS to be honest, I'm not sure spending time fixing what we're encountering locally is even helpful.

I can close this one if people wish to try 0.24.0. I've just attempted it and it's failing to compile for me on OSX, so I don't think it's ready just yet.

I'm not sure spending time fixing what we're encountering locally is even helpful.
I think it would be great to see why Hydra isn't outputting any logs once it starts the build. My current branch has tracing enabled, so if we can get that to run on Hydra, we'll at least see where it stops.

I agree that the Hydra Darwin builds outputting no logs could be a different issue / PR though. That assumes it's a general issue with Hydra and Darwin and not something specific to building Bazel.

I've quickly hacked it to build 0.24.0. Same issue as 0.23.2. It builds fine for me with sandbox=false. Fails to build with sandbox=true.

@Profpatsch I'd appreciate it if you could kick off another build so that we can see the trace output (if any) on Hydra.

Ready for review and Hydra builds. I've left TODOs with the tracing code which can be removed after testing Darwin builds on Hydra.

@Profpatsch Sorry to bother, any chance you could kick off a build for this please? Or tag somebody else with appropriate permissions.
I'd like to confirm why Hydra builds are failing.

Oh sorry, missed this notification. Sure
@GrahamcOfBorg build bazel

Update on the situation. It appears that both Hydra and my local sandbox=true builds are failing at the same place. The error just manifests itself differently.
Tracing output from Hydra:
++ test -z /nix/store/9qyzw01bnla43457iwp6l9r6fbp7gk4y-zulu1.8.0_121-<IP_ADDRESS>
++ JAVAC=/nix/store/9qyzw01bnla43457iwp6l9r6fbp7gk4y-zulu1.8.0_121-<IP_ADDRESS>/bin/javac
++ [[ -x /nix/store/9qyzw01bnla43457iwp6l9r6fbp7gk4y-zulu1.8.0_121-<IP_ADDRESS>/bin/javac ]]
+++ /nix/store/9qyzw01bnla43457iwp6l9r6fbp7gk4y-zulu1.8.0_121-<IP_ADDRESS>/bin/javac -version
building of '/nix/store/qqiyp494jprdxbbak4z6w5zg68kbw3vm-bazel-0.24.0.drv' timed out after 600 seconds of silence
error: build of '/nix/store/qqiyp494jprdxbbak4z6w5zg68kbw3vm-bazel-0.24.0.drv' failed

Compare that to my failure on Mac:
++ test -z /nix/store/9qyzw01bnla43457iwp6l9r6fbp7gk4y-zulu1.8.0_121-<IP_ADDRESS>
++ JAVAC=/nix/store/9qyzw01bnla43457iwp6l9r6fbp7gk4y-zulu1.8.0_121-<IP_ADDRESS>/bin/javac
++ [[ -x /nix/store/9qyzw01bnla43457iwp6l9r6fbp7gk4y-zulu1.8.0_121-<IP_ADDRESS>/bin/javac ]]
+++ /nix/store/9qyzw01bnla43457iwp6l9r6fbp7gk4y-zulu1.8.0_121-<IP_ADDRESS>/bin/javac -version
++ JAVAC_VERSION='dyld: Library not loaded: /System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
  Referenced from: /nix/store/9qyzw01bnla43457iwp6l9r6fbp7gk4y-zulu1.8.0_121-<IP_ADDRESS>/bin/javac
  Reason: no suitable image found.  Did find:
        /System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa: file system sandbox blocked stat()
        /System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa: file system sandbox blocked stat()'
+ run_atexit_handlers 134
+ local exit_code=134

You can see that both fail when running javac -version inside the sandbox. Locally it fails with SIGABRT and on Hydra it just hangs for some reason. I'm guessing that the way that the Hydra sandbox environment is setup for Darwin, means that loading native libraries like this might be blocked at a low-level or the same SIGABRT is happening, but not being propagated to the logs somehow.
Any suggestions @Profpatsch ? We now know what the error is and can reproduce it locally and on Hydra. This is good progress I think.
Any ideas how this ever could have worked in a sandbox?

javac is linked to Cocoa so I guess we need to use install_name_tool or build the jdk from source:
MacBook-Pro-6:nixpkgs uri$ otool -L /nix/store/9qyzw01bnla43457iwp6l9r6fbp7gk4y-zulu1.8.0_121-<IP_ADDRESS>/bin/javac
/nix/store/9qyzw01bnla43457iwp6l9r6fbp7gk4y-zulu1.8.0_121-<IP_ADDRESS>/bin/javac:
	/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa (compatibility version 1.0.0, current version 19.0.0)
	/System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 55179.11.0)
	/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices (compatibility version 1.0.0, current version 45.0.0)
	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.5)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0)
	/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
	/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 945.16.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 744.18.0)

javac is linked to Cocoa so I guess we need to use install_name_tool or build the jdk from source:

Yes. That seems like it should be done as part of the jdk nixpkg. I guess there is a reason that it's not been done.
I guess another alternative would be to use the Bazel binary distributions of Bazel (at least for Darwin). I see that Ant, Gradle, Maven and SBT are all using binary distributions. As a result, they don't execute javac inside the sandbox.
I guess building from source is the Nix preference, but there are official binary Linux, Darwin and Windows builds of Bazel available.

I had completely forgotten about extra-sandbox-paths: https://github.com/NixOS/nix/issues/2732#issuecomment-474150868
It's possible the value of that setting on hydra changed recently, but on master bazel is ok because it was built from before the change, so it's still cached.
@LnL7 Do you know where I can look up the value of that setting on hydra.nixos.org? I couldn't find anything in https://github.com/NixOS/nixos-org-configurations.

on master bazel is ok because it was built from before the change, so it's still cached.

I'm trying to confirm if 0.22 still builds on master now with: https://github.com/NixOS/nixpkgs/pull/58557
Just needs someone to kick-off a Hydra build.
Good tip about extra-sandbox-paths

Looks like extra-sandbox-paths almost does the trick. Bazel 0.24.0 gets right to the end of the build with:
sandbox = true
extra-sandbox-paths = /System/Library/Frameworks /System/Library/PrivateFrameworks /usr/lib

What is the policy for setting those values on Hydra? Or are those settings not permitted on Hydra?
It now fails with:
Starting local Bazel server and connecting to it...
Server crashed during startup. Now printing /private/var/folders/0f/v0lxnbcx4v94rbhtz37d6f2m0000gn/T/nix-build-bazel-0.24.0.drv-0/_bazel_groodt/561765db2a1d1b8606e4b7b3fd145d4b/server/jvm.out
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.protobuf.UnsafeUtil (file:/private/var/folders/0f/v0lxnbcx4v94rbhtz37d6f2m0000gn/T/nix-build-bazel-0.24.0.drv-0/_bazel_groodt/install/74381fd39e30d52ac531dc05c55dda59/_embedded_binaries/A-server.jar) to field java.nio.Buffer.address
WARNING: Please consider reporting this to the maintainers of com.google.protobuf.UnsafeUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
I/O Error: Failed to bind

That's Bazel trying to start a local server and failing to connect to a port I think. Another sandboxing change?

@uri-canva None of the darwin machines used by hydra currently have sandboxing enabled.

@LnL7 Would you mind kicking off this build on Darwin please: https://github.com/NixOS/nixpkgs/pull/58557

@GrahamcOfBorg build bazel

Going through the comments I see a bunch of mostly unrelated comments and I see a bunch of unrelated issues that's probably confusion.
Firstly sandboxing is currently not fully functional on darwin and not enabled by default, I have been spending some time to get it back on track but it's not a requirement for merge requests. Hydra and ofborg are 2 different things. Hydra only builds reviewed changes and these will be pushed to the cache eg. https://hydra.nixos.org/job/nixpkgs/trunk/bazel.x86_64-darwin, while ofborg runs for merge requests and uses different hardware. I enabled partial darwin sandboxing there to get an overview of what still needs fixing, like this case.
There are also some functional differences between sandboxing on linux vs darwin.

/tmp is global and not namespaced per build, which means if 2 consecutive builds try to write the same file there without cleaning it up nicely the second will fail because builds run as different unprivileged users. It's better to make the build use NIX_BUILD_TOP instead since that's managed by nix and will be cleaned up automatically regardless if the build fails or not.
Similarly localhost is not namespaced either, if multiple builds try to listen on the same port at the same time one of them will fail. Because of this the nix sandbox needs explicit whitelisting of builds that are allowed to do this with __darwinAllowLocalNetworking = true;.

@GrahamcOfBorg build bazel

ofborg runs for merge requests and uses different hardware. I enabled partial darwin sandboxing there to get an overview of what still needs fixing, like this case.

This explains it! Thank you! I've been pulling my hair out trying to figure out how 0.22 could ever have succeeded with sandboxing enabled on Darwin. The answer is that it wouldn't. It would fail just as it fails on my local machine.
@LnL7 do you have any insights into why these Darwin failures with ofberg manifest as "time outs" rather than actual sandbox errors which they are likely to be?

I enabled partial darwin sandboxing there to get an overview of what still needs fixing, like this case.

Is the configuration available anywhere? Nothing jumps to my eye in https://github.com/NixOS/ofborg.
We're trying to repro the failure in ofborg on our laptops but we can't get the same errors. At least we now know it's a sandbox error due to jdk linking in all system libraries and we also know how come it doesn't cause any issues in hydra, but since neither of us can trigger darwin builds on ofborg it's still very hard for us to debug the derivation.

If I have sandbox enabled, and change the reference to the link section with install_name_tool, I get:
+++ /nix/store/fn6gwllqy98qhmizikmkzdfmpq05nyqr-zulu1.8.0_121-<IP_ADDRESS>/bin/javac -version
++ JAVAC_VERSION='dyld: Library not loaded: /nix/store/hdsb3b20czzy8424yzhx3xkfyr11mmh7-apple-framework-Cocoa/Library/Frameworks/Cocoa.framework/Cocoa
  Referenced from: /nix/store/fn6gwllqy98qhmizikmkzdfmpq05nyqr-zulu1.8.0_121-<IP_ADDRESS>/bin/javac
  Reason: no suitable image found.  Did find:
        /nix/store/hdsb3b20czzy8424yzhx3xkfyr11mmh7-apple-framework-Cocoa/Library/Frameworks/Cocoa.framework/Cocoa: file system sandbox blocked stat()
        /System/Library/Frameworks/Cocoa.framework/Cocoa: file system sandbox blocked stat()'

The reference to the system framework might come from some other dynamic library loaded in javac, but it looks like even the library from nixpkgs cannot be accessed in the sandbox.

Darwin success: https://github.com/NixOS/nixpkgs/pull/58147/checks?check_run_id=88814348
Was the sandbox disabled?

Darwin success: https://github.com/NixOS/nixpkgs/pull/58147/checks?check_run_id=88814348
Was the sandbox disabled?

@uri-canva I imagine it must have been disabled by @LnL7 or somebody else.
In summary @uri-canva all of our struggling with Darwin specific sandbox issues was unnecessary. Still valuable information, but ultimately unnecessary. It should have been possible to merge 0.23.0 or 0.23.2 once we had them building locally on Mac and remotely with Ofborg.
Main important things I've learned:

Hydra and Ofborg are different running on different hardware for different purposes.
On Hydra, sandbox is not enabled for Darwin builds. Precisely because of issues like we've encountered. sandbox simply isn't ready yet for Darwin.
Ofborg builds can have sandbox enabled or disabled for Darwin, so it can be flaky. I think it's not intended for Ofborg to block merges to master, but rather to give higher confidence and catch obvious issues.
There is an issue with Ofborg build failure messages on Darwin. When it encounters a sandbox related issue like we have, it is manifesting as a timeout incorrectly.

@Profpatsch I've squashed all my commits into 1 now and removed all TODOs. So the PR is ready for review.
Ironically, it is now the linux build failing. It builds Bazel and gets all the way to the tests and then fails with:
ERROR: /build/_bazel_nixbld/96c321f6d4067432c019ff2725c18acb/external/bazel_tools/tools/jdk/BUILD:218:1: no such package '@remote_java_tools//': java.io.IOException: Error downloading [https://mirror.bazel.build/bazel_java_tools/java_tools_pkg-0.5.1.tar.gz] to /build/_bazel_nixbld/96c321f6d4067432c019ff2725c18acb/external/remote_java_tools/java_tools_pkg-0.5.1.tar.gz: Unknown host: mirror.bazel.build and referenced by '@bazel_tools//tools/jdk:JacocoCoverageRunner'
ERROR: Analysis of target '//examples/cpp:hello-success_test' failed; build aborted: no such package '@remote_java_tools//': java.io.IOException: Error downloading [https://mirror.bazel.build/bazel_java_tools/java_tools_pkg-0.5.1.tar.gz] to /build/_bazel_nixbld/96c321f6d4067432c019ff2725c18acb/external/remote_java_tools/java_tools_pkg-0.5.1.tar.gz: Unknown host: mirror.bazel.build

@groodt the linux build succeeds when the sandbox is turned off (it already is on Darwin). The reason is that in order to be sandbox friendly, you need to prefetch dependencies, since there is no network access inside the sandbox. We already need to do this, so you probably just need to add new  tarballs to srcDeps (grep Bazel's default.nix).

Status update.
Building successfully on Darwin with these settings in /etc/nix/nix.conf:
sandbox = true
extra-sandbox-paths = /System/Library/Frameworks /System/Library/PrivateFrameworks /usr/lib

Failing on linux. Investigating now, but it looks like a standard issue.

@groodt the linux build succeeds when the sandbox is turned off (it already is on Darwin). The reason is that in order to be sandbox friendly, you need to prefetch dependencies, since there is no network access inside the sandbox. We already need to do this, so you probably just need to add new tarballs to srcDeps (grep Bazel's default.nix).

@mboes This is true somewhat, but if you look at the game of whack-a-mole I've been playing in this thread, it's not so straightforward. Ofborg has been running with sandboxing enabled, so Darwin builds have been "timing out" (actually SIGABRT sandbox errors).
This resulted in us chasing our tails and making changes to the already working derivation of 0.22. It might be that some of these changes I've made to make Darwin sandbox happy, are causing it to fail on linux.
The early builds of this derivation were passing on linux and "timed out" on Darwin. Now it is working on Darwin and failing on Linux. 👍
Anyway, it's probably just a matter of downloading additional files or rolling back some of my changes that made Darwin sandbox happy. I'll take a look today.

One thing I thought of in the meantime: skipping 0.23.x could make it harder for users to upgrade, since Bazel keeps only one version of backwards compatibility. We should have two separate upgrade commits, even if they're one right after the other, so that people who pin nixpkgs can upgrade to 0.23.2 first and then to 0.24.0.

I reopened https://github.com/NixOS/nixpkgs/pull/58116 in case we want to go ahead and merge that first.

Having a Hydra build of bazel 0.23.2 for pinning sounds good to me.
I'm currently testing linux sandboxed builds of this PR, but I think it's also ready now.

This is now building and ready for review @Profpatsch @mboes Got there in the end!
Please see @uri-canva comment about thoughts of doing 0.22 -> 0.23.2 first. I don't mind if that is done first. I don't see too much benefit in sticking to an old Bazel for very long, so we should get this one merged shortly after if possible.
See below table for how I've tested this derivation on local Darwin and Linux. I'm confident that it will work on Hydra now. Linux should work on Ofborg and Hydra. Darwin might fail on Ofborg, but should succeed on Hydra given the testing that I've done.

Status
OS
sandbox
Notes

✅
Linux
false
single-user Nix 2.2.1 on privileged debian:jessie

✅
Linux
true
single-user Nix 2.2.1 on privileged debian:jessie

✅
Darwin
false
sandbox = false

✅
Darwin
true
sandbox = trueextra-sandbox-paths = /System/Library/Frameworks /System/Library/PrivateFrameworks /usr/lib

@GrahamcOfBorg build bazel

@profpatsch As far as I'm concerned we can merge.

@profpatsch Any feedback? There is typically a new Bazel release every 2-4
weeks, so maybe we should land this one after all. If somebody needs an
older version, they can call a custom derivation. Anybody using Bazel has
to stay up to date with new versions anyway.
On Sun, 31 Mar 2019 at 23:26, Mathieu Boespflug<EMAIL_ADDRESS>wrote:

@profpatsch As far as I'm concerned we can merge.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/NixOS/nixpkgs/pull/58147#issuecomment-478337330, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAU9d0uoR4vSn6NiVSrYri-lz5cAcdcYks5vcKmPgaJpZM4cEZQ6
.

Yep, we should merge even if we skip 0.23, the releases are too fast for the manpower we have here.

Should this be backported to 19.03 as well?

Should this be backported to 19.03 as well?

I'd like this too, but it's a bit late in the game for 19.03 I think. They've been working really hard to get it out the door. I don't know what the policy is for late inclusions. I guess with Nix it should be safe, but 🤷♂

Nah, it’s a major, incompatible update, so we shouldn’t backport it.


Automatically enable App SDK 0.8 coroutine support when present

Fixes https://github.com/microsoft/WindowsAppSDK/issues/957 by automatically including the header from the App SDK
Keeps behavior consistent with other Microsoft-provided dispatcher types, and allows the App SDK to version the header themselves still
Testing methodology:

Create a new console app
Install CppWinRT and WinUI 3 NuGet packages
Replace cppwinrt.exe from NuGet with self-built binary
Ensure the following code compiles:#include <winrt/Microsoft.UI.Dispatching.h>

winrt::fire_and_forget test(const winrt::Microsoft::UI::Dispatching::DispatcherQueue& queue)
{
    co_await queue;
}

We've decided against adding support for additional dispatchers. Those from outside libraries can instead ship their own helper libraries.

This presents several problems though:

When the switch to C++20 modules is done, libraries won't be able to extend the winrt module in any way, leading to the need to continue using headers, or to ship a different module. This is counterproductive to the adoption of modules.
Something that was expected to "just work", due to precedent with existing types, won't work anymore. And since it's a poorly documented change/requirement (it is written absolutely nowhere that I need to #include <microsoft.ui.dispatching.co_await.h> for coroutines to work), it will lead to user confusion, like in the original issue linked above.

If this is the way to go, I think there should be a standard way for libraries to extend cppwinrt's codegen, as to inject their own "strings" into generated headers.
This PR just includes the header from the NuGet package if available, so it can still be updated and versioned outside of cppwinrt, and doesn't add any code specific to that dispatcher directly in the core cppwinrt codebase.


Enable manual trigger for release workflow.

Signed-off-by: Zelin Hao<EMAIL_ADDRESS>Description
Enable manual trigger for release workflow.
Issues Resolved
List any issues this PR will resolve, e.g. Closes [...].
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #2480 (1e33a85) into main (856b691) will increase coverage by 0.06%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##               main    #2480      +/-   ##
============================================
+ Coverage     94.63%   94.70%   +0.06%
============================================
  Files           211      151      -60
  Lines          4322     4039     -283
  Branches         29        0      -29
============================================
- Hits           4090     3825     -265
+ Misses          226      214      -12
+ Partials          6        0       -6

Impacted Files
Coverage Δ

src/system/temporary_directory.py
85.71% <0.00%> (-11.43%)
:arrow_down:

src/system/os.py
93.75% <0.00%> (-6.25%)
:arrow_down:

deployment/lib/buckets.ts

...tifactsQualifier_OpenSearch_Dashboards_Jenkinsfile

tests/jenkins/jobs/GetManifestSHA_Jenkinsfile

tests/jenkins/jobs/Messages_Jenkinsfile

tests/jenkins/jobs/PublishNotification_Jenkinsfile

tests/jenkins/jobs/BuildShManifest_Jenkinsfile

...enkins/jobs/BuildDockerImage_Qualifier_Jenkinsfile

tests/jenkins/jobs/AssembleUpload_Jenkinsfile

... and 52 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


Adding to Conda-Forge

I would like to know if it would be possible to get this added to Conda-Forge.  I am willing to do the work to do it, if it is allowed.  I will have to learn how first, but it is something that I have been wanting to learn.

Hi I was looking to use conda-forge too  - to install and use powerbiclient, the first thing we would need is a source distribution at https://pypi.org/project/powerbiclient/3.1.1/#files - which is currently missing.
@orshemesh16 @KotanaSai21 is that some that can be done.. to enable conda users. I can help with the conda-forge feedstock and recipe if needed.

Add here
https://ideas.fabric.microsoft.com/ideas/idea/?ideaid=2fc7ca31-88a2-ef11-95f6-000d3a7a93ec


ROMFS not rebuilt correctly when ROMFS files are 'touch'ed

@dagar: related to discovery in PR: #9067.
If one of the files in ROMFS/px4_common/ (or whatever directory you specify as your ROMFS src directory) is touched, then the ROMFS is not rebuilt - it is rebuilt ONLY if an airframe is touched (or a file that starts with a number). This is because the add_custom_command in "ROMFS/CMakeLists.txt" does not depend on the contents of the ROMFS/px4_common/ directory. It only depends on the air frames (files that start with a number).
file(GLOB_RECURSE init_airframes ${PX4_SOURCE_DIR}/ROMFS/${config_romfs_root}/*/[1-9]*)
add_custom_command(OUTPUT ${romfs_temp_dir}/init.d/rcS ${romfs_temp_dir}/init.d/rc.autostart
        COMMAND cmake -E copy_directory ${romfs_src_dir} ${romfs_temp_dir}
        COMMAND ${PYTHON_EXECUTABLE} ${PX4_SOURCE_DIR}/Tools/px_process_airframes.py
-a ${romfs_temp_dir}/init.d
-s ${romfs_temp_dir}/init.d/rc.autostart
--board ${BOARD}
        COMMAND ${PYTHON_EXECUTABLE} ${PX4_SOURCE_DIR}/Tools/px_romfs_pruner.py
--folder ${romfs_temp_dir} --board ${BOARD}
        DEPENDS
${romfs_src_files}
${init_airframes}
${PX4_SOURCE_DIR}/ROMFS/${config_romfs_root}/init.d/rcS
${PX4_SOURCE_DIR}/Tools/px_process_airframes.py
        )

Note: although, the add_custom_command() depends on ${romfs_src_files}, this variable is never set anywhere; it is an empty dependency. I suspect that ${romfs_src_files} was supposed be set with a file(GLOB_RECURSE ...), but this has just not been done.
Note: As it currently stands, if you make a change to any of the rc.* files other than rcS, then the ROMFS will not be rebuilt.

What we could do is remove the file(GLOB_RECURSE init_airframes ${PX4_SOURCE_DIR}/ROMFS/${config_romfs_root}/*/[1-9]*) line and replace it with file(GLOB_RECURSE romfs_src_files ${romfs_src_dir} ${romfs_src_dir}/*)
We could then remove the dependence on ${init_airframes}, since it would be redundant. And as far as I can see, its only purpose is to cause a rebuild of the ROMFS if an airframe file is touched.
Tell me if you think I should submit a PR for this.

I think we might need to explicitly list all the files. Globbing generally doesn't work very well in cmake. It will miss added files (you need to know how to manually kick off another cmake configure), and it's perhaps even more annoying when you remove a file that was previously globbed.

I'd be up for requiring all the ROMFS files to be explicitly listed in a CMakeLists.txt. With some simple directory hierarchy in place in won't be too overwhelming.

I'd be up for requiring all the ROMFS files to be explicitly listed in a CMakeLists.txt

Agreed. cmake recommends not to use GLOB and it will improve incremental builds.

current master after flashing into fmu_V4 (vtol) all tuning values get lost or don´t load, probably the same source as mentioned above. Now I flashed back to master from 8 th of march that still was o.k.

Conversation continued here:  #9107.


Mthree

Adding Mthree mitigation method (See Equation (5) in https://arxiv.org/pdf/2108.12518.pdf for MTHREE implementation) and fixing a bug in tensor_rank_mult
It says I'm behind main by 1 commit. I tried rebasing manually, but cant seem to fix it. Let me know if you need me to pull fresh changes and start clean again or if you can guide me to fix it

Hey @rajaiitp feels like this can be merged.

I wanted to push the changes that you requested in the review along with majority vote, sorry about the delay. I am done, will push in a few mins and request review

Feel free to ping me for a huddle if you want me to walk you through the idea


[RFC][DNM] DMM: Skeleton for a DMA Memory Management subsystem

This is a hopefully-to-be-discussed proposal for an actual implementation of what was discussed in #57220 (please read that before keep reading).
TL;DR: Zephyr is currently missing a way to allocate and prepare buffers to be used by DMA engines and DMA-enabled peripherals on platforms with a complex memory organization. This PR is a proposal for a subsystem interface dealing with that.

Can we use device tree to describe the requirement for a buffer

That idea I like very much.  Puts a lot of preprocessor work onto the platform/controller layer, but cleanly extends to all sorts of nonsense like alternate memory spaces (even madness like iommu's or 24 bit ISA DMA!) that are otherwise really hard to abstract.  And it falls back to obvious behavior for "it's just memory" systems that tend to be the common case.

@teburd, do I understand correctly that you suggest to move the responsibility to manage memory buffers from a device driver to its user? If that's the case, let's consider following application code:
struct spi_buf_set prepare_spi_tx_buffer(void) {
  static const struct spi_buf buf = {
    .buf = "Hello world",
    .len = sizeof("Hello world"),
  };

  struct spi_buf_set set = {
    .buffers = &buf,
    count = 1,
  }

  return set;
}

If SPI device driver takes care of copying buf content to a bounce buffer accessible by SPI's DMA, like @carlocaione suggested, then this code is portable. However, if the SPI API user was responsible for managing memory buffers, it would need to be modified to sth like:
struct spi_buf_set prepare_spi_tx_buffer(void) {
  static const struct spi_buf buf = {
    .buf = allocate_spi_buf(sizeof("Hello world")),
    .len = sizeof("Hello world"),
  };

  if (!buf.buf) {
    // Handle fault
  }

  memcpy(buf.buf, "Hello world", sizeof("Hello world"));

  cache_write_back(buf.buf, sizeof("Hello world"));

  buf.buf = translate_cpu_address_to_bus_address(buf.buf);

  struct spi_buf_set set = {
    .buffers = &buf,
    count = 1,
  }

  return set;
}

void release_spi_tx_buffer(struct spi_buf_set set) {
  void *cpu_address = translate_bus_address_to_cpu_address(set.buffers[0].buf); // If such translation is possible
  free_spi_buf(cpu_address);
}

Portability is an important software attribute, especially in hardware agnostic projects like Zephyr. That's why I think the responsibility for managing memory buffers for DMA should be hidden from application code and implemented in device drivers.
Applications should have a possibility to improve performance and use buffer allocation methods meeting buffer memory management requirements to prevent device drivers from copying data to bounce buffers. But it should not be an option, not a requirement for an application.

@hubertmis the problem with that kind of scheme is that once we decide on a dynamic allocation API we can't go back.  This isn't linux, and we don't get to assume we're running on devices with copious virtual memory available for stuff.  Existing Zephyr paradigms work very hard (to be fair, occasionally too hard) to enable static allocation for everything an application might want.
@teburd's idea is very much in that vein: it's an API that allows an app to say "I want a buffer that will work for DMA with this device", and get it, without having to do much more than know that the device exists and call whatever acquire/release API we decide on around code that mutates it.

@teburd , could you draft your proposal? I would like to understand it better to be able to compare properties of both solutions

Thanks @teburd for these details. I like the idea of describing memory buffer requirements in the device tree and creating macros to easily create buffers meeting them.
It looks the main open question in this discussion is about the device driver model. I don't think Zephyr's device driver model currently puts any requirements on the memory buffers passed to the drivers' functions by their users: users can pass any buffers allocated in any memory (writable or not, cacheable or not, aligned or not, dynamically or statically allocated, ...) and they expect the drivers to correctly perform requested operations using the passed memory buffers.
It seems we are discussing several options on how to incorporate DMA requirements regarding used memory buffers in a unified way for all device drivers in Zephyr:

Keep the device driver model as it is. Create responsibility for drivers of devices with DMA with specific requirements to meet these requirements.
Modify the device driver model to require driver users to pass to the drivers only the memory buffers created with a designated (static or dynamic) allocator.
Keep the device driver model as it is. Create an opportunity for applications to allocate buffers meeting DMA requirements. Create responsibility for device drivers to check if the passed buffer meets the requirements, if not the driver would be responsible for copying the buffer's content to a bounce buffer.

The proposed device tree properties and macros can be used in each of these solutions. The difference would be which layer in the system would be able and/or responsible to use the macros to allocate the buffers correctly.
Is this summary correct? Do you happen to see any other options I didn't list?
When we decide the list of options is around being completed, we should start listing properties of these options that would help us to find out which one is the most aligned with Zephyr's needs.

@teburd, thanks for your comment.
User code
I'm looking at selected Zephyr samples to assess changes needed if we decided to modify the device driver model so that the application is responsible for correct buffer allocation.
spi_bitbang
https://github.com/zephyrproject-rtos/zephyr/blob/c48422f52360a6026ef64e8c13387dbd09a9c9aa/samples/drivers/spi_bitbang/src/main.c#L32-L35
Currently buff is used directly in the tx_buf structure. I understand with your recommendation this sample (and all other code using SPI driver API) would need to be modified to something like
 DT_BUF(dma0, dev) uint16_t buff[5] = { 0x0101, 0x00ff, 0x00a5, 0x0000, 0x0102};
 int len = 5 * sizeof(buff[0]);

 struct spi_buf tx_buf = { .buf = buff, .len = len };

  dt_buf_acquire(buff); // To do cache management

I'm not sure if the DT_BUF() macro definition can somehow describe the padding of buff, so it might be necessary to use another macro:
DT_BUF(dma0, dev) uint16_t buff[DT_BUF_SIZE(dma0, dev, 5)] = { ... };

Using sizeof(buff) would not return the intended size of the buffer anymore. This change of the sizeof() return value could result in obscure bugs.
uart/echo_bot
This sample includes both RX and TX paths of UART communication.
RX
It's pretty easy because the following code
https://github.com/zephyrproject-rtos/zephyr/blob/c48422f52360a6026ef64e8c13387dbd09a9c9aa/samples/drivers/uart/echo_bot/src/main.c#L24
would need to be modified to
DT_BUF(dma0, uart_dev) static char rx_buf[DT_BUF_SIZE(dma0, uart_dev, MSG_SIZE)];

assuming these macros would work correctly for global static objects.
And some _acquire() / _release() calls for proper cache management.
TX
We can't use anymore just any char *buf to be printed like in
https://github.com/zephyrproject-rtos/zephyr/blob/c48422f52360a6026ef64e8c13387dbd09a9c9aa/samples/drivers/uart/echo_bot/src/main.c#L64-L71
To use this kind of API the sample code would need to allocate a bounce buffer meeting the UART TX  and use it here:
DT_BUF(dma1, uart_dev) static char tx_buf[DT_BUF_SIZE(dma1, uart_dev, 1)];

void print_uart(char *buf)
{
	int msg_len = strlen(buf);

	for (int i = 0; i < msg_len; i++) {
                tx_buf[0] = buf[I];
                dt_buf_tx_acquire(tx_buf);
		uart_poll_out(uart_dev, tx_buf);
                dt_buf_tx_release(tx_buf);
	}
}

This bounce buffer would be allocated by the application for any platform, even if given UART does not have DMA, or includes a DMA capable of printing any application. This potentially unnecessary bounce buffer makes usage of driver APIs harder and is wasteful.
blinky_pwm
Some hardware peripherals like PWM, even though it's a rare case, can also include DMA. To be consistent with the device driver model and potentially handle DMA here, the API user would need to pass period and pulse arguments to the pwm_set_dt() function in a buffer meeting DMA requirements. So we could again allocate a bounce buffer in the application to do that. However, it's not obvious how the period and pulse fields shall be placed in the buffer, because each hardware peripheral might require a different data structure.
uart/echo_bot again
Assume we have a UART peripheral that uses DMA to read data from RAM in the following structure:
Byte 1: the size of the transfer
Byte 2: the 1st byte to be transferred
Byte 3: the 2nd byte to be transferred
Byte 4: ...
The solution discussed above with DT_BUF(dma1, uart_dev) static char tx_buf[DT_BUF_SIZE(dma1, uart_dev, 1)]; would not work with a peripheral like this. The tx_buf buffer would miss byte 1 containing the size of the transfer. The device driver would anyway allocate another bounce buffer and copy data around to add byte 1 at the beginning.
Conclusion
It seems impossible to impose responsibility on an application to allocate correct buffers for each potential DMA usage in any system. It would require an application (or a subsystem) to structure data in a buffer in such a way that is required for its target platform, rendering the application non-portable.
This makes option 2 I described in https://github.com/zephyrproject-rtos/zephyr/pull/57602#issuecomment-1545700327 invalid.
Do you agree?

@teburd, thanks for your comment.
User code with modified device driver model
I'm looking at selected Zephyr samples to assess changes needed if we decided to modify the device driver model so that the application is responsible for correct buffer allocation.
spi_bitbang
https://github.com/zephyrproject-rtos/zephyr/blob/c48422f52360a6026ef64e8c13387dbd09a9c9aa/samples/drivers/spi_bitbang/src/main.c#L32-L35
Currently buff is used directly in the tx_buf structure. I understand with your recommendation this sample (and all other code using SPI driver API) would need to be modified to something like
 DT_BUF(dma0, dev) uint16_t buff[5] = { 0x0101, 0x00ff, 0x00a5, 0x0000, 0x0102};
 int len = 5 * sizeof(buff[0]);

 struct spi_buf tx_buf = { .buf = buff, .len = len };

  dt_buf_acquire(buff); // To do cache management

I'm not sure if the DT_BUF() macro definition can somehow describe the padding of buff, so it might be necessary to use another macro:
DT_BUF(dma0, dev) uint16_t buff[DT_BUF_SIZE(dma0, dev, 5)] = { ... };

Using sizeof(buff) would not return the intended size of the buffer anymore. This change of the sizeof() return value could result in obscure bugs.
uart/echo_bot
This sample includes both RX and TX paths of UART communication.
RX
It's pretty easy because the following code
https://github.com/zephyrproject-rtos/zephyr/blob/c48422f52360a6026ef64e8c13387dbd09a9c9aa/samples/drivers/uart/echo_bot/src/main.c#L24
would need to be modified to
DT_BUF(dma0, uart_dev) static char rx_buf[DT_BUF_SIZE(dma0, uart_dev, MSG_SIZE)];

assuming these macros would work correctly for global static objects. And some _acquire() / _release() calls for proper cache management.
TX
We can't use anymore just any char *buf to be printed like in
https://github.com/zephyrproject-rtos/zephyr/blob/c48422f52360a6026ef64e8c13387dbd09a9c9aa/samples/drivers/uart/echo_bot/src/main.c#L64-L71
To use this kind of API the sample code would need to allocate a bounce buffer meeting the UART TX and use it here:
DT_BUF(dma1, uart_dev) static char tx_buf[DT_BUF_SIZE(dma1, uart_dev, 1)];

void print_uart(char *buf)
{
	int msg_len = strlen(buf);

	for (int i = 0; i < msg_len; i++) {
                tx_buf[0] = buf[I];
                dt_buf_tx_acquire(tx_buf);
		uart_poll_out(uart_dev, tx_buf);
                dt_buf_tx_release(tx_buf);
	}
}

This bounce buffer would be allocated by the application for any platform, even if given UART does not have DMA, or includes a DMA capable of printing any application. This potentially unnecessary bounce buffer makes usage of driver APIs harder and is wasteful.
blinky_pwm
Some hardware peripherals like PWM, even though it's a rare case, can also include DMA. To be consistent with the device driver model and potentially handle DMA here, the API user would need to pass period and pulse arguments to the pwm_set_dt() function in a buffer meeting DMA requirements. So we could again allocate a bounce buffer in the application to do that. However, it's not obvious how the period and pulse fields shall be placed in the buffer, because each hardware peripheral might require a different data structure.
uart/echo_bot again
Assume we have a UART peripheral that uses DMA to read data from RAM in the following structure:
Byte 1: the size of the transfer Byte 2: the 1st byte to be transferred Byte 3: the 2nd byte to be transferred Byte 4: ...
The solution discussed above with DT_BUF(dma1, uart_dev) static char tx_buf[DT_BUF_SIZE(dma1, uart_dev, 1)]; would not work with a peripheral like this. The tx_buf buffer would miss byte 1 containing the size of the transfer. The device driver would anyway allocate another bounce buffer and copy data around to add byte 1 at the beginning.

Then change the macro
/**
 * Defines a buffer with a given symbolic name such that it is aligned, placed, sized, and
 * cached appropriately. The symbol buf_name may not be the actual start address of the buffer
 * if the device requires a prefixed space for metadata about the operation.
 */
#define DT_BUF_DEFINE(node, buf_name, buf_len) \
    DT_BUF_ATTRS(node) uint8_t __buf_name[DT_BUF_LEN(node, buf_len)];
    uint8_t *buf_name = &__buf_name[DT_BUF_OFFSET(node)];

Just like that, I have a macro appropriately created. How could we verify the buffer is correctly done though for that device you ask? A marker could be prefixed or suffixed and asserted on when CONFIG_ASSERT=y.

Conclusion
It seems impossible to impose responsibility on an application to allocate correct buffers for each potential DMA usage in any system. It would require an application (or a subsystem) to structure data in a buffer in such a way that is required for its target platform, rendering the application non-portable.

That's the whole point of applying macros and device tree to the problem. To make it portable.

This makes option 2 I described in #57602 (comment) invalid.
Do you agree?

No I do not agree, I don't think we've fully explored options to statically allocate buffers correctly.

I have a macro appropriately created

Thanks, indeed it would cover much more cases.
I'll reiterate some and discuss a few more examples to see if we have a common understanding of how this macro would change the Zephyr codebase.
blinky_pwm
I believe if a PWM peripheral uses DMA then the PWM API user could not create a buffer and fill it with period and pulse because the buffer structure is unknown to the user.
It would mean that the allocation of buffers by the API user would be available only for the device drivers that are streaming data (UART, SPI, I2C, USB, radios, ...), but for other drivers, it would be the driver's responsibility to allocate a bounce buffer and fill necessary data like period and pulse.
UART (again) with a dynamic payload offset in DMA buffer
Some peripherals have a dynamic structure of data consumed by DMA. One example would be UART which requires the following structure if the transfer is shorter than 255:
{length, byte 0, byte 1, byte 2, ...}

and following structure if the transfer is longer:
{0xff, length 0, length 1, length 2, length 3, payload 0, payload 1, ...}

Do you think we could somehow capture it in a macro, or would we need to treat such peripherals as exceptions and let device drivers allocate bounce buffers?
UART transferring constant data
Let's discuss an application that transfers constant byte-strings using UART. Currently, the application code would define these byte-strings as something like:
static const uint8_t str1[] = {0xde, 0xad, 0xbe, 0xef};
static const uint8_t str2[] = {0xca, 0xfe, 0xfa, 0xce};

Potentially placing data in NVM where some DMA have access, but others not necessarily.
With the new macro definition, I understand the code would need to be refactored to something like
DT_BUF_DEFINE(dev, str1_buf, sizeof(str1));
DT_BUF_DEFINE(dev, str2_buf, sizeof(str2));

static void copy_buf_content(void) {
  memcpy(str1_buf, str1, sizeof(str1));
  memcpy(str2_buf, str2, sizeof(str2));
}

what is functionally correct. However, it would cause unnecessary RAM buffers allocation and memcpy overhead for platforms where UART can access NVM directly.
spi_flash
SPI flash sample uses SPI (potentially with DMA) indirectly: main -> drivers/flash -> drivers/spi.
I'm not sure which module should allocate SPI DMA buffers in this sample.
I don't think it should be done by main, because main is decoupled from the const struct device *spi - it is aware of const struct device *flash only. So it cannot define a buffer like DT_BUF_DEFINE(spi, spi_buf, size);.
An alternative would be to define a bounce buffer in drivers/flash and copy the payload passed by main, but I understand your goal is to avoid bounce buffers. Do you think we can somehow achieve it in a more complex DMA usage like this one?

It seems like there's some special metadata header that needs to exist inline with the data in the cases you've laid out. Fair enough, lets work through it.

I have a macro appropriately created

Thanks, indeed it would cover much more cases.
I'll reiterate some and discuss a few more examples to see if we have a common understanding of how this macro would change the Zephyr codebase.
blinky_pwm
I believe if a PWM peripheral uses DMA then the PWM API user could not create a buffer and fill it with period and pulse because the buffer structure is unknown to the user.

This seems like a pretty special case, and perhaps a specialized PWM set of buffer setup macros would be useful that understand the period and pulse. We have C and while that limits our ability to express some compile time things, I think this could be done with PWM specific macros.

It would mean that the allocation of buffers by the API user would be available only for the device drivers that are streaming data (UART, SPI, I2C, USB, radios, ...), but for other drivers, it would be the driver's responsibility to allocate a bounce buffer and fill necessary data like period and pulse.
UART (again) with a dynamic payload offset in DMA buffer
Some peripherals have a dynamic structure of data consumed by DMA. One example would be UART which requires the following structure if the transfer is shorter than 255:
{length, byte 0, byte 1, byte 2, ...}

and following structure if the transfer is longer:
{0xff, length 0, length 1, length 2, length 3, payload 0, payload 1, ...}

Do you think we could somehow capture it in a macro, or would we need to treat such peripherals as exceptions and let device drivers allocate bounce buffers?

Why is a bounce buffer needed here vs multiple length prefixed transfers in a list? Heap allocations and memcpy aren't free.

UART transferring constant data
Let's discuss an application that transfers constant byte-strings using UART. Currently, the application code would define these byte-strings as something like:
static const uint8_t str1[] = {0xde, 0xad, 0xbe, 0xef};
static const uint8_t str2[] = {0xca, 0xfe, 0xfa, 0xce};

Potentially placing data in NVM where some DMA have access, but others not necessarily.
With the new macro definition, I understand the code would need to be refactored to something like
DT_BUF_DEFINE(dev, str1_buf, sizeof(str1));
DT_BUF_DEFINE(dev, str2_buf, sizeof(str2));

static void copy_buf_content(void) {
  memcpy(str1_buf, str1, sizeof(str1));
  memcpy(str2_buf, str2, sizeof(str2));
}

what is functionally correct. However, it would cause unnecessary RAM buffers allocation and memcpy overhead for platforms where UART can access NVM directly.
spi_flash
SPI flash sample uses SPI (potentially with DMA) indirectly: main -> drivers/flash -> drivers/spi. I'm not sure which module should allocate SPI DMA buffers in this sample.
I don't think it should be done by main, because main is decoupled from the const struct device *spi - it is aware of const struct device *flash only. So it cannot define a buffer like DT_BUF_DEFINE(spi, spi_buf, size);.
An alternative would be to define a bounce buffer in drivers/flash and copy the payload passed by main, but I understand your goal is to avoid bounce buffers. Do you think we can somehow achieve it in a more complex DMA usage like this one?

blinky_pwm

This seems like a pretty special case,

I would like to highlight here that every single peripheral in the system could be a special case like this. It is possible that any peripheral, including basic ones like PWM, GPIO, or timers, can use DMA internally to offload a CPU from accessing peripheral registers.

perhaps a specialized PWM set of buffer setup macros would be useful that understand the period and pulse

Do I understand correctly that it would require deprecating pwm_set_dt() and similar functions and replacing them with something like
DT_PWM_DATA(dev, name);

void set_pwm(period, pulse) {
  DT_PWM_SET_PERIOD(name, period);
  DT_PWM_SET_PULSE(name, pulse);
  pwm_set_apply(dev, name);
}

Solutions like this would cause an overhead I described in "UART transferring constant data" for every single device driver.

Is there some datasheet/ref manual or driver in the tree that does this today I could look at?

An example of PWM using DMA to load pulse values is here: https://infocenter.nordicsemi.com/topic/ps_nrf5340/pwm.html?cp=4_0_0_6_22
However, nRF53 has a basic memory hierarchy so no special handling of memory access is needed there (no cache: no need for alignment and padding and writing-back; common addressing across the system: no need for address translation; PWM's EasyDMA can access all memories in the system). If the memory hierarchy was more complicated, e.g. caches present, then it would be more difficult to handle and currently we are missing a systematic way in Zephyr to do that.

UART (again) with a dynamic payload offset in DMA buffer

Why is a bounce buffer needed here vs multiple length prefixed transfers in a list?

I'm not sure if a bounce buffer is needed. However, if the transfer size is determined in run-time, I cannot imagine a macro that could set the size and the offsets in the static buffer correctly. Maybe it would be possible with more complicated macros.

Is there some hardware datasheets or drivers I can look at?

I'm not aware of any publicly available datasheet or driver. Assuming an 802.15.4 radio's DMA expects PHR immediately followed by PSDU in RAM (like nRF5 radios), for 802.15.4 MSK PHY described in 17.1.1 of IEEE 802.15.4-2020, the length of PHR (encoding of the frame length) is one or two bytes long depending on the payload (PSDU) length. It's similar to the example I drafted in the previous comment.

UART transferring constant data
However, it would cause unnecessary RAM buffers allocation and memcpy overhead for platforms where UART can access NVM directly.

That's true today to with bounce buffers isn't it?

Some SoCs require special handling of RAM buffers for DMAs, others do not. Currently, Zephyr works perfectly fine with devices that do not require any special handling. DMAs in devices like this can access static const uint8_t str1[] = {0xde, 0xad, 0xbe, 0xef}; buffers directly, and Zephyr's driver model allows them to do that.
The RFC created by @carlocaione and this PR aim to design a systematic way to support SoCs requiring special handling of DMA RAM buffers. To answer your question: Yes, in these more complicated devices the additional buffer overhead and memcpy is most likely required for both bounce buffers and application-managed buffers solutions.
But my point is that we should avoid creating this overhead for SoCs which work fine without it.
I'm curious if we could define a driver API that allows its user to avoid the discussed overhead in simple SoCs, but still statically allocate buffers to avoid bounce buffers in complex SoCs.

spi_flash

In this case wouldn't the flash device tree node define things appropriately for applications? I'm a little lost on this example in particular.

I'll try to show my concern in a little more detailed example
Currently, the spi_flash sample contains code like:
const uint8_t expected[] = { 0x55, 0xaa, 0x66, 0x99 };
uint8_t buf[sizeof(expected)];
rc = flash_write(flash_dev, SPI_FLASH_TEST_REGION_OFFSET, expected, len);
rc = flash_read(flash_dev, SPI_FLASH_TEST_REGION_OFFSET, buf, len);

The expected and buf buffers are later translated by a chosen flash driver to something like data below (using spi_nor.c as an example):
        uint8_t buf[5] = { 0 };
	struct spi_buf spi_buf[2] = {
		{
			.buf = buf,
			.len = 1,
		},
		{
			.buf = data,
			.len = length
		}
	};

I understand the spi_nor.c code would need to be modified a little to use a static allocator like DT_BUF_DEFINE() for the buf buffer:
        DT_BUF_DEFINE(&driver_cfg->spi, buf, 5);
        memset(buf, 0, 5);

	struct spi_buf spi_buf[2] = {
		{
			.buf = buf,
			.len = 1,
		},
		{
			.buf = data,
			.len = length
		}
	};

However, it's not clear to me how the data buffer could be allocated with DT_BUF_DEFINE().
One option would be to allocate a buffer in the main.c:
const uint8_t expected[] = { 0x55, 0xaa, 0x66, 0x99 };
DT_BUF_DEFINE(/* how to get spi dev, if it's decoupled from flash API user? */, expected_buf, sizeof(expected));
memcpy(expected_buf, expected, sizeof(expected));

rc = flash_write(flash_dev, SPI_FLASH_TEST_REGION_OFFSET, expected_buf, len);

But to make it work we would need to increase coupling between software modules so that flash API user gets access to the bus device to which the flash module is connected. I don't think increasing coupling like this is a good idea.
Another solution would be to define a bounce buffer in spi_nor.c:
        DT_BUF_DEFINE(&driver_cfg->spi, bounce_buf, length);
        memcpy(bounce_buf, data, length);

But this solution relies on a bounce buffer that we discuss how to avoid.

My 2 cents.
Shortly, I think that the problem here is that there are cases where the application layer has no idea about the requirement at driver / hardware level about buffer allocation / overhead / metadata / etc...
Now, in many cases this information can indeed be obtained from DT, but there are cases where (for example the overhead, the internal buffer organization, etc...) the information is only known at run-time and not compile-time. For all these cases a static buffer allocation is probably not optimal (because we would end up introducing a different macro for each exception) or not achievable at all, so we must resort to a dynamic allocation of sorts (aka bounce buffers).
Now, I understand @andyross @teburd  concerns but I don't see an easy way out. If the buffer cannot be statically defined at app layer, then it must be allocated by drivers and filled up with a user buffer provided by app (when needed, plus all the other needed DMA bits and pieces).

My 2 cents.
Shortly, I think that the problem here is that there are cases where the application layer has no idea about the requirement at driver / hardware level about buffer allocation / overhead / metadata / etc...

That's what should be solved, making that available to the application.

Now, in many cases this information can indeed be obtained from DT, but there are cases where (for example the overhead, the internal buffer organization, etc...) the information is only known at run-time and not compile-time. For all these cases a static buffer allocation is probably not optimal (because we would end up introducing a different macro for each exception) or not achievable at all, so we must resort to a dynamic allocation of sorts (aka bounce buffers).

All the cases in Zephyr I've seen thus far can be solved with a macro that does just that. Please help me understand where this is coming from.

Now, I understand @andyross @teburd concerns but I don't see an easy way out. If the buffer cannot be statically defined at app layer, then it must be allocated by drivers and filled up with a user buffer provided by app (when needed, plus all the other needed DMA bits and pieces).

To repeat what @andyross said, this isn't Linux. To have this setup where the default would be to malloc+memcpy silently on a peripheral API call just because the application author didn't have the tools to setup a buffer right is massively wasteful and I'll -1 it all day. If you have special hardware with special needs because it has some unique logic as being described here, well that to me that seems like a vendor specific problem not a Zephyr problem. That should be solved in a vendor specific way not in a broad way that implies this is how Zephyr wishes to handle DMA buffers generally by default.
I'm happy to be wrong if examples can be found and given or some docs can be linked so I can better understand what problem is trying to be solved here. I haven't seen any good examples in the tree after some grepping around.
UART:

nrf has a memcpy from the source to a ram buffer, seemingly to avoid doing DMA from flash? Solvable with DT_BUF seemingly.

SPI:

esp32_spim mallocs and copies if the buffer isn't aligned. Solvable with DT_BUF, should not be done this way.
nrfx_spim copies if not in RAM. Solvable with DT_BUF

I2C:

nrfx_twim has a memcpy to seemingly work with easydma (which I believe does not have a traditional transfer list). Unsolvable in a general way. Unclear exactly to me the reasoning.

FLASH: (harder as many peripherals allow for mapping the flash into a bus address space...)

nrf_qspi does a memcpy to seemingly add a prefix. Solvable with DT_BUF potentially.
xmc4xxx does a memcpy to align. DT_BUF solvable
nios2 qspi does a memcpy to add a length prefix from a quick glance. Potentially DT_BUF solvable with a flash wrapper (as would nrf_qspi).

Happy to be shown something differently but I just don't see where this problem actually is. I do see the problem with setting up aligned and placed buffers. That is a well trod problem I think anyone who's written a test case with DMA involved can attest to. But a macro could solve it. I've linked to ones I know of first hand already.

malloc+memcpy silently on a peripheral API call just because the application author didn't have the tools to setup a buffer right is massively wasteful and I'll -1 it all day.

Agreed. But from the analysis I presented in previous comments, it seems forcing the application authors to use DT_BUF on the platforms in which such allocation is not necessary is also massively wasteful, so I could -1 it all day :)
I'm curious if we can modify any of these solutions or find a new one that is not wasteful in systems where DMA can use any buffers, and in systems where DMAs have special needs.

Maybe it will be easier to find common ground if we decide on the basic expected properties of the system we would like to get.
I believe for simplicity we can discuss an application that sends through UART two strings:

"Hello world": static const uint8_t hello_world[]  = "Hello world";
a generated random number: static uint8_t rand_num[RN_LEN];

Both buffers are configured statically by an application.
Now, let's consider two types of SoCs for which we compile this application

STM32 where UART driver sends data byte by byte using USART's TDR (Transmit data register)
nRF52 which contains UARTE requiring data to be placed in RAM
I would love to discuss a device with a data cache, but could not find one supported by Zephyr yet.

For STM32 we would like to:

Have hello_world buffer stored in ro-data in flash. The UART device driver copies data byte by byte from flash to USART's TDR.
Have rand_num buffer stored in bss in RAM. The UART device driver copies data byte by byte from this buffer to USART's TDR.

For nRF52 we would like to:

Copy hello_world from flash to RAM and configure UARTE's EasyDMA engine to point to the copy.
Have rand_num buffer stored in bss in RAM. The UARTE's EasyDMA engine points directly to rand_num buffer without copying.

Do you agree these are the system properties we would like to get in a single portable application?

I was thinking a little about the discussion we had during the last arch meeting.
We would like to achieve multiple quality attributes of the system:

low memory consumption
stability and accessibility
performance
developer-friendly API
and more

Changing priorities of these attributes leads to different solutions.
I'm thinking if we could merge (make available) all proposed solutions to make it possible for the application developer to choose which of the attributes is the highest priority for the application.
Here is what came to my mind looking at the example we discussed: audio application: getting audio data from SPI and sending them to I2S output.
Proposal 1: drivers responsible to manage buffers
The application code would be something like this:
uint8_t audio_frame[AUDIO_FRAME_SIZE];

spi_read(spi_dev, audio_frame, AUDIO_FRAME_SIZE);
i2s_send(i2s_dev, audio_frame, AUDIO_FRAME_SIZE);

User-friendliness is perfect - code is easy to write, understand, and maintain. Performance varies: in some socs spi_read() and/or i2s_send() would need to allocate bounce buffers and memcpy() data around; in other socs audio_frame buffer could be used directly by DMAs. Memory consumption and stability vary, depending if bounce buffer allocators are static or dynamic.
Proposal 2: application responsible to manage buffers
Code example:
DMM_BUFFER(spi_dev, spi_frame, AUDIO_FRAME_SIZE);
DMM_BUFFER(i2s_dev, i2s_frame, AUDIO_FRAME_SIZE);

spi_read(spi_dev, spi_frame, AUDIO_FRAME_SIZE);
memcpy(i2s_frame, spi_frame, AUDIO_FRAME_SIZE);
i2s_send(i2s_dev, i2s_frame, AUDIO_FRAME_SIZE);

User-friendliness is a little worse - using macros to allocate buffers is not intuitive. Performance is worse: we always have at least one memcpy() even if it's not needed in a target SoC. Stability is perfect with static memory allocation we cannot run out of buffers. Memory consumption may be better or worse than in proposal 1 - it depends if the target SoC would require bounce buffers.
Merged proposal: the application may manage buffers using macros, but the responsibility is in the drivers.
With this approach, the application could be implemented in multiple ways. One example would be exactly like "Proposal 1", another exactly like "Proposal 2", sharing all of its attributes.
One more which I think is the most interesting would be:
// Allocate statically buffer for SPI, because buffers written by DMA are usually more restricted than those read by DMA
DMM_BUFFER(spi_dev, audio_frame, AUDIO_FRAME_SIZE);

// Application ensures that the buffer sent to read SPI data can be used by SPI's DMA
spi_read(spi_dev, audio_frame, AUDIO_FRAME_SIZE);
i2s_send(i2s_dev, audio_frame, AUDIO_FRAME_SIZE);
// The buffer passed to I2S is not guaranteed to be usable by I2S's DMA

User-friendliness is similar to "Proposal 2" - using unintuitive macros to allocate buffers. Performance is perfect - the SoC-specific code in the drivers will ensure that we avoid memcpy() if possible. Stability and memory consumption vary between platforms depending on the buffer allocation strategy for I2S.
But what I think is most important in this solution is that we do not enforce "stability over performance" or "performance over user-friendliness". It's the application developer who decides which property is the most important for their application.
Other benefits of the merged proposal are:

no changes in the device driver model and existing device driver APIs
no changes in existing device drivers
no changes needed in existing samples or subsystems

Required changes:

new macros in device driver APIs to allow applications to allocate DMA-compliant buffers


Add string literal mutations

Mutates things like 'foo' to 'foo__mutest__'.
This now matches what we do with symbols.
Also removes a dead constant in the symbol mutator I noticed while implementing this.

[also for historical context]
My follow up thought to that is that while the symbol case is probably more useful, mutating a string still make a lot of sense for when you use string hash keys, arguments to a method that accepts a string (imagine a config option takes strings), etc. even if something more like an error message might not need the precision of asserting the entire string.
I'm running it on our work codebase now to see if we get any alive mutations, but I think this is definitely worth adding in general. :)


New union-based shard headers/commitments representation

Based on discord discussion, special thanks to @Nashatyrev for the suggestion of swapping pending-headers and confirmation data with a Union. This PR implements that, along with a special state for unconfirmable headers, and updates of the processing functions.
The state now contains a single field, a table with:

256 slots (TBD) lookback. This is a ring-buffer, so merkle proofs are stable throughout the slots (until the data is overwritten, beyond 256 slots).
a List per slot, to just store the active-shards, instead of a super large vector
a Union per shard/slot combination, to switch data depending on the shard-header-status

The selector can equal 3 status values:

0, UNCONFIRMED_SHARD_DATA: when pending shard headers are not confirmed within time, they get stale (no attestations can increase the vote bitfield anymore). The state-transition cleans this up by switching it to unconfirmed. It's a sane default as well. We also use this to mark shard/slot combinations that can never get a pending shard-header to begin with (when there is no committee assigned to this (shard, slot) combi).
1, CONFIRMED_SHARD_DATA: when a 2/3 majority is reached in the committee on any of the pending shard headers, it's confirmed, or the most-voted header at the end of the epoch. We reduce it to just track the commitment. The position within the state implies the slot and shard.
2, PENDING_SHARD_DATA: when we are processing attestations, we may see multiple different shard headers for the same (slot, shard). These are likely slashable, but the committee can still confirm any of them.

Additionally, this PR:

renames epoch processing functions to read like {type of change}_{status of target data}_shard_{name of change}. Previously the "header" wording was confusing with non-sharding functions that live in the same namespace.
Fix doc section structure

This PR is a work in progress, using the new state representation, I'm also trying to address related issues:

Not all active shards may have a committee (also see #2452) Mostly resolved I think. 2452 can address the P2P part.
The list-length of active shards per slot needs to be initialized correctly Done.
The "dummy" pending shard-header is weird. Either need to remove it and handle it differently, or change the confirmation process (dummy headers should not be rewarded, and should be marked as unconfirmed) Done.
Maybe we should change get_active_shard_count to refer to the state. It is not ideal to keep the state data-structure in sync with the configuration.

In terms of beacon-state serialization, especially if we have a long period of headers, I prefer to minimize the unconfirmed headers to 1 byte (the Union selector). And it also makes a merkle proof to the Union selector sufficient to check the confirmation status. The empty PendingShardHeader() works to track the special case during beacon-processing, but I think we shouldn't impose that on users who just need the status of the shard data.

Work in progress:

Effective balance can change between epochs during shard confirmation, and thus the pending-header weight can get inaccurate, due to different inclusion time of attestations
An effective balance change during epoch transition can affect confirmation status. But confirmation processing happens before effective balance changes, so confirmation may be delayed to an epoch later in an awkward way
When shard data is explicitly unconfirmed, attestations for that data must still be valid. The assertion of pending-shard-header would break that.


Fixed broken joystick axis mappings under Linux

Description
Fixes broken joystick axis mappings on Linux. Without the change ioctl returns -1 and errno is set to INVALID_ADDRESS
This PR is related to the issue #3166
Tasks

[X] Tested on Linux
[ ] Tested on Windows
[ ] Tested on macOS
[ ] Tested on iOS
[ ] Tested on Android

How to test this PR?
Just run the Joystick example and look at the output for each axis.

I tested this locally on Ubuntu with the same joystick device as @AlexAUT, I was able to reproduce the issue. Once the fix was applied it worked as intended. Good find! 🎉

SFMLBuildMaster: Build this please.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

0 of 1 (0.0%)  changed or added relevant line in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.008%) to 55.902%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

src/SFML/Window/Unix/JoystickImpl.cpp
0
1
0.0%

Totals

Change from base Build<PHONE_NUMBER>:
-0.008%

Covered Lines:
11700

Relevant Lines:
19698

💛  - Coveralls

The two failing CI jobs have just been fixed so I need you to rebase on master then we can merge this once CI succeeds. Looks like you opened your PR from your own master branch which means I'm not allowed to update it myself.

Rebased

SFMLBuildMaster: Build this please.


[Bug] When I use Doris' hive catalog to query the hive table, the Doris be process crashes

Search before asking

[X] I had searched in the issues and found no similar issues.

Version
1.2.7
What's Wrong?
Preconditions:
1.Modify the hive field from bigint to string;
Is normal when the hive field type havent chenaged.
When I use Doris' hive catalog to query the hive table, the Doris be process crashes,Simultaneously obtaining the following exceptions:
RpcException, msg: org.apache.doris.rpc.RpcException: io.grpc.StatusRuntimeException: UNAVAILABLE: Network closed for unknown reason
Using presto to query for errors as follows:
Malformed ORC file. Can not read SQL type varchar from ORC stream .age of type LONG
What You Expected?
Is normal to use doris hive catalog,without exception.
How to Reproduce?
1.Create a hive table with orc type:
CREATE EXTERNAL TABLE IF NOT EXISTS test_qa.testcatalog(     idstring,    namestring,    age bigint    )   ROW FORMAT SERDE     'org.apache.hadoop.hive.ql.io.orc.OrcSerde'   STORED AS INPUTFORMAT     'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'   OUTPUTFORMAT     'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat';
2.Insert into test_qa.testcatalog values('a','a',1);
3.Query by Doris is ok:
select * from hive.test_qa.testcatalog;
4.Change the table field age type,from bigint to string:
alter table test_qa.testcatalog change column age age string;
5.Query by Doris,I get a exception,bucause be crashes:
RpcException, msg: org.apache.doris.rpc.RpcException: io.grpc.StatusRuntimeException: UNAVAILABLE: Network closed for unknown reason
Anything Else?
No response
Are you willing to submit PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Can you provide be.out ?


Deletion of certain sections from a pattern

The new function delete_range was added in pattern.py. To test it, another function test_delete_range was implemented in test_pattern.py.

lgtm


Export from script to database: Auto assing table

In https://github.com/dbeaver/dbeaver/issues/2855 you added export from script. When I use this feature to export to another database, however, column automapping does not work even if the query is as simple as select * from table, because dbeaver tries to map to destination table "select * from table" instead of just "table" (see screenshot below)

The information should be available, since if I export the same script to csv, and use ${table.name}, dbeaver is capable to name the output file after the exported table (TABLE.timestamp.csv, without "SELECT * FROM").
If relevant, my database is oracle, and I'm fully qualifying tables to avoid issues in metadata retrieval (e.g. schema.table)

When you export from database to database you can create a custom name for a new table (double click on target field or press New btn).

The different ticket exists for export to CSV with a custom name https://github.com/dbeaver/dbeaver/issues/8033


Hello can this project support the cowboy REST part?

According what I learned, the cowboy REST are not directly define the HTTP method such as "PUT", "GET", "POST", It uses the methods such as  allowed_methods. I just wander that this app can supports the cowboy REST.
Thanks a lot.

@T-tssxuan I think what you're looking for is something like sumo REST.

thanks, a lot.


Code is ugly, rewrite it completely

It wasn't that bad :)  I've seen substantiallllly worse little projects ;)


Use SMART resolver strategy when parsing timestamps and dates

This one is really hard to deal with. Since internally we have used yyyy-MM-dd HH:mm:ss.S as a default SimpleDateFormat there's no way to keep STRICT parsing and not accidentally cause a failure. This will require rewriting data with any timestamps that have csv

Apologies for the typo - fixed along with the tests.


Peppermint, Blood Red Dying, Cleanup and assorted things

Added Peppermint
Rebalanced the Trick or Treat loot table
Switched the food stats of Rare Candy and Winged Strawberry Candy
Added British English and Australian English translations
Changed Spooktober command to say 'Spooktober 31st'
Added translations for biomes that were missing them
Reformatted code

Throwing stuff into blood now dyes it red
Abstracted the fluid crafting stuff
Properly added fluid tags
More cleanup


Fix for Swapchain VK_SUBOPTIMAL_KHR crash on Linux

A possible fix for my Issue #99.
By recreating the Swapchain as a result of VK_SUBOPTIMAL_KHR in addition to VK_ERROR_OUT_OF_DATE_KHR the next call to
vkAcquireNextImageKHR doesn't fail anymore.
In addition i prevented the validation error "VUID-VkSwapchainCreateInfoKHR-imageExtent-01274" to cause an assertion error.
This is neccesary due to an "oversight" in the specification https://github.com/KhronosGroup/Vulkan-ValidationLayers/issues/1340.
If the window is resized between the call to get the new window size and the creation of the Swapchain this validation error is triggered.
This cant be caught, but the application recovers (if we don't kill it via assertion).
It may be a good idea to not log a critical message in check  for VK_SUBOPTIMAL_KHR, this I have not implemented.

Thanks for the fix and the reference to the other issue in the ValidationLayers. :+1:
I think it's okay to still log this message, since it happens quite rarely and is a good clue to the underlying problem. I changed the log level for VK_SUBOPTIMAL_KHR from critical to warn now.


Set the raw data: Any other reason besides setting a password?

    Any other reason besides setting a password?

I extended the Entity class by adding private properties. abandoned the magic of setters/getters - created get* set*
Now setAttributes cannot call set*
So fill() is the better solution.
Regarding the password hash - this operation is done in the beforeInsert Model or by setting the hash yourself
Originally posted by @neznaika0 in https://github.com/codeigniter4/CodeIgniter4/issues/2011#issuecomment-1274160221

Maybe return solution? And ohter drivers
        if (is_subclass_of($className, Entity::class)) {
            return empty($data = $this->fetchAssoc()) ? false : (new $className())->fill($data)->syncOriginal();
        }

If you completely remove the check on the Entity (and the private class), then fetch_object () does not call setters, it will immediately assign values to private properties. Therefore, the situation needs to be corrected.
If I do fill() then everything works fine - private properties are set and $attributes are filled in my setters

It's probably more of a mistake. I posted on the forum


Fix Formatting public and Plugin myinfo

Format Document:

Hello, thanks your for this, but I don't get it, your picture "Format Document" shows that this doesn't work,
public Plugin myinfo =
    {
    name = "Test",
    author = "Test",
    description = "Test",
    version = "1.0.0", // "<IP_ADDRESS>",
    url = "Test"}

is not correct

Picture "Format Document" displays how it is currently formatting the document.
This fix correctly formats the document: "public Plugin myinfo", and removes the problem of lines 2 to 6.
Current format document:

Fix format document:

I have tested your code incorrectly, I don't know how.
This works great, I'm merging this.
Thanks again!


fix : Fixing Flight A detection when having jump boost

FastPlace Fix : Tested on a factions server, a player builds sand walls while being in a minecart and keeping the right click pressed creating false positives with the check.
Flight A Fix : Like Flight B, Flight A must take into account the jumpboost effect. When a player takes velocity (for example when a custom monster sends him into the air with an attack) the check does not take into account this velocity. Also, when a player is teleported back like in "safezone" it is consigned as in flight.
Motion D : A player moving at high speed in a boat makes false positives.
Interact B : When a player places blocks (use bug blocks) in a region where the BlockPlaceEvent is cancelled the player is considered to be in fly. It really flies for the server (2/3 blocks high) but it overloads with reports for an action very often performed by players (especially in factions).
OmniSprint : False positives when a player is moving diagonally in liquid ? Should we really check the speedhack in the water?
I have disabled a lot of detections that are not essential in my situation (Jesus, badpackets, fastclimb, inventory, and more...) because we are using a modified client and a custom spigot. I am doing this pull request only on the "types of detections" that I have had the opportunity to test. I use several of your detections in my personal anti-cheat.
These detections are tested on a network with +1000 players connected simultaneously.
Good luck!

Thanks for testing it on a real server and sharing your issues as well as fixing them for me, I greatly appreciate this and I give my thanks!

Thank you for disabling checks that might false flag with a custom client many thanks. I hope more people like you come to this community.

No problem, you do a good job, it's the least you can do to participate when you use someone else's work!


Package requires newer setup tools

Python 3.7.3 on windows includes  setuptools 40.8.0 which is too low for some tenserflow components.
Before installing the package run
%>pip install -U setuptools
This will install setuptools 41.0.0 so the install runs smoothly.

Thanks @eduh
For future reference: Solve this issue by upgrading setuptools with  pip install --upgrade setuptools>=41.0.0


Exam monitoring: Initial dashboard

Checklist
General

[ ] I tested all changes and their related features with all corresponding user types on a test server.
[x] Language: I followed the guidelines for inclusive, diversity-sensitive, and appreciative language.
[x] I chose a title conforming to the naming conventions for pull requests.

Client

[x] I followed the coding and design guidelines.
[x] I added multiple integration tests (Jest) related to the features (with a high test coverage), while following the test guidelines.
[x] I added authorities to all new routes and checked the course groups for displaying navigation elements (links, buttons).
[x] I documented the TypeScript code using JSDoc style.
[x] I added multiple screenshots/screencasts of my UI changes.
[x] I translated all newly inserted strings into English and German.

Motivation and Context

As part of my thesis, we also want to introduce the client dashboard for the exam monitoring. This pull request presents the first initial version. This version will be updated and extended in several further pull requests.
Description

Don't be afraid of the amount of new code added - there are also many parameterized tests :D

To view the dashboard, you need to enable exam monitoring and ...

... go to the Exam Management and see the newly added button.

The Dashboard contains multiple (sub)sections.

Overview: Shows the most important information of each subsection (clickable header, real-time charts, ...)

The exercise bar charts are updated every 5 seconds (to avoid too many re-renders)
The action line charts are updated every 10 seconds (to avoid too many re-renders)

Exercise/Activity-log: Will be implemented in a follow up (since this PR is already too big)
...

Each action performed by the student is sent to the server via websockets and then sent to the exam monitoring dashboard via websockets. As you can see, we have two functions which updates the data of each chart. At the moment, both functions behave very similarly. This will be changed in a follow-up to improve performance and to allow a reload/initial load.

initData: This functions is called on the initial load of the chart. We receive all performed actions and prepare them for the chart (implemented in a follow up).
updateData: This function is called on every update. It should only update the chart and not perform any data preparation (since we don't get any initial data at the moment, this function behaves like initData and will be adjusted in a follow up).

Future PRs:
In order to get this PR merged (and since this PR is already really large), we move some topics into additional PRs.

Add initial loading
Improve performance (if required)
Add additional sections (exercises, activity log, ...)
Improve tooltips
Fine-tuning of actions
...

Steps for Testing

Prerequisites:

1 Instructor
Multiple Students
1 Exam with multiple exercises

Log in to Artemis
Navigate to Course Administration
Create an Exam with multiple Exercises and multiple registered Students
View the Exam Monitoring Dashboard as Instructor during the exam
Participate with multiple students during the exam
Verify that the collected actions are presented in the Dashboard

Review Progress

Code Review

[ ] Review 1
[ ] Review 2

Manual Tests

[ ] Test 1
[ ] Test 2

Test Coverage

Class/File
Branch
Line

exam-monitoring-websocket.service.ts
57.14%
71.42%

exam-monitoring.component.ts
58.33%
100%

exam-monitoring.service.ts
50%
81.81%

chart.title.component.ts
100%
100%

monitoring-chart.ts
83.87%
97.5%

actions-chart.component.ts
100%
100%

average-actions-chart.component.ts
75%
94.73%

category-actions-chart.component.ts
75%
96.87%

total-actions-chart.component.ts
75%
95.23%

exercise-chart.component.ts
100%
93.75%

exercise-group-chart.component.ts
87.5%
95.23%

exercise-navigation-chart.component.ts
100%
95.45%

exercise-submission-chart.component.ts
100%
95.45%

exercise-template-chart.component.ts
100%
100%

monitoring-card.component.ts
100%
100%

monitoring-overview.component.ts
75.67%
100%

Screenshots

Exam Monitoring Dashboard (without data)

Exam Monitoring Dashboard (connected to server)

Tested on TS2, so far so good. Code also looks nice. Some suggestions:

Add a Beta label next to the page title, so one immediately notices that the feature is not finished
Fix the button layout, looks very weird

@ge65cer thanks for your review! This feature is only usable if you are admin right now - so i guess we don't need a beta tag (but i like the suggestion, maybe we will still add one) :D

We perform an additional testing session tomorrow evening - feel free to hit me up if you want to join :+1: (@ reviewers)

Thanks for participating in our two testing session for this PR @babyygemperor @TheTrueVester @rhubinger @mallgaier - some last re-approvals and we are ready to merge 🥳

Re-testing and re-approve is not required as this was only a merge conflict with no impact on the exam monitoring :+1:

We performed an additional testing session with @willergeorg and @akesfeden and everything works as expected :+1:


is there a way to parallel image wait and CPLD read operation?

Now I'm using single shot mode. It takes ~100ms to get a 320x240 image ready and ~150ms for me to read it out via 8M SPI. I want to make this two steps in parallel.  When I try to adjust the sequence of operation, I can't figure out how to deal with the clear_fifo_flag() operation.
Below is a pseudo process.
while(1)
    1. Trigger
    2. wait for image ready (100ms)
    3. read length
    4. burst read data (150ms)
    5. clear fifo

Change to
0. Trigger (first image)
while(1)
    1. wait for image ready (only first time need 100ms)
    2. read length
    3. trigger (a new image)
    4. burst read data (150ms)
    5. clear fifo (this is key..if I do not clear fifo, I will get corrupt data, if I clear fifo I will lost the new image and wait at 1) forever)

CPLD does not support this synchronization mode currently , and we may develop this mode in a future release.

@xiongyu0523
If the quantity required is large, we can customer it for you.

Thank you. I'm just doing a prototype.


Handle lists and non-null properly

Hello,
Right now gql-gen doesn't check if ! (non-nullable) applies to a list or elements inside the list:
https://github.com/dotansimha/graphql-code-generator/blob/f3f6e0aa4d482bf407e507267424e363d1d32758/packages/graphql-codegen-core/src/schema/resolve-type.ts#L10
As a result, the list is considered required even when it isn't:

GraphQL type
Expected Typescript Type
Actual Generated Type

[String]
(string | null)[] | null
string[] | null

[String]!
(string | null)[]
string[]

[String!]
string[] | null
string[]

[String!]!
string[]
string[]

More generally, GraphQL allows arbitrary nesting of list and non-null modifiers according to http://graphql.org/learn/schema/#lists-and-non-null .

I ran into the same problem.

Yea just encountered this too. Should be pretty simple fix seeing that code, something like lastIndexOf(']!') should be sufficient in my opinion.
Unfortunately, it feels like @dotansimha is rather busy lately, he is not responding to issues much. Looks like custom fork will be necessary, but that's going to be a pain considering monorepo structure.

I made the PR. It's a working solution, I have a rather big schema and nothing was broken by that change.

@FredyC Hi, I'd like to try your PR. Could you tell me a little bit about how to use it in my project.
package.json
"graphql-code-generator": "git://github.com/FredyC/graphql-code-generator.git#FredyC-patch-1",

I executed yarn install, but gql-gen command was not created in ./node_modules/.bin/.

@FredyC @dotansimha Thank you 🙌


Refactor names and descriptions of test cases

For example change
"have the standard admin" in {
to
"have the standard admin" should "fail with/succeed with ...." in {

It is okay as it is, because we are using an akka tester.


Embedded graph database mode

Provide ability to choose between stored and in-memory graph db

Grakn no longer supports in-memory. This feature will need to wait for #7


Point color dimension option for MapBox map

Dimension associated color scheme option for MapBox point map. Continuous and categorical color themes would add value to the plot besides just the one fixed color option.

@stanbiryukov please fill out the issue template. :)


Implement the collectionFormat attribute (fixes #23)

Note that, strictly speaking, this change is not backward-compatible: At the moment, swagger1st implicitly uses a limited form of collectionFormat: multi for every array parameter, regardless of the specified collection format (and it breaks on one-element arrays). The Swagger spec defines collectionFormat: csv as the default, though, so this commit changes the default behavior accordingly.

Coverage increased (+0.4%) to 84.342% when pulling 85e38c6504ff58d48dabbc7e985bea2b917a402c on sebastianpoeplau:feature/collection-format into ffe3da5503e1a6fe4b2f16b1a7b1bb79d20602b5 on sarnowski:master.

:+1:

👍

:+1:


CLOUD-63074 only consider DATANODE when counting the replication factor

@akanto
This PR allows every hg which does not have a DATANODE to scale down to 1 node.

Refer to this link for build results (access rights to CI server needed):
http://ci.sequenceiq.com//job/cloudbreak-pull-request/2925/

Refer to this link for build results (access rights to CI server needed):
http://ci.sequenceiq.com//job/mock-integration-test/420/


How to build jar file

i managed to get the clone of the project, but i did not understood the next steps, what do you mean by
Configure where your Android SDK resides ? how do i do that? Please help

You need to have the Android SDK installed on the build-machine, see e.g. the section "Get just the command line tools" at https://developer.android.com/studio/index.html and when compiling you need to define the location where you installed the SDK to.


git-server: migrate from warp to axum

Both git clone and rad sync work with the new implementation. However I was not able to test any request which used PeerId in its Uri to check the correctness.
In what case is PeerId used?


Add upgrade timeout rpc to 04-channel protobuf Msg service

Summary

It is possible for the channel upgrade process to stall indefinitely on UPGRADE_TRY if the UPGRADE_TRY transaction simply cannot pass on the counterparty; for example, the upgrade feature may not be enabled on the counterparty chain.

In this case, we do not want the initializing chain to be stuck indefinitely in the UPGRADE_INIT step. Thus, the UpgradeInit message will contain a TimeoutHeight and TimeoutTimestamp. The counterparty chain is expected to reject UpgradeTry message if the specified timeout has already elapsed.

A relayer must then submit an ChanUpgradeTimeoutMsg message to the initializing chain which proves that the counterparty is still in its original state. If the proof succeeds, then the initializing chain shall also restore its original channel and cancel the upgrade.

rpc ChannelUpgradeTimeout(MsgChannelUpgradeTimeout) returns (MsgChannelUpgradeTimeoutResponse);

Implement the ChannelUpgradeTimeout functionality as defined in ICS004 - Upgrade Handshake:

[x] Add ChannelUpgradeTimeout functionality to IBC core Msg server implementation, #3749
[x] Add ChanUpgradeTimeout to 04-channel keeper #3747
[x] #3748
[x] #3750

For Admin Use

[ ] Not duplicate issue
[ ] Appropriate labels applied
[ ] Appropriate contributors tagged/assigned

seems like this wasn't closed, closing!