IPC-550: Finality vote tally

Closes #550
Adds an STM enabled VoteTally component to fendermint_vm_topdown.
The way it works is that it keeps track of the finalized parent chain as observed by this node, that is, a single vector of block hashes that is never expected to be rolled back. This is slightly controversial, as there can be reorgs on Lotus, however it's not any more controversial than the way we handle finality: by waiting until a block is sufficiently embedded, then executing the messages up to that point in our instant-finality forward-only ledger. The ledger itself couldn't handle reorgs either.
What that in mind, it is expected that the top-down syncer will add new finalized blocks to this component, while it will also receive votes from the gossip channel from other validators. Some of these votes will be about blocks we have not observed, which is why the votes go into a map indexed by height and hash until the time comes to sum them up looking for a quorum. At that point we go from the latest block we observed back towards the last finalized block in the ledger and sum up the votes until we find a height which has quorum. Every vote is implicitly voting on all previous heights.
Note that even if we know about votes in future blocks, unless these are heights that we consider final, we don't consider them. That is because there is no evidence that these are on the came chain as the one we see. On the other hand if we get to the same height and see the same hash, then necessarily all previous hashes must match as well. We trust the syncer to feed us a chain. So, even if the syncer knows more blocks that lead to this vote, unless they are final this component ignores them.
A conceptual illustration:
PC: B0 - B1 - B2 - B3 - B4
--------------------------
V1:           B2                     B7
V2:      B1   B2                 B6
V3:      B1             B4   B5

Say we observe B0 - B4 on the parent chain, and below that we see the votes we received from validators V1, V2 and V3. Say our last finalized block was B0. In this case we'll find quorum for B3 because V1 and V2 explicitly voted for it, and V3 voted for B4, and we see that B2 was indeed an ancestor of B4 in the PC chain. However, we don't consider B5 final, because even though V1 and V2 have voted for B7 and B6, we don't actually know whether there has been a network partition until we ourselves see these blocks being added to PC.
Even if we proposed B5, we couldn't execute it until we have it in PC, so if the validators want to finalize these blocks they can do it on their own, at which point we'll be forced to wait until we have it, but that's a different story.
TODO:

[x] Add a view of the parent chain from our perspective
[x] Add votes coming from validators (including us)
[x] Find a block that we think is final and has enough votes to pass quorum
[x] Detect equivocations
[x] Clear out finalized blocks
[x] Update power table
[x] Property based testing

In this case we'll find quorum for B3 because V1 and V2 explicitly voted for it

I think you mean to say B2, right?


Use a raw syscall for memfd_create on gnu

I think this is related to https://github.com/nix-rust/nix/issues/1972
I want this because I'm trying to use ctrlc in the compiler (https://github.com/rust-lang/rust/pull/111769), which uses this crate, and the feature set that ctrlc uses pulls in this function.
The compiler is primarily built as a shared object, and it is linked against an old version of glibc (2.17) so there is no memfd_create symbol available when it is built. Using a raw syscall provides compatibility with such old glibc versions.

Are there plans to merge this MR? I've also hit this issue and would appreciate this change being merged.

No, because I still think it's the wrong solution.  As described in the issue, I think it's a linker problem, and we need somebody to explore the linker more.

I agree with you, when nix is being used in a binary. But rustc is compiled as a shared library so linker dead code elimination is significantly less effective.

Wait, rustc itself is consuming Nix?  That's surprising.  It seems like a dependency loop situation.  I thought that rustc had very few external dependencies.

I am trying to add this crate to the compiler's list of third-party dependencies. This crate is a dependency of ctrlc which is a very nice way to set up the signal handling that I need for the rustc PR that I linked in the description of this PR: https://github.com/rust-lang/rust/pull/111769
I don't think it's quite accurate to say that the compiler has very few external dependencies. It certainly does not have a lot and dependencies are added rather carefully, but everyone who works on the compiler and the core tooling have done a lot of work to enable people to build a robust package ecosystem, so it doesn't really make sense to deprive ourselves of the fruits of our labor.

My use case is similar to the one mentioned in the ticket. I'm building a static library in Rust that exposes C bindings for downstream C/C++ libraries to use. The consumers of my library build on CentOS7 with glibc 2.17, which causes a linking failure when using my library due to the missing memfd_create symbol.
I would say this is more of a portability issue than a linker issue. Using the memfd_create function forces a minimum libc version of 2.27 on libraries that use this crate.

My use case is similar to the one mentioned in the ticket. I'm building a static library in Rust that exposes C bindings for downstream C/C++ libraries to use. The consumers of my library build on CentOS7 with glibc 2.17, which causes a linking failure when using my library due to the missing memfd_create symbol.

This bug makes a little more sense if it only affects cdylib projects.

I would say this is more of a portability issue than a linker issue. Using the memfd_create function forces a minimum libc version of 2.27 on libraries that use this crate.

Do you mean glibc 2.27?

I still don't like hacking Nix to workaround a glibc bug.  But looking closer, I realize that memfd_create probably shouldn't be gated by the fs feature.  I'm not sure why it is.  It's really closer to the mman feature.  Not that it's a 100% solution, but would changing it's feature gate fix your problem, @saxon-parker ?  I've already verified that it would fix the problem for the ctrlc crate.

This is not a glibc bug. We are both targeting CentOS 7, which predates memfd_create. That system call was added in Linux 3.17 or glibc 2.27, and CentOS 7 is Linux 3.10 or glibc 2.17.

I agree that memfd_create is probably under the wrong feature, but I didn't initially propose changing that because I figured it would be a breaking change. One possible way around that is to give the fs feature a sub-feature that turns on everything except memfd_create, then have ctrlc depend only on that new sub-feature (I was thinking fs-minimal or fs-compat or something like that).

This is not a glibc bug. We are both targeting CentOS 7, which predates memfd_create. That system call was added in Linux 3.17/glibc 2.27, and CentOS 7 is Linux 3.10/glibc 2.17.

Yeah, but new syscalls get added all the time.  Why is this the only one that causes linker failures?

I still don't like hacking Nix to workaround a glibc bug. But looking closer, I realize that memfd_create probably shouldn't be gated by the fs feature. I'm not sure why it is. It's really closer to the mman feature. Not that it's a 100% solution, but would changing it's feature gate fix your problem, @saxon-parker ? I've already verified that it would fix the problem for the ctrlc crate.

Unfortunately  moving memfd_create to the mman feature wouldn't solve my problem. nix is brought into my library as a dependency of the mmap-rs crate, which uses the mman feature.
I can get around this issue in other ways, so if this MR doesn't get merged it's not a huge deal for me.

Yeah, but new syscalls get added all the time. Why is this the only one that causes linker failures?

The Rust standard library handles use of system calls by using a combination of weak linkage and syscall: https://github.com/rust-lang/rust/blob/fcf3006e0133365ecd26894689c086387edcbecb/library/std/src/sys/unix/weak.rs

Yeah, but new syscalls get added all the time. Why is this the only one that causes linker failures?

The Rust standard library handles use of new system calls by using a combination of weak linkage and syscall: https://github.com/rust-lang/rust/blob/fcf3006e0133365ecd26894689c086387edcbecb/library/std/src/sys/unix/weak.rs

I mean that new syscalls get added to Nix all the time.  For example, I can build a cdylib that uses Nix with the "fs" feature, which enables the fspacectl function, on FreeBSD13.  That works even though the fspacectl wasn't added until FreeBSD 14.  And I can link that cdylib into a running program without problems.  That's why I think this bug involves linker settings on your Linux distro.  Or else why is memfd_create the only syscall that causes this problem?

You must pass the linker flag --no-allow-shlib-undefined in order to get this linker error. I don't know the first thing about FreeBSD so I can't comment there, but the only syscall wrapper I can find that this create has added since glibc 2.17 that is actually used in all the Rust programs installed on my machine (I just ran nm | grep over all of them) is copy_file_range, and this crate implements that by calling libc::syscall.

How do you pass that linker flag?  For me, the following program builds fine as a cdylib on both FreeBSD and Linux.  Can you cause it to fail with that flag?
mod ffi {
    extern "C" {
        pub fn does_not_exist() -> libc::c_int;
    }
}

pub extern "C" fn foo() -> i32 {
    unsafe { ffi::does_not_exist() }
}

You must pass the linker flag --no-allow-shlib-undefined in order to get this linker error. I don't know the first thing about FreeBSD so I can't comment there, but the only syscall wrapper I can find that this crate has added since glibc 2.17 that is actually used in all the Rust programs installed on my machine (I just ran nm | grep over all of them) is copy_file_range, and this crate implements that by calling libc::syscall.

This does not appear to be true when actually linking the final binary using that shared library with missing symbols (on both my nixos and the debian  rust:latest container. While the dylib builds fine unless I manually pass --no-undefined (with RUSTFLAGS=-Clink-arg=-Wl,--no-undefined), the final binary fails by default with both binutils ld and ld.lld.
Here's a repro: https://github.com/rust-lang/rust/pull/111769#issuecomment-1669525104

Oooh, it's "dylib" crates that you're worried about.  I thought it was "cdylib".  Now that I try using a dylib I can reproduce the problem.  But I like the #[inline] solution better than the syscall one, because it preserves type safety.  It also has the advantage that it will rightly fail at link time on systems that lack memfd_create, if the crate actually tries to use that function.  Could you restore the PR to that version please?


Regenerate with latest gir

Depends on https://github.com/gtk-rs/gio/pull/258

@GuillaumeGomez This is good to go, CI will fail until the examples are fixed. Please merge :)


Changed inline code from Powershell to PMC

Fixes #15069
Suggestion only :)

Thank you for the PR; however, PowerShell is what we should be using to correctly identify the cmdlets being executed in PMC. Our syntax highlighting system actually doesn't recognize PMC as a valid value.


strace-4.15 hangs on launch

Your Windows build number: 16215

What you're doing and what's happening:

$ strace -V
strace -- version 4.15
$ strace ls
<hangs>

Strace of the failing command, if applicable:

Using strace-4.9 to trace strace-4.15
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fe7bec809d0) = 14731
rt_sigaction(SIGTTOU, {SIG_IGN, [], SA_RESTORER, 0x7fe7be693030}, NULL, 8) = 0
rt_sigaction(SIGTTIN, {SIG_IGN, [], SA_RESTORER, 0x7fe7be693030}, NULL, 8) = 0
rt_sigaction(SIGHUP, {0x7fe7bee75990, [], SA_RESTORER, 0x7fe7be693030}, NULL, 8) = 0
rt_sigaction(SIGINT, {0x7fe7bee75990, [], SA_RESTORER, 0x7fe7be693030}, NULL, 8) = 0
rt_sigaction(SIGQUIT, {0x7fe7bee75990, [], SA_RESTORER, 0x7fe7be693030}, NULL, 8) = 0
rt_sigaction(SIGPIPE, {0x7fe7bee75990, [], SA_RESTORER, 0x7fe7be693030}, NULL, 8) = 0
rt_sigaction(SIGTERM, {0x7fe7bee75990, [], SA_RESTORER, 0x7fe7be693030}, NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [HUP INT QUIT PIPE TERM], NULL, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
wait4(-1, [{WIFSTOPPED(s) && WSTOPSIG(s) == SIGSTOP}], __WALL, NULL) = 14731
rt_sigprocmask(SIG_BLOCK, [HUP INT QUIT PIPE TERM], NULL, 8) = 0
ptrace(PTRACE_GETREGSET, 14731, NT_PRSTATUS, [{0x7fe7bf1066a0, 216}]) = -1 EINVAL (Invalid argument)
ptrace(PTRACE_GETREGS, 14731, 0, 0x7fe7bf1066a0) = 0
ptrace(PTRACE_SETOPTIONS, 14731, 0, PTRACE_O_TRACESYSGOOD|PTRACE_O_TRACEEXEC|PTRACE_O_TRACEEXIT) = -1 EINVAL (Invalid argument)
fstat(2, {st_mode=S_IFCHR|0660, st_rdev=makedev(4, 1), ...}) = 0
ioctl(2, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
write(2, "/usr/bin/strace: PTRACE_SETOPTIO"..., 53/usr/bin/strace: PTRACE_SETOPTIONS: Invalid argument
) = 53
getpid()                                = 14729
kill(14731, SIGCONT)                    = 0
kill(14731, SIGTERM)                    = 0
wait4(14731,

The wait on the subprocess hangs, but only after ptrace(PTRACE_SETOPTIONS, child, ..., PTRACE_O_TRACESYSGOOD|PTRACE_O_TRACEEXEC|PTRACE_O_TRACEEXIT) fails with EINVAL
This appears to be similar to #548, but I cannot reopen it.

It looks like the primary difference between the trace of 4.12 and the trace of 4.15 is PTRACE_O_TRACEEXIT.

It is being tracked in #555. There's further discussion about strace-4.recent in #1578 (message).

Marking as a duplicate of #555.


Let SCF iteration converge for this test, because an SCF convergence …

…failure is now an error and causes an abort.

This PR works fine for me locally using this docker file and it should resolve the current issue with the regression test for the AiiDA-CP2K plugin.
@yakutovicha Please, could you have a look?

I'm afraid the test is still rather flaky as it fails roughly every other time.

Is there no other way? If you mock out cp2k then this test will be far less useful.
Maybe we could just set IGNORE_CONVERGENCE_FAILURE to restore the old behavior?


Unable to build in WSL2

Click to expand!
Issue Type
Build/Install
Have you reproduced the bug with TF nightly?
Yes
Source
source
Tensorflow Version
2.11
Custom Code
No
OS Platform and Distribution
Linux Ubuntu 22.04 in WSL2
Mobile device
No response
Python version
3.10
Bazel version
5.3.0
GCC/Compiler version
11.3.0
CUDA/cuDNN version
11.2/8.1
GPU model and memory
1080Ti with 11GB memory
Current Behaviour?
First, it got two warnings:

WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/tensorflow/runtime/archive/071fc48c32248196b17912cdab206f29fa6ded38.tar.gz failed: class java.io.FileNotFoundException GET returned 404 Not Found
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/llvm/llvm-project/archive/1ec021467da65c06cd4aa24ce81898b0759bd16b.tar.gz failed: class java.io.FileNotFoundException GET returned 404 Not Found

Then, got errors:

ERROR: An error occurred during the fetch of repository 'local_config_cuda':
   Traceback (most recent call last):
        File "/home/someone/code/tensorflow/third_party/gpus/cuda_configure.bzl", line 1430, column 38, in _cuda_autoconf_impl
                _create_local_cuda_repository(repository_ctx)
        File "/home/someone/code/tensorflow/third_party/gpus/cuda_configure.bzl", line 1007, column 35, in _create_local_cuda_repository
                cuda_config = _get_cuda_config(repository_ctx, find_cuda_config_script)
        File "/home/someone/code/tensorflow/third_party/gpus/cuda_configure.bzl", line 747, column 52, in _get_cuda_config
                compute_capabilities = compute_capabilities(repository_ctx),
        File "/home/someone/code/tensorflow/third_party/gpus/cuda_configure.bzl", line 473, column 32, in compute_capabilities
                auto_configure_fail("Invalid compute capability: %s" % capability)
        File "/home/someone/code/tensorflow/third_party/gpus/cuda_configure.bzl", line 353, column 9, in auto_configure_fail
                fail("\n%sCuda Configuration Error:%s %s\n" % (red, no_color, msg))
Error in fail:
Cuda Configuration Error: Invalid compute capability: compute_112
ERROR: /home/someone/code/tensorflow/WORKSPACE:15:14: fetching cuda_configure rule //external:local_config_cuda: Traceback (most recent call last):
        File "/home/someone/code/tensorflow/third_party/gpus/cuda_configure.bzl", line 1430, column 38, in _cuda_autoconf_impl
                _create_local_cuda_repository(repository_ctx)
        File "/home/someone/code/tensorflow/third_party/gpus/cuda_configure.bzl", line 1007, column 35, in _create_local_cuda_repository
                cuda_config = _get_cuda_config(repository_ctx, find_cuda_config_script)
        File "/home/someone/code/tensorflow/third_party/gpus/cuda_configure.bzl", line 747, column 52, in _get_cuda_config
                compute_capabilities = compute_capabilities(repository_ctx),
        File "/home/someone/code/tensorflow/third_party/gpus/cuda_configure.bzl", line 473, column 32, in compute_capabilities
                auto_configure_fail("Invalid compute capability: %s" % capability)
        File "/home/someone/code/tensorflow/third_party/gpus/cuda_configure.bzl", line 353, column 9, in auto_configure_fail
                fail("\n%sCuda Configuration Error:%s %s\n" % (red, no_color, msg))
Error in fail:
Cuda Configuration Error: Invalid compute capability: compute_112
INFO: Found applicable config definition build:cuda in file /home/someone/code/tensorflow/.bazelrc: --repo_env TF_NEED_CUDA=1 --crosstool_top=@local_config_cuda//crosstool:toolchain --@local_config_cuda//:enable_cuda
INFO: Found applicable config definition build:cuda in file /home/someone/code/tensorflow/.bazelrc: --repo_env TF_NEED_CUDA=1 --crosstool_top=@local_config_cuda//crosstool:toolchain --@local_config_cuda//:enable_cuda
WARNING: The following configs were expanded more than once: [cuda]. For repeatable flags, repeats are counted twice and may lead to unexpected behavior.
ERROR: @local_config_cuda//:enable_cuda :: Error loading option @local_config_cuda//:enable_cuda:
Cuda Configuration Error: Invalid compute capability: compute_112

Standalone code to reproduce the issue
./configure

bazel build --config=cuda //tensorflow/tools/pip_package:build_pip_package

Relevant log output
No response

Can you try with Cuda 11.8 and cuDNN 8.6?

@nitins17 I had tried 11.8 + 8.7 and got the same error. Do I still need to try 11.8 + 8.6?
Combinations I tried:
cuda 11.8 + cuDNN 8.7,
cuda 11.4 + cuDNN 8.7,
cuda 11.2 + cuDNN 8.1.

I solved this problem. The compute compatibility should be 6.1, but I set 11.8. You can close this issue.


config support both path and names

Signed-off-by: husharp<EMAIL_ADDRESS>Part of https://github.com/tikv/pd/pull/5886. We need to update client-go first to update TiDB. Should be merged after https://github.com/tikv/pd/pull/5886
And for integration test, need to change TiDB temporarily

/test

Need to update go-version to 1.19


docs: update "How is the MessageShareCommitment generated?"

Update payment docs to account for how message share commitment is generated post ADR 008.
Note: targets feature branch

Thanks for proposing all the fixes @SweeXordious !


Update React to v15.6.1, update enzyme, add prop-types

https://github.com/patientslikeme/react-calendar-heatmap/issues/34

thanks for the PR @easingthemes!

Published as https://github.com/patientslikeme/react-calendar-heatmap/releases/tag/v1.2.1


ColorArray as face_color errors

🐛 Bug
When trying to use a ColorArray for different point colors an error is raised by napari. Perhaps related to #762.
To Reproduce
points = np.array([[10, 10], [20, 20]])
ca = vispy.color.ColorArray(['red', 'blue'])

with napari.gui_qt():
    v = napari.view_points(points, face_color=ca)  # ValueError

Interestingly, a list as face_color works, i.e. face_color=['red', 'blue'].
Expected behavior
The ColorArray should grant each point its designated color.
Environment
napari 0.2.6, python 3.7, vispy 0.6.2
Additional context
The full traceback from the code is the following, perhaps suggesting that napari doesn't transfer the right array to vispy:
Traceback (most recent call last):
  File "/home/hagai/Code/napari/nan_example.py", line 24, in <module>
    v = napari.view_points(points, face_color=ca)
  File "/home/hagai/Code/napari/napari/view_layers.py", line 233, in view_points
    visible=visible,
  File "/home/hagai/Code/napari/napari/components/viewer_model.py", line 666, in add_points
    self.add_layer(layer)
  File "/home/hagai/Code/napari/napari/components/viewer_model.py", line 386, in add_layer
    self.layers.append(layer)
  File "/home/hagai/Code/napari/napari/util/list/_model.py", line 56, in append
    self.events.added(item=object, index=len(self) - 1)
  File "/home/hagai/Code/napari/napari/util/event.py", line 508, in __call__
    self._invoke_callback(cb, event)
  File "/home/hagai/Code/napari/napari/util/event.py", line 529, in _invoke_callback
    cb_event=(cb, event),
  File "/home/hagai/Code/napari/napari/util/event.py", line 523, in _invoke_callback
    cb(event)
  File "/home/hagai/Code/napari/napari/_qt/qt_viewer.py", line 148, in _add_layer
    vispy_layer = create_vispy_visual(layer)
  File "/home/hagai/Code/napari/napari/_vispy/util.py", line 33, in create_vispy_visual
    visual = layer_to_visual[type(layer)](layer)
  File "/home/hagai/Code/napari/napari/_vispy/vispy_points_layer.py", line 29, in __init__
    self._on_data_change()
  File "/home/hagai/Code/napari/napari/_vispy/vispy_points_layer.py", line 89, in _on_data_change
    scaling=True,
  File "/home/hagai/miniconda3/envs/napari_dev/lib/python3.7/site-packages/vispy/visuals/markers.py", line 553, in set_data
    face_color = ColorArray(face_color).rgba
  File "/home/hagai/miniconda3/envs/napari_dev/lib/python3.7/site-packages/vispy/color/color_array.py", line 148, in __init__
    rgba = _user_to_rgba(color, clip=clip)
  File "/home/hagai/miniconda3/envs/napari_dev/lib/python3.7/site-packages/vispy/color/color_array.py", line 55, in _user_to_rgba
    color = np.atleast_2d(color).astype(np.float32)
ValueError: setting an array element with a sequence.

From some basic debugging, it seems like set_data here is expecting a string or a single tuple as the face_color argument, but instead it has a ColorArray at hand.

Hmm yeah I guess we hardcoded some assumptions there. Given that we currently all people to pass vispy colormaps to the colormap property for custom colormaps, it probably makes sense to allow people to pass a vispy ColorArray to our color like properties.
@kevinyamauchi @jni do you agree with this? In which case we should fix and standardize across all the layers which accept colors (Vectors, Shapes, Points) and add tests to each layer

I'll be happy to tackle this PR myself with a bit of guidance.

Yep, I'm 👍 on this proposal. Thanks for reporting @HagaiHargil and even more for offering to fix! =)

Thanks for raising this, @HagaiHargil ! I agree that we should allow people to pass custom colors to set *_color properties in our layers. I have listed a few thoughts below. Please let me know what you all think!

Do we want users to pass ColorArrays or is it preferable to allow users to pass RGBA tuples (or some other format)? As far as I can tell, from the user perspective, the vispy ColorArray would basically be a container for the RGBA tuple. If that is the case, accepting the RGBA tuple directly would save the user an import.
It would be nice if there was a "transparent" default so that users can pass something simple (e.g., None or 'transparent') and the color is set with alpha=0 (see #762).
I am not sure the best thing to display in the viewer dropdown menus when the user sets the face_color with a custom ColorArray. Perhaps we add something generic to the list (e.g., "custom") and that gets set when the user sets the parameter with a ColorArray? It could be cool if clicking the color swatch opened up a UI (not a popup :P ) for setting the RGBA tuple. Thoughts?

@HagaiHargil, it would be awesome if you took this on! I am happy to chat more if you have questions.

The only reason I was even aware of vispy's ColorArray is that the documentation for face_color states that users can pass a ColorArray instance for the colors of individual points.
I think that we should be able to allow list-like containers of 3- and 4-tuples, as well as ColorArrays, since they're quite easily constructed from numpy arrays and lists. I'll start tackling this issue soon.

Sounds great! Let me know if you have any questions or run into any issues!


[WIP] Add test setup convenience methods

Adds two convenience methods generate_and_send_coins() and create_spending_transaction(). Using those methods removes a lot of the duplicate code in the notebooks that exists simply to setup the tests.
I've added a chapter 2.0 that explains the structure of the 2.x notebooks and introduces the convenience methods. I've also updated chapter 2.1 to use the convenience methods.
If people agree that this is an improvement, I'll update 2.2, 2.3 and 2.4 to use these methods.
Fixes #92

Addressed all of @bitschmidty's feedback (thanks!) : https://github.com/bitcoinops/taproot-workshop/compare/f849dddd175d8ee1979baecd4b56d6205ee75786..cd5f378fa2754f716b67834c3a0564ebf9d2baf1

Rebased on master

I've rebased this on #126, because that makes changes in a lot of places. Please review that first.

Thanks for the review @bitschmidty ! I've fixed all your issues.

Addressed @jachiang's feedback (thanks for the review!)

Thanks @bitschmidty !


KYLIN-4697: Fix User info update logic use copyForWrite

Proposed changes
Describe the big picture of your changes here to communicate to the maintainers why we should accept this pull request. If it fixes a bug or resolves a feature request, be sure to link to that issue.
Types of changes
What types of changes does your code introduce to Kylin?
Put an x in the boxes that apply

[x ] Bugfix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Documentation Update (if none of the other choices apply)

Checklist
Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

[x ] I have create an issue on Kylin's jira, and have described the bug/feature there in detail
[ ] Commit messages in my PR start with the related jira ID, like "KYLIN-0000 Make Kylin project open-source"
[ ] Compiling and unit tests pass locally with my changes
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] If this change need a document change, I will prepare another pr against the document branch
[ ] Any dependent changes have been merged

Further comments
If this is a relatively large or complex change, kick off the discussion at user@kylin or dev@kylin by explaining why you chose the solution you did and what alternatives you considered, etc...

Pull Request Test Coverage Report for Build 6417

0 of 17 (0.0%)  changed or added relevant lines in 5 files are covered.
10 unchanged lines in 2 files lost coverage.
Overall coverage decreased (-0.01%) to 28.041%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

server-base/src/main/java/org/apache/kylin/rest/controller/UserController.java
0
1
0.0%

server-base/src/main/java/org/apache/kylin/rest/security/KylinUserManager.java
0
1
0.0%

server-base/src/main/java/org/apache/kylin/rest/service/KylinUserService.java
0
1
0.0%

server-base/src/main/java/org/apache/kylin/rest/service/KylinUserGroupService.java
0
3
0.0%

server-base/src/main/java/org/apache/kylin/rest/security/KylinAuthenticationProvider.java
0
11
0.0%

Files with Coverage Reduction
New Missed Lines
%

server-base/src/main/java/org/apache/kylin/rest/security/KylinAuthenticationProvider.java
1
0%

core-metrics/src/main/java/org/apache/kylin/metrics/lib/impl/MetricsSystem.java
9
60.0%

Totals

Change from base Build 6416:
-0.01%

Covered Lines:
26251

Relevant Lines:
93616

💛  - Coveralls


Clean unnecessary custom_imports in entrypoints

As MMCV already supports custom_imports since PR https://github.com/open-mmlab/mmcv/pull/606. The entrypoints in MMDet no longer need to handle the custom_imports manually. The codes like below should be cleaned.
https://github.com/open-mmlab/mmdetection/blob/c76ab0eb3c637b86c343d8454e07e00cfecc1b78/tools/train.py#L94-L96

Hi, @RangiLyu can I work on this issue?

Hi, @RangiLyu can I work on this issue?
Sure! Welcome to open a PR!


Update README.md

For a quick and better localization of all the repos, all them have been sorted alphabetically in the readme file.

Thanks!

Thanks!

Thank you Arthur!

Thank you Arthur!


Compilation error for python3_script.cc.

Describe the Problem
When compiling on a Mac, I get the following compilation error from python3_script.cc:
python3_script.cc:179:55: error: ordered comparison between pointer and zero ('PyObject *' (aka '_object *') and 'int')
if ( PyRun_String(command, Py_file_input, Dict, Dict) < 0 )  {
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~
Expected Behavior
I expect the code to compile without error.
Environment
Describe your runtime environment:

Machine: Mac laptop
OS: Catalina (v10.15.3)
MET master_v9.0 development branch.
Compiler version number:
/usr/bin/g++ --version
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Apple clang version 11.0.0 (clang-1<IP_ADDRESS>)
Target: x86_64-apple-darwin19.3.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

To Reproduce
Compile MET version 9.0 on a Mac.
Relevant Deadlines
Please include fix in the next met-9.0 bugfix release.
Funding Source
NONE
Define the Metadata
Assignee

[X] Select engineer(s) or no engineer required
[X] Select scientist(s) or no scientist required

Labels

[X] Select component(s)
[X] Select priority
[X] Select requestor(s)

Projects and Milestone

[X] Review projects and select relevant Repository and Organization ones
[X] Select milestone

Committed fixes to the master_v9.0 and develop branches.
No code review necessary for this simple bugfix.


perf: reuse allocated buffer. add config parameter

Added parameter -buffer-size. Reusing pre-allocated buffer for connection handling.

Hi, thanks for your contribution. however, i don't know if we really need to add another option to set the buffer size.

We can not add the buffer-size parameter to the args. How about reworking the socket reading function. In this branch, an improvement was made to reuse the allocated buffer within a single connection

Yeah it's all good but we can just set the buffer-size to 1024 and remove the command line option


Revert "[ConstraintSystem] Extend availability check to cover unavailable ext…"

Reverts apple/swift#58466
Resolves: rdar://92540971

@swift-ci please smoke test

@swift-ci please smoke test Windows platform


[Bug]: Cannot build devbox dockerfile on arm64

Current Behavior (bug)
When trying to build a multi-platform image with devbox and nix there is a eBPF issue causing the dockerfile to fail to build. I have been able to reproduce this by running the base Devbox rootUser dockerfile in this repo. This does not seem to happen if I use the non-root Dockerfile
docker buildx build  --platform linux/amd64,linux/arm64  . -f internal/impl/generate/tmpl/DevboxImageDockerfileRootUser

The exact error can be seen here:
51.36 ~~> Setting up shell profiles for Fish with conf.d/nix.fish inside /etc/fish /usr/local/etc/fish /opt/homebrew/etc/fish /opt/local/etc/fish
51.36
51.42 ~~> Setting up the default profile
51.42
51.48 ---- sudo execution ------------------------------------------------------------
51.48 I am executing:
51.48
51.48     $ sudo HOME=/root /nix/store/9670sxa916xmv8n1kqs7cdvmnsrhrdjv-nix-2.17.0/bin/nix-env -i /nix/store/9670sxa916xmv8n1kqs7cdvmnsrhrdjv-nix-2.17.0
51.48
51.48 to install a bootstrapping Nix in to the default profile
51.48
52.00 installing 'nix-2.17.0'
52.25 error:
52.25        … while setting up the build environment
52.25
52.25        error: unable to load seccomp BPF program: Invalid argument
52.38
52.38 ---- oh no! --------------------------------------------------------------------
52.38 Oh no, something went wrong. If you can take all the output and open
52.42 an issue, we'd love to fix the problem so nobody else has this issue.
52.42
52.42 :(
52.42
52.43 We'd love to help if you need it.
52.43
52.43 You can open an issue at
52.43 https://github.com/NixOS/nix/issues/new?labels=installer&template=installer.md
52.43
52.43 Or get in touch with the community: https://nixos.org/community
52.43

Expected Behavior (fix)
Devbox Docker images should build cross platform
Additional context
I have currently tried building this on an amd64 Macbook and ubuntu VM, this might work if built on an arm64 macbook

@mohsenari IIRC you'd implemented the Dockerfile root user functionality. Any leads on this?

I wonder if you managed to solve this for users on a Mac building this image locally?

Sorry for the confusion, this is still not solved for root images if we want to build cross platform images with gha for instance. The non root image seems to work well now

@mohsenari can we open this issue back up? this is still an issue

@mohsenari can we open this issue back up? this is still an issue

Thanks for letting me know. I'll try to reproduce this issue.
For context, are you running an ubuntu vm on an m1 mac and you're trying to build docker image inside the vm?

@mohsenari thanks for taking a look! I can reproduce this by running an amd64 Ubuntu VM trying to build an arm64 image with the following command
docker buildx build  --platform linux/arm64  . -f internal/impl/generate/tmpl/DevboxImageDockerfileRootUser


Checked existing words + Added more words

Checked existing arabic flagged words + Added newer ones

Thank you @hamzaleroi for your help!

No problem. I will be glad to help again.


Enable deep-inspection for Chains in tekton config

This config change enables Chains to inspect task level results making sure the index manifests for an image index get attested too and not just the image index iself.
Resolved CVP-4070

@arewm @enarha  @lcarva Could you please take a look?

/test test-pipeline-service-deployment-ocp-414

debug fix ^^

FYI @yashvardhannanavati the various metadata for the test started at https://github.com/openshift-pipelines/pipeline-service/blob/main/operator/test/test.sh#L150
is over in https://github.com/openshift-pipelines/pipeline-service/tree/main/operator/test/manifests/test/tekton-chains

also, we'll want to see passing app studio e2e's from https://github.com/redhat-appstudio/infra-deployments/pull/3543 before merging this PR here

OK we have some basic debug  @yashvardhannanavati ... I'll attach the log here.
I got it by watching the test-pipeline-service-deployment-ocp-414 pipelinerun over at https://console-openshift-console.apps.stone-prd-rh01.pg1f.p1.openshiftapps.com/pipelines/ns/tekton-ci/pipeline-runs
I'm assuming this will make sense to you all more than me, but with this config change, the test pipeline step at https://github.com/openshift-pipelines/pipeline-service/blob/main/operator/test/manifests/test/tekton-chains/simple-copy-pipeline.yaml#L49-L73 fails to create the pod.
[ERROR] Pipeline failed to complete successful
+ kubectl get pipelineruns/simple-copy-run-9vjp6 -n plnsvc-tests
+ echo /dev/stdout
+ prefix=pipelineruns/
NAME                    SUCCEEDED   REASON   STARTTIME   COMPLETIONTIME
simple-copy-run-9vjp6   False       Failed   5m          4m46s
/dev/stdout
+ string_to_prune=pipelineruns/simple-copy-run-9vjp6
+ tkn_param=simple-copy-run-9vjp6
+ tkn pr logs simple-copy-run-9vjp6 -n plnsvc-tests
task image-copy has failed: Failed to create pod due to config error
/dev/stdout
Tasks Completed: 1 (Failed: 1, Cancelled 0), Skipped: 0

does this part of the test:
echo "Bypass image push temporarily"
 echo "foobar"  > "$(results.IMAGE_URL.path)"
 echo "foobar"  > "$(results.IMAGE_DIGEST.path)"

violate the deep inspection?  If so, then yeah you will need to update the test and/or the sample pipelinerun etc. so that we can properly pass the tests at https://github.com/openshift-pipelines/pipeline-service/blob/main/operator/test/test.sh#L189-L264
If you need more debug, you can add kubectl calls to https://github.com/openshift-pipelines/pipeline-service/blob/main/operator/test/test.sh#L89-L103 as part of your PR here to get events from the test namespace or the chains controller logs etc when the CI runs again.
Or, if debugging through our CI proves to onerous, you bring up your own OCP cluster and deploy your branch to it.  The instructions for that are

run git push -m origin enable_deep_inspect ... the -m is key, as that allows gitops to pull your branch's changes when it deploys things to your cluster
from your clone of the pipeline server repo, run ./developer/openshift/dev_setup.sh --use-current-branch ... this will bootstrap your cluster with openshift pipelines, gitops, and this repo's configuration for openshift pipelines, including your modification to the chains config
then login via the console or cli and debug, where you replicate the test from  https://github.com/openshift-pipelines/pipeline-service/blob/main/operator/test/test.sh#L158-L264 either using our script, or making calls manually, however is better for you.

full test logs
test-pipeline-service.log

/test test-pipeline-service-deployment-ocp-414

I'm trying to manually verify the change and while the upgrade test here succeeds, deploying the change over the baseline, in ArgoCD I get
one or more objects failed to apply, reason: error when patching "/dev/shm/1568775355": admission webhook "webhook.operator.tekton.dev" denied the request: mutation failed: cannot decode incoming new object: json: cannot unmarshal string into Go struct field Chain.spec.chain.artifacts.pipelinerun.enable-deep-inspection of type bool. Retrying attempt #6 at 1:01PM.

There was and issue with using true vs “true” for the field that the
operator webhook complains about
Long term the operator should be more flexible
Short term,
Look for my pr in infra deployment that sets the field with true but not in
quotes
On Sun, Apr 14, 2024 at 9:01 AM Emil Natan @.***> wrote:

I'm trying to manually verify the change and while the upgrade test here
succeeds, deploying the change over the baseline, in ArgoCD I get
one or more objects failed to apply, reason: error when patching "/dev/shm/1568775355": admission webhook "webhook.operator.tekton.dev" denied the request: mutation failed: cannot decode incoming new object: json: cannot unmarshal string into Go struct field Chain.spec.chain.artifacts.pipelinerun.enable-deep-inspection of type bool. Retrying attempt #6 at 1:01PM.
—
Reply to this email directly, view it on GitHub
https://github.com/openshift-pipelines/pipeline-service/pull/985#issuecomment-2054054826,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AA3NU5FNM7Q4PRGTRZJDQXTY5J43PAVCNFSM6AAAAABFVQYBO2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANJUGA2TIOBSGY
.
You are receiving this because you commented.Message ID:
@.***>

this was handled in https://github.com/redhat-appstudio/infra-deployments/pull/3580
closing


Implement pattern support in SVG parser/renderer/serializer

Fully support pattern element. This will fix ~14 tests from SVG 1.1 Test suite.

When it will be in master?


Containers drop off bridge networks unexpectedly

Description
In support we have seen some recent cases where containers are removed from the bridge network unexpectedly.
Steps to reproduce the issue:
TBD
Describe the results you received:

balena inspect ${CONTAINER_ID} will still show up on the proper network.
balena network inspect ${CONTAINER_ID} the network does not include the container

Describe the results you expected:
Additional information you deem important (e.g. issue happens only occasionally):
Issue happens frequently since upgrading from v2.58.4 but can be resolved by restarting the container.
Output of balena-engine version:
Client:
 Version:           19.03.13-dev
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        074a481789174b4b6fd2d706086e8ffceb72e924
 Built:             Mon Feb  1 20:12:05 2021
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          19.03.13-dev
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.17
  Git commit:       074a481789174b4b6fd2d706086e8ffceb72e924
  Built:            Mon Feb  1 20:12:05 2021
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          1.2.0+unknown
  GitCommit:
 runc:
  Version:
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 balena-engine-init:
  Version:          0.13.0
  GitCommit:        949e6fa-dirty

Output of balena-engine info:
Client:
 Debug Mode: false

Server:
 Containers: 12
  Running: 12
  Paused: 0
  Stopped: 0
 Images: 15
 Server Version: 19.03.13-dev
 Storage Driver: aufs
  Root Dir: /var/lib/docker/aufs
  Backing Filesystem: extfs
  Dirs: 331
  Dirperm1 Supported: true
 Logging Driver: journald
 Cgroup Driver: systemd
 Plugins:
  Volume: local
  Network: bridge host null
  Log: journald json-file local
 Swarm:
  NodeID:
  Is Manager: false
  Node Address:
 Runtimes: bare runc
 Default Runtime: runc
 Init Binary: balena-engine-init
 containerd version:
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: 949e6fa-dirty (expected: fec3683b971d9)
 Kernel Version: 5.8.18-yocto-standard
 Operating System: balenaOS 2.68.1+rev1
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 7.691GiB
 Name:<PHONE_NUMBER>9f.videolink.io
 ID: ODX3:BQOU:LFIR:MXUE:WZX4:L37E:BY42:VT3K:6AZL:FDZI:RDVT:UQ2B
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  <IP_ADDRESS>/8
 Live Restore Enabled: false

Additional environment details (device type, OS, etc.):
ID="balena-os"
NAME="balenaOS"
VERSION="2.68.1+rev1"
VERSION_ID="2.68.1+rev1"
PRETTY_NAME="balenaOS 2.68.1+rev1"
MACHINE="genericx86-64"
VARIANT="Production"
VARIANT_ID=prod
META_BALENA_VERSION="2.68.1"
RESIN_BOARD_REV="cd52766"
META_RESIN_REV="e658a4e"
SLUG="intel-nuc"

I know this is the engine repo and not the Supervisor but on devices running balenaOS, the Supervisor manages the containers on the engine. It can be possible that the Supervisor is responsible for removing the network from the containers which seems unlikely because it would only do that if the target state has changed. I just wanted to add we can confirm it's not the Supervisor because the Supervisor would delete the existing container and create a new one. Therefore, if we can reproduce this issue then deploy your containers and note the created_at field for the container with access to the network. Perform the steps to get the network to be removed and verify the container no longer is on the network. Once there check if the created_at has changed.
The supervisor logs would also indicate that it's going to recreate the container. This is the only way it would remove the network from a container is be recreating it.

This could be happening when a privileged / network: host container with NetworkManager inside is not configured to ignore veth* interfaces on the host.
Those map to containers which are connected to bridge interfaces which then form the "docker network".
A log like this then means that NM removed the container from the network, leading to this scenario:
NetworkManager[1016]: <info>  [1626122797.1098] device (vethe79bead): released from master device br-f43124f982ae

balenaOS' own NetworkManger is configured to ignore those veth interfaces.

This could be related to https://github.com/balena-os/balena-engine/issues/261. I am looking at a device that is refusing to update with error HTTP code 404) -- no such container: sandbox when renaming a container and the affected container is also missing from the default network.
balena-engine version
Client:
 Version:           19.03.13-dev
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        074a481789174b4b6fd2d706086e8ffceb72e924
 Built:             Wed Mar 17 07:23:03 2021
 OS/Arch:           linux/arm64
 Experimental:      false

Server:
 Engine:
  Version:          19.03.13-dev
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.17
  Git commit:       074a481789174b4b6fd2d706086e8ffceb72e924
  Built:            Wed Mar 17 07:23:03 2021
  OS/Arch:          linux/arm64
  Experimental:     true
 containerd:
  Version:          1.2.0+unknown
  GitCommit:
 runc:
  Version:
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 balena-engine-init:
  Version:          0.13.0
  GitCommit:        949e6fa-dirty

balena-engine info
Client:
 Debug Mode: false

Server:
 Containers: 5
  Running: 3
  Paused: 0
  Stopped: 2
 Images: 37
 Server Version: 19.03.13-dev
 Storage Driver: overlay2
  Backing Filesystem: <unknown>
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: journald
 Cgroup Driver: systemd
 Plugins:
  Volume: local
  Network: bridge host null
  Log: journald json-file local
 Swarm:
  NodeID:
  Is Manager: false
  Node Address:
 Runtimes: bare runc
 Default Runtime: runc
 Init Binary: balena-engine-init
 containerd version:
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: 949e6fa-dirty (expected: fec3683b971d9)
 Kernel Version: 5.4.83-v8
 Operating System: balenaOS 2.73.1+rev1
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 962.2MiB
 Name: f9a83fc
 ID: EDQD:QP5A:T6RN:3W7Z:A7UF:GKSK:UTAM:ESZH:PSR6:XSIW:FFJD:QPXH
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  <IP_ADDRESS>/8
 Live Restore Enabled: false

WARNING: No swap limit support

This happened to them on at least 10% of their fleet of 10000 devices

@pipex just FYI this seems to still be happening with every update to a number of devices.
Pushing a random fleet variable such as FOO=BAR to all services seems to allow them to get past this issue but that's very annoying to have to do manually each time

I'm also seeing this behavior randomly on about ~250 of our 2200 devices in Balena. We also have a privileged container in host networking mode used for configuring our devices networking over bluetooth. This container also has the balena socket  exposed and when this occurs the socket is no longer accessible from that container. Another container that has the docker/balena socket exposed as well (also privileged but not in host networking mode) can still access the socket

I reviewed the 7 support tickets we have attached to this issue. We don't usually have all the data to check if they were cases of Engine startup timeouts, but for 4 tickets (from two different users) there is strong evidence this was indeed the case. Two other tickets (from two other users) were more difficult to analyze and probably involved more than one single issue -- but we have noticed unexpected behavior after reboot (in of them, it caught my attention the application was using 11 containers, which could translate to a higher Engine startup time).
Still, very importantly, in one ticket we have some good evidence that a container got dropped off the network without a reboot or Engine restart (the Engine had a 49-day uptime on this case).
So, startup timeouts seem to be a common cause of this issue, but not the only one.


stdenv: throw from ghcjs.cc.outputs rather than ghcjs.cc

Description of changes
pkgsCross.ghcjs.stdenv.cc currently throws, and it is (a good) part of the Nix semantics that exceptions transmit no information to an enclosing tryEval.  This makes it impossible to express the fact that the answer to "is the C compiler from GNU?" should be false when no compiler exists.
The reason why pkgsCross.ghcjs.stdenv.cc is a throw is explained here:
https://github.com/NixOS/nixpkgs/blob/aa371581dd110352b217e214867dd2a83660fa3c/pkgs/stdenv/cross/default.nix#L66-L71
which refers to this check in splice.nix:
https://github.com/NixOS/nixpkgs/blob/aa371581dd110352b217e214867dd2a83660fa3c/pkgs/top-level/splice.nix#L58-L63
Since the result of that check is always passed to getOutputs, which is strict in its argument's outputs attribute, we can simply move the throw deeper into the stdenv.cc attrset.  This leaves room to provide a useful value for attributes like isGNU and isClang when the compiler is "none".
Hopefully this removes the need for 1ee0f4c2aa2f6f70dca68c0d9bc51663a923bf2b from #208947.
Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.05 Release Notes (or backporting 22.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[ ] Fits CONTRIBUTING.md.

This might still be a good idea, but note both of you that I made a stdenv.hasCC which can be used to guard accesses to e.g. stdenv.cc.isClang.

This might still be a good idea, but note both of you that I made a stdenv.hasCC which can be used to guard accesses to e.g. stdenv.cc.isClang.

Yeah, I used that in an earlier revision of this PR, but it really uglifies the caller because (for some reason I didn't track down) stdenv is sometimes missing the hasCC attribute.  So callers need to do (!stdenv.hasCC or true) || stdenv.cc.isGNU.
And of course, callers have to remember to do that.  Which I'm sure they won't, unless they are using ghcjs.
So I went looking for a more automatic way to accomplish the goal and came up with this.

@ofborg eval
The Darwin builders appear to be stuck.


HTML node can not autoscale when canvas scaled!

RT,when i use HTML custom node,and scroll to scale the whole canvas:

the normal node can scale perfectly
the html node stay the origin size,can not scale.

not support yet


Change dead_letter_queue_max_size_in_bytes / size_in_bytes from counter to gauge metrics

Fixes issue #198

@Aloshi I will leave it on master, without releasing a new tag for a moment
I hope I will have some time on this weekend to do a better check on this PR and release it in a new tag


Removed preset data (non functioning) and standard option is made blank

With these changes in, the javascript from the project, generalized in #2 , doesn't work anymore.


Ngram featurizer fitting noise

Rasa NLU version (e.g. 0.10.1):
Used backend / pipeline (..., intent_featurizer_ngrams, ...):
Issue:
I think the ngram featurizer should use reasonable ngrams by default, e.g. 2grams or 3grams, and only diverge when there is a strong enough signal. Otherwise we might simply fit to some noise in the training data. See below
DEBUG:rasa_nlu.featurizers.ngram_featurizer:Evaluating usage of 1 ngrams. Score: 0.363302749387
DEBUG:rasa_nlu.featurizers.ngram_featurizer:Evaluating usage of 2 ngrams. Score: 0.363302749387
DEBUG:rasa_nlu.featurizers.ngram_featurizer:Evaluating usage of 3 ngrams. Score: 0.363302749387
DEBUG:rasa_nlu.featurizers.ngram_featurizer:Evaluating usage of 4 ngrams. Score: 0.338078286517
DEBUG:rasa_nlu.featurizers.ngram_featurizer:Evaluating usage of 5 ngrams. Score: 0.358586897329
DEBUG:rasa_nlu.featurizers.ngram_featurizer:Evaluating usage of 6 ngrams. Score: 0.36000907244
DEBUG:rasa_nlu.featurizers.ngram_featurizer:Evaluating usage of 7 ngrams. Score: 0.361385903733
DEBUG:rasa_nlu.featurizers.ngram_featurizer:Score without ngrams: 0.0
INFO:rasa_nlu.featurizers.ngram_featurizer:Best score with 1 ngrams: 0.363302749387

I'm not sure what you mean thomas - can you please explain in more detail?

Sure! So, the ngram_featurizer tests which ngrams work best. Which is a great idea for domain adaption. There might be some domains where one works much better than the other. However, it might also not make a big difference, as in my case above. In fact, unigrams, bigrams, and trigrams had the same score and the rasa picked the unigrams which basically translates to features like a,b,c,d,e,... right? Actually I have two points here.

I assume unigrams are rarely helpful. As there are only a handful of them and there presence is probably relatively uniform across sentences of English thus giving little useful information to the intent classifier.

bigrams or trigrams should be a reasonable default for most domains. It translates into quite a number of features, that actually contain some information. 4-7 grams might be too fine-grained in many cases. They translate into a huge amount of features that rarely occur (sparsity). That is why, we shouldn't make a move to use them if there isn't a strong signal for them. Right now they would be used even if they just have an 0.000001 edge over bi- or trigrams. However, bi- and trigrams will usually generalize much better to test data than the specialized highgrams (lol, just made up this word ^^).

The change I propose is to a) cut unigrams completely because I think they are not helpful and thus might confuse users, and b) pick either bi- or trigrams as the default and only switch if there is enough evidence that another ngram is actually better in this domain. This might mean a couple percentage points in difference

Actually unigrams might make sense for Chinese 🤔

also longer ngrams make sense for german compound words - virtually all of which are out of vocabulary

@twerkmeister you are interpreting the logging incorrectly, in
DEBUG:rasa_nlu.featurizers.ngram_featurizer:Evaluating usage of 2 ngrams. Score: 0.36330274938

The 2 ngrams is not referring to ngrams of length 2 but to "use two ngrams" whos length is unknown. The model creates ngrams of different lengths, sorts them according to there usefullness and then starts to add them to the feature vector one after another until the model doesn't get better anymore. So the first added ngram could be "en" the second "bart" the third "be" and so one. The result after adding the fourth ngram is this line:
DEBUG:rasa_nlu.featurizers.ngram_featurizer:Evaluating usage of 4 ngrams. Score: 0.338078286517

Ok, good to know :+1:, closing this then!

About unigram making sense: could '$' generate a feature for sentences generally containing an amount of money?
RegexFeaturize could be used to but I think it only works with the crf.


WIP: fix/userField

Work in Progress Pull Request

Pull Request Test Coverage Report for Build 1125

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 77.35%

Totals

Change from base Build 1116:
0.0%

Covered Lines:
2047

Relevant Lines:
2524

💛  - Coveralls


feat(iroh-net)!: Create our own connection struct

Description
We used to re-export quinn::Connection, this makes this an owned
connection so we can add our own methods to this.  This adds the
.remote_peer_id() method and removes the helper function for this.
Breaking Changes
iroh_net::endpoint::get_remote_node_id has been removed.  Use
iroh_net::endpoint::Connection::remote_node_id instead.
Notes & open questions

We have talked about wanting to do this for so long, decided to just
give this a go.

Strictly speaking we do not need to remove get_remote_node_id
right now.

This completely removes the ability of iroh-blobs to work with
upstream Quinn connections.  It now only works with iroh.  We could
probably use the h3::quic traits in iroh-blobs to make it generic
over QUIC implementation.

The code just lives in iroh_net::endpoint now.  That's probably
fine, but it's a large file by now.  Easy enough to move it later
though.

The public API is at iroh_net::endpoint::Connection.  Perhaps it
should also be exported at iroh_net::Connection just like
Endpoint is.  Though what belongs there?  What about all the other
types?  Hence I'm leaving this as the status quo for this PR and
leave it where it is.

  This probably
should have happened earlier, we knew we were not exporting enough
of those but never did a proper check on what is needed.  Now maybe
everything is there, bar possibly some things from quinn-udp still
missing.

Change checklist

[x] Self-review.
[x] Documentation updates following the style guide, if relevant.
[ ] Tests if relevant.
[x] All breaking changes documented.

Seen from iroh_net it makes sense. Is there anything preventing us from implementing Into<quinn::Connection> or AsRef or similar so that it doesn't affect iroh_blobs as much?

Seen from iroh_net it makes sense. Is there anything preventing us from implementing Into<quinn::Connection> or AsRef or similar so that it doesn't affect iroh_blobs as much?

This is a good idea!

Seen from iroh_net it makes sense. Is there anything preventing us from implementing Into<quinn::Connection> or AsRef or similar so that it doesn't affect iroh_blobs as much?

This is a good idea!

Note that there are no code changes since iroh-blobs already uses iroh_net::nedpoint::Connection as import.  I'd love to get @rklaehn's opinion on this before changing that back to iroh_quinn::Connection.  Because the real breakage was done at the point when we transitioned from quinn to iroh-quinn.

Back to this, is there any reason to not implement Deref and DerefMut so that we don't have to keep updating docs and migrating new functions? Sometimes docs for Dereferencing types is not great, but otherwise it seems appropriate

Back to this, is there any reason to not implement Deref and DerefMut so that we don't have to keep updating docs and migrating new functions? Sometimes docs for Dereferencing types is not great, but otherwise it seems appropriate

I'm not really a fan of Deref for users, it always confuses me as a user.  Plus it gives us less flexibility.  Yes we need to keep this in sync when we update the dependency, but we also get to benefit from better integrated documentation - I did make changes to the docs here.  So I think I'd prefer this approach.

I'm not really a fan of Deref for users

Yeah, I wish rust had a better way to do or at least show this when all we need is delegation.
Agree with the conclussion

Can you elaborate your future plans with this? If the only thing this allows us to do is to change the get_remote_node_id fn into a member fn, I would say it is not worth it.
It would help with making examples more approachable, but you could do the same (except for an extra import) with an extension method.

Can you elaborate your future plans with this?

This can enable a number of things which we have shied away from in the past simply because taking the step of adding this struct was not yet done:

Connection type and changes to it are currently on on the Endpoint, they also belong on the Connection.
Connection information, like which paths are used, the latencies of those paths, etc, also belongs on the Connection.
Some APIs don't work well for us, e.g. Connection::local_ip or Connection::remote_address are not that meaningful.  This allows us to remove those and replace them with more appropriate APIs.
It is generally not great to expose types which you have no control over in your public API.  This doesn't solve that fully, but it does tackle an important struct.

Probably some more things I'm not remembering right now.
Admittedly this PR doesn't tackle many of these things.  But it makes it so much easier to do in the future.

docs: https://n0-computer.github.io/iroh/pr/2768/docs/iroh_net/index.html

Given that we have to hide all of quinn for 1.0 anyway, I think this is reasonable to do.

PR came first checking approved ones (candidates for merging) but this has a long list of conflicts. Since this has essentially passed the approval bar, here's a friendly ping to refresh it @flub

Yeah, I was waiting for the dust to settle on all the renames and moving crates before getting back to this.  Doing backwards-incompatible changes was very difficult at some point during this churn.
Converted it back to draft for now, will pick it up after the next release probably.


Fix incorrect 1123label validation

Please provide a description of this PR:

@hanxiaop: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

unit-tests-arm64_istio
a2f1cbe5d53971fd95a8bd697080f6e542e9a428
link
true
/test unit-tests-arm64

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@hanxiaop: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

unit-tests-arm64_istio
a2f1cbe5d53971fd95a8bd697080f6e542e9a428
link
true
/test unit-tests-arm64

unit-tests_istio
a2f1cbe5d53971fd95a8bd697080f6e542e9a428
link
true
/test unit-tests

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@hanxiaop: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

unit-tests-arm64_istio
a2f1cbe5d53971fd95a8bd697080f6e542e9a428
link
true
/test unit-tests-arm64

unit-tests_istio
a2f1cbe5d53971fd95a8bd697080f6e542e9a428
link
true
/test unit-tests

lint_istio
a2f1cbe5d53971fd95a8bd697080f6e542e9a428
link
true
/test lint

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

This seems like a breaking change. Is there any more context on why we should do this?

@howardjohn I was trying to add some dns1123label validation to iop fields in validate, and I think the uppercase should be checked as well. I'll close this one and do it in another way.


Origin of micro_speech audio_preprocessor_test.py expected var values?

Hi,
I'm walking through the process of creating my own models (for a cat meowing).  One thing I don't understand is the way audio_preprocessor_test.py works.  Specifically, how do you get the values used in the expected variables (below).  Is there some other pre-process I need to run to get them?
Thank you,
-T
  def testFeatureOutputYes(self):
    feature_params = audio_preprocessor.FeatureParams()
    audio_pp = audio_preprocessor.AudioPreprocessor(feature_params)
    audio_pp.load_samples(Path(self.sample_prefix_path, 'yes_30ms.wav'))
    feature = audio_pp.generate_feature_using_tflm(audio_pp.samples)
    feature_list = feature.numpy().tolist()
    expected = [
        124, 105, 126, 103, 125, 101, 123, 100, 116, 98, 115, 97, 113, 90, 91,
        82, 104, 96, 117, 97, 121, 103, 126, 101, 125, 104, 126, 104, 125, 101,
        116, 90, 81, 74, 80, 71, 83, 76, 82, 71
    ]
    self.assertSequenceEqual(feature_list, expected)

  def testFeatureOutputNo(self):
    feature_params = audio_preprocessor.FeatureParams()
    audio_pp = audio_preprocessor.AudioPreprocessor(feature_params)
    audio_pp.load_samples(Path(self.sample_prefix_path, 'no_30ms.wav'))
    feature = audio_pp.generate_feature_using_tflm(audio_pp.samples)
    feature_list = feature.numpy().tolist()
    expected = [
        126, 103, 124, 102, 124, 102, 123, 100, 118, 97, 118, 100, 118, 98,
        121, 100, 121, 98, 117, 91, 96, 74, 54, 87, 100, 87, 109, 92, 91, 80,
        64, 55, 83, 74, 74, 78, 114, 95, 101, 81
    ]
    self.assertSequenceEqual(feature_list, expected)

@gamename
The audio_preprocessor_test.py serves as a smoke test for the operators that the audio preprocessor uses.  The expected values were generated by feeding the audio preprocessor model and sample data to either TensorFlow or the TfLite interpreter (I don't remember which one I used).
An example of using the TfLite interpreter is available in the hello_world example evaluate scripts.
For the TFLM signal operator plugins to be available to TensorFlow/TfLite, you must use Bazel to build/run your python scripts.


PasswordBox: The password cannot be unmasked per keyboard

Describe the bug
The documentation states the following:
Windows 10, version 1607, introduces a new keyboard shortcut for accessibility. When the focus is in the PasswordBox and the password reveal button is visible, the user can press and hold Alt+F8 to reveal their password. When they let go of either key press, the password is hidden again.
This does not appear to work.
Steps to reproduce the bug

Open the XAML Controls Gallery and go to the PasswordBox page
Focus a PasswordBox and start typing into it so that the reveal button is shown
Hold Alt+F8

Observe that the password is not unmasked, even though it should.
Expected behavior
Password should be unmasked while holding Alt+F8.

Windows app type:

UWP
Win32

Windows 10 version
Saw the problem?

Insider Build (xxxxx)

May 2020 Update (19041)
Yes

November 2019 Update (18363)

May 2019 Update (18362)

October 2018 Update (17763)

April 2018 Update (17134)

Fall Creators Update (16299)

Creators Update (15063)

Device form factor
Saw the problem?

Desktop
Yes

Xbox

Surface Hub

IoT

That's strange, I wasn't able to repro this both in the XCG nor the MUXC app. In both cases, pressing alt+f8 reveals the entered text.

Hmmm...I tested other Alt+ keyboard combinations in VS (like Alt+F2 for opening the Performance Profiler) and they don't work either. Perhaps this is a Dell device configuration issue (I have a Dell laptop)?
I am searching the Internet right now to see if I can find something here but so far, no real luck. Alt+F4 works, heh, but it's a bit unfortunate that apparently in-app Alt+Fn keyboard shortcuts don't appear to work on an out-of-the-box Dell system.

alt F8 is only supposed to be working in the scenarios where the reveal button is visible.  @chingucoding are you seeing something different?

Its possible that you have software on your computer that is overriding these commands. For instance on my home PC alt F8 opens the GForceExperience app and doesn't show the password box.

@Austin-Lamb or @MikeHillberg do you happen to know details about when/why alt F8 could be disabled?


updated the themes and footer

Issue #23: Implement Dark and Light Theme Toggle
Description:
Closes #23
Users should have the option to toggle between dark and light themes for better user experience, especially during different times of the day. The theme toggle will use a sun and moon icon to represent light and dark modes respectively.
Changes proposed:
Add a theme toggle icon that switches between light and dark modes. Use the Font Awesome icons fa-moon for dark mode and fa-sun for light mode.
Implement CSS variables to manage the styles for both dark and light modes.
Save the selected theme in localStorage so it persists between sessions.
On page load, check localStorage for the user's preferred theme and apply it.
Update the icon based on the active theme.
Screenshots:
Add screenshots showing:

https://github.com/user-attachments/assets/df529691-4d6c-4301-a83c-893f1f817e30
Note to reviewers:
Ensure that the theme toggle works smoothly.
Confirm that the user's preference is stored and persists after reloading the page.
Issue #32: Dynamic Footer Year
Description:
Closes #32
The footer should display the current year dynamically to avoid manual updates. This ensures that the copyright notice is always accurate.
Changes proposed:
Add JavaScript to automatically insert the current year in the footer.
Ensure that the year is updated correctly on January 1st each year.
Screenshots:

Note to reviewers:
Confirm that the footer year updates automatically based on the current date.
Test on different browsers to ensure compatibility.

now this is a PR submission, wow!

Hey @Ham12-3. I did the dark mode in here. If you're open, I can raise a separate issue for the footer.

I am still open so can I close or are you merging

I am still open so can I close or are you merging

Merging.

Ok, merge

I am deeply sorry for replying you late. I have merged it already from my side


.keyof() relies on key order, resulting in unsound .options tuple type

Saw this awesome trick by @ecyrbe transforming object keys into string tuple ordered by source declaration, but beware TS treats key order differently than JS engines:
z.object({ a: z.string(), '1': z.string() }).keyof().options()
// TS says it's ['a', '1']
// JS runtime it's ['1', 'a']

I believe .options is public API for enums, therefore this typing is unsound.
A simple solution that comes to mind is having an UnorderedZodEnum like
type UnorderedZodEnum<T extends [string, ...string[]]> = Omit<z.ZodEnum<T>, 'options'> & { options: Set<T> }

Or just downgrade the keyof to return ZodSet instead of ZodEnum.
A * mostly * non breaking change could be something like:
type UnorderedZodEnum<T> = Omit<ZodEnum<T>, 'options'> & { options: Repeat<typeof T[number], T['length']> }

type Repeat<N extends number, T, Acc extends T[] = []> = N extends Acc['length']
  ? Acc
  : Repeat<N, T, [T, ...Acc]>

Will still be breaking change in use-cases relying on contravariance or assignability, but those are exactly the cases that would be unsound in current version, eg:
const Enum = z.object({ a: z.string(), b: z.string() }).keyof()

// user relying on ordering of current API
const first: 'a' = Enum.options[0]
// Type '"a" | "b"' is not assignable to type '"a"'.

// needs to be cast instead
const first = Enum.options[0] as 'a'

A more non-breaking solution would be to try mimicking JS engines, beware of typescript keyof though, it seems to sometimes result in desired order (i really didn't get the rule on that), but only with object literals, not when the object comes from a generic param. Seems the safer would be to force the engine ordering with something like:
type ObjectKeys<T> = [
  ...Sort<enumUtil.UnionToTupleString<IntString<keyof T>>>,
  ...enumUtil.UnionToTupleString<Exclude<keyof T, IntString<keyof T>>>
]

But to implement a 'Sort' for that would need some more possibly unsound tricks

Managed to do an implementation of the last approach for reference
type ObjectKeys<T> = [
  ...Sort<enumUtil.UnionToTupleString<IntString<keyof T>>>,
  ...enumUtil.UnionToTupleString<Exclude<keyof T, IntString<keyof T>>>,
]

type Head<T> = T extends [infer H, ...any[]] ? H : T extends `${infer H}${string}` ? H : never
type Tail<T> = T extends [any, ...infer Rest] ? Rest : T extends `${string}${infer Rest}` ? Rest : never
type Take<N extends number, T extends any[], Acc extends any[] = []> = N extends Acc['length']
  ? Acc
  : T extends [infer Head, ...infer Rest]
  ? Take<N, Rest, [...Acc, Head]>
  : never

type Digits = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

type IntString<T> = T extends '0'
  ? T
  : T extends `${Exclude<Digits[number], 0>}${infer Rest}`
  ? `${Head<T>}${DigitString<Rest>}`
  : never

type DigitString<T> = T extends `${Digits[number]}${infer Rest}` ? `${Head<T>}${DigitString<Rest>}` : T

type Sort<T extends Array<any>, Acc extends Array<any> = []> = T extends []
  ? Acc
  : Insert<Head<T>, Acc> extends infer G
  ? Sort<Tail<T>, G extends Array<any> ? G : never>
  : never

type Insert<N, Acc extends Array<any>> = Acc extends []
  ? [N]
  : Compare<N, Head<Acc>> extends 'lt' | 'eq'
  ? [N, ...Acc]
  : Insert<N, Tail<Acc>> extends infer G
  ? [Head<Acc>, ...(G extends Array<any> ? G : never)]
  : never

type Compare<A, B> = CompareLength<A, B> extends 'eq'
  ? Head<A> extends Head<B>
    ? Compare<Tail<A>, Tail<B>>
    : CharToInt<Head<A>> extends Take<CharToInt<Head<B>>, Digits>[number]
    ? 'lt'
    : 'gt'
  : CompareLength<A, B>

type CharToInt<C> = Extract<{ [N in Digits[number]]: [N, `${N}`] }[Digits[number]], [any, C]>[0]

type CompareLength<A, B> = [A, B] extends ['', '']
  ? 'eq'
  : [A, B] extends [`${infer _}${string}`, '']
  ? 'gt'
  : [A, B] extends ['', `${infer _}${string}`]
  ? 'lt'
  : CompareLength<Tail<A>, Tail<B>>

So changing https://github.com/colinhacks/zod/blob/59768246aa57133184b2cf3f7c2a1ba5c3ab08c3/src/types.ts#L1944
to
  keyof(): ZodEnum<ObjectKeys<T>> {

Would probably work, but damn, guess the Set one makes more sense

No keyof is designed to feed zod enum schemas, that is why originaly i named it .enum() to conveys the use case. Using a set will defeat the purpose of it.
Plus like the name suggest, it's used to represent enums not tuples. Enum have no order , your code should not rely on enum order, JavaScript and typescript gives no guaranty on object Key order, no point trying to develop workerounds to satisfy V8 algorithm when order can be completely different depending on the JavaScript engine. Typescript being unsound is completely expected and can't be evoided since typescript !==runtime JavaScript engine

To clarify, even options() is expected to be unsound since you can't have guaranties on enum order

I think Collin naming it keyof was a mistake, because it's not the keys(tuple) we are returning it's an enum.
@colinhacks if you want to comment.

M'y proposition would then be to rename it into .enum() and add précision in the documentation about it's use case and it being unsound being expected

@ecyrbe

options() is expected to be unsound
That already satisfies it for me, indeed I suspected it was the case that enum is unordered, just was surprised by the ordered tuple signature it has

Also I agree that .enum would be better naming in this feature as well, .keyof result doesn't reflect neither JS Object.keys nor TS keyof very well


Missing sandship

My dowsing function is set to the trial gates so the ship didn't spawn. I checked around the spot it is supossed to be with the cannon, but no luck. I also couldn't enter the ship yet at the place you are normally able to.

As long as you have gotten the ancient Sea Chart, you can dowse for the Sandship, even if the dowsing icon isn't updated correctly (top option on the dowsing wheel)

As long as you have gotten the ancient Sea Chart, you can dowse for the Sandship, even if the dowsing icon isn't updated correctly (top option on the dowsing wheel)

I see, thanks for the reply. I had visited the skipper after getting the sea chart bit dowsing again hadn’t come to mind. This worked as intended as I was just able to find the ship via dowsing.

I see, thanks for the reply. I had visited the skipper after getting the sea chart bit dowsing again hadn’t come to mind. This worked as intended as I was just able to find the ship via dowsing.


[wip] user profile related api changes

Coverage increased (+0.4%) to 86.354% when pulling af2a790f0aff540478d4110defd2f4231fd91bd1 on userProfile into 2c0769ea2273fe96efee24006607e8f9bf934b95 on master.


Model Loading Stuck (in ray ?)

python = 3.11.5
torch = 2.1.0 + cu121
vllm = 0.2.2
GPU: L40 * 4
I install vllm by "pip install vllm".
It will STUCK when loading vicuna-7b-v1.5 model using the vllm framework, while the fastchat framework work well.
When I arise a KeyboardInterrupt, it is stuck at
the ./ray/_private/worker.py, line 769, in get_objects
data_metadata_pairs = self.core_worker.get_objects(
File "python/ray/_raylet.pyx" in line 3211, in ray._raylet.CoreWorker.get_objects
File "python/ray/_raylet.pyx" in line 449, in ray._raylet.check_status
KeyboardInterrupt

What's your nvidia driver version and topology? You can get it via nvidia-smi and nvidia-smi topo

Thanks. I was suspecting this is the case of #1801 but doesn't seem like it. Can you paste your full command line argument or script? Additionally, can you run ray stack when it is stuck and paste in the output so we can see what's stuck here? Lastly, a version of nccl would be useful.
python -c "import torch;print(torch.cuda.nccl.version())"

Thanks. I was suspecting this is the case of #1801 but doesn't seem like it. Can you paste your full command line argument or script? Additionally, can you run ray stack when it is stuck and paste in the output so we can see what's stuck here? Lastly, a version of nccl would be useful.
python -c "import torch;print(torch.cuda.nccl.version())"

This is what the terminal show, when the code is stuck for a long time:

(vv) xxx@vision11:~/llm_projects$ python "/home/xxx/llm_projects/vllm_test.py"
2023-11-30 09:49:16,217 INFO worker.py:1673 -- Started a local Ray instance.
INFO 11-30 09:49:17 llm_engine.py:72] Initializing an LLM engine with config: model='/data2_vision5/xxx/vicuna-7b-v1.5/', tokenizer='/data2_vision5/xxx/vicuna-7b-v1.5/', tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=2, quantization=None, seed=0)
(RayWorker pid=1080652) [E ProcessGroupNCCL.cpp:474] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=ALLREDUCE, NumelIn=1, NumelOut=1, Timeout(ms)=1800000) ran for 1800265 milliseconds before timing out.
(RayWorker pid=1080652) [E ProcessGroupNCCL.cpp:488] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
(RayWorker pid=1080652) [E ProcessGroupNCCL.cpp:494] To avoid data inconsistency, we are taking the entire process down.
(RayWorker pid=1080652) [E ProcessGroupNCCL.cpp:915] [Rank 1] NCCL watchdog thread terminated with exception: [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=ALLREDUCE, NumelIn=1, NumelOut=1, Timeout(ms)=1800000) ran for 1800265 milliseconds before timing out.
(RayWorker pid=1080652) [2023-11-30 10:19:23,102 E 1080652 1080830] logging.cc:97: Unhandled exception: St13runtime_error. what(): [Rank 1] NCCL watchdog thread terminated with exception: [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=ALLREDUCE, NumelIn=1, NumelOut=1, Timeout(ms)=1800000) ran for 1800265 milliseconds before timing out.
 To troubleshoot the problem, check the logs for the dead worker. RayTask ID: ffffffffffffffffa79f2a2a0976a2c753c732ae01000000 Worker ID: 45477f19145fb3424019689328b818fbf84c547b27e6eb4ffa8fc060 Node ID: 9c6f827858d621904b3d1cfb06057c57480da46c3edab524030f1e58 Worker IP address: <IP_ADDRESS> Worker port: 45575 Worker PID: 1080652 Worker exit type: SYSTEM_ERROR Worker exit detail: Worker unexpectedly exits with a connection error code 2. End of file. There are some potential root causes. (1) The process is killed by SIGKILL by OOM killer due to high memory usage. (2) ray stop --force is called. (3) The worker is crashed unexpectedly due to SIGSEGV or other unexpected errors.
Traceback (most recent call last):
File "/home/xxx/llm_projects/vllm_test.py", line 35, in
llm = LLM(
File "/home/xxx/anaconda3/envs/vv/lib/python3.10/site-packages/vllm/entrypoints/llm.py", line 93, in init
self.llm_engine = LLMEngine.from_engine_args(engine_args)
File "/home/xxx/anaconda3/envs/vv/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 231, in from_engine_args
engine = cls(*engine_configs,
File "/home/xxx/anaconda3/envs/vv/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 108, in init
self._init_workers_ray(placement_group)
File "/home/xxx/anaconda3/envs/vv/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 181, in _init_workers_ray
self._run_workers(
File "/home/xxx/anaconda3/envs/vv/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 704, in _run_workers
all_outputs = ray.get(all_outputs)
File "/home/xxx/anaconda3/envs/vv/lib/python3.10/site-packages/ray/_private/auto_init_hook.py", line 24, in auto_init_wrapper
return fn(*args, **kwargs)
File "/home/xxx/anaconda3/envs/vv/lib/python3.10/site-packages/ray/_private/client_mode_hook.py", line 103, in wrapper
return func(*args, **kwargs)
File "/home/xxx/anaconda3/envs/vv/lib/python3.10/site-packages/ray/_private/worker.py", line 2565, in get
raise value
ray.exceptions.RayActorError: The actor died unexpectedly before finishing this task.
class_name: RayWorker
actor_id: a79f2a2a0976a2c753c732ae01000000
pid: 1080652
namespace: 7cfbf24e-bae4-49d1-9ac2-d4db1cb83a89
ip: <IP_ADDRESS>
The actor is dead because its worker process has died. Worker exit type: SYSTEM_ERROR Worker exit detail: Worker unexpectedly exits with a connection error code 2. End of file. There are some potential root causes. (1) The process is killed by SIGKILL by OOM killer due to high memory usage. (2) ray stop --force is called. (3) The worker is crashed unexpectedly due to SIGSEGV or other unexpected errors.
 To troubleshoot the problem, check the logs for the dead worker. RayTask ID: ffffffffffffffffba61971047f999e0f9ceaab101000000 Worker ID: 0e4b1eeade4874f7080db5807059e066cc128f27de4d1404a6057750 Node ID: 9c6f827858d621904b3d1cfb06057c57480da46c3edab524030f1e58 Worker IP address: <IP_ADDRESS> Worker port: 34011 Worker PID: 1080651 Worker exit type: SYSTEM_ERROR Worker exit detail: Worker unexpectedly exits with a connection error code 2. End of file. There are some potential root causes. (1) The process is killed by SIGKILL by OOM killer due to high memory usage. (2) ray stop --force is called. (3) The worker is crashed unexpectedly due to SIGSEGV or other unexpected errors.
(RayWorker pid=1080651) [E ProcessGroupNCCL.cpp:474] [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=ALLREDUCE, NumelIn=1, NumelOut=1, Timeout(ms)=1800000) ran for 1800305 milliseconds before timing out.
(RayWorker pid=1080651) [E ProcessGroupNCCL.cpp:488] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
(RayWorker pid=1080651) [E ProcessGroupNCCL.cpp:494] To avoid data inconsistency, we are taking the entire process down.
(RayWorker pid=1080651) [E ProcessGroupNCCL.cpp:915] [Rank 0] NCCL watchdog thread terminated with exception: [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=ALLREDUCE, NumelIn=1, NumelOut=1, Timeout(ms)=1800000) ran for 1800305 milliseconds before timing out.
(RayWorker pid=1080651) [2023-11-30 10:19:23,561 E 1080651 1080832] logging.cc:97: Unhandled exception: St13runtime_error. what(): [Rank 0] NCCL watchdog thread terminated with exception: [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=ALLREDUCE, NumelIn=1, NumelOut=1, Timeout(ms)=1800000) ran for 1800305 milliseconds before timing out.
(RayWorker pid=1080651) [2023-11-30 10:19:23,568 E 1080651 1080832] logging.cc:104: Stack trace:
(RayWorker pid=1080651)  /home/xxx/anaconda3/envs/vv/lib/python3.10/site-packages/ray/raylet.so(+0xf199fa) [0x7fecd8c0c9fa] ray::operator<<()
(RayWorker pid=1080651) /home/xxx/anaconda3/envs/vv/lib/python3.10/site-packages/ray/raylet.so(+0xf1c1b8) [0x7fecd8c0f1b8] ray::TerminateHandler()
(RayWorker pid=1080651)  [repeated 7x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/ray-logging.html#log-deduplication for more options.)
(RayWorker pid=1080651) /home/xxx/anaconda3/envs/vv/bin/../lib/libstdc++.so.6(+0xdbbf4) [0x7fecd7b9ebf4] execute_native_thread_routine
(RayWorker pid=1080651) /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7fecda955609] start_thread
(RayWorker pid=1080651) /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7fecda720133] __clone
(RayWorker pid=1080651) *** SIGABRT received at time=1701310763 on cpu 72 ***
(RayWorker pid=1080651) PC: @     0x7fecda64400b  (unknown)  raise
(RayWorker pid=1080651)     @     0x7fecd7b7435a  (unknown)  __cxxabiv1::__terminate() [repeated 2x across cluster]
(RayWorker pid=1080651)     @     0x7fecd7b74070  (unknown)  (unknown)
(RayWorker pid=1080651) [2023-11-30 10:19:23,568 E 1080651 1080832] logging.cc:361: *** SIGABRT received at time=1701310763 on cpu 72 ***
(RayWorker pid=1080651) [2023-11-30 10:19:23,568 E 1080651 1080832] logging.cc:361: PC: @     0x7fecda64400b  (unknown)  raise
(RayWorker pid=1080651) [2023-11-30 10:19:23,568 E 1080651 1080832] logging.cc:361:     @     0x7fecd7b7435a  (unknown)  __cxxabiv1::__terminate() [repeated 2x across cluster]
(RayWorker pid=1080651) [2023-11-30 10:19:23,569 E 1080651 1080832] logging.cc:361:     @     0x7fecd7b74070  (unknown)  (unknown)
(RayWorker pid=1080651) Fatal Python error: Aborted

python -c "import torch;print(torch.cuda.nccl.version())":  (2, 18, 1)
I'm not the sudoer, it seems like I can't run .

Hi, I got the same problem when using --tensor-parallel-size in vllm, have you fixed it?

We have the same problem running inference with fastchat vllm-worker.
With a standard fastchat-worker using accelerate of HuggingFace the parallel inference is working.
But with the vllm-worker the problem occurs. accelerate and vllm using the cuda-Lib in different ways, but perhaps vllm has to change anything.

There is something wrong with the connection between 2 specific gpus. When I use the other 2 gpus, the code can work well.

It does work for me too, but is weird🧐 what's wrong with the connection?

We have the same problem running inference with fastchat vllm-worker. With a standard fastchat-worker using accelerate of HuggingFace the parallel inference is working. But with the vllm-worker the problem occurs. accelerate and vllm using the cuda-Lib in different ways, but perhaps vllm has to change anything.

Do you also use L40 to run vllm ?

There is something wrong with the connection between 2 specific gpus. When I use the other 2 gpus, the code can work well.

It does work for me too, but is weird🧐 what's wrong with the connection?

Do you also use L40 to run vllm ?

When I add "export NCCL_P2P_DISABLE = 1" in the ~/.bashrc , the code can also work using the previous 2 GPUs.

Do you also use L40 to run vllm ?

No, we are using 2xA100 (40GB), no linking. And we tried that out with Tesla T4 on another machine.

When I add "export NCCL_P2P_DISABLE = 1" in the ~/.bashrc , the code can also work using the previous 2 GPUs.
great. This is working on both machines. With A100 and Tesla T4.
Does this have any performance impact? Now vllm is doing all controlling work between the GPUs?

There is something wrong with the connection between 2 specific gpus. When I use the other 2 gpus, the code can work well.

It does work for me too, but is weird🧐 what's wrong with the connection?

Do you also use L40 to run vllm ?

No I'm using RTX3090*8, but this works for me, also the  "export NCCL_P2P_DISABLE = 1" one works. But I think  "export NCCL_P2P_DISABLE = 1" has impact on performance. You can check  https://github.com/NVIDIA/nccl-tests/issues/117#issuecomment-1474324074, it is used on 4090

There is something wrong with the connection between 2 specific gpus. When I use the other 2 gpus, the code can work well.

It does work for me too, but is weird🧐 what's wrong with the connection?

Do you also use L40 to run vllm ?

No I'm using RTX3090*8, but this works for me, also the "export NCCL_P2P_DISABLE = 1" one works.

When I add "export NCCL_P2P_DISABLE = 1" in the ~/.bashrc , the code can also work using the previous 2 GPUs.

great. This is working on both machines. With A100 and Tesla T4. Does this have any performance impact? Now vllm is doing all controlling work between the GPUs?

I think "export NCCL_P2P_DISABLE = 1" has impact on performance. You can check #117, it is used on 4090.

When I add "export NCCL_P2P_DISABLE = 1" in the ~/.bashrc , the code can also work using the previous 2 GPUs.

great. This is working on both machines. With A100 and Tesla T4. Does this have any performance impact? Now vllm is doing all controlling work between the GPUs?

I think "export NCCL_P2P_DISABLE = 1" has impact on performance. You can check issue NVIDIA/nccl-tests#117, it is used on 4090.

There is something wrong with the connection between 2 specific gpus. When I use the other 2 gpus, the code can work well.

how to change gpus? For example ,if  I have 8 gpus in a machine,how can I specify to  use gpu 3 and gpu 4?

There is something wrong with the connection between 2 specific gpus. When I use the other 2 gpus, the code can work well.

how to change gpus? For example ,if I have 8 gpus in a machine,how can I specify to use gpu 3 and gpu 4?

Maybe CUDA_VISIBLE_DEVICES=3,4 is helpful ?

I am using docker
docker run --runtime nvidia --gpus '"device=1,2"'

I am using docker docker run --runtime nvidia --gpus '"device=1,2"'

Well sorry I don't know much about the docker command.

There is something wrong with the connection between 2 specific gpus. When I use the other 2 gpus, the code can work well.

how to change gpus? For example ,if I have 8 gpus in a machine,how can I specify to use gpu 3 and gpu 4?

Maybe CUDA_VISIBLE_DEVICES=3,4 is helpful ?

thank you

I am using docker docker run --runtime nvidia --gpus '"device=1,2"'

Well sorry I don't know much about the docker command.

The command CUDA_VISIBLE_DEVICES=3,4 doesn't seem to be effective. Although I've set it, the script continues to run on devices 1 and 2. I believe specifying the devices directly in the Docker run command would be more useful

I am not sure if you are using the CUDA_VISIBLE_DEVICES the right way. Have a look at https://github.com/vllm-project/vllm/discussions/691#discussioncomment-7246279


Windows supports

For supporting windows container in Cattle, the following task list needs to be finished:

[ ] https://github.com/rancher/agent/pull/87 should be merged. This PR addressed the point that host-api should be migrated into agent code and can be run inside agent process.

[ ] agent runs in container and the way it talks to the daemon in the host is by using network stack. This means that users have to explicitly configure their docker daemon before registering their host in rancher.By default windows container talks to a named pipe and that named pipe can not be bind into container. Not sure if it is good to run agent.exe directly on the host.

[ ] we need a clean template for windows environment, the infra stack doesn't apply to window. So for windows environment, maybe don't add any infrastructure stack to it now.

[ ] add network mode nat,  transparent to cattle API, hide other linux network mode when in windows environment

[ ] Add parsing volume support in windows style. (C:/foo:C:/bar)

[ ] agent is still receiving config.update in windows host. Maybe we could stop sending this event by telling different host OS types.

[ ] https://github.com/StrongMonkey/agent-image. Dockerfile for rancher/agent on windows is ready to be reviewed.

[ ] fix UI for windows environment(Changing the registration command, hiding linux specific fields and windows container stats until it get supported in 1.13)

[ ] LB refactor for windows.

[ ] Add CI for windows

@cjellick @ibuildthecloud @will-chan @deniseschannon

@sangeethah Can you follow the steps in UI and start testing some basic functions(like start container, stop container and remove container) in Windows?

Tested with latest build from master:
Able to add Windows hosts successfully.
Able add/delete services.
Able to do exec contaners , container stats and logs.
Host stats are also functional.
Following are some issues seen when testing this feature:
https://github.com/rancher/rancher/issues/7098
https://github.com/rancher/rancher/issues/7097
https://github.com/rancher/rancher/issues/7101


Update GithubRepositoryType.php

In relation to problem #115 it seems that this was not using the github private access token.
I have added a line to set the configured token in the GithubRepositoryType constructor, this works fine.

Have you been able to review this change suggestion?
Thank you very much.


[MRG] Improve docs on Dockerfile escape hatch

I was struggling to find how to get the generated Dockerfile contents. When I found it, I thought I'd open a PR to add a link from the first place I looked for the next person who struggles :)

I experienced some unexpected behavior and added a warning to the docs :point_up:

I feel like more documentation is needed. When I create a Dockerfile following these instructions and attempt to build, there are errors because the Dockerfile references paths that aren't in the build context:
Step 26/60 : COPY --chown=1002:1002 build_script_files/-2fhome-2fmfisher-2f-2eminiconda3-2fenvs-2frepo2docker-2flib-2fpython3-2e11-2fsite-2dpackages-2frepo2docker-2fbuildpacks-2fconda-2factivate-2dconda-2esh-cf5cc0 /etc/profile.d/activate-conda.sh
COPY failed: file not found in build context or excluded by .dockerignore: stat build_script_files/-2fhome-2fmfisher-2f-2eminiconda3-2fenvs-2frepo2docker-2flib-2fpython3-2e11-2fsite-2dpackages-2frepo2docker-2fbuildpacks-2fconda-2factivate-2dconda-2esh-cf5cc0: file does not exist(repo2docker)

When I remove those steps, later steps fail because mamba and conda are both unavailable. What else can I do to document the process of switching from a non-Dockerfile config to a Dockerfile-based config?

The debug output is just for debugging, it's not intended to be buildable since you also need the full context that's created on the fly, and I don't think we should recommend it as a way to create a Dockerfile.
I've got a playground/proof-of-concept in https://github.com/manics/repo2shellscript if you're interested in seeing the full complexity.

Thanks @manics! Is there anything that can be documented about migrating from a config-file based build to a Dockerfile-based build? Does repo2docker provide any value add for that use case? I'm struggling with a build that has broken man pages, and I'm not seeing a repo2docker path to fixing it right now (since I think I need to do stuff as root to fix the issue).

https://github.com/jupyterhub/repo2docker/issues/700

Here's how I solved my problem: https://github.com/CryoInTheCloud/hub-image/pull/65
As you know, I totally missed the "Can I use repo2docker to bootstrap my own Dockerfile?" FAQ on my first time through :laughing: I've added a link to it where I would have seen it my first time through and replaced my bad suggestions with hopefully better ones, in addition to a new FAQ to resolve the specific problem with man pages.


DS18B20 Temperature will not be shown

Hi,
I've a connect DS18B20 to a Sonoff Basic.
Under the object the temperature will not be shown. I have restarted the adapter several time, but it doesn't help.
The sonoff is sending:
10:03:24 MQT: tele/sonoff/STATE = {"Time":"2018-06-15T10:03:24","Uptime":"0T00:10:14","Vcc":3.208,"POWER1":"OFF","Wifi":{"AP":1,"SSId":"S04_mesh","RSSI":84,"APMac":"EC:08:6B:F9:07:94"}} 10:03:24 MQT: tele/sonoff/SENSOR = {"Time":"2018-06-15T10:03:24","DS18B20":{"Temperature":0.0},"TempUnit":"C"}
The iobroker is is receiving the temperature:

sonoff.0
2018-06-15 11:03:24.811
debug
stateChange sonoff.0.Sonoff_Basic_1.DS18B20_Temperature:  {"val":0,"ack":true,"ts":1529053404803,"q":0,"from":"system.adapter.sonoff.0","lc":1529052803660}

sonoff.0
2018-06-15 11:03:24.792
debug
[Sonoff_Basic_1] Received: tele/sonoff/SENSOR = {"Time":"2018-06-15T10:03:24","DS18B20":{"Temperature":0.0},"TempUnit":"C"}

sonoff.0
2018-06-15 11:03:24.772
debug
stateChange sonoff.0.Sonoff_Basic_1.Wifi_RSSI:  {"val":84,"ack":true,"ts":1529053404757,"q":0,"from":"system.adapter.sonoff.0","lc":1529053404757}

sonoff.0
2018-06-15 11:03:24.771
debug
stateChange sonoff.0.Sonoff_Basic_1.POWER1:  {"val":false,"ack":true,"ts":1529053404755,"q":0,"from":"system.adapter.sonoff.0","lc":1529052446688}

sonoff.0
2018-06-15 11:03:24.758
debug
stateChange sonoff.0.Sonoff_Basic_1.Vcc:  {"val":3.208,"ack":true,"ts":1529053404748,"q":0,"from":"system.adapter.sonoff.0","lc":1529052453690}

sonoff.0
2018-06-15 11:03:24.737
debug
[Sonoff_Basic_1] Received: tele/sonoff/STATE =  {"Time":"2018-06-15T10:03:24","Uptime":"0T00:10:14","Vcc":3.208,"POWER1":"OFF","Wifi":{"AP":1,"SSId":"S04_mesh","RSSI":84,"APMac":"EC:08:6B:F9:07:94"}}

`
Regards
Sebastian

Sorry I cannot reproduce you error. I even built your log into tests and everything is OK.
Sure that you updated the browser? F5?


refactor: made logger clearer

Change Description:
Before:

After:

Manually merged into the current feat/extract-info-from-tiff branch to avoid conflict


fix(versions): update Activiti/activiti-cloud-full-chart-security versions

UpdateBot pushed helm dependency: activiti-cloud-full-example to: 7.1.329

UpdateBot commands:
updatebot push-version --kind helm activiti-cloud-full-example 7.1.329


怎么卸载啊

有没有一键卸载

重新运行一键脚本，有卸载选项
Sent from ProtonMail Mobile
On Thu, Dec 31, 2020 at 13:31, Sharklet<EMAIL_ADDRESS>wrote:

有没有一键卸载
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.