Portable packed SIMD vector types

This RFC extends Rust with portable packed SIMD vector types. The RFC is forked from the closed RFC 2366 by @gnzlbg.
Rendered

I have so many, many thoughts, the first and most important of which is:
Not a single SIMD crate has a good and complete version of portable SIMD, and we don't even have full Neon support in Nightly, so this is probably far too early to RFC.

So if your position is "here's a junk drawer of stuff that you maybe could use" rather than "here's a complete and ready solution", well that's not wrong I suppose.
I think that if you want to expose LLVM's intrinsic concepts for normal Rust users to build with then that's super cool, but we should do that as directly as possible if that is the goal.
Currently the packed_simd crate is "long-term-blocked" on a lack of Const Generics in Rust. The ideal design (with const generics) would be a breaking change from the current design. For just one particular example: the inner field of the Simd type is commented as something that's supposed to be private but that can't be because the shuffle! macro needs it public. If there was const generics then shuffle could be made into a method and the field could be made private. Obviously we should not stabilize anything that has planned breaking changes in its future.
Speaking personally, when I tried to understand packed_simd enough to use it, I could not get a handle on it because all operations funnel through a single Simd<T> type, and so all methods and traits for all vector lengths of all types end up on a single page. This makes the docs effectively unnavigable. Instead I just used core::arch directly and trusted the auto-vectorizer to not screw up too badly for non-intel platforms.
I'm not at all saying that Rust is fine without portable SIMD. That's important to have in the long term.
I am saying that packed_simd, in its current state, is not what we need.

So if your position is "here's a junk drawer of stuff that you maybe could use" rather than "here's a complete and ready solution", well that's not wrong I suppose.

I think that's an inappropriate characterization. Rather, my position is that packed_simd is ready for real-world use, so we should make it available for real-world use and add more stuff later instead of not making it available while figuring out what else is possible to add.

I think that if you want to expose LLVM's intrinsic concepts for normal Rust users to build with then that's super cool, but we should do that as directly as possible if that is the goal.

I want to expose the concept of portable SIMD that LLVM has without exposing LLVM-specific idiosyncrasies but making the concept work well on the Rust level. That's what packed_simd is. Also, I gather than so far rustc developers have resisted exposing LLVM's intrinsics directly.

The ideal design (with const generics) would be a breaking change from the current design.

I understand the desire for ideal design, but real artists ship, and actually being able to use stuff is an important feature. The application programmer should not look at what u8x16, u16x8, etc. desugar into. I'd be fine with some compiler hack to hide what they desugar into for now so that changing the desugaring once const generics arrive wouldn't be a breaking change. (No one should be paying attention to what core::arch::x86_64::__m128i desugars into, either, and that one is on stable.)

Speaking personally, when I tried to understand packed_simd enough to use it, I could not get a handle on it because all operations funnel through a single Simd<T> type

In general, it's not useful to pay attention to Simd<T> but to use u8x16, u16x8, etc. The documentation pages for these look understandable enough to me.

I am saying that packed_simd, in its current state, is not what we need.

Implementing portable SIMD is a lot of work that has already been done. I'm a user of portable SIMD, so I care, I'm not going have the bandwidth to implement portable SIMD in a different way. I think we should take the work that has been done instead of blocking it by saying "not like that".

The other big question I'm not seeing an answer to in the RFC is: Why does it matter if this is in std rather than a separate crate? The Motivation section seems to be entirely about why higher-level crates like packed_simd need to exist, but the fact that packed_simd is usable and useful today doesn't automatically mean it'd benefit from being in std. Even this RFC is citing multiple examples of crates built atop packed_simd, seemingly showing that its vocabulary types are effective as a crate already.
There are a few entries in the FAQ section that touch on this, but they just confused me. For example, this one basically says that packed_simd was "abandoned due to the lack of roadmap commitment from the core stakeholders to get into the standard library," which seems highly misleading at best since it leaves out the whole "waiting for const generics" aspect. Then there's these two which appear to answer my question directly:

Why does this need to go into the standard library instead of living in the crate ecosystem?
The functionality fundamentally relies on the compiler backend (presently LLVM) and, previously, there has been reluctance to expose the kind of compiler intrinsics that packed_simd depends on. Furthermore, the types provided by the crate are important vocabulary types, so asking them to live in the crate ecosystem is similar to asking u32 and u64 to live in the crate ecosystem.
Can't this be implemented in the crate ecosystem on top of std::arch instead of compiler intrinsics?
In theory, yes. In practice, no. It would be a massive undertaking. It makes no sense to redevelop functionality of compiler optimizers and backends that already exist and work just in order to work around Rust's process issues.

But I don't get how promoting packed_simd to std would resolve the disagreement over intrinsics, or over whether packed_simd's vocabulary types are the optimal ones for high-level SIMD, or how it would make the implementation undertaking any less massive. If anything, these seem like additional reasons why we're not ready to put anything high-level into std yet.

From the view of a consumer, who wants to use SIMD in rust, today barriers exist - you choose nightly and packed simd (which from my view, was a great user experience honestly), or you need to use stable with std::arch - reading std::arch's SIMD intrinsics was really daunting and I don't think I'd be able to be effective in using them.
I don't think that packed_simd was confusing to use at all, a good hour with the docs and I had working examples that were able to improve real workloads I have. Like anything it's a slider - do you use the "lowest level" and get all the control? Or do you use a "high level" implementation (ie the faster crate) but lose some of the control? Or worse, rely on autovectorisation and have no control? I think that packed simd appears a good middle ground on this scale. It's low enough to expose the operations that exist on hardware, to solve real problems, but without being so low level that I'm required to read an intel architecture manual.

Currently the packed_simd crate is "long-term-blocked" on a lack of Const Generics in Rust. The ideal design (with const generics) would be a breaking change from the current design. For just one particular example: the inner field of the Simd type is commented as something that's supposed to be private but that can't be because the shuffle! macro needs it public. If there was const generics then shuffle could be made into a method and the field could be made private. Obviously we should not stabilize anything that has planned breaking changes in its future.

Waiting for Rust to be "perfect" before making a change like this, seems like the wrong answer. This is why tools like macro backtrace, and ASAN remain locked behind nightly, despite being hugely useful tools. If the insistence is that shuffle! needs const generics, then why not move shuffle behind a feature flag for nightly, hide the Simd implementation so that it's completely internal to the library, and then allow something useful to be accessible to consumers. When const generics happens, it can be changed internally in the library, and shuffle exposed to users at that point.
From my view, I support this PR and seeing packed_simd become part of std.

From the view of a consumer, who wants to use SIMD in rust, today barriers exist - you choose nightly and packed simd (which from my view, was a great user experience honestly), or you need to use stable with std::arch - reading std::arch's SIMD intrinsics was really daunting and I don't think I'd be able to be effective in using them.

That's good reason to get more SIMD operations stabilized for other architectures, so that the packed_simd crate can run on stable. I could be mistaken, but it seems like with the new inline assembly set to be stabilized, it might not even be necessary to get every SIMD function stabilized: only the basic register types for each architecture?
There's still no good motivation presented for why the portable-SIMD API needs to be in std though. There is an extremely high bar for APIs in std which is necessary for Rust's long-term commitment to stability.

The other big question I'm not seeing a clear answer to in the RFC is: Why is it important to get this into std in the near future?

My motivation is that Firefox has been shipping first with simd and then with packed_simd for years, and the apparent abandonment of packed_simd poses a risk to the product I'm working on and that uses packed_simd.
The Rust community reacts with shock every time someone new discovers how this is accomplished by working around Rust's stability story (and, too ofter, with hostility, too).

The Motivation section seems to be entirely about why higher-level crates like packed_simd need to exist, but the fact that packed_simd is usable and useful today doesn't automatically mean it'd benefit from being in std.

The compilation of packed_simd needs access to the kind of compiler internals that are not exposed on stable. To avoid exposing those compiler internals on stable. It belongs in std for the same reason count_ones() belongs in std.

Even this RFC is citing multiple examples of crates built atop packed_simd, seemingly showing that its vocabulary types are effective as a crate already.

As noted, if you are OK with compiling with nightly features enabled, packed_simd is effective and works.

There are a few entries in the FAQ section that touch on this, but they mostly left me confused. For example, this one basically says that packed_simd was "abandoned due to the lack of roadmap commitment from the core stakeholders to get into the standard library," which seems highly misleading at best since it leaves out the whole "waiting for const generics" aspect.

I guess both reasons apply.

But I don't get how promoting packed_simd to std would resolve the disagreement over intrinsics,

It would resolve it to the same extent as providing count_ones() resolves the issue in the scalar domain by not exposing an LLVM intrinsic but by exposing an operation internally implemented in terms of an LLVM intrinsic.

or over whether packed_simd's vocabulary types are the optimal ones for high-level SIMD,

They are on the same level of abstraction as u32 and u64. They are lower level than bignums are in the scalar domain. They are appropriate for a language that considers it appropriate to have u32 and u64.

I could be mistaken, but it seems like with the new inline assembly set to be stabilized, it might not even be necessary to get every SIMD function stabilized: only the basic register types for each architecture?

The optimizer has less visibility into inline asm than into ISA-specific intrinsics. Both are ISA-spefic and, therefore, don't remove the need for being able to write portable code for SIMD like we get to write portable code for scalars.

There's still no good motivation presented for why the portable-SIMD API needs to be in std though.

It needs to be in std, because it exposes compiler capabilities that can't be accessed (on non-nightly) by the crate ecosystem (unless the compiler team's stance on exposing implementation details of the compiler changes radically).
That is, it needs to be in std for the same reason count_ones() is in std as opposed to being in the crate ecosystem.

FWIW, the plan was always to put a platform independent SIMD API into std. It cannot live outside of std because its implementation relies on LLVM specific bits. This is not a simple matter of "let's stabilize more intrinsics for other architectures so that people can write things like packed_simd in crates on stable."
(I haven't had a chance to read the RFC yet, but wanted to chime in on at least that point now. I hope to comment more later.)

The compilation of packed_simd needs access to the kind of compiler internals that are not exposed on stable.

Okay, that's much more interesting. So, what exactly are these internals? And why can't we expose them more directly? There must be an awful lot of baggage associated with them if "stabilize all of packed_simd in its current form that we know will be obsolete someday" is somehow a better choice than anything smaller or more fine-grained that'd allow packed_simd to work on stable, but the RFC text doesn't explain any of this.

Okay, that's much more interesting, and brand new information to me. So, what exactly are these internals?

packed_simd has:
#![feature(
    repr_simd,
    const_fn,
    platform_intrinsics,
    stdsimd,
    aarch64_target_feature,
    arm_target_feature,
    link_llvm_intrinsics,
    core_intrinsics,
    stmt_expr_attributes,
    mmx_target_feature,
    crate_visibility_modifier,
    custom_inner_attributes
)]

The key perma-unstable internal things are platform_intrinsics, link_llvm_intrinsics, and core_intrinsics. This list isn't complete, but the sort of things that these provide access to are LLVM internals that allow the use of LLVM to generate code for statically-expressed shuffels and allow the use of LLVM to figure out things like "I'd like to do a lane-wise addition of two vectors of 8 lanes of 16-bit signed integers. Go figure out what that means for the current target CPU".
packed_simd is about exposing these back end capabilities plus

Mask vectors (covered in the FAQ)
Obvious Rustic things like having safe ways to do the bitwise casts that are actually safe.
Obvious Rustic things like mapping the back end capabilities that map to Rust traits e.g. for addition.

Considering how thin the layer is on top of the intrinsics, I find it odd how there is resistance to having the Rustic layer and suggestions to expose the compiler internals instead. In the scalar domain, the Rust principles have gone in the other direction: Exposing capabilities like count_ones() in std but not letting the crate ecosystem depend on the compiler internals directly.

Things like repr(simd) have also been considered historically perma-unstable.
In addition to what @hsivonen said, the other problem with dealing with the llvm features directly is that they are difficult to use correctly, and can easily lead to instruction selection errors at monomorphization time, which is something rustc doesn't do today.
If you try to think about what it takes to expose llvm's platform independent SIMD support without exposing llvm's quirks, then something like packed_simd is what you get.

The performance could by further improved by requiring the input data to be aligned to a 16-byte boundary, and/or by handling the elements before the next 16-byte boundary in a special way.

Rust could face this growing alignment requirements problem with optional alignment annotations, as in Zig:
https://andrewkelley.me/post/unsafe-zig-safer-than-unsafe-rust.html
If that's too much heavy for Rust, then perhaps we could introduce some kind of alignment type in the std lib:
fn average_fast256(xs: &[Aligned<f32, 16>]) -> f32 {

If that's too much heavy for Rust, then perhaps we could introduce some kind of alignment type in the std lib (this requires const generics):
    fn average_fast256(xs: &[Aligned<f32, 16>]) -> f32 {

Rust already has a way to align memory, it's just really obscure and requires you to call the allocator with a layout.  packed_simd also has a way to avoid this with it's types of aligned/unaligned input. Which while not optimal, does "side step" this issue.
I think that "waiting" on const generics is not a solution, because we will potentially be waiting for years here instead.
As an external viewer to this, an under-stated part of this issue seems to be "fear" that once included the api of "std::simd" can never ever possibly be changed at all under and circumstances. When the reality is Rust has editions, and over time this can be changed and can be improved. We should not be stopped that it's not perfect today, but it's inclusion can be made better over time, there are path ways to achieve this over time, and that the inclusion of this library opens many doors to developers.
Perfect is an impossibly high bar to set for everything, especially given that packed simd is already high quality, in production use, and the discussion laid out by @hsivonen and others is very well thought out.

When the reality is Rust has editions, and over time this can be changed and can be improved.

Not that I necessarily disagree with your broader point, but there's an important clarification to make here. Editions do not permit breaking changes to std in their current form. The same version of std must work across all editions. If we create std::simd and get it wrong enough that const generics would provide significant quality of life improvements, but cannot be done in a compatible way, then our main approach at that point would be deprecation of an old API and adding a new one.

I should note here that actually const generics are not required to move packed_simd to a usable "final design". There's another route available to us.
We can simply offer the necessary operations that require compile time constant arguments using the rustc_const_arg attribute which is heavily used already in core::arch. It makes the function minorly unusual but not in an overly difficult to use way.
If we took that path, we could have packed_simd usable in the standard library in Nightly "soon" and we could optimistically aim for a stabilization of packed_simd within the standard library "by the end of the year".

We can simply offer the necessary operations that require compile time constant arguments using the rustc_const_arg attribute which is heavily used already in core::arch.

I'm unsure if it's because of GitHub infra issues right now, but I failed to find an example of  rustc_const_arg. Could you point me to an example, please?

@hsivonen I think @Lokathor meant rustc_args_required_const. Here's an example: https://doc.rust-lang.org/src/core/up/stdarch/crates/core_arch/src/x86/sse42.rs.html#503-512
Specifically, the imm8 parameter needs to be a constant, known at compile time. The constify_imm8 macro tricks the compiler into think it's a constant, even if it isn't, by using exhaustive case analysis.

Ah, yes, BurntSushi guessed my meaning correctly :)

So, haven't fully read the RFC, but based upon what I read, I'm not really on favour of disabling certain types on certain platforms. The only other major types I know that do this are atomics, but there's a key difference here from atomics: a correct implementation for atomics is actually not possible if they aren't supported by the underlying hardware, but SIMD vectors can be emulated just fine without it, albeit slower.
Obviously, it is ideal to optimised based upon what the target architecture supports, but I feel like it'd be nice to be able to, for example, have a 4-D vector struct wrap an f32x4 type and just fall back if it's not supported. The logical meaning of the SIMD types is IMHO useful and I feel like explicitly excluding it if the hardware doesn't support it isn't ideal.
Precedent for this: we have all sorts of methods on primitives that may or may not have hardware support, but that doesn't stop us from using a software implementation of sincos for example.

So, haven't fully read the RFC, but based upon what I read, I'm not really on favour of disabling certain types on certain platforms.

Does the RFC accidentally suggest that somewhere?

...yup, I see now a direct reference to say that polyfills will be added. I thought there was, but I guess that I skimmed too fast. Carry on. :sweat_smile:

I'm concerned the masking semantics are too leaky. m32x4 should not be guaranteed to be 128bits wide. The only thing that type needs to specify is that it can be used to mask any ?32x4 type. This leaves the implementation open to use the avx512 mask registers when available.

Two cents:

I'm kind of hyped for portable SIMD to be added to std. This is something really cool that the graphics/comp. geometry/physics programmer in me loves to see. I fully support having it on stable eventually.
If anything, the discussion so far tells us that the const-generics team could use some support. Last I heard, CG is partially blocked on Lazy Normalization. If anyone knows about that topic, and can spare some hours, please go help :)

+ 1 extra cent:
Thank you @hsivonen for writing this RFC. It is no minor effort :heart:

Also once the select function is expressed in terms of mask registers, then clang can turn existing sequences of op + select functions into the same instructions as the mask/maskz variants.
See https://godbolt.org/z/8rNdKW

I'm concerned the masking semantics are too leaky. m32x4 should not be guaranteed to be 128bits wide.

Not guaranteeing mask type width would be a problem for in terms of zero-cost bitcasts to integer vectors and zero-cost transmute to core::arch types.

Thank you @hsivonen for writing this RFC. It is no minor effort heart

@gnzlbg did the hard work. I'm tying to promote the work and added the FAQ.

I'm concerned the masking semantics are too leaky. m32x4 should not be guaranteed to be 128bits wide.

Not guaranteeing mask type width would be a problem for in terms of zero-cost bitcasts to integer vectors and zero-cost transmute to core::arch types.

I agree that would be the case. But as proposed, on systems that support EVEX encoding your mask types are non-zero-cost for masking, which I would consider a more import consideration.
In the current draft, how would the m1xN values be created? Would every comparison function have two variants?

avx512 is quite simply not a portable CPU feature set.

I wanted to go meta on my previous objections to guaranteed mask layout. I feel like the responses got bogged down in the niche-ness of AVX512. I feel the standard library should be extremely conservative in the guarantees it makes. Another RFC can always add more guarantees at a later date if there proves a real need. The eventual RFC guaranteeing the size of bool seems like a good comparison.
To add a another data point (also niche) back to the original argument. The draft spec for the Risc-V packed SIMD has mask registers that don't follow the convention assumed in this RFC. They are the full width, but only use 1 bit per element.
On the implementation side, if this library is a wrapper for LLVM intrinsics, both the fcmp and icmp intrinsics return N x i1. Adding on a stricter requirement to the return type risks non-zero-cost implementation on some ISA in the future.
To address responses from the RFC submitter:

Transmute to core::arch types: If the client code wants to perform platform-specific ops then they should already know the platform specific size and format of the mask. The library shouldn't need to guarantee anything for this.
Cast to unsigned int vector: The guarantee of a particular bit-pattern can be part From/Into implementation between masks and unsigned vector types.

Can you please explain why portable SIMD has to be tied to LLVM intrinsics and not implemented in terms of arch intrinsics? I understand that it will be easier to start with them, but IIUC there are no fundamental blockers for the latter. One problem which comes to mind is that by default we use per-compiled std, so changing target features will not affect portable SIMD code implemented via arch intrinsics as part of std, but I would say it's an argument for keeping portable SIMD in a separate crate outside of std.
Also I think this RFC lacks explanation of how portable SIMD will work with runtime detection. This problem is not an easy one, especially if we want to compile several versions of an algorithm with different target features coming from a separate crate. Ideally I would like to see something like this, but even it does not solve the issue with a separate crate.

The RFC doesn't lack an explanation of runtime detection, it explicitly rejects the idea: https://github.com/hsivonen/rfcs/blob/ppv/text/0000-ppv.md#shouldnt-the-portable-simd-implementation-detect-instruction-set-extensions-at-run-time-and-adapt

The linked part talks about runtime detection at individual operation level, while my post is about algorithm level. I agree that that detection should not be performed for individual operations, since it can seriously degrade performance, but it's a very common expectations for SIMD accelerated code to be able to switch between different implementations at runtime depending on hardware capabilities. So I believe SIMD proposals should keep such use cases in mind.
popcnt is influenced by #[target_feature(enable="..")], so if portable SIMD will be implemented in terms of LLVM intrinsics, it should not be a problem (although note that the result is not ideal, e.g. foo1 does not get inlined and adding -C target-feature=+popcnt does not remove the detection code). But if we are to implement portable SIMD in terms of arch intrinsics, then IIUC it will not work as expected for default build.

Operations exposed by this low layer of portable SIMD are each too small to use runtime detection. This RFC is not generally proposing any operation big enough to warrant a feature check and then branch to one variant or the other.
Even sin and cos don't really benefit enough from feature variations to offset the cost of the check for a higher feature level.

Can you please explain why portable SIMD has to be tied to LLVM intrinsics and not implemented in terms of arch intrinsics? I understand that it will be easier to start with them, but IIUC there are no fundamental blockers for the latter.

This is covered in the FAQ. While there is no theoretical obstacle, there's a huge practical obstacle: Rewriting a large chunk of code that LLVM has (and, I believe, GCC has) and Cranelift either has or is going to have makes no sense as a matter of realistic allocation of development effort and its hard to motivate anyone to do the work given that the work could be avoided by delegating to an existing compiler back end.
More bluntly: Those who suggest re-implementing the functionality on top of core::arch haven't yet actually volunteered to do it and shown existence proof of the result. It's far easier to say that the implementation should be done that way than to actually do it. OTOH, the implementation that delegates to an existing compiler back end actually exists.

Basic operations like addition and insertion are currently implemented in corearch using the simd_* platform intrinsics. These get forwarded to the appropriate generic llvm intrinsics by rustc_codegen_llvm. I have implemented most of them in rustc_codegen_cranelift, as there are only like 10 of them that are generic over input and output vector sizes. For the more platform specific simd intrinsics corearch directly calls platform specific llvm intrinsics. These are different for every platform and vector size. corearch uses more than 100 for x86 alone. This means that they have to be reimplemented by the backend for every combination of platform and vector size, while for the simd_* platform intrinsics only a handful have to be implemented to work on every platform with every vector size. (If performance matters, you will still have to handle every platform and vector size combination individually, but when you only want completeness, it is very easy to implement them all)
Preferably the portable packed SIMD vector types only expose operations that can be implemented using simd_* platform intrinsics. This makes it both easier to handle all platforms when implementing it (just call simd_* and don't worry about the current platform) and makes it much easier for alternative codegen backends to support it.

More bluntly: Those who suggest re-implementing the functionality on top of core::arch haven't yet actually volunteered to do it and shown existence proof of the result.

Actually, I have. I just stopped after f32x4 because my one user didn't need anything else and I got kinda bored with it.

More bluntly: Those who suggest re-implementing the functionality on top of core::arch haven't yet actually volunteered to do it and shown existence proof of the result.

Actually, I have. I just stopped after f32x4 because my one user didn't need anything else and I got kinda bored with it.

This isn't really a constructive piece of input. @hsivonen's point still stands that an alternative doesn't exist especially if you have become bored with the continued development of your version. What is currently offered by packed simd not only exists today, but is in (probably production) use today as mentioned which goes a long way to proving it's value even if not "perfect".

Rust is not generally in the business of stabilizing things into the standard library before they're ready. This is what keeps our standard library quite high quality. I stand by this policy.
The biggest limit to Stable portable SIMD via explicit intrinsics, in terms of serving x86/x64 and ARM/Aarch64 (the majority of processors in use), is the fact that the ARM intrinsics aren't on Stable, and they aren't even complete on Nightly. They're like 11% complete on Nightly. It's a bit of a drag, but the answer here is to submit PRs to stdarch and add more intrinsics so that we can move ARM intrinsics towards being Stable.
Also, packed_simd can become available on Stable another way: instead of evading the Nightly feature usage by just throwing it into the standard library, consider petitioning the lang team to stabilize all of the individual Nightly features that it needs for it to build on Stable without being in the standard library. There's only like 3 of them, and then the whole Rust world would benefit from the additional language abilities.

I think what @Lokathor says makes a lot of sense:

When you stabilize something into the stdlib, you're more or less forced to maintain what you've delivered. Breaking changes are... not a thing there? Maybe when getting into an Edition, but otherwise, no, as far as I'm aware. And you don't want to chain this feature to a lesser version of it just because it was not ready and fully thought and tested through before it got published.

The shortest path to using this on Stable is to stabilize the parts that it needs, and not to make it a part of the stdlib.

And to these points, I'd like to add mine:

Anything that's good enough, important enough and mainstream enough, will eventually become part of the stdlib. That's how things tend to work, and I think that's great - this means that an amazing library used by everyone could eventually become a standard and be included in the language itself (well the stdlib anyway). This particular feature doesn't need to be a part of the stdlib right away - it can become part of it later. In the future it will probably be completed, polished, and brought to such a good level that it becomes obvious that it needs to be a part of the stdlib. In other words, I think it will be part of the stdlib, but in a farther future than is maybe proposed here.

I'm trying to understand what the real blockers to moving forward with this RFC are. There's an implicit motivation of getting packed_simd's API to build on stable Rust, but the RFC itself is focused on offering a portable SIMD API in std.

How relevant is the question of whether to build on LLVM intrinsics vs std::arch for offering a portable SIMD API in std?
From a public API perspective it seems like that only becomes significant when we're suggesting building on std::arch as a way to get a portable API that compiles on stable Rust that's "only" blocked on a lot of work that's well-defined? If that's the case then it doesn't seem like it's really a blocker for this RFC, since the plan has always been to offer a portable API in std.

The question then is why not this proposed API?
There seems to be a general impression in the comments here that the API proposed is not ideal. What would a more ideal one look like? From what I can see, it looks like the RFC isn't proposing the vector types as an alias of some generic Simd type, so there's no surprises there. There's the potential of improving shuffle with const generics but that doesn't seem like a blocker for std, which already has precedent for hiding unstable details behind macros.

the current packed_simd does use a generic SIMD type as the type that backs all other types. So, if this RFC is proposing that we just suddenly accept packed_simd as is, that's what we'd have.

I'd like to note that all of this should hopefully go into core, not just std.

I'm trying to understand what the real blockers to moving forward with this RFC are.

AFAICT, the main blocker is that this lacks a champion within the libs team. (I don't have the bandwidth to become that person at this time.)

How relevant is the question of whether to build on LLVM intrinsics vs std::arch for offering a portable SIMD API in std?

As noted earlier, I believe in terms of the amount of work and who is willing to do the work, the only realistic path forward is to use the LLVM (or in the future other back end) facilities as packed_simd currently does. Previous decisions indicate that exposing those LLVM facilities directly is a no-go, as it would tie Rust LLVM. packed_simd is just enough abstraction to allow for other back ends in the future, which is why I think packed_simd itself in the form of core::simd (yes, this would belong in core rather than std) is the right abstraction.
While building on std::arch is theoretically possible, I believe no one is going to volunteer to reimplement things that LLVM already provides for the whole breadth of packed_simd. (A demo of one or two features isn't enough. It's already known that it's theoretically possible. The issue is doing all the work.)

I do not believe that it would tie rust particularly to LLVM, it would just tie that one crate using the LLVM intrinsics to LLVM until an alternate backend also had the same intrinsics. Which is arguably less disruptive than what happens with packed_simd in core, because with packed_simd in core now everyone would need to wait on alternative backends supporting all the packed_simd intrinsics, instead of just whoever is uses one particular crate off of crates.io.
Incidentally wide got a new release over the weekend and it now supports all the 128-bit types and for many of their desired ops. Runs on Stable and everything. Sure, it's not immediate 100% coverage of every possible SIMD thing, but there's only so much time in the day so you gotta start somewhere. If there was a more realistic path forward because of being unblocked by the general SIMD blockers I'm pretty sure that it would be easy enough to get additional people interested as well.

I do not believe that it would tie rust particularly to LLVM, it would just tie that one crate using the LLVM intrinsics to LLVM until an alternate backend also had the same intrinsics.

It is very likely to have at least one crate in the dependency graph that needs a certain intrinsic, which is actually quite disruptive. For example the rand crate indirectly depended on ppv-lite86 which always requires certain x86 specific intrinsics on x86 when the simd feature is enabled. Older rand versions always enabled it, but newer versions don't anymore due to some confusion about if simd or no-simd is the default. I recently implemented enough llvm intrinsics in cg_clif to make ppv-lite86 work both with and without simd enabled.

Which is arguably less disruptive than what happens with packed_simd in core, because with packed_simd in core now everyone would need to wait on alternative backends supporting all the packed_simd intrinsics, instead of just whoever is uses one particular crate off of crates.io.

All packed_simd methods are likely to be #[inline]. This causes codegen to be deferred to the calling crate, which means that you only need to support their implementation when called. This is also how I could compile core::arch with cg_clif without having to implement all intrinsics. packed_simd also doesn't use any platform specific llvm intrinsics, so the amount of intrinsics necessary to support any platform is much smaller. As an additional bonus the non-platform specific llvm intrinsics have consistent names with respect to vector and lane sizes.

I’ve opened #2977 to establish a project group to work on std::simd. Check it out, see what you think of the scope, and please reach out if you’d like to be involved 😃
Thanks everyone for all your effort so far!

Update on the project group
The Portable SIMD group (@rust-lang/project-portable-simd) has been established and very busy! We’re mostly working in the new https://github.com/rust-lang/stdsimd repository, scaffolding the core::simd API.
We’ve also published a new patched version of packed_simd on crates.io as packed_simd_2 that you can patch in to your existing projects using the package field in the meantime:
[dependenciea.packed_simd]
package = “packed_simd_2”


Texture Mask Missing in RoadMaster - My beautiful road wear lines are gone!!

CARLA version: 0.9.12
Platform/OS: Ubuntu / Windows
Problem you have experienced: Road wear lines are missing
What you expected to happen: Road wear lines to be present
Steps to reproduce: Load the Town10HD map from either the package or built from source, compare to prior versions.
Other information (documentation you consulted, workarounds you tried):
The RoadMaster-based Material Instances in the 0.9.12 version of CARLA are the Scalar Parameter Values / Texture Parameter Values that enable blending a Texture Mask to add road wear lines to the map are missing. This feature is documented in https://carla.readthedocs.io/en/0.9.12/tuto_M_custom_road_painter/#paint-the-road. The effect can readily be seen by comparing Town10HD in 0.9.12 to prior versions.
The below two images show the material instance + Town10HD map in CARLA 0.9.10, which shows off the road wear lines

The below two images show the material instance + Town10HD map in CARLA 0.9.12, which does not have road wear lines.

Hi @jimmyw404,
We are now working on finishing a maintenance release, which will soon come out. Where this problem will be fixed. If you need to fix it you just have to select the material of the road and put the mask of the Town10.

Sorry for the inconvenience it may have caused.

Hi @jimmyw404,
We are now working on finishing a maintenance release, which will soon come out. Where this problem will be fixed. If you need to fix it you just have to select the material of the road and put the mask of the Town10.
Sorry for the inconvenience it may have caused.

Works great, thanks!!!!
Closing this ticket...


getting started connectivity test fails on arm64 (cilium 1.10.0-rc0

Bug report

General Information

Cilium version v1.10.0-rc0 (helm chart, no options)
Kernel version: 5.4.0-1030-raspi #33-Ubuntu SMP PREEMPT Wed Feb 24 11:20:11 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux
Orchestration system version in use: Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0-rc1+k3s1", GitCommit:"2705431d9645d128441c578309574cd262285ae6", GitTreeState:"clean", BuildDate:"2021-04-21T23:32:49Z", GoVersion:"go1.16.2", Compiler:"gc", Platform:"linux/arm64"}

How to reproduce the issue
To reproduce, follow the k3s getting started doc on an arm64-based kubernetes cluster, which will fail at the connectivity test step
The following images are used in the connectivity test yaml, and lack arm64 variants. I'm following the k3s Getting started doc on a rapsberry pi 4 cluster.

byrnedo/alpine-curl (there was a marged PR for this in its repo, but it doesn't seem to actually be available from that travis config)
cilium/json-mock

This is a followup for issue #15840, and I believe the final missing piece for arm64 support in 1.10.0

We are still experiencing this issue. Is this resolved? Is there a workaround?
cilium-cli: v0.12.6 compiled with go1.19.1 on darwin/amd64
cilium image (default): v1.12.2
cilium image (stable): v1.12.3
cilium image (running): v1.12.3


Welcome messages can now be set to be pop-up windows or fullscreen.

Welcome messages can now be set to be either a pop-up window or full-screen sized in the admin console. The default is to be a pop-up window.

A pop-up sized welcome message:

A full-screen welcome message:

On mobile phones (devices smaller than 576px), the welcome message will display full-screen regardless. This is because it looks neater on a smaller screen.
Before the changes, the welcome message modal on mobile phones was a pop-up:

Full-screen modal on a mobile phone:

Closes #79

Cheers Paul, references to Microsoft Graph removed. I have no clue how they got in there - I definitely didn't put them in 🙃


Add anoncreds issuance and presentation support [DEV-4592]

Task linked: DEV-4592 Complete credential issuance flow with ACA-Py


Multiple Charts 1 Canvas

It would be cool if I could put two chart types in one canvas, for example a line graph and a bar graph....

Example (possible usage):
var data = {
    labels: ["January", "February", "March", "April", "May", "June", "July"],
    datasets: [
        {
            type: "Line",
            label: "My Line Graph",
            data: [65, 59, 80, 81, 56, 55, 40]
        },
        {
            type: "Bar",
            label: "My Bar Graph",
            data: [28, 48, 40, 19, 86, 27, 90]
        }
    ]
};

@TheColorRed  Can I get zip file?

@vajrayanu http://www.chartjs.org/docs/#chart-configuration-mixed-chart-types


Ontario/Canada Recently Showing Zero Weekly Cases

When I checked on April 9th, the microCOVID score for Canada/Ontario was 0.1 - the Details showed zero weekly cases as of April 8th.  The data was refreshed as of April 10th but is still showing zero weekly cases.  The last time I checked on April 2, the numbers were reasonable.   I spot-checked some health units in Ontario - they look reasonable.  The other provinces/territories are also showing reasonable microCOVID data.
I checked cases_timeseries_prov.csv - the Ontario and other provincial case data looks good.  There is a note on the COVID-19 Canada Open Data Working Group site that they are making changes but those only go into effect on April 15th.

As of April 15th, I am seeing microcovid.org data for Ontario.  However. 39,625 weekly cases and a 31.9% positivity rate seems high.  https://www.publichealthontario.ca/en/data-and-analysis/infectious-disease/covid-19-data-surveillance/covid-19-data-tool is reporting 40699 cases over the last 14 days while cases_timeseries_prov.csv is showing about 24000 weekly cases.  The weekly positivity rate is about 19.9% according to the Public Health Infobase.
I spot checked some of the Ontario health units.  Their weekly cases numbers look reasonable but they all showed a positivity of 31.9% which given the spread in weekly cases/100K does not seem right.

As of this morning, the data for Canada, the provinces, and the Ontario Health Units is still showing an April 15 date.  This is when the Canada Open Data Working Group made changes to their reporting.

Many other regions also stopped updating on April 15th.

As of this morning, Canada/Ontario (including health units) is current as of April 30 and the case counts/positivity look reasonable.


Line Indentation is ignored, no way to enable it.

I'm using Tesseract 3.05.02 (packaged in Fedora 29), and I'm using it to read text in a code editor (same monospace font, different colors and styles -monospace/italic-)
The command I'm using is tesseract -c preserve_interword_spaces=1 -psm6 <input> <output>. I need plain text, not HTML.
I didn't find a way to preserve the line indentation, even a relative one.
So if I read a text editor that contains
a {
    color: red;
}

Tesseract tells me it
a {
color: red;
}

and ignores to whitespaces before the line.
It would be a good addition to have an option like preserve_prefix_whitespaces=1 to get the prefix indentation part of the text.

PDF and hOCR output have text positions, so that requirement would only make sense for text output. I know no other OCR software which preserves leading whitespace, and even if it were implemented, it would not fit your needs because it could output relative whitespaces, but not the exact number of blanks needed for code indentation.
Why don't you use a code formatter to fix the indentation after the OCR?

relative whitespaces would be perfect in my case.
I'm using Tesseract to test the rendering of a text editor I'm working on (Eclipse IDE in that case), and I would like from a given editor to automatically take a screenshot of the text area (that's provided by internal APIs) and then invoke OCR and compare what I get from OCR matches the text I want to see in the editor, ideally including leading whitespaces (even if that's just relative).


[json-dto] updated to v.0.2.9.

json_dto library updated to the latest version.

/azp run

Thanks for the update, @eao197! This can be merged as soon as CI goes through. Also for anyone looking at this in the future, the new GitHub repo is official AFAICT


Improve notifications/messages in deployment get command

As a:
Operator
I want:

[x] deployment get to show a message "No deployments found" when no deployments available to show instead of nothing.

[x]  When introspection is not configured in spk config, the generic message "You need to run spk init to initialize." is shown which can be improved by including details like what section/variable is missing in the configuration and inform running the spk init again. Similar to Validate messages would be useful.

[x] Hide the message (node:11557) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.

So that:
Know exactly what configuration is missing in spk config file.
** Repro steps **
Run spk deployment get with following setup

no table exist in the storage account
table exist but empty

From both scenarios, spk deployment get shows the following message.
(node:11557) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
Describe the solution you'd like:
Acceptance Criteria:
Describe alternatives you've considered:
Additional context:
Does this require updates to documentation?:

@sarath-p can you please size the task?

There's already a No deployments found for specified filters message shown when no deployments are found.

From testing these both scenarios 1) no table exist 2) table exist but no data, spk deployment get shows the following message.
(node:11557) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.

@sarath-p size estimate?

@sarath-p That warning comes from a library that's being included by the azure library, which is why a lot of other spk commands are seeing it too

@samiyaakhtar can we hide or wrap azure library messages such as

(node:11557) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.

?

This has been fixed with the new error codes that were added to all of spk.


Deep network a la Covington paper

Recently came across this project. Massively impressed by the clean and intuitive API 👏
I started with the basic Pooling method since it seemed to be the simplest. Then I noticed the Covington reference -- the paper seems to imply a deeper network whereas the PoolNet uses a single layer. Am I completely misreading it (feel free to tell me to shut up :-D )
Wondering if you experimented with adding additional layers? Would be happy to dig into it otherwise.
Best
Asif

You can definitely try stacking more layers on top. You can do this by writing your own version of the pooling layer, then passing it as the representation argument of the model constructor.
Assuming you are using the sequential models, I would strongly recommend building on top of the LSTM-based representation: it gets much better results.

Thanks for the tip. And I am using the seq models as you'd guessed. I will start with LSTM 👍
I guess what tripped me up with Pooling was that the target and input were sharing the same item_embeddings -- this may(?) not necessarily be true if one were to add layers. Covington et al. seems to suggest that the penultimate layer becomes pseudo-embeddings of sort.

As you say, the default implementations share the embeddings for the input and output layers. You can change that with your own representation layer. Whether or not the input and output embeddings are shared is unrelated to the number of layers; you can have a deep model without tying the embeddings. It may be worth trying both and seeing which one works better!


Allow infinite contract size

Description
This is a near trivial change to simply expose the ethereumjs option for allowing infinite contract size
Tasks

[ ] Add the option to the createTevm options
[ ] Write a test
[ ] Run pnpm all in the vm/vm folder to regenerate docs and run linter

@roninjin10 can we close as completed?


Introduction To CSS

Description

All about CSS
Why CSS is so important?

Domain
Frontend Dev HTML/CSS/JS
Type of Contribution
Documentation
Code of Conduct

[X] I follow Contributing Guidelines & Code of conduct of this project.

/assign

can I work on this issue?

/assign


Help actually using this to train YOLO

This might just be something that I am missing that should be super easy to figure out, but how can I actually use this dataset with YOLOv4? I was able to run the script (though I had to remove the f-strings because I am on Ubuntu 16.04 and python3.5) and get all the images. It seems to me that I should somehow be able to use the images and annotations pretty directly when plugging into YOLO, but the only instructions (or these) I found on training for the custom objects seemed like I would have to do the bounding boxes manually for each. My understanding is that the end goal of this is to get a new .weights file that represents both COCO and your data.
This definitely feels like something that would be super easy for someone who knew their way around YOLO and I really don't want to have to manually label all these points. Please help

Hi, now I recommend using YOLOv5 or even YOLOv7 because they have a PyTorch code representation. Annotations (bounding boxes) for all UAVVaste images are in annotations.json file in COCO format.
To use this data with the YOLO algorithm, you should convert them from COCO format to YOLO format (here is an example repository that does it).
Then when you have correctly converted data (I suggest previously visualizing sample bounding boxes for both COCO and YOLO formats), starting the training process is really straightforward - follow instructions from the aforementioned repositories.


doc: single logo, reformat source

Codecov Report

Merging #27 (c27d696) into main (8d40b73) will decrease coverage by 0.05%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main      #27      +/-   ##
==========================================
- Coverage   97.94%   97.88%   -0.06%
==========================================
  Files         169      169
  Lines        5349     5349
  Branches      799      799
==========================================
- Hits         5239     5236       -3
- Misses         63       66       +3
  Partials       47       47

Impacted Files
Coverage Δ

xpresso/binders/_extractors/body/discriminated.py
70.73% <0.00%> (-2.44%)
:arrow_down:

xpresso/binders/_extractors/body/file.py
96.00% <0.00%> (-2.00%)
:arrow_down:

xpresso/binders/_extractors/body/form.py
95.71% <0.00%> (-1.43%)
:arrow_down:

Codecov Report

Merging #27 (2df17d5) into main (8d40b73) will decrease coverage by 0.05%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main      #27      +/-   ##
==========================================
- Coverage   97.94%   97.88%   -0.06%
==========================================
  Files         169      169
  Lines        5349     5349
  Branches      799      618     -181
==========================================
- Hits         5239     5236       -3
- Misses         63       66       +3
  Partials       47       47

Impacted Files
Coverage Δ

xpresso/binders/_extractors/body/discriminated.py
70.73% <0.00%> (-2.44%)
:arrow_down:

xpresso/binders/_extractors/body/file.py
96.00% <0.00%> (-2.00%)
:arrow_down:

xpresso/binders/_extractors/body/form.py
95.71% <0.00%> (-1.43%)
:arrow_down:


[CARBONDATA-2487] Block filters for lucene with more than one text_match udf

[x] Any interfaces changed?
NA
[x] Any backward compatibility impacted?
NA
[x] Document update required?
NA
[x] Testing done
UT added
[x] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
NA

Build Failed with Spark 2.2.1, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder/4759/

Build Failed  with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder1/5913/

SDV Build Fail , Please check CI http://<IP_ADDRESS>:8080/job/ApacheSDVTests/4946/

Build Success with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder1/5919/

Build Success with Spark 2.2.1, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder/4765/

SDV Build Success , Please check CI http://<IP_ADDRESS>:8080/job/ApacheSDVTests/4951/

Build Failed with Spark 2.2.1, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder/4807/

Retest this please

Build Success with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder1/5965/

SDV Build Success , Please check CI http://<IP_ADDRESS>:8080/job/ApacheSDVTests/4991/

Build Success with Spark 2.2.1, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder/4812/

Build Success with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder1/5994/

SDV Build Success , Please check CI http://<IP_ADDRESS>:8080/job/ApacheSDVTests/5012/

Build Success with Spark 2.2.1, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder/4835/

LGTM


Minor update SA-Med2d 20M dataset

Added an additional logic to choose a fraction of samples per dataset for segmentation, expect the datasets with a small size already.

Thanks for spotting! The argument is now exposed to both default_segmentation_dataset and default_segmentation_loader now. GTG from my side!

Should we set verify_paths=False when default_segmentation_dataset is called in get_sa_med_dataset?

Yes, this definitely makes sense to add it to the dataset itself. Thanks for the mention!


[AzureClient] Expose feature gates for AzureClient users

Description
This PR exposes feature gates for AzureClient users to allow the use of experimental features.
Please see Design Doc (MSFT Internal) for context.
Reviewer Guidance
I would appreciate feedback on:

Using ExperimentalFlags instead of IConfigProviderBase (see design doc "Alternative Approaches" section for more info)
Implementation details
Where should supporting documentation go?

Misc
AB#3946

@scottn12 Mind if I add a changeset to this PR before you merge?

Please do @tylerbutler, thanks!


⚠ ClamAV has degraded performance

In da6ccc8, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 53 ms

Resolved: ClamAV performance has improved in 0c0fa90.


Fix ASCII logo not shown on openSUSE

On openSUSE the ASCII logo was not shown.

I hope it's the right place to fix it.
There was also a typo. ascii_Distro="antix"

:+1:


Empty log output after start minikube cluster

Podman Desktop version: 1.10.3
Minikube extension version: v0.3.0
Minikube version: v1.33.1

Did some research.
This is different kind of logs and won't be displayed in this tab.


[Bug] UI totally freezes when resizing window smaller than Medium (MD)

Describe the bug
UI Totally freezes when resizing window.
To Reproduce
Steps to reproduce the behavior:

Get the latest version of Blazor Hero.
Run it.
Open the Developper Console. Stretch de developper console until the MD breakpoint is hit.
UI Totally Freezes

Expected behavior
No Freezes when UI is resized
Screenshots

I Found the offending code.
When I comment Line 43 to 52 in MainLayout.razor The freezes Stops
 @* <MudHidden Breakpoint="Breakpoint.SmAndDown" Invert> *@
@*     <LanguageSelector/> *@
@*     <MudMenu Color="Color.Inherit" Dense Direction="Direction.Right"<EMAIL_ADDRESS>OffsetY="true"> *@
@*
 *@
@*             <MudTooltip Text="@_localizer["Toggle Dark Mode"]"> *@
@*                 <MudIconButton Color="Color.Inherit"<EMAIL_ADDRESS>OnClick="@((e) => DarkMode())"/> *@
@*             </MudTooltip> *@
@*
 *@
@*     </MudMenu> *@
@* </MudHidden> *@

The issue is coming from the Tooltip on the Darkmode Icon.
This now works perfectly:
<MudHidden Breakpoint="Breakpoint.SmAndDown" Invert>
<LanguageSelector/>
<MudMenu Color="Color.Inherit" Dense Direction="Direction.Right"<EMAIL_ADDRESS>OffsetY="true">

  <MudIconButton Color="Color.Inherit"<EMAIL_ADDRESS>OnClick="@((e) => DarkMode())"/>

</MudMenu>
</MudHidden>


SDK not on start.branch.io

https://start.branch.io/#/integrate-sdk/steps
No tab for the web :-1:

+1

This is still the case.

Thanks for the feedback on this! It is definitely appreciated, and has not been ignored.
We're speccing out a project to revamp this entirely, so this will be revisited in that project.  Since that process will be entirely different, I'm closing out this issue.


EASY: Add property names to the Model object

Currently, in the model object schema, there are some properties like hasParameter (function), name (string), and parameters (object). I think it would be cool that in the file where we create the model object, we also add in the columns that the user defined for the model.
You might be wondering what the use case for this would be, but it would be mainly used for what I'm going to show below. Using this below could make the code much more readable and simple to understand, but in the end, User.id would also have the value of "id", so the two would be identical in function.
const userData = User.findOne({
    where: Condition.Equals("id", 1)
})

Turns into
const userData = User.findOne({
    where: Condition.Equals(User.id, 1)
})

Hey, William. I can do this change as you mentioned above.

Sounds good @Deepu178!


Typo?

In 'Overridden access from parent' chapter you use extends instead of extend. This might be typo.

Yup.
Thanks :)


Issue 11435 - -O optimization flag causes invalid 32 bit codegen

Do not emit a push from memory instruction for 8 or 16-byte words, ever.
@WalterBright Is there some way to detect it's pushing from a stack slot? (and therefore is always safe)
https://issues.dlang.org/show_bug.cgi?id=11435

@yebblies ping

Eh I can't be bothered with this right now.


[BUG] Image upload does not start when HTTP Proxy is configured

Describe the bug
When a HTTP Proxy is configured in harvester, the VM image upload does not work. The upload does not start and errors with a timeout after 60 seconds. There are no obvious related errors in the logs.
When the HTTP Proxy settings are removed from Harvester, the image upload works without a problem. The no_proxy setting contained the cluster internal IP range <IP_ADDRESS>/8.
To Reproduce

Configure a HTTP Proxy in Harvester
Upload a VM image from your local machine through the UI

Expected behavior
The VM image is successfully uploaded
Environment

Harvester ISO version: 1.0.2

@futuretea Shouldn't these be added to noProxy if necessary automatically by harvester?
According to the docs: https://docs.harvesterhci.io/v1.0/settings/settings/#http-proxy

Harvester appends necessary addresses to user configured no-proxy to ensure the internal traffic works. i.e., localhost,<IP_ADDRESS>,<IP_ADDRESS>,<IP_ADDRESS>/8,cattle-system.svc,.svc,.cluster.local

Verified fixed on v1.0-483ee6b3-head (7/21). Close this issue.
Result
After the HTTP proxy is configured, we can upload image from local file correctly without error

Test Information

Test Environment: 1 node harvester on local kvm machine
Harvester version: v1.0-483ee6b3-head (7/21)

Verify Steps

Clone ipxe-example vagrant project https://github.com/harvester/ipxe-examples
Edit settings.yml
Set harvester_network_config.offline=true
Create a one node air gapped Harvester with a HTTP proxy server
Access Harvester settings page
Add the following http proxy configuration

{
  "httpProxy": "http://<IP_ADDRESS>:3128",
  "httpsProxy": "http://<IP_ADDRESS>:3128",
  "noProxy": "localhost,<IP_ADDRESS>,<IP_ADDRESS>,<IP_ADDRESS>/8,cattle-system.svc,<IP_ADDRESS>/16,.svc,.cluster.local,example.com"
}

7. Wait for 5 minutes for Harvester to apply the network setting
8. Go to images page
9. Upload image from local file

Verified fixed on master-1c060275-head (7/21). Close this issue.
Result
After the HTTP proxy is configured, we can upload image from local file correctly without error

Test Information

Test Environment: 1 node harvester on local kvm machine
Harvester version: master-1c060275-head (7/21)

Verify Steps

Clone ipxe-example vagrant project https://github.com/harvester/ipxe-examples
Edit settings.yml
Set harvester_network_config.offline=true
Create a one node air gapped Harvester with a HTTP proxy server
Access Harvester settings page
Add the following http proxy configuration

{
  "httpProxy": "http://<IP_ADDRESS>:3128",
  "httpsProxy": "http://<IP_ADDRESS>:3128",
  "noProxy": "localhost,<IP_ADDRESS>,<IP_ADDRESS>,<IP_ADDRESS>/8,cattle-system.svc,<IP_ADDRESS>/16,.svc,.cluster.local,example.com"
}

Wait for 5 minutes for Harvester to apply the network setting
Go to images page
Upload image from local file


Ensure CSRF_TRUSTED_ORIGINS and ALLOWED_HOSTS are specified

This PR specifies a CSRF_TRUSTED_ORIGINS and ALLOWED_HOSTS in settings.py, and thus fixes #1288.

I think all of these should depend on the deployment environment.

ah, and I see on line 42 that we use os.getenv to set ALLOWED_HOSTS. If we merged this change, it would overwrite that. Let me see what I can do to properly address this.


Governance. Lock. "Unlockable" has no loading state

Description
When user switch between networks, "Unlockable" state will be shown until new data came from new network
Steps to Reproduce

Has 2 networks with unlockable and not unlockable state
Switch between that networks

Expected Behavior
When switch networks - loading state apply for clock plate
Actual Behavior
Loading state apply only for lock amount, Unlockable text will be shown until data from the chain will be received
https://github.com/user-attachments/assets/ab46ac3e-307e-4142-a67e-0094078109f9
Environment
https://github.com/novasamatech/nova-spektr/commit/769e2c447f837eea9cdc8d946a1bd416cacb1fc2

Status verification: ✅


Could not load file or assembly 'DocumentFormat.OpenXml, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=8fb06cb64d019a17'

I have 2 different .net (c#) projects on my VS2019: console app and class library project
I use DocumentFormat.OpenXml (<IP_ADDRESS>) and OpenXmlPowerTools (<IP_ADDRESS>) on both projects.
Both project were compilated successful. Conscole app works correct. But Class library project warn me the next error message:
Exception thrown: 'System.IO.FileNotFoundException' in WordAssistant.dll
An exception of type 'System.IO.FileNotFoundException' occurred in WordAssistant.dll but was not handled in user code
Could not load file or assembly 'DocumentFormat.OpenXml, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=8fb06cb64d019a17' or one of its dependencies. The system cannot find the file specified.
CopyLocal =true and I see all libraries in output folder
app.config:
<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <runtime>
    <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
      <dependentAssembly>
        <assemblyIdentity name="Newtonsoft.Json" publicKeyToken="30ad4fe6b2a6aeed" culture="neutral" />
        <bindingRedirect oldVersion="<IP_ADDRESS>-<IP_ADDRESS>" newVersion="<IP_ADDRESS>" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="DocumentFormat.OpenXml" publicKeyToken="8fb06cb64d019a17" culture="neutral" />
        <bindingRedirect oldVersion="<IP_ADDRESS>-<IP_ADDRESS>" newVersion="<IP_ADDRESS>" />
      </dependentAssembly>
    </assemblyBinding>
  </runtime>
</configuration>

packages.config
<package id="DocumentFormat.OpenXml" version="2.11.3" targetFramework="net472" />
  <package id="Microsoft.Office.Interop.Word" version="15.0.4797.1003" targetFramework="net472" />
  <package id="Newtonsoft.Json" version="12.0.3" targetFramework="net472" />
  <package id="Open-Xml-PowerTools" version="4.4.0" targetFramework="net472" />
  <package id="System.IO.FileSystem.Primitives" version="4.3.0" targetFramework="net472" />
  <package id="System.IO.Packaging" version="5.0.0-rc.2.20475.5" targetFramework="net472" />

The same behaviour was explained here at first Solution
I already updated both library from nuget. pls help

Can you share a repro project? This appears to be a problem with binding redirects but I don't see any issue with what you have here

Also, how are you running this? Is this a console/web/winforms/wpf/pluging app? Each of those handle this differently. If you provide a repro project, happy to give pointers.


HHH-15454 Retrieve primitives from tuples

Same as #5218 for main branch (the same issue existed here as well)

Great work, thanks!


Vcpkg support

What does "vcpkg support" mean? I prefer that the people who are interested in having the package for a particular package manager, who know the package system well, are the ones who maintain the package. I don't know much about vcpkg, and I'm not enthusiastic about learning. If Boost.JSON is good enough and users find it helpful then they can package it.
I also don't want to be promoting Boost.JSON in a package before it has gone through review, because we are calling it "Boost" JSON but it has not been accepted yet, and that's bad form.

Boost is already packaged, and if/when Boost.JSON is accepted, will get packaged right along with all the other Boost libraries. So there is not really anything actionable here.

@vinniefalco Copy that


Docs: fix commandline doc create.md and run.md

Signed-off-by: Qiang Huang<EMAIL_ADDRESS>

LGTM

LGTM

ping @moxiegirl

changes so far look good; have you checked if the man-pages also need the same change? (sorry, typing on my phone now, otherwise I would have checked)

@thaJeztah Yes man-pages need to be changed too, updated.

LGTM Thanks @hqhq nice job. Back to @thaJeztah for the final check and merge

LGTM! Thanks @hqhq (and all in alphabetical order; make the OCD side of me happy :smile:)!


Intellisense in: when calling method just above method declaration

Type: Bug
// showing Intellisense for MyMethod
Console.WriteLine();
// NOT showing Intellisense for MyMethod
void MyMethod() { }
// showing Intellisense for MyMethod
VS Code version: Code 1.86.0 (05047486b6df5eb8d44b2ecd70ea3bdf775fd937, 2024-01-31T10:28:19.990Z)
OS version: Windows_NT x64 10.0.19045
Modes:

System Info

Item
Value

CPUs
Intel(R) Pentium(R) CPU G840 @ 2.80GHz (2 x 2794)

GPU Status
2d_canvas: enabledcanvas_oop_rasterization: disabled_offdirect_rendering_display_compositor: disabled_off_okgpu_compositing: enabledmultiple_raster_threads: disabled_offopengl: enabled_onrasterization: unavailable_offraw_draw: disabled_off_okskia_graphite: disabled_offvideo_decode: enabledvideo_encode: enabledvulkan: disabled_offwebgl: enabledwebgl2: unavailable_offwebgpu: enabled

Load (avg)
undefined

Memory (System)
7.85GB (1.37GB free)

Process Argv
--crash-reporter-id 16d6b610-0e87-48b1-a7d7-f3d8239b07cb

Screen Reader
no

VM
0%

Extensions (9)

Extension
Author (truncated)
Version

rest-client
hum
0.25.1

ilspy-vscode
ics
0.17.0

docomment
k--
1.0.0

csdevkit
ms-
1.3.10

csharp
ms-
2.15.30

vscode-dotnet-runtime
ms-
2.0.1

vscodeintellicode-csharp
ms-
0.1.26

material-icon-theme
PKi
4.33.0

msbuild-project-tools
tin
0.6.0

A/B Experiments
vsliv368:30146709
vspor879:30202332
vspor708:30202333
vspor363:30204092
vswsl492cf:30256860
vscorecescf:30445987
vscod805cf:30301675
binariesv615:30325510
vsaa593cf:30376535
py29gd2263:30899288
c4g48928:30535728
azure-dev_surveyone:30548225
vscrpc:30673769
962ge761:30951796
pythongtdpath:30769146
welcomedialogc:30910334
pythonidxpt:30866567
pythonnoceb:30805159
asynctok:30898717
pythontestfixt:30902429
pythonregdiag2:30936856
pyreplss1:30897532
pythonmypyd1:30879173
pythoncet0:30885854
pythontbext0:30879054
dsvsc016:30899300
dsvsc017:30899301
dsvsc018:30899302
dsvsc019b:30953937
edj9j583:30958191
3ef8e399:30949928

/extC#

It looks like this is caused by the C# extension. Please file the issue to the C# extension repository. Make sure to check their issue reporting template and provide them relevant information such as the extension version you're using. See also our issue reporting guidelines for more information.
Happy Coding!

It looks like this is caused by the C# extension. Please file the issue to the C# extension repository. Make sure to check their issue reporting template and provide them relevant information such as the extension version you're using. See also our issue reporting guidelines for more information.
Happy Coding!

Thank you! Let's try this way.


Load .plotly or .json file for display

Hi,
At the moment I believe there is only the possibility of displaying Plotly with embedded code inside the dataviewjs. Plotly allows to save a serialized version of their plot (JSON format, with .plotly) extension and was wondering if we could make it possible to load such file for display. This would simply a lot the notes I believe
something like  should work I believe
function readTextFile(file, callback) {
    var rawFile = new XMLHttpRequest();
    rawFile.overrideMimeType("application/json");
    rawFile.open("GET", file, true);
    rawFile.onreadystatechange = function() {
        if (rawFile.readyState === 4 && rawFile.status == "200") {
            callback(rawFile.responseText);
        }
    }
    rawFile.send(null);
}

readTextFile("PlotlyFileSomehwere.plotly", function(text){
    var data = JSON.parse(text);
    dv.paragraph(data);
    window.renderPlotly(this.container, data)
});

Hello!
That is correct - as of now, charts can only be created via plotly and dataviewjs code blocks.
With dataviewjs block, it is already posible to load and draw plots from any file (but I haven't tested your code yet).
I support your idea to simplify this process and I see two approaches to that:

by providing additional block processor (plotly-file or something)
by providing user a single function for loading and rendering file

Other ideas and suggestions are appreciated.

Also, as a sidenote, I think users should have a way to define their own custom functions that they would reuse across their plotly charts. Maybe in settings menu.


cmd/link: wrong loader/interpreter used with cross-compiled toolchain for ppc64le

What version of Go are you using (go version)?

$ go version
go version devel go1.19-4484c30f78 Wed

Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (go env)?
go env Output
$ go env
ppc64le

What did you do?
Built a cross toolchain on x86 host targeting ppc64le. By default CGO_ENABLED=0 for cross builds.
Moved the toolchain to a ppc64le machine. Many tests work fine except those that use -buildmode=pie. In those cases the test fails.
What did you expect to see?
Tests using -buildmode=pie work.
What did you see instead?
$ go build -buildmode=pie hello.go
$ ./hello
bash: ./hello: No such file or directory
$ file hello
hello: ELF 64-bit LSB shared object, 64-bit PowerPC or cisco 7500, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, with debug_info, not stripped
Same result if I set GO_LDSO=/libd64/ld64.so.2 before the go build command above.
If I set GO_LDSO=/lib64/ld64.so.2 when building the cross toolchain then it uses the correct path to the loader.
Also, if I build a cross compiled toolchain with a cross gcc on the build machine and set CGO_ENABLED=1, then the go binary is built dynamic but is unusable for the same reason.. With CGO_ENABLED=0, the go binary is static so doesn't have that problem.

Built a cross toolchain on x86 host targeting ppc64le. By default CGO_ENABLED=0 for cross builds.
Moved the toolchain to a ppc64le machine.

Could you show the commands you run for building and copying the toolchain? Is it just GOARCH=ppc64le ./make.bash then copying using scp or something? Or it is something else?

Thanks. During make.bash, cmd/dist writes defaultGO_LDSO in internal/buildcfg/zbootstrap.go ( https://cs.opensource.google/go/go/+/master:src/cmd/dist/buildruntime.go;l=69 ). And the linker picks up that value as the default dynamic linker path.
The linker already does not use defaultGO_LDSO when cross compiling at link time ( https://cs.opensource.google/go/go/+/master:src/cmd/link/internal/ld/elf.go;l=1771 ). But  in this case the toolchain is cross-compiled but  at link  time the host and target are the same so it is not cross compiling (and there is no way for the linker to know it). So it still uses defaultGO_LDSO.
cmd/dist also writes in internal/buildcfg/zbootstrap.go defaultGOOS and defaultGOARCH. But they are just runtime.GOOS and runtime.GOARCH, i.e. the current host, so it doesn't help. Maybe cmd/dist could write a per-platform defaultGO_LDSO, so the linker can choose to use it only when targeting that platform.
cc @ianlancetaylor

This doesn't need to target Go 1.19. I think it needs more work to ship a cross-compiled binary release, and we're probably not  going to do it for Go 1.19.

On a related note, I noticed this line in cmd/link/internal/ppc64/obj.go:
                Linuxdynld: "/lib64/ld64.so.1",

This is wrong for ppc64le, which is what the default should be since linux/ppc64 doesn't generate pie or dynamic binaries.  It didn't seem worthy of its own issue since it has always been this way so this field must never be used, but it should still contain the correct string. It could be fixed along with the fix for this issue in Go 1.20. Probably when using the external linker the interpreter is set in a different way, and that's the way pies were generated up until @pmur s recent changes to the internal linker.

@laboger thanks. Yes, this should be fixed. Feel free to send a CL. What should it be?
This string is still used in some cases, for example, when cross compiling (not with a cross-compiled toolchain, but like GOARCH=ppc64le go build -buildmode=pie hello.go on a different architecture), or when GO_LDSO is not set.

The C cross toolchain on my machine uses /lib64/ld64.so.2. Maybe this is the correct vale?

The C cross toolchain on my machine uses /lib64/ld64.so.2. Maybe this is the correct value?

Yes, it should be 2 since ppc64le is ELF v2. I can create a CL.

Thanks for the CL for the default. I just ran into the GO_LDSO cross-compilation issue too, and I'm going to send a general fix for Go 1.20.

This issue is now a duplicate of either #54196 and #54197, depending on which way we fix the GO_LDSO problem (probably both ways).

With #54197, if you don't explicitly set GO_LDSO when running make.bash, the resulting linker will use Linuxdynld , which, with the CL above, should points to the correct path. I think we can close this. Thanks.


Issues with prop reactivity & default values when using reactive props destructure

Environment

Operating System: Darwin
Node Version:     v20.9.0
Nuxt Version:     3.13.1
CLI Version:      3.13.1
Nitro Version:    2.9.7
Package Manager<EMAIL_ADDRESS>Builder:          -
User Config:      compatibilityDate, devtools
Runtime Modules:  -
Build Modules:    -

Reproduction
https://stackblitz.com/edit/github-yd89uq?file=app.vue,components%2FFoo.vue,components%2FBar.vue
Describe the bug
I've noticed a few issues with component props after updating from Nuxt 3.12.3 to the latest version. I suspect it might be caused by enabling reactive props destructuring by default in Vue 3.5.

setting default values for boolean props doesn't work (though it does for string props, for example)

const {
    foo = true, // will be `false` if not provided
    bar = 'something', // will be `'something'` if not provided
} = defineProps<{
    foo?: boolean
    bar?: string
}>()

seemingly, all prop reactivity is lost when using reactive destructure (when changing the prop values, they do not update in the component)

// It works correctly if defined using the "old" way, though
const props = defineProps<{
    foo?: boolean
    bar?: string
}>()

The behaviour was the same whether or not I had vite.vue.script.propsDestructure set to true in nuxt.config (though it should not be relevant now that it should be enabled by default?)
Additional context
I tested it in vue SFC playground and it worked correctly there, so that's why I assume this might be a Nuxt issue
https://play.vuejs.org/#eNqNUlFLwzAQ/ishLyqMFpm+zG6iMkEfdKiPAanttWamuZCkdTD6372kdA5x4tvd9313/fJdt/zKmKRrgc945gorjWcOfGsWQsvGoPXsBhvDKosNO0rS0AT50YXQQmfpMEJiajw0RuUeqGMsK2UXCyrjhnSA0wHP0j01n3DvCtSVrJO1Q01etkEseEGTUoF9NF6idoLPWGQClyuFn/cR87aFyYgX71B8/IKv3SZggq8sOLAdCL7jfG5r8AO9fH6ADdU7ssGyVaT+g3wCh6oNHgfZdatLsr2ni27vYqJS1y9uufGg3fioYDQo+6gXnBIOoR16+rfdaXIW54TuKcXxOj+PyVSu6znN0Zp4Kwrb+WFfhcjmY1I9lSVUUsPKonHZqLicsTdEBTnN9ovjk/8f/xaRjG/jZ3qyefgXeO3AhkDI/DQ5T055/wUEUtzf
Logs
No response

this is enabled by default in next nuxt release - for now, see https://github.com/nuxt/nuxt/issues/28829#issuecomment-2329411615.


add "unpack_bits" function

for discussion: is the name and syntax acceptable?
@havardpe
@lesters
@bratseth
@jobergum

Do we need the "_from_in8" suffix? Otherwise LGTM!

LGTM, great!

Great! Thanks for adding the test showing compatibility with the packing done by the Colbert embedder.
The numpy "equivalent" names are unpackbits and [packbits](https://numpy.org/doc/stable/reference/generated/numpy.packbits.html].
I think the name is fair since it doesn't make sense for other tensor cell types?

some changes and rebased now:
function name is just "unpack_bits"
type resolving checks that input has int8 cells (easy to work-around or lift restriction later)
default output cell type is "float", default endian is "big"

unpack_bits(attribute(foo))
unpack_bits(attribute(foo), float)
unpack_bits(attribute(foo), float, big)

Perfect @arnej27959 !


Aliu/enqueue eth

~will remove prints and try CI. I'll ping people to review when it's ready~

clean CI: https://github.com/tenstorrent-metal/tt-metal/actions/runs/7642158485


[ZT] Editing a device profile requires a connection restart

Publish after CLIENT-8858 is resolved.


Allow getting PluginGroup and installing plugins with PluginGroup with vMAJOR or vMAJOR.MINOR or vMAJOR.MINOR.PATCH versions

What this PR does / why we need it

Allow getting PluginGroup and installing plugins with PluginGroup with vMAJOR or vMAJOR.MINOR or vMAJOR.MINOR.PATCH versions

Which issue(s) this PR fixes

Fixes #493
Describe testing done for PR

Added Unit tests and E2E tests

$ export TANZU_CLI_PLUGIN_DISCOVERY_IMAGE_SIGNATURE_VERIFICATION_SKIP_LIST=localhost:9876/tanzu-cli/plugins/central:small

$ tz plugin source update default --uri localhost:9876/tanzu-cli/plugins/central:small
[i] Reading plugin inventory for "localhost:9876/tanzu-cli/plugins/central:small", this will take a few seconds.
[!] Skipping the plugins discovery image signature verification for "localhost:9876/tanzu-cli/plugins/central:small"

[ok] updated discovery source default

$ tz plugin group search --show-details
[i] The tanzu cli essential plugins are outdated and are being updated now. The update may take a few seconds.

name: vmware-tmc/tmc-user
description: Desc for vmware-tmc/tmc-user:v9.9.9
latest: v9.9.9
versions:
    - v0.0.1
    - v9.9.9

name: vmware-tanzucli/essentials
description: Desc for vmware-tanzucli/essentials:v9.9.9
latest: v9.9.9
versions:
    - v0.0.1
    - v9.9.9

name: vmware-tkg/default
description: Desc for vmware-tkg/default:v9.9.9
latest: v9.9.9
versions:
    - v0.0.1
    - v9.9.9

name: vmware-tkg/shortversion
description: Desc for vmware-tkg/shortversion:v1.2.0
latest: v1.2.0
versions:
    - v1.1.0-beta.1
    - v1.1.0
    - v1.1.1
    - v1.2.0

$ tz plugin group get vmware-tkg/shortversion:v1.1
Plugins in Group:  vmware-tkg/shortversion:v1.1.1
  NAME     TARGET      VERSION
  cluster  kubernetes  v1
  package  kubernetes  v1
  secret   kubernetes  v1

$ tz plugin group get vmware-tkg/shortversion:v1
Plugins in Group:  vmware-tkg/shortversion:v1.2.0
  NAME     TARGET      VERSION
  cluster  kubernetes  v2.3
  package  kubernetes  v2.3
  secret   kubernetes  v2.3

$ tz plugin group get vmware-tkg/shortversion:v2                                                                                (base)
[x] : plugin-group "vmware-tkg/shortversion:v2" cannot be found

$ tz plugin group get vmware-tkg/shortversion:v1.1.0-beta.1
Plugins in Group:  vmware-tkg/shortversion:v1.1.0-beta.1
  NAME     TARGET      VERSION
  cluster  kubernetes  v1.9.2-beta.1
  package  kubernetes  v1.9.2-beta.1
  secret   kubernetes  v1.9.2-beta.1

$ tz plugin install --group vmware-tkg/shortversion:v1.1
[i] Installing plugins from plugin group 'vmware-tkg/shortversion:v1.1.1'
[i] Installing plugin 'cluster:v1.11.3' with target 'kubernetes'
[i] Plugin binary for 'cluster:v1.11.3' found in cache
[i] Installing plugin 'package:v1.11.3' with target 'kubernetes'
[i] Plugin binary for 'package:v1.11.3' found in cache
[i] Installing plugin 'secret:v1.11.3' with target 'kubernetes'
[i] Plugin binary for 'secret:v1.11.3' found in cache
[ok] successfully installed all plugins from group 'vmware-tkg/shortversion:v1.1.1'

$ tz plugin install --group vmware-tkg/shortversion:v1
[i] Installing plugins from plugin group 'vmware-tkg/shortversion:v1.2.0'
[i] Installing plugin 'cluster:v2.3.5' with target 'kubernetes'
[i] Plugin binary for 'cluster:v2.3.5' found in cache
[i] Installing plugin 'package:v2.3.5' with target 'kubernetes'
[i] Plugin binary for 'package:v2.3.5' found in cache
[i] Installing plugin 'secret:v2.3.5' with target 'kubernetes'
[i] Plugin binary for 'secret:v2.3.5' found in cache
[ok] successfully installed all plugins from group 'vmware-tkg/shortversion:v1.2.0'

$ tz plugin install --group vmware-tkg/shortversion:v2
[x] : unable to find plugin group with name 'vmware-tkg/shortversion' matching version 'v2'

$ tz plugin install --group vmware-tkg/shortversion:v1.1.0-beta.1
[i] Installing plugins from plugin group 'vmware-tkg/shortversion:v1.1.0-beta.1'
[i] Installing plugin 'cluster:v1.9.2-beta.1' with target 'kubernetes'
[i] Installing plugin 'package:v1.9.2-beta.1' with target 'kubernetes'
[i] Installing plugin 'secret:v1.9.2-beta.1' with target 'kubernetes'
[ok] successfully installed all plugins from group 'vmware-tkg/shortversion:v1.1.0-beta.1'

Release note

Allow getting/installing plugins by specifying plugin-group version as `vMAJOR` or `vMAJOR.MINOR` or `vMAJOR.MINOR.PATCH` with `tanzu plugin install --group` and `tanzu plugin group get` commands

Additional information
Special notes for your reviewer

The behaviour when using shortened versions with air-gap download should be discussed (the change is probably from Allow installing plugins with vMAJOR or vMAJOR.MINOR or vMAJOR.MINOR.PATCH versions #486).  From what I can see the group  vmware-tkg/shortversion:v1.1.1 contains plugins with a shortened version of v1. Using that group, when I run a tz plugin download-bundle --group vmware-tkg/shortversion:v1.1.1 I see that it download all plugins that match version v1.  I was wondering if it should instead only download the latest patch that matches v1, as we do for plugin install.

Thanks for pointing this out. I would agree with the suggestion of only installing latest minor/patch version of the plugin in this case. I have made minor changes in the latest commit to handle this situation. Please take a look.

While trying this out I notice that when we use a shortened version (for plugins or groups), a pre-release is considered newer than the previous patch.  I wonder if that is what we want?  For example, in the local test repos we have the secret plugin with versions v1.9.1 and v1.9.2-beta.1 and when install the shorted version v1.9 it installs v1.9.2-beta.1.

From an end-user perspective, I don't think this is correct. If you agree, we can open an issue for that and not delay this PR.

I will file a separate issue to handle this as this might require more discussion.


human readable description of AssertionError

It would be helpful if the AssertionError contains a human readable error message so that it can be presented to the client.
Traceback (most recent call last):
    response = self.sp.parse_authn_request_response(message, binding, self.outstanding_queries, self.outstanding_certs)
  File "/usr/lib/python2.7/dist-packages/saml2/client_base.py", line 573, in parse_authn_request_response
    binding, **kwargs)
  File "/usr/lib/python2.7/dist-packages/saml2/entity.py", line 949, in _parse_response
    response = response.verify(key_file)
  File "/usr/lib/python2.7/dist-packages/saml2/response.py", line 853, in verify
    res = self._verify()
  File "/usr/lib/python2.7/dist-packages/saml2/response.py", line 391, in _verify
    assert self.issue_instant_ok()
AssertionError

Best would be exceptions for each error type, e.g. ResponseLifetimeExceed.

Agree, it's on my list of things to do. Unfortunately it's been there for a while.

Even worse is raising of Exception itself. This cannot be handled without checking str(exc).
    response = self.sp.parse_authn_request_response(message, binding, self.outstanding_queries, self.outstanding_certs)
  File "/usr/lib/python2.7/dist-packages/saml2/client_base.py", line 573, in parse_authn_request_response
    binding, **kwargs)
  File "/usr/lib/python2.7/dist-packages/saml2/entity.py", line 949, in _parse_response
    response = response.verify(key_file)
  File "/usr/lib/python2.7/dist-packages/saml2/response.py", line 864, in verify
    if self.parse_assertion(key_file):
  File "/usr/lib/python2.7/dist-packages/saml2/response.py", line 838, in parse_assertion
    if not self._assertion(assertion, assertion in res):
  File "/usr/lib/python2.7/dist-packages/saml2/response.py", line 772, in _assertion
    if not self.condition_ok():
  File "/usr/lib/python2.7/dist-packages/saml2/response.py", line 576, in condition_ok
    validate_before(conditions.not_before, self.timeslack)
  File "/usr/lib/python2.7/dist-packages/saml2/validate.py", line 97, in validate_before
    raise Exception("Can't use it yet %d <= %d" % (nbefore, now))
Exception: Can't use it yet<PHONE_NUMBER> <=<PHONE_NUMBER>

I bumped into this while testing an older version, and the broad exception raising seems to have been corrected in newer versions along with adding a "slack" factor. Here is the fix commit from @knaperek :
https://github.com/IdentityPython/pysaml2/commit/9a3b3849df2b8a337f9d130de68024fec13363ee#diff-b22d07287a1f7d4a4ae80402fa1bd832

I bumped into this confusing error while testing an older version, and the broad exception raising seems to have been corrected in newer versions. Here is the fix commit from @rohe
https://github.com/IdentityPython/pysaml2/commit/5518bd69f572600eea035c5b9958354ea11b58e0#diff-b22d07287a1f7d4a4ae80402fa1bd832

I am closing this, as it has been fixed. Thanks for looking into this.


Add custom domain option to support GitHub Enterprise

This PR is WIP for adding "custom domain" functionality
Please read the comments for details

Will close this until there is a chrome fix available


threads management and single node of  toy configuration of Torque in the local machine?

I have 18 human samples of which are paired-end RNA-seq data. I used the default setting --profile base to perform NGI-RNAseq procedure. However, the program quickly consumed a lot of hardware source. And I checked it using 'top' command and found that the load was above 70%.  Although I specified using the process.cpus = 40 to restrict the max CPU usage from my understanding. But it failed and the number of CUPS reached ~60 and my local machine is 64 (multi-core).
I then used the Torque (4.2.7) and Maui (3.3) to simulate the 'PBS' environment.  I edited the base config file and specified executor to 'pbs' and cpus = 40. However, When I run the script, I used the qstat to check all submitted work, but most work is in sleep model (in queue, Q).  and I also used the top command to check the cpu and memory usage, they all were in the low load.
I think I missed something in configuration file either in Torque or nextflow config?
Here is the output from Torque (qmgr -c 'p s'):
torque.txt
Here is the nextflow config adapted from the base and modified according to my understanding.
wanglab.txt
Any suggestions? Thanks in advance.

Hi @zhenyisong,
I'll reply to specific parts of your message:

Although I specified using the process.cpus = 40 to restrict the max CPU usage from my understanding. But it failed and the number of CUPS reached ~60 and my local machine is 64 (multi-core).

process.cpus sets the number of CPUs to use per process, not in total. So Nextflow will have been trying to run all 18 samples in parallel with 40 CPUs each. If you're using the local executor then executor.cpus will set the total max cpus to use (see the nextflow docs) as you expected.

I then used the Torque (4.2.7) and Maui (3.3) to simulate the 'PBS' environment. I edited the base config file and specified executor to pbs and cpus = 40. However, When I run the script, I used the qstat to check all submitted work, but most work is in sleep model (in queue, Q).

I guess that this is the same problem as above - you've told every process to ask Torque for 40 cpus, but you only have 64 available. As such, there is only space for one job to run at a time.

I also used the top command to check the cpu and memory usage, they all were in the low load.

Tools such as FastQC are not multithreaded and will only use one cpu, even if you allocate 40. If it was just FastQC running, then the cpu and memory usage will be low.
A couple of general points when looking at your config file:
Note that process.cpus sets the default value for all processes. This is overwritten by any process-specific configuration parameters. In your config you have the following:
process.cpus     = 40
process {
    $star {
        cpus = 4
    }
}

Here the default is 40 cpus, but the STAR process is then only using 4. Also, you have a few typos - the FastQC and TrimGalore processes have cpu = 8 instead of cpus = 8.
I hope this helps! Let us know how you get on..
Phil

Quick question: if we do not use the 'Torque' to mimic the cluster in a single node machine, and use the 'local' model to forward the task, and if task 1 uses the executor.cpus = 40, and task 2 uses the excutor.cpus = 40 then, will this result in overlaod? I mean, our local machine has 64 cpus, the combine task (1&2) will consume all (64 cpus)?
Put another way, is there any thread managment procedure implemened or controled in the local, base script? Or do we have to add extra specification to perform the single task to avoid overload? Thanks again.

Hi @zhenyisong,
executor.cpus should only be stated once and applies to the entire pipeline. It shouldn't be set for each process, unlike process.cpus. The two should be used in combination - process.cpus says what each step requires. excecutor.cpus says that maximum that should be used at any time. If the pipeline wants to run processes with process.cpus that sum to more than executor.cpus, jobs will be held back by Nextflow until earlier ones have completed and there are more cpus available again. Does this make sense?
Exactly how you run the pipeline is kind of up to you. However, if you have a single machine and not many users, my suggestion would be to use the base profile in local mode. I would create my own custom config file that looks as follows:
/* wanglab.config */
executor {
    name = 'local'
    cpus = 40
}

Then run the pipeline as follows:
nextflow run SciLifeLab/NGI-RNAseq -profile base -c /path/to/wanglab.config  # then other normal flags here...

The base profile config will determine the number of CPUs and amount of memory allocated for every step in the pipeline (default 2 cpus, used by tasks such as FastQC). The executor statement above will ensure that a maximum of 40 cpus will be requested at any time, meaning that there will always be at least 24 cpus left free for other users on the server.
Does this make sense and answer your question?
Phil

I did not clarify my meaning. Supposed that two different users in our local single-node machine and they submit two jobs and both configuration files specify the executor number, which is equal to 30. And supposed that local machine is 64-cores. If running these two jobs at same time, the cpus consumed on the tasks are 30 at max or 60 (30 +30 ) at maximum?

@ewels thanks.


Chrome Driver Problem

Im install this Program but when im start,i have problem

m

Should be fixed with this commit.


amazing micky mouse face animation

Describe the Feature you want to end

i want to add this amazing micky mouse face animation
Expected behavior

Screenshots (optional)

https://github.com/pranjay-poddar/Dev-Geeks/assets/120590200/43af6b79-d6b1-4769-8f1f-ff5a5b4ae3dd
Additional context (optional )

@MohitGupta121  @TusharKesarwani please assign this issue to me

@MohitGupta121 @TusharKesarwani please assign this to me

@MohitGupta121 @TusharKesarwani please review it asap i have raised the pr


"Execute + jump to next cell" brings me to two cells further instead of one

What steps will reproduce the problem?

This behaviour suddenly occured in an old spyder 3.X version.
After some time I reinstalled Spyder (now having version 5.1.5) but the problem remained?
here some system info:
Python 3.9.12 64-bit | Qt 5.9.7 | PyQt5 5.9.2 | Windows 10
Mandatory:
atomicwrites >=1.2.0          :  1.4.0 (OK)
chardet >=2.0.0               :  4.0.0 (OK)
cloudpickle >=0.5.0           :  2.0.0 (OK)
cookiecutter >=1.6.0          :  1.7.3 (OK)
diff_match_patch >=20181111   :  20200713 (OK)
intervaltree >=3.0.2          :  3.1.0 (OK)
IPython >=7.6.0               :  8.2.0 (OK)
jedi >=0.17.2;<0.19.0         :  0.18.1 (OK)
jsonschema >=3.2.0            :  4.4.0 (OK)
keyring >=17.0.0              :  23.4.0 (OK)
nbconvert >=4.0               :  6.4.4 (OK)
numpydoc >=0.6.0              :  1.2 (OK)
paramiko >=2.4.0              :  2.8.1 (OK)
parso >=0.7.0;<0.9.0          :  0.8.3 (OK)
pexpect >=4.4.0               :  4.8.0 (OK)
pickleshare >=0.4             :  0.7.5 (OK)
psutil >=5.3                  :  5.8.0 (OK)
pygments >=2.0                :  2.11.2 (OK)
pylint >=2.5.0;<2.10.0        :  2.9.6 (OK)
pyls_spyder >=0.4.0           :  0.4.0 (OK)
pylsp >=1.2.2;<1.3.0          :  1.2.4 (OK)
pylsp_black >=1.0.0           :  None (OK)
qdarkstyle =3.0.2             :  3.0.2 (OK)
qstylizer >=0.1.10            :  0.1.10 (OK)
qtawesome >=1.0.2             :  1.0.3 (OK)
qtconsole >=5.1.0             :  5.3.0 (OK)
qtpy >=1.5.0                  :  2.0.1 (OK)
rtree >=0.9.7                 :  0.9.7 (OK)
setuptools >=49.6.0           :  61.2.0 (OK)
sphinx >=0.6.6                :  4.4.0 (OK)
spyder_kernels >=2.1.1;<2.2.0 :  2.1.3 (OK)
textdistance >=4.2.0          :  4.2.1 (OK)
three_merge >=0.1.1           :  0.1.1 (OK)
watchdog >=0.10.3             :  2.1.6 (OK)
zmq >=17                      :  22.3.0 (OK)
Optional:
cython >=0.21                 :  0.29.28 (OK)
matplotlib >=2.0.0            :  3.5.1 (OK)
numpy >=1.7                   :  1.21.5 (OK)
pandas >=1.1.1                :  1.4.2 (OK)
scipy >=0.17.0                :  1.7.3 (OK)
sympy >=0.7.3                 :  1.10.1 (OK)

Hi @MichaelMAB2020 thank you for the feedback! Could it be possible for you to share an example code snippet to reproduce this behavior? Also, could you try to update to at least Spyder 5.2.2 and check again? Let us know!

Hi Daniel,
thanks for looking into my problem!
I just tried to update Spyder to the latest version - unfortunately that failed (also deinstallation of spyder) :-( . Tried it via the anaconda navigator as normal user and as admin ... - no success. I remember also the last time when I updated spyder (from 3.X to 5.X) at the end I had to totally deinstall anaconda and starting from scratch.
I attach an example file here. It includes the example python code and a spyder screenshot, showing the code and  also its results in the Konsole when I

run the full script at once wit the "play" button [F5]): In [29]
run it from the first cell (cell #0) with the "Execute + jump to next cell" (Shift+Return) button: In [30] to In [33]
run it from the second cell (cell #1) with the "Execute + jump to next cell" (Shift+Return) button: In [34] to In [36]
You see the jumping cell numbers executed and also the different result ...

If you think it's worth to upgrade I am willing to try, but as I wrote: the problem "survived" also the upgrade from 3.X to my current version 5.1.5 ...  :-(
spyderTest.zip

Thanks for the new info @MichaelMAB2020 ! I was unable to reproduce this locally with Spyder 5.2.2 or 5.3.3:
Checking with Spyder 5.2.2 using the toolbar buttons

If your Spyder installation is on your base conda environment I would suggest to update to Spyder 5.2.2 and check again by running from an Anaconda prompt with your base env activated something like:
conda update anaconda
conda install spyder=5.2.2

Let us know!

Hi Daniel,
the two command lines from your last comment worked so that I was able to install spyder 5.2.2 (in contrast to my update trials via the anaconda navigator :-( ) and ... it works again.
Thank you & best regards!!!

Awesome! Glad to hear Spyder is working for you now. Thank you for letting us know and then I will close this one :)