Wasm plugin system

EDIT: This is a proposal the discuss the implementation, nothing here is fixed yet
the current protocol is implemented here: https://github.com/astrale-sharp/wasm-minimal-protocol
you can send any number of &str from typst and you get back a single String
this is the source used for compiling hello.wasm (target : wasm32-unknwon-unknown)
use wasm_minimal_protocol::*;
initiate_protocol!();

#[wasm_func]
pub fn hello() -> String {
    String::from("Hello from wasm!!!")
}

#[wasm_func]
pub fn double_it(arg: &str) -> String {
    format!("{}.{}", arg, arg)
}

#[wasm_func]
pub fn concatenate(arg1: &str, arg2: &str) -> String {
    format!("{}*{}", arg1, arg2)
}

#[wasm_func]
pub fn shuffle(arg1: &str, arg2: &str, arg3: &str) -> String {
    format!("{}-{}-{}", arg3, arg1, arg2)
}

for me what's missing right now to make a real use of plugins is the possiblility of returning bytes from the plugin (a bytes type in typst?) and then interepret it as an image.
from a plugin you'd need to be able to create images, svgs etc, right now this is not possible

for me what's missing right now to make a real use of plugins is the possiblility of returning bytes from the plugin (a bytes type in typst?) and then interepret it as an image.

I recently added Bytes as a built-in Value. It's not yet user exposed but we could do that. read() would need to gain support for raw bytes And the image function, too of course.

Do you know whether wasm works when the host itself runs in WebAssembly? And if it does, whether it uses the browser runtime or whether we would run a WebAssembly interpreter in WebAssembly?

May help https://github.com/wasmerio/wasmer-js/tree/main

@laurmaedje
I haven't tested it, what flags do you use for compiling typst?
I'm pretty sure it works if you have the "js" feature enabled, but it's not easy to test for me since there's no wasm test or similar (maybe could be added so i can test it)

for me what's missing right now to make a real use of plugins is the possiblility of returning bytes from the plugin (a bytes type in typst?) and then interepret it as an image.

I recently added Bytes as a built-in Value. It's not yet user exposed but we could do that. read() would need to gain support for raw bytes And the image function, too of course.

As scoped function, would that make sense? image.fromRaw(bytes) && read.toRaw(path) Asking cause I'd also be interested to send a PR for that

Like, when you compile typst to wasm, what's your precise target?

With features  "js", "std", it compiles, but I haven't been able to run it for now

I'm a bit unsure how to add completions (knowing plugin_instance contains a vec of &str of all the function name that it can call, snippet completion doesn't seem adequate, but I don't think I should be passing a Value::Str(function_name) either (hard to test right now))

The target is wasm32-unknown-unknown. We use wasm-pack. For read, I'm not sure whether I'd prefer a scope function or an extra argument. For image, I'm also not sure as it would be possible to overload based on whether it's a string or bytes. But that could be strange with SVGs. If it's a scoped function, it should use kebab-case.

I'll have a look tomorrow regarding autocomplete.

In this case, it's probably best to just push the completion yourself instead of using one of the helper functions:
for k in plugin.iter_functions() {
    ctx.completions.push(Completion {
        kind: CompletionKind::Func,
        label: k.as_str().into(),
        apply: None,
        detail: None,
    });
}

Thanks for the review! I'll adress the points soon.
A point investigated right now is that wasmer (and wasmtime) are not interpreter, they recompile wasm to native and that's not gonna work if you compile it to wasm32
So for typst-cli wasmer or wasmtime could be acceptable but for typst on the web, wasmi should be prefered, I'll be working on creating an host with wasmi.

So for typst-cli wasmer or wasmtime could be acceptable but for typst on the web, wasmi should be prefered, I'll be working on creating an host with wasmi.

From the documentation it seems like it should just work 🤔

Do you have a way of running the typst test suite from wasm to be sure?cause it's not easy to see if wasmer
something as naive as cargo test --all --target wasm32-unknown-unkown wouldn't work

I just tried compiling your branch with the web app and got a ton of compiler errors, even with wasmer's js feature enabled and sys disabled. I also noticed that wasmer adds >200 new dependencies. Is there a lighter-weight alternative? I think having a small simple engine would be worth more than having the best possible performance.
If necessary, we can also write our own wrapper crate that uses one of the Rust engines locally and js-sys on the web.

Do you have a way of running the typst test suite from wasm to be sure?cause it's not easy to see if wasmer
something as naive as cargo test --all --target wasm32-unknown-unkown wouldn't work

Yea wasmer is really heavy :/
we have a working host in wasmtime as well (less heavy) and in wasmi (very lightweigth but doens't support reading from wasm32-wasi which might be okay (plus there's the possibilty to stub the wasi functions)

I've been wondering about the WASI thing. As plugins should be pure transformations without any permission to talk to the operating system, why would we care about that?

When porting a project, it might be much easier to compile it to wasi in some circumstances, however, stubbing the wasi function should be enough to be able to run it like it isn't wasi, if the plugin doesn't work at all, then it needed to access files to work which shouldn't happen anyway.
let's go with wasi!

It works! It's tested! It may not panic unless a mutex is poisoned (I could even try to prevent that if you'd like, by reloading the plugin with the stored bytes if a mutex is poisoned (possibly with a limit and an error on limit reached))
wasmi compiles to wasm32-unknown-unknown no problems but getrandom does not, I assume you use a modify version of the Cargo.toml that uses the right features.
Completions have not been tested yet as typst-lsp doesn't support 0.0.6 yet (https://github.com/nvarner/typst-lsp/issues/186)

Before I make this ready for review, do you want me to publish host-wasmi to crates.io ? maybe under typst-host-wasmi ?

Before I make this ready for review, do you want me to publish host-wasmi to crates.io ? maybe under typst-host-wasmi ?

I'm wondering whether we want to have everything here in one place, perhaps as a typst-plugin crate or something? Or do you think that's a bad idea. I'm not sure how the dev experience for creating the correct WASM file for a plugin with Rust/C/Zig is right now.

host-wasmi could totally be inlined in typst!
In typst there should also be a description of the protocol that wasm file must implement.
wasm-minimal-protocol is just a bunch of examples and help tools and tips to easily implement this protocol so I think it has an interest of being outside.
for initial development it's been easier for me to have everything in one repo and test it in typst, if it's a typst crate I should still be able to import it from anywhere to help with local dev of plugins so I don't see a problem with that.
where should that crate go ? crates/plugin-host?
I think only typst-library needs it so maybe crates/typst-library/plugin-host ?

Either crates/typst-plugin or just directly inline it into crates/typst/src/eval/plugin.rs. I think I'd prefer the latter for now. If it grows we can still extract it.
The problem with separate crates is that it always makes trait impls and dependencies more complicated and creates the need for extra types. The Plugin type currently defined in typst depends on a bunch of definitions in Typst, so it needs to be in typst or in a crate that depends on typst. As host_wasmi::PluginInstance is pretty simple, I think we skip the extra wrapper here and implement its features directly in Typst's Plugin type.

If you can move all of the plugin code into typst-library instead of typst, that would be fine, too. I'd say it's currently a bit misplaced in data.rs.

even moving it out of crates/typst/src/eval.rs?

Yeah, but I have no strong opinion whether it should live in typst or typst-library. It doesn't matter all that much.

I think I successfully put it in typst-library! it looks pretty good to me

are you also okay with the test in tests/typ/compute/data.typ ?

I think I'm going to move them in compiler/plugin.typ

What's next
A guide to make wasm plugin, for now this could simply refer to wasm-minimal-protocol or simply copy things from there.

I think I successfully put it in typst-library! it looks pretty good to me

I'm a bit confused as it is in typst and not in typst-library now. Which is okay, but different. The plugin_func itself should move to the library though, maybe to construct.rs.

For the record, I'm still hesitating between trying to have wasmtime (at least for the non wasm target and have the browser help for the wasm target) instead of wasmi cause it supports wasm32-wasi (and you have control over) but it is heavier 120 dependencies approx).

Why would we want to support wasi targets? It does not make sense to me as system access is explicitly disallowed.

I think the PR can be merged as is (style comments aside), but we should remain open minded about changing the wasi support in particular for the following reasons:

compiling to wasi is sometimes way easier
stubbing the functions before running might not always be enough (needs investigation)
wasmtime might give enough control that we can support wasi by " faking it " without really offering OS capabilities (difference of behaviour with stubbing to be proven)

I think we should just see what kind of problems people run into while trying to make plugins and adapt then, starting with the pr as is seems okay to me
Possible addition to the PR: automatically stub the wasi functions from the code might be a good workaround and someone is on the job. This might come with a perf price for initialisation tho so it could guarded behind a flag plugin(data.wasm, stub-wasi : true)
We could also add all this PR behind a default feature for people wanting to embed typst in minimal context.

ignore this for a while, I accidentaly pushed force on it..

by the way, there's no need to force push as it's all squashed anyway. in fact, not force pushing makes it easier to review the differences.

The current docs we have for the plugin can currently be found here. It could be copied as is but I'm not sure where I could put it, certainly not above the plugin function.
It would describe the protocol from a wat perspective, it should also mention that there is a rust lib with a wrapper to help write these plugins and a few example as well.
For me all the example and stuff should be kept out of typst/typst. If wasm-minimal-protocol becomes outdated, typst should point elsewhere for examples (maybe under a typst/plugins).
I will also add a MIT LICENSE to make sure w-m-p can be used then (just realized it had no license).
You can use string as bytes but not bytes as string, currently the string returned is verified utf8, if we return bytes, we need to either:

provide a check function in typst (which is a bit annoying in my opinion)
have a way in the protocol to specify the type you're returning (in which case from typst you wouldn't know if you're receiving one or the other which is not a big deal in my opinion)

I'm also wondering whether we want to go with strings as the minimal protocol primitive or whether we might just as well drop one level down and only allow bytes. I imagine that on the wasm side that's already supported because a string is just UTF-8 bytes. However, on the Typst side, it would require a bit of work on integrating a bytes type.

I wonder if it'd be more appropriate to define some form of ABI for some different kinds of Typst values (even if just the most basic ones, such as strings, integers, arrays and whatnot) in order to turn the plugin development process more seamless. I don't intend to suggest to overcomplicate the process at this point, however - we could have some MVP here with an ABI which just supports strings and bytes, and then add more as we go.

I would be really careful with defining an ad-hoc ABI. I do not have an experience in this domain and I imagine it's pretty easy to make mistakes that have been made often enough. Serialization/deserialization on both sides seems relatively sane to me as long as we provide the necessary primitives (i.e. json to/from string).

What about the documentation/spec for plugin, where would I put it/ where should it appear? (basically, what about my previous comment ^^)

I'm not sure I agree that the docs should be kept out of typst/typst since the Typst compiler itself mandates accordance to the protocol. If we consider it to developer-y for the main docs, we could have a file like docs/dev/plugins.md with details and link to that from the plugin function?

I would be really careful with defining an ad-hoc ABI.

Sounds fair. What if we use something like BSON (or some other common binary data exchange format)? The compiler could then automatically (de)serialize the basic scripting types.

I'm convinced! I'll move the docs where you said (soon)!

Still not mergeable,
@arnaudgolfouse should be credited for their work on documentation/implementation on this PR as well

@laurmaedje
Okay, I added documentation and a basic guide for plugin development, it links to wasm-minimal-protocol for examples and tools.
The test are currently not working since they are still using strings since the bytes type is not exposed yet. For this to be in merging state, we need at least a way to convert from/to string/bytes.
If you prefer the guide in another PR or not at all, feel free to tell me!

Adding the string <-> byte conversions should be simple enough. I think the design was more or less settled on Discord. If no one else does it first, I can implement it sometime next week.

Having a guide is nice. I wonder, if we have one, why not put the protocol stuff in there, too and get rid of the docs/dev file.

done

Loading Image from String, See #1810

I imagine use cases, where an array or dict would be returned from a plugin. Are you planning to allow structured return values, later? If not, would every structured return data need to be stringified / parsed by json?

The idea is that it will be stringified and then either parsed as json or with as Typst code with eval. Making the plugin interface very minimal makes things so much easier that it is worth it in my opinion.

@astrale-sharp @Beiri22 I will implement all missing things for a bytes type tomorrow to unblock this PR and the image decoding one.

The idea is that it will be stringified and then either parsed as json or with as Typst code with eval. Making the plugin interface very minimal makes things so much easier that it is worth it in my opinion.

I forgot about eval, a very handy tool. But did you think of automatically serializing / deserializing json? So like: Plugin method returns some serializable type; system on plugin side serializes; your plugin-call-function auto-deserializes? Or is that too heavy again?

Usage of a binary format, such as BSON, would probably be useful here as well, especially when transmitting binary data directly (or numbers). Not sure how easy it would be to implement, however.

Usage of a binary format, such as BSON, would probably be useful here as well, especially when transmitting binary data directly (or numbers). Not sure how easy it would be to implement, however.

https://docs.rs/bson/latest/bson/

Thanks @PgBiel for your grammar nitpicks and other comments, very useful :)
Thanks @Beiri22 for the implementation of encode/decode
It's unlikely that I'll have time today but I'll definitely try to complete this tomorrow !

The bytes type is now available.

I'd like to make it so you can pass string and they get converted to bytes automatically if you think that's possible?
I'd also like to experiment with a more performant implementation discussed on the discord to see how it could work

Just to make sure I understand: are we planning to simply have plugins take and return raw bytes then (not necessarily strings)? If so, I wonder if there could be a way to safely manipulate some primitive types, e.g. integers, that way (we could perhaps add a conversion from bytes to integers, and vice-versa).

Also, an important point: it'd be nice to have the decoding functions be more predictable. By this I mean that we should be able to "catch" errors while decoding, perhaps with some "try-decode" thing (or a fallible parameter or something), though that is made harder with the absence of an optional type in Typst. Maybe we can just return none for those, since I don't think you'd expect none for e.g. integers anyway.

Finally, if we aren't going to enforce something like BSON, we could at least (on a separate PR) implement a BSON encoder/decoder as well (akin to json() and friends) to make the lives of plugin creators easier.

Just to make sure I understand: are we planning to simply have plugins take and return raw bytes then (not necessarily strings)?

Yes.

If so, I wonder if there could be a way to safely manipulate some primitive types, e.g. integers, that way (we could perhaps add a conversion from bytes to integers, and vice-versa).

I would like to see real-world demand for this before we add it prematurely. You already have three options for this: (1) return your number as string bytes and parse them, (2) encode your output as json/whatever bytes and parse them, (3) slice four bytes, convert the result into an array and do the little/big-endian byte to int conversion yourself. I think raw integer decoding will be a very rare use case to be honest.

Also, an important point: it'd be nice to have the decoding functions be more predictable. By this I mean that we should be able to "catch" errors while decoding ...

While I agree that some mechanism for fallibility would be nice in general, I don't think it is particularly important for this PR. After all, you would expect your plugin to return well-behaved output.

Finally, if we aren't going to enforce something like BSON, we could at least (on a separate PR) implement a BSON encoder/decoder as well (akin to json() and friends) to make the lives of plugin creators easier.

The only benefit is see with BSON as opposed to just JSON is that it can represent byte arrays, which might be useful if you want to return a bunch of byte arrays and don't deal with decoding them yourself. None of the existing encoding formats can handle them. I'm not sure how often you would need that, but if there is sufficient demand, we can add BSON support.
Overall, I feel like the issue is with encoding/decoding isn't nearly as big as one could think from the discussion. If you want to return a custom image, then just return its raw bytes. If you want to return more structured data, there are multiple good options, including returning JSON bytes and returning Typst code to be evaluated. Only the case where you want to return multiple structured binary blobs (multiple images at once or something like that) isn't well covered yet, but even that would be relatively simple to handle in user code if the structure isn't too crazy.

I'd like to case to ToBytes when eating argument so that everything that can be converted to bytes is accepted. However, this is defined in typst-library and I can't access it from there since typst doesn't depend on it, any advice?

I'm trying integrate a protocol change but since the store is behind a mutex, I'm getting reference to temporary data problems, if anyone has an idea how to fix that, it would be nice!

I'd like to case to ToBytes when eating argument so that everything that can be converted to bytes is accepted. However, this is defined in typst-library and I can't access it from there since typst doesn't depend on it, any advice?

I'm not sure this is desirable anyway. The plugin API is very low level and should be wrapped in an idiomatic Typst function. Making the conversions explicit is good in my opinion. I considered adding an implicit string -> bytes conversion and decided against it. The return value can't be coerced anyway and I think it's not bad if both input and output work the same way.

Last bit, using the new protocol version depends on result being dropped at some time, I'm not sure typst does that so the tests are freezing, I don't think this problem has an easy solution so we would need to either

Go back to the previous protocol
Find a way to not depend on when results are dropped (which I'm not sure is possible)

Opinions? @arnaudgolfouse @camiyoru @laurmaedje

For the record, discussions have happened here

Reading the actual code, yes I think this is useless anyway, since values are immediately copied anyway:
let s = self.call_inner(function_name, byte_args).at(span)?;
Ok(Value::Bytes(s.get().into())) // <-- This makes a new allocation

Although this drops s (of type ReturnedData) on the very next line, so I'm not sure the issues you're experiencing are related

Didn't update the protocol in the end since I don't think it was possible to make work, if perf becomes a problem, i suggest we try modifying the protocol then!
I consider this finally ready!

Didn't update the protocol in the end since I don't think it was possible to make work, if perf becomes a problem, i suggest we try modifying the protocol then! I consider this finally ready!

What about versioning the protocol? The version is probably some global function or variable injected by wasm-minimal-protocol which returns a utf-8 version string so that protocol can upgrade like typst package's @preview.

Didn't update the protocol in the end since I don't think it was possible to make work, if perf becomes a problem, i suggest we try modifying the protocol then!

The version is probably some global function or variable injected by wasm-minimal-protocol, which returns a utf-8 version string.

Some possiblity for versioning might be nice. Either this way or by appending some suffix like _v1 do all functions. (Not sure whether a function that returns a version string gives full flexibility in changing the protocol?)
However, I think this mechanism should only be used as a last resort or way in the future. This kind of binary protocol is something you'd ideally just not have to break. I am not too concerned about copy overhead, but deferring an important protocol API design decision to later seems unwise to me. We should figure out now what we want to do.
Could someone give me an update which two API designs were proposed and what the problem with each is? I didn't follow the discussion closely.

= First iteration

Two functions are provided by the typst host (they should be imported by plugins):

wasm_minimal_protocol_write_args_to_buffer(*mut u8): copies the arguments for the function into the provided plugin-allocated buffer.
wasm_minimal_protocol_send_result_to_host(*const u8, usize): sends the result to the host, that copies it into a buffer of its own.

It has one major downside: every function calls incurs two copies of (possibly large) buffers back and forth between the plugin and the host.

= Second iteration

The imported functions are the same as above, but wasm_minimal_protocol_send_result_to_host does not copy the result into the host (and so the plugin should not free this buffer).
The plugin should also export a function wasm_minimal_protocol_free_byte_buffer(*mut u8, usize) that is used by the host to free the buffer returned by wasm_minimal_protocol_send_result_to_host
In theory, the advantage is that we don't need the 'copy buffer from wasm into host' part: we can directly use the host memory.
But in practice, the Bytes type owns its content anyway, so it cannot hold a reference to the wasm memory.

The arguments are no longer concatenated!

I did a little refactoring and also decided that the plugin function itself is the best place to document the protocol after all (as opposed to a guide).
I also tried to perform the plugin build step myself with the Zig example and came to the conclusion that it would be very helpful if the examples in wasm-minimal-protocol would somewhere (ideally at the top of the file) just show the build command. I had to dig into the test-runner to find it. Also, wasm_minimal_protocol_free_byte_buffer was still present in the Zig example which isn't needed anymore from my understanding.

Thanks so much for your work on this! I think we landed on a really nice result. :)

btw. this #1935 might be useful when using the wasm plugin system to transfer structured data...

Thanks for merging! It was really nice contributing, I'll adress your point about wasm-minimal-protocol soon!

Looking through the implementation, it seems there is no WASI support. In particular, it seems impossible to run an external command line tool or link to a native dynamic library. I suspect this is by design (for plugin portability, safety, etc.), but it also means that existing tools cannot be easily integrated into Typst. Currently, for a tool to be converted to a Typst plugin, all of its transitive dependencies must support being compiled to WebAssembly, which is not always the case. Can we have a native plugin counterpart, with the same interface, but as a native dynamic library (.so or .dll) instead of WebAssembly? I believe the end user should have the final decision on security issues, so it would not do any harm as long as native plugins are opt-in.

The strong encapsulation is indeed by design. There are no plans for native plugins at the moment, to ensure not only security but also portability (across platforms and in the web app). Native plugin support would fragment the ecosystem into things that only work when you trust a specific native plugin, things that only work on some OS, etc., which is something we seek to prevent. See also https://github.com/typst/typst/issues/1056 for extensive arguments on both sides.

Thanks for the explanation. It makes perfect sense to me. Unfortunately, in my specific use case the program for content processing is written in Haskell, which I have little confidence to compile to WASM without WASI. My understanding is that the query and preprocess API is the officially supported way for such use cases, but it has the downside that I would need a separate build system to manage caching and dependencies.

As long as your program doesn't actually try to read files etc. you can still compile to WASI and then stub it.


handle case where no cdn loaded based on domain name in ssl keys api

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/incubator-trafficcontrol-PR/268/

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/incubator-trafficcontrol-PR-trafficops-test/226/


Catch ConnectionError Exceptions on modules with bad connections.

I think it is fine to catch all ConnectionErrors instead of any specific subclasses. I am not sure that we could recover the connection in any case.

I think it is fine to catch all ConnectionErrors instead of any specific subclasses. I am not sure that we could recover the connection in any case.

Agreed, I figured we would want to disconnect regardless of the type of disconnect error. I'm just not sure yet if we need to add more try/except blocks to catch connection errors elsewhere, e.g. when sending a message. So far this is the only connection error I've actually seen occur.

It may be beneficial to have message manager try to send an ack or something periodically to prune any modules that are dead and not sending or subscribed.

Interesting idea but we'd have to think a bit about how to do that properly. Some modules can be idle without anything being wrong, e.g. they pause subscriptions before they stop reading. Our Executive module does this. But I agree it would be good to prune crashed matlab modules that don't disconnect or unsubscribe.
Maybe we can keep track of how many failed messages are sent to each module in a given amount of time, and if some threshold is reached the connection would be dropped.

It may be beneficial to have message manager try to send an ack or something periodically to prune any modules that are dead and not sending or subscribed.

Interesting idea but we'd have to think a bit about how to do that properly. Some modules can be idle without anything being wrong, e.g. they pause subscriptions before they stop reading. Our Executive module does this. But I agree it would be good to prune crashed matlab modules that don't disconnect or unsubscribe.
Maybe we can keep track of how many failed messages are sent to each module in a given amount of time, and if some threshold is reached the connection would be dropped. This would be good for a new issue/enhancement.

I don't think the module would even need to reply, the fact that message went through and we got no error would indicate the socket is valid. We only need message manager to initiate the write, I believe we would get an error if the module crashed and we tried to write. It probably wouldn't show up as ready to write though from select.

I guess we could get an error if we try to send a message to a module immediately after it disconnects, as a race condition:

Module A connects and subscribes to message type sent from module B
New iteration of message manager's run loop begins
Message manager gets rlist from select, module A may or may not be in it
Message manager gets wlist from select, module A is in it
Module A disconnects or crashes
Message manager receives a message from module B before attempting to read from module A. Attempts to forward message to module A.
Connection error may occur when writing to socket

The error that I've actually seen and handled is that after a module disconnects, it still appears in the rlist (maybe because it sent the Disconnect message?), but then a connection error occurs when it tries to read it. Since the sockets are shuffled and I've only been doing simple tests with a single client, I suppose the situation above could happen. I can try to test for it.
I would think the way to fix that would be to put a try/except around every socket write, not to send extra ACK messages for this purpose. Since we still use a lot of matlab, I think the most common "crash" is when a matlab module errors out (or is ctrl+c'd) and so matlab returns to the command line but does not close the process or socket. So the socket is still valid but no longer being read. The module would continue to appear in the wlist until the buffer is full, and message manager already detects that and starts sending FAILED_MESSAGE and printing 'x's. It would be an improvement to automatically close the socket after a certain number of failed messages are sent in a row, but I'm not sure if there's much value in adding a special ACK message to accomplish that.

I just verified this race condition can happen. Right now, since process_message is in the same try block, it will be caught in the same except block and incorrectly disconnect the sender instead of the receiver. I'll fix this.

Caught and ignored ConnectionErrors when writing to a socket seem to immediately result in a second ConnectionError on the next read, so I think it will be sufficient to put each write in a try block, catch and log and connection errors, but only force a disconnect on the read error.

Fixed in 3e29118. Message manager now seems to be robust to crashing on connection errors on either writes or reads. Errors on writes are logged and ignored, errors on reads are logged and the module is disconnected.
I might add sending a failed message when the error occurs on the write (currently that only happens if module is not in wlist). Otherwise I think this issue can be closed, and pruning inactive modules could be moved to a new issue.

Added the failed_message in aab1f0d5. I'm pretty happy with this. I added a statement to send_failed_message to return early if the type of the failed message is a FailedMessage, to prevent infinite recursion e.g if a logger module crashes.
I also saw that sometimes the error will happen on a write, and then the disconnect signal will be successfully read on the next read WITHOUT an error. So I feel like that's probably good enough justification for only forcing disconnections on failed reads, and not failed writes.
I can open another issue for pruning inactive modules (which generally should not appear in wlist), so I think it's separate from the ConnectionError issue. I'll close this issue.


Add port/fork for Cloudflare Pages

---EDIT---
moving to Discussions section
---EDIT---
Summary
Would be great to see this great project ported/forked over to support Cloudflare Pages (in addition to Netifly), with similar features such as _headers and _redirects and _functions and the new /functions/ folder and supported functions within. Support for preview environments, environment variables are all available, so it seems a feasible port that would be beneficial for us deeply invested in Cloudflare and wish to use this platform. I'm not that familiar with modifying the npm build process to ensure that Cloudflare's build process properly generate the correct files and ensure they fall into correct folders in the right format and so forth.
General documentation for Pages:
https://developers.cloudflare.com/pages/platform/
Existing migration guide from Netifly to Cloudflare Pages that might be useful:
https://developers.cloudflare.com/pages/migrations/migrating-from-netlify/

+1

Deployment guides (for a/o Cloudflare Pages) are in the making (including official Hyas plugins if applicable + example repos)

Exciting, thanks for the update!

Awesome, Waiting for the update.


Not working in Android Studio 3.6.1

Android Studio 3.6.1
Build #AI-192.7<IP_ADDRESS>41897, built on February 27, 2020
Runtime version: 1.8.0_212-release-1586-b04 amd64
VM: OpenJDK 64-Bit Server VM by JetBrains s.r.o
Windows 10 10.0
GC: ParNew, ConcurrentMarkSweep
Memory: 1237M
Cores: 8

Pressing ctrl + shift + g & toggles the CodeGlance plug-in


Query filter doesn't match if property's value is 0

I tried issuing a KYCCountryOfResidenceCredential with countryCode: 0 (yes, I know this is not a valid country code).
In the verification step, the auth request query against this was a check against sanctioned countries, e.g. {countryCode: {$nin: [1, 2, 3, 4]}} (again, these aren't necessarily valid country codes).
In the call to CredentialWallet.findCredentialsByQuery, this failed to match the issued credential. Tracing it through, it appears to be a bug in FilterQuery.execute where it returns false if the credential property value is false-y: https://github.com/0xPolygonID/js-sdk/blob/main/src/storage/filters/jsonQuery.ts#L113
The code is:
    const credentialPathValue = resolvePath(credential, this.path);
    if (!credentialPathValue) {
      return false;
    }

Testing this in node console:
jsSdk = require("@0xpolygonid/js-sdk");
obj = {foo: 42};
fq = new jsSdk.FilterQuery("foo", (a, b) => (a === b), 42)
fq.execute(obj)
=> true

but:
obj = {foo: 0};
fq = new jsSdk.FilterQuery("foo", (a, b) => (a === b), 0)
fq.execute(obj)
=> false

For context: I wanted to test whether $in and $nin work when the value being tested is 0, since that's the default value in the circuit's values input and also in the public signals (the number of values used is not indicated there).


Added new quotes and deleted few duplicates

Removed/Replaced some duplicate quotes and added few
@NikhilNamal17

@Aravind22 your branch has conflicts, pls resolve and pr again

@Aravind22 your branch has conflicts, pls resolve and pr again

I think i have resolved it, please check.


Ingress v1

Fixes https://github.com/k8gb-io/k8gb/issues/847 (we can't deploy on<EMAIL_ADDRESS>and newer)
All the details are described in the issue ^
This is an API breaking change, because the ingress@v1 resource has a slightly different structure and we use that sub-tree in our own gslb CRD.
Nginx ingress-controller was updated because version 3 works w/ the old ingress (v1beta1)
The idea is that for Ingress@v1beta1 (kubernetes version 1.21 and lower) use k8gb in version 0.8.8 or lower and for Ingress@v1 (since k8s@1.19), use the > 0.8.8.

@k0da another part of this should be also bumping the version of external-dns deployment, right? This might go in a separate PR afterwards.
Although we don't use the Ingresses nor Services for communicating the DNS records for ext-dns (we use CRs). I can imagine there might be some expectations baked in the external-dns binary. however the tests are green with external-dns:rfc-ns1 img.
Would it be hard to rebase our changes on the latest external-dns? What was the repo for this image absaoss/external-dns:rfc-ns1, is it this branch? I can't find external-dns repo under absaoss GH org, nor under k8gb-io org.
side note: we might use similar compatibility matrix as ext-dns have in their readme: https://github.com/kubernetes-sigs/external-dns/pkgs/container/external-dns%2Fexternal-dns#kubernetes-version-compatibility

@jkremser We simply can switch to latest released upstream. As change was finally merged a while ago


Font Not Showing

I've just installed your Theme (which I like) but the Cartograph CF font that you used is not showing (I have it on my PC). How Can I solve this?
I've attached you a screenshot.

Thanks

@c0mm0dore Hey, man! I had the same problem.


Replace from_utc with DateTime::from_naive_utc_and_offset

DateTime::<Tz>::from_utc is deprecated now.
see log here: https://github.com/confidential-containers/attestation-service/actions/runs/6531296026/job/17732254060?pr=147

cc @Xynnn007

seems like this should not be a independent change


Observations

Why Atom is as slow as Molasses in the Winter
Auto encapsulating my tags

The number of packages installed on Atom cause a delay at startup
In settings under packages look at each package.
If you see a settings button for that particular package click it. It will tell you for ex.
This package added 101ms to startup time.
Some of these packages also interfere with each other.
I spent most of the day working through the material for MART 341, and wishing that Atom would perform like Notepad ++
I ended up killing a few of the packages by pushing disable.
I wanted the encapsulation that I have with NP++.
I ended up finding a package called auto-complete html.
I changed the settings in the force section to "*"
Find that Package Information Here
The result is now I do not have to bloody type nearly half as much and my production speed increases.
For those of you who have used NP++ you know what I am talking about. So I figured if anyone wanted
that functionality there it is.

Yeah atom is slow for me too it really drags down performance.


Can't use terminal to full capabilities. Default shortcuts are getting in the way.

Does this issue occur when all extensions are disabled?: Yes

VS Code Version: 1.76.0
OS Version: Ubuntu 22.10 x64

Steps to Reproduce:

Open the terminal. Any shell (bash, zsh, Windows CMD, etc)
Try to pass Alt + V (for example) to a program
Watch the menu pop up when nobody asked

Alt + V is not passed to the program and it really throws off my flow. Is it possible to implement a "zen terminal" mode, perhaps, that disables all keyboard shortcuts so they can be passed to the terminal?
VS Code's built in terminal is the best one available to me (full ASCII RGB color support, what more could I ask for?!). I'd like it even more if I could use all my programs without having to change their bindings, like the one that uses Alt + V.

See https://code.visualstudio.com/docs/terminal/advanced#_keybinding-and-the-shell
You can set terminal.integrated.sendKeybindingsToShell to false to prefer the terminal which I think is what you're after. There's also fine tuning you can do with terminal.integrated.commandsToSkipShell, terminal.integrated.allowMnemonics and terminal.integrated.allowChords

Oh that's interesting. Thanks.


Initial support for London + Difficulty bomb delay (EIP-3554)

Issue #233

Do you have EIP-3541 inplemented?

No, not yet.


Comments handled incorrectly

return Column(
  children: [
    Container(), //
    Container(),
  ],
);

The second container will be recognized as a comment which is incorrect.

This issue should be fixed on the latest version!


Organize and group aliases better

Like, move this:
https://github.com/mhulse/dotfizzles/blob/master/.bash/aliases.bash#L87-L88
In the same group as the other related aliases at the top.

Done.


BAU: stop Rails 5 automatically disabling submit buttons

Rails 5 automatically disables submit buttons after they’ve been clicked on once. If you go back on our pages it remembers the disabled state, thus breaking the system. We can turn this functionality off globally.

This was added to Rails with this PR: https://github.com/rails/rails/pull/21135

Looks good 👍


Crash loading ConiqlPlugin

When you start using the ConiqlPlugin with coniql running, frequently you get a crash.
Somehow a VType with value null has arrived. We could fix this in vtypeToString(), but it would be nice to understand what's happening.
cc @skyfrench @TimGuiteDiamond

I think I can reproduce it by:

set up coniql
open a new tab (opening the console now is a good idea)
go to localhost:3000
click on the coniql link


fix: docs(ui/steps): use the setup script for exmple and docs

Checklist

[ ] Fix linting errors
[ ] Tests have been added / updated (or snapshots)

Change information

Codecov Report

Merging #505 (734c27c) into docs/script-setup (8a3eed9) will not change coverage.
The diff coverage is n/a.

@@                Coverage Diff                 @@
##           docs/script-setup     #505   +/-   ##
==================================================
  Coverage              96.42%   96.42%
==================================================
  Files                    217      217
  Lines                   5148     5148
  Branches                 944      944
==================================================
  Hits                    4964     4964
  Misses                   184      184

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8a3eed9...734c27c. Read the comment docs.


Setting create_origin_access_control to true does not work to successfully create an OAC based Cloudfront

Description
I am trying to enable OAC for my distribution using this module. Using create_origin_access_control = true with or without setting create_origin_access_identity causes terraform plan to fail with the error:
│ Error: Missing required argument
│
│   with module.my_cloudfront.aws_cloudfront_distribution.this[0],
│   on .terraform\modules\my_cloudfront\main.tf line 27, in resource "aws_cloudfront_distribution" "this":
│   27: resource "aws_cloudfront_distribution" "this" {
│
│ The argument "origin.0.s3_origin_config.0.origin_access_identity" is required, but no definition was found.

[X] ✋ I have searched the open/closed issues and my issue is not listed.

Versions

Module version [Required]: ~> 3.2.1
Terraform version: Terraform v1.4.2
Provider version(s):

+ provider registry.terraform.io/hashicorp/aws v4.67.0
+ provider registry.terraform.io/hashicorp/random v3.5.1

Reproduction Code [Required]
################################################################################
# My Cloudfront
################################################################################

module "my_cloudfront" {
  source = "git::https://github.com/terraform-aws-modules/terraform-aws-cloudfront.git?ref=v3.2.1"

  comment             = "My Cloudfront"
  enabled             = true
  is_ipv6_enabled     = true
  price_class         = "PriceClass_100"
  retain_on_delete    = false
  wait_for_deployment = false
  aliases             = "achme.mydomain.com"
  http_version        = "http2and3"

  create_origin_access_control = true
  origin_access_control = {
    my_common_oac = {
      description      = "Origin Access Control for My Cloudfront"
      origin_type      = "s3"
      signing_behavior = "always"
      signing_protocol = "sigv4"
    }
  }

  # create_origin_access_identity = false
  # origin_access_identities = {
  #   my_common_oai = "Origin Access Identity for My Cloudfront"
  # }

  origin = [
    {
      domain_name = module.s3_one.s3_bucket_bucket_regional_domain_name
      s3_origin_config = {
        origin_access_control = "my_common_oac"
      }
      origin_id = module.s3_one.s3_bucket_bucket_regional_domain_name
    # },
    # {
    #   domain_name = module.s3_one.s3_bucket_bucket_regional_domain_name
    #   s3_origin_config = {
    #     origin_access_identity = "my_common_oai"
    #   }
    #   origin_id = module.s3_one.s3_bucket_bucket_regional_domain_name
    }
  ]

  default_cache_behavior = {
    target_origin_id       = "s3_one"
    viewer_protocol_policy = "redirect-to-https"
    allowed_methods        = ["HEAD", "GET", "OPTIONS"]
    cached_methods         = ["GET", "HEAD"]
    compress               = true
    query_string           = false
    min_ttl                = 0
    default_ttl            = 0
    max_ttl                = 86400
  }

  ordered_cache_behavior = [
    {
      path_pattern           = "Default/*"
      target_origin_id       = "s3_one"
      viewer_protocol_policy = "redirect-to-https"
      allowed_methods        = ["HEAD", "GET", "OPTIONS"]
      cached_methods         = ["GET", "HEAD"]
      compress               = true
      query_string           = false
      min_ttl                = 0
      default_ttl            = 0
      max_ttl                = 86400
    }
  ]

  # viewer_certificate = {
  #   acm_certificate_arn      = var.my_cdn_acm_globalsign_certificate_arn
  #   ssl_support_method       = "sni-only"
  #   minimum_protocol_version = var.my_cdn_MinProtocolVersion
  # }

  custom_error_response = [{
    error_code            = 403
    response_code         = 200
    error_caching_min_ttl = 3600
    response_page_path    = "/index.html"
    }, {
    error_code            = 404
    response_code         = 404
    error_caching_min_ttl = 3600
    response_page_path    = "/index.html"
    }, {
    error_code            = 500
    response_code         = 500
    error_caching_min_ttl = 3600
    response_page_path    = "/index.html"
    }, {
    error_code            = 502
    response_code         = 502
    error_caching_min_ttl = 3600
    response_page_path    = "/index.html"
    }, {
    error_code            = 503
    response_code         = 503
    error_caching_min_ttl = 3600
    response_page_path    = "/index.html"
  }]

  geo_restriction = {
    restriction_type = "none"
  }

  tags = local.tags
}

######
# ACM
######
# data "aws_route53_zone" "this" {
#   name = local.domain_name
# }
# module "acm" {
#   source  = "terraform-aws-modules/acm/aws"
#   version = "~> 4.0"
#
#   domain_name               = local.domain_name
#   zone_id                   = data.aws_route53_zone.this.id
#   subject_alternative_names = ["${local.subdomain}.${local.domain_name}"]
# }

#############
# S3 buckets
#############
module "s3_one" {
  source  = "terraform-aws-modules/s3-bucket/aws"
  version = "~> 3.0"
  bucket        = "s3-one-${random_pet.this.id}"
  force_destroy = true
}

Expected behavior
Because I have set create_origin_access_control = true, I expect the distribution to be able to setup and configure the OAC without force-requiring OAI to be used
Actual behavior
OAC is not getting created. Terraform plan fails with the following error:
│ Error: Missing required argument
│
│   with module.my_cloudfront.aws_cloudfront_distribution.this[0],
│   on .terraform\modules\my_cloudfront\main.tf line 27, in resource "aws_cloudfront_distribution" "this":
│   27: resource "aws_cloudfront_distribution" "this" {
│
│ The argument "origin.0.s3_origin_config.0.origin_access_identity" is required, but no definition was found.

NVM - I am not a smart man. Realized the origin section needs to be a map, not a list. This issue can be closed as PEBKAC


[BUG] Cannot scroll on swap with server-sent events

Important Notice
using  sse chat example can not scroll on  sse message

Did you try sse_rand_scroll.py? That one works for me. If it doesn't, create a new issue in which you fill out the form properly so we can actually help.

sse_rand_scroll.py

thanks I am going to try

tried sse_rand_scroll.py and not working
the problem is you have to render markdown code then sse_rand_scroll is not working


[WIP] librem-ec-acpi: init at 0.9.1

This is more or less copied from the system76-acpi package, whose
upstream is the basis for librem-ec-acpi.
Motivation for this change
This kernel module is required for power management on Purism Librem 14
laptops because there is no mainline support yet.
Things done
Basically nothing is done yet, because I still need to learn about how kernel modules are actually built and managed. Using nix-shell and my own checkout I was able to follow the development guide in the manual and build a module for my kernel, which was enough for me to configure the charging thresholds and ensure I can use my laptop.

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[ ] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
21.11 Release Notes (or backporting 21.05 Relase notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Thanks! FWIW the patch is still incorrect I didn't even adjust the sha256 from what i cargo culted it as so I'm just committed your suggestions based on blind faith, I hope I didn't waste your time looking at something so incomplete


Add automatic installation of pysap-mri, pysap and pynufft / pynfft to docker image

Hello,
This PR basically adds apt.txt and requirements.txt so that direct launch with binder would allow for use of pysap and pynfft libraries without any installation.

@philouc you can squash merge this.
There could be minor updates to codes to point to where the classes exist now than where they were earlier. But we can surely fix and get it running on any machine.


WIP: Support AWS policy for weighted records

This PR is based on the proposal here: https://github.com/kubernetes-incubator/external-dns/pull/331. It assumes we're only doing the "cluster" portion now. Changes to the registry will be required to do the "global" part.
I have tested it and it is fully functional. I would appreciate feedback especially around the plan and the registry changes. I feel like there is a lot of logic there and wondering if you have ideas on how to make that part more straightforward. I was considering splitting that out even to its own package but thought I'd run it by here first. There's also bits of logic spread around with regards to using the AWS-specific record sets which I do not like, but I have not come up with any great ideas on how to get around that.
Also note that the changes to the inmemory zone are necessary because it's possible to have multiple records with the same dns name (and is necessary for weighting).

What's the status on this merge? I'm really looking forward to policy endpoints.

Since external-dns has support for v1.ServiceTypeNodePort in master, could this work be extended to cover weighted records for this type of service as well ?

The PR is stale for a long time. I'm closing it. Feel free to create a new PR.


Matchy Email Resend

We dun goofed and should resend the emails so that peeps will get the correct link

xD "we dun goofed"

S


Beautify JSON

Hey, how about beautifying the JSON and using <pre> instead of <div> for better styling?

This might be a quite long for bigger states and frequent actions being dispatched. It might be much more interesting to have a kind of collapsing in there instead of showing everything it all by default. But I guess that will be covered in more advanced monitors as they emerge for sure.

I'm inclined to merge this until somebody implements collapsing.
Any objections?

For bigger states, select prop helps to focus on a particular part. <DevTools select={state => state.todos} />

I played with it some more.. I'm not sure this is very helpful.
Deeply nested JSON is still painful to view.
I have another idea which I'll post shortly.
(Thanks for effort though!)

How about this? Horizontal scrolling for better JSON display?

I agree that a drop down (similar to the Chrome extension, JSON Formatter) would be most helpful.

Though its even less space horizontally now..

Some thoughts here: https://github.com/gaearon/redux-devtools/issues/38.
Maybe these should be two separate monitors.

I imagine your proposition would be the best solution, though its a bit more complex to implement. The current monitor as well as this proposition here are merely temporary solutions.

I think I like the last commit best, what do you think? Until the dock-monitor is done, we can still polish the simple-monitor.

Thank you! Let's hear what others have to say. With a bit of design polish I think this is a good temporary solution.

This is what it looks like now, horizontally scrollable, separately expandable action and state using <pre/> and a disable/enable label:

I'll leave it like that for now, waiting for feedback.

@bkniffler I love it! As a temporary solution it's definitely very good. +1 for merge

Hmm, here's how I see TodoMVC example in your branch:

Another problem is lots of linting errors:

Thanks for your effort! In the end we went with #48 because it also had a tree inspector.

Ah, no worries, it looks a lot better!


Fix typo in method name MmapAllocator::SizeClass::allocateFromMappdFree

@waitinfuture Would you rebase this PR on top of latest main?


Remove lock release typechain

Please review and merge https://github.com/decentdao/fractal-interface/pull/1663 first.

Liking these series of PRs

😁 ME TOO


A few suggestions

The last line can be changed to:
'print("Accuracy: ", sum(y_guess == y_test)/float(len(y_guess)))'
as I've got 0 in Python 2.7
Increace EPOCHES to 20 or more to get a more fair accuracy because of the differences in platform(such as hyper-parameters) and weights initialization and randomness with SGD.
Is the data need to zero mean normalization first to get a more resaonable result?

Regarding 3. instead of doing the equivalent of tf.image.per_image_standardization ((x - mean) / adjusted_stddev), the pixel intensities are just re-scaled. This probably degrades performance but shouldn't affect the relative difference between frameworks. I will play around to see if it makes a big difference.

Also, actually it might be worth to have a few iterations of "burn-in" before you time them, especially with the cuDNN auto-tune some of the frameworks will run the procedure when you build the computations while other will do it on the first call to any execution. Thus it is unfair to the latter if you just time them. I would suggest no more than 10 iterations for such burn-in phase, this should give a very tiny overhead on the whole execution.

I have a question regarding your RNN tensorflow implementation. Since you padded sequences with zero,  don't you need to implement some kind of masking mechanism, so that the loss contribution coming from pad character neutralized? How do you handle with that without implementing mask matrix? Does Tensorflow do it automatically?


⚠ Bofrost.de has degraded performance

In 716d897, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 1613 ms

Resolved: Bofrost.de performance has improved in 0cffa07 after 9 minutes.


request for additional configuration option for native_function_invocation

I've finally been getting around to using php-cs-fixer 2.x (not waiting around for #2553 to be resolved), and I ended up experimenting with it on vimeo/psalm#152, but some of the tests produce a rather verbose diff file ([nearly 7000 lines)(https://gist.github.com/SignpostMarv/840cac5070e7b5e840588b8e51903c18#file-native_function_invocation-only-diff)).
The argument behind native_function_invocation is to speed up resolving, and most things that improve performance are good things, but having such large diffs in pull requests is probably a bad idea (for several reasons that I won't go into at the moment).
So a thought occurred to me; use function strtolower; produces the same opcodes as \strtolower(); see 3v4l.org: backslash vs. 3v4l.org: use function.
Since use function has been available since 5.6, would it be possible to have a configuration option added to native_function_invocation to enable smaller diffs to be generated with the same performance improvement? i.e.:
$rules = [
    'native_function_invocation' => [
        'import' => true, // defaults to false to preserve existing behaviour
    ],
];

for convenience/those not wanting to click on 3val.org links:
<?php

namespace Foo;

use function strtolower;

strtolower('Foo');

Finding entry points
Branch analysis from position: 0
Jump found. (Code = 62) Position 1 = -2
filename:       /in/IUsaC
function name:  (null)
number of ops:  4
compiled vars:  none
line     #* E I O op                           fetch          ext  return  operands
-------------------------------------------------------------------------------------
   7     0  E >   INIT_FCALL                                               'strtolower'
         1        SEND_VAL                                                 'Foo'
         2        DO_ICALL
         3      > RETURN                                                   1

<?php

namespace Foo;

\strtolower('bar');

Finding entry points
Branch analysis from position: 0
Jump found. (Code = 62) Position 1 = -2
filename:       /in/gvU56
function name:  (null)
number of ops:  4
compiled vars:  none
line     #* E I O op                           fetch          ext  return  operands
-------------------------------------------------------------------------------------
   5     0  E >   INIT_FCALL                                               'strtolower'
         1        SEND_VAL                                                 'bar'
         2        DO_ICALL
         3      > RETURN                                                   1

Sounds like a good idea to me 👍

It was already discussed once with author of rule - @localheinz - as nice config option to have, but was postponed to undefined future. ref https://github.com/FriendsOfPHP/PHP-CS-Fixer/pull/2462
feel free to open a PR proposal.

Is there an existing rule that'd be a good starting point to look at to trigger the insertion of use statements ?

Instead of having this rule configurable, I would rather make new generic rule (that is executed after this, existing one) that would import all used classes from global namespaces (configurable to work with classes/functions/traits)
+use Foo\Bar;

-$x = new \Foo\Bar();
+$x = new Bar();

@SignpostMarv I can't remind any

import_absolute_references ?

sth like this, name could always be adjusted during review, if one would figure out better name (now, I cant ;) )

I was poking around the php-cs-fixer source last night, thinking "no_absolute_references" under PhpCsFixer\Fixer\Import might be a good fit ?
Also found something that may or may not be a bit of a problem - getPriority(); we'd want this to run after NativeFunctionInvocationFixer, but before NoUnusedImportsFixer & OrderedImportsFixer, might need to change things ?

as it's suppose to import things, import namespace is ok
priority, as long as you know what you want to achieve, won't be a problem

wouldn't changing the priority on a bunch of other fixers be a problem for others ?

priorities are to set relations between fixers, like fixer A has to be executed before fixer B.
as long as integration tests will be fine after your changes, everything is fine

poking around in the code a bit more today, so I can probably figure out how to add new use foo; statements, but I'm not sure how to identify the appropriate insertion point- I'm looking at what OrderedImportsFixer does with Tokens::fromCode() & $tokens->overrideRange()
presumably we want to insert tokens on the line after one of

"<?" or  "<?php" (first available)
namespace Foo; (used instead of opening tag position)
use Foo;

would appreciate help if I'm looking at the wrong example to figure things out from :D

I would suggest to do it step by step.
First step: create fixer that is not doing what you wanna achieve :D at least not fully.
GeneralImportAddFixer, that will receive configuration what to import and it will add imports for it.
So here, without detecting what to import. After it is done, you could reuse it in new fixer like NoFQCNWhenImportedFixer, that will detect what to import, ask first fixer to import it, and drop FQCN from usage.` But let us come back to this fixer after first part is ready and working.
I would ignore:

multi-namespaced files (we already have method for it: Tokens->findKinds... and count it)
non monolitic files (=multiple <?php) (we already have method for it as well: isMonolitic)
like we already do it some fixers.

then like you said, search for last existing imports, if it's not there go for namespace, if it's not there go for opening tag and skip following declare/docblock
You have the right place to add things. Now you need to build tokens for things you want to import and Tokens->insertAt them (no need to override).
You could convert string use Foo\Bar; via fromCode, but I would rather suggest to build Token items manually (parsing is fast, and we do know what string we want to achieve so we could prepare the tokens manually)
Don't be shy to reach me over gitter if you have any questions !

This would really be nice as an option to the native_function_invocation fixer (which I think is what has been proposed).  Used with the scope option set to namespaced may be a requirement, as PHP doesn't like the use function X statements when the code isn't namespaced.
https://3v4l.org/oicuF

Warning: The use statement with non-compound name 'strtolower' has no effect in /in/oicuF on line 3

done in https://github.com/FriendsOfPHP/PHP-CS-Fixer/pull/4355


feat: [Consign] Add medium select to price estimate

Addresses https://artsyproduct.atlassian.net/browse/GRO-91
This updates our price estimate search to include a medium drop-down to get more specific results.
The flow:

initial search makes query to priceInsights, returns a list of items with a bunch of mediums
mediums are pushed into select
when user selects medium, it fires off a new query to marketPriceInsights, with the artistID and medium
since the same piece of state is updated with the same response shape, things refresh without many changes

@damassi What does the zero state look like for an artist that doesn't have all categories?

Wow so fast! I'm wondering about the flow you have outlined here - any reason we can't query for all the markets to start?? I guess I'm confused about the relationship between the priceInsights and marketPriceInsights - I thought they were roughly equal. What I was expecting was for a single query to be issued that returns all the markets and then the select interaction simply causing a re-render in React, not a network request. 🤔

@williardx

What does the zero state look like for an artist that doesn't have all categories
We're pulling categories directly from the response, so we'll only ever see categories applicable to the artist (the category list updates after each fetch)

@jonallured
Sorry for being an idiot lol -- all updated and ready for a re-review!

LGTM! Thanks @damassi


[YUNIKORN-2448] Implemented licenses modal

What is this PR for?
This PR adds a menu option to display licenses used by the project. Clicking that menu option opens a modal dialog where all licenses are listed.
A list of licenses is generated on build time. For dev preview, a short example file is used.
What type of PR is it?

[ ]  - Bug Fix
[x]  - Improvement
[ ]  - Feature
[ ]  - Documentation
[ ]  - Hot Fix
[ ]  - Refactoring

What is the Jira issue?
YUNIKORN-2448
Questions?

Should we append at the beginning of the licenses preview dialog information that the project is licensed under the Apache License 2.0?

Screenshots (if appropriate)

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 73.07%. Comparing base (db71be7) to head (f1f6a5b).

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #175   +/-   ##
=======================================
  Coverage   73.07%   73.07%
=======================================
  Files           1        1
  Lines          26       26
=======================================
  Hits           19       19
  Misses          4        4
  Partials        3        3

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

@dcoric thanks for this contribution!


HDFS-16785. DataNode hold BP write lock to scan disk

Description of PR
When patching the fine-grained locking of datanode, I  found that addVolume will hold the write block of the BP lock to scan the new volume to get the blocks. If we try to add one full volume that was fixed offline before, i will hold the write lock for a long time.
The related code as bellows:
for (final NamespaceInfo nsInfo : nsInfos) {
  String bpid = nsInfo.getBlockPoolID();
  try (AutoCloseDataSetLock l = lockManager.writeLock(LockLevel.BLOCK_POOl, bpid)) {
    fsVolume.addBlockPool(bpid, this.conf, this.timer);
    fsVolume.getVolumeMap(bpid, tempVolumeMap, ramDiskReplicaTracker);
  } catch (IOException e) {
    LOG.warn("Caught exception when adding " + fsVolume +
        ". Will throw later.", e);
    exceptions.add(e);
  }
}

And I noticed that this lock is added by HDFS-15382, means that this logic was not with lock before.

@MingXiangLi @Hexiaoqiao Sir, can you help me review this PR?

I think it's necessary to hold the pool lock.Or make sure the following scenarios not have problem without hold block pool lock for example.
1、When fsVolume.getVolumeMap() scan the bock from disk to add block metadata，it may add new block metadata when another thread add block.
2、How we deal conflict when remove BlockPool operating occur at same time.

1、When fsVolume.getVolumeMap() scan the bock from disk to add block metadata，it may add new block metadata when another thread add block.

Great point. +1 from my side. So volume level lock will be more proper?

2、How we deal conflict when remove BlockPool operating occur at same time.

Not sure if this is block issue. This improvement seems not involve global meta management, only prepare phase here? So any conflict here?

Great point. +1 from my side. So volume level lock will be more proper?
volume level is ok, It will hold the block pool read lock, to avoid case2 conflict  occur。
Not sure if this is block issue. This improvement seems not involve global meta management, only prepare phase here? So any conflict here?
addVolume() may invoke by refresh Configuration.It may conflict by FsDatasetImpl#shutdownBlockPool.There are two stages in FsDatasetImpl#shutdownBlockPool.For example conflict will occur like this.
 FsDatasetImpl#shutdownBlockPool*volumeMap.cleanUpBlockPool(bpid); FsDatasetImpl#addVolume*fsVolume.getVolumeMap(bpid, tempVolumeMap, ramDiskReplicaTracker) FsDatasetImpl#shutdownBlockPool*volumes.removeBlockPool(bpid, blocksPerVolume);

addVolume() may invoke by refresh Configuration.It may conflict by FsDatasetImpl#shutdownBlockPool.There are two stages in FsDatasetImpl#shutdownBlockPool.For example conflict will occur like this.
1.FsDatasetImpl#shutdownBlockPoolvolumeMap.cleanUpBlockPool
2.FsDatasetImpl#addVolumefsVolume.getVolumeMap
3.FsDatasetImpl#shutdownBlockPool*volumes.removeBlockPool

Got it. What I mean above is that fsVolume.getVolumeMap(bpid, tempVolumeMap, ramDiskReplicaTracker); is absolutely local variable operation, so is it necessary to hold lock here?

@MingXiangLi @Hexiaoqiao Sir, thanks for the warm discussions.

1、When fsVolume.getVolumeMap() scan the bock from disk to add block metadata，it may add new block metadata when another thread add block.

This volume is still not added into the FsVolumeList here, means another thread can not add new block into this volume?  So this case not exists?

2、How we deal conflict when remove BlockPool operating occur at same time.

RemoveBlockPool just remove the blocks in the memory replicasMap, will not delete blocks on the disk. So remove block pool operation will not affect scanning disk. Let's see some stages:
Case 1: It's ok.

fsVolume.addBlockPool();
volumeMap.cleanUpBlockPool(bpid);
volumes.removeBlockPool(bpid, blocksPerVolume); Here will remove the blockPoolSlice
fsVolume.getVolumeMap(); here will throw IOException, because the blockPoolSlice is null.

Case2:

addVolume get NamespaceInfo
volumeMap.cleanUpBlockPool(bpid);
volumes.removeBlockPool(bpid, blocksPerVolume); Here will remove the blockPoolSlice
fsVolume.addBlockPool();
fsVolume.getVolumeMap(); here will be ok
activateVolume(); here will add the removed bpid into the replicamap again. Maybe is not expected. But it's another problem, should be fixed by other PR.

From my point, scanning the disk with a lock for a long time is not allowed. If there are some conflict here, we can find them and fix by other solution.

fsVolume.getVolumeMap(); here will throw IOException, because the blockPoolSlice is null.

I think we can change this lock to read lock.This not influences most of read/write operate.And can avoid conflict cause by volume/block pool remove or add.

try (AutoCloseDataSetLock l = lockManager.readLock(LockLevel.VOLUME, bpid, fsVolume.getStorageID())) {
        fsVolume.addBlockPool(bpid, this.conf, this.timer);
        fsVolume.getVolumeMap(bpid, tempVolumeMap, ramDiskReplicaTracker);
      } catch (IOException e) {
        LOG.warn("Caught exception when adding " + fsVolume +
            ". Will throw later.", e);
        exceptions.add(e);
      }

Changing the code as above?  Emm.. Holding the BP read lock for a long time will have a great impact on the operations that need to acquire the BP write lock, such as: invalidate, recoverAppend, createTemporary.
The current logic uses IOException to avoid the conflict case, I think it's ok. And there is no lock before HDFS-15382, means it's ok. If we can find one conflict case, we can use IOException to fix it.

LGTM @Hexiaoqiao any good suggestion?

@MingXiangLi Sir, thanks for your review.

Case2: It's wrong, but not caused by this lock.

About this case, I will create one new PR to fix it.

And can avoid conflict cause by volume/block pool remove or add.

We can add one synchronized lock to addBlockPool and shutdownBlockPool as before to avoid the conflict caused by volume/blockPool remove or add. And this synchronized just lock volume/blockPool remove or add, so it will not block other operations, such as read or write from client. If ok, I will fix it in a new ticket with Case2 together.
@Hexiaoqiao looking forward your good ideas.

I totally agree that we should not hold lock when ever IO operation, especially scan the whole disk, it will be one terrible disaster even at refresh volume. Of course it does not include during restart DataNode instance.
Back to this case. I think the point is that we should only hold block pool lock (maybe write lock here) when get/set BlockPoolSlice rather than one coarse grain lock.
So should we split the following segment and only hold lock for BlockPoolSlice. And leave other logic without any level locks. I think it could be acceptable if meet any conflicts or other exceptions about only one volume which is being added.
      try (AutoCloseDataSetLock l = lockManager.writeLock(LockLevel.BLOCK_POOl, bpid)) {
        fsVolume.addBlockPool(bpid, this.conf, this.timer);
        fsVolume.getVolumeMap(bpid, tempVolumeMap, ramDiskReplicaTracker);
      }

WDYT? cc @MingXiangLi @ZanderXu

  final FsVolumeImpl fsVolume =
        createFsVolume(sd.getStorageUuid(), sd, location);
    // no need to add lock
    final ReplicaMap tempVolumeMap = new ReplicaMap();
    ArrayList<IOException> exceptions = Lists.newArrayList();

    for (final NamespaceInfo nsInfo : nsInfos) {
      String bpid = nsInfo.getBlockPoolID();
      try (AutoCloseDataSetLock l = lockManager.writeLock(LockLevel.BLOCK_POOl, bpid)) {
        fsVolume.addBlockPool(bpid, this.conf, this.timer);
        fsVolume.getVolumeMap(bpid, tempVolumeMap, ramDiskReplicaTracker);
      } catch (IOException e) {
        LOG.warn("Caught exception when adding " + fsVolume +
            ". Will throw later.", e);
        exceptions.add(e);
      }
    }

The fsVolume here is a local temporary variable and still not be added into the volumes, and add/remove bp operations just use the volume in volumes, so there is no conflicts. So here doesn't need the lock for BlockPoolSlice.
@Hexiaoqiao Sir, can check it again?

@Hexiaoqiao @tomscut @haiyang1987 @MingXiangLi Sir, can you help me review this PR?
It will block the DataNode for a long time when we dynamically add one disk with many blocks.

  final FsVolumeImpl fsVolume =
        createFsVolume(sd.getStorageUuid(), sd, location);
    // no need to add lock
    final ReplicaMap tempVolumeMap = new ReplicaMap();
    ArrayList<IOException> exceptions = Lists.newArrayList();

    for (final NamespaceInfo nsInfo : nsInfos) {
      String bpid = nsInfo.getBlockPoolID();
      try (AutoCloseDataSetLock l = lockManager.writeLock(LockLevel.BLOCK_POOl, bpid)) {
        fsVolume.addBlockPool(bpid, this.conf, this.timer);
        fsVolume.getVolumeMap(bpid, tempVolumeMap, ramDiskReplicaTracker);
      } catch (IOException e) {
        LOG.warn("Caught exception when adding " + fsVolume +
            ". Will throw later.", e);
        exceptions.add(e);
      }
    }

The fsVolume here is a local temporary variable and still not be added into the volumes, and add/remove bp operations just use the volume in volumes, so there is no conflicts. So here doesn't need the lock for BlockPoolSlice.
@Hexiaoqiao Sir, can check it again?

I agree with @ZanderXu here. +1 from my side.

LGTM. +1 from my side. sorry for the late response.
cc @MingXiangLi any more comments here?

re-trigger jenkins and let's wait what it will say.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
46m  0s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
43m  8s

trunk passed

+1 :green_heart:
compile
1m 30s

trunk passed

+1 :green_heart:
checkstyle
1m 17s

trunk passed

+1 :green_heart:
mvnsite
1m 37s

trunk passed

+1 :green_heart:
javadoc
1m 45s

trunk passed

+1 :green_heart:
spotbugs
3m 49s

trunk passed

+1 :green_heart:
shadedclient
26m  7s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m 23s

the patch passed

+1 :green_heart:
compile
1m 18s

the patch passed

+1 :green_heart:
javac
1m 18s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
1m  1s

the patch passed

+1 :green_heart:
mvnsite
1m 26s

the patch passed

+1 :green_heart:
javadoc
1m 28s

the patch passed

+1 :green_heart:
spotbugs
3m 32s

the patch passed

+1 :green_heart:
shadedclient
25m 31s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
347m 49s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch passed.

+1 :green_heart:
asflicense
1m 16s

The patch does not generate ASF License warnings.

506m 33s

Reason
Tests

Failed junit tests
hadoop.hdfs.server.namenode.ha.TestHAStateTransitions

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4945/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/4945

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux 35e578afdaa6 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 61a5d265943a4682c93a3d48f23eba8c7d44cf37

Default Java
Red Hat, Inc.-1.8.0_352-b08

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4945/2/testReport/

Max. process+thread count
2237 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4945/2/console

versions
git=2.9.5 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

The failed UT hadoop.hdfs.server.namenode.ha.TestHAStateTransitions doesn't relate to this PR and it works well locally.

Committed to trunk. Thanks @ZanderXu for your works. And@MingXiangLi @tomscut for your reviews.


Support external OIDC identity provider

Is your request related to a new offering from AWS?
yes , aws now supports external oidc identity provider - https://aws.amazon.com/blogs/containers/introducing-oidc-identity-provider-authentication-amazon-eks/
https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eks_identity_provider_config
Is your request related to a problem? Please describe.
I am trying associating the oidc identity provider as post step after the cluster is created and is taking a long time
ref: https://github.com/aws/containers-roadmap/issues/1438
Describe the solution you'd like.
the external oidc provider is associated with the eks cluster during the build time so that the api server is not restarted to associate the oidc provider

Looking into docs this is using resource https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eks_identity_provider_config
so you can achieve this outside module by just:
resource "aws_eks_identity_provider_config" "example" {
  cluster_name = module.eks_cluster.cluster_id

  oidc {
    client_id                     = "your client_id"
    identity_provider_config_name = "example"
    issuer_url                    = "your issuer_url"
  }
}

Yes i am doing that now, the problem is it takes like 25-35 mins to associate and disassociate the identity provider increasing the overall time to build a new cluster and teardown it

how embedding this into module will influence those timings?

Currently this is done as post step after the cluster is built out, causing the api server to restart . I was wondering if we can hook this identity provider config as part of the cluster build rather than post step

I don`t see any option (in webui) to add this during a creation, so seems you must first have cluster control planes available and then attach it.
I don`t think there will be any way to make it faster as seems be AWS limitation.

I also checked AWS API here https://docs.aws.amazon.com/eks/latest/APIReference/API_CreateCluster.html and here https://docs.aws.amazon.com/eks/latest/APIReference/API_AssociateIdentityProviderConfig.html so as per me there is no way to make this in other way than proposed initially

For the time being, I don't see us adding even more resources like aws_eks_identity_provider_config into this module even if it has a potentially better experience for the end-users. There is already a lot of magic in this module which we need to improve first (see #635, for e.g.).
@bseenu I agree with this comment - https://github.com/terraform-aws-modules/terraform-aws-eks/issues/1483#issuecomment-909230653 - you should be able to accomplish this outside of the module.
Closing this issue.

I'm trying to achieve something like this, let me know if this is possible to achieve with this module in 2022. @antonbabenko
data "tls_certificate" "eks"{
        url = aws_eks_cluster.demo.identity[0].oidc[0].issuer
}

resource "aws_iam_openid_connect_provider" "eks"{
    client_id_list  = ["sts.amazonaws.com"]
    thumbprint_list = [data.tls_certificate.eks.certificates[0].sha1_fingerprint]
    url             =  aws_eks_cluster.demo.identity[0].oidc[0].issuer
}

resource"aws_iam_policy" "test-policy" {
    name="test-cluster-autoscaler-policy"
    policy = jsonencode({
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "autoscaling:SetDesiredCapacity",
                "autoscaling:TerminateInstanceInAutoScalingGroup"
            ],
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "aws:ResourceTag/k8s.io/cluster-autoscaler/${var.eks_cluster_name}": "owned"
                }
            }
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "autoscaling:DescribeAutoScalingInstances",
                "autoscaling:DescribeAutoScalingGroups",
                "ec2:DescribeLaunchTemplateVersions",
                "autoscaling:DescribeTags",
                "autoscaling:DescribeLaunchConfigurations"
            ],
            "Resource": "*"
        }
    ]
    })
}

resource"aws_iam_role" "test_oidc"{
    assume_role_policy = data.aws_iam_policy_document.test_oidc_assume_role_policy.json
    name = "test-oidc"
}

resource "aws_iam_role_policy_attachment" "test_attach"{
    role = aws_iam_role.test_oidc.name
    policy_arn = aws_iam_policy.test-policy.arn
}

data "aws_iam_policy_document" "test_oidc_assume_role_policy" {
  statement {
    actions = ["sts:AssumeRoleWithWebIdentity"]
    effect  = "Allow"

    condition{
      test     = "StringEquals"
      variable = "${replace(aws_iam_openid_connect_provider.eks.url,"https://","")}:sub"
      values   = ["system:serviceaccount:kube-system:cluster-autoscaler"]
    }

    principals{
      identifiers      = [aws_iam_openid_connect_provider.eks.arn]
      type             = "Federated"
    }
  }
}

thanks @bryantbiggs I was able to replicate this but the problem with latest eks module's version is when I'm trying to execute it with the following code:
Main "variables" to take in consideration are:

create_iam_role = true
create_cloudwatch_log_group = false
iam_role_use_name_prefix = false

`module "eks" {
source = "terraform-aws-modules/eks/aws"
version      = "18.21.0"
cluster_name = var.eks_cluster_name
cluster_version = var.kubernetes_version
enable_irsa = true
subnet_ids = module.vpc.private_subnets
cluster_endpoint_private_access = true
vpc_id = module.vpc.vpc_id
eks_managed_node_group_defaults = {
instance_types    = var.node_instance_type
disk_size   = var.node_ami_disk_size
ami_type = var.node_ami_type
}
eks_managed_node_groups = {
nodes = {
desired_size = var.nodes_desired_capacity
max_size     = var.nodes_max_capacity
min_size     = var.nodes_min_capacity
instance_types = var.node_instance_type
capacity_type = var.node_capacity_type
  k8s_labels = {
    Environment = "on_demand"
    GithubRepo  = "terraform-aws-eks"
    GithubOrg   = "terraform-aws-modules"
  }
  additional_tags = {
    ExtraTag = "on_demand-node"
  }
}

create_iam_role = true
create_cloudwatch_log_group = false
iam_role_use_name_prefix = false

}
}
data "aws_eks_cluster" "cluster" {
name = module.eks.cluster_id
}
data "aws_eks_cluster_auth" "cluster" {
name = module.eks.cluster_id
}
data "aws_iam_openid_connect_provider" "cluster_oidc_arn" {
arn = module.eks.oidc_provider_arn
}
data "aws_iam_openid_connect_provider" "cluster_oidc_url" {
url = module.eks.oidc_provider
}`
I'm getting the following error:

bash-4.2# terraform apply
╷
│ Error: expected length of name_prefix to be in the range (1 - 38), got create_cloudwatch_log_group-eks-node-group-
│
│   with module.eks.module.eks_managed_node_group["create_cloudwatch_log_group"].aws_iam_role.this[0],
│   on .terraform/modules/eks/modules/eks-managed-node-group/main.tf line 435, in resource "aws_iam_role" "this":
│  435:   name_prefix = var.iam_role_use_name_prefix ? "${local.iam_role_name}-" : null
│
╵
╷
│ Error: expected length of name_prefix to be in the range (1 - 38), got iam_role_use_name_prefix-eks-node-group-
│
│   with module.eks.module.eks_managed_node_group["iam_role_use_name_prefix"].aws_iam_role.this[0],
│   on .terraform/modules/eks/modules/eks-managed-node-group/main.tf line 435, in resource "aws_iam_role" "this":
│  435:   name_prefix = var.iam_role_use_name_prefix ? "${local.iam_role_name}-" : null

I struggled a lot with the "parent eks" module as well as with the child "eks-managed-node-group" module but can't figure this out.

I think you have an indentation issue
module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "18.21.0"

  cluster_name = var.eks_cluster_name
  cluster_version = var.kubernetes_version
  enable_irsa = true
  subnet_ids = module.vpc.private_subnets
  cluster_endpoint_private_access = true
  vpc_id = module.vpc.vpc_id

  # This should be at cluster level
  create_cloudwatch_log_group = false

  # I don't know if you meant this for the cluster IAM role or the node IAM role, moved to cluster level
  iam_role_use_name_prefix = false

  eks_managed_node_group_defaults = {
    instance_types    = var.node_instance_type
    disk_size   = var.node_ami_disk_size
    ami_type = var.node_ami_type
  }

  eks_managed_node_groups = {
    nodes = {
      desired_size = var.nodes_desired_capacity
      max_size     = var.nodes_max_capacity
      min_size     = var.nodes_min_capacity
      instance_types = var.node_instance_type
      capacity_type = var.node_capacity_type

      k8s_labels = {
        Environment = "on_demand"
        GithubRepo  = "terraform-aws-eks"
        GithubOrg   = "terraform-aws-modules"
      }
      additional_tags = {
        ExtraTag = "on_demand-node"
      }
    }
  }
}

Thanks, @bryantbiggs  for the quick resolution here, it was my bad that I hadn't double-checked the indentation of the block.
But again I have another problem, while using the following:
data "aws_iam_openid_connect_provider" "cluster_oidc_url" {   url = module.eks.cluster_oidc_issuer_url }
I'm getting the following error on terraform apply:

│ Error: error finding IAM OIDC Provider by URL (https://oidc.eks.us-east-1.amazonaws.com/id/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx): not found
│
│   with data.aws_iam_openid_connect_provider.cluster_oidc_url,
│   on eks-cluster.tf line 84, in data "aws_iam_openid_connect_provider" "cluster_oidc_url":
│   84: data "aws_iam_openid_connect_provider" "cluster_oidc_url" {

I checked in the aws console the same "URL" is being created under "OpenID Connect provider URL".
Please let me know what I'm missing here.

I would look it up using the ARN instead:
data "aws_iam_openid_connect_provider" "this" {
  arn = module.eks.oidc_provider_arn
}

Oh I see but I'm trying to do achieve something like:
`resource "aws_iam_policy" "test-policy" {
name="test-cluster-autoscaler-policy"
policy = jsonencode({
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"autoscaling:SetDesiredCapacity",
"autoscaling:TerminateInstanceInAutoScalingGroup"
],
"Resource": "",
"Condition": {
"StringEquals": {
"aws:ResourceTag/k8s.io/cluster-autoscaler/${var.eks_cluster_name}": "owned"
}
}
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": [
"autoscaling:DescribeAutoScalingInstances",
"autoscaling:DescribeAutoScalingGroups",
"ec2:DescribeLaunchTemplateVersions",
"autoscaling:DescribeTags",
"autoscaling:DescribeLaunchConfigurations"
],
"Resource": ""
}
]
})
}
resource"aws_iam_role" "test_oidc"{
assume_role_policy = data.aws_iam_policy_document.test_oidc_assume_role_policy.json
name = "test-oidc"
}
resource "aws_iam_role_policy_attachment" "test_attach"{
role = aws_iam_role.test_oidc.name
policy_arn = aws_iam_policy.test-policy.arn
}
data "aws_iam_policy_document" "test_oidc_assume_role_policy" {
statement {
actions = ["sts:AssumeRoleWithWebIdentity"]
effect  = "Allow"
condition{
  test     = "StringEquals"
  variable = "${data.aws_iam_openid_connect_provider.cluster_oidc_arn.arn}:sub"
  values   = ["system:serviceaccount:kube-system:cluster-autoscaler"]
}

principals{
  identifiers      = [data.aws_iam_openid_connect_provider.cluster_oidc_url.url]
  type             = "Federated"
}

}
}
output "test_policy_arn" {
value = aws_iam_role.test_oidc.arn
}`
And "data sources" as:

data "aws_iam_openid_connect_provider" "cluster_oidc_arn" {
arn = module.eks.oidc_provider_arn
}
data "aws_iam_openid_connect_provider" "cluster_oidc_url" {
url = module.eks.cluster_oidc_issuer_url
}

Please let me know what I'm missing here.

Why not use https://github.com/terraform-aws-modules/terraform-aws-iam/tree/master/modules/iam-role-for-service-accounts-eks
module "cluster_autoscaler_irsa_role" {
  source      = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"
  version     = "~> 5.0"

  role_name                        = "cluster-autoscaler"
  attach_cluster_autoscaler_policy = true
  cluster_autoscaler_cluster_ids   = [module.eks.cluster_id]

  oidc_providers = {
    ex = {
      provider_arn               = module.eks.oidc_provider_arn
      namespace_service_accounts = ["kube-system:cluster-autoscaler"]
    }
  }

  tags = local.tags
}

@bryantbiggs beautiful ❤
Now, my only questions are:

Can we implement/deploy/achieve -> Cluster Autoscaler and Metrics Server using this module?
If not, what's the best way to achieve these two?

This module does not provision cluster level resources (aws-auth configmap is the only cluster level management performed in this module). You can wrap the helm charts in the Terraform helm chart provider like https://github.com/clowdhaus/eks-reference-architecture/blob/e2a8cdc6405a4eff69bb6119ec57dc729f6ab8f1/karpenter/us-east-1/karpenter.tf#L37 or you can use a GitOps based approach with something like ArgoCD or Flux (would be the recommended way)

@bryantbiggs  thanks a ton for the help.
I've one more query regarding "cluster version update" while using terraform and eks module I'm trying to change the existing k8s cluster version to a newer one but I'm not getting the expected behaviour.

From the screen please check when I applied the changes for k8s cluster version update (in this case IAM user has the "UpdateCluster" permission attached as well" but instead of making the output as "waiting for update OR updating...) it is just throwing an error and when I checked the "aws console" it was showing "updating" status and updated the cluster's version.
The problem here is the terraform's output on this particular activity.
Please let me know if you can provide me with an explanation for this.

@bryantbiggs Oh, I have the following permissions set for the IAM user.

Not sure whether the rest (on the left are also required), the problem is there isn't any perfect list for required permissions at all.

right, and the DescribeUpdate is probably necessary for Terraform to keep checking if the update has succeeded (or it will hit the timeout set in the provider and error out if the update doesn't progress within the window)

Got it, thanks a lot for the clarification. @bryantbiggs


feat: calculate crc32 sum for files

Why
Need to add crc32 sum for file encryption so that integrity can be checked on files
How
Added the crc32 sum calculation and bumped the versions
Checklist

[x] Commit messages are formatted as per the convention, and breaking changes are marked

:tada: This PR is included in version 3.11.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


"Cannot find type 'AnyHashableSendable' in scope" bug on build on 1.0.2

Description

experiencing "Cannot find type 'AnyHashableSendable' in scope" on build
using swift version 6
using composable architecture 1.12.1
Checklist

[ ] I have determined whether this bug is also reproducible in a vanilla SwiftUI project.
[ ] If possible, I've reproduced the issue using the main branch of this package.
[X] This issue hasn't been addressed in an existing GitHub issue or discussion.

Expected behavior
No response
Actual behavior
No response
Reproducing project
No response
Sharing version information
1.0.2
Destination operating system
Vision OS 2.0
Xcode version information
Version 16.0 (16A242d)
Swift Compiler version information
Apple Swift version 6.0 (swiftlang-<IP_ADDRESS>.10 clang-16<IP_ADDRESS>)

Hi @ckse93, you cannot use an hold version of TCA and this library. You will need to wait until the PR I mentioned previously is merged and released.
But also, there is no reason for you to use this library with TCA 1.12. You can just keep using 1.12 alone since it already has all of the state sharing tools in it.
Since this isn't an issue with the library I am going to convert it to a discussion. Please feel free to continue the conversation over there!


Support for other languages (Spanish, German, etc..)

Hi there!
Is there any version of the pre-trained weights the was trained with data other than english? for example using https://arxiv.org/abs/2108.13897 mMARCO: A Multilingual Version of the MS MARCO ?

We haven't trained a version like this ourselves, but check out PrimeQA! They have multi-lingual colbert models iirc.


Add delete match button

resolves #40

Are you going to clean this up?

Yeah I'm almost done doing the checkbox thing but I've been pretty busy recently. All I really have to do is fix the splicing loop and it'll be done

Splicing loop is still broken because I wasn't really able to work on it over the long weekend. It loops through all the checked matches' places in the data array and splices them. Problem is that once it's spliced one, they're all out of order and it splices things it's not supposed to.
I mapped it out in google docs:

Ok I did the checkbox thing and changed the button take a look now.

@ErikBoesen @AndrewLester

@ianzullo
Two things:
Do you mind hiding the trash button if no data is found? Just add it to the else block on line 17 of data.js
Also can you write nothing to the data file if the mat list is empty, because right now it will write an empty array which causes the data window to assume that the file isn't empty, making the no-data message not show. (Maybe just an if statement checking the length attribute of the mat variable which you are writing to the file)


Solved Issue Number #726 #727 (Added left shift and right shift operator)

Closes: #[issue number that will be closed through this PR]
Issue Number #726  #727
Describe the changes you've made
Added case for left shift and right shift operator
Checklist:

[x] My code follows the style guidelines of this project.
[x] I have performed a self-review of my own code.
[x] My changes generate no new warnings.

Screenshots

Original
Updated

You just added the option?
Have you applied the working?

yes


Flipping bits [Hackerrank - Easy]

Aim
I would like to add the solution for Flipping bits [Easy level] from Hackerrank.
Details
Link to the problem: https://www.hackerrank.com/challenges/flipping-bits/problem?h_l=interview&playlist_slugs[]=interview-preparation-kit&playlist_slugs[]=miscellaneous
I would like to add my solution to this problem in Python in an efficient approach.
Please assign this issue to me under HCSSoC (Hack Club SRM Summer of Code) .
@Ayush7614 @Amit366
Programming language

[ ] C
[ ] C++
[ ] Java
[ ] Python

Can I try in c++ and python