cmd/compile: allow the use of go:wasmimport globally

Background
#38248 defined a new compiler directive, go:wasmimport, for declaring a dependency on a function provided by the runtime environment. It also defined a restriction on the use of this compiler directive, such that it cannot be used in code outside of the runtime and syscall/js standard library packages. #58141 relaxed this restriction to allow the use of the directive in the syscall package to allow accessing WASI defined syscall APIs necessary to support WASI.
The go:wasmimport directive is defined as follows:
//go:wasmimport importmodule importname

Where importmodule is the Wasm module of the function and importname is the name of the function within the module. The effect of this compiler directive is that any Go function calls of the function will be replaced with a call instruction to the function identified by the module and function name during compilation. Parameters and return values to the Go function are translated to Wasm according to the following table:

Go value

Wasm value

int8, uint8, int16, uint16, int32, uint32

i32

int64, uint64

i64

float32

f32

float64

f64

Pointer

i32

For more information see the ssagen compiler package.
Proposal
We propose that the package restriction on the use of the go:wasmimport directive be removed entirely.
Rationale
Much of the discussion around the WASI proposal (#58141) revolved around the question of how and when the WASI target should move between syscall APIs (i.e. from depending on the wasm_snapshot_preview1 API to the wasm_snapshot_preview2 API and beyond). A major reason for this anxiety is because of the conflict between the desire for stability and the desire for access to newer runtime features afforded by newer syscall APIs.
One way to help avoid this conflict would be to provide access to newer APIs through purpose built third-party packages (e.g. golang.org/x/wasi/preview2/net) implementing various syscall level APIs through the use of go:wasmimport and exposing Go native types implementing interfaces such as net.Listener and net.Conn. Users who want to opt-in to new functionality can import one of these packages with the understanding that it may be unstable. Users who prefer to wait to use this functionality until it’s in the standard library can stay on the older API version. We’re exploring adding functionality to wit-bindgen to generate Go files with go:wasmimport directives and function signatures.
In addition to this, it would also allow users or vendors to create their own packages around vendor specific runtime extensions (e.g. WasmEdge defines its own sockets API, Fermyon defines a custom HTTP handler API) and expose them as easily consumable Go modules. This would help make Go more flexible and useful in different Wasm runtime environments.
Discussion
Rationale for the existing restriction
The existing restriction was added as a last minute addition to the original proposal. The primary rationale for this seems to have been an anxiety around exposing this widely without having had any experience of using it. If changes had to be made, it could break existing users.
We believe this concern should be smaller now that we have experience of using the directive while implementing go:wasmimport and working on implementing the WASI target.
Regardless, we do want to reserve the right to make breaking changes to this directive until such a point as the wasm architecture itself is declared stable. This is in line with the reservations made in #38248.
Effect on js/wasm
The js/wasm target has similar functionality exposed via the use of the syscall/js standard library package. The existing standard library interfaces will remain the preferred way of accessing functions defined on the JavaScript host. The primary use case within the js/wasm target would be as a means of accessing functions defined by other Wasm modules loaded into the same environment.
A note on wasm module names
The Wasm Core 1.0 spec defines a name as used in module and function names as a sequence of UTF-8 codepoints. The current definition of the go:wasmimport directive uses a space to separate the module name from the function name, making it impossible to specify a module or function name with a space in it. This proposal does not suggest changing the definition to support this use case.
Future work
Richer type support for imported functions
The current implementation of go:wasmimport supports only functions with integer, floating point or pointer parameters and a single return value. In the future, we would like to support types such as strings and structs and multiple return values. This would require defining a more elaborate calling convention for the imported functions than the one currently used.
The current mechanism follows the same semantics in use by other Wasm compilers such as Clang, which effectively treat Wasm the same as C, requiring the caller and callee to share the same ABI to be able to decode values in memory.
Authors
@johanbrandhorst, @Pryz, @evanphx, @achille-roussel

Pointer
i32

Currently the implementation uses 32-bit linear memory. But that might change in the future? Does this have any impact to this? Thanks.

@cherrymui There is the 64bit memory proposal for web assembly, so in the future yes. One reason we used i32 here is to remain similar to what clang and friends use for pointers. We certainly could use i64 today to avoid future concerns though.

Thanks. Using i32 for now is probably fine. The concern is that if later we change to 64-bit address, will this break any user? Maybe it will not if the Wasm engine is backward compatible? Do you think there is any compatibility issue? If not, this is probably okay.

I've raised https://github.com/golang/go/issues/59156 which should clarify some things around behavior of pointers when using go:wasmimport. This proposal should only go ahead once #59156 has been resolved.

I've added a section to the proposal that disallows any parameter types other than those that easily map to Wasm types. This restriction could be relaxed in a hypothetical future wasm32 port, but will likely remain in the current port until we have a better idea of what a 64 bit Wasm runtime means. See #59156 for more details on this discussion, but I believe this unblocks further discussion in this proposal.

Thanks. SGTM. Maybe add a note that if Wasm moves to 64-bit pointers, the ABI translation rule for unsafe.Pointer might change?
A minor question: if the directive is seen when compiling to a non-Wasm platform, is it an error or just silently ignored?

Thank you, I updated the proposal with a section on future 64 bit Wasm runtimes.
The current behavior is to cause a compile time error when using the directive on a non-Wasm platform, and the current plan is to keep that behavior.

One thing to note is that there will be some users who tries to put go:wasmimport on global variables from my experience with Ziglang dev https://github.com/ziglang/zig/issues/4866 since import in Wasm can be applied to "wasm globals" and others as well, not only functions.
Support for Importing global variables will be far more complex than just functions, and standard compilers usually disable it AFAIK. So maybe we might want to add safeguard on this.

Also, two things to consider:

do we want to do incompatible duplicated go:wasmimport directives at linktime? (somehow Wasm spec allows it, but in reality, such binaries are not loadable/usable).
wasm spec allows empty names both on module and symbol names. do we want to support them with go:wasmimport? I guess in practice no one would request, but it's good to be aware of what's possible in spec vs in Go runtime.

Support for Importing global variables will be far more complex than just functions, and standard compilers usually disable it AFAIK. So maybe we might want to add safeguard on this.

Yeah, if we support go:wasmimport only on functions, we should emit an error if the directive is applied to a variable.

do we want to raise an error on incompatible duplicated go:wasmimport directives at linktime? (somehow Wasm spec allows it, but in reality, such binaries are not loadable/usable).

Either way it is fine for me. If at link time we don't naturally check for duplications when generating the table, I think it is fine not to. Does other linker check this?

wasm spec allows empty names both on module and symbol names. do we want to support them with go:wasmimport? I guess in practice no one would request, but it's good to be aware of what's possible in spec vs in Go runtime.

I guess that will require changing the syntax of the directive? As noted above, we don't support space in module names, so I guess it is okay to leave empty name out as well. If there is a need in the future we can extended the syntax.
Empty names seem very confusing, though. How does the other side provide the symbol?

Hm, theoretically, you could have libraries that import the same host function independently, right? Could you clarify what sort of problem this might cause?

well, what if two different packages import the functions with different signatures but with the same import names? E.g.
package foo

//go:wasmimport env hostfn
func hostfn(uint32)

package bar

//go:wasmimport env hostfn
func hostfn(float32)

if we have objects like ^^, then the resulted wasm binary would be like:
(module
  (type (;0;) (func (param i32)))
  (type (;1;) (func (param f32)))
  (import "env" "hostfn" (func (;0;) (type 0)))
  (import "env" "hostfn" (func (;1;) (type 1)))
)

which is somehow valid in the sense of Wasm spec, but in reality the binary is unusable. So my question is whether or not we want to raise an error when linking such objects. But I guess as @cherrymui mentioned, we don't need to do anything.

Empty names seem very confusing, though. How does the other side provide the symbol?

in Wasm spec, names are arbitrary length utf-8 strings, so technically the hosts can define an empty module instance (zero length utf-8), and export the empty name entries (e.g. function).
agreed it's better to just rule out the empty names for now as it seems to require syntax changes as opposed to the practical use cases.

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

@mathetake

well, what if two different packages import the functions with different signatures but with the same import names?

Since there will be a single signature eventually available at runtime, the program will likely fail at runtime when resolving imports.
The compiler could ensure that all uses of go:wasmimport are consistent, but then they could also be all invalid or incompatible with the host functions available at runtime. I'm not arguing that we shouldn't add this type of validation, just reflecting on the limits of what can be verified at compile time.
Thanks for raising this point!

Agree that it's ok to defer this error checking to the WASM runtime if other toolchains do the same. As https://github.com/golang/go/issues/59149#issuecomment-1509939837 says, we can only check a subset of issues at compile time anyway. Runtime mismatches between imported symbols and actual host signatures are likely a much common problem, and this is something the compiler can't do anything about.

Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group


demultiplexing

Folie can treat received data as coming from multiple streams.
A simple approach would be to use \<n> to switch to stream <n> (n = 1..5, perhaps).
Can only work if received data is plain text, never binary data containing one of these codes.

Consider using SLIP escape sequence, which supports full 8-bit binary pass-through.

There's some overlap with #6 (telnet escapes): from folie to µC needs out-of-band info to control the DTR, RTS, and parity settings for firmware uploaing, while from µC to folie needs out of band info to switch streams and indicate which thread the following data is coming from.
It's a bit odd to use telnet escapes one way and slip the other. No clear solution comes to mind.

Yes, I think SLIP is the way to go. It needs to escape $C0 and $DB, but apart from that it's 100% transparent - and best of all: when not used, the remote end doesn't have to be SLIP-aware.

The question is how to demultiplex, and what to do with those streams. I'm trying to keep Folie as state- and config-less as possible (so that it always behaves the same, in any project).
One idea would to publish text preceded by <$DB><n> to an MQTT topic (perhaps /folie/stream<n> on localhost as default).
But that's not really a good match: MQTT is not really character-stream-based, and this approach would not have any packet structure, it would simply continue to stream to that topic until a new <$DB><n2> marker is received, switching it to stream n2. This introduces state across host and remote - probably not a good idea (i.e. on remote: "where will my next output be going?").
I'd like to allow each task in a Forth-based multi-tasking app to send out whatever text it wants to the console with emit, type, etc - with a lower layer detecting the task, and inserting stream-switching escapes automatically as needed.
The <$DB><n> mechanism in itself would be enough for that, but then what? I don't want to implement a "stream management" user interface in Folie, where you say: stream 1 goes to the console, stream 2 goes to filex, stream 3 gets sent to a net socket, stream 4 goes to a named pipe, etc.

Hey, @tve and @tht, maybe you have some thoughts about the above https://github.com/jeelabs/folie/issues/7#issuecomment-274183474 ?

Does SLIP not allow the endpoint to use IP socket numbers  and then folie to pass the socket stream to the routed by the OS?

Ah, no , I'm not transferring TCP/IP packets here - just using SLIP as serial line encoding mechanism, to be able to delimit packet boundaries and send some other messages. Doing TCP/IP would require implementing an TCP/IP stack on the remote end.

I'm not sure what the use-cases here are. Lemme try:

multiple tasks printing debug info, would be nice to pretty-print it, i.e., ensure it doesn't get completely garbled
input directed at the REPL
input directed at any task
tasks outputting MQTT messages
incoming MQTT messages forwarded to tasks
other stream coming out of a task being forwarded on some unix file descriptor (file, pipe, socket, ...)
other stream coming in from some unix FD being forwarded to a task
???

Which of these do you want to solve in folie? ;-)
Here's a stab:

each SLIP pkt has a stream ID, chars outside of a packet are assigned stream ID 0
each stream has a type that gets set via explicit command in the stream
task "std" output gets assigned a line-buffered type
repl output gets assigned either a line-buffered or special repl type
folie uses type to pretty-print or route output
if a task wants to output mqtt, it has to change the type of its output or create a new stream and set its type
stdin goes to the repl stream
the mqtt stream type supports commands, such as subscribe
add as much additional complexity as you feel compelled to... ;-)

Thx. Now I need to think about whether tasks are the best way forward - they are relatively expensive in terms of RAM use. Another option would be to have two or three fixed tasks, not per-app: one for the REPL, one for background activity, timer management, maybe even GC, who knows. Possibly a third task for application-specific (foreground?) use.

Enumerating multiple comm ports seems like a  reasonably hardware independent approach, prior art in the BMP code.

Which of these do you want to solve in folie? ;-)

My main goal is to keep a responsive command interpreter as much as possible, even when "the app" is doing stuff in the background. Very much like the JS console in the browser or in Node (or Tcl, for that matter). That's essentially just another way of saying that things need to be event-driven.
At 500+ bytes of RAM per task (with 64-word data and return stacks), things could get a bit cramped on an STML052 if everything ends up being a separate task. I think I'd much rather have JavaScript's setTimeout and setInterval callback mechanism, that should be sufficient for most repetitive remote node activities, as well as to drive a PureData-like engine.
I think I'll go for two tasks: a foreground task being the Mecrisp command interpreter, and a background task for everything else. With multi.fs it's easy to implement those, and to enable/disable the background activity while fooling around with the code. It's also very easy to see which of the two tasks is running (: fg? up @ boot-task = ;). That in turn, allows implementing a version of emit which injects a "task prefix byte" into the output stream when the output is not coming from the same task as before.
So my plan now is to insert CTRL-F and CTRL-B bytes into the output stream, and to de-multiplex this in Folie. This does assume that binary data is never emitted. Not sure yet if that's good enough - it could be solved by using SLIP or Telnet escapes as markers instead.
A simple modified hook-emit word would be all that's needed to implement this: it compares fg? to the last known state and injects a prefix whenever it has changed.
The result: Folie always sends text assuming that it will be consumed by the Mecrisp (foreground) command interpreter, and splits all incoming data (i.e. outbound from Mecrisp) into two streams - one is treated as responses from Mecrisp, i.e. command echo and replies to command requests, the other is for asynchronous "events", and could be sent into MQTT (one event per line) for example.
This is about as simple as I can make it to get an async Node-like command prompt, I think.

FWIW, none of the above strikes me as particularly elegant. Yet another ad-hoc mechanism...

Are we certain that enumerating a second USB logical port would not be more open other user level code. Are we fixed to using serial, might not a more binary type interface be interesting.

Yes, you have a point. I was going to say that this won't solve the issue for when a serial rf-to-rf console gets implemented, but I now realise that the node could easily be made to talk in two different ways (header bits, or net group) to separate app comms from the interactive prompt.
I'm not looking forward to the prospect of adding stuff to the suf/usb.fs code, but that's just me. If there's a way to do it (like the Black Magic Probe's two serial ports), it could be worthwhile after all.


Support launch as Non-Root user in all published container images

This is a more generic requirement to all the container images created here. Many Kubernetes clouds have security standard policy that doesn't allow running as root in the container image. So we think it'd be better that the container images that released here, just like the image embedding-tei did, to add the support of running the image as not-root user(normally would be uid 1000).
Normally, we can add the following into the very top of the Dockerfile, i.e.
FROM xxx:yy

# Create a custom user with UID 1000 and GID 1000
RUN groupadd -g 1000 user && useradd -m -u 1000 -g user user

USER user

<rest of your Dockerfile>

The following is a list of Dockerfile which needs to add this support besides ASR/Whisper which is mentioned in issue #300:
$ grep -Lr USER $(find -name "Docker*")
./comps/lvms/llava/Dockerfile_hpu
./comps/lvms/llava/Dockerfile
./comps/lvms/Dockerfile
./comps/tts/speecht5/Dockerfile_hpu
./comps/tts/speecht5/Dockerfile
./comps/tts/Dockerfile
./comps/reranks/langchain-mosec/mosec-docker/Dockerfile
./comps/embeddings/langchain-mosec/mosec-docker/Dockerfile
./comps/web_retrievers/langchain/chroma/docker/Dockerfile
./comps/llms/utils/lm-eval/Dockerfile.cpu
./comps/llms/text-generation/vllm/docker/Dockerfile.hpu
./comps/llms/text-generation/vllm-xft/docker/Dockerfile
./comps/llms/text-generation/ray_serve/docker/Dockerfile.rayserve
./comps/llms/text-generation/vllm-ray/docker/Dockerfile.vllmray
./comps/asr/Dockerfile
./comps/asr/whisper/Dockerfile_hpu
./comps/asr/whisper/Dockerfile

Hi @ZePan110, I drafted #330 per a previous request (#300) adding non-root user for ASR & whisper dockerfiles only. If you have planned to provide a unified update for all the dockerfiles, I'll close my PR.

@ZailiWang  OK. You can close the PR, I am currently doing this.

The issue is solved by https://github.com/opea-project/GenAIComps/pull/406 and https://github.com/opea-project/GenAIComps/pull/460


Add Huggingface SmolLM models

Huggingface SmolLM would be a great addition to Candle. From the config, it's similar to LLama-based models. Can we use it directly with the candle quantisation example of LLAma/Tinyllama? if not I can give it a try.
It's also a good candidate for Wasm/Webgpu for client-side browser inferencing.
https://huggingface.co/collections/HuggingFaceTB/smollm2-6723884218bcda64b34d7db9

I think it should actually work out of the box, e.g. with:
cargo run --features cuda --profile=release-with-debug --example llama -- --model-id HuggingFaceTB/SmolLM2-1.7B --which v32-1b

(note that it's an unquantised version though)


bower home $modulename does not open $modulename homepage

As in the title.
$ bower search XMLHttpRequest
Search results:

    FakeXMLHttpRequest git://github.com/trek/FakeXMLHttpRequest.git
    xmlhttprequest git://github.com/ilinsky/xmlhttprequest.git

$ bower home FakeXMLHttpRequest
bower FakeXMLHttpRequest#*  not-cached https://github.com/trek/FakeXMLHttpRequest.git#*
bower FakeXMLHttpRequest#*     resolve https://github.com/trek/FakeXMLHttpRequest.git#*
bower FakeXMLHttpRequest#*    download https://github.com/trek/FakeXMLHttpRequest/archive/1.3.0.tar.gz
bower FakeXMLHttpRequest#*     extract archive.tar.gz
bower FakeXMLHttpRequest#*     invalid-meta fake-xml-http-request is missing "ignore" entry in bower.json
bower FakeXMLHttpRequest#*         resolved https://github.com/trek/FakeXMLHttpRequest.git#1.3.0

$ bower help

Usage:

    bower <command> [<args>] [<options>]
Commands:

    cache                   Manage bower cache
    help                    Display help information about Bower
    home                    Opens a package homepage into your favorite browser
   ...

Hi @leegee , I tried the same. bower home FakeXMLHttpRequest opens https://github.com/pretenderjs/FakeXMLHttpRequest in browser. what bower version are you are using?

Thanks, @rajzshkr . I'm sorry I can't tell you --- I'm on Windows 10, rather than my usual OSX or Linux. I have no idea how to find out anything on Windows: it's horrible.
My default is set to Chrome (latest), but sometimes Windows seems to prefer to use Edge (no idea why). I also have the latest Firefox.
If I can help, please let me know.

@leegee he meant/said bower version, not browser ;)
you can find out with bower -v in your terminal.

@rajzshkr lol sorry about that -- you can imagine what fun I have debugging with eyesight like this...!
$ bower -v
1.7.7

I've updated opn package to 4.0.0. I hope it'll fix issue on Windows (did it ever work)?


refactor: Disallow cancelled 'Integration Requests' on checkout page

part of: https://github.com/frappe/erpnext/pull/43689

needs backporting to v15


Kwlt 6 ability to export private keys

This PR:

Adds functionality to show a modal, confirm accountId and view private key
Adds an icon to the "Remove Account from Wallet" button

Figma link

This is looking great @esaminu! Not a blocker but just something I noticed / thought of:

Should we specify that the key is the 'key on this machine' and not the key from the users recovery method? Seems one might assume it's the recovery method key otherwise
Would be nice with ability to copy the key. Simple thing would be to just wrap the key div in a ClickToCopy component.

cc @heycorwin


add experimental hybrid based on arsd master

I slapped together this hybrid after thinking about it a bit in the blog Friday. It isn't fully tested but does OK so far so I thought it might be fun to add here and see how it stacks up.
I also fixed an old buffering bug in my write function that harmed performance across the board (it did buffering with the concat operator, already an inefficient way to do it.... but then ignored the buffer and wrote the pieces in a loop anyway, leading to like 8 syscalls when only one was necessary! This killed performance in the hello world case, fixing that helped the existing things a lot too.)
I'll be quite amused if it moves up to the middle of the pack with less than one day's work and no api change :)

BTW the hybrid just is the thread server with -version=cgi_use_fiber and -version=cgi_use_fork combined.
So it starts as a thread server then forks off a copy of itself. The reason for this is while one copy is stopped doing a GC collection, the other can keep going so the back queue doesn't get as long (it really only helps the rare top 0.1% of cases but still).
Then the thread server, with -version=cgi_use_thread, instead of sending connections off to worker threads to process directly, instead sends connections off to worker fibers. The worker threads then pick up and resume worker fibers as they become ready. (this means the fibers can jump around threads!)
Not actually a whole lot of code to implement this but there's a lot of risk of breakage so I'm not quite ready to formally release yet, hence the dependency on ~master instead of a numbered version.

Hi Adam, thanks for this.
I'll try to rerun the tests over the night (as it takes pretty long already) with other changes too and will post the results tomorrow (hopefully).
Only thing I'd like to make sure of is the test category of this. As currently I've only published singleCore tests and marked arsd singleCore too even though it's really not (as it's using fork or multiple threads). But as it hasn't used epoll/io_uring it seems fair enough to me to mark it as that and compare this traditional approach with the modern async event loops.
But with this change on the first look I'm not so sure if it still holds that, and I'm more inclined to multiCore category here.
I should probably modify other tests to return same total response size as arsd as currently it returns the largest response and will be affected by it in a negative way.

On Mon, Sep 28, 2020 at 10:20:01AM -0700, Tomáš Chaloupka wrote:

Only thing I'd like to make sure of is the test category of this. As currently I've only published singleCore tests and marked arsd singleCore too even though it's really not (as it's using fork or multiple threads).

Yeah, I was kinda curious about that too. I searched your code and
didn't even really understand what the difference was.
If anything, I'd probably call epoll single core (though it can
obviously be spread across worker threads too)...

I should probably modify other tests to return same total response size as arsd as currently it returns the largest response and will be affected by it in a negative way.

This might have changed since part of the buffer bug was it sending a
bit more than it needed to.
But the other changes might be my default header set - I also put out no
cache headers and such. See:
$ curl -v http://localhost:4000

Trying <IP_ADDRESS>:4000...
TCP_NODELAY set
Connected to localhost (<IP_ADDRESS>) port 4000 (#0)

GET / HTTP/1.1
Host: localhost:4000
User-Agent: curl/7.66.0
Accept: /

Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Mon, 28 Sep 2020 17:40:58 GMT
< Cache-Control: private, no-cache="set-cookie"
< Expires: 0
< Pragma: no-cache
< Content-Type: text/html; charset=utf-8
< Content-Length: 13
<
Connection #0 to host localhost left intact
Hello, world!

That combination is probably obsolete - it made a difference back with
IE6, this is an old library! - but still it was a sane default for a
long time and doesn't hurt much really. Especially with the buffer bug
fixed so it all goes into one tcp packet again now.

Ok, so I've got all tests to output 192B for the whole response (including header).
About single/multiCore.
My idea was to primarily test performance when run on the single core and single thread (as all but arsd used epoll/io_uring loops in a single thread). This is also easier to test on the localhost as load generator won't interfere too much with the test.

C epoll, uring, nginx - all are using just one worker process with single thread event loop (nginx has a variant for multiCore too)
D - uring, epoll, eventcore, hunt, lighttp, vibe-core, vibe-d - all uses only single thread event loop

Hunt - well with this I'm not so sure, using ioThreadSize to set 1 worker, but hunt-http is too slow and hunt-pico pretty fast - seems odd

D - photon - I should investigate this and probably patch it as it seems to work with multiple threads based on this
.Net Core - should work with one thread (options.IOQueueCount = threadCount)
Go - using GOMAXPROCS to set workers number
Rust - using workers setter to set to single worker

I would like to add generic prefork runner that'll just fork multiple processes (and pin them to the available cores) for normally just single thread tests with shared port and let the others use their own multi core scheduler.
I haven't seen this scenario tested in other http benchmarks, but it seems to me that this is too an interesting scenario to see how it behaves ie without shedulers to pass work between threads - especially with just one thread epoll/io_uring event loops.

On Mon, Sep 28, 2020 at 12:10:11PM -0700, Tomáš Chaloupka wrote:

About single/multiCore.
My idea was to primarily test performance when run on the single core and single thread (as all but arsd used epoll/io_uring loops in a single thread). This is also easier to test on the localhost as load generator won't interfere too much with the test.

Yeah, I have ways to do it for debugging purposes too:
-version=cgi_no_threads
with the thread thing and a static ctor/custom main that changes processPoolSize
for processes. (I think I'll document that for the formal release and
put it in the new config struct so it is easier to set. Then do
something similar for number of worker threads as well.)
But the only documentation these have is // NEVER USE THIS because it
is kinda terrible for any actual use.
Ruby on Rails' godawful default WebBRICK is single threaded like this
too and it is actually really easy to lock it up, even when just testing
code locally. (That awful thing will outright deadlock if you have a
thing that issues a request to itself which I learned the hard way last
year trying to test some trivial code. Pretty sure at least the vibe
style event loop can handle that case, but it is still prone to problems
if you call any blocking function, whether third party i/o or just slow
cpu crunching code. This is why I was so hesitant to go event at all for
the core server, it must be done very carefully to actually work as well
as it claims and most user code simply isn't that careful. And even when
it is, you miss out on most libraries that aren't that careful.)

I would like to add generic prefork runner that'll just fork multiple processes (and pin them to the available cores) for normally just single thread tests with shared port and let the others use their own multi core scheduler.

You might hit "address already in use" problems depending on how they
bind. It is best to fork after listening so they can share that.
And on Windows, of course, you don't have fork() at all... you must
thread and/or fiber there. (It's built in thread and fiber facilities
are excellent though.)
If the framework doesn't offer some built-in facility to handle this
itself, I'd put that as a very serious negative, unless, of course, it
uses CGI (or its simple/fast variants), which explicitly are designed to
allow pushing it off to another layer :)

I haven't seen this scenario tested in other http benchmarks, but it seems to me that this is too an interesting scenario to see how it behaves ie without shedulers to pass work between threads - especially with just one thread epoll/io_uring event loops.

Aye. Kinda interesting to test simulated slow connections with larger
responses and slow code too, just to see how it is handled. (gzipping
larger responses is very useful!) One slow user or endpoint shouldn't
break others. Even a slow worker thread in an async/await type situation
shouldn't affect the latency of other users - thread prioritization
might be needed.
At some point you might just say "well fix your code" but there is value
in a framework being friendlier too.

It would be great If we can make a new hybrid configuration in both variants.
I've run just a fast check with 8 clients on my main rig (8core 16 threads AMD processor) and:

Language
Framework
Category
Name
Req
RPS
BPS
max
50%
75%
90%
99%

dlang
photon
micro

42322122
352684
67715395
22.82
0.019
0.023
0.03
1.21

dlang
arsd
platform
processes
41127973
342733
65804756
9.35
0.021
0.026
0.028
0.053

c
epoll
micro
raw - lvl
23457862
195482
37532579
3.4
0.035
0.04
0.065
0.077

c
io_uring
micro
raw
22295220
185793
35672352
2.86
0.042
0.045
0.046
0.051

c
epoll
micro
raw - edge
22277576
185646
35644121
2.75
0.042
0.044
0.046
0.051

c
nginx
platform

17275853
143965
27641364
8.67
0.055
0.057
0.06
0.066

dlang
arsd
platform
hybrid
8235759
68631
13177214
13.31
0.028
0.89
1.9
4.35

dlang
arsd
platform
threads
7986066
66550
12777705
13
0.027
0.837
1.66
3.91

Run with: _suite/runner.d bench --tool wrk -c 8 -d 120 --type singleCore arsd c/ photon
It's another evidence that photon isn't just a single core one.
And arsd/processes is on top probably just because it scales between cores and so all the HTTP parsing, routing, etc is done in other core for each client but for raw tests it's all handled in the single core (well basically raw tests don't handle anything related to HTTP, they are just checking that the received data ends with \r\n\r\n and send data from prepared buffer - that's why it's suspicious if some other test beats these).
But still pretty high above hybrid/threads variants.

Yeah, that buffering bug was killing my processes performance before. Fixing that made a huge difference, turned 8 syscalls into 1. (Though it still has the downside of a slow / idle connection of being able to monopolize a worker. I'd kinda love if I can dynamically detect that and spawn new workers... but idk).
The beauty of prefork processes is the Linux kernel does basically all the work and it does it in very few context switches. When you call blocking accept/send/recv, you're added to the kernel's event loop and then you're notified when the operation is done. One system call, one context switch. Furthermore the userland is simplified since there's no dynamic allocations, no fiber state management, etc.
With my hybrid approach, there's four system calls to do the same thing: receive, see it would block and epoll_ctrl, epoll_wait until it is free, then receive again to retry. Accept is similar but include the fiber setup.
Even with the pure single core model like vibe (I'm pretty sure anyway) uses, there's still at least two: epoll_wait, then some userland processing before you receive again. System calls are expensive when you aren't doing much else.
And the kernel itself is a common baseline for all of them so it is hard to beat just letting it do 90% of the work!
Add in the crash protection of processes and library compatibility and they're legitimately hard to beat (on Linux anyway).... until you have a huge number of mostly-idle connections. That's where the userspace event systems shine.


Redirect image dependencies to "old_io" branch

The image library will transition to the new standard library io design on the master branch, and has created an "old_io" branch to keep existing projects running.
Put a branch = "old_io" in the Cargo.toml, for example:
[dependencies.image]
git = "https://github.com/pistondevelopers/image"
default-features = false
branch = "old_io"

This was never done because everything got ported over to the new io.


[9.x] Wrong directory permission

Laravel Version: 9.0.0-beta.3
PHP Version: 8.1.1

Description:
Storage::makeDirectory('test-dir') create directory with 0700 permission
It should be 0755
Steps To Reproduce:

php artisan tinker
\Illuminate\Support\Facades\Storage::makeDirectory('test-dir')
exit
stat storage/app/test-dir
Access: (0700/drwx------)

Ok,
It seems that default file visibility changed to private in laravel 9
I just didn't touch my filesystems.php config file since laravel 7
I just added 'visibility' => 'public' to filesystems.disks.local


add client.BareDo(ctx context.Context, req *http.Request) (*Response, error) func to allow user to handle the body

This allows the user to also treat bodies as plain go streams.
For more flexibility.
~Before this, passing a nil body would do the request but close the~
~body. Now it is up to the caller to close resp.Body.~
~I mainly wanted to be able to do this to stream bodies and to re-use the same HTTP client in order to download release files.~
Edit:
This PR enables using the client as an http client with Github Specific error handling and throttling management.

Hmmm... I'm concerned that this change might be too disruptive for users of this package.  If I understand correctly, it would force all users to add code to manually close the body.

Hmmm... I'm concerned that this change might be too disruptive for users of this package.  If I understand correctly, it would force all users to add code to manually close the body.

I had the same thought... this seems like a pretty risky change.

I had the same thought... this seems like a pretty risky change.

Yes, that is correct ! Is that an issue even for v34 ?

Yes, that is correct ! Is that an issue even for v34 ?

To be precise it would force users that do nothing with the response's body.

To be precise it would force users that do nothing with the response's body.

Yes, that is correct ! Is that an issue even for v34 ?

Yes. While bumping version numbers is indeed used for breaking API changes, this change I feel is too disruptive for the majority of users of this package to be generally useful.
I'm happy to leave this issue/PR open for discussion, but if there is not much interest in it, I think we will most likely close it.
Alternatively, if you could come up with a PR that would default to existing behavior but as an option people could enable your proposed new behavior, that might be more palatable for inclusion in this repo.

Yes, that is correct ! Is that an issue even for v34 ?

Yes. While bumping version numbers is indeed used for breaking API changes, this change I feel is too disruptive for the majority of users of this package to be generally useful.
I'm happy to leave this issue/PR open for discussion, but if there is not much interest in it, I think we will most likely close it.
Alternatively, if you could come up with a PR that would default to existing behavior but as an option people could enable your proposed new behavior, that might be more palatable for inclusion in this repo.

Hey @gmlewis,

this change I feel is too disruptive for the majority of users of this package to be generally useful.

Gotcha, I understand 🙂 . The api doesn't change so unless you read the changelog... Its an easy trap.

but if there is not much interest in it, I think we will most likely close it.

After some thinking I still would like to reuse the same client to download binaries from Github. In case they start adding api limitations or something.  So I still would like to make this work. Also I like to handle the body like if I was doing a plain http request.
I was thinking of adding a new function that looks like so:
// ___ Is just like Do but will let you handle the body.
// Make sure to close the Response Body.
func (c *Client) ___(ctx context.Context, req *http.Request) (*Response, error) {

In terms of name I was thinking of something like BareDo ? DoWithContext ( meh ).
I'm also happy to implement something with your idea:
So allowing to call:
client.Do(ctx, req, github.LeaveBody)

Ah naming is hard :D Do you have a suggestion there ?
( I like my suggestion a tad more, because it feels simple and sort of godoc friendly, I will implement it to gain some time but feel free to make me change it ).

Hey @gmlewis,

this change I feel is too disruptive for the majority of users of this package to be generally useful.

Gotcha, I understand 🙂 . The api doesn't change so unless you read the changelog... Its an easy trap.

but if there is not much interest in it, I think we will most likely close it.

After some thinking I still would like to reuse the same client to download binaries from Github. In case they start adding api limitations or something.  So I still would like to make this work. Also I like to handle the body like if I was doing a plain http request.
I was thinking of adding a new function that looks like so:
// ___ Is just like Do but will let you handle the body.
// Make sure to close the Response Body.
func (c *Client) ___(ctx context.Context, req *http.Request) (*Response, error) {

In terms of name I was thinking of something like BareDo ? DoWithContext ( meh ).
I'm also happy to implement something with your idea:
So allowing to call:
client.Do(ctx, req, github.LeaveBody)

Ah naming is hard :D Do you have a suggestion there ?
( I like my suggestion a tad more, because it feels simple and sort of godoc friendly, I will implement it to gain some time but feel free to make me change it ).

Hello there, commenting here just to say that this works for me. I had a tiny issue where the auth token from github was bein set again to a request done to aws which was 400'ing because it want only one auth token, I was able to fix this using the following code:
fuunc something() {
	ts := oauth2.StaticTokenSource(
		&oauth2.Token{AccessToken: tk},
	)
	tc = &http.Client{
		Transport: &HostSpecificTokenAuthTransport{
			TokenSources: map[string]oauth2.TokenSource{
				"api.github.com": ts,
			},
		},
	}
}

type HostSpecificTokenAuthTransport struct {
	TokenSources map[string]oauth2.TokenSource

	// Transport is the underlying HTTP transport to use when making requests.
	// It will default to http.DefaultTransport if nil.
	Transport http.RoundTripper

	Base http.RoundTripper
}

// RoundTrip authorizes and authenticates the request with an
// access token from Transport's Source.
func (t *HostSpecificTokenAuthTransport) RoundTrip(req *http.Request) (*http.Response, error) {
	source, found := t.TokenSources[req.Host]
	if found {
		reqBodyClosed := false
		if req.Body != nil {
			defer func() {
				if !reqBodyClosed {
					req.Body.Close()
				}
			}()
		}

		if source == nil {
			return nil, errors.New("transport's Source is nil")
		}
		token, err := source.Token()
		if err != nil {
			return nil, err
		}

		token.SetAuthHeader(req)

		// req.Body is assumed to be closed by the base RoundTripper.
		reqBodyClosed = true
	}

	return t.base().RoundTrip(req)
}

func (t *HostSpecificTokenAuthTransport) base() http.RoundTripper {
	if t.Base != nil {
		return t.Base
	}
	return http.DefaultTransport
}

This seems to be related with #804 and #246 would you like me to contribute this change here ?

Hello there, commenting here just to say that this works for me. I had a tiny issue where the auth token from github was bein set again to a request done to aws which was 400'ing because it want only one auth token, I was able to fix this using the following code:
fuunc something() {
	ts := oauth2.StaticTokenSource(
		&oauth2.Token{AccessToken: tk},
	)
	tc = &http.Client{
		Transport: &HostSpecificTokenAuthTransport{
			TokenSources: map[string]oauth2.TokenSource{
				"api.github.com": ts,
			},
		},
	}
}

type HostSpecificTokenAuthTransport struct {
	TokenSources map[string]oauth2.TokenSource

	// Transport is the underlying HTTP transport to use when making requests.
	// It will default to http.DefaultTransport if nil.
	Transport http.RoundTripper

	Base http.RoundTripper
}

// RoundTrip authorizes and authenticates the request with an
// access token from Transport's Source.
func (t *HostSpecificTokenAuthTransport) RoundTrip(req *http.Request) (*http.Response, error) {
	source, found := t.TokenSources[req.Host]
	if found {
		reqBodyClosed := false
		if req.Body != nil {
			defer func() {
				if !reqBodyClosed {
					req.Body.Close()
				}
			}()
		}

		if source == nil {
			return nil, errors.New("transport's Source is nil")
		}
		token, err := source.Token()
		if err != nil {
			return nil, err
		}

		token.SetAuthHeader(req)

		// req.Body is assumed to be closed by the base RoundTripper.
		reqBodyClosed = true
	}

	return t.base().RoundTrip(req)
}

func (t *HostSpecificTokenAuthTransport) base() http.RoundTripper {
	if t.Base != nil {
		return t.Base
	}
	return http.DefaultTransport
}

This seems to be related with #804 and #246 would you like me to contribute this change here ?

I'm not sure I fully understand how you expect BareDo to be used since most (all?) endpoints in this repo call Do directly.
Do you expect users of this library to call it directly from their own code?

I'm not sure I fully understand how you expect BareDo to be used since most (all?) endpoints in this repo call Do directly.
Do you expect users of this library to call it directly from their own code?

That's what I'd like to do, yes. I need all the nice error handling and throttling management but would like to handle the bodies myself, just like if I was using an plain http.Client that knows a little more about the Github API. It makes this library even more powerful to me. Because it'd be sort of replaceable.

That's what I'd like to do, yes. I need all the nice error handling and throttling management but would like to handle the bodies myself, just like if I was using an plain http.Client that knows a little more about the Github API. It makes this library even more powerful to me. Because it'd be sort of replaceable.

Okay I updated the PR Thanks for reviewing 🙂

Okay I updated the PR Thanks for reviewing 🙂

Good idea, I added a func to test this 🙂

Good idea, I added a func to test this 🙂

Thank you, @wesleimp !
Merging.

Thank you, @wesleimp !
Merging.

Nice ! :D thanks for your time !

Nice ! :D thanks for your time !


Can we close app manually??

Flutter version

Flutter 1.8.2 • channel master • https://github.com/flutter/flutter.git
Framework • revision xxxxxxxxxx (x weeks ago) • xxxxxxxxxxxxxxxxxxxxxxxx
Engine • revision xxxxxxxxxx
Tools • Dart 2.x.x (build xxxxxxxxxx)

It is not an issue. If we want to close app manually in flutter like navigator.pop how can we achieve in go flutter. Thanks Regards.

flutter.PopBehavior(flutter.PopBehaviorClose)


Warning [parser.cc:779] XML Element[instanced_actor]

Hello!Firstly thanks for your wonderful job!
I have build sdf provided by you.However,when I launched the node with the libmenge_plugin_randomized.so, there is a problem that "Warning [parser.cc:779] XML Element[instanced_actor], child of element[sdf] not defined in SDF. Ignoring[instanced_actor]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
 ".
I have check the sdf,and there already is a instance_actor.sdf.
So could you tell me how you build the sdformat?
Thanks for any advices!

Hi,
I am also facing similar problems. Were you able to solve this?


IDF Monitor refers to host source (IDFGH-2430)

see https://github.com/espressif/esp-idf/issues/4541

0x40085fbd: strlen at /builds/idf/crosstool-NG/.build/HOST-x86_64-apple-darwin12/xtensa-esp32-elf/src/newlib/newlib/libc/machine/xtensa/strlen.S:84
0x40085fcd: strlen at /builds/idf/crosstool-NG/.build/HOST-x86_64-apple-darwin12/xtensa-esp32-elf/src/newlib/newlib/libc/machine/xtensa/strlen.S:96

It's the ".build/HOST-x86_64-apple-darwin12" part that worries me. I'm not an expert on crosstool-NG, but I believe we're dealing with target libraries here.

In case this is normal, still doesn't invalidate #4541

crosstool-NG puts the build directory into ${CT_WORK_DIR}/${CT_HOST:+HOST-${CT_HOST}/}${CT_TARGET}, which makes the build directory specific for the given host-target combination (https://github.com/espressif/crosstool-NG/blob/72ba393991f2eac2df3dbd3207d4eec79b8c42a3/config/global/paths.in#L45). The source code of GCC and newlib is extracted into this directory and patched. When newlib is compiled, the directory name gets embedded into the debug information.
I suppose it would be more clear if the default was ${CT_WORK_DIR}/${CT_HOST:+HOST-${CT_HOST}/}TARGET-${CT_TARGET}, resulting in a directory like .build/HOST-x86_64-apple-darwin12/TARGET-xtensa-esp32-elf/. But this option is not configurable, so we can't override it at build time.


Too many different -Scope parameters

Currently there are several different uses of -Scope as a parameter in core PowerShell modules:

As a string to identify a PowerShell invocation scope (e.g. when using cmdlets that work with aliases, variables, PSDrives, and modules)
As a string to identify the profile to work with (e.g. when working with Package Management commands, you can specify CurrentUser or AllUsers, and now when working with experimental features)
As an ExecutionPolicyScope enumeration to identify the execution policy scope (e.g. when working with the *-ExecutionPolicy cmdlets)

Personally I've found it confusing when I see that parameter name, assume it means one thing, and then find out in the documentation it means something else. Before this confusion continues, especially within core PowerShell modules, perhaps it would be better if at least some of these used a more explicit parameter name, with -Scope as an alias for backward compatibility.
For example, in package management and experimental features, -UserScope would be more appropriate than simply using -Scope. For *-ExecutionPolicy cmdlets, -PolicyScope would be more appropriate and accurate than simply using -Scope.
Thoughts?

Given these are parameters, we can change the parameter name and assign the alias Scope to each of these both as a shorthand and for backwards compatibility. I see no reason not to clarify the meaning of these parameters, it's something that's bothered me for quite a while.

@KirkMunro

-UserScope would be more appropriate than simply using -Scope.

Interesting. Personally I find -Scope user much more readable than -UserScope. Also, as a parameter argument, you can easily pass it as a value without having to splat.
@vexx32

we can change the parameter name and assign the alias Scope to

You can't use aliases like that. You can't alias -UserScope to -Scope User.

see no reason not to clarify the meaning of these parameters,

That's what documentation is for. Is there a problem with the current docs? If so, please open a docs bug. Thanks.

Hmm. It's not clear, but I read it as saying "we should use -UserScope as a direct replacement for -Scope for that cmdlet; for example -UserScope User or -UserScope System (or whatever that one is, I gotta say I don't install a whole lot system wide).
The name might want to be changed if that is the case, though... maybe -InstallScope or something.

@bpayette That's not what I meant.
My issue is that there are three distinct parameter types for -Scope parameters, and when I see -Scope I need to go to the docs to see what it's for.
If the *-ExecutionPolicy cmdlets instead had a -PolicyScope parameter (with an alias of Scope for backward compatibility), and if the PowerShellGet/PackageManagement cmdlets and the *-ExperimentalFeature cmdlets instead had a -UserScope parameter (with an alias of Scope for backward compatibility), then the parameters would be more self-explanatory.
Does that make more sense? Maybe the part that confused you was the -UserScope parameter name proposal. I suppose it could be called -ProfileScope instead, but having it just called -Scope by default with values of CurrentUser and AllUsers when other cmdlets in PowerShell have -Scope parameters that take values such as Global, Local, Script, or a numeric value is a confusion point that could be eliminated.
Case in point: when I was looking at *-ExperimentalFeature cmdlets and I saw -Scope, I initially presumed that it was referring to a PowerShell invocation scope where you could enable the feature, and didn't realize it was about enabling those features for the current user or for all users on the local system instead.

Historical reasoning:
Set-ExecutionPolicy -scope ...

Scope in this case is clearly qualified by the noun so -PolicyScope is redundant
Set-Variable -Scope ...

Again, -Scope is clearly qualified by the noun making -VariableScope redundant.
And then it all goes off the rails with Install-Module:
Install-Module -Scope ...
Import-Module -Scope ...

This is extremely unfortunate because here the qualifying noun doesn't provide sufficient context. Maybe something like
Install-Module -Repository ...

would be more semantically significant.

take values such as Global, Local, Script, or a numeric value is a confusion point that could be eliminated.

For Import-Module -Scope ... , Get-Variable -Scope ... and $global:foo = ... , in all cases, the word "scope" is referring to exactly the same thing: the activation stack of the currently running script.  The use of numeric values allows more specific non-local references than "script" or "global".

with *-ExperimentalFeature ... thought it was referring to a PowerShell invocation scope where you could enable the feature, and didn't realize it was about enabling those features for the current user or for all users on the local system instead

I totally get that. This should probably just be fixed. Maybe -EnableFor ? Or just -For ? Install-Module -For CurrentUser would work too as would Set-ExecutionPolicy -For process.

I ❤ the -For parameter name suggestion. That's very expressive! I've updated an RFC I have open proposing optional features to use that parameter name because it makes the commands self-documenting.
I also like that parameter name for *-ExecutionPolicy, *-ExperimentalFeature, and the PowerShellGet/PackageManagement commands. Thanks for that suggestion @bpayette!

@SteveL-MSFT: This should be a fairly quick decision to make for or against this in the Committee. Could you table this at a meeting? If the Committee buys in, I'll submit the PRs.
The tl;dr of this is:
PowerShell uses "scope" to refer to the activation stack of the currently running script. Because of that, a -Scope parameter name on Import-Module, Get-Variable, etc. makes sense. There are other uses of -Scope as a parameter name that cause confusion: Enable-ExperimentalFeature and Disable-ExperimentalFeature, for example, as well as commands in PowerShellGet/PackageManagement and the Get-/Set-ExecutionPolicy cmdlets. I think we should consider changing the name of -Scope in those commands to something more appropriate, with -Scope as an alias for backwards compatibility. @BrucePay's suggestion of -For is a really good alternative parameter name that makes commands self-documenting in all of those cases.

I'm not a fan of the -For idea because I use  -For interactively for -Force.   Also, while this is "fluent" in nature, a preposition doesn't quite fit the PowerShell idiom for parameter names IMO.  I'm certainly up for changing the parameter name on Enable-ExperimentalFeature.

@rkeithhill Prepositions are already in use as parameter names in PowerShell commands. Some common examples that come to mind are the -From and -To parameters in Send-MailMessage, or the -Before and -After parameters in `Get-Job.

Fair enough.  I guess down the road I'd still like to still be able to recognize the language.  I'm all for additions and strategic changes but changing things that have existed for a decade or more to make it "slightly better" ... yeah, I'm not a fan of that.

Not all of these commands have been there for a decade or more, and among those discussed here, the commands that are a dozen years old are not the ones where changes are being proposed here, aside from Get-/Set-ExecutionPolicy which are usually just fire-and-forget commands.
I look at issues like this as something that wasn't readily apparent early on, but that later became more of a problem due to additions to the language, at which time it makes sense to at least question the current direction, and perhaps change it based on the current landscape of features/commands.

Plus, to speak to the "slightly better" comment and give some background behind where this discussion started, I have an RFC open for commands for use with optional features. In an earlier version of the draft, I was looking at using a -Scope parameter, but in those cmdlets that's very confusing because they have two notions of scope: the users for whom they are enabled/disabled or by default (e.g. CurrentUser or AllUsers) and/or the activation stack in which they are enabled or disabled. It's during the authoring of that RFC that I came into this challenge, and to maintain discoverability and consistency started to question this more broadly.
This is not just a trivial I-have-nothing-better-to-do-with-my-time-so-let's-go-fix-something-that-isn't-important-to-make-something-slightly-better scenario.

I have an RFC open for commands for use with optional features.

And it's a great conversation to have for new commands.  I'm just not so keen on the "let's also change well-established commands for consistency" aspect of this.  BTW I don't think the ExperimentalFeature commands are well-established.

Scope as a parameter to me makes sense & changing this in all the areas seems to not provide all that much benefit whilst adding more unnecessary confusion, especially as now you are moving into the realms of parameter naming guidance, whereas good Parameter Help should be where people go to learn and understand what the parameter means more than it's simple name, along with good examples.
Also Scope is used heavily in Az Commands for a manner of different contextual usecases.
I'm siding with @rkeithhill on this as to me doesn't seem to add value to either new or existing scripters and would in fact likely cause more confusion than anything else.

@PowerShell/powershell-committee reviewed this (@BrucePay included) and we don't feel anything we do here is going to add much value:

Adding a descriptor to -Scope is going to be repetitive with the noun (e.g. -ModuleScope or -InstallationScope for Install-Module doesn't seem helpful)
Where disambiguation is needed between two scope concepts, it's perfectly fine to use a prefix for that disambiguation
-For is a "fluent" parameter that might read better, but it's not a pattern we've used often in PowerShell parameters, and it's not any less ambiguous than -Scope
Tab completion works fine for all the -Scope values across the cmdlets
There was 50/50 agreement in the room between whether using -For for -Force was a valid concern. I personally use -fo interactively every day, and this would drive me crazy (especially on Install-Module).


Add Mongotron project to index.html

This pull-request adds the Mongotron project to the #Built on Electron section of the homepage.

Mongotron is a database-management tool for MongoDB. It's built on Electron and Angular JS.
Website: http://mongotron.io/
Github : https://github.com/officert/mongotron
Logo : http://mongotron.io/src/ui/images/logo.png

Awesome! Thanks for the screenshot :100:


Update datatypes-string-operators.md

Add note about the _cs suffix on operators for clarification.

@lesterw1 : Thanks for your contribution! The author(s) have been notified to review your proposed change.

@lesterw1 thanks so much for your contribution. I'm merging your changes.


It's me again!

Hello Unam! I'm already ashamed to ask you for something, so I apologize in advance for taking up your time. Yesterday I asked about the shellcode, so that's what I needed it for. As I understand it, shellcode can be sewn into macros so that when the macro is activated, it runs. I watched the video and nowhere did I see the use of the binary code that gives me .bin in the manner. Then I started Googling shellcode injection, but there is very little information in my language, so I searched in English and here is the problem: either I didn't understand anything (unlikely), or the information is not what I need. Please help me, show me which way to move and where to look for information. I described the idea above. I also found out about .dll that you can use them to work https://github.com/aslitsecurity/CVE-2021-40444_builders . Then I started looking for injection software and found analog.builder, if you know, you'll understand, but I didn't find anything worthwhile

No that has something to do with encodings.

I have python 3.9 but i thought it has no matter


fix(docker): use debian:buster-slim for slim docker image

Closes #1575

Shall we use the latest stable version bullseye? What version are you using now @Rory-Z

Shall we use the latest stable version bullseye? What version are you using now @Rory-Z

Usually I will use debian:11-slim


Liveonly tests , the cert was in another form in both of them…

…, cleanup the recording
Pull Request Checklist
Please leverage this checklist as a reminder to address commonly occurring feedback when submitting a pull request to make sure your PR can be reviewed quickly:
See the detailed list in the contributing guide.

[ ] C++ Guidelines
[ ] Doxygen docs
[ ] Unit tests
[ ] No unwanted commits/changes
[ ] Descriptive title/description

[ ] PR is single purpose
[ ] Related issue listed

[ ] Comments in source
[ ] No typos
[ ] Update changelog
[ ] Not work-in-progress
[ ] External references or docs updated
[ ] Self review of PR done
[ ] Any breaking changes?

filed https://github.com/Azure/azure-sdk-for-cpp/issues/5522

/azp run cpp - keyvault


apple_tv: Rate limit play status updates

Description:
Currently the apple_tv spams the log with around 5-6 updates per second (while playing) which seems excessive. Maximum of 1 per second seems reasonable since that's the speed media plays at.
I could make it configurable but that seemed like overkill, let me know if that's desired.
Checklist:
If the code communicates with devices, web services, or third-party tools:

[x] Local tests with tox run successfully. Your PR cannot be merged unless tests pass
[x] New dependencies have been added to the REQUIREMENTS variable (example).
[x] New dependencies are only imported inside functions that use them (example).
[x] New dependencies have been added to requirements_all.txt by running script/gen_requirements_all.py.

I would like to know the root cause of all updates, i.e. what are the content of these updates. The PR you propose is not a fix, it's a workaround that might break legitimate updates (since calls with real metadata updates might be dropped). So I would not like to have this merged before we have sorted things out. It also adds an additional dependency which we should avoid (you also missed adding it to REQUIREMENTS).
First off, what updates are you referring to? Is it state updates or just callbacks from the device? Any examples you might share? Which generation of device are you using? Is it the same in all apps or just one/some?
If we are receiving updates that aren't really updates (e.g. not changes to what is playing, but something else), then we should filter those updates in pyatv and not post them to the listener. All events generated by the device should be delivered to listener, but changes to for instance current play time should not.

If you want to see the raw requests, you can enable logging like this:
logger:
  default: info
  logs:
    pyatv.net: debug
    pyatv.daap: debug

Totally agree with your points and acknowledge this is kind of a hack.
Upon doing some testing it looks like it happens in some apps and not others. The Netflix app doesn't really report it's position just playing or not playing, so this issue happens in the YouTube app and iTunes.
Here's some logs from those apps where you can see it updating 5-6 times a second.
 So that is what has to be fixed.


Publish rest of the comic

E03:
Part 01
Part 02
E04:
Part 01
Part 02
E05:
Part 01
Part 02
E06:
Part 01
Part 02
Part 03
E07:
Part 01*
Part 02*
E08:
Part 01*
Part 02*
E09:
Part 01*
Part 02*
E10:
Part 01*
Part 02*
E11:
Part 01*
Part 02*
Part 03*

@andreasgangso created a new object for e03 and a new link in admin but app shows "couldnt load content". what did i miss?
https://github.com/bcc-code/bcc-media-platform/blob/master/web/src/components/comics/Comic.vue


remove never used TopicInfoService in ZKService.

I think TopicInfoService should not appear in this ZKService.

Yeah u r right,but not a good push.thanks for u contribute.


Alex's Shell Backend

Test failing because of differing path in Travis, will fix before merge to master


Introduce the send event once to the IterativeTelemetryLogger class

It will be used by datachain to send events to the telemetry server

Thank you for fixing mypy/lint errors.


Visualization of recalculated aggregates

The expandAllRows feature is very helpful and I've added an auto expand functionality in my app to make sure rows are expanded each time a condition changes i.e. new grouping, filtering, sorting, recalculated aggregates.
Expanding all rows creates a visual blink as the grid is repainted.
I guess it's fine when setting new groups or applying a new filter/sort but it's annoying when recalculating the aggregates.
My aggregates are calculated via a parametrized function and these parameters can change regularly.
Right now, i call setAggregatesGroup with the same parameters each time one of the parameters change. Is there a better way to recalculate aggregates?
This visual blink can also be my own doing when inappropriately calling either grid.behavior.reindex(), grid.behavior.changed() or grid.repaint()
Can you please remind me in which situation I should call which?

Re: Visual Blink
Reindex should be called when the state of your pipeline objects changes or on a setData call
Behavior Changed should be called when the state of a row, column or columns is changed. Such as when you resize a column, change their order, toggle a row or set a dynamic property. Many of this is already down internally on your behalf if you use the appropriate API method.
Repaint is called whenever you have updated the state of the grid. The repaint is not immediate however, but a flag that is set and checked on an interval. You may also make use of the continuousRepaint flag which leverages requestAnimationFrame

Re: Parametized Aggregations function
For all interested readers. Please note that we are going to be dropping support for our aggregations, filtering, sorting and analytics services. Our reasoning is defined here. We will provide an API to allow you to update the grid's view of nested data and or subgrids, but the actual grouping or aggregations etc, must be done on your end. In short, Hypergrid is a view. That being said our existing analytics services are being exported out of Openfin's github domain, but they will still be available for private forking by those interested.
The Hyper-analytics Aggregations module's internal state tracks an array of functions as aggregates and builds a schema to describe the columns (see here). If you wish to avoid resending duplicative functions for the sake of one function, you could certainly fork the module and provide easier helper utilities.

For all interested readers. Please note that we are going to be dropping support for our aggregations, filtering, sorting and analytics services. Our reasoning is defined here. We will provide an API to allow you to update the grid's view of nested data and or subgrids, but the actual grouping or aggregations etc, must be done on your end. In short, Hypergrid is a view. That being said our existing analytics services are being exported out of Openfin's github domain, but they will still be available for private forking by those interested.

@nclsclsn We are replacing support for aggregations with support for a tree column api. Please review the example here

@Dwaynekj the provided tree-column api demo URL is broken/not working. Can we get a working demo?


Pre-select user language in onboarding and move line item to the top of the list

In Javascript, we can get the user's language with the following command window.navigator.language, which would output something similar to **"en-US"*.
It has been reported that window.navigator.language is not 100% accurate so I don't recommend we by pass the language step, but instead we can use window.navigator.language to autoselect the language that is likely true and move it to the top of the list. The user more so has to confirm the pre-selected option instead of digging though the list to find their language then.
I believe only one thing needs to be added to the code to get this working.
Compare window.navigator.language to the options on the language onboarding step. If one matches, select it and move it to the top of the list. There likely needs to be some type of mapping logic put in place as well, e.g.

if language === "en-US" {
  $('#sp').attr('checked', true);
  $('.js-homeModal-languageList').prepend($('.rowItem[data-code="sp"]').parent());
}else if(...```

This introduces some light maintenance as we add new languages, but it seems worth it.

@jjeffryes please add any thoughts you may have to this.

This should be fairly easy to do, and would work well with how the language model already works.

Though I should point out it will only work for languages we have already added to the client.

@djbooth007

Completed


Simple Line Chart Class

In order to test and further establish decisions in the implementation, an extension should be created for the Line Chart. It should extend from the abstract base class (#14).
To handle:

[x] Put component in it's right category (Charts?)
[x] Give proper name to component
[x] Set up and document all needed references for the component (e.g. description) for component
[x] Construct a basic LineChart

To do for later (versioning.js and YaVersion.java): #20

Both versions for the simple Line Chart are done.
Main version: #18

Basic Line Chart implementation done. Tests should be written (#19)


Typo on stoptUpdatingLocation method name

I’d presume the method name should be stopUpdatingLocation and not stoptUpdatingLocation?
Of course XCode is able to suggest the right method.

Oh yes, would you be able to fix it please?


Problem with duplicate column headings

There seems to be an issue with the duplicate column detection when two (or more) tables are converted. This worked differently in v. 0.4.0 at least.
0.6.0 Output:
[     [{         "Column1": "C1",         "Column2": "C2"     }],     [{         "Column1_2": "C1",         "Column2_2": "C2"     }] ]
0.4.0 Output (correct):
[     [{         "Column1": "C1",         "Column2": "C2"     }],     [{         "Column1": "C1",         "Column2": "C2"     }] ]

Thanks so much for submitting and with details, it seemed like there was something up.
This is super helpful.
I'll spend some time at the weekend to create tests and resolve these.

I still have not gotten around to this, but see also #26 and #18

This issue will be fixed with version 0.9.2. Thanks for raising and describing it. I'm going to push the fix in the next minutes.


Parallel diagram: Cannot remove existing visual map via setOption

Version
5.0.2
Reproduction link
https://jsfiddle.net/3opxjg2v/
Steps to reproduce
Open the example at https://jsfiddle.net/3opxjg2v/
Click button "Remove visualMap".
What is expected?
The visual map should be removed and the default color should be shown
What is actually happening?
The visual map is not removed, the lines are still colored

Tried as well with setting visualMap to undefined or to an empty array. Neither did work.

Just found out that setOption with the default value of notMerge (false) will not remove components according to the documentation. I assume this is the case in the above example, therefore I'll close this issue. Sorry for the inconveniences.


Added/updated available data sources

@NickyvVr : Thanks for your contribution! The author(s) have been notified to review your proposed change.

#label:"aq-pr-triaged"

Hi, @NickyvVr -- thanks so much for pointing this out! I'm going to close your PR because I ended up making a slightly different change to this article, linking instead to the article that lists all the paginated report data sources, and updating the reference there.  Thanks for getting the ball rolling! Changes should be live this afternoon Pacific time, Fri Dec 17, 2021.


Vite example didn't work - 2 bugs : Element is missing end tag and Unknown word

Describe the bug
Hello,
I just would like to try the mardown example.
I encounter 2 error messages :
[vite] Internal server error: Element is missing end tag.
Plugin: vite:vue
File: C:/Users/X/Desktop/vite/README.md:28:1
26 |    plugins: [
27 |      Vue({
28 |        include: [/.vue$/, /.md$/], // <-- allows Vue to compile Markdown files
|   ^
29 |      }),
[vite] Internal server error: C:/Users/W/Desktop/vite/pages/index.md:6:10: Unknown word
Plugin: vite:vue
File: C:/Users/X/Desktop/vite/pages/index.md:35:10
34 |  }
35 |
36 |
|   ^
I am available for further questions
Reproduction
Minimal reproduction is on examples/vite folder
System Info
Win10 64bits,
Node 18.18.0
Npm 9.8.1
Firefox 118.0.1

------------
  System:
    OS: Windows 10 10.0.19045
    CPU: (12) x64 AMD Ryzen 5 2600 Six-Core Processor
    Memory: 18.63 GB / 31.92 GB
  Binaries:
    Node: 18.18.0 - C:\Program Files\nodejs\node.EXE
    npm: 9.8.1 - C:\Program Files\nodejs\npm.CMD
    pnpm: 8.8.0 - C:\Program Files\nodejs\pnpm.CMD
  Browsers:
    Chrome: 117.0.5938.92
    Edge: Spartan (44.19041.1266.0), Chromium (117.0.2045.43)
    Internet Explorer: 11.0.19041.1566

Used Package Manager
npm
Validations

[X] Follow our Code of Conduct
[X] Read the Contributing Guide.
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[X] Check that this is a concrete bug. For Q&A, please open a GitHub Discussion instead.
[X] The provided reproduction is a minimal reproducible of the bug.

I noticed that removing the  and  tags from README.md fixes this issue


Compatibility with atom-ide

Atom now has packages such as atom-ide-ui, and other languages have a *-ide package which works with it. I was wondering if Juno features could be made useable with the atom-ide-ui system. For example, I think I had to disable the linter I previously used in favor of atom-ide-diagnostics which does linting and other jobs.

This provides some LSP functionality, but proper Juno integration isn't trivial. We'll definitely take a look at this soonish though.

Close since now atom-ide-ui is dead and we definitely don't want to go along with it.


Is it possible to support multiple users ?

Would it be possible to support multiple users? something similar to how A1111/forge/fooocus does it using gradio?  we can basically have a list of credentials file reading the user/password pairs.
Our goal is to provide some minimal privacy for users sharing the same instance of ComfyUI, when they login we can retrieve their username to modify the output path to something like "output/[username]/date...". We do the same thing for Loras folder, so users get to have their own loras individually on the same instance.
Would this be possible to achieve using this custom node ?
Thank you.

My suggestion and best practice is to use docker to isolate different ComfyUI instance.
You can try my docker image: https://hub.docker.com/layers/sidaliu/comfyui/comfy-cli/images/sha256-0274da457d48876b6ecd37765870b1ab1c988b94ac8bfc4cd517310ed066808a?context=repo
An example of using it via docker compose is here:
https://github.com/liusida/ComfyUI-dockers (main file: docker-compose.yml)

FYI: this image: sidaliu/comfyui:comfy-cli uses comfy-cli to install brandnew ComfyUI instances in docker containers.

The only issue with that is queue management, for example with the other UIs that are gradio based, since users share the GPU instance, if they generate an image at the same time, they get in a queue as opposed to having their inference time slowed down, this is why we didn't go towards multi instance path. So far this has worked for us perfectly on A1111, Forge, Fooocus. We just need to achieve the same with ComfyUI.

Yes. queue management will be a pain.
ComfyUI is very versatile, especially with ComfyUI-Manager, people can install custom nodes that can do basically anything, including running dangerous programs on server. What's your thought on that? Provide only limited functionalities instead of the full fledged ComfyUI?

I've shared my ComfyUI instance with my friend earlier, and he's basically changed the basic UI by installing some translation nodes! That's why I started using docker.

I was thinking about a restricted mode recently: https://github.com/comfyanonymous/ComfyUI/issues/3740
But I'm not quite sure if this is the right direction.

Yes. queue management will be a pain.
ComfyUI is very versatile, especially with ComfyUI-Manager, people can install custom nodes that can do basically anything, including running dangerous programs on server. What's your thought on that? Provide only limited functionalities instead of the full fledged ComfyUI?

Yeah I did consider that, the solution I had in mind for now is simply to disable ComfyUI-Manager when running a shared instance, just like with A1111 you have the option to not allow non localhost users to install extensions and change important settings.

ComfyUI-Manager is just one major way to modify the server. Other nodes can do this too, like this very node, ComfyUI-Login, also modifies some of the core system... If we don't allow users to use their favorite nodes, ComfyUI becomes much less attractive.
Need to come up with a better solution. I'll sleep on it.

In our specific case it is really not that important, because we plan to provide our artists with specific pre-made workflows. Advanced users who will want to tinker and make new workflows (or import them from internet), will usually have an installed instance locally. So just like with A1111 we locked things like extensions, models, controlnets... we plan to do the same with ComfyUI, so we really don't mind having an instance with precise custom nodes that we maintain for all the users.
In fact in our case we plan to go further by exposing these specific workflows through a simplified UI (for now we're thinking of using mixlab for this).
The solution you try to find seems to be more suited for online services that are geared to power users who just don't have the GPUs to do the work, like this one https://comfy.icu/ which for now I think they also maintain specific versions of the most commonly used custom nodes.

Thanks for sharing your use case. I think a frozen/restricted mode can be helpful in your case.
It might be like that:
if there is a specified .yml file, the system will enter a frozen mode. The system will exam each workflow so that only nodes on the whitelist can be executed. And no custom nodes can be installed under frozen mode. But the users can still adjust the links and provided nodes, adjusting the parameters, so they can generate their own version of art.
Do you think this might help?

or should i call it the guest mode?

That's a nice idea, and I like all those names "frozen", "restricted", "guest".
The user/password input for me is necessarily especially for output and LoRA directories, we want each artist to have their own output folder (an artist can have access rights to their output folder only, but the instance can write to all of them), it will look like output/username/...
same thing for Loras, it would look like models/username/loras (or models/loras/username).
This is how we provide a required amount of privacy to our artists.


GLP Strats Dashboard seems to be throwing a "KeyError"

Hi Coin Data School,
I just wanted to let you know that your GLP Strats Dashboard seems to be throwing a "KeyError".  It says you can look in the logs to see details

I would love to try the dashboard.
Thanks,
Larry

This is caused by dune recently updated their graphql api url. I searched around and couldn't find the new url. Dune is rolling out an official API, and that's probably why. Will let you know once a solution emerges.

Thank you!
Larry
On Nov 26, 2022 at 12:17 AM -0700, Coin Data School @.***>, wrote:

This is caused by dune recently updated their graphql api url. I searched around and couldn't find the new url. Dune is rolling out an official API, and that's probably why. Will let you know once a solution emerges.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: @.***>

The Arbitrum side is fixed. The Avalanche side is still broken.

Thanks for the update.
Larry
On Mon, Dec 19, 2022 at 5:27 AM Coin Data School @.***>
wrote:

The Arbitrum side is fixed. The Avalanche side is still broken.
—
Reply to this email directly, view it on GitHub
https://github.com/coindataschool/GLP-strats-comp-dashboard/issues/1#issuecomment-1357596817,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/APD3ZBQKZYKSADHNQP4RKFTWOBIC3ANCNFSM6AAAAAASLVVPT4
.
You are receiving this because you authored the thread.Message ID:
@.***>

both sides are working now. Take a look and let me know otherwise.

Thank you!
On Thu, Dec 22, 2022 at 12:00 AM Coin Data School @.***>
wrote:

both sides are working now. Take a look and let me know otherwise.
—
Reply to this email directly, view it on GitHub
https://github.com/coindataschool/GLP-strats-comp-dashboard/issues/1#issuecomment-1362485214,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/APD3ZBVYKST543HXK5UJWDLWOP4AHANCNFSM6AAAAAASLVVPT4
.
You are receiving this because you authored the thread.Message ID:
@.***>

--
Sent from Gmail Mobile


Add temporary repo-level README

For context to those who come across the repository.

Good points! I updated based on those comments.

Looks good to me. Should help users navigate this repo until we decide to split it apart


Added TailWind command to npm scripts

Added the command to build the Tailwind CSS to NPM scripts.
In the client folder you can now run the command below to build/compile the tailwind css.
npm run tailwind

In the project root folder you can run the command. Which will start the React Client, The Tailwind CSS builder, and the server at the same time.
npm start

In both cases the base command that runs tailwind is:
npx tailwindcss build -i ./src/index.css -o ./dist/output.min.css --watch --minify

Great! Many thanks!


join order and CBO meta-thread

Currently the Quokka execution plan should support CBO and join ordering. This is a great issue for someone who wants to impress a Snowflake or Redshift interviewer! You can say you implemented a CBO join optimizer almost from scratch haha.
What needs to be done:
Currently Quokka's optimizer implements predicate pushdown early projection and map fusion.
After these three steps, most of the nodes in the system should be MapNodes and StatefulNodes (aka joins). Some of those joins might already be broadcast joins with materialized arguments, that's okay.

Join Fusion. We are now going to replace trees of join nodes with join nodes with multiple operands. This will greatly simplify the tree and expose different choice of joins. We will do predicate pull up from the individual join nodes. This should be a pretty straightforward optimization pass.

Cardinality Estimation. Input nodes now should have all the relevant early projection and pushed down filter, and can estimate how much data will be produced based on this. We are going to estimate the number of unique values for each column as well as the total expected size of the output from this node. This estimate will be pushed up the tree as far as possible.
If the estimated size is smaller than some threshold, the entire input will be materialized as a constant variable in the query plan. The relevant joins will be converted to broadcast joins at this stage.
This should also be a pretty straightforward pass on the graph. The hardest part will be to implement the actual CBO that answers the cardinality estimates given a predicate.

Join order selection. The join node will now figure out the order in which to do the joins. If some of its inputs are materialized it will join them in the main thread first. We are going to restrict ourselves to left deep plans and enumerate all n! possible plans. We will use all the nice independence assumptions to compute the join cost of each ordering. If some of the tables have already been materialized, they will be joined first to produce bigger/smaller materialized tables. This could potentially lead to a size blowup, which we don't address.

Check staged-execution branch


All Pods goes into CrashLoopBackOff, getting invalid argument (without BTF k/v) error

Hello @k8spacket,
I have setup K8spacket in our Dev account and is not starting. Getting below error for all the pods:
Defaulted container "k8spacket" out of: k8spacket, init-k8spacket (init)
2024/07/09 11:42:17 [k8spacket] [plugins] Cannot find plugin TLSConsumerPlugin, gave up. plugin: symbol TLSConsumerPlugin not found in plugin github.com/k8spacket/plugins/nodegraph
2024/07/09 11:42:17 [k8spacket] [plugins] Cannot find plugin TCPConsumerPlugin, gave up. plugin: symbol TCPConsumerPlugin not found in plugin github.com/k8spacket/plugins/tls-parser
2024/07/09 11:42:17 [k8spacket] [api] Serving requests on port 6676
2024/07/09 11:42:17 [k8spacket] INIT inet
2024/07/09 11:42:27 [k8spacket] [tc-loop] Refreshing interfaces for capturing...
Getting k8s resources
2024/07/09 11:42:27 [k8spacket] [tc] Loading objects: field TcFilter: program tc_filter: map output_events: map create: invalid argument (without BTF k/v)

Setup Details:
EKS Version: 1.28
Kernel Version: 5.10.214-202.855.amzn2.x86_64
k8spacket version:           image: "docker.io/k8spacket/k8spacket:2.0.4"
OS Details:
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"

Additional Details wrt bpf:
[root@IPAddress bin]# $ ls /sys/kernel/btf/vmlinux
bash: $: command not found
[root@IPAddress bin]# ls /sys/kernel/btf/vmlinux
/sys/kernel/btf/vmlinux
[root@IPAddress bin]# egrep '(CONFIG_BPF|CONFIG_BPF_SYSCALL|CONFIG_BPF_JIT|CONFIG_BPF_EVENTS)=' /boot/config-$(uname -r)
CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
CONFIG_BPF_JIT=y
CONFIG_BPF_EVENTS=y
[root@IPAddress bin]# egrep '(CONFIG_HAVE_BPF_JIT|CONFIG_HAVE_EBPF_JIT)=' /boot/config-$(uname -r)
CONFIG_HAVE_EBPF_JIT=y
[root@IPAddress bin]# cat /boot/config-$(uname -r) | grep CONFIG_IKHEADERS
CONFIG_IKHEADERS=m

Let me know if any other info is required.

Hi @shail248
Message: map output_events: map create: invalid argument (without BTF k/v) means that your kernel has a problem with creating ringbuf map output_events: https://github.com/k8spacket/k8spacket/blob/master/ebpf/tc/bpf/tc.bpf.c#L53
Based on the information you shared, I think it should work, because this kind of map has been added in version 5.8 of the Linux kernel: https://github.com/iovisor/bcc/blob/master/docs/kernel-versions.md#map-types
What you can check more (if you have or can install bpftool) is the command:
bpftool feature list_builtins map_types

to see if ringbuf map type is known on your kernel
Additionally, could you share the response to the command:
uname -a

I see that you shared the kernel version, but I'm curious because k8spacket was tested on AMI2 (amzn2-ami-kernel-5.10-hvm-2.0.20240131.0-x86_64-gp2) with the kernel:
5.10.205-195.807.amzn2.x86_64

Thanks for the quick response @k8spacket , Below is requested output:
bpftool feature list_builtins map_types isn't working but below is what I get while looking for ringbuf:
bpftool feature probe|grep ringbuf
eBPF map_type ringbuf is available
        - bpf_ringbuf_output
        - bpf_ringbuf_reserve
        - bpf_ringbuf_submit
        - bpf_ringbuf_discard
        - bpf_ringbuf_query
        - bpf_ringbuf_output
        - bpf_ringbuf_reserve
        - bpf_ringbuf_submit
        - bpf_ringbuf_discard
        - bpf_ringbuf_query
        - bpf_ringbuf_output
        - bpf_ringbuf_reserve
        - bpf_ringbuf_submit
        - bpf_ringbuf_discard
        - bpf_ringbuf_query
        - bpf_ringbuf_output
        - bpf_ringbuf_reserve
        - bpf_ringbuf_submit
        - bpf_ringbuf_discard
        - bpf_ringbuf_query
        - bpf_ringbuf_output
        - bpf_ringbuf_reserve
        - bpf_ringbuf_submit
        - bpf_ringbuf_discard
        - bpf_ringbuf_query
        - bpf_ringbuf_output
        - bpf_ringbuf_reserve
        - bpf_ringbuf_submit
        - bpf_ringbuf_discard
        - bpf_ringbuf_query
        - bpf_ringbuf_output
        - bpf_ringbuf_reserve
        - bpf_ringbuf_submit
        - bpf_ringbuf_discard
        - bpf_ringbuf_query
        - bpf_ringbuf_output
        - bpf_ringbuf_reserve
        - bpf_ringbuf_submit
        - bpf_ringbuf_discard
        - bpf_ringbuf_query
        - bpf_ringbuf_output
        - bpf_ringbuf_reserve
        - bpf_ringbuf_submit
        - bpf_ringbuf_discard
        - bpf_ringbuf_query
        - bpf_ringbuf_output
        - bpf_ringbuf_reserve
        - bpf_ringbuf_submit
        - bpf_ringbuf_discard
        - bpf_ringbuf_query
        - bpf_ringbuf_output
        - bpf_ringbuf_reserve
        - bpf_ringbuf_submit
        - bpf_ringbuf_discard
        - bpf_ringbuf_query
        - bpf_ringbuf_output
        - bpf_ringbuf_reserve
        - bpf_ringbuf_submit
        - bpf_ringbuf_discard
        - bpf_ringbuf_query
        - bpf_ringbuf_output
        - bpf_ringbuf_reserve
        - bpf_ringbuf_submit
        - bpf_ringbuf_discard
        - bpf_ringbuf_query
        - bpf_ringbuf_output
        - bpf_ringbuf_reserve
        - bpf_ringbuf_submit
        - bpf_ringbuf_discard
        - bpf_ringbuf_query
        - bpf_ringbuf_output
        - bpf_ringbuf_reserve
        - bpf_ringbuf_submit
        - bpf_ringbuf_discard
        - bpf_ringbuf_query
        - bpf_ringbuf_output
        - bpf_ringbuf_reserve
        - bpf_ringbuf_submit
        - bpf_ringbuf_discard
        - bpf_ringbuf_query
        - bpf_ringbuf_output
        - bpf_ringbuf_reserve
        - bpf_ringbuf_submit
        - bpf_ringbuf_discard
        - bpf_ringbuf_query
        - bpf_ringbuf_output
        - bpf_ringbuf_reserve
        - bpf_ringbuf_submit
        - bpf_ringbuf_discard
        - bpf_ringbuf_query
        - bpf_ringbuf_output
        - bpf_ringbuf_reserve
        - bpf_ringbuf_submit
        - bpf_ringbuf_discard
        - bpf_ringbuf_query
        - bpf_ringbuf_output
        - bpf_ringbuf_reserve
        - bpf_ringbuf_submit
        - bpf_ringbuf_discard
        - bpf_ringbuf_query
        - bpf_ringbuf_output
        - bpf_ringbuf_reserve
        - bpf_ringbuf_submit
        - bpf_ringbuf_discard
        - bpf_ringbuf_query
        - bpf_ringbuf_output
        - bpf_ringbuf_reserve
        - bpf_ringbuf_submit
        - bpf_ringbuf_discard
        - bpf_ringbuf_query
        - bpf_ringbuf_output
        - bpf_ringbuf_reserve
        - bpf_ringbuf_submit
        - bpf_ringbuf_discard
        - bpf_ringbuf_query
        - bpf_ringbuf_output
        - bpf_ringbuf_reserve
        - bpf_ringbuf_submit
        - bpf_ringbuf_discard
        - bpf_ringbuf_query
        - bpf_ringbuf_output
        - bpf_ringbuf_reserve
        - bpf_ringbuf_submit
        - bpf_ringbuf_discard
        - bpf_ringbuf_query
        - bpf_ringbuf_output
        - bpf_ringbuf_reserve
        - bpf_ringbuf_submit
        - bpf_ringbuf_discard
        - bpf_ringbuf_query

uname -a:
Linux ip-10-220-233-95.ap-south-1.compute.internal 5.10.214-202.855.amzn2.x86_64 #1 SMP Tue Apr 9 06:57:12 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Hi @shail248
Could you try this command (it runs prepared k8spacket docker image for testing without k8s and go-plugins):
sudo docker run -it --cap-add=CAP_SYS_ADMIN --cap-add=CAP_NET_ADMIN --env=K8S_PACKET_K8S_RESOURCES_DISABLED=true --env=K8S_PACKET_TCP_LISTENER_INTERFACES_REFRESH_PERIOD=2s --env=K8S_PACKET_TCP_LISTENER_INTERFACES_COMMAND="echo 'eth0' | tr '\\n' ','" -v /sys/kernel/debug/tracing:/sys/kernel/tracing/ --user 0 --network host k8spacket/k8spacket:disablek8s

and from another terminal call f.e.
curl https://ebpf.io

Then in logs, you should get
2024/07/10 14:45:07 [k8spacket] [tc-loop] Refreshing interfaces for capturing...
2024/07/10 14:45:08 [k8spacket] TLS Event: {"Client":{"Addr":"<IP_ADDRESS>","Port":34890,"Name":"N/A","Namespace":""},"Server":{"Addr":"<IP_ADDRESS>","Port":443,"Name":"","Namespace":""},"TlsVersions":[],"Ciphers":[49200,49196,49192,49188,49172,49162,165,163,161,159,107,106,105,104,57,56,55,54,136,135,134,133,49202,49198,49194,49190,49167,49157,157,61,53,132,49199,49195,49191,49187,49171,49161,164,162,160,158,103,64,63,62,51,50,49,48,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"ServerName":"ebpf.io","UsedTlsVersion":771,"UsedCipher":49195}
2024/07/10 14:45:09 [k8spacket] TCP Event: {"Client":{"Addr":"<IP_ADDRESS>","Port":34890,"Name":"N/A","Namespace":""},"Server":{"Addr":"<IP_ADDRESS>","Port":443,"Name":"","Namespace":""},"TxB":813,"RxB":288208,"DeltaUs":1585}
2024/07/10 14:45:09 [k8spacket] [tc-loop] Refreshing interfaces for capturing...

I ran an instance in your area and got the VM:
Linux ip-172-31-6-99.ap-south-1.compute.internal 5.10.219-208.866.amzn2.x86_64 #1 SMP Tue Jun 18 14:00:06 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

It looks similar to yours and works as expected

@k8spacket Apologies for late response, getting below error while running docker command itself and getting same error:
root@ip-10-220-160-11 bin]# docker run -it --cap-add=CAP_SYS_ADMIN --cap-add=CAP_NET_ADMIN --env=K8S_PACKET_K8S_RESOURCES_DISABLED=true --env=K8S_PACKET_TCP_LISTENER_INTERFACES_REFRESH_PERIOD=2s --env=K8S_PACKET_TCP_LISTENER_INTERFACES_COMMAND="echo 'eth0' | tr '\\n' ','" -v /sys/kernel/debug/tracing:/sys/kernel/tracing/ --user 0 --network host k8spacket/k8spacket:disablek8s
Unable to find image 'k8spacket/k8spacket:disablek8s' locally
disablek8s: Pulling from k8spacket/k8spacket
ec99f8b99825: Pull complete
787733fc55b4: Pull complete
7d0a9b42f31c: Pull complete
4f4fb700ef54: Pull complete
8f885d38541b: Pull complete
376dad55b17b: Pull complete
Digest: sha256:5f219bf8b149d5afd226071829b4834a90f3d0133004f0929a526bf606f69e5c
Status: Downloaded newer image for k8spacket/k8spacket:disablek8s
2024/07/18 08:18:44 [k8spacket] [api] Serving requests on port
2024/07/18 08:18:44 [k8spacket] INIT inet
2024/07/18 08:18:46 [k8spacket] [tc-loop] Refreshing interfaces for capturing...
Getting k8s resources is disabled
2024/07/18 08:18:46 [k8spacket] [tc] Loading objects: field TcFilter: program tc_filter: map output_events: map create: invalid argument (without BTF k/v)

@k8spacket Did you got time to look into it

Hi @shail-jaiswar24
Not yet, sorry.
This is one man side project and I'm on holidays curently. I'll back to you on Monday.
I cannot do more at this moment, sorry.

Hi @shail-jaiswar24
Could you run the command and share the result?
sudo docker run -it --cap-add=CAP_SYS_ADMIN --cap-add=CAP_NET_ADMIN --env=K8S_PACKET_K8S_RESOURCES_DISABLED=true --env=K8S_PACKET_TCP_LISTENER_INTERFACES_REFRESH_PERIOD=2s --env=K8S_PACKET_TCP_LISTENER_INTERFACES_COMMAND="echo 'eth0' | tr '\\n' ','" -v /sys/kernel/debug/tracing:/sys/kernel/tracing/ --user 0 --network host k8spacket/k8spacket:kernelversion

It's almost the same as before, except docker image name. I want to check the Linux kernel that is visible in the application.
My response:
...
Status: Downloaded newer image for k8spacket/k8spacket:kernelversion
2024/07/29 13:03:01 [k8spacket] Linux 5.10.219-208.866.amzn2.x86_64 #1 SMP Tue Jun 18 14:00:06 UTC 2024
2024/07/29 13:03:01 [k8spacket] [api] Serving requests on port
2024/07/29 13:03:01 [k8spacket] INIT inet
...

Hi guys, any idea for this issue, because i have the same issue

Hi @ddysmhz
Could you share linux kernel version of your host?
uname -a

and try command from comment https://github.com/k8spacket/k8spacket/issues/62#issuecomment-2255892823 and share the response also.

Hi @shail248 , @shail-jaiswar24
I prepared the host with the same Linux Kernel as you have
[root@ip-172-31-46-8 ~]# uname -a
Linux ip-172-31-46-8.ap-south-1.compute.internal 5.10.214-202.855.amzn2.x86_64 #1 SMP Tue Apr 9 06:57:12 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

When I ran the command from the comment https://github.com/k8spacket/k8spacket/issues/62#issuecomment-2255892823, I got a working response:
# sudo docker run -it --cap-add=CAP_SYS_ADMIN --cap-add=CAP_NET_ADMIN --env=K8S_PACKET_K8S_RESOURCES_DISABLED=true --env=K8S_PACKET_TCP_LISTENER_INTERFACES_REFRESH_PERIOD=2s --env=K8S_PACKET_TCP_LISTENER_INTERFACES_COMMAND="echo 'eth0' | tr '\\n' ','" -v /sys/kernel/debug/tracing:/sys/kernel/tracing/ --user 0 --network host k8spacket/k8spacket:kernelversion
2024/08/12 08:33:27 [k8spacket] Linux 5.10.214-202.855.amzn2.x86_64 #1 SMP Tue Apr 9 06:57:12 UTC 2024
2024/08/12 08:33:27 [k8spacket] [api] Serving requests on port
2024/08/12 08:33:27 [k8spacket] INIT inet
2024/08/12 08:33:29 [k8spacket] [tc-loop] Refreshing interfaces for capturing...
Getting k8s resources is disabled
...
2024/08/12 08:33:45 [k8spacket] [tc-loop] Refreshing interfaces for capturing...
2024/08/12 08:33:45 [k8spacket] TLS Event: {"Client":{"Addr":"<IP_ADDRESS>","Port":41972,"Name":"N/A","Namespace":""},"Server":{"Addr":"<IP_ADDRESS>","Port":443,"Name":"","Namespace":""},"TlsVersions":[],"Ciphers":[49200,49196,49192,49188,49172,49162,165,163,161,159,107,106,105,104,57,56,55,54,136,135,134,133,49202,49198,49194,49190,49167,49157,157,61,53,132,49199,49195,49191,49187,49171,49161,164,162,160,158,103,64,63,62,51,50,49,48,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"ServerName":"ebpf.io","UsedTlsVersion":771,"UsedCipher":49195}
2024/08/12 08:33:47 [k8spacket] TCP Event: {"Client":{"Addr":"<IP_ADDRESS>","Port":41972,"Name":"N/A","Namespace":""},"Server":{"Addr":"<IP_ADDRESS>","Port":443,"Name":"","Namespace":""},"TxB":813,"RxB":286236,"DeltaUs":2146}
2024/08/12 08:33:47 [k8spacket] [tc-loop] Refreshing interfaces for capturing...
...

I started EKS in ap-south-1 location with Amazon Linux 2 nodes and didn't get problems with k8spacket either. It's hard to say, why this problem reached your machine. Did you do some Linux kernel upgrades manually?

@k8spacket Thanks for the update, I haven't tried any as suggested above but now it has started working not sure how. No kernel or EKS upgrade but its working pretty fine.
Thanks for all the help, will keep you posted if observed nay anomaly.

There is no more activity in the thread. I'm going to close the issue.