Multi-process support

This PR adds multi-process support (clone syscall, execve syscall, exit syscall) for lind within wasmtime.

Implemented features and related files:

moved lind_syscall interface from wasi_preview1 to a standalone module lind-common and add cageid parameter to lind_syscall. Related files are crates/lind-common/src/lib.rs
Implemented clone syscall and exit syscall. Related files are crates/lind-multi-process/src/clone_constants.rs, crates/lind-multi-process/src/lib.rs, crates/wasmtime/src/runtime/externals.rs, crates/wasmtime/src/runtime/func.rs, crates/wasmtime/src/runtime/func/typed.rs, crates/wasmtime/src/runtime/store.rs
Created a global cage counter to make sure the process will only exit after all cages were exited (instead of main process exiting). Related files are crates/lind-utils/src/lib.rs
Implemented execve syscall. Related files are crates/lind-multi-process/src/lib.rs, src/commands/run.rs

Unfortunately its in our best interests to split this PR as well.
Realistically it should be
PR1: the changes needed to interface with the base rustposix PR and the base glibc PR. This should go to main.
PR2: the changes for multi-processing.

Splitted. This PR now contains mainly multi-process stuff only


Error while Decompression in CyberChef

I compressed a base 64 text using ZIP in cyberchef and when I open the cyberchef in a new sessionto decompress the same compressed Text format it throws an error "END OF CENTRAL CIRECTORY RECORD NOT FOUND". Can anyone help me with this erro in CyberChef?

Hi @lbhardwaj, there are a number of ways this error could have come about. From the way you've described it, I suspect it's due to you copying over the raw output of the Zip operation into a new instance of CyberChef. Some bytes are not printable, so CyberChef replaces them with dots (.) to allow users to see that there is an unprintable byte there. This means that if you just copy that raw output, you will lose the actual values of those bytes, which will make unzipping the data impossible and lead to the error message you described. If you need to copy the zip output somewhere, you should either convert it to a text-based format like hexadecimal or base64, or save it to a file.
I hope this solves your problem. If not, please let us know and include the input and recipe which can replicate the error.


compile error

Suddenly got this error, seems related to the new release. Note that I'm not using this crate directly. Any ideas?
Compiling flate2 v1.0.10
error[E0658]: use of unstable library feature 'rustc_private': this crate is being loaded from the sysroot, an unstable location; did you mean to load this crate from crates.io via Cargo.toml instead?
--> /home/rust/.cargo/registry/src/github.com-1ecc6299db9ec823/flate2-1.0.10/src/lib.rs:89:1
|
89 | extern crate libc;
| ^^^^^^^^^^^^^^^^^^
|
= note: for more information, see https://github.com/rust-lang/rust/issues/27812
error[E0658]: use of unstable library feature 'rustc_private': this crate is being loaded from the sysroot, an unstable location; did you mean to load this crate from crates.io via Cargo.toml instead?
--> /home/rust/.cargo/registry/src/github.com-1ecc6299db9ec823/flate2-1.0.10/src/ffi.rs:52:20
|
52 |     pub use libc::{c_int, c_uint, c_void, size_t};
|                    ^^^^^
|
= note: for more information, see https://github.com/rust-lang/rust/issues/27812

Right, this seems to have been a result of me moving libc to optional features in my miniz_oxide commit (as it wasn't needed at all any longer when using the rust backend.)
If you compile the crate with --no-default-features and don't specify any features, it breaks as libc will not have been specified in Cargo.toml. I presume it doesn't break on miniz_sys in the same way since that resides as a subfolder. What crate is it that's using flate2-rs in this way?

Hm I don't think this crate has ever compiled successfully with --no-default-features, @sallyyu0 can you indicate how you're depending on this crate?

It compiles with --no-default-features when I test with the 0.2.9 tag, while 1.0.0 breaks.

Oh it breaks with --no-default-features + --features miniz_oxide

In my project it's actix-http 0.2.9 (via latest actix-web) that depends on flate2 with default-features = false:
https://github.com/actix/actix-web/blob/979c4d44f4626e4346714753106d2e94c9b845d1/actix-http/Cargo.toml#L93
If I put flate2 = "1.0.10" in my Cargo.toml without any feature flags, then the libc crate is added as a dependency of flate2 in Cargo.lock and the project builds fine.

I presume the actix author's intention is to allow chosing between what flate2 backend to use with two different features in the actix-http crate. The the non-rust one is specified as using the miniz-sys feature (which does not exist).
Doing cargo test --no-default-features --features miniz-sys works fine on 1.0.9, but not on 1.0.10.
cargo test --no-default-features --features "miniz-sys libc" does work on 1.0.10 on the other hand.
Is this how cargo is supposed to work?

Yes Cargo is working as intended here, and it's a bug in this crate basically that there wasn't a dedicated feature for turning on the miniz backend. It was implicitly left as miniz-sys, and looks like we need to keep that working. I've posted https://github.com/alexcrichton/flate2-rs/pull/206 to fix things.

I suppose a change to add feature and make it optional would need a major version bump then. Wasn't aware of the possibility of using a dependency as a feature in this way.

Thanks all for the input. Here is what I did to pass the build.

added flate2 = "=1.0.10" in Cargo.toml (I did not have this and from the logs of last build, it was 1.0.9), this fixed cargo build but still get a failure in cargo test

updated rust compiler from 1.33 to 1.34, passed cargo test.

Heh no worries @oyvindln, I also didn't realize that your change was technically a breaking change! We can fix this via other methods with renaming dependencies maybe, but it's basically easiest for now to just keep things working as they were before.


Mcedit <IP_ADDRESS> Not updating command Block Coordinates Upon Paste

Every time I paste a schematic containing command blocks with static coordinates the Static coordinates do not get updated in ANY of the Command Blocks. This is started happening in the newest version (<IP_ADDRESS>). The command do retain their data, simply they do not get updated. Thank-you for any solutions that can be offered.

Please, answer of the following question, even if some may look stupid:

What is the game version of the world you want to import schematics in?
If you know it, what is the game version used to create the schematics?
In the import tool panel, did you check the box Update Command Block Coords?

Also, note that, if the option Cancel Command Block Offset is enabled, this will prevent all coordinates in command blocks to be updated. You can find the related checkbox in the main options panel.

Hello, the world is on minecraft game version: 1.11.2 for both worlds (paste and schematic)
I did make sure to check the box - Update Command Block Coords.
I will Try with that last note you gave me! Thank-you!

We've changed the game IDs handling in MCEdit-Unified v<IP_ADDRESS> to make it more friendly with different game versions. It is still a work in progress feature, and will be enhanced in future versions.
Having the files needed by the program to handle correctly the game v1.11.2 may solve your issue.
To get them, just copy the 1.11 folder as 1.11.2 in the mcver one you'll find in your MCEdit installation directory.
If you run on Mac, you'll have to enter the .app.
If you run on Linux, creating a link to the 1.11 folder shall do the job.

In the future this feature may be enhanced to automatically fall back to the next lowest version to prevent this (1.9.2  using the 1.9 defs etc.) I think some improvements were made for <IP_ADDRESS> if I remember correctly though.

Right.
The files can now "call" other ones when loading. The MCEdit v<IP_ADDRESS> files for game v1.11.2 contain one definition each to make the 1.11 files be loaded.
An automatic fall back to the next lowest version will be also added.
Note that PE also has its own directory and files :smile:

Hello, I have been unable to find the mcver folder and the new update has not fixed the problem. When I do not rotate the ship the commands remain within the command blocks but are un-changed (even with the command update option on) When I rotate the schematic the commands are all wiped. I would really appreciate any suggestion on how to fix these specific problems.
Thank-you

The mcver folder shall be included in the installers...
We'll have a look at this.
You can try the following steps:

Open this link: https://github.com/Khroki/MCEdit-Unified/tree/4246e24d58d8cfb5b83f6328362b3ee674e46686
It is the code which was used to build v<IP_ADDRESS>.
Click on the Clone or Download button, then on Download ZIP.
Save or open the archive. You'll have the mcver folder in.
Put the mcver in the folder where MCEdit is installed.
If you run on Mac, you'll have to open the .app.
If you run on Windows, put the folder alongside mcedit.exe.
If you run on Linux, put the folder alongside mcedit.sh.

I did exactly that and yet the problem remains the same. I run on windows - Here is a screen shot of where the folder is placed:

@Julienc249 Did you try MCEdit v<IP_ADDRESS>?

I am sorry to have to reopen this thread again, however, lately, every time I rotate and paste a schematic all nbt data is wiped (command blocks are wiped, chests, paintings, signs go invisible etc.) I cannot understand why this is happening. Could it have to do with the fact that the world the schematics were taken from is from 1.8 and the world I am pasting them in is 1.9 - 1.11.2?
Thank-you for any help you can provide and fix you can come up with.

I don't experience this on my side.
Which MCEdit version are you using?

Sorry for the late response, I am using the latest version of mcedit unified - <IP_ADDRESS> I believe.

To know the exact version of MCEdit, just look in the console, or the title bar :)
On my side, as I said, I don't experience this when importing MC 1.8 schematics I created.
Can you send your's, please?
We'll have a look at, and try to find the cause.

Ok, thank-you very much, it is indeed the latest version (<IP_ADDRESS>) and the main schematic that seems to exhibit the problem is the one that you should receive through gmail. (hopefully if this one's cause can be tracked down, my other can be fixed as well)
Again, thank-you very much!

No incoming stuff from gmail for now...
I guess you can upload it here, though.

U.S.S. Valiant.zip

After doing some testing, I realized that maybe what is happening is that the command blocks are not getting rotated but actually replaced, with empty nbt data. Maybe this could be due to the 1.9 updates where command block rotation and facing began to have an impact on their functionality.

Hello, does anyone know how this problem is going?

To anyone that is interested: I believe now that this issue is directly linked to the issue with paintings and item frames issue that has been observed before. This issue is only occurring in the latest version of mcedit. One version lower and you can avoid the problem. However, all paintings are cleared and command blocks are not updated despite the update command block coords being selected. We can only hope that this will be fixed soon.

@Julienc249 I've made some tests on my side with the 'U.S.S. Valiant' schematic you provided.
The fact is some command blocks do not have any command in the schematic. 5I guess you already knew that.)
Also, the NBT data looks good for paintings and item frames and are imported by MCEdit, but once loaded in MC 1.12-pre7, they are removed...
When importing the schematic in a 1.10.2 world, the signs are still there, but no visible.
From which game version the schematic has been created?

Yes, many command blocks are empty and that is normal, however all of them get cleared (you already know that) the schematic was taken in 1.8.8, just before the release of 1.9. Thank-you very much for the work you have done already tracking this issue down.


Removing deep imports and type alias, merging commons and utils

Merging commons and utils together because in another PR I want to add typechecking to database seeds. This in turn leads to commons and utils requiring each other for typings.
I also make exports for all vars in commons. this allows to remove all type alias and all deep imports

@fguitton I don't undrstand what should be in the package.json entry point is not /dist/index.js.
They still have to be compiled when publish right?


ci: changed action secret

ci: changed action secret
Changed the gh-action runner secret to give permissions to the gh-action[bot]

:warning: Please install the  to ensure uploads and comments are reliably processed by Codecov.
Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:
:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.


nfhl_pipeline.py fails when using the Python 3.9 Dataflow base image

Beam SDK disconnects will cause the Dataflow pipeline to fail

Can you specify the command and input data you are using? It's not possible to debug with the information provided.

Sorry, yeah this is a bit sparse. Basically I followed the instructions in the README, the only thing I did differently was use the Python 3.9 base image for data flow and that causes the process to fail outright. I made no other changes to the current process with the exception of removing that extra argument in the previous PR you merged before. I think if you were to rerun the pipeline with the latest data flow Python 3.9 image you should be able to replicate the issue. If it doesn't occur anymore then great, maybe it was just a temporary issue with the image.


Fix: link to troubleshooting guideline in docs

Q
A

Bug fix?
no

New feature?
no

API breaks?
no

Deprecations?
no

License
Apache 2.0

What's in this PR?
Troubleshooting guideline link is not rendering properly. This edit fixes link and blockquote.
Why?
Issues with  documentation

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Error: Cannot find module 'node:url'

Hi,
I seem to be missing something. Do I need to install something extra? Running my code gives me;
Error: Cannot find module 'node:url'
Require stack:
- ...node_modules/http-cookie-agent/dist/http/create_cookie_agent.js
- ...node_modules/http-cookie-agent/dist/http/index.js
- ...node_modules/http-cookie-agent/http/index.js
- ...server/client.js

im importing your package via; const { HttpCookieAgent, HttpsCookieAgent } = require('http-cookie-agent/http'); and using
...
this.instance = axios.create({
      baseURL: 'https://members-ng.iracing.com',
      httpAgent: new HttpCookieAgent({ cookies: { jar } }),
      httpsAgent: new HttpsCookieAgent({ cookies: { jar } }),
    })
...

I do not have any messages or errors in the IDE, it just fails on start.
Any pointers on what I'm doing wrong would be great.
Thanks in advance

Use Node.js >= v14.18.0 or >= v16.0.0 (except v15.x)
See https://nodejs.org/api/modules.html#core-modules

v16.0.0, v14.18.0 | Added node: import support to require(...).

Hi, I am using v16.15.1, so should be okay. i'll play around with nvm, maybe its a version issue

I had many node issue, all resolved now. thanks for the help, and package!

This still breaks with jest@26, even on supported Node versions

It was added here, recently: https://github.com/facebook/jest/pull/11331


As an SRE, I want to be able to execute against Taqueria running in a container so that I can improve efficiency by integrating Taqueria into my CI/CD process

Implementation:
Docker image

The Dockerfile is complete and functional. It might change again slightly at a later stage if the github action becomes more complex.

This looks good to me, is there a need for more review or can we close this?

@jchenche please will you review this and provide an approval here?

@russellmorton Disclaimer: not very fluent with Docker image 😅 but it looks good to me!


Add chat completion method

Supersede #581. Thanks to @Wauplin, I can import the types from "@huggingface/tasks"
I've followed the pattern for textGeneration and textGenerationStream.

One question @coyotte508 , do we want to abstract the /v1/chat/completions endpoint if the user uses chatCompletion with our hosted models, let's say mistralai/Mistral-7B-Instruct-v0.2, shall this also work?
const stream = hf.chatCompletionStream({
				model: "mistralai/Mistral-7B-Instruct-v0.2",
				messages: [{ role: "user", content: "Complete the equation 1+1= ,just the answer" }],
				max_tokens: 500,
				return_full_text: false,
				temperature: 0.1,
				seed: 0,
			});

The only issue is now all models are served/compatible with /v1/chat/completions we might ended up on the same

@radames not quite exactly. TGI-served models expose a /v1/chat/completions route which is fully compatible with the chat completion API. transformers-served models on the other hand don't have a /v1/chat/completions route but they still accept to receive a list of messages as input in their text-generation pipeline. In this case messages are rendered server-side which makes figure from https://github.com/huggingface/huggingface_hub/pull/2094#issue-2173830914 obsolete. I made the change in https://github.com/huggingface/huggingface_hub/pull/2258 if you are interested to look at it. In particular you'll see that the transformers response is simply a text string since it is still the text-generation pipeline. To return a proper ChatCompletionOutput object, you'll have to populate fields manually (in my case a lot of fields are set to "dummy"). Hope this will help, please let me know if you have some remaining questions.

Oh and btw, TGI-served models have a /info route exposed (see https://api-inference.huggingface.co/models/meta-llama/Meta-Llama-3-70B-Instruct/info). With this endpoint you can be 100% sure whether a model is TGI or transformers backed. In the current huggingface_hub implementation we rely on errors to "guess/infer" if a model is not TGI-served but don't do this :grimacing: I'll open a PR to fix this on hfh-side as well.

The only issue is now all models are served/compatible with /v1/chat/completions we might ended up on the same

@radames not quite exactly. TGI-served models expose a /v1/chat/completions route which is fully compatible with the chat completion API. transformers-served models on the other hand don't have a /v1/chat/completions route but they

thanks @Wauplin, yes I miswrote my sentence, and I wanted to say that not all models are served with mixedbread-ai/mxbai-embed-large-v1 exactly your point!
and I think we should copy this logic to complete the model url, in case ones provide only a model id
           model_url = self._resolve_url(model)
            if not model_url.endswith("/chat/completions"):
                model_url += "/v1/chat/completions"

Thanks @coyotte508, @Wauplin got some inspiration from your Python implementation. Much simpler for now, but please let me know what you think.

great point @julien-c , I'll add docs , and I agree with you this method will be quite used, I have a few projects waiting for this to replace openai client. The PR is good to go, I just wanted @Wauplin to have a final look, but we could merge and deploy, so we can test it in the wild?

Yes for sure @radames please update the README in the root and in packages/inference and then we cna merge

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@coyotte508 Done, feel free to check and merge thanks! I'll push the demo list soon after the package is published 🚀

Can you update the tests in HfInference.spec.ts to use the chatCompletionStream method instead of streamingRequest?
(also check my README changes)
It should be good afterwards

Can you update the tests in HfInference.spec.ts to use the chatCompletionStream method instead of streamingRequest?
(also check my README changes)
It should be good afterwards

@coyotte508 for the external providers, it has to be streamingRequest since they don't support receiving extra arguments that comes with the chatCompletionStream , which sends options params, that are compatible with our serveless API and TGI

@coyotte508 here is the issue, when we use predefined tasks, such as textGeneration, which is compatible with our serveless API, https://huggingface.co/docs/api-inference/en/detailed_parameters, option parameter is valid. That's why on my example I use the custom request, since it won't send anything options
https://github.com/huggingface/huggingface.js/blob/b66fcf345e03e72754aa20d0d4bb123b0d3d2f44/packages/inference/src/lib/makeRequestOptions.ts#L114-L117

Great point. We might not need otherOptions. It seems like we can do it via headers, and we're already doing it.

Great point. We might not need otherOptions. It seems like we can do it via headers, and we're already doing it.

@radames can you run tests again with VCR_MODE=cache to record the new tapes.json?
Hopefully it should be good afterwards

thanks @coyotte508 done! I ran the tests only for the new api VCR_MODE=cache pnpm exec vitest run  -t "OpenAI Specs"

I think is just the final ok from @coyotte508 it's ready to be merged and released. I'm also waiting for this, to build a couple of things.

Thanks @radames !


Implement engagement transfer in Interactor

This PR implements engagement transfer callback in Interactor.
Engagement transfer callback changes the state to being engaged with a new first available operator and notifies all listening parties of the change.
MUIC-244

If you guys have any idea how to write unit tests for this I would be thankful. It seems like Operator cannot be mocked due to init protection level, so I didn't find any easy to write unit test that state does change when engagement transfer callback gets called

If you guys have any idea how to write unit tests for this I would be thankful. It seems like Operator cannot be mocked due to init protection level, so I didn't find any easy to write unit test that state does change when engagement transfer callback gets called

Let's wait until Operator has public init. That will enable mocking for it.

Added tests


Add Parallax Scrolling Effect to the images

Hi Can I contribute by adding Parallax Scrolling Effect to the latest trends and best sellers section of your web site as a part of Hacktoberfest ?

sure @soumya495