Running model from a GGUF file, only

Describe the bug
Running model from a GGUF file using llama.cpp is very straightforward, just like that:
server -v -ngl 99 -m Phi-3-mini-4k-instruct-Q6_K.gguf
and if model is supported, it just works.
I tried to do the same using mistral.rs, and I've got that:
mistralrs-server gguf -m . -t . -f .\Phi-3-mini-4k-instruct-Q6_K.gguf
2024-05-17T07:21:48.581660Z  INFO mistralrs_server: avx: true, neon: false, simd128: false, f16c: true
2024-05-17T07:21:48.581743Z  INFO mistralrs_server: Sampling method: penalties -> temperature -> topk -> topp -> multinomial
2024-05-17T07:21:48.581820Z  INFO mistralrs_server: Loading model `.` on Cuda(CudaDevice(DeviceId(1)))...
2024-05-17T07:21:48.581873Z  INFO mistralrs_server: Model kind is: quantized from gguf (no adapters)
2024-05-17T07:21:48.583625Z  INFO hf_hub: Token file not found "C:\\Users\\[REDACTED]\\.cache\\huggingface\\token"
2024-05-17T07:21:48.583707Z  INFO mistralrs_core::utils::tokens: Could not load token at "C:\\Users\\[REDACTED]/.cache/huggingface/token", using no HF token.
thread 'main' panicked at mistralrs-core\src\pipeline\gguf.rs:290:58:
File "tokenizer.json" not found at model id "."
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Why it asks me for a tokenizer file, when it's included in the GGUF file? I understand having this as an option (if I wanted to try out different tokenizer / configuration), but by default it should just use information provided in the gguf file itself.
Next attempt, when I copied tokenizer.json from the original model repo:
mistralrs-server gguf -m . -t . -f .\Phi-3-mini-4k-instruct-Q6_K.gguf
2024-05-17T07:28:34.987235Z  INFO mistralrs_server: avx: true, neon: false, simd128: false, f16c: true
2024-05-17T07:28:34.987332Z  INFO mistralrs_server: Sampling method: penalties -> temperature -> topk -> topp -> multinomial
2024-05-17T07:28:34.987382Z  INFO mistralrs_server: Loading model `.` on Cuda(CudaDevice(DeviceId(1)))...
2024-05-17T07:28:34.987431Z  INFO mistralrs_server: Model kind is: quantized from gguf (no adapters)
2024-05-17T07:28:34.989190Z  INFO hf_hub: Token file not found "C:\\Users\\[REDACTED]\\.cache\\huggingface\\token"
2024-05-17T07:28:34.989270Z  INFO mistralrs_core::utils::tokens: Could not load token at "C:\\Users\\[REDACTED]/.cache/huggingface/token", using no HF token.
2024-05-17T07:28:35.532371Z  INFO mistralrs_core::pipeline::gguf: Loading `"tokenizer.json"` locally at `".\\tokenizer.json"`
thread 'main' panicked at mistralrs-core\src\pipeline\gguf.rs:290:58:
File "config.json" not found at model id "."
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

And another attempt, after copying config.json (which I think is also unnecessary, as llama.cpp works fine without it):
mistralrs-server gguf -m . -t . -f .\Phi-3-mini-4k-instruct-Q6_K.gguf
2024-05-17T07:30:00.352139Z  INFO mistralrs_server: avx: true, neon: false, simd128: false, f16c: true
2024-05-17T07:30:00.352236Z  INFO mistralrs_server: Sampling method: penalties -> temperature -> topk -> topp -> multinomial
2024-05-17T07:30:00.352301Z  INFO mistralrs_server: Loading model `.` on Cuda(CudaDevice(DeviceId(1)))...
2024-05-17T07:30:00.352344Z  INFO mistralrs_server: Model kind is: quantized from gguf (no adapters)
2024-05-17T07:30:00.354085Z  INFO hf_hub: Token file not found "C:\\Users\\[REDACTED]\\.cache\\huggingface\\token"
2024-05-17T07:30:00.354168Z  INFO mistralrs_core::utils::tokens: Could not load token at "C:\\Users\\[REDACTED]/.cache/huggingface/token", using no HF token.
2024-05-17T07:30:00.601055Z  INFO mistralrs_core::pipeline::gguf: Loading `"tokenizer.json"` locally at `".\\tokenizer.json"`
2024-05-17T07:30:00.814258Z  INFO mistralrs_core::pipeline::gguf: Loading `"config.json"` locally at `".\\config.json"`
2024-05-17T07:30:00.814412Z  INFO hf_hub: Token file not found "C:\\Users\\[REDACTED]\\.cache\\huggingface\\token"
2024-05-17T07:30:00.814505Z  INFO mistralrs_core::utils::tokens: Could not load token at "C:\\Users\\[REDACTED]/.cache/huggingface/token", using no HF token.
2024-05-17T07:30:01.022055Z  INFO mistralrs_core::pipeline: Loading `".\\Phi-3-mini-4k-instruct-Q6_K.gguf"` locally at `".\\.\\Phi-3-mini-4k-instruct-Q6_K.gguf"`
thread 'main' panicked at mistralrs-core\src\pipeline\gguf.rs:290:58:
RequestError(Status(401, Response[status: 401, status_text: Unauthorized, url: https://huggingface.co/api/models/revision/main]))
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

I wanted to give mistral.rs a shot, but it's a really painful experience for now.
Latest commit
ca9bf7d1a8a67bd69a3eed89841a106d2e518c45 (v0.1.8)

Did you add the HuggingFace Token? I got the same error RequestError(Status(401, Response[status: 401, status_text: Unauthorized, url: https://huggingface.co/api/models/revision/main])) until I added the token.
Here are the ways you can add it https://github.com/EricLBuehler/mistral.rs?tab=readme-ov-file#getting-models-from-hf-hub.
Also, this chat helped me as I was getting a 403 Error after that https://discuss.huggingface.co/t/error-403-what-to-do-about-it/12983. I had to accept the Llama license.

@joshpopelka20 I want to run model from a local GGUF file, only - exactly the same way as in llama.cpp. Communication with HF (or any other) servers shouldn't ever be required for that.

A recent issue also showed UX issues with this: https://github.com/EricLBuehler/mistral.rs/issues/295#issuecomment-2106450931
I found the README a bit confusing too vs llama-cpp for local GGUF (it doesn't help that it refers to terms you need to configure, but then uses short option names, the linked CLI args output also appears outdated from what a git build shows).
I was not able to use absolute or relative paths in a way that mistral.rs would seem to understand / accept, so based on the linked issue above, I had to ensure the binary was adjacent to the model (and forced tokenizer.json + config.json files)...

Still like you it fails, but here is the extra output for knowing why:
$ RUST_BACKTRACE=1 ./mistralrs-server --token-source none gguf -m . -t . -f model.gguf

2024-05-18T01:41:28.388727Z  INFO mistralrs_server: avx: true, neon: false, simd128: false, f16c: true
2024-05-18T01:41:28.388775Z  INFO mistralrs_server: Sampling method: penalties -> temperature -> topk -> topp -> multinomial
2024-05-18T01:41:28.388781Z  INFO mistralrs_server: Loading model `.` on Cuda(CudaDevice(DeviceId(1)))...
2024-05-18T01:41:28.388828Z  INFO mistralrs_server: Model kind is: quantized from gguf (no adapters)
2024-05-18T01:41:28.388869Z  INFO hf_hub: Token file not found "/root/.cache/huggingface/token"
2024-05-18T01:41:28.658484Z  INFO mistralrs_core::pipeline::gguf: Loading `"tokenizer.json"` locally at `"./tokenizer.json"`
2024-05-18T01:41:29.024145Z  INFO mistralrs_core::pipeline::gguf: Loading `"config.json"` locally at `"./config.json"`
2024-05-18T01:41:29.024256Z  INFO hf_hub: Token file not found "/root/.cache/huggingface/token"
2024-05-18T01:41:29.333151Z  INFO mistralrs_core::pipeline: Loading `"model.gguf"` locally at `"./model.gguf"`
thread 'main' panicked at mistralrs-core/src/pipeline/gguf.rs:290:58:
RequestError(Status(401, Response[status: 401, status_text: Unauthorized, url: https://huggingface.co/api/models/revision/main]))
stack backtrace:
   0: rust_begin_unwind
   1: core::panicking::panic_fmt
   2: <mistralrs_core::pipeline::gguf::GGUFLoader as mistralrs_core::pipeline::Loader>::load_model_from_hf
   3: tokio::runtime::context::runtime::enter_runtime
   4: tokio::runtime::runtime::Runtime::block_on
   5: mistralrs_server::main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

Note that --token-source none has no effect (it must be given prior to the gguf subcommand where it is not considered a valid option), the code path still goes through load_model_from_hf (which will then forward to load_model_from_path if it didn't panic):
https://github.com/EricLBuehler/mistral.rs/blob/ca9bf7d1a8a67bd69a3eed89841a106d2e518c45/mistralrs-core/src/pipeline/gguf.rs#L303
https://github.com/EricLBuehler/mistral.rs/blob/ca9bf7d1a8a67bd69a3eed89841a106d2e518c45/mistralrs-core/src/pipeline/mod.rs#L180-L187
https://github.com/EricLBuehler/mistral.rs/blob/ca9bf7d1a8a67bd69a3eed89841a106d2e518c45/mistralrs-core/src/pipeline/mod.rs#L210
--token-source does accept none and refuses None or not-a-valid-variant, so I'm not sure why the output suggests it's still trying to use the default TokenSource::CacheToken? But it should be TokenSource::None (EDIT: Confirmed, this is a check hf_hub crate does regardless)
https://github.com/EricLBuehler/mistral.rs/blob/ca9bf7d1a8a67bd69a3eed89841a106d2e518c45/mistralrs-server/src/main.rs#L94-L95

Initial attempt
Let's follow the problem from CLI to hugging face API call with the token:

EDIT: Collapsed for brevity (not relevant)
https://github.com/EricLBuehler/mistral.rs/blob/ca9bf7d1a8a67bd69a3eed89841a106d2e518c45/mistralrs-server/src/main.rs#L245
https://github.com/EricLBuehler/mistral.rs/blob/ca9bf7d1a8a67bd69a3eed89841a106d2e518c45/mistralrs-server/src/main.rs#L284-L296
https://github.com/EricLBuehler/mistral.rs/blob/ca9bf7d1a8a67bd69a3eed89841a106d2e518c45/mistralrs-core/src/pipeline/gguf.rs#L278-L303
https://github.com/EricLBuehler/mistral.rs/blob/ca9bf7d1a8a67bd69a3eed89841a106d2e518c45/mistralrs-core/src/pipeline/macros.rs#L133-L138
https://github.com/EricLBuehler/mistral.rs/blob/ca9bf7d1a8a67bd69a3eed89841a106d2e518c45/mistralrs-core/src/pipeline/gguf.rs#L31
https://github.com/EricLBuehler/mistral.rs/blob/ca9bf7d1a8a67bd69a3eed89841a106d2e518c45/mistralrs-core/src/utils/tokens.rs#L15-L18
https://github.com/EricLBuehler/mistral.rs/blob/ca9bf7d1a8a67bd69a3eed89841a106d2e518c45/mistralrs-core/src/utils/tokens.rs#L53
The macro calls the huggingface API and adds the token .with_token(Some(get_token($token_source)?)), which is an empty string for TokenSource::None. The upstream crate .with_token() expects an option, so it's being wrapped with Some() to always assume a valid token is provided to it?
https://docs.rs/hf-hub/latest/hf_hub/api/sync/struct.ApiBuilder.html#method.with_token
https://github.com/huggingface/hf-hub/blob/9d6502f5bc2e69061c132f523c76a76dad470477/src/api/sync.rs#L143-L157
    /// Sets the token to be used in the API
    pub fn with_token(mut self, token: Option<String>) -> Self {
        self.token = token;
        self
    }

    fn build_headers(&self) -> HeaderMap {
        let mut headers = HeaderMap::new();
        let user_agent = format!("unkown/None; {NAME}/{VERSION}; rust/unknown");
        headers.insert(USER_AGENT, user_agent);
        if let Some(token) = &self.token {
            headers.insert(AUTHORIZATION, format!("Bearer {token}"));
        }
        headers
    }

Because the empty string was passed in, it passes that conditional and we add the authorization HTTP header with Bearer " (empty value). If it was instead None value passed to the API here, it'd skip the header and that error would be avoided.
Next up, back in mistral.rs with that same macro, the expected tokenizer.json and config.json files are presumably being enforced by the logic here:
https://github.com/EricLBuehler/mistral.rs/blob/ca9bf7d1a8a67bd69a3eed89841a106d2e518c45/mistralrs-core/src/pipeline/macros.rs#L147-L154

Workaround
I'm terrible at debugging, so I sprinkled a bunch of info! lines to track where the logic in the macro was failing:
https://github.com/EricLBuehler/mistral.rs/blob/ca9bf7d1a8a67bd69a3eed89841a106d2e518c45/mistralrs-core/src/pipeline/macros.rs#L180-L191
The api_dir_list at this point fails with due to the 401 response:
https://github.com/EricLBuehler/mistral.rs/blob/ca9bf7d1a8a67bd69a3eed89841a106d2e518c45/mistralrs-core/src/pipeline/macros.rs#L2-L18
I'm not familiar with what this part of the code is trying to do, but for local/offline use the HF API shouldn't be queried at all... but it seems to be enforced?
Since 404 errors have an exception to panic, if you do the same for 401 then it's happy (tokenizer_config.json must also be provided though):
https://github.com/EricLBuehler/mistral.rs/blob/ca9bf7d1a8a67bd69a3eed89841a106d2e518c45/mistralrs-core/src/pipeline/macros.rs#L15
# Example:
if resp.into_response().is_some_and(|r| !(matches!(r.status(), 401 | 404))) {

Proper solution is probably to opt-out of the HF API entirely though?

Hi @MoonRide303!
Our close integration with the HF hub is intentional, as generally it is better to use the official tokenizer. However, I agree that it would be nice to enable loading from only a GGUF file. I'll begin work on this, and it shouldn't be too hard.
@polarathene:

Note that --token-source none has no effect (it must be given prior to the gguf subcommand where it is not considered a valid option), the code path still goes through load_model_from_hf (which will then forward to load_model_from_path if it didn't panic):

I think this behavior can be improved, I'll make a modification.

Agree with https://github.com/EricLBuehler/mistral.rs/issues/326#issuecomment-2119166130. The prior PR change was the minimal chnages needed to load a known HF model from local. It is an awkward UX to have for a local-only model.

I think there is a strong use case for loading from file without access to hugging face. HF is good! But, if you're trying to use an LLM in production, it's another failure point if your access to HF goes down. Also, there is always the risk that the creators of the LLM model might deny access to the repo at some point in the future.
Anyways, trying to get this to work locally now with the rust library. load_model_from_path requires the ModelPaths object, which doesn't seem to be importable from mistralrs/src/lib.rs.

I think there is a strong use case for loading from file without access to hugging face. HF is good! But, if you're trying to use an LLM in production, it's another failure point if your access to HF goes down. Also, there is always the risk that the creators of the LLM model might deny access to the repo at some point in the future.

Yes, especially when using a GGUF file as otherwise, there is always ISQ. I'm working on adding this in #345.

Anyways, trying to get this to work locally now with the rust library. load_model_from_path requires the ModelPaths object, which doesn't seem to be importable from mistralrs/src/lib.rs.

Ah, sorry, that was an oversight. I just merged #348, which both exposes those, and also exposes the Device, DType and a few other useful types so that you do not need to explicitly depend on our Candle branch.

@MoonRide303, @polarathene, @Jeadie, @joshpopelka20, @ShelbyJenkins
I just merged #345, which enables using the GGUF tokenizer. The implementation is tested against the HF tokenizer in CI, so you have a guarantee that it is correct. This is the applicable readme section.
Here is an example:
cargo run --release --features ... -- -i --chat-template <chat_template> gguf -m . -f Phi-3-mini-128k-instruct-q4_K_M.gguf

I would appreciate your thoughts on how this can be improved!

Yes, unfortunately, that's a problem higher up in the dependency graph. There's a PR for that here: coreylowman/cudarc#238, and I'll let you know when it gets merged.

I am currently working on a project where I need to use the gguf model locally. However, I I am not very familiar with calling Rust libraries directly.
Could you please provide an example of how to invoke gguf locally in Rust? A simple example would be very helpful for my understanding.
Thank you for your assistance!

Could you please provide an example of how to invoke gguf locally in Rust? A simple example would be very helpful for my understanding.

Absolutely, here is a simple example of running a GGUF model purely locally:

https://github.com/EricLBuehler/mistral.rs/blob/9273f2a9157ae9f646d08a3f91e799548c700765/mistralrs/examples/gguf_locally/main.rs#L10-L64

Please feel free to let me know if you have any questions!

@EricLBuehler
Thank you for providing the example!
I tried running it, but I encountered the following error:
RequestError(Status(401, Response[status: 401, status_text: Unauthorized, url: https://huggingface.co/api/models/revision/main]))

Since this is a local example, I assumed that the HuggingFace Token wouldn't be necessary. Is this not the case?

HI @solaoi, that should be fixed now, can you please try it again after a git pul?

@MoonRide303

Not strictly related to this issue, but I updated to current (12.5) CUDA version few days ago, and mistral.rs (as of v0.1.11) no longer compiles.

cudarc just merged 12.5 support, so this should compile now.

@EricLBuehler Pros: it compiles. Cons: doesn't work.
mistralrs-server.exe gguf -m . -f Mistral-7B-Instruct-v0.3-Q6_K.gguf
thread 'main' panicked at C:\Users\[REDACTED]\.cargo\registry\src\index.crates.io-6f17d22bba15001f\cudarc-0.11.2\src\driver\sys\mod.rs:43:71:
called `Result::unwrap()` on an `Err` value: LoadLibraryExW { source: Os { code: 126, kind: Uncategorized, message: "Nie można odnaleźć określonego modułu." } }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Not sure what might be causing this - but llama.cpp compiles and works without issues, so I'd assume my env is fine.

@MoonRide303, it looks like you are using Windows. This issue has been reported here (coreylowman/cudarc#219) and here (huggingface/candle#2175). Can you add the path to your libcuda.so to LD_LIBRARY_PATH?

@EricLBuehler .so ELFs and LD_LIBRARY_PATH won't work on Windows. I am compiling and using CUDA-accelerated llama.cpp builds without issues, so CUDA .dlls should be in my path already.

Right, sorry, my mistake. On Windows, do you know if you have multiple CUDA installations, can you run:
$ dir /c/Program Files/NVIDIA GPU Computing Toolkit/CUDA/

@EricLBuehler Got some, but those were just empty dirs from old versions:
$ ls "/c/Program Files/NVIDIA GPU Computing Toolkit/CUDA/"
v11.7/  v11.8/  v12.1/  v12.4/  v12.5/

I removed all except 12.5, and it didn't help in any way. But it shouldn't matter as long as necessary .dll are in the path (and they are).

that should be fixed now, can you please try it again after a git pull?

I did a fresh git clone and build in a container, so I'm not sure why I'm still encountering the 401.
@EricLBuehler I assume the reason you're not experiencing that is because the default for --token-source is cache which you've probably been utilizing? You need to add --token-source none and should also hit the same problem?
I pointed out the 401 issue earlier. It can be bypassed after a patch, but proper solution would be to skip calling out to HF in the first place?

The loaders HF method isn't doing much beyond getting the paths and then calling the local method directly after implicitly with that extra data?:
https://github.com/EricLBuehler/mistral.rs/blob/527e7f5282c991d399110e21ddbef6c51bba607c/mistralrs-core/src/pipeline/gguf.rs#L282-L291
What is the actual minimum amount of paths needed? Can that whole method be skipped if paths are provided locally? Is the chat template required or can it fallback to a default (perhaps with warning)? I'm not sure what llama-cpp does but their local GGUF loader support doesn't require much upfront to run.
Otherwise this macro is presumably juggling conditions of API call vs fallback/alternative? (also I'm not too fond of duplicating the macro to adjust for local GGUF):
https://github.com/EricLBuehler/mistral.rs/blob/527e7f5282c991d399110e21ddbef6c51bba607c/mistralrs-core/src/pipeline/macros.rs#L156-L244
Ah I see the paths struct here:
https://github.com/EricLBuehler/mistral.rs/blob/527e7f5282c991d399110e21ddbef6c51bba607c/mistralrs-core/src/pipeline/mod.rs#L98-L111
How about this?:

Some condition to opt-out of HF API when providing local file paths?
Based on that condition handle either:

Assign any user supplied paths from CLI
Get all paths info via HF API

Apply fallback paths for any mandatory paths that are still None, or fail.
load_model_from_path() can be called now. The load_model_from_hf() would be changed to only return the paths, instead of minor convenience of calling load_model_from_path() internally?

I am a bit more familiar with this area of the project now, I might be able to take a shot at it once my active PR is merged 😅
Original response
Perhaps I am not using the command correctly:

Attempts
Can ignore most of this, mistralrs-server gguf -m . -f model.gguf fails with 401 unauthorized.

From the mistral.rs git repo at /mist, absolute path to model location for -m (EDIT: this probably should have been part of -f):
$ RUST_BACKTRACE=1 target/release/mistralrs-server gguf -m /models/Hermes-2-Pro-Mistral-7B.Q4_K_M -f Hermes-2-Pro-Mistral-7B.Q4_K_M.gguf

2024-05-30T00:06:47.470010Z  INFO mistralrs_core::pipeline::gguf: Loading model `/models/Hermes-2-Pro-Mistral-7B.Q4_K_M` on Cuda(CudaDevice(DeviceId(1)))...
2024-05-30T00:06:47.508433Z  INFO mistralrs_core::pipeline::gguf: Model config:
general.architecture: llama
general.file_type: 15
general.name: jeffq
general.quantization_version: 2
llama.attention.head_count: 32
llama.attention.head_count_kv: 8
llama.attention.layer_norm_rms_epsilon: 0.00001
llama.block_count: 32
llama.context_length: 32768
llama.embedding_length: 4096
llama.feed_forward_length: 14336
llama.rope.dimension_count: 128
llama.rope.freq_base: 10000
thread 'main' panicked at mistralrs-core/src/pipeline/gguf_tokenizer.rs:65:31:
no entry found for key
stack backtrace:
   0: rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::panicking::panic_display
   3: core::option::expect_failed
   4: <mistralrs_core::pipeline::gguf::GGUFLoader as mistralrs_core::pipeline::Loader>::load_model_from_path
   5: <mistralrs_core::pipeline::gguf::GGUFLoader as mistralrs_core::pipeline::Loader>::load_model_from_hf
   6: tokio::runtime::context::runtime::enter_runtime
   7: mistralrs_server::main

Error "no entry found for key"

From the model directory, absolute path to mistralrs-server:
$ RUST_BACKTRACE=1 /mist/target/release/mistralrs-server gguf -m . -f Hermes-2-Pro-Mistral-7B.Q4_K_M.gguf

thread 'main' panicked at mistralrs-core/src/pipeline/gguf.rs:282:58:
RequestError(Status(401, Response[status: 401, status_text: Unauthorized, url: https://huggingface.co/api/models/revision/main]))
stack backtrace:
   0: rust_begin_unwind
   1: core::panicking::panic_fmt
   2: <mistralrs_core::pipeline::gguf::GGUFLoader as mistralrs_core::pipeline::Loader>::load_model_from_hf
   3: tokio::runtime::context::runtime::enter_runtime
   4: mistralrs_server::main

401 unauthorized.

Just to double check I copied the mistralrs-server executable to the same folder which is how I previously tried to run it in past comments:
$ RUST_BACKTRACE=1 ./server gguf -m . -f Hermes-2-Pro-Mistral-7B.Q4_K_M.gguf

thread 'main' panicked at mistralrs-core/src/pipeline/gguf.rs:282:58:
RequestError(Status(401, Response[status: 401, status_text: Unauthorized, url: https://huggingface.co/api/models/revision/main]))
stack backtrace:
   0: rust_begin_unwind
   1: core::panicking::panic_fmt
   2: <mistralrs_core::pipeline::gguf::GGUFLoader as mistralrs_core::pipeline::Loader>::load_model_from_hf
   3: tokio::runtime::context::runtime::enter_runtime
   4: mistralrs_server::main

401 unauthorized again.

Adding --token-source none doesn't help. Nor -t ., both were the two other args I had originally used in past comments but AFAIK aren't necessary anymore? Produces same error as above with 401.
If I don't use -t ., but use -m with an absolute path to the directory for the -f it'll give the first output I shared in this comment. So I figured maybe the same for the -t which then results in:

$ RUST_BACKTRACE=1 ./server --token-source none gguf -m /models/Hermes-2-Pro-Mistral-7B.Q4_K_M -t /models/Hermes-2-Pro-Mistral-7B.Q4_K_M -f Hermes-2-Pro-Mistral-7B.Q4_K_M.gguf

thread 'main' panicked at mistralrs-core/src/pipeline/gguf.rs:282:58:
RequestError(Status(404, Response[status: 404, status_text: Not Found, url: https://huggingface.co//models/Hermes-2-Pro-Mistral-7B.Q4_K_M/resolve/main/tokenizer.json]))
stack backtrace:
   0: rust_begin_unwind
   1: core::panicking::panic_fmt
   2: <mistralrs_core::pipeline::gguf::GGUFLoader as mistralrs_core::pipeline::Loader>::load_model_from_hf::{{closure}}
   3: <mistralrs_core::pipeline::gguf::GGUFLoader as mistralrs_core::pipeline::Loader>::load_model_from_hf
   4: tokio::runtime::context::runtime::enter_runtime
   5: mistralrs_server::main

So it's still trying to connect to HF 🤷♂ (because of the mandatory -m arg I guess when I don't use .)
This model was one that you mentioned had a duplicate field (that error isn't being encountered here, although previously I had to add a patch to bypass a 401 panic, which you can see above).

@MoonRide303, @polarathene, the following command works on my machine after I merged #362:
cargo run --release --features cuda -- -i --token-source none --chat-template chat_templates/mistral.json gguf -m . -f mistral-7b-instruct-v0.1.Q4_K_M.gguf

Note: as documented in the README here, you need to specify the model id, file, and chat template when loading a local GGUF model without using the HF tokenizer. If you are using the HF tokenizer, you may specify -t/--tok-model-id which is a HF/local model ID to the tokenizer.json and tokenizer_config.json.

The loaders HF method isn't doing much beyond getting the paths and then calling the local method directly after implicitly with that extra data?:

Yes, it just queries the HTTP side and if that failes treats them as local paths. My thinking was that we should always try HTTP first, but maybe you can flip that in a future PR?

Otherwise this macro is presumably juggling conditions of API call vs fallback/alternative?

Not really, the api_dir_list! and api_get_file! macros handle that. get_paths_gguf! just handles the tokenizer loading differences between GGUF and anything else. I don't love it though, maybe we can use akin or something like that to deduplicate it? I haven't looked into that area.

Some condition to opt-out of HF API when providing local file paths?

That seems like a great idea, perhaps --local in the CLI and a flag in the MistralRs builder so that the Python and Rust APIs can accept it? Happy to accept a PR for that too.

Tried a few other GGUF models from HF and some also failed with no entry found for key, yet these seem to work with llama-cpp so probably not quite there yet? 🤔 (Here's the Hermes model I tried that gave the above failure)

Yeah, we only support the llama/replit GGUF tokenizer model for now as they are both unigram. I just merged #363, which allows the default unigram UNK token (0) in case it is missing. After I merge #356, I'll add support for chat template via the GGUF file (I don't want to cause any more rebases :)), but until then, you should provide the Hermes chat template in the chat template file. In this case, it would be:
{
    "chat_template": "{{bos_token}}{% for message in messages %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}"
}

@MoonRide303, I think it should be fixed now, https://github.com/coreylowman/cudarc/pull/240 was merged and there are reports that it works for others. Can you please try it again after a git pull and cargo update?

Yeah, we only support the llama/replit GGUF tokenizer model for now as they are both unigram.

I don't know much about these, but after a git pull I can confirm Hermes is working now while the models that aren't now report an error about the actual tokenizer missing support which is good to see 👍

I don't know much about these, but after a git pull I can confirm Hermes is working now while the models that aren't now report an error about the actual tokenizer missing support which is good to see 👍

Great! For reference, see these docs: https://github.com/ggerganov/ggml/blob/master/docs/gguf.md#ggml

@EricLBuehler tested (full rebuild) on current master (1d21c5f2d8a75545135741d615fbd7c41106d5d7), result:
mistralrs-server.exe gguf -m . -f Mistral-7B-Instruct-v0.3-Q6_K.gguf

thread 'main' panicked at C:\Users\[REDACTED]\.cargo\registry\src\index.crates.io-6f17d22bba15001f\cudarc-0.11.3\src\curand\sys\mod.rs:51:9:
Unable to find curand lib under the names ["curand", "curand64", "curand64_12", "curand64_125", "curand64_125_0", "curand64_120_5"]. Please open GitHub issue.
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Proper name for this one (as of CUDA 12.5 on Windows) should be:
$ which curand64_10.dll
/c/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.5/bin/curand64_10.dll

@MoonRide303 I opened an issue: coreylowman/cudarc#242. I'll let you know when a fix gets merged.

@EricLBuehler I've noticed the change in cudarc was merged, so I tried to rebuild - it seems problems with CUDA dlls are now solved. But it still asks me for tokenizer_config.json file:
mistralrs-server.exe gguf -m . -f Mistral-7B-Instruct-v0.3-Q6_K.gguf
thread 'main' panicked at mistralrs-core\src\pipeline\gguf.rs:294:58:
File "tokenizer_config.json" not found at model id "."
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

When I copied this config from the original HF repo and added port parameter, I managed to launch the server:
mistralrs-server.exe -p 1234 gguf -m . -f Mistral-7B-Instruct-v0.3-Q6_K.gguf
2024-06-01T20:00:38.909745Z  INFO mistralrs_core::pipeline::gguf: Loading model `.` on Cuda(CudaDevice(DeviceId(1)))...
2024-06-01T20:00:39.046703Z  INFO mistralrs_core::pipeline::gguf: Model config:
general.architecture: llama
general.file_type: 18
general.name: Mistral-7B-Instruct-v0.3
general.quantization_version: 2
llama.attention.head_count: 32
llama.attention.head_count_kv: 8
llama.attention.layer_norm_rms_epsilon: 0.00001
llama.block_count: 32
llama.context_length: 32768
llama.embedding_length: 4096
llama.feed_forward_length: 14336
llama.rope.dimension_count: 128
llama.rope.freq_base: 1000000
llama.vocab_size: 32768
2024-06-01T20:00:39.072195Z  INFO mistralrs_core::pipeline::gguf_tokenizer: GGUF tokenizer model is `llama`, kind: `unigram`, num tokens: 32768, num added tokens: 0, num merges: 0, num scores: 32768
2024-06-01T20:00:40.665088Z  INFO mistralrs_core::pipeline::paths: `tokenizer_config.json` does not contain a chat template, attempting to use specified JINJA chat template.
2024-06-01T20:00:40.665937Z  INFO mistralrs_core::pipeline::paths: No specified chat template. No chat template will be used. Only prompts will be accepted, not messages.
2024-06-01T20:00:40.718935Z  INFO mistralrs_core::pipeline::chat_template: bos_toks = "<s>", eos_toks = "</s>", unk_tok = <unk>
2024-06-01T20:00:40.721333Z  INFO mistralrs_server: Model loaded.
2024-06-01T20:00:40.721698Z  INFO mistralrs_core: GEMM reduced precision in BF16 not supported.
2024-06-01T20:00:40.747338Z  INFO mistralrs_core::cublaslt: Initialized cuBLASlt handle
2024-06-01T20:00:40.748197Z  INFO mistralrs_server: Serving on http://<IP_ADDRESS>:1234.

Though it seems it didn't read chat template at all (not from the GGUF, and not from the separate config file). Trying to chat after running python chat.py (from examples/server/) resulted in
openai.UnprocessableEntityError: Error code: 422 - {'message': 'Received messages for a model which does not have a chat template. Either use a different model or pass a single string as the prompt'}

Great work! I just re-started implementing for my crate. For reference I'm giving two options for loading GGUFs. Both are designed to be as easy as possible (for the Python immigrants).
Option 1)  from presets  with pre-downloaded tokenizer.json, tokenizer_config, and config.json. Given the user's vram, it then downloads the largest quant that will fit in the vram. The tokenizer.json is no longer required since you've implemented tokenizer from GGUF (legend), and presumably I can use that interface or use the code in my crate for my tokenizer needs.
Option 2) User supplies the full hf url or file path to the quantized model.
Option 3) not implemented - safe tensors -> ISQ :D
I then plan to pass those paths after loading to mistral.rs just like I currently do with llama.cpp.

IMO the GGUFLoaderBuilder or the GGUFLoader interface could be simplified for clarity/discoverability and maintenance.

Split into two code paths - one for loading from hf_hub, and one from loading from local.
Or combine with something like this

impl GGUFLoaderBuilder {
    pub fn new() -> Self {
        let kind = ModelKind::Quantized {
            quant: QuantizationKind::Gguf,
        };

        Self {
            config: GGUFSpecificConfig { repeat_last_n: 64 },
            kind,
            ..Default::default()
        }
    }
    /// Load the model from Hugging Face.
    /// A full path to the quantized models download link
    /// OR it could be repo_id/model_id + filename
    pub fn from_hf(mut self, model_url: &str) -> Self {
        self.model_url = Some(model_url.to_string());
    }

    /// Load the model from a local path.
    pub fn from_local_path(mut self, model_path: &str) -> Self {
        self.model_path = Some(model_path.to_string());
    }

    /// A config for a GGUF loader.
    pub fn repeat_last_n(mut self, repeat_last_n: usize) -> Self {
        self.config.repeat_last_n = repeat_last_n;
        self
    }
    /// The Jinja chat template to use.
    /// Not required if loading chat template from GGUF implemented.
    pub fn chat_template(mut self, chat_template: &str) -> Self {
        self.chat_template = Some(chat_template.to_string());
        self
    }
    /// The file name and path of the chat template json file
     /// Not required if loading chat template from GGUF implemented.
    pub fn chat_template_path(mut self, chat_template_path: &str) -> Self {
        self.chat_template = Some(chat_template_path.to_string());
        self
    }
    pub fn build(self) ...
}

I would implement this and submit a PR, but I haven't looked at the downstream code enough to understand the implications. If you think it's a good idea, I can make an attempt.
On chat templates - If we could implement loading the chat template from GGUF, I'm not sure we'd need any thing else. The reason llama.cpp doesn't do this is because they don't/won't implement a Jinja parser, but as most models are including the chat template in the GGUF now I'm not sure if there is a reason to manually load it. I don't mind manually adding a chat_template.json to the presets I have, but it makes loading a model from a single file more difficult. Another option might be to accept the chat template as a string.

@EricLBuehler I've noticed the change in cudarc was merged, so I tried to rebuild - it seems problems with CUDA dlls are now solved. But it still asks me for tokenizer_config.json file:
mistralrs-server.exe gguf -m . -f Mistral-7B-Instruct-v0.3-Q6_K.gguf
thread 'main' panicked at mistralrs-core\src\pipeline\gguf.rs:294:58:
File "tokenizer_config.json" not found at model id "."
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

When I copied this config from the original HF repo and added port parameter, I managed to launch the server:
mistralrs-server.exe -p 1234 gguf -m . -f Mistral-7B-Instruct-v0.3-Q6_K.gguf
2024-06-01T20:00:38.909745Z  INFO mistralrs_core::pipeline::gguf: Loading model `.` on Cuda(CudaDevice(DeviceId(1)))...
2024-06-01T20:00:39.046703Z  INFO mistralrs_core::pipeline::gguf: Model config:
general.architecture: llama
general.file_type: 18
general.name: Mistral-7B-Instruct-v0.3
general.quantization_version: 2
llama.attention.head_count: 32
llama.attention.head_count_kv: 8
llama.attention.layer_norm_rms_epsilon: 0.00001
llama.block_count: 32
llama.context_length: 32768
llama.embedding_length: 4096
llama.feed_forward_length: 14336
llama.rope.dimension_count: 128
llama.rope.freq_base: 1000000
llama.vocab_size: 32768
2024-06-01T20:00:39.072195Z  INFO mistralrs_core::pipeline::gguf_tokenizer: GGUF tokenizer model is `llama`, kind: `unigram`, num tokens: 32768, num added tokens: 0, num merges: 0, num scores: 32768
2024-06-01T20:00:40.665088Z  INFO mistralrs_core::pipeline::paths: `tokenizer_config.json` does not contain a chat template, attempting to use specified JINJA chat template.
2024-06-01T20:00:40.665937Z  INFO mistralrs_core::pipeline::paths: No specified chat template. No chat template will be used. Only prompts will be accepted, not messages.
2024-06-01T20:00:40.718935Z  INFO mistralrs_core::pipeline::chat_template: bos_toks = "<s>", eos_toks = "</s>", unk_tok = <unk>
2024-06-01T20:00:40.721333Z  INFO mistralrs_server: Model loaded.
2024-06-01T20:00:40.721698Z  INFO mistralrs_core: GEMM reduced precision in BF16 not supported.
2024-06-01T20:00:40.747338Z  INFO mistralrs_core::cublaslt: Initialized cuBLASlt handle
2024-06-01T20:00:40.748197Z  INFO mistralrs_server: Serving on http://<IP_ADDRESS>:1234.

Though it seems it didn't read chat template at all (not from the GGUF, and not from the separate config file). Trying to chat after running python chat.py (from examples/server/) resulted in
openai.UnprocessableEntityError: Error code: 422 - {'message': 'Received messages for a model which does not have a chat template. Either use a different model or pass a single string as the prompt'}

@MoonRide303 that was a bug and should be fixed now.

Great work! I just re-started implementing for my crate. For reference I'm giving two options for loading GGUFs. Both are designed to be as easy as possible (for the Python immigrants).

Hi @ShelbyJenkins! That's exciting, I look forward to seeing future developments.

I would implement this and submit a PR, but I haven't looked at the downstream code enough to understand the implications. If you think it's a good idea, I can make an attempt.

That sounds good. If you could also be sure to implement it for the Normal and GGML loaders too, that would be great.

On chat templates - If we could implement loading the chat template from GGUF, I'm not sure we'd need any thing else. The reason llama.cpp doesn't do this is because they don't/won't implement a Jinja parser, but as most models are including the chat template in the GGUF now I'm not sure if there is a reason to manually load it. I don't mind manually adding a chat_template.json to the presets I have, but it makes loading a model from a single file more difficult. Another option might be to accept the chat template as a string.

Yes, that is something which will be easy to do. I'll add support for that in the coming days and will write here when I do.

@ShelbyJenkins I just merged support for multiple GGUF files in #379.

IMO the GGUFLoaderBuilder or the GGUFLoader interface could be simplified for clarity/discoverability and maintenance.

I'll be refactoring this, it's on my todo.

I would implement this and submit a PR, but I haven't looked at the downstream code enough to understand the implications. If you think it's a good idea, I can make an attempt.

That sounds good. If you could also be sure to implement it for the Normal and GGML loaders too, that would be great.

Again heads up that this is something I'll be tackling 😅
My interests are in simplifying the existing code for maintenance, I also am likely to change the ModelKind enum/type as part of this.
After that's been handled, it would be a better time for you to PR your own improvements for UX. Just mentioning this to avoid us both working on conflicting changes at the same time for this portion of mistral.rs.

I just merged support for multiple GGUF files in #379.

I'm having trouble following this requirement?
Where is the example of sharded GGUF? Was there a misunderstanding with what @ShelbyJenkins was describing? (Multiple GGUF files that are distinct by their quantization to support lower VRAM requirements)
Your PR changed quite a bit and I'm lacking clarity as to why it was needed as it seems otherwise redundant complexity?

@EricLBuehler I've noticed the change in cudarc was merged, so I tried to rebuild - it seems problems with CUDA dlls are now solved. But it still asks me for tokenizer_config.json file:
mistralrs-server.exe gguf -m . -f Mistral-7B-Instruct-v0.3-Q6_K.gguf
thread 'main' panicked at mistralrs-core\src\pipeline\gguf.rs:294:58:
File "tokenizer_config.json" not found at model id "."
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

When I copied this config from the original HF repo and added port parameter, I managed to launch the server:
mistralrs-server.exe -p 1234 gguf -m . -f Mistral-7B-Instruct-v0.3-Q6_K.gguf
2024-06-01T20:00:38.909745Z  INFO mistralrs_core::pipeline::gguf: Loading model `.` on Cuda(CudaDevice(DeviceId(1)))...
2024-06-01T20:00:39.046703Z  INFO mistralrs_core::pipeline::gguf: Model config:
general.architecture: llama
general.file_type: 18
general.name: Mistral-7B-Instruct-v0.3
general.quantization_version: 2
llama.attention.head_count: 32
llama.attention.head_count_kv: 8
llama.attention.layer_norm_rms_epsilon: 0.00001
llama.block_count: 32
llama.context_length: 32768
llama.embedding_length: 4096
llama.feed_forward_length: 14336
llama.rope.dimension_count: 128
llama.rope.freq_base: 1000000
llama.vocab_size: 32768
2024-06-01T20:00:39.072195Z  INFO mistralrs_core::pipeline::gguf_tokenizer: GGUF tokenizer model is `llama`, kind: `unigram`, num tokens: 32768, num added tokens: 0, num merges: 0, num scores: 32768
2024-06-01T20:00:40.665088Z  INFO mistralrs_core::pipeline::paths: `tokenizer_config.json` does not contain a chat template, attempting to use specified JINJA chat template.
2024-06-01T20:00:40.665937Z  INFO mistralrs_core::pipeline::paths: No specified chat template. No chat template will be used. Only prompts will be accepted, not messages.
2024-06-01T20:00:40.718935Z  INFO mistralrs_core::pipeline::chat_template: bos_toks = "<s>", eos_toks = "</s>", unk_tok = <unk>
2024-06-01T20:00:40.721333Z  INFO mistralrs_server: Model loaded.
2024-06-01T20:00:40.721698Z  INFO mistralrs_core: GEMM reduced precision in BF16 not supported.
2024-06-01T20:00:40.747338Z  INFO mistralrs_core::cublaslt: Initialized cuBLASlt handle
2024-06-01T20:00:40.748197Z  INFO mistralrs_server: Serving on http://<IP_ADDRESS>:1234.

Though it seems it didn't read chat template at all (not from the GGUF, and not from the separate config file). Trying to chat after running python chat.py (from examples/server/) resulted in
openai.UnprocessableEntityError: Error code: 422 - {'message': 'Received messages for a model which does not have a chat template. Either use a different model or pass a single string as the prompt'}

@MoonRide303 that was a bug and should be fixed now.

Doesn't look fixed as of v0.1.15 (9712da61e89b828925b6e6993a94be2c13a9c09f) - it still asks for tokenizer config in separate file:
mistralrs-server.exe -p 1234 gguf -m . -f Mistral-7B-Instruct-v0.3-Q6_K.gguf
thread 'main' panicked at mistralrs-core\src\pipeline\gguf.rs:272:58:
File "tokenizer_config.json" not found at model id "."
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

And even when provided in separate file (which shouldn't be needed) - yes, the server starts without warnings:
mistralrs-server.exe -p 1234 gguf -m . -f Mistral-7B-Instruct-v0.3-Q6_K.gguf
2024-06-05T06:42:12.616525Z  INFO mistralrs_core::pipeline::gguf: Loading model `.` on Cuda(CudaDevice(DeviceId(1)))...
2024-06-05T06:42:12.801370Z  INFO mistralrs_core::gguf::gguf_tokenizer: GGUF tokenizer model is `llama`, kind: `unigram`, num tokens: 32768, num added tokens: 0, num merges: 0, num scores: 32768
2024-06-05T06:42:20.837169Z  INFO mistralrs_core::pipeline::chat_template: bos_toks = "<s>", eos_toks = "</s>", unk_tok = <unk>
2024-06-05T06:42:20.839905Z  INFO mistralrs_server: Model loaded.
2024-06-05T06:42:20.840295Z  INFO mistralrs_core: GEMM reduced precision in BF16 not supported.
2024-06-05T06:42:20.898012Z  INFO mistralrs_core::cublaslt: Initialized cuBLASlt handle
2024-06-05T06:42:20.899032Z  INFO mistralrs_server: Serving on http://<IP_ADDRESS>:1234.

But the chat template still doesn't work:
python chat.py
Enter system prompt >>> Just be yourself.
>>> Who are you?
Traceback (most recent call last):
  File "D:\repos-git\mistral.rs\examples\server\chat.py", line 47, in <module>
    completion = openai.chat.completions.create(
  File "D:\anaconda3\lib\site-packages\openai\_utils\_utils.py", line 277, in wrapper
    return func(*args, **kwargs)
  File "D:\anaconda3\lib\site-packages\openai\resources\chat\completions.py", line 590, in create
    return self._post(
  File "D:\anaconda3\lib\site-packages\openai\_base_client.py", line 1240, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
  File "D:\anaconda3\lib\site-packages\openai\_base_client.py", line 921, in request
    return self._request(
  File "D:\anaconda3\lib\site-packages\openai\_base_client.py", line 1005, in _request
    return self._retry_request(
  File "D:\anaconda3\lib\site-packages\openai\_base_client.py", line 1053, in _retry_request
    return self._request(
  File "D:\anaconda3\lib\site-packages\openai\_base_client.py", line 1005, in _request
    return self._retry_request(
  File "D:\anaconda3\lib\site-packages\openai\_base_client.py", line 1053, in _retry_request
    return self._request(
  File "D:\anaconda3\lib\site-packages\openai\_base_client.py", line 1020, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.InternalServerError: Error code: 500 - {'message': 'invalid operation: Conversation roles must alternate user/assistant/user/assistant/... (in chat_template:1)'}

@polarathene, great, looking forward to those refactors.
Regarding the sharded GGUF files, here is one such example. That PR also reduced GGUF code duplication by centralizing the get_metadata and loading from a reader.

Hi @MoonRide303, I just merged #385 which adds more verbose and hopefully more helpful logging during loading.
I think there are a few things going on here.

We don't support loading the chat template directly from the GGUF file. As such, you should provide a chat template file or a Hugging Face tokenizer model ID. I will add support for loading the chat template from the GGUF file soon (tracking issue is #386).
The Mistral v0.1 Instruct model does not accept a system prompt:
From the HF Jinja template:

raise_exception('Only user and assistant roles are supported!')

@EricLBuehler Model itself knows how to handle system role, it's just the limitation of the default template. But you're right that's a separate issue, and it can be worked around by providing custom template accepting system role. As of this issue - it seems that #386 is the last missing part, then.

That PR also reduced GGUF code duplication by centralizing the get_metadata and loading from a reader.

I was actually going to handle that (and do it better IMO), but your changes would have caused heavy conflicts to resolve, so I'm somewhat glad I was checking development activity as I was about to start around the time I discovered the merged PR 😓
@ShelbyJenkins if you want to tackle your own improvements to the loader go ahead. The pace of development in this area is a bit too frequent for me to want to try touch it for the time being.

I was actually going to handle that (and do it better IMO), but your changes would have caused heavy conflicts to resolve, so I'm somewhat glad I was checking development activity as I was about to start around the time I discovered the merged PR 😓

I will roll back those changes, looking forward to seeing your implementation!

 The pace of development in this area is a bit too frequent for me to want to try touch it for the time being.

I don't plan on working much in that area for the foreseeable future, aside from the rollback, which I'll do shortly. The pace of development there should not be excessive.

@polarathene @EricLBuehler Ok, I'll take look at this over this week and weekend. It's not a high priority for my project, but I think it's valuable to people consuming this library 👍

Hi @MoonRide303! I just merged #416 which enables loading the GGUF chat template from a GGUF file as well as #397 which adds support for the GPT2 (BPE) tokenizer type which extends support. This command now works:
cargo run --release --features cuda -- --token-source none -i gguf -m . -f Meta-Llama-3-8B-Instruct.Q4_K_M.gguf

Output
2024-06-10T12:38:05.106472Z  INFO mistralrs_server: avx: true, neon: false, simd128: false, f16c: true
2024-06-10T12:38:05.106510Z  INFO mistralrs_server: Sampling method: penalties -> temperature -> topk -> topp -> multinomial
2024-06-10T12:38:05.106523Z  INFO mistralrs_server: Model kind is: quantized from gguf (no adapters)
2024-06-10T12:38:05.132855Z  INFO mistralrs_core::pipeline::paths: Loading `"Meta-Llama-3-8B-Instruct.Q4_K_M.gguf"` locally at `"./Meta-Llama-3-8B-Instruct.Q4_K_M.gguf"`
2024-06-10T12:38:05.180204Z  INFO mistralrs_core::pipeline::gguf: Loading model `.` on Cuda(CudaDevice(DeviceId(1)))...
2024-06-10T12:38:06.002567Z  INFO mistralrs_core::pipeline::gguf: Model config:
general.architecture: llama
general.file_type: 15
general.name: models
general.quantization_version: 2
llama.attention.head_count: 32
llama.attention.head_count_kv: 8
llama.attention.layer_norm_rms_epsilon: 0.00001
llama.block_count: 32
llama.context_length: 8192
llama.embedding_length: 4096
llama.feed_forward_length: 14336
llama.rope.dimension_count: 128
llama.rope.freq_base: 500000
llama.vocab_size: 128256
2024-06-10T12:38:06.281404Z  INFO mistralrs_core::gguf::gguf_tokenizer: GGUF tokenizer model is `gpt2`, kind: `Bpe`, num tokens: 128256, num added tokens: 0, num merges: 280147, num scores: 0
2024-06-10T12:38:06.292431Z  INFO mistralrs_core::gguf::chat_template: Discovered and using GGUF chat template: `{% set loop_messages = messages %}{% for message in loop_messages %}{% set content = '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n'+ message['content'] | trim + '<|eot_id|>' %}{% if loop.index0 == 0 %}{% set content = bos_token + content %}{% endif %}{{ content }}{% endfor %}{% if add_generation_prompt %}{{ '<|start_header_id|>assistant<|end_header_id|>\n\n' }}{% endif %}`
2024-06-10T12:38:08.128146Z  INFO mistralrs_core::pipeline::paths: Using literal chat template.
2024-06-10T12:38:08.399407Z  INFO mistralrs_core::pipeline::chat_template: bos_toks = "<|begin_of_text|>", eos_toks = "<|end_of_text|>", "<|eot_id|>", unk_tok = `None`
2024-06-10T12:38:08.414974Z  INFO mistralrs_server: Model loaded.
2024-06-10T12:38:08.415363Z  INFO mistralrs_core: GEMM reduced precision in BF16 not supported.
2024-06-10T12:38:08.429916Z  INFO mistralrs_core: Enabling GEMM reduced precision in F16.
2024-06-10T12:38:08.432349Z  INFO mistralrs_core::cublaslt: Initialized cuBLASlt handle
2024-06-10T12:38:08.432442Z  INFO mistralrs_server::interactive_mode: Starting interactive loop with sampling params: SamplingParams { temperature: Some(0.1), top_k: Some(32), top_p: Some(0.1), top_n_logprobs: 0, frequency_penalty: Some(0.1), presence_penalty: Some(0.1), stop_toks: None, max_len: Some(4096), logits_bias: None, n_choices: 1 }

cargo run --release --features cuda -- --token-source none -i gguf -m . -f Meta-Llama-3-8B-Instruct.Q4_K_M.gguf

Something is wrong with tokenizer - it fails at What is 3333+777? test (using ac1537d858644aec197b7fe64c761d3251582854):
PS D:\repos-git\mistral.rs> cargo run --release --features cuda -- --token-source none -i gguf -m . -f E:\ML-models\Meta-Llama-3-8B-Instruct-GGUF\Meta-Llama-3-8B-Instruct-Q6_K.gguf
    Finished `release` profile [optimized] target(s) in 0.42s
     Running `target\release\mistralrs-server.exe --token-source none -i gguf -m . -f E:\ML-models\Meta-Llama-3-8B-Instruct-GGUF\Meta-Llama-3-8B-Instruct-Q6_K.gguf`
2024-06-10T14:51:47.647104Z  INFO mistralrs_server: avx: true, neon: false, simd128: false, f16c: true
2024-06-10T14:51:47.647227Z  INFO mistralrs_server: Sampling method: penalties -> temperature -> topk -> topp -> multinomial
2024-06-10T14:51:47.647344Z  INFO mistralrs_server: Model kind is: quantized from gguf (no adapters)
2024-06-10T14:51:47.648949Z  INFO hf_hub: Token file not found "C:\\Users\\[REDACTED]\\.cache\\huggingface\\token"
2024-06-10T14:51:47.649053Z  INFO hf_hub: Token file not found "C:\\Users\\[REDACTED]\\.cache\\huggingface\\token"
2024-06-10T14:51:47.951484Z  INFO mistralrs_core::pipeline::paths: Loading `"E:\\ML-models\\Meta-Llama-3-8B-Instruct-GGUF\\Meta-Llama-3-8B-Instruct-Q6_K.gguf"` locally at `"E:\\ML-models\\Meta-Llama-3-8B-Instruct-GGUF\\Meta-Llama-3-8B-Instruct-Q6_K.gguf"`
2024-06-10T14:51:48.475791Z  INFO mistralrs_core::pipeline::gguf: Loading model `.` on Cuda(CudaDevice(DeviceId(1)))...
2024-06-10T14:51:49.520064Z  INFO mistralrs_core::pipeline::gguf: Model config:
general.architecture: llama
general.file_type: 18
general.name: Meta-Llama-3-8B-Instruct
general.quantization_version: 2
llama.attention.head_count: 32
llama.attention.head_count_kv: 8
llama.attention.layer_norm_rms_epsilon: 0.00001
llama.block_count: 32
llama.context_length: 8192
llama.embedding_length: 4096
llama.feed_forward_length: 14336
llama.rope.dimension_count: 128
llama.rope.freq_base: 500000
llama.vocab_size: 128256
2024-06-10T14:51:49.712946Z  INFO mistralrs_core::gguf::gguf_tokenizer: GGUF tokenizer model is `gpt2`, kind: `Bpe`, num tokens: 128256, num added tokens: 0, num merges: 280147, num scores: 0
2024-06-10T14:51:49.721485Z  INFO mistralrs_core::gguf::chat_template: Discovered and using GGUF chat template: `{% set loop_messages = messages %}{% for message in loop_messages %}{% set content = '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n'+ message['content'] | trim + '<|eot_id|>' %}{% if loop.index0 == 0 %}{% set content = bos_token + content %}{% endif %}{{ content }}{% endfor %}{% if add_generation_prompt %}{{ '<|start_header_id|>assistant<|end_header_id|>\n\n' }}{% endif %}`
2024-06-10T14:51:51.559330Z  INFO mistralrs_core::pipeline::paths: Using literal chat template.
2024-06-10T14:51:51.761652Z  INFO mistralrs_core::pipeline::chat_template: bos_toks = "<|begin_of_text|>", eos_toks = "<|eot_id|>", "<|eot_id|>", unk_tok = `None`
2024-06-10T14:51:51.778165Z  INFO mistralrs_server: Model loaded.
2024-06-10T14:51:51.778496Z  INFO mistralrs_core: GEMM reduced precision in BF16 not supported.
2024-06-10T14:51:51.802510Z  INFO mistralrs_core: Enabling GEMM reduced precision in F16.
2024-06-10T14:51:51.806780Z  INFO mistralrs_core::cublaslt: Initialized cuBLASlt handle
2024-06-10T14:51:51.806864Z  INFO mistralrs_server::interactive_mode: Starting interactive loop with sampling params: SamplingParams { temperature: Some(0.1), top_k: Some(32), top_p: Some(0.1), top_n_logprobs: 0, frequency_penalty: Some(0.1), presence_penalty: Some(0.1), stop_toks: None, max_len: Some(4096), logits_bias: None, n_choices: 1 }
> What is 3333+777?
Let me calculate that for you!

33,333 + 777 = 34,110
>

Same model file used in llama.cpp:

@MoonRide303 that is actually correct :). Closing this as the feature is complete, please feel free to reopen!


Add config params to configure log level

Is there a related GitHub Issue?
#1925
What is this change about?
Adds config params that allows the api and controller log levels to be changed on the fly. The default log levels are info.
Does this PR introduce a breaking change?
no
Acceptance Steps
See issue
Tag your pair, your PM, and/or team
@davewalter

Besides the notes I left above, I do not see the feature being really tested in an integration manner.I wonder how expensive it would be to have a suite that changes the log level and verifies by looking at the logs whether logs reflect the new value.

Thanks for the thoughts/suggestions Danail. We weren't happy with where we left it last night, but wanted to get feedback from CI to see what failed.

Besides the notes I left above, I do not see the feature being really tested in an integration manner.I wonder how expensive it would be to have a suite that changes the log level and verifies by looking at the logs whether logs reflect the new value.

We think it would be difficult to add an e2e test since they don't interact directly with k8s. However, we are adding some tests which verify the functionality of the directory watching component. We will also add tests for the functions that change the log level.

I was wondering if you considered using something like this: https://github.com/spf13/viper#watching-and-re-reading-config-files
It uses fsnotify as well, but seems to handler the low level details of event detection and transmission.

I was wondering if you considered using something like this: https://github.com/spf13/viper#watching-and-re-reading-config-files
It uses fsnotify as well, but seems to handle the low level details of event detection and transmission.

Looks interesting. We've pulled this out into a separate chore.


[8.x] Custom channels for model broadcasting are ignored

Laravel Version: 8.80.0
PHP Version: 7.4.24
Database Driver & Version: not pertinent

Description:
Model broadcasting seemed like a very convenient way for real-time updates on the frontend, so I gave it a try.
Ran into an issue when I wanted to change the channel the events are broadcasted on.
As per docs, I added BroadcastsEvents trait on the model along with
    public function broadcastOn($event)
    {
        return [new PrivateChannel('foo-bar-baz')];
    }

This did not yield expected results, as the events kept broadcasting only on the default model channel.
I dug around for a bit and the culprit seems to be in https://github.com/laravel/framework/blob/8940ff3c9c643f8abcca026a6341fdfcfc83e983/src/Illuminate/Database/Eloquent/BroadcastsEvents.php#L112
We check if (! empty($this->broadcastOn($event)) || ! empty($channels)) { to ensure that there are channels to broadcast on, however:
return broadcast($instance->onChannels(Arr::wrap($channels)));
we completely ignore the channels we defined in broadcastOn. It's only broadcasted on $channels.
Changing https://github.com/laravel/framework/blob/ac6fc32c2933d5868a2c0d2aea4af9405d710378/src/Illuminate/Database/Eloquent/BroadcastsEvents.php#L119
to return broadcast($instance->onChannels(array_merge(Arr::wrap($channels), $this->broadcastOn($event)))); rectifies the issue.
Steps To Reproduce:

Set up model broadcasting
Add broadcastOn to the model definition as shown in the description above
Create/update/delete the model
Model will be sent on the default auto-generated channel instead of the provided one

See the closed PR by Taylor.


Backend code refactoring

A start to #60 and #65.

@markus2330 Is it ok to change the error response in the backend? Previously ResponseBody was used for errors as well. That however has no benefit at all as data was always empty. In this PR I change the response to be just a string. If this is a problem I'll change it back.
Generally I would prefer to not use ResponseBody as the message part of that struct is mostly unnecessary in my opinion. Currently the only thing that is sent in message is ok, but we can just use the status code to convey that the request was successful.
This would however require a change in the frontend as well so I don't know how to handle that.

Is it ok to change the error response in the backend?

Yes, totally. Looking forward to see a nice error concept by a Rust expert! :1st_place_medal:
Can you write a small technical document on how to handle errors in PermaplanT for Rust novices?

mostly unnecessary

When is it necessary? Is it worth to have differences in the APIs in these situations?

This would however require a change in the frontend as well so I don't know how to handle that.

Due to typeshare+typescript it will give you compile errors.

Can you write a small technical document on how to handle errors in PermaplanT for Rust novices?

I would make that part of the high level doc using mdBook.

mostly unnecessary

When is it necessary? Is it worth to have differences in the APIs in these situations?

This would however require a change in the frontend as well so I don't know how to handle that.

Due to typeshare+typescript it will give you compile errors.

Maybe create a separate issue for that.

In general we are missing code comments. Most of the devs are new to Rust, so some explanations of the code are useful.

This will be part of #60 (which I'm already working on). Part of that forces you to write doc for every method in the project.

Please don't self-merge, see also #62. But I was just waiting for your answer and would have merged then, so nothing happened.

Oh sorry about that then. You approved the request so I thought it would be fine.


etcdmain: print out version information on startup

Fix #3332
example output
./etcd
2015-08-20 14:49:29.875955 I | etcdmain: etcd Version: 2.2.0-alpha.1+git
2015-08-20 14:49:29.876090 I | etcdmain: Git SHA: 7cf9770
2015-08-20 14:49:29.876095 I | etcdmain: Go Version: go1.5
2015-08-20 14:49:29.876099 I | etcdmain: Go OS/Arch: darwin/amd64

/cc @brianredbeard

The change lgtm.
#3332 also mentions to use git describe to replace hard-coded version in code. This is good, so it doesn't require to bump the version each time, which saves time. I could handle the work.

I will open an issue myself. LGTM


Control utils 3d cdn is not working

Please make sure that this is a bug and also refer to the troubleshooting, FAQ documentation before raising any issues.
System information (Please provide as much relevant information as possible)

Have I written custom code (as opposed to using a stock example script provided in MediaPipe):
Mostly the same code, just added the angle calculations for my purposes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04, Android 11, iOS 14.4):
Windows
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
Haven't tested on a mobile device
Browser and version (e.g. Google Chrome, Safari) if the issue happens on browser:
Version 97.0.4692.99 (Official Build) (64-bit) google chrome
Programming Language and version ( e.g. C++, Python, Java):
Javascript
MediaPipe version:

Bazel version (if compiling from source):
NA
Solution ( e.g. FaceMesh, Pose, Holistic ):
Pose
Android Studio, NDK, SDK versions (if issue is related to building in Android environment):
NA
Xcode & Tulsi version (if issue is related to building for iOS):
NA
Describe the current behavior:
app.js:6 Uncaught ReferenceError: LandmarkGrid is not defined
at app.js:6:14 in console
Status Code: 404  (from disk cache) in network tab
Describe the expected behavior:
Landmark grid should work
Standalone code to reproduce the issue:
<EMAIL_ADDRESS>- just visit here, 404 for the file.
Provide a reproducible test case that is the bare minimum necessary to replicate the problem. If possible, please share a link to Colab/repo link /any notebook:
<EMAIL_ADDRESS>- fix this
Other info / Complete Logs :
Include any logs or source code that would be helpful to
diagnose the problem. If including tracebacks, please include the full
traceback. Large logs and files should be attached

Hi @faraznisar, Could you please verify control_utils_3d.js is actually loading into your script tag as mentioned here https://github.com/google/mediapipe/issues/2388#issuecomment-895407600

Hi @faraznisar, Could you please verify control_utils_3d.js is actually loading into your script tag as mentioned here #2388 (comment)

There is the entire list of CDNs that I am using.

Also the screenshot of the network tab and as you can see control_utils_3d is not loading.

You are using "drawing_utils/control_utils_3d.js" instead of "control_utils_3d/control_utils_3d.js".
Also, when checking the Network tab, I recommend using "disable cache", since otherwise the browser may not try to make the network fetches again, but may simply return cached responses (e.g. HTTP 304's).

Fix the solution api for javascript on your website.
https://google.github.io/mediapipe/solutions/pose.html#javascript-solution-api

Now I see this issue. I see your comment here https://github.com/google/mediapipe/issues/2388#issuecomment-1033319297
Does this mean that the solution won't work till the new version is released?
If not, is there a workaround for the time being?
In both cases, do we have an ETA on when the new version will be released?

Please refer the "disable cache" recommendation as mentioned in the above https://github.com/google/mediapipe/issues/3055#issuecomment-1033038996. ETA was about the document  update of api fix which is mentioned above.

No effect whatsoever.

The scripts

         <video class="input_video"></video>         <canvas class="output_canvas" width="1280px" height="720px"></canvas>

The HTML
const videoElement = document.getElementsByClassName("input_video")[0]; const canvasElement = document.getElementsByClassName("output_canvas")[0]; const canvasCtx = canvasElement.getContext("3d"); const landmarkContainer = document.getElementsByClassName('landmark-grid-container')[0]; const grid = new LandmarkGrid(landmarkContainer);
The elements
`
function onResults(results) {
if (!results.poseLandmarks) {
grid.updateLandmarks([]);
return;
}
let landmarks = results.poseLandmarks
const canvasCtx = canvasElement.getContext("3d")
canvasCtx.save();
canvasCtx.clearRect(0, 0, canvasElement.width, canvasElement.height);
canvasCtx.drawImage(results.image,
    0,
    0,
    canvasElement.width,
    canvasElement.height
)
drawConnectors(canvasCtx,
    results.poseLandmarks, POSE_CONNECTIONS,
    { color: 'green', lineWidth: 5 });
drawLandmarks(canvasCtx, results.poseLandmarks,
    { color: 'red', lineWidth: 3, radius: 2 });
drawLandmarks.poseLandmarks
canvasCtx.restore();
grid.updateLandmarks(results.poseWorldLandmarks);

}
const pose = new Pose({
locateFile: (file) => {
return https://cdn.jsdelivr.net/npm/@mediapipe/pose/${file};
},
});
pose.setOptions({
modelComplexity: 1,
smoothLandmarks: true,
enableSegmentation: true,
smoothSegmentation: true,
minDetectionConfidence: 0.5,
minTrackingConfidence: 0.5,
});
pose.onResults(onResults);
const camera = new Camera(videoElement, {
onFrame: async () => {
await pose.send({ image: videoElement });
},
width: 1280,
height: 720
});
camera.start();
`
The rest of the code.
I removed the custom code that I needed for my purposes. If needed I will add this to a repository because so far I tried everything you said but still no result.

Hello @faraznisar,
We are upgrading the MediaPipe Legacy Solutions to new MediaPipe solutions However, the libraries, documentation, and source code for all the MediaPipe Legacy Solutions will continue to be available in our GitHub repository and through library distribution services, such as Maven and NPM.
You can continue to use those legacy solutions in your applications if you choose. Though, we would request you to check new MediaPipe solutions which can help you more easily build and customize ML solutions for your applications. These new solutions will provide a superset of capabilities available in the legacy solutions.


Support for subtitle indexing (.scc, .srt, etc)

Allow for indexing of individual / batch of closed captioning files for assets.  Potentially expose via API to allow for MAM integration, and Kibana for search ex: "Search this Phrase" would return all occurrences (timecode, assetid) of the phrase in an asset.

Thank you for your suggestion. We've carefully considered your feedback and have decided not to include this feature in a future release.


Sunrise, sunset, location 'attributes' and the services endpoint.

When using Rules (or mobile app Routines) we can work with Sunrise and Sunset times for our Locations, as well as a selection of Location 'attributes' covering things like temperature and current weather as well as the current modes and security status.
It would help development of Rules if we could get the current values assembled in one place via the CLI. Sunrise and sunset is particularly important as users have become used to seeing that in the IDE with the legacy platform.
(This issue originally had two separate requests. The deleted one has been moved to #493.)

Fresh.


<EMAIL_ADDRESS>Fix hash

Closes #168

All changes look good.
Wait for review from human collaborators.
vlc

[x] Description
[x] License
[x] Hashes
[x] Checkver
[x] Autoupdate
[x] Autoupdate Hash Extraction


Show "No transaction history" for empty wallets

Not sure how we should display the empty wallet notification, if it should be some friendly graphics, icon or text? Suggestions are welcome!

Implemented like this, we can consider some improvements later on, with perhaps some tips on how to buy coins.


Added bio field to the Profile struct

made changes in the following areas
anchor/tests/pubkey-protocol-profile.spec.ts
anchor/programs/pubkey-protocol/src
-utils.rs
- errors.rs
- constant.rs
anchor/programs/pubkey-protocol/src/state

profile.rs
anchor/programs/pubkey-protocol/src/instructions/profile
profile_create.rs
profile update.rs
repo https://github.com/Josh0007-sunday/pubkey-protocol.git
.
This pull request was created for https://app.gib.work/bounties/0d295331-b491-4356-8bc5-046a618dcfd8 in an attempt to solve a bounty #51 . Payment for the bounty is immediately sent to the contributor after merge.

Thanks for giving this a stab! I'm closing the PR as the functionality has been added by #64


Adding array_is_list function

Hope this helps
-- Flo

Thanks. I tidied it up a bit to fit with my own style, and moved it up to the 8.1 features section of the file. I'll make it clearer that I have the file in PHP version order.
Really appreciate the contribution. Thanks.


Helm Chart：Unnecessary RBAC permissions

Description
The bug is that the Deployment dapr-operator in the charts has too much RBAC permission than it needs. The service account of dapr-operator is bound to a clusterrole (operator.yaml#L15) with the following permissions:

get/list verb of the secrets resource (ClusterRole)
delete verb of the services resource (ClusterRole)

After reading the source code of dapr-operator, I didn't find any Kubernetes API usages using list secrets and delete services permissions. Besides, some of these unused permissions may have potential risks. For example, if malicious users gain control of a Kubernetes node running a dapr-operator pod, they can list all the names of the secrets, and with the name, they can get the details of all the secrets objects (since this is declared in a ClusterRole).
Therefore, these permissions should be rechecked to determine if they are truly unnecessary. If they are, the issue should be fixed by removing the unnecessary permissions or or other feasible methods.
To Reproduce
Use helm chart with default values.

Thanks, would you be interested in opening a PR with the changes you think can be done to the cluster roles? We have automated tests on Kubernetes so if anything's not behaving properly we'll know right away.

These ClusterRole's and ClusterRoleBindings are a concern for us too


improvedocx

Community Contribution
Link to live project, video, or image:

Sharing

[ ] I would be happy to have my project shared on The Coding Train's social media!

Instagram:
Twitter:

Thanks for your suggestion! However _config.yml is named that way because it's predetermined by Jekyll. So changing it would break the build system even if it makes things more clear to developers.
If you're interested in contributing to the Coding Train website, I'd suggest contributing a link to a project you've made based on one of our videos. You can see more instructions on doing that in the community contributions guide.
You can also practice using GitHub with our Rainbow-Poem repository which was covered in the Git/GitHub series. Unfortunately there's a rather large backlog of pull requests open there so it's unlikely it that any new pull requests will be merged soon.


Improve division and multiplication mixing process the data

What changes were proposed in this pull request?
spark-sql> select<PHONE_NUMBER>123456789012 /<PHONE_NUMBER>1234567890120) *<PHONE_NUMBER>1234567890120;
NULL
spark-sql> select<PHONE_NUMBER>1234567890 / 123) * 123;
NULL
when the length of the getText is greater than 19, The result is not what we expected.
but mysql handle the value is ok.
mysql> select<PHONE_NUMBER>123456789012 /<PHONE_NUMBER>1234567890120) *<PHONE_NUMBER>1234567890120;
+------------------------------------------------------------------------------+
|<PHONE_NUMBER>123456789012 /<PHONE_NUMBER>1234567890120) *<PHONE_NUMBER>1234567890120 |
+------------------------------------------------------------------------------+
|<PHONE_NUMBER>123456789012.0000 |
+------------------------------------------------------------------------------+
1 row in set (0.00 sec)
mysql>
mysql> select<PHONE_NUMBER>1234567890 / 123) * 123;
+------------------------------------+
|<PHONE_NUMBER>1234567890 / 123) * 123 |
+------------------------------------+
|<PHONE_NUMBER>1234567890.0000 |
+------------------------------------+
1 row in set (0.00 sec)
How was this patch tested?
Supplement the unit test.

Can one of the admins verify this patch?

I don't think this makes any sense:

The parser is not the place where you should make this change. This should be done in TypeCoercion. Now we create different semantics between SQL and the DataFrame API.
This is a breaking change to result type of Divide, and will probably break Hive compatibility. This is going break to user code in a lot of places. I also think it is weird that we are going to always return Double when a Decimal is completely valid.
I think MySql returns a Decimal and not a Double.

So I am going to -1 this.

@hvanhovell
thank you for review it.
This is an extreme case. must be with Longtype type and Decimal type mixing test.
However, when test case:
spark-sql> select<PHONE_NUMBER>123456789012 /<PHONE_NUMBER>1234567890120)
0.1
spark-sql> select 0.1 *<PHONE_NUMBER>1234567890120;
1234567890123456789012
the result is ok.
For this PR, It is not the best treatment method.
close it for a moment.


[Storage] [Typing] [Blob] Misc files (_serialize.py, _deserialize.py, _download.py)

Remaining errors on:
_deserialize.py:
azure\storage\blob_deserialize.py:74: error: Need type annotation for "parsed_result" (hint: "parsed_result: Dict[, ] = ...")  [var-annotated]

For this one I am unsure of the correct annotation as it seems to re-use parsed_result to act as a placeholder for 2 different things, but I couldn't completely decipher how using two variables would give us the same expected behavior.

_download.py:
azure\storage\blob_download.py:137: error: Missing return statement  [return]

Cannot get the return type quite right for yield

azure\storage\blob_download.py:608: error: Incompatible return value type (got "Sequence[object]", expected "bytes")  [return-value]
azure\storage\blob_download.py:608: error: Incompatible return value type (got "Sequence[object]", expected "str")  [return-value]
azure\storage\blob_download.py:637: error: Argument "non_empty_ranges" to "_ChunkDownloader" has incompatible type "Optional[List[Dict[str, int]]]"; expected "List[Any]"  [arg-type]
azure\storage\blob_download.py:668: error: Incompatible return value type (got "str", expected "bytes")  [return-value]
azure\storage\blob_download.py:669: error: Incompatible return value type (got "bytes", expected "str")  [return-value]
azure\storage\blob_download.py:680: error: Argument 1 to "readinto" of "StorageStreamDownloader" has incompatible type "BytesIO"; expected "IO[str]"  [arg-type]
azure\storage\blob_download.py:683: error: Incompatible return value type (got "str", expected "bytes")  [return-value]
azure\storage\blob_download.py:683: error: Argument 1 to "decode" of "bytes" has incompatible type "Union[bytes, str]"; expected "str"  [arg-type]
azure\storage\blob_download.py:684: error: Incompatible return value type (got "bytes", expected "str")  [return-value]
azure\storage\blob_download.py:758: error: No overload variant of "write" of "IO" matches argument type "bytes"  [call-overload]
azure\storage\blob_download.py:758: note: Possible overload variants:
azure\storage\blob_download.py:758: note:     def write(self, str, /) -> int
azure\storage\blob_download.py:777: error: Argument "non_empty_ranges" to "_ChunkDownloader" has incompatible type "Optional[List[Dict[str, int]]]"; expected "List[Any]"  [arg-type]

These are all related. I just can't get the stream type right (but I didn't touch the code), and then the return values are confusing for generic T.
A little bit more of the same of the above

EDIT: Passing now, just failing on a quck query recording??

API change check
APIView has identified API level changes in this PR and created following API reviews.
azure-storage-blob

/azp run python - storage - ci


scipy.optimize.minimize bug with _minimize_slsqp in python3?

The following code runs perfectly using python 2.7. Running the same code using python3 it crashes.
import numpy as np
from scipy.optimize import minimize

def find_admissible(C, c, D, d):
    """Gibt einen zulaessigen Punkt x mit
        Cx + c = 0  # falls C nicht None
        Dx + d <= 0
    zurueck"""
    def f(x):
        return x.dot(x)

    def g(x):
        return C.dot(x) + c

    def h(x):
        return -D.dot(x) - d

    cons = {'type': 'ineq', 'fun': h}
    if C is not None:
        cons = ({'type': 'eq', 'fun': g}, cons)

    res = minimize(f, np.zeros(2), method='SLSQP', constraints=cons)
    if not res.success:
        print("cannot find admissible point")
        exit()
    return res.x

def run_test():
    C = np.eye(2)
    c = np.array([0, 0])
    D = np.concatenate((np.eye(2), np.ones((1, 2))))
    d = np.array([0, 0, 0])
    print(find_admissible(C, c, D, d))

if __name__ == "__main__":
    run_test()

Adding a print(a_eq) to line 394 in _minimize_slsqp in slsqp.py shows, that it seem that the inequality constraints are also taken as equality constraints.
I have tested this with scipy 0.12.0 and scipy 0.13.3.

Upgrade to 0.14.1 or newer.

Thank you.


What's runScript in TestEnvironment

I'm using Jest CustomEnvironment. According to Jest docs, CustomEnvironment must export a class contains setup, teardown and runScript member methods.
So what is runScript,  when will be called, and what are parameters of this function.
Jest: 26.1.0

I'd suggest to ask in Jest Discord channel, you will get help easier there

I'd suggest to ask in Jest Discord channel, you will get help easier there


server port

Hi, looking on http://knexjs.org/ i've no found the syntax to create connections to mysql with port different from default.. i think it's:
client: 'mysql',
connection: {
        host: '<IP_ADDRESS>',
        port: '3307',
        user: 'birdbase',
        password: 'password',

        database: 'birdbase',
        charset: 'utf8',

    }

Is it correct ? or port: 3307,

You can define the port as a number and as a string.
The example you described is correct.

look correct.

Hi! I'm using a port different from the default with postgress.
I was able to address a different port by just adding it to the initialization area. The knex documentation doesn't really state what other settings you can modify in that section, so it makes sense why you couldn't find it. Here's my code. (Note that the password and database name needs to be filled in to connect, I removed mine for security reasons.)


Which codes can repoduce the Figure4 Runtime overhead in paper?

I wonder how we can evaluate Figure 4, Table4 and 5  in the paper: Runtime overhead in relation to the inter-/intra-
connectivity ratio of mini-batches, both for serial and concurrent history access patterns.
I am sorry that I cannot find it in the repo.

You are right that this part is not included in this repo, I am sorry.
You should be able to run "serial" mode by setting _async=False in the ScalableGNN base class.


Enable CNI

Please use this Epic to track any issues encountered with the CNI.

FYI:  istio CNI install instructions
On initial review of integrating CNI here:

[ ] add a version of istio-cni helm chart to this repo
[ ] modify bin/install.sh

add a command flag for install_cni
what should install-all do w.r.t. install_cni?

proposal:  have install_cni be a separate step and modify install_control to detect whether istio-cni is installed to set helm value for istio_cni.enabled=true when sidecar injector configmap is created (iop istio-autoinject)

how to detect which k8s env we're on?  e.g. istio-cni on GKE requires a specific setting for cni-bin-dir


Refactor tsi_ssl_credentials target.

Splits tsi_ssl_credentails into two targets tsi_ssl_session_cache and tsi_ssl_credentials for internal use.
@ZhenLian

Hi Zhan, can you please sign the CLA please?
Also, do you want to only change the BUILD file, or you want some code changes as well to come after? If latter, it's probably better to put it in the same PR; so that we can see clearly what is the purpose of making this change.

Cleaned up the dependencies for tsi_ssl_session_cache.

All failures seem irrelevant:
Artifact Build MacOS - https://github.com/grpc/grpc/issues/27932
Bazel C/C++ Dbg MacOS - irrelevant failures and tool failures
Merging now.

Per offline discussion and experimenting, this is NOT a bug in TSI or gRPC and this change is no longer needed.


Getting banned on 500followers.com

Hi, just wondering if 500followers is actually a legit site? or it a phishing page ? keeps throwing banned account when I click authorise app

This is unrelated to the Swifter project, and will thus be closed.