Ollama stuck after few runs

I updated Ollama from  0.1.16 to 0.1.18 and encountered the issue.
I am using python to use LLM models with Ollama and Langchain on Linux server(4 x A100 GPU).
There are 5,000 prompts to ask and get the results from LLM.
With Ollama 0.1.17, the Ollama server stops in 1 or 2 days.
Now it hung in 10 minutes.

This is the Ollama server message when it stops running.
It happens more when Phi 2 runs then when Mixtral runs
After the freeze, exit the server and run it again, then the prompt and the LLM answer is successfully received.
The environment
Linux: Ubuntu 22.04.3 LTS
python: 3.10.12
Ollama: 0.1.18
Langchain: 0.0.274
Mixtral: latest
Phi 2: latest
GPU: NVIDIA A100-SXM4-80GB x 4
Prompt size: ~10K
# of Prompts: 5K

Read these articles, https://github.com/jmorganca/ollama/issues/1853, https://github.com/jmorganca/ollama/issues/1688
But none of them are works here.
Also, if there are any way to install previous version of Ollama (0.1.16), let me know

@jadhvank
for previous version you can install the docker ollama/hub

Hi @jadhvank sorry you hit this, looking into it
In the meantime an easy way to install 0.1.17 is
curl https://ollama.ai/install.sh | sed 's#https://ollama.ai/download#https://github.com/jmorganca/ollama/releases/download/v0.1.17#' | sh

I think this is realted to https://github.com/jmorganca/ollama/issues/1691

I also experience this issue with 2x 3090 GPUs. The server just stops generating.

I updated the Ollama to version 0.1.19 and the stuck happened again in 5 min.
Removed the 0.1.19 and installed 0.1.16.
The stuck occurred after 6 hours (better!)

I think I have the same problem. After a few runs, the ollama server crashes and stops to generate text. I'm using windows 11 (wsl ubuntu) and langchain. I have a rtx 4090 and I tried from 0.1.16 to 0.1.19, but all of them have this issue in my case.
instead, on a laptop with windows 10 and with an nvidia T500, I don't have this problem.

me too, same problem, stop generation after random time.

"Similarly, it halts after approximately 100 iterations."

wanted to see if anyone is still running into this issue with ollama v0.1.22

wanted to see if anyone is still running into this issue with ollama v0.1.22

I confirm i still have this problem with 0.1.22

wanted to see if anyone is still running into this issue with ollama v0.1.22

I confirm i still have this problem with 0.1.22

I confirm also (on MacBook Pro 2,6 GHz Intel Core i7  and on a cpu-only server)

I could confirm that issue with 0.1.23 (on WSL)
I ran the script with 100 requests and saw in the logs that 6/10 requests were frozen and never received a response :(

+1
I run a community leaderboard for Julia code generation and I've run 10s of thousands of samples in the past (with failures, but not unreasonable).
Recently, I've updated and haven't been able to run anything anymore... Same machine/setup
Behavior:

I updated to 0.1.2x (not sure) and I couldn't run more than a few samples of Qwen-72b FP16. It kept freezing, it would stop using GPU, etc
Updated to 0.1.23 -> cannot run more than 1-2 samples and it just hangs (need to kill process)
Tried 0.1.24 pre-release -> same as above
Tried 0.1.19 -> better, I could run c. 100 samples and then it froze
I'm now on 0.1.16 and it's been a few hundred samples and it's still going. So clearly the issue didn't exist back then!

Workload:

This benchmark: https://github.com/svilupp/Julia-LLM-Leaderboard
Different models, prompts, test cases, but never have any empty or unexpected inputs (the workload hasn't changed, it just doesn't run anymore). I'm even running models that ran fine previously
Using ollama via /api/chat endpoint

System:

Debian-based x86
4x RTX 4090

Ollama header:

2024/02/08 17:20:55 images.go:857: INFO total blobs: 140
2024/02/08 17:20:55 images.go:864: INFO total unused blobs removed: 0
2024/02/08 17:20:55 routes.go:950: INFO Listening on <IP_ADDRESS>:11434 (version 0.1.22)
2024/02/08 17:20:55 payload_common.go:106: INFO Extracting dynamic libraries...
2024/02/08 17:20:57 payload_common.go:145: INFO Dynamic LLM libraries [rocm_v6 cpu rocm_v5 cuda_v11 cpu_avx2 cpu_avx]
2024/02/08 17:20:57 gpu.go:94: INFO Detecting GPU type
2024/02/08 17:20:57 gpu.go:236: INFO Searching for GPU management library libnvidia-ml.so
2024/02/08 17:20:57 gpu.go:282: INFO Discovered GPU libraries: [/usr/lib/x86_64-linux-gnu/libnvidia-ml.so.535.86.10]
2024/02/08 17:20:57 gpu.go:99: INFO Nvidia GPU detected
2024/02/08 17:20:57 gpu.go:140: INFO CUDA Compute Capability detected: 8.9

Hi @jadhvank sorry you hit this, looking into it嗨，抱歉你碰到了这个，正在调查它
In the meantime an easy way to install 0.1.17 is同时安装 0.1.17 的简单方法是
curl https://ollama.ai/install.sh | sed 's#https://ollama.ai/download#https://github.com/jmorganca/ollama/releases/download/v0.1.17#' | sh

Could it have anything to do with GPU memory management?
My experience is that if you use a 12g gpu to load the llama13b model, the output will basically get stuck if it exceeds 200 tokens.

This should be fixed as of 0.1.24. Please let me know if that isn't the case, and we'll re-open this (and get it fixed once and for all 😊). Sorry about this!

@jmorganca Unfortunately, it isn't fixed in 0.1.25.
OS: Ubuntu 22.04.2 LTS
GPU: NVIDIA RTX A6000 (Driver Version: 530.41.03, CUDA Version: 12.1)
Model: Tested mixtral:8x7b-instruct-v0.1-q4_K_M, mixtral:8x7b-instruct-v0.1-q6_K, llama2:7b-chat-q4_0
Env: Official Docker
/api/generate and /api/chat hangs complitely while version or tags info works well.

Observed this behavior sometimes with 0.1.23, but 0.1.25 makes things even worse - hangs approximately every hour.

@jmorganca, Likewise still seeing this issue after a small number of iterations on v0.1.25

I think I have the same problem. After a few runs, the ollama server crashes and stops to generate text. I'm using windows 11 (wsl ubuntu) and langchain. I have a rtx 4090 and I tried from 0.1.16 to 0.1.19, but all of them have this issue in my case. instead, on a laptop with windows 10 and with an nvidia T500, I don't have this problem.

I confirm this problem with 0.1.25

@jmorganca Can you please reopen this issue?

Same here, issue still persists on fresh install (calling multiple times in a loop).

I am seeing this with 0.1.27 running on docker on linux. Docker has a limit of 8GB of RAM but the container is using only 1.
The container just hangs and shows nothing in logs. I am using open-webui as a frontend.

I confirm alors on 0.1.27 on Mac OS X, Fedora with GPU (RTX), and Ubuntu (without GPU). In a fastapi + langchain env with 2 endpoints invoking 2 different ollama models , after I succeed in receiving responses from the first endpoint, I'm stuck when I try the 2nd endpoint. I have to restart the ollama service to see my response.

I confirm that this problem occurs with versions 0.1.24 and 0.1.27. After adding a prompt about the output limitation, it seems to be resolved.
Does that mean no [EOS] token has been generated? Using the 'STOP' options will also resolve this problem, but sometimes it may not achieve the ideal result.
OS: Ubuntu 22.04.2 LTS
GPU: NVIDIA L4 (Driver Version: 535.154.05, CUDA Version: 12.2)
Model: Mixtral8x7b-instruct-v0.1-q5_K_M

For example
Limitation prompt
    text_to_gpt = (
        f'[INST] You are a helpful assistant. Your task is to read the following context.'
        f'You will respond with a JSON object containing the entire sentence only if it contains abnormal words,'
        f' along with the confidence score. '
        f' Display only five objects with a high confidence score (greater than 0.6).'    #  Adding limitation
        f'Abnormal words are defined as 'error', 'unknown', 'fail', 'alert', etc. Do not provide explanations.'
        f'context """\n{input_text}\n"""\n'
    )

Use options
        response = self.client.chat(
            model=self.model,
            messages=[
                {
                    'role': 'user',
                    'content': input_text,
                },
            ],
            options={
                'top_k': 10,
                'temperature': 0.6,
                'stop': ['\n']
            }
        )

Has anyone come up with a minimal repro with curl or equivalent?  I'll try to repro and get to the bottom of this.

I confirm this problem with 0.1.27 on Windows 10 without WSL. May be related with format=json and stream=false?

Could someone help us. This issue still persists

In my case, it becomes unresponsive right after the initial interaction.

Gets stuck also with:
$ ollama -v
ollama version is 0.1.28

I'm getting the same thing with 0.1.26, 0.1.27, 0.1.28 on an M2 Max (64gb ram)
This is both in the cli ollama run llava and via python APIs (chat and generate). It will work fine on one or two images, then stall out. There is no crash, it just stops streaming new tokens and hangs.
Server logs are as follows for a sample run via Python:
time=2024-03-07T07:46:10.314+09:00 level=INFO source=images.go:710 msg="total blobs: 14"
time=2024-03-07T07:46:10.315+09:00 level=INFO source=images.go:717 msg="total unused blobs removed: 0"
time=2024-03-07T07:46:10.316+09:00 level=INFO source=routes.go:1021 msg="Listening on <IP_ADDRESS>:11434 (version 0.1.28)"
time=2024-03-07T07:46:10.316+09:00 level=INFO source=payload_common.go:107 msg="Extracting dynamic libraries..."
time=2024-03-07T07:46:10.338+09:00 level=INFO source=payload_common.go:146 msg="Dynamic LLM libraries [metal]"
[GIN] 2024/03/07 - 07:46:10 | 200 |      45.458µs |       <IP_ADDRESS> | HEAD     "/"
[GIN] 2024/03/07 - 07:46:10 | 200 |       578.5µs |       <IP_ADDRESS> | POST     "/api/show"
[GIN] 2024/03/07 - 07:46:10 | 200 |     373.083µs |       <IP_ADDRESS> | POST     "/api/show"
time=2024-03-07T07:46:15.526+09:00 level=INFO source=dyn_ext_server.go:90 msg="Loading Dynamic llm server: /var/folders/2t/ln_5pr392nd9dytmbrtvwhgw0000gp/T/ollama2139231423/metal/libext_server.dylib"
time=2024-03-07T07:46:15.526+09:00 level=INFO source=dyn_ext_server.go:150 msg="Initializing llama server"
loading library /var/folders/2t/ln_5pr392nd9dytmbrtvwhgw0000gp/T/ollama2139231423/metal/libext_server.dylib
{"function":"load_model","level":"INFO","line":380,"msg":"Multi Modal Mode Enabled","tid":"0x171ef7000","timestamp":1709765175}
ggml_metal_init: allocating
ggml_metal_init: found device: Apple M2 Max
ggml_metal_init: picking default device: Apple M2 Max
ggml_metal_init: default.metallib not found, loading from source
ggml_metal_init: GGML_METAL_PATH_RESOURCES = /var/folders/2t/ln_5pr392nd9dytmbrtvwhgw0000gp/T/ollama2139231423
ggml_metal_init: loading '/var/folders/2t/ln_5pr392nd9dytmbrtvwhgw0000gp/T/ollama2139231423/ggml-metal.metal'
ggml_metal_init: GPU name:   Apple M2 Max
ggml_metal_init: GPU family: MTLGPUFamilyApple8  (1008)
ggml_metal_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_init: GPU family: MTLGPUFamilyMetal3  (5001)
ggml_metal_init: simdgroup reduction support   = true
ggml_metal_init: simdgroup matrix mul. support = true
ggml_metal_init: hasUnifiedMemory              = true
ggml_metal_init: recommendedMaxWorkingSetSize  = 51539.61 MB
ggml_backend_metal_buffer_type_alloc_buffer: allocated buffer, size =   595.50 MiB, (  596.44 / 49152.00)
key clip.vision.image_grid_pinpoints not found in file
key clip.vision.mm_patch_merge_type not found in file
key clip.vision.image_crop_resolution not found in file
ggml_backend_metal_buffer_type_alloc_buffer: allocated buffer, size =    32.89 MiB, (  629.33 / 49152.00)
llama_model_loader: loaded meta data with 24 key-value pairs and 291 tensors from /Users/jono/.ollama/models/blobs/sha256:170370233dd5c5415250a2ecd5c71586352850729062ccef1496385647293868 (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.name str              = liuhaotian
llama_model_loader: - kv   2:                       llama.context_length u32              = 32768
llama_model_loader: - kv   3:                     llama.embedding_length u32              = 4096
llama_model_loader: - kv   4:                          llama.block_count u32              = 32
llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 14336
llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 32
llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 8
llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  10:                       llama.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  11:                          general.file_type u32              = 2
llama_model_loader: - kv  12:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  13:                      tokenizer.ggml.tokens arr[str,32000]   = ["<unk>", "<s>", "</s>", "<0x00>", "<...
llama_model_loader: - kv  14:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv  15:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1
llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2
llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0
llama_model_loader: - kv  19:            tokenizer.ggml.padding_token_id u32              = 0
llama_model_loader: - kv  20:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  21:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  22:                    tokenizer.chat_template str              = {{ bos_token }}{% for message in mess...
llama_model_loader: - kv  23:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:   65 tensors
llama_model_loader: - type q4_0:  225 tensors
llama_model_loader: - type q6_K:    1 tensors
llm_load_vocab: special tokens definition check successful ( 259/32000 ).
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = llama
llm_load_print_meta: vocab type       = SPM
llm_load_print_meta: n_vocab          = 32000
llm_load_print_meta: n_merges         = 0
llm_load_print_meta: n_ctx_train      = 32768
llm_load_print_meta: n_embd           = 4096
llm_load_print_meta: n_head           = 32
llm_load_print_meta: n_head_kv        = 8
llm_load_print_meta: n_layer          = 32
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 4
llm_load_print_meta: n_embd_k_gqa     = 1024
llm_load_print_meta: n_embd_v_gqa     = 1024
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: n_ff             = 14336
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 1000000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx  = 32768
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: model type       = 7B
llm_load_print_meta: model ftype      = Q4_0
llm_load_print_meta: model params     = 7.24 B
llm_load_print_meta: model size       = 3.83 GiB (4.54 BPW)
llm_load_print_meta: general.name     = liuhaotian
llm_load_print_meta: BOS token        = 1 '<s>'
llm_load_print_meta: EOS token        = 2 '</s>'
llm_load_print_meta: UNK token        = 0 '<unk>'
llm_load_print_meta: PAD token        = 0 '<unk>'
llm_load_print_meta: LF token         = 13 '<0x0A>'
llm_load_tensors: ggml ctx size =    0.22 MiB
ggml_backend_metal_buffer_from_ptr: allocated buffer, size =  3847.58 MiB, ( 4476.91 / 49152.00)
llm_load_tensors: offloading 32 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 33/33 layers to GPU
llm_load_tensors:        CPU buffer size =    70.31 MiB
llm_load_tensors:      Metal buffer size =  3847.57 MiB
................................
....................................
..............................
llama_new_context_with_model: n_ctx      = 2048
llama_new_context_with_model: freq_base  = 1000000.0
llama_new_context_with_model: freq_scale = 1
ggml_metal_init: allocating
ggml_metal_init: found device: Apple M2 Max
ggml_metal_init: picking default device: Apple M2 Max
ggml_metal_init: default.metallib not found, loading from source
ggml_metal_init: GGML_METAL_PATH_RESOURCES = /var/folders/2t/ln_5pr392nd9dytmbrtvwhgw0000gp/T/ollama2139231423
ggml_metal_init: loading '/var/folders/2t/ln_5pr392nd9dytmbrtvwhgw0000gp/T/ollama2139231423/ggml-metal.metal'
ggml_metal_init: GPU name:   Apple M2 Max
ggml_metal_init: GPU family: MTLGPUFamilyApple8  (1008)
ggml_metal_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_init: GPU family: MTLGPUFamilyMetal3  (5001)
ggml_metal_init: simdgroup reduction support   = true
ggml_metal_init: simdgroup matrix mul. support = true
ggml_metal_init: hasUnifiedMemory              = true
ggml_metal_init: recommendedMaxWorkingSetSize  = 51539.61 MB
ggml_backend_metal_buffer_type_alloc_buffer: allocated buffer, size =   256.00 MiB, ( 4733.59 / 49152.00)
llama_kv_cache_init:      Metal KV buffer size =   256.00 MiB
llama_new_context_with_model: KV self size  =  256.00 MiB, K (f16):  128.00 MiB, V (f16):  128.00 MiB
llama_new_context_with_model:        CPU input buffer size   =    13.02 MiB
ggml_backend_metal_buffer_type_alloc_buffer: allocated buffer, size =   164.02 MiB, ( 4897.61 / 49152.00)
llama_new_context_with_model:      Metal compute buffer size =   164.00 MiB
llama_new_context_with_model:        CPU compute buffer size =     8.00 MiB
llama_new_context_with_model: graph splits (measure): 2
clip_model_load: model name:   openai/clip-vit-large-patch14-336
clip_model_load: description:  image encoder for LLaVA
clip_model_load: GGUF version: 3
clip_model_load: alignment:    32
clip_model_load: n_tensors:    377
clip_model_load: n_kv:         19
clip_model_load: ftype:        f16

clip_model_load: loaded meta data with 19 key-value pairs and 377 tensors from /Users/jono/.ollama/models/blobs/sha256:72d6f08a42f656d36b356dbe0920675899a99ce21192fd66266fb7d82ed07539
clip_model_load: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
clip_model_load: - kv   0:                       general.architecture str              = clip
clip_model_load: - kv   1:                      clip.has_text_encoder bool             = false
clip_model_load: - kv   2:                    clip.has_vision_encoder bool             = true
clip_model_load: - kv   3:                   clip.has_llava_projector bool             = true
clip_model_load: - kv   4:                          general.file_type u32              = 1
clip_model_load: - kv   5:                               general.name str              = openai/clip-vit-large-patch14-336
clip_model_load: - kv   6:                        general.description str              = image encoder for LLaVA
clip_model_load: - kv   7:                        clip.projector_type str              = mlp
clip_model_load: - kv   8:                     clip.vision.image_size u32              = 336
clip_model_load: - kv   9:                     clip.vision.patch_size u32              = 14
clip_model_load: - kv  10:               clip.vision.embedding_length u32              = 1024
clip_model_load: - kv  11:            clip.vision.feed_forward_length u32              = 4096
clip_model_load: - kv  12:                 clip.vision.projection_dim u32              = 768
clip_model_load: - kv  13:           clip.vision.attention.head_count u32              = 16
clip_model_load: - kv  14:   clip.vision.attention.layer_norm_epsilon f32              = 0.000010
clip_model_load: - kv  15:                    clip.vision.block_count u32              = 23
clip_model_load: - kv  16:                     clip.vision.image_mean arr[f32,3]       = [0.481455, 0.457828, 0.408211]
clip_model_load: - kv  17:                      clip.vision.image_std arr[f32,3]       = [0.268630, 0.261303, 0.275777]
clip_model_load: - kv  18:                              clip.use_gelu bool             = false
clip_model_load: - type  f32:  235 tensors
clip_model_load: - type  f16:  142 tensors
clip_model_load: CLIP using Metal backend
clip_model_load: text_encoder:   0
clip_model_load: vision_encoder: 1
clip_model_load: llava_projector:  1
clip_model_load: model size:     595.49 MB
clip_model_load: metadata size:  0.14 MB
clip_model_load: params backend buffer size =  595.49 MB (377 tensors)
clip_model_load: compute allocated memory: 32.89 MB
{"function":"initialize","level":"INFO","line":433,"msg":"initializing slots","n_slots":1,"tid":"0x171ef7000","timestamp":1709765176}
{"function":"initialize","level":"INFO","line":445,"msg":"new slot","n_ctx_slot":2048,"slot_id":0,"tid":"0x171ef7000","timestamp":1709765176}
time=2024-03-07T07:46:16.208+09:00 level=INFO source=dyn_ext_server.go:161 msg="Starting llama main loop"
{"function":"update_slots","level":"INFO","line":1565,"msg":"all slots are idle and system prompt is empty, clear the KV cache","tid":"0x17349f000","timestamp":1709765176}
[GIN] 2024/03/07 - 07:46:16 | 200 |  5.709566167s |       <IP_ADDRESS> | POST     "/api/chat"
update check failed - TypeError: fetch failed
time=2024-03-07T07:46:41.119+09:00 level=INFO source=dyn_ext_server.go:171 msg="loaded 1 images"
{"function":"launch_slot_with_data","level":"INFO","line":826,"msg":"slot is processing task","slot_id":0,"task_id":0,"tid":"0x17349f000","timestamp":1709765201}
{"function":"update_slots","level":"INFO","line":1825,"msg":"kv cache rm [p0, end)","p0":0,"slot_id":0,"task_id":0,"tid":"0x17349f000","timestamp":1709765201}
encode_image_with_clip: image embedding created: 576 tokens

encode_image_with_clip: image encoded in   223.26 ms by CLIP (    0.39 ms per image patch)
{"function":"print_timings","level":"INFO","line":264,"msg":"prompt eval time     =    5730.16 ms /     1 tokens ( 5730.16 ms per token,     0.17 tokens per second)","n_prompt_tokens_processed":1,"n_tokens_second":0.17451528815004544,"slot_id":0,"t_prompt_processing":5730.157,"t_token":5730.157,"task_id":0,"tid":"0x17349f000","timestamp":1709765223}
{"function":"print_timings","level":"INFO","line":278,"msg":"generation eval time =   16943.76 ms /   411 runs   (   41.23 ms per token,    24.26 tokens per second)","n_decoded":411,"n_tokens_second":24.25672181205148,"slot_id":0,"t_token":41.225686131386865,"t_token_generation":16943.757,"task_id":0,"tid":"0x17349f000","timestamp":1709765223}
{"function":"print_timings","level":"INFO","line":287,"msg":"          total time =   22673.91 ms","slot_id":0,"t_prompt_processing":5730.157,"t_token_generation":16943.757,"t_total":22673.914,"task_id":0,"tid":"0x17349f000","timestamp":1709765223}
{"function":"update_slots","level":"INFO","line":1635,"msg":"slot released","n_cache_tokens":412,"n_ctx":2048,"n_past":1971,"n_system_tokens":0,"slot_id":0,"task_id":0,"tid":"0x17349f000","timestamp":1709765223,"truncated":false}
[GIN] 2024/03/07 - 07:47:03 | 200 | 22.698469125s |       <IP_ADDRESS> | POST     "/api/chat"
time=2024-03-07T07:47:03.825+09:00 level=INFO source=dyn_ext_server.go:171 msg="loaded 1 images"
{"function":"launch_slot_with_data","level":"INFO","line":826,"msg":"slot is processing task","slot_id":0,"task_id":414,"tid":"0x17349f000","timestamp":1709765223}
{"function":"update_slots","level":"INFO","line":1825,"msg":"kv cache rm [p0, end)","p0":0,"slot_id":0,"task_id":414,"tid":"0x17349f000","timestamp":1709765223}

I am facing the same issue after running it on gpu. I had no issues previously when running it only on cpu. Using 0.1.28 and llama index. Eagerly waiting for a solution.

Same issue here :(

Has anyone come up with a minimal repro with curl or equivalent? I'll try to repro and get to the bottom of this.

Regretfully, haven't managed to do so. Nevertheless, I went through the logs and noticed that when Ollama hangs, instead of the normal functions sequence:
launch_slot_with_data: slot processing task -> update_slots: slot progression, kv cache -> print timings: prompt, generation ->  update slots: slot release
it goes through the following:
launch_slot_with_data: slot processing task -> update_slots: slot progression, kv cache -> update_slots: slot context shift
The last function executes infinitely until I stop the server and relaunch it. So, might be related to another issue I found (sorry I haven't pinpointed the number) related to infinite context shifting.
Hope the above provides a bit of assistance.

Here are the goroutines from a stuck ollama serve that stops answering queries after a few.

Same issue. Running ollama 0.1.28 on M1 Max.
My observation:
I work on a large number of text chunks as input for a RAG algorithms - and the task for the LLM (mixtral in my case) is to extract keywords and concepts from the chunks. The document is rather large; so it makes quite a diffference if I set the character count for the text split to produce chunks of a length of 1.000 or 2.000 characters. These chunks are served to Mixtral as USER_PROMPT - and the SYSTEM_PROMPT by itself is also rather long.
Now the key observation: the failure seems to be functionally dependent on the length of the overall prompt.
If I set text-split to 2.000 characters, the overall prompt lenghth is much longer - and the failure occurs much quicker (5-10 generations) than if the text-split is set to 1.000 characters (around 15-20 generations). Unfortunately, the algorithsm ought to work its way thru > 400 to 800 chunks.... which it doesn't.
Long story short: the occurence of the bug seems to be a function of the number of tokens being served to the LLM thru Ollama.

I'm observing this issue in 0.1.28 on Ubuntu 22.04 with a 3060+16GB RAM. I am running TheBloke's Q6 Mistral Instruct v0.2. GGUF
I'm encountering this only when I'm setting "format"="json". I am using the model for a classification task and only got through 5 queries before it hung up and I had to restart ollama. I was able to reproduce this consistently and it always failed at the 6th query
I reran it, this time without the "format"="json" param, and I am 4k+ requests in without a crash

This will likely be resolved with #3218 but I'll leave this open until we can verify the health check logic is sufficient to catch this hang scenario.

My first feedback after the last Ollama update: the situation has improved a lot, but has not gone away alltogether.
I tried it out on several text sizes - and now it works on longer texts, but still gets eventually stuck on very long texts.
Before the last update, I would not get more than 30 generations in a row, feeding the algorithms with text chunks of the size of 1.000 characters. Now, it works up to around 100 generations and slightly north of that.
But e.g. processing text consisting of 200 or more chunks still gets the process stuck eventually.

I have to stop docker. rm the docker instance and run it again to solve this issue. I hope someone fixes this soon.

here is an example of my local ollama in docker hallucinating:
 docker exec -it ollama ollama run starcoder2
>>> write a simple hello world program in golang
 and build an image from it.

This is the command that will be used:
```sh
docker run -it --name hello-go-app --rm -v ${PWD}:/go/src/hello-go-app golang:1.7 go run /go/src/hello-go-app/*.go

Building the image
The first step is to create a Docker file that will contain the commands for building and running our application. The Dockerfile for our app will have two sections. We will use COPY and ENTRYPOINT for
this. Copy will copy the contents of our app folder into /go/src/hello^C

what is this?
 docker exec -it ollama ollama run starcoder2
>>> write a simple hello world program in golang.
 1.

Go is a compiled language. You can't compile Go programs with gcc or
clang on Linux. So the first thing you need to do is install a C compiler.
In this case, you will use GCC. If you don’t know how to do that, please consult
the previous post: Installing^C

>>> Send a message (/? for help)

the same issue!

my system:
apple m1 max 64G
At the initial run, everything works fine. When questions are asked continuously, the system will be stuck, the CPU usage will continue to increase, and the GPU will be reduced to 0 at the same time.
Once the CPU reaches 100%, ollama will stop working. I have tried many methods to no avail!

I ran into the same snag when I was working on summarizing text. Tweaking the prompt words sorted it out for me. I added a bit in the prompt to make sure the summary stays between 100 and 200 words.

This happened to me as well. After some research and test, I found setting the option num_kepp to 0 fixed this issue.
Possibly related to #2805 , #2225
Python code I used:
def send_summarization_request(text, system_prompt):
    url = summarization_endpoint
    payload = {
        "model": model,
        "prompt": text,
        "system": system_prompt,
        "stream": False,
        "keep_alive": "5m",
        "options": {
            "num_keep": 0,
            "num_batch": 8
        }
    }
    response = requests.post(url, json=payload)
    summary = json.loads(response.text)

    response_text = summary["response"]

    return response_text

I ran into the same snag when I was working on summarizing text. Tweaking the prompt words sorted it out for me. I added a bit in the prompt to make sure the summary stays between 100 and 200 words.

This happened to me as well. After some research and test, I found setting the option num_kepp to 0 fixed this issue.
Possibly related to #2805 , #2225
Python code I used:
def send_summarization_request(text, system_prompt):
    url = summarization_endpoint
    payload = {
        "model": model,
        "prompt": text,
        "system": system_prompt,
        "stream": False,
        "keep_alive": "5m",
        "options": {
            "num_keep": 0,
            "num_batch": 8
        }
    }
    response = requests.post(url, json=payload)
    summary = json.loads(response.text)

    response_text = summary["response"]

    return response_text

Looked all over the place to find what num_keep does but with no avail. All I found is that num_keep default value is 0. Version 1.31 hangs even more often for me. Team ollama is doing a great work, but this bug is destroying the experience.
Working on a Mac Studio M2 Max 32GB, running many summarization tasks in sequence - if it helps.

I added the num_keep parameter but the bug still exists. For now, I'm using a timeout to kill the ollama process as a workaround to complete the batch summary tasks.
I use a bash script to start the 'ollama serve &' process, checking every 2 seconds to see if the ollama process exists, and if not, start it.
When calling the API, I add a timeout limit of 1 minute. If a timeout exception occurs, I kill the ollama process, then wait for 2 seconds, remove the task causing the timeout, and start the loop again.

I also have this bug... :( Unfortunately sometimes even restarting the ollama server is not helping it
It just hangs

This happens to me too so I wrote a bash script that manages and just restarts it every 10 minutes....
start_ollama() {
    nohup ollama serve &  # Start ollama serve in the background and ignore hangups
    echo "Ollama serve started."
}

while true; do

    ollama_pid=$(pgrep -f 'ollama')

    if [ ! -z "$ollama_pid" ]; then
        echo "Killing ollama serve process: $ollama_pid"
        kill $ollama_pid
        sleep 2
    fi
    start_ollama

    sleep 600
done

Hitting the stability issue here as well.  Had to add a reset action in my
neovim so I could poke it awake more easily :
https://github.com/jdonaldson/dotfiles/blob/main/.config/lvim/config.lua#L45
On Tue, Apr 9, 2024 at 9:17 AM Kevin Urrutia @.***>
wrote:

This happens to me too so I wrote a bash script that manages and just
restarts it every 10 minutes....
start_ollama() {
nohup ollama serve &  # Start ollama serve in the background and ignore hangups
echo "Ollama serve started."
}
while true; do
ollama_pid=$(pgrep -f 'ollama')

if [ ! -z "$ollama_pid" ]; then
    echo "Killing ollama serve process: $ollama_pid"
    kill $ollama_pid
    sleep 2
fi
start_ollama

sleep 600

done
—
Reply to this email directly, view it on GitHub
https://github.com/ollama/ollama/issues/1863#issuecomment-2045588515,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAKEB2XJ24THSMCWDGVFFLY4QICDAVCNFSM6AAAAABBSZT5G6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBVGU4DQNJRGU
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

Got same issue , seem to be problems caused by using codelama 34b . But its not exactly , just iterative check

Just to give more notes, I use ollama on mac and linux.  On linux for me it seems for stable. On a mac studio m1 and mac book pro m1, I have to restart it every dozen or so requests because it just freezes. I want to run this on my mac studio as a server, but its too unstable. I am going to just add a restart script every hour to see if that fixes it.

Same issue running on Macbook 16 M3 Max with 36 GB of ram, ollama hangs after an hour or so of usage, logs don't seem to indicate nothing wrong happened.. seems that i'll need to use a cronjob to restart it every hour

We are seeing the same issue with Ubuntu 20.04 LTS and 2 x A100. So far I am taking a timeout check and restart approach by running the following script every 10 minutes with cron, ollama-check:
#! /bin/sh
# check if ollama api is not responding and restart service after timeout
# Dan Wilcox, ZKM | Hertzlab, zkm.de

# host
HOST=YOURSERVER:11434

# desired model
MODEL=llama2

# timeout in seconds
TIMEOUT=5

# make api call with timeout
curl --connect-timeout $TIMEOUT --max-time $TIMEOUT -d "{
  \"model\": \"$MODEL\",
  \"prompt\": \"Why is the sky blue?\",
  \"format\": \"json\",
  \"stream\": false
}" http://${HOST}/api/generate 2> /dev/null

# force restart ollama on *any* non-zero exit code:
# https://everything.curl.dev/cmdline/exitcode.html
# ex. to check just timeout, use [ "$?" = "28" ]
if [ "$?" != "0" ] ; then
  killall ollama 2> /dev/null
  systemctl stop ollama 2> /dev/null
  systemctl start ollama
fi

By modifying this with the latest version, I have not had any more problems. If anyone tries it too and it works for them I can do PR.
---
 llm/server.go | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/llm/server.go b/llm/server.go
index 0e084d5..f32e25c 100644
--- a/llm/server.go
+++ b/llm/server.go
@@ -503,6 +503,11 @@ type CompletionResponse struct {
 }

 func (s *LlamaServer) Completion(ctx context.Context, req CompletionRequest, fn func(CompletionResponse)) error {
+	// Set a timeout for the request
+	var cancelFunc context.CancelFunc
+      ctx, cancelFunc = context.WithTimeout(ctx, 60*time.Second)
+      defer cancelFunc()
+
 	request := map[string]any{
 		"prompt":            req.Prompt,
 		"stream":            true,
---

Please give 0.1.32 a try and let us know if you're still seeing unrecoverable hangs.

0.1.32 seems to have fixed the issue for me. 2000+ iterations so far without a hang. Previously 5-10 iterations would cause Ollama to hang.

Thanks to everyone for reporting and testing this. Marking this as resolved for now pending any more reports.

Hangs after about 400 long context requests on mixtral and same with llama3
ollama --version
ollama version is 0.1.32

Please give 0.1.32 a try and let us know if you're still seeing unrecoverable hangs.

{"function":"update_slots","level":"INFO","line":1601,"msg":"slot context shift","n_cache_tokens":2048,"n_ctx":2048,"n_discard":1023,"n_keep":1,"n_left":2046,"n_past":2047,"n_system_tokens":0,"slot_id":0,"task_id":105393,"tid":"140517846056960","timestamp":1714056803}
{"function":"update_slots","level":"INFO","line":1601,"msg":"slot context shift","n_cache_tokens":2048,"n_ctx":2048,"n_discard":1023,"n_keep":1,"n_left":2046,"n_past":2047,"n_system_tokens":0,"slot_id":0,"task_id":105393,"tid":"140517846056960","timestamp":1714056823}
{"function":"update_slots","level":"INFO","line":1601,"msg":"slot context shift","n_cache_tokens":2048,"n_ctx":2048,"n_discard":1023,"n_keep":1,"n_left":2046,"n_past":2047,"n_system_tokens":0,"slot_id":0,"task_id":105393,"tid":"140517846056960","timestamp":1714056843}

Still having the problem here on version 0.1.32.  I am running batches of image annotations with llava and it will just hang after a few dozen images or so.
RTX 4090
Ubuntu 22.04
Running via Docker container
Restarting the container kicks it back into submission but looking for a a more reliable answer.

Also still experiencing hangs when calling /api/chat endpoint. Running on HumanEval benchmark (164 samples), usually fails about 70-80 calls. Requires ollama serve restart to recover. Mostly happens with CodeLlama-70b rather than the smaller models (13b, 7b; only tested these).
Running v0.1.32 on Ubuntu 22.04.2 with NVIDIA RTX A6000, driver 530.30.02, CUDA 12.1, using a Docker container
The specific part where it seems to loop indefinitely is the update_slots function
with "msg":"slot context shift" line from ollama serve logs (full logs below):
{"function":"update_slots","level":"INFO","line":1601,"msg":"slot context shift","n_cache_tokens":2048,"n_ctx":2048,"n_discard":1023,"n_keep":1,"n_left":2046,"n_past":2047,"n_system_tokens":0,"slot_id":0,"task_id":7729,"tid":"139817808982016","timestamp":1713984503}

Code that calls the endpoint:
    url = "http://localhost:11434/api/chat"
    params = {
        'model': 'codellama:70b-instruct',
        'options': {
            'num_ctx': 4096
        },
        'messages': [
            {'role': 'system', 'content': sys_prompt},
            {"role": "user", "content": user_prompt},
        ],
        'options': {
            'seed': 123,
            'temperature': 0.2,
        },
        'stream': False,
        'keep_alive': 10
    }
    response = requests.post(url, json=params).json()

Full log of the run below:
time=2024-04-24T14:30:43.820-04:00 level=INFO source=gpu.go:121 msg="Detecting GPU type"
time=2024-04-24T14:30:43.820-04:00 level=INFO source=gpu.go:268 msg="Searching for GPU management library libcudart.so*"
time=2024-04-24T14:30:43.824-04:00 level=INFO source=gpu.go:314 msg="Discovered GPU libraries: [/tmp/ollama1496169119/runners/cuda_v11/libcudart.so.11.0]"
time=2024-04-24T14:30:44.105-04:00 level=INFO source=gpu.go:126 msg="Nvidia GPU detected via cudart"
time=2024-04-24T14:30:44.105-04:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-04-24T14:30:44.387-04:00 level=INFO source=gpu.go:202 msg="[cudart] CUDART CUDA Compute Capability detected: 8.6"
time=2024-04-24T14:30:44.437-04:00 level=INFO source=gpu.go:121 msg="Detecting GPU type"
time=2024-04-24T14:30:44.437-04:00 level=INFO source=gpu.go:268 msg="Searching for GPU management library libcudart.so*"
time=2024-04-24T14:30:44.440-04:00 level=INFO source=gpu.go:314 msg="Discovered GPU libraries: [/tmp/ollama1496169119/runners/cuda_v11/libcudart.so.11.0]"
time=2024-04-24T14:30:44.440-04:00 level=INFO source=gpu.go:126 msg="Nvidia GPU detected via cudart"
time=2024-04-24T14:30:44.440-04:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-04-24T14:30:44.505-04:00 level=INFO source=gpu.go:202 msg="[cudart] CUDART CUDA Compute Capability detected: 8.6"
time=2024-04-24T14:30:44.536-04:00 level=INFO source=server.go:127 msg="offload to gpu" reallayers=81 layers=81 required="38351.2 MiB" used="38351.2 MiB" available="45354.3 MiB" kv="640.0 MiB" fulloffload="324.0 MiB" partialoffload="348.0 MiB"
time=2024-04-24T14:30:44.536-04:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-04-24T14:30:44.536-04:00 level=INFO source=server.go:264 msg="starting llama server" cmd="/tmp/ollama1496169119/runners/cuda_v11/ollama_llama_server --model /root/.ollama/models/blobs/sha256-1436d66b69757a245f02d000874c670507949d11ad5c188a623652052c6aa508 --ctx-size 2048 --batch-size 512 --embedding --log-disable --n-gpu-layers 81 --port 38491"
time=2024-04-24T14:30:44.536-04:00 level=INFO source=server.go:389 msg="waiting for llama runner to start responding"
{"function":"server_params_parse","level":"INFO","line":2603,"msg":"logging to file is disabled.","tid":"139817808982016","timestamp":1713983444}
{"build":1,"commit":"7593639","function":"main","level":"INFO","line":2819,"msg":"build info","tid":"139817808982016","timestamp":1713983444}
{"function":"main","level":"INFO","line":2822,"msg":"system info","n_threads":32,"n_threads_batch":-1,"system_info":"AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | ","tid":"139817808982016","timestamp":1713983444,"total_threads":64}
llama_model_loader: loaded meta data with 22 key-value pairs and 723 tensors from /root/.ollama/models/blobs/sha256-1436d66b69757a245f02d000874c670507949d11ad5c188a623652052c6aa508 (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.name str              = codellama
llama_model_loader: - kv   2:                       llama.context_length u32              = 2048
llama_model_loader: - kv   3:                     llama.embedding_length u32              = 8192
llama_model_loader: - kv   4:                          llama.block_count u32              = 80
llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 28672
llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 64
llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 8
llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  10:                       llama.rope.freq_base f32              = 10000.000000
llama_model_loader: - kv  11:                          general.file_type u32              = 2
llama_model_loader: - kv  12:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  13:                      tokenizer.ggml.tokens arr[str,32016]   = ["<unk>", "<s>", "</s>", "<0x00>", "<...
llama_model_loader: - kv  14:                      tokenizer.ggml.scores arr[f32,32016]   = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv  15:                  tokenizer.ggml.token_type arr[i32,32016]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1
llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2
llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0
llama_model_loader: - kv  19:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  20:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  21:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:  161 tensors
llama_model_loader: - type q4_0:  561 tensors
llama_model_loader: - type q6_K:    1 tensors
llm_load_vocab: mismatch in special tokens definition ( 264/32016 vs 260/32016 ).
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = llama
llm_load_print_meta: vocab type       = SPM
llm_load_print_meta: n_vocab          = 32016
llm_load_print_meta: n_merges         = 0
llm_load_print_meta: n_ctx_train      = 2048
llm_load_print_meta: n_embd           = 8192
llm_load_print_meta: n_head           = 64
llm_load_print_meta: n_head_kv        = 8
llm_load_print_meta: n_layer          = 80
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 8
llm_load_print_meta: n_embd_k_gqa     = 1024
llm_load_print_meta: n_embd_v_gqa     = 1024
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 28672
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx  = 2048
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: model type       = 70B
llm_load_print_meta: model ftype      = Q4_0
llm_load_print_meta: model params     = 68.98 B
llm_load_print_meta: model size       = 36.20 GiB (4.51 BPW)
llm_load_print_meta: general.name     = codellama
llm_load_print_meta: BOS token        = 1 '<s>'
llm_load_print_meta: EOS token        = 2 '</s>'
llm_load_print_meta: UNK token        = 0 '<unk>'
llm_load_print_meta: LF token         = 13 '<0x0A>'
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:   yes
ggml_cuda_init: CUDA_USE_TENSOR_CORES: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA RTX A6000, compute capability 8.6, VMM: yes
llm_load_tensors: ggml ctx size =    0.55 MiB
llm_load_tensors: offloading 80 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 81/81 layers to GPU
llm_load_tensors:        CPU buffer size =   140.70 MiB
llm_load_tensors:      CUDA0 buffer size = 36930.21 MiB
....................................................................................................
llama_new_context_with_model: n_ctx      = 2048
llama_new_context_with_model: n_batch    = 512
llama_new_context_with_model: n_ubatch   = 512
llama_new_context_with_model: freq_base  = 10000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init:      CUDA0 KV buffer size =   640.00 MiB
llama_new_context_with_model: KV self size  =  640.00 MiB, K (f16):  320.00 MiB, V (f16):  320.00 MiB
llama_new_context_with_model:  CUDA_Host  output buffer size =     0.15 MiB
llama_new_context_with_model:      CUDA0 compute buffer size =   324.00 MiB
llama_new_context_with_model:  CUDA_Host compute buffer size =    20.01 MiB
llama_new_context_with_model: graph nodes  = 2566
llama_new_context_with_model: graph splits = 2
{"function":"initialize","level":"INFO","line":448,"msg":"initializing slots","n_slots":1,"tid":"139817808982016","timestamp":1713983453}
{"function":"initialize","level":"INFO","line":457,"msg":"new slot","n_ctx_slot":2048,"slot_id":0,"tid":"139817808982016","timestamp":1713983453}
{"function":"main","level":"INFO","line":3064,"msg":"model loaded","tid":"139817808982016","timestamp":1713983453}
{"function":"main","hostname":"<IP_ADDRESS>","level":"INFO","line":3267,"msg":"HTTP server listening","n_threads_http":"63","port":"38491","tid":"139817808982016","timestamp":1713983453}
{"function":"update_slots","level":"INFO","line":1578,"msg":"all slots are idle and system prompt is empty, clear the KV cache","tid":"139817808982016","timestamp":1713983453}
{"function":"process_single_task","level":"INFO","line":1506,"msg":"slot data","n_idle_slots":1,"n_processing_slots":0,"task_id":7726,"tid":"139817808982016","timestamp":1713984223}
{"function":"log_server_request","level":"INFO","line":2734,"method":"GET","msg":"request","params":{},"path":"/health","remote_addr":"<IP_ADDRESS>","remote_port":40168,"status":200,"tid":"139814867263488","timestamp":1713984223}
{"function":"process_single_task","level":"INFO","line":1506,"msg":"slot data","n_idle_slots":1,"n_processing_slots":0,"task_id":7727,"tid":"139817808982016","timestamp":1713984223}
{"function":"log_server_request","level":"INFO","line":2734,"method":"GET","msg":"request","params":{},"path":"/health","remote_addr":"<IP_ADDRESS>","remote_port":40168,"status":200,"tid":"139814867263488","timestamp":1713984223}
{"function":"log_server_request","level":"INFO","line":2734,"method":"POST","msg":"request","params":{},"path":"/tokenize","remote_addr":"<IP_ADDRESS>","remote_port":40168,"status":200,"tid":"139814867263488","timestamp":1713984223}
{"function":"process_single_task","level":"INFO","line":1506,"msg":"slot data","n_idle_slots":1,"n_processing_slots":0,"task_id":7728,"tid":"139817808982016","timestamp":1713984223}
{"function":"log_server_request","level":"INFO","line":2734,"method":"GET","msg":"request","params":{},"path":"/health","remote_addr":"<IP_ADDRESS>","remote_port":40168,"status":200,"tid":"139814867263488","timestamp":1713984223}
{"function":"launch_slot_with_data","level":"INFO","line":830,"msg":"slot is processing task","slot_id":0,"task_id":7729,"tid":"139817808982016","timestamp":1713984223}
{"function":"update_slots","ga_i":0,"level":"INFO","line":1809,"msg":"slot progression","n_past":4,"n_past_se":0,"n_prompt_tokens_processed":300,"slot_id":0,"task_id":7729,"tid":"139817808982016","timestamp":1713984223}
{"function":"update_slots","level":"INFO","line":1836,"msg":"kv cache rm [p0, end)","p0":4,"slot_id":0,"task_id":7729,"tid":"139817808982016","timestamp":1713984223}
{"function":"update_slots","level":"INFO","line":1601,"msg":"slot context shift","n_cache_tokens":2048,"n_ctx":2048,"n_discard":1023,"n_keep":1,"n_left":2046,"n_past":2047,"n_system_tokens":0,"slot_id":0,"task_id":7729,"tid":"139817808982016","timestamp":1713984352}
{"function":"update_slots","level":"INFO","line":1601,"msg":"slot context shift","n_cache_tokens":2048,"n_ctx":2048,"n_discard":1023,"n_keep":1,"n_left":2046,"n_past":2047,"n_system_tokens":0,"slot_id":0,"task_id":7729,"tid":"139817808982016","timestamp":1713984428}
{"function":"update_slots","level":"INFO","line":1601,"msg":"slot context shift","n_cache_tokens":2048,"n_ctx":2048,"n_discard":1023,"n_keep":1,"n_left":2046,"n_past":2047,"n_system_tokens":0,"slot_id":0,"task_id":7729,"tid":"139817808982016","timestamp":1713984503}

I dont understand the hurry to close this issue without getting feedback first. where have you learned this @BruceMacD ? or is this normal procedure in your dev workflow?

I think I have the same problem. After a few runs, the ollama server crashes and stops to generate text. I'm using windows 11 (wsl ubuntu) and langchain. I have a rtx 4090 and I tried from 0.1.16 to 0.1.19, but all of them have this issue in my case. instead, on a laptop with windows 10 and with an nvidia T500, I don't have this problem.

I still have this problem with 0.1.32 with rtx 4090 and windows 11 (on wsl ubuntu).
After a few run (less then 10), the Ollama server is stuck and i can't access anymore to "http://localhost:11434/" unless i kill the process

Got the same bug with A100 on Ubuntu 22.04. ollama version is 0.1.32.

Thanks to everyone for reporting and testing this. Marking this as resolved for now pending any more reports.

This issue has not been resolved. I encountered the same congestion issue yesterday (ollama 0.1.32, A100, Ubuntu 22.04) and had to restart to resolve it.

Same bug here. Mac M2 Max Studio hangs after several questions being asked.

Thanks for the reports, re-opening this.
Couple of questions to help me reproduce:

What models are people seeing this on?
Are you inputting long prompts/context when it gets stuck?

Thanks for the reports, re-opening this.
Couple of questions to help me reproduce:

What models are people seeing this on?
Are you inputting long prompts/context when it gets stuck?

mistral, mistral:instruct, llama3
I usually use the LLM model to summarize text. The inputs are normally less than 50 ~500 words.

Thanks for the reports, re-opening this.
Couple of questions to help me reproduce:

What models are people seeing this on?
Are you inputting long prompts/context when it gets stuck?

llama2 7b and 13b, llama3 8b
I use about 300 words in the prompts

The pre-release for 0.1.33 is available now, which should resolve these long context hang/loop problems.

@dhiltgen Great news, thank you: will try asap after I have the update installed.
FYI, the situation has alread improved considerably - but hangups still there are with 0.1.32.
I experienced a hangup after  having Ollama / Mixtral churn thru a large text file for > 12 h, extracting semanting information from it.
Setup: M1 Powerbook with 64 GB RAM and Ollama 0.1.32.
The text had 623 chunks with 1000 characters each (plus another ca. 400 characters prompt size) - and the hangup occured after processing 517 of these chunks.

I can still encounter this problem on 0.1.33: ollama gets stuck after 15 minutes of embeddings processing (using nomic-embed-text). Last log lines:
ollama[8937]: [GIN] 2024/04/28 - 20:12:17 | 200 |  155.784253ms |  <IP_ADDRESS> | POST     "/api/embeddings"
ollama[9198]: {"function":"process_single_task","level":"INFO","line":1506,"msg":"slot data","n_idle_slots":1,"n_processing_slots":0,"task_id":31361,"tid":"140302648860672","timestamp":1714335137}
ollama[9198]: {"function":"log_server_request","level":"INFO","line":2734,"method":"GET","msg":"request","params":{},"path":"/health","remote_addr":"<IP_ADDRESS>","remote_port":41614,"status":200,"tid":"140301227372544","timestamp":1714335137}
ollama[9198]: {"function":"process_single_task","level":"INFO","line":1506,"msg":"slot data","n_idle_slots":1,"n_processing_slots":0,"task_id":31362,"tid":"140302648860672","timestamp":1714335137}
ollama[9198]: {"function":"log_server_request","level":"INFO","line":2734,"method":"GET","msg":"request","params":{},"path":"/health","remote_addr":"<IP_ADDRESS>","remote_port":41614,"status":200,"tid":"140301227372544","timestamp":1714335137}
ollama[9198]: {"function":"launch_slot_with_data","level":"INFO","line":830,"msg":"slot is processing task","slot_id":0,"task_id":31363,"tid":"140302648860672","timestamp":1714335137}
ollama[9198]: {"function":"update_slots","level":"INFO","line":1836,"msg":"kv cache rm [p0, end)","p0":0,"slot_id":0,"task_id":31363,"tid":"140302648860672","timestamp":1714335137}
ollama[9198]: {"function":"update_slots","level":"INFO","line":1640,"msg":"slot released","n_cache_tokens":103,"n_ctx":2048,"n_past":103,"n_system_tokens":0,"slot_id":0,"task_id":31363,"tid":"140302648860672","timestamp":1714335137,"truncated":false}
ollama[9198]: {"function":"log_server_request","level":"INFO","line":2734,"method":"POST","msg":"request","params":{},"path":"/embedding","remote_addr":"<IP_ADDRESS>","remote_port":41618,"status":200,"tid":"140301247180800","timestamp":1714335137}
ollama[8937]: [GIN] 2024/04/28 - 20:12:17 | 200 |   49.744207ms |  <IP_ADDRESS> | POST     "/api/embeddings"
ollama[9198]: {"function":"process_single_task","level":"INFO","line":1506,"msg":"slot data","n_idle_slots":1,"n_processing_slots":0,"task_id":31366,"tid":"140302648860672","timestamp":1714335137}
ollama[9198]: {"function":"log_server_request","level":"INFO","line":2734,"method":"GET","msg":"request","params":{},"path":"/health","remote_addr":"<IP_ADDRESS>","remote_port":41618,"status":200,"tid":"140301247180800","timestamp":1714335137}
ollama[9198]: {"function":"process_single_task","level":"INFO","line":1506,"msg":"slot data","n_idle_slots":1,"n_processing_slots":0,"task_id":31367,"tid":"140302648860672","timestamp":1714335137}
ollama[9198]: {"function":"log_server_request","level":"INFO","line":2734,"method":"GET","msg":"request","params":{},"path":"/health","remote_addr":"<IP_ADDRESS>","remote_port":41618,"status":200,"tid":"140301247180800","timestamp":1714335137}
ollama[9198]: {"function":"launch_slot_with_data","level":"INFO","line":830,"msg":"slot is processing task","slot_id":0,"task_id":31368,"tid":"140302648860672","timestamp":1714335137}

GPU: NVIDIA GeForce RTX 3060
Driver version: 550.54.14
CUDA Version: 12.4

Just to add to this a bit - looks like we are experiencing same issue.
Running llava model, gets stuck every 15 - 20 minutes. Sometimes gets stuck after just 4 requests.
We are using API to call the endpoint. running version 0.1.33 though a docker container.

+1
`
You 05/05/2024 18:18
то ти розуміеш украйську мову?
profile
Llama3:Latest
Корисно запитання!
Я можу зрозуміти основні фрази і слова на українській мові, оскільки я вивчав великий корпус текстів різних мов, включно з українською.
Однак, моя комп'ютерна профікувність українській мові може бути обмеженою порівняно з профікувністю кореспондента-українця. Я можу зробити помилки в граматиці, морф
`
remote_port":47034,"status":200,"tid":"139900095553536","timestamp":1714925939} {"function":"log_server_request","level":"INFO","line":2737,"method":"POST","msg":"request","params":{},"path":"/tokenize","remote_addr":"<IP_ADDRESS>","remote_port":47034,"status":200,"tid":"139900095553536","timestamp":1714925939} {"function":"process_single_task","level":"INFO","line":1509,"msg":"slot data","n_idle_slots":1,"n_processing_slots":0,"task_id":832,"tid":"139900887961600","timestamp":1714925939} {"function":"log_server_request","level":"INFO","line":2737,"method":"GET","msg":"request","params":{},"path":"/health","remote_addr":"<IP_ADDRESS>","remote_port":47034,"status":200,"tid":"139900095553536","timestamp":1714925939} {"function":"log_server_request","level":"INFO","line":2737,"method":"POST","msg":"request","params":{},"path":"/tokenize","remote_addr":"<IP_ADDRESS>","remote_port":47034,"status":200,"tid":"139900095553536","timestamp":1714925939} {"function":"process_single_task","level":"INFO","line":1509,"msg":"slot data","n_idle_slots":1,"n_processing_slots":0,"task_id":833,"tid":"139900887961600","timestamp":1714925939} {"function":"log_server_request","level":"INFO","line":2737,"method":"GET","msg":"request","params":{},"path":"/health","remote_addr":"<IP_ADDRESS>","remote_port":47036,"status":200,"tid":"139900087160832","timestamp":1714925939} {"function":"log_server_request","level":"INFO","line":2737,"method":"POST","msg":"request","params":{},"path":"/tokenize","remote_addr":"<IP_ADDRESS>","remote_port":47036,"status":200,"tid":"139900087160832","timestamp":1714925939} {"function":"process_single_task","level":"INFO","line":1509,"msg":"slot data","n_idle_slots":1,"n_processing_slots":0,"task_id":834,"tid":"139900887961600","timestamp":1714925939} {"function":"log_server_request","level":"INFO","line":2737,"method":"GET","msg":"request","params":{},"path":"/health","remote_addr":"<IP_ADDRESS>","remote_port":47036,"status":200,"tid":"139900087160832","timestamp":1714925939} {"function":"log_server_request","level":"INFO","line":2737,"method":"POST","msg":"request","params":{},"path":"/tokenize","remote_addr":"<IP_ADDRESS>","remote_port":47036,"status":200,"tid":"139900087160832","timestamp":1714925939} {"function":"process_single_task","level":"INFO","line":1509,"msg":"slot data","n_idle_slots":1,"n_processing_slots":0,"task_id":835,"tid":"139900887961600","timestamp":1714925939} {"function":"log_server_request","level":"INFO","line":2737,"method":"GET","msg":"request","params":{},"path":"/health","remote_addr":"<IP_ADDRESS>","remote_port":47036,"status":200,"tid":"139900087160832","timestamp":1714925939} {"function":"launch_slot_with_data","level":"INFO","line":830,"msg":"slot is processing task","slot_id":0,"task_id":836,"tid":"139900887961600","timestamp":1714925939} {"function":"update_slots","ga_i":0,"level":"INFO","line":1812,"msg":"slot progression","n_past":1085,"n_past_se":0,"n_prompt_tokens_processed":307,"slot_id":0,"task_id":836,"tid":"139900887961600","timestamp":1714925939} {"function":"update_slots","level":"INFO","line":1839,"msg":"kv cache rm [p0, end)","p0":1085,"slot_id":0,"task_id":836,"tid":"139900887961600","timestamp":1714925939}

A quick feedback: from my perspective, the bug is solved as far as Ollama running on Mac Silicon is concerned. I was never able to to process more than ca. 120 text chunks of a size of 1.000 characters in one go on an M1 Pro Mac. Now, with the upate to 0.1.33, the computer ran for 24 h nonstop, processing ca. 630 text chunks of a larger document to extract information from it.... and did so to the very end.
Simply: Thanks !

Still happens to me with llava models @ ollama v0.1.34
Interestingly, Ollama only freezes up when I use the /chat endpoint with both image and text data. It works fine when I only send text.
Setup:
2x RTX 4090
13900k

The same. still, have issues
RTXA6000
Llama3:70b
Hangs on <10 requests

Has happened with multiple models for me. My prompt is usually just a line. I have seen that it happens if I kept it idle for sometime. When I come back the responses are stuck. I am running on CPU but I don't see that as issue because the first prompt after starting ollama serve command always gets quick response.

Hi all, give this fix a go: https://github.com/ollama/ollama/issues/4604#issuecomment-2130436000

I'm not using Docker so I don't think this fix is applicable to me.

I'm experiencing a similar issue. I'm running on a virtualized VM with a Tesla P40. After booting the VM everything works, but after a while, when the server idles, it stops working. Neither requests from a frontend nor from the cli (e.g. ollama run llama3) work. With the cli it just never starts up. The log files dont show anything suspicous. "service ollama restart" does nothing. The only thing that is maybe not aligned is, that I'm having CUDA 12.2 installed but the runner is using v11.

Thanks for the reports, re-opening this.
Couple of questions to help me reproduce:
* What models are people seeing this on?

* Are you inputting long prompts/context when it gets stuck?

i'm seeing the same problem on Windows. I'm using llava:v1.6 and pass the images through bind(...) as base64 and then invoke the promp. works fine for few prompts and then stops.
i'm at the moment checking if maybe something else could be wrong, but as my experience fits to what i read here i want to mention it.

I'm seeing the same issues with vLLM which indicates a problem with some underlying libraries, e.g. torch or maybe something CUDA. What is fascinating is, that this has apparently nothing to do with time between requests or even "going into sleep mode" since I managed to perform a query to vLLM which was working perfectly with around 15 tokens per second and then slowed to a crawl with 0.1 tokens per second. So even while the generation is running this stuff happens.

I have resolved my issue. It had nothing todo with ollama, vllm, or any other part of the software stack. It was a LICENSING  issue. I simply forgot to aquire a license for the vGPU the VM was using. So after a while the nvidia driver degraded the performance of the vGPU to become basically unusable.

@jak4 thank you for letting us know about this. May I ask what was the VM provider so we know in the future what to lookout for?

@mchiang0610 I'm unsure what you mean bei VM provider, but I'm running a homelab with PROXMOX as the VM host and Tesla P40. The guest is a Debian 12 instance.

I am having the same issue running:
ollama  0.1.41
M2 Max with 64gb ram
I was initially using the 'llava' model, which would hang after ~10 image and prompt pairs. Now I have switched to 'llava-llama3' and am able to process ~20 requests before it hangs.

 Downgrading to v0.1.38 solved this for me: both models behave properly as before. (Both versions built through the Archlinux build process.)

I haven't been able to reproduce this yet
On an M3 mac, the following loops at least 80+ times without problem:
C=0; while ~/ollama-darwin run llava --verbose please describe the contents of the following image ./image.jpg ; do C=$(($C+1)); echo $C; done

A cuda windows system with this also loops cleanly
$C=0; while($true) { &ollama run --verbose llava please describe the contents of the following image ./image.jpg ; ++$C; write-output $C }

Perhaps there are some non-default settings being passed via API clients that is causing the hang?  Can anyone share a minimal curl loop that repro's?

It looks like it's way easier to break with a model derived from llama3:8b with a longer system prompt (I'm not able to share) than with the plain llama3:8b but eventually a chat (OpenWebUI) with the llama3:8b also broke down: chat-Unseen Backyard Secret Revealed.txt
Just guessing as a longer system prompt model breaks earlier I'd guess the bigger the amount of text, the earlier it breaks.
After that linked chat even ollama runs are producing garbage:

Run Output
# ollama run llama3:8b "Please write a story of a girl wandering into the forest, discovering all the secrets of the forest. Include her feelings and throughts. Ignore length limits."
As she stepped out of a secret.

As Sophia had always felt a name her age of the woods behind her heart was a Whispering Woods' whispers of course I'd ever saw before - ancient trees towered withered clearing
surrounded by towering trees, their branches that most vividly colored leaves and wildflowers, there was a massive stone statue of twisted tree, its trunk twisted with
age-wrinkled with a small streams of the sun began to me felt like my feetprints of hidden treasures: shiny rocks that had made their in a cozy little den beneath the ancient
tree; a
, a hidden pools of crystal clear and filled with crystal-clear water crystal clear; a tiny fish swimming holes in old, abandoned cabin that seemed carved withered from the
perfect resting place for centuries.

As I could be enjoyed.
As I spent hours passed the stories like any old trees or creatures, though those secrets are home to the whispers. As you see things like, fireflies across the sky on clear
nights; heard the moon cast my hair-raised magic on the wings as they flit about the treetoak
The forest itself – the whispers that faint murmurs of leaves that seem to come from nowhere and everywhere at the same time. Some people might be back inched secret glade of
hidden trails that crisscross the forest, each one day, I've been, while others are overgrown and require a veil or camouflaged by mist. But as you follow the forest is sharing
its secrets with me, one trail at a time.
As I could gothic, there are also dangers lurking in the shadows – dark creatures that canny corners of the forest; packs, and packs of wild wolves prowling through the
fringes, yelping like embers. But even the forest magic and I've to respect their space, to both to trust my wits its depths.
As I' That's the stories  – so many creatures! From the forest sprites whooping owls who taught me about the interconnectedness; the mischievous vouches the importance of
quick-thinking; and the majestic deer, gentle giants us her wisdom with me. And the universe. They all these are you see, my friend, it'nt just waiting to me, like magic,
danger, and endless possibility. I feel so lucky to have stumbled upon its secrets that it able to explore its secrets and wherever I do so much deeper truths that part of

model: llama3
when after some runs, it got stuck, and after a long time, it outputed, and when stuck, the GPU still be used

This happens frequently with 0.1.45 and dolphin-mixtral, 26 gb version. Ollama is running from command prompt on my XTX. Does not seem to happen with smaller models.

I was having the same issue, on a Mac, what I just found out that if I run (0.1.48)
 ollama serve &
from terminal then I incur in the issue of having the server crashing after 5-6 requests with  /api/generate , and the .ollama/logs/server.log is not created/populated.
while if I run
open /Applications/Ollama.app &
the log is created and the server starts working flawlessly (I left it all night and we're now at 600 logged requests), and the Ollama icon appears on MacOS menu bar (where you can also quit it).
So there is definitely some difference between the sole launch of the ollama server and the startup of the app, although I can't say what it is with precision that causes the issue, and blocking the logging, but at least it sounds like there's a workaround. I find this behavior to be consistent.
Could someone on Linux and Windows check out too?

Honestly from all my personal experience and talking to lots of other developers, Ollama is really good for quick prototyping and testing models, but due to ongoing issues like this, is not really meant for production. For production vllm is much more stable and seems to be used by lots of companies.  — Sent from my mobileOn Jun 30, 2024, at 1:49 AM, hybra @.***> wrote:
I was having the same issue, on a Mac, what I just found out that if I run (0.1.48)
ollama serve &
from terminal then I incur in the issue of having the server crashing after 5-6 requests with  /api/generate , and the .ollama/logs/server.log is not created/populated.
while if I run
open /Applications/Ollama.app &
the log is created and the server starts working flawlessly (I left it all night and we're now at 600 logged requests), and the Ollama icon appears on MacOS menu bar (where you can also quit it).
So there is definitely some difference between the sole launch of the ollama server and the startup of the app, although I can't say what it is with precision that causes the issue, and blocking the logging, but at least it sounds like there's a workaround. I find this behavior to be consistent.
Could someone on Linux and Windows check out too?
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>

Honestly from all my personal experience and talking to lots of other developers, Ollama is really good for quick prototyping and testing models, but due to ongoing issues like this, is not really meant for production. For production vllm is much more stable and seems to be used by lots of companies.

vLLM seems to be Linux only. But we're OT now

It seems to me that using LLaVA Ollama freezes completely after a couple of hundreds requests. I can't find any errors or other issues which could occur

On may Mac m1, it runs since hours serving hundrets of requests. On my Hetzner server with Nvidia RTX 4000, it gets in stuck after some requests.

After 10 mins or so, it starts to hang on Linux running a RTX 4000. On Mac m1, it runs properly through since 24 hours.
https://github.com/ollama/ollama/issues/6380

Same issue here, 4 V100 32G GPU on Linux, docker ollama, llama2 70B. It hangs on for two hours after some iterations. But 4 GPU are activate (all about 30%) during it hangs on. It generates smoothly for 100 iterations, and hangs on, then generates some again, and hangs on ......

it amazes me that this issue is persisting for over 8 months now and nobody knows how to fix it.

yeah, i think the problem is that there nobody has a clue why it is happening or couldnt debug it properly.
do the ollama alternatives have the same problem or are vLLM or localAI working without problems for the same tasks? because this would really move me to move to another system...

yeah, i think the problem is that there nobody has a clue why it is happening or couldnt debug it properly.

I think the devs aren't aware of the fact that they are building an app that does not run properly. the commits show that this repo is active. but that doesnt do anything if the app is not running for many people.
looks like wrong priorization of tasks.

I think the devs aren't aware of the fact that they are building an app that does not run properly. the commits show that this repo is active. but that doesnt do anything if the app is not working for many people.

Well, it does run. The problem seems to only happen with the very large models. if you use llama <7B you dont have these Problems. At least for me it only happens with the large models (for my applications i do not need the big ones).

I wonder if that happens with llama.cpp too. Does anyone have any insights on that?

I wonder if that happens with llama.cpp too. Does anyone have any insights on that?

we use llama.cpp meanwhile and there it works perfeclty.

We may finally have a solution to this. For those that are experiencing the problem and are able to build from source, there is a new runner module that is currently being tested. Instructions for building it are here:
https://github.com/ollama/ollama/blob/main/docs/development.md#transition-to-go-runner

I wonder if that happens with llama.cpp too. Does anyone have any insights on that?

we use llama.cpp meanwhile and there it works perfeclty.

Yes, the llama.cpp server works well without similar issue. However, I'm looking for tool calling api support recently, it seems llama.cpp server is lacking behind as I saw related issues are still opening.
And glad to see ollama team is addressing on this issue. Well done! Thank you!

We may finally have a solution to this. For those that are experiencing the problem and are able to build from source, there is a new runner module that is currently being tested. Instructions for building it are here: https://github.com/ollama/ollama/blob/main/docs/development.md#transition-to-go-runner

is there a timing when you think an official build will be provided?

We may finally have a solution to this. For those that are experiencing the problem and are able to build from source, there is a new runner module that is currently being tested. Instructions for building it are here: https://github.com/ollama/ollama/blob/main/docs/development.md#transition-to-go-runner

In my case it seems to have helped — i was able to leave ollama running overnight, processing batches of embeddings via /api/embed endpoint. It was running little under 8 hours with no freezes or error logs.

Thanks for taking the problem into consideration. I am mostly working with Google colab, and I can not run the ollama for a long run.
happy to test the new feature if I could.

I have a similar problem, I use an extension in vscode called "Continue - Codestral, Claude, and more" which uses ollama as an llms provider, I have configured the tabAutocomplete function which makes queries to ollama to autocomplete code, but since version 0.4.0 ollama gets stuck after some queries, running ollama run codegemma:code I saw that when making some queries it keeps generating tokens without stopping and I can believe that is the cause of the hangup, and also that prevents the model from stopping with ollama stop codegemma:code, ollama ps shows that stopping ... but it never ends.

@willypaz243 you might be experiencing the same thing as #7645 - With OLLAMA_DEBUG=1 set, when it gets stuck we see periodic context limit hit - shifting in the logs and the ollama_llama_server processes saturates 1 CPU core.

I think the original issue is fixed but it sounds like there is a new issue with somewhat similar symptoms. I'm going to close this issue so we can track it in a single place in #7645. For those that are running into this, we have made further improvements in this area so it would be helpful it you can test with 0.4.3-rc0 (or later) and report the results in the other bug.


Are generated schemas from values without any added @schema annotations usable?

I'm thinking about helm-schema to our default pipeline for building & publishing Helm carts, making it available in many charts not necessarily maintained by myself.
Is it a good idea to start (and even release) with a generate schema solely based on the values, without any @schema hints?
Can it break something in certain situations, where currently used values suddenly wouldn't be valid anymore?
When I discovered this project, I wasn't really sure if helm-schema wouldn't just assume the tightest type possible, making every string to be an enum unless explicitly configuring it - I'm glad that it's not that way :)
One case I can think of, that could potentially (even likely) break charts without further configuration of @schema would be anyOf types like resources blocks.
I think it's a good idea to

put a recommendation in the docs, either warning about or recommending some approach about onboarding charts / using default config
list common anyOf properties / gotchas
maybe handle certain cases directly within helm-schema? resources.limits.cpu (as well as the other paths ofc) are a very common pattern, for which a default type of integer|string could be set; this could be done for everything resembling known Kubernetes anyOf properties (probably behind some --flag)

Helm validates the values.yaml against the schema by default, unless --skip-schema-validation is given. And yes, this will break stuff for users if you just publish the generated schema as-is.
For example, the grafana-oncall Helm Chart has some null values by default, which are interpreted as the only valid value by this tool. This leads to many errors like these when running Helm:
Error: Error: values don't meet the specifications of the schema(s) in the following chart(s):
oncall:
- externalPostgresql.existingSecret: Invalid type. Expected: null, given: string
- externalPostgresql.host: Invalid type. Expected: null, given: string
- externalPostgresql.passwordKey: Invalid type. Expected: null, given: string

For this reason I would strongly recommend against publishing the generated schema, unless you add proper annotations to all attributes.


Empty Screen.

I've followed the directions, went perfectly smooth. I go to localhost:5000 and all I have is a screen that has "Welcome to Project ". Nothing else.

Have you granted yourself all the Roles? It seems like you dont have any roles assigned.
For this navigate to the Administration in the menu at the top. In the user list select the star next to your user and in the dialog grant yourself all the roles:

I've assigned roles, same problem.
On 3/23/21, Steffen Nörtershäuser @.***> wrote:

Have you granted yourself all the Roles? It seems like you dont have any
roles assigned.
For this navigate to the Administration in the menu at the top. In the user
list select the star next to your user and in the dialog grant yourself all
the roles:

--
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
https://github.com/steffendx/GoNorth/issues/55#issuecomment-805208666

Nevermind. Fixed. I logged out and logged back in. IT's fixed. Thanks.
On 3/23/21, Steffen Nörtershäuser @.***> wrote:

Have you granted yourself all the Roles? It seems like you dont have any
roles assigned.
For this navigate to the Administration in the menu at the top. In the user
list select the star next to your user and in the dialog grant yourself all
the roles:

--
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
https://github.com/steffendx/GoNorth/issues/55#issuecomment-805208666

Ah yes, sorry I forgot to mention that you will have to sign out and back in again.
Happy to hear it works now!


Add InteractionType: SsoSilent

Add typeInteractionStatus: SsoSilent to distinguish between silent and ssoSilent both in logs and PWB cases.

Codecov Report

Merging #6504 (022e257) into dev (81d34b4) will increase coverage by 10.64%.
Report is 329 commits behind head on dev.
The diff coverage is n/a.

Flag
Coverage Δ

*Carryforward flag

msal-angular
96.73% <ø> (+0.22%)
:arrow_up:
Carriedforward from a51a29b

msal-browser
?

msal-common
?

msal-core
?

msal-node
?

msal-node-extensions
?

msal-react
94.24% <ø> (-0.45%)
:arrow_down:

node-token-validation
?

*This pull request uses carry forward flags. Click here to find out more.

Files Changed
Coverage Δ

lib/msal-angular/src/constants.ts
100.00% <ø> (ø)

lib/msal-angular/src/msal.broadcast.service.ts
100.00% <ø> (ø)

lib/msal-angular/src/msal.guard.ts
90.78% <ø> (+0.64%)
:arrow_up:

lib/msal-angular/src/msal.interceptor.ts
100.00% <ø> (ø)

lib/msal-angular/src/msal.module.ts
100.00% <ø> (ø)

lib/msal-angular/src/msal.navigation.client.ts
93.33% <ø> (+0.47%)
:arrow_up:

lib/msal-angular/src/msal.redirect.component.ts
100.00% <ø> (ø)

lib/msal-angular/src/msal.service.ts
100.00% <ø> (ø)

lib/msal-angular/src/packageMetadata.ts
100.00% <ø> (ø)

... and 219 files with indirect coverage changes


support for crafting

As crafting is a core part of the game, i believe valenece should in some way include support for handling crafting.
Ideally this would be implemented in a way that makes it possible to craft items in any menu (not just the vanilla menus).
Maybe a CraftingRecipe struct could look like this:
struct CraftingRecipe {
    /// Items to be used for crafting (slots in the OpenInventory, item stack)
    cost: HashMap<u16, ItemStack>,
    /// The item(s) that will be crafted (slots in the OpenInventory, item stack)
    result: HashMap<u16, ItemStack>,
    /// If the item(s) should be crafted as soon as all the required items are in the slots (if false, you will have to take one of the result items for the cost items to be used)
    instant_craft: bool,
    // some more stuff
    ...
}

and for furnaces some extra logic (maybe a seperate struct) would be required.
those recipes would be attached to OpenInventories as a component (maybe like a Recipes Component)
Any thoughts on that?

I agree, it would be silly not to support crafting. I don't think attaching the recipes to the inventory entities themselves is a good idea. There should be a crafting registry that can be used to easily lookup the inputs to see if they match a recipe.
Take a moment to list out all the requirements before we discuss implementation. We'll end up with a much better first draft. To get us started:

Recipes take items to produce a result item
Would be nice to be able to support forge item tags in recipes
Some mods have recipes that take more than one item in a particular slot
Crafting tables

produce the result instantly
has both shaped and shapeless recipes
shape is position independent -- eg. a torch can be crafted in any 2 slots, as long as its a coal above a stick
some shapeless recipes are more dynamic -- mods have recipes that let you repair tools by putting 2 damaged ones in the table, and you get a new tool with the durability combined.

Furnace

consumes fuel
produces result after a time

I also agree that it would be nice!
One thing that should be considered too is that it might be useful to have client dependant crafting recipes, (I can think of cases where teams etc get different results for the same items for example), Maybe it makes sense that its a component on the Client with the available recipes for that client? I'm not sure about the best design is here, but it would be awesome if the implementation allowed for such flexibility.
Also I think its wise to define the scope of this implementation and if it considers implementing any recipe book related stuff too or not. (I'm guessing its out of scope with the recipe book sync)

One thing that should be considered too is that it might be useful to have client dependant crafting recipes, (I can think of cases where teams etc get different results for the same items for example), Maybe it makes sense that its a component on the Client with the available recipes for that client? I'm not sure about the best design is here, but it would be awesome if the implementation allowed for such flexibility.

i fully agree with that.
I think crafting should be implemented so that its flexible enough to allow for something like this:
Workbench A can only craft item X, and Workbench B can only craft item Y
not sure if there is any use for something like that, but i do think you should be able to do something like that.
And i also think crafting should be possible in non-crafting inventories, like chests for example.

https://github.com/andrewgazelka/hyperion/blob/a7f2db4ddd36fef98c17f2fc908786827c421004/crates/hyperion-crafting/src/lib.rs


OverflowText (marquee) does not work or adjust its width correctly when its container resizes

Confirming I have seen this a lot as well.


media_seen() return 'ok' , but target story not marked as seen

Before submitting an issue, make sure you have:

[x] Updated to the lastest version v1.6.0
[x] Read the README and docs
[x] Searched the bugtracker for similar issues including closed ones
[x] Reviewed the sample code in tests and examples

Which client are you using?

[x] app (instagram_private_api/)
[ ] web (instagram_web_api/)

Describe your Question/Issue:
I'm using  media_seen() for marking story items as seen, but it's not working.
media_seen() calls 'media/seen/' but web version of Instagram calls 'https://www.instagram.com/stories/reel/seen' maybe it's a problem?

Code:

result = api.user_reel_media('212572123')
api.media_seen(result['items'])

any update?

Facing the same issue...


Node.js: Set/unset client seqid for json_protocol and compact_protocol

Like binary_protocol does.

This change needs a Jira ticket; please see https://thrift.apache.org/docs/HowToContribute.  Also if you can rebase on master and push that will kick a new CI build; CI needs to pass (or failures easily explained) to merge.

This change needs a Jira ticket; please see https://thrift.apache.org/docs/HowToContribute. Also if you can rebase on master and push that will kick a new CI build; CI needs to pass (or failures easily explained) to merge.

@bananer is this worth pulling forward if it passes a build?

@jeking3 To be honest, I don't know why this is useful/necessary. I think there is no need for a seqid because the connection between request and response is handled through callbacks automatically.

In general I want to make sure all clients send a seqid and all servers put that seqid in the result.  Without this we won't be able to have clients that can have multiple outstanding requests in the future.

This should have updates added to it that were done in https://issues.apache.org/jira/browse/THRIFT-4480, otherwise the server will spam logs for every requests if they come from clients that send seqid = 0 (perl does this)

Original author is not responsive and deleted his upstream branch/repo.  I am closing this.


Empty Sierra file

Create a test for an empty Sierra export file - this may happen if no new orders/brief bibs have been added on the previous day.
NightShift should create a SourceFile record without any attached resources, etc. Should silently then exit.

No issues concering empty Sierra exports discovered.
Added two tests (unit and integration) for this situation.


⚠ Bofrost.lu has degraded performance

In eb8672c, Bofrost.lu (https://www.bofrost.lu) experienced degraded performance:

HTTP code: 200
Response time: 2173 ms

Resolved: Bofrost.lu performance has improved in f9ed1bb after 9 minutes.


Set minimum deployment target to iOS and tvOS 16

Pull request
Description
This PR updates the minimum deployment target for Pillarbox to iOS and tvOS 16.
Changes made

Update package manifest.
Update demo to use modern navigation APIs.
Make controls togglable (not directly related but still nice).

Checklist

[x] Your branch has been rebased onto the main branch.
[x] APIs have been properly documented (if relevant).
[x] The documentation has been updated (if relevant).
[x] New unit tests have been written (if relevant).
[x] The demo has been updated (if relevant).

🌙 Available in TestFlight nightlies
The most recent state can be tested in the following builds:

Platform
Version

iOS
-

tvOS
1.0.0 (62)


Map also empty dictionaries to file

This PR fixes issues with Cython coverage plugin - see https://github.com/cython/cython/issues/3515.
This PR partialy reverts commit https://github.com/nedbat/coveragepy/commit/106828c2cc8bbce1e5fb31c6a89ea3ac025225c1 which introduces this issue.
Fixes #972

This looks like a very simple fix. I wonder why it hasn't received any traction unless its incorrect?

This does look like a simple fix.  I wish I understood more about Cython and its plugin.  But I guess it is reasonable to continue processing files that have had no data collected.  I will merge this, and hope that we don't discover why the if v was there in the first place.

This is now released as part of coverage 6.4.3.

I wish I understood more about Cython and its plugin.

There's was a short comment at the top of https://github.com/cython/cython/blob/master/Cython/Coverage.py that I now expanded to explain the mechanics in an "obvious" place.

@matusvalo #1538 points out that this change has a performance impact.  Can we try to understand more about why Cython needs this change when other situations don't?

Can we try to understand more about why Cython needs this change when other situations don't?

I will try to investigate but it is pretty long time ago I dig into this code so I need some time. Maybe @scoder  or @da-woods can help here too.

This is now released as part of coverage 7.1.0.

This is now released as part of coverage 7.2.7.


Make a Recommendation system .

You have to make a recommendation system

@Niketkumardheeryan  Is there any specific type of recommendation system we have to make or we are free to choose?

You are free to choose.. @ayushgoyal9991

I am interested in contributing as an LGM-SOC participant.

@ayushgoyal9991 go for it

Hello, I have completed a content-based recommendation system and uploaded the same in the forked repo. Can you please tell me what do I need to do to make changes in the parent repo?

Please assign me thia issue

Any update ? Can I work on this issue?


Iterator factory.  Resolves #1075

I'm unsure of this particular feature.  It adds complication for little gain.  The java driver already exposes an API to monitor its performance.  The mapping code is about to undergo a drastic update and, at best, this should wait until that is done.  I'm going to close this for now.


Layout and resizing need improvement

Sorry to say this, but the resizing is useless. Once you resize a window, it goes into floating mode while other windows adjust to the grid and hide under it. It should be also resized by arrow keys because TWM's are focused around keyboard use, and sudden need to switch to mouse/touchpad can be a pain in the ass (other TWMs are allowing to change this with keys)
The layouts itself are fine, but there should be also shortcuts for each of them. Having to constantly circle through them is annoying
LeftWM is overall great, but current resizing makes me want to switch to something else. The default layouts itself are not that great for work, and the config file keeps bricking if you make a single error (leftwm returns to stock keybinds and stays there)

hey @Qwart376
A lot of the layouts support the commands (IncreaseMainWidth, DecreaseMainWidth) I use these to resize my windows instead of trying to make them float.
Here is what I have in my config.
[[keybind]]
command = "IncreaseMainWidth"
value = "5"
modifier = ["modkey"]
key = "semicolon"
[[keybind]]
command = "DecreaseMainWidth"
value = "5"
modifier = ["modkey"]
key = "j"

I would also recommend running the command leftwm-check I run it every time I make a change to my config. that way it tells me when I do something wrong in the config.
Hope this helps,
lex148

hey @Qwart376
A lot of the layouts support the commands (IncreaseMainWidth, DecreaseMainWidth) I use these to resize my windows instead of trying to make them float.
Here is what I have in my config.
[[keybind]]
command = "IncreaseMainWidth"
value = "5"
modifier = ["modkey"]
key = "semicolon"
[[keybind]]
command = "DecreaseMainWidth"
value = "5"
modifier = ["modkey"]
key = "j"

I would also recommend running the command leftwm-check I run it every time I make a change to my config. that way it tells me when I do something wrong in the config.
Hope this helps,
lex148

Thanks and sorry for closing this so late. It would be nice if there was also command to change height of the window


Prefix more instances of unused variables with _

Should I do anything? Any comments?

Sorry, I hasn't too much time the last days and thanks for your contribution. I will try to review the PR as soon as possible.

Thank you for your contribution.


For driver-load-order dependencies, see How To Control Device Driver Load Order.

"For driver-load-order dependencies, see How To Control Device Driver Load Order."
Link is broken

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 89caeb22-0006-6d94-24dc-b08bef273905
Version Independent ID: 030be807-fb37-503f-f62a-1df6d19303c0
Content: Device management namespace objects
Content Source: windows-driver-docs-pr/bringup/device-management-namespace-objects.md
Product: windows-hardware
GitHub Login: @windows-driver-content
Microsoft Alias: windowsdriverdev

Thanks for pointing this out, @Shyamal-V
@barrygolden can you look into this? I think the KB article is gone, but you might find that install/specifying-driver-load-order is a valid resource.

@Shyamal-V the link has been updated to point to Specifying Driver Load Order.


SignalConst: Improve placement of first point

Fix wrong index rounding in SinglaConst (#4083).

Thanks for this fix @StendProg! Note that the recommendation here was implemented in #4082 by @KroMignon which has already been merged, but I'm going to use this PR to track updating the changelog to reflect this contribution.
Thanks again for both of your help on this!


Any idea to disabe auto save for some file type?

Like some log file, I may edit it accidentally, so I don't wanna save it automatically, also git commit, I might wanna drop it instead of saving it.

Hello, @johnzeng
This seems to be a duplicate of #33

You can apply a solution from this comment to implement the behaviour you need.


display git branch in statusbar

I'm trying to use '#(shell-command)' to display the status of my git branch, but output is blank.
to debug, i replaced it with '#(ls -l)', and refreshed the theme through VIM. it showed '0', which makes sense because we're inside VIM.
exit vim, the count is still 0 even though that's not true.
Looks like the commands are not updating, which is surprising since #W changes upon path, and time and date update as well.
Any ideas?

I'm not sure if it's exactly what you're looking for, but if you just want to display the branch name in the tmuxline status, this worked for me:
#(cd #{pane_current_path}; git rev-parse --abbrev-ref HEAD)
Here is my tmuxline_preset from my .vimrc:
let g:tmuxline_preset = {
    \'a'    : '#S',
    \'b'    : '#W',
    \'c'    : '#(cd #{pane_current_path}; git rev-parse --abbrev-ref HEAD)',
    \'cwin' : ['#I', '#W'],
    \'win'  : ['#I', '#W'],
    \'x'    : '#{?client_prefix,Prefix,      }',
    \'y'    : ['%a', '%e-%b-%Y', '%l:%M%p'],
    \'z'    : '#h'}


Llama 3 based Gorilla

Are there any plans to release Gorilla based on Llama 3? Is there any readme where we can find instructions on fine-tuning Llama 3 for Gorilla?

As per the https://github.com/ShishirPatil/gorilla/blob/main/inference/README.md , based on the Llama agreement, we will have to merge the Gorilla weights with the base Llama weights using the https://github.com/ShishirPatil/gorilla/blob/main/inference/apply_delta.py.
As per Lllama 3 documentation here https://github.com/meta-llama/llama3, the weights are available in both native llama3 format and huggingface format. which should we be using for the apply_delta.py?
Would the Gorilla Delta weights work with Llama 3 native weights?
If at all we have all the above queries taken care of, should we be applying Gorilla delta on base Lllama model weights or Instruct based model weights?

+1 to this question

+1 to this question


[Bug]: Textarea not autoresize

Before submitting...

[X] I have searched for duplicate or closed issues.
[X] I have read the CONTRIBUTING document and my issue is following the guidelines.
[X] I have read the template completely before filling it in.

Context
Textarea input not resize when add a line break.
Current Behavior
No response
Expected behavior
No response
Possible Solutions or Causes
No response
Steps to reproduce
No response
Your Environment

Version used: 2.0.3 alpha
Browser Name and version: firefox / chrome
Operating System and version (desktop or mobile): Win11
Additional information you want to tell us:

I was able to solve this by explicitly initialising each text area:
document.querySelectorAll('.materialize-textarea').forEach(function(element){
    M.Forms.InitTextarea(element);
});

Current version 2.0.3: textarea still do not resize when enter a long text or enter line break...

I was able to solve this by explicitly initialising each text area:
document.querySelectorAll('.materialize-textarea').forEach(function(element){
    M.Forms.InitTextarea(element);
});

Document do not mention about init textarea for autoresize when enter long text which must be default for textarea :|

it also dont work at docs


Create deployment documentation for Word Count pipeline

closes #75

thanks for creating the screenshot! ✨
we will have to update it to include the connector and Redis DB after https://github.com/bakdata/streams-explorer/issues/288