Yet another "segmentation fault" issue with AMD GPU

What is the issue?
Error: llama runner process has terminated: signal: segmentation fault (core dumped). It occurs while loading larger models, that are still within the VRAM capacity. Here I'm trying to load command-r:35b-08-2024-q4_K_M (19GB), on an RX 7900 XTX with 24GB of VRAM. Smaller models load fine.
Edit: even with gemma2:27b-instruct-q4_K_M 16 GB I still get the error. It seems that max model size that can be loaded is 13 GB, eg codestral:22b-v0.1-q4_K_M.
From the logs: ollama clearly says that available vram is 23.5 GiB
Sep 11 09:41:57 computer ollama[71334]: time=2024-09-11T09:41:57.987-06:00 level=INFO source=types.go:107 msg="inference compute" id=0 library=rocm variant="" compute=gfx1100 driver=0.0 name=1002:744c total="24.0 GiB" available="23.5 GiB"
Error:
Sep 11 09:43:18 computer ollama[71334]: time=2024-09-11T09:43:18.408-06:00 level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server not responding" Sep 11 09:43:26 computer ollama[71334]: time=2024-09-11T09:43:26.324-06:00 level=ERROR source=sched.go:456 msg="error loading llama server" error="llama runner process has terminated: signal: seg
I could've swore that models of that size used to load just fine on older ollama version, but unfortunately, not sure which was the latest ollama version that worked.
OS
Linux
GPU
AMD
CPU
AMD
Ollama version
0.3.9

hi @remon-nashid can you retry using ollama 0.3.10 ?

@igorschlum will do as soon as it lands in archlinux packages.

I don't have an identical setup, but I tried to repro on windows on an 7900 XTX with 0.3.9 and 0.3.10 and they both load this model OK.  @remon-nashid are you specifying any custom parameters like context size?  Server logs might help narrow things down as well.

Hi @dhiltgen , thanks for looking into this. I'm not specifying neither a custom context size nor other parameters.
Below are the logs from my attempt to ollama run command-r:35b-08-2024-q4_K_M.
Sep 11 18:18:59 computer ollama[67030]: time=2024-09-11T18:18:59.630-06:00 level=INFO source=payload.go:30 msg="extracting embedded files" dir=/tmp/ollama1037403190/runners
Sep 11 18:19:03 computer ollama[67030]: time=2024-09-11T18:19:03.676-06:00 level=INFO source=payload.go:44 msg="Dynamic LLM libraries [cpu rocm]"
Sep 11 18:19:03 computer ollama[67030]: time=2024-09-11T18:19:03.676-06:00 level=INFO source=gpu.go:200 msg="looking for compatible GPUs"
Sep 11 18:19:03 computer ollama[67030]: time=2024-09-11T18:19:03.676-06:00 level=WARN source=gpu.go:669 msg="unable to locate gpu dependency libraries"
Sep 11 18:19:03 computer ollama[67030]: time=2024-09-11T18:19:03.676-06:00 level=WARN source=gpu.go:669 msg="unable to locate gpu dependency libraries"
Sep 11 18:19:03 computer ollama[67030]: time=2024-09-11T18:19:03.700-06:00 level=WARN source=gpu.go:669 msg="unable to locate gpu dependency libraries"
Sep 11 18:19:03 computer ollama[67030]: time=2024-09-11T18:19:03.707-06:00 level=WARN source=amd_linux.go:59 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
Sep 11 18:19:03 computer ollama[67030]: time=2024-09-11T18:19:03.708-06:00 level=INFO source=amd_linux.go:348 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=11.0.0
Sep 11 18:19:03 computer ollama[67030]: time=2024-09-11T18:19:03.709-06:00 level=INFO source=amd_linux.go:274 msg="unsupported Radeon iGPU detected skipping" id=1 total="512.0 MiB"
Sep 11 18:19:03 computer ollama[67030]: time=2024-09-11T18:19:03.709-06:00 level=INFO source=types.go:107 msg="inference compute" id=0 library=rocm variant="" compute=gfx1100 driver=0.0 name=1002:744c total="24.0 GiB" available="22.7 GiB"
Sep 11 18:19:31 computer ollama[67030]: [GIN] 2024/09/11 - 18:19:31 | 200 |      50.265µs |       <IP_ADDRESS> | HEAD     "/"
Sep 11 18:19:31 computer ollama[67030]: [GIN] 2024/09/11 - 18:19:31 | 200 |    63.60154ms |       <IP_ADDRESS> | POST     "/api/show"
Sep 11 18:19:31 computer ollama[67030]: time=2024-09-11T18:19:31.435-06:00 level=INFO source=sched.go:715 msg="new model will fit in available VRAM in single GPU, loading" model=/var/lib/ollama/blobs/sha256-c264671c5e5347f66317f2d92db914e46dd242de0fe50f5a534991a0a15a2656 gpu=0 parallel=4 available=24412168192 required="21.7 GiB"
Sep 11 18:19:31 computer ollama[67030]: time=2024-09-11T18:19:31.436-06:00 level=INFO source=memory.go:309 msg="offload to rocm" layers.requested=-1 layers.model=41 layers.offload=41 layers.split="" memory.available="[22.7 GiB]" memory.required.full="21.7 GiB" memory.required.partial="21.7 GiB" memory.required.kv="1.2 GiB" memory.required.allocations="[21.7 GiB]" memory.weights.total="18.1 GiB" memory.weights.repeating="16.5 GiB" memory.weights.nonrepeating="1.6 GiB" memory.graph.full="1.1 GiB" memory.graph.partial="2.1 GiB"
Sep 11 18:19:31 computer ollama[67030]: time=2024-09-11T18:19:31.439-06:00 level=INFO source=server.go:391 msg="starting llama server" cmd="/tmp/ollama1037403190/runners/rocm/ollama_llama_server --model /var/lib/ollama/blobs/sha256-c264671c5e5347f66317f2d92db914e46dd242de0fe50f5a534991a0a15a2656 --ctx-size 8192 --batch-size 512 --embedding --log-disable --n-gpu-layers 41 --no-mmap --parallel 4 --port 33599"
Sep 11 18:19:31 computer ollama[67030]: time=2024-09-11T18:19:31.440-06:00 level=INFO source=sched.go:450 msg="loaded runners" count=1
Sep 11 18:19:31 computer ollama[67030]: time=2024-09-11T18:19:31.440-06:00 level=INFO source=server.go:591 msg="waiting for llama runner to start responding"
Sep 11 18:19:31 computer ollama[67030]: time=2024-09-11T18:19:31.440-06:00 level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server error"
Sep 11 18:19:31 computer ollama[67338]: INFO [main] build info | build=3535 commit="1e6f6554a" tid="127366016851008" timestamp=1726100371
Sep 11 18:19:31 computer ollama[67338]: INFO [main] system info | n_threads=16 n_threads_batch=-1 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | " tid="127366016851008" timestamp=1726100371 total_threads=32
Sep 11 18:19:31 computer ollama[67338]: INFO [main] HTTP server listening | hostname="<IP_ADDRESS>" n_threads_http="31" port="33599" tid="127366016851008" timestamp=1726100371
Sep 11 18:19:31 computer ollama[67030]: llama_model_loader: loaded meta data with 34 key-value pairs and 322 tensors from /var/lib/ollama/blobs/sha256-c264671c5e5347f66317f2d92db914e46dd242de0fe50f5a534991a0a15a2656 (version GGUF V3 (latest))
Sep 11 18:19:31 computer ollama[67030]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
Sep 11 18:19:31 computer ollama[67030]: llama_model_loader: - kv   0:                       general.architecture str              = command-r
Sep 11 18:19:31 computer ollama[67030]: llama_model_loader: - kv   1:                               general.type str              = model
Sep 11 18:19:31 computer ollama[67030]: llama_model_loader: - kv   2:                               general.name str              = C4Ai Command R 08 2024
Sep 11 18:19:31 computer ollama[67030]: llama_model_loader: - kv   3:                            general.version str              = 08-2024
Sep 11 18:19:31 computer ollama[67030]: llama_model_loader: - kv   4:                           general.basename str              = c4ai-command-r
Sep 11 18:19:31 computer ollama[67030]: llama_model_loader: - kv   5:                         general.size_label str              = 32B
Sep 11 18:19:31 computer ollama[67030]: llama_model_loader: - kv   6:                            general.license str              = cc-by-nc-4.0
Sep 11 18:19:31 computer ollama[67030]: llama_model_loader: - kv   7:                          general.languages arr[str,10]      = ["en", "fr", "de", "es", "it", "pt", ...
Sep 11 18:19:31 computer ollama[67030]: llama_model_loader: - kv   8:                      command-r.block_count u32              = 40
Sep 11 18:19:31 computer ollama[67030]: llama_model_loader: - kv   9:                   command-r.context_length u32              = 131072
Sep 11 18:19:31 computer ollama[67030]: llama_model_loader: - kv  10:                 command-r.embedding_length u32              = 8192
Sep 11 18:19:31 computer ollama[67030]: llama_model_loader: - kv  11:              command-r.feed_forward_length u32              = 24576
Sep 11 18:19:31 computer ollama[67030]: llama_model_loader: - kv  12:             command-r.attention.head_count u32              = 64
Sep 11 18:19:31 computer ollama[67030]: llama_model_loader: - kv  13:          command-r.attention.head_count_kv u32              = 8
Sep 11 18:19:31 computer ollama[67030]: llama_model_loader: - kv  14:                   command-r.rope.freq_base f32              = 4000000.000000
Sep 11 18:19:31 computer ollama[67030]: llama_model_loader: - kv  15:     command-r.attention.layer_norm_epsilon f32              = 0.000010
Sep 11 18:19:31 computer ollama[67030]: llama_model_loader: - kv  16:                          general.file_type u32              = 15
Sep 11 18:19:31 computer ollama[67030]: llama_model_loader: - kv  17:                      command-r.logit_scale f32              = 0.062500
Sep 11 18:19:31 computer ollama[67030]: llama_model_loader: - kv  18:                command-r.rope.scaling.type str              = none
Sep 11 18:19:31 computer ollama[67030]: llama_model_loader: - kv  19:                       tokenizer.ggml.model str              = gpt2
Sep 11 18:19:31 computer ollama[67030]: llama_model_loader: - kv  20:                         tokenizer.ggml.pre str              = command-r
Sep 11 18:19:31 computer ollama[67030]: llama_model_loader: - kv  21:                      tokenizer.ggml.tokens arr[str,256000]  = ["<PAD>", "<UNK>", "<CLS>", "<SEP>", ...
Sep 11 18:19:31 computer ollama[67030]: time=2024-09-11T18:19:31.691-06:00 level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server loading model"
Sep 11 18:19:31 computer ollama[67030]: llama_model_loader: - kv  22:                  tokenizer.ggml.token_type arr[i32,256000]  = [3, 3, 3, 3, 3, 3, 3, 3, 1, 1, 1, 1, ...
Sep 11 18:19:31 computer ollama[67030]: llama_model_loader: - kv  23:                      tokenizer.ggml.merges arr[str,253333]  = ["Ġ Ġ", "Ġ t", "e r", "i n", "Ġ a...
Sep 11 18:19:31 computer ollama[67030]: llama_model_loader: - kv  24:                tokenizer.ggml.bos_token_id u32              = 5
Sep 11 18:19:31 computer ollama[67030]: llama_model_loader: - kv  25:                tokenizer.ggml.eos_token_id u32              = 255001
Sep 11 18:19:31 computer ollama[67030]: llama_model_loader: - kv  26:            tokenizer.ggml.padding_token_id u32              = 0
Sep 11 18:19:31 computer ollama[67030]: llama_model_loader: - kv  27:               tokenizer.ggml.add_bos_token bool             = true
Sep 11 18:19:31 computer ollama[67030]: llama_model_loader: - kv  28:               tokenizer.ggml.add_eos_token bool             = false
Sep 11 18:19:31 computer ollama[67030]: llama_model_loader: - kv  29:           tokenizer.chat_template.tool_use str              = {{ bos_token }}{% if messages[0]['rol...
Sep 11 18:19:31 computer ollama[67030]: llama_model_loader: - kv  30:                tokenizer.chat_template.rag str              = {{ bos_token }}{% if messages[0]['rol...
Sep 11 18:19:31 computer ollama[67030]: llama_model_loader: - kv  31:                   tokenizer.chat_templates arr[str,2]       = ["tool_use", "rag"]
Sep 11 18:19:31 computer ollama[67030]: llama_model_loader: - kv  32:                    tokenizer.chat_template str              = {{ bos_token }}{% if messages[0]['rol...
Sep 11 18:19:31 computer ollama[67030]: llama_model_loader: - kv  33:               general.quantization_version u32              = 2
Sep 11 18:19:31 computer ollama[67030]: llama_model_loader: - type  f32:   41 tensors
Sep 11 18:19:31 computer ollama[67030]: llama_model_loader: - type q4_K:  240 tensors
Sep 11 18:19:31 computer ollama[67030]: llama_model_loader: - type q6_K:   41 tensors
Sep 11 18:19:32 computer ollama[67030]: llm_load_vocab: special tokens cache size = 42
Sep 11 18:19:32 computer ollama[67030]: llm_load_vocab: token to piece cache size = 1.8428 MB
Sep 11 18:19:32 computer ollama[67030]: llm_load_print_meta: format           = GGUF V3 (latest)
Sep 11 18:19:32 computer ollama[67030]: llm_load_print_meta: arch             = command-r
Sep 11 18:19:32 computer ollama[67030]: llm_load_print_meta: vocab type       = BPE
Sep 11 18:19:32 computer ollama[67030]: llm_load_print_meta: n_vocab          = 256000
Sep 11 18:19:32 computer ollama[67030]: llm_load_print_meta: n_merges         = 253333
Sep 11 18:19:32 computer ollama[67030]: llm_load_print_meta: vocab_only       = 0
Sep 11 18:19:32 computer ollama[67030]: llm_load_print_meta: n_ctx_train      = 131072
Sep 11 18:19:32 computer ollama[67030]: llm_load_print_meta: n_embd           = 8192
Sep 11 18:19:32 computer ollama[67030]: llm_load_print_meta: n_layer          = 40
Sep 11 18:19:32 computer ollama[67030]: llm_load_print_meta: n_head           = 64
Sep 11 18:19:32 computer ollama[67030]: llm_load_print_meta: n_head_kv        = 8
Sep 11 18:19:32 computer ollama[67030]: llm_load_print_meta: n_rot            = 128
Sep 11 18:19:32 computer ollama[67030]: llm_load_print_meta: n_swa            = 0
Sep 11 18:19:32 computer ollama[67030]: llm_load_print_meta: n_embd_head_k    = 128
Sep 11 18:19:32 computer ollama[67030]: llm_load_print_meta: n_embd_head_v    = 128
Sep 11 18:19:32 computer ollama[67030]: llm_load_print_meta: n_gqa            = 8
Sep 11 18:19:32 computer ollama[67030]: llm_load_print_meta: n_embd_k_gqa     = 1024
Sep 11 18:19:32 computer ollama[67030]: llm_load_print_meta: n_embd_v_gqa     = 1024
Sep 11 18:19:32 computer ollama[67030]: llm_load_print_meta: f_norm_eps       = 1.0e-05
Sep 11 18:19:32 computer ollama[67030]: llm_load_print_meta: f_norm_rms_eps   = 0.0e+00
Sep 11 18:19:32 computer ollama[67030]: llm_load_print_meta: f_clamp_kqv      = 0.0e+00
Sep 11 18:19:32 computer ollama[67030]: llm_load_print_meta: f_max_alibi_bias = 0.0e+00
Sep 11 18:19:32 computer ollama[67030]: llm_load_print_meta: f_logit_scale    = 6.2e-02
Sep 11 18:19:32 computer ollama[67030]: llm_load_print_meta: n_ff             = 24576
Sep 11 18:19:32 computer ollama[67030]: llm_load_print_meta: n_expert         = 0
Sep 11 18:19:32 computer ollama[67030]: llm_load_print_meta: n_expert_used    = 0
Sep 11 18:19:32 computer ollama[67030]: llm_load_print_meta: causal attn      = 1
Sep 11 18:19:32 computer ollama[67030]: llm_load_print_meta: pooling type     = 0
Sep 11 18:19:32 computer ollama[67030]: llm_load_print_meta: rope type        = 0
Sep 11 18:19:32 computer ollama[67030]: llm_load_print_meta: rope scaling     = none
Sep 11 18:19:32 computer ollama[67030]: llm_load_print_meta: freq_base_train  = 4000000.0
Sep 11 18:19:32 computer ollama[67030]: llm_load_print_meta: freq_scale_train = 1
Sep 11 18:19:32 computer ollama[67030]: llm_load_print_meta: n_ctx_orig_yarn  = 131072
Sep 11 18:19:32 computer ollama[67030]: llm_load_print_meta: rope_finetuned   = unknown
Sep 11 18:19:32 computer ollama[67030]: llm_load_print_meta: ssm_d_conv       = 0
Sep 11 18:19:32 computer ollama[67030]: llm_load_print_meta: ssm_d_inner      = 0
Sep 11 18:19:32 computer ollama[67030]: llm_load_print_meta: ssm_d_state      = 0
Sep 11 18:19:32 computer ollama[67030]: llm_load_print_meta: ssm_dt_rank      = 0
Sep 11 18:19:32 computer ollama[67030]: llm_load_print_meta: model type       = 35B
Sep 11 18:19:32 computer ollama[67030]: llm_load_print_meta: model ftype      = Q4_K - Medium
Sep 11 18:19:32 computer ollama[67030]: llm_load_print_meta: model params     = 32.30 B
Sep 11 18:19:32 computer ollama[67030]: llm_load_print_meta: model size       = 18.43 GiB (4.90 BPW)
Sep 11 18:19:32 computer ollama[67030]: llm_load_print_meta: general.name     = C4Ai Command R 08 2024
Sep 11 18:19:32 computer ollama[67030]: llm_load_print_meta: BOS token        = 5 '<BOS_TOKEN>'
Sep 11 18:19:32 computer ollama[67030]: llm_load_print_meta: EOS token        = 255001 '<|END_OF_TURN_TOKEN|>'
Sep 11 18:19:32 computer ollama[67030]: llm_load_print_meta: PAD token        = 0 '<PAD>'
Sep 11 18:19:32 computer ollama[67030]: llm_load_print_meta: LF token         = 136 'Ä'
Sep 11 18:19:32 computer ollama[67030]: llm_load_print_meta: max token length = 1024
Sep 11 18:19:35 computer ollama[67030]: ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
Sep 11 18:19:35 computer ollama[67030]: ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
Sep 11 18:19:35 computer ollama[67030]: ggml_cuda_init: found 1 ROCm devices:
Sep 11 18:19:35 computer ollama[67030]:   Device 0: AMD Radeon RX 7900 XTX, compute capability 11.0, VMM: no
Sep 11 18:19:35 computer ollama[67030]: llm_load_tensors: ggml ctx size =    0.31 MiB
Sep 11 18:19:36 computer ollama[67030]: time=2024-09-11T18:19:36.916-06:00 level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server not responding"
Sep 11 18:19:37 computer ollama[67030]: time=2024-09-11T18:19:37.232-06:00 level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server loading model"
Sep 11 18:19:37 computer ollama[67030]: llm_load_tensors: offloading 40 repeating layers to GPU
Sep 11 18:19:37 computer ollama[67030]: llm_load_tensors: offloading non-repeating layers to GPU
Sep 11 18:19:37 computer ollama[67030]: llm_load_tensors: offloaded 41/41 layers to GPU
Sep 11 18:19:37 computer ollama[67030]: llm_load_tensors:      ROCm0 buffer size = 18873.16 MiB
Sep 11 18:19:37 computer ollama[67030]: llm_load_tensors:  ROCm_Host buffer size =  1640.62 MiB
Sep 11 18:19:37 computer ollama[67030]: time=2024-09-11T18:19:37.683-06:00 level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server not responding"
Sep 11 18:19:47 computer ollama[67030]: time=2024-09-11T18:19:47.855-06:00 level=ERROR source=sched.go:456 msg="error loading llama server" error="llama runner process has terminated: signal: segmentation fault (core dumped)"
Sep 11 18:19:47 computer ollama[67030]: [GIN] 2024/09/11 - 18:19:47 | 500 | 16.529325999s |       <IP_ADDRESS> | POST     "/api/chat"

If it helps, here are the ROCm-related packages locally:
yay -Qs rocm
local/comgr 6.0.2-1
    Compiler support library for ROCm LLVM
local/hip-runtime-amd 6.0.2-4
    Heterogeneous Interface for Portability ROCm
local/hipblas 6.0.2-1
    ROCm BLAS marshalling library
local/hsa-rocr 6.0.2-2
    HSA Runtime API and runtime for ROCm
local/ollama-rocm 0.3.9-3
    Create, run and share large language models (LLMs) with ROCm
local/rocblas 6.0.2-1
    Next generation BLAS implementation for ROCm platform
local/rocm-core 6.0.2-2
    AMD ROCm core package (version files)
local/rocm-device-libs 6.0.2-1
    ROCm Device Libraries
local/rocm-llvm 6.0.2-1
    Radeon Open Compute - LLVM toolchain (llvm, clang, lld)
local/rocm-opencl-runtime 6.0.2-1
    OpenCL implementation for AMD
local/rocminfo 6.0.2-1
    ROCm Application for Reporting System Info
local/rocsolver 6.0.2-1
    Subset of LAPACK functionality on the ROCm platform
local/rocsparse 6.0.2-2
    BLAS for sparse computation on top of ROCm

It might be a different crash, but my suspicion is it's memory related.  To workaround this until we find the root cause, you can set num_gpu to a smaller value (try 40, 39, ...) or use the new env-var OLLAMA_GPU_OVERHEAD to reserve some VRAM so our algorithm calculates less layers to load.  Let us know how many layers do load successfully, or if this turns out not to be the root cause and the crash isn't memory related.

@remon-nashid you can run ollama serve --help to get a short description of the configuration variables, and some are discussed in more depth in various markdown files in the docs directory.

Found this issue because I was experiencing a similar thing. Gemma2 Q4_K_M would load just fine some time ago, but now in a conversation with a few back and forths between me and the AI. Setting GPU overhead to 1 GiB 'fixes' it. Loads 33 out of 47 layers.

Well, I've tried setting OLLAMA_GPU_OVERHEAD=1073741824 (the value is in bytes, right?) but the segmentation fault persists. Perhaps I need to try with different values.

The value is in bytes. Try increasing it until it doesn't crash. And if that doesn't work, manually lower num_gpu parameter in the modelfile until it loads.

Thanks @ProjectMoon for your help but I wonder, assuming that due to a bug, ollama doesn't calculate the available VRAM well, shouldn't it try to offload some layers to CPU? It used to behave that way in older versions but that as well seems to have regressed lately.

Just wanted to update you guys that I've tried an overhead up to 10 GB but still wouldn't load the model.

It should offload to CPU yes. Especially if you manually lower the number of GPU layers.
But if 10 GB overhead didn't let the model load, it sounds like something else is going on. Either overhead parameter not set correctly, not working in the code, or possibly something else entirely.
Are you using the ollama supplied ROCm or a system installed one?

I'm experiencing this as well. I'm using

Ollama 0.3.10
AMD Radeon Pro W7900 (48 GB of VRAM)
Fedora 40 with the latest hipblas and rocm installed via dnf

I get the same thing with Ubuntu 22.04 LTS

I can load Llama 3.1 8B but I can't do any of the 70B models. I'll try the suggestions here, but I think there needs to be some way for Ollama to detect and deal with this.
In the logs I just get a message that says the LLM didn't "respond" and the exit sequence starts.

Update: this still did not work, although I can't tell if it did anything. I kept on increasing the GPU overhead and restarted the server but there was no difference in behavior.

Are you using the ollama supplied ROCm or a system installed one?

I'm not aware of that ollama suppoed ROCm binaries. I have arch linux's package installed https://archlinux.org/packages/extra/x86_64/ollama-rocm/

I'm using the Fedora-supplied ROCm packages. And it works since the log shows it was compatible. I can also run Llama 3.1 8b, just not 70b

One interesting finding revealed by sending requests to ollama in parallel:

Send a request to two larger models, eg mistral-small:22b and internlm2:20b. Ollama would load one model, generate, unload it, load the next one, generate. Everything is fine.
Send a request to one larger model and two smaller models eg gemma2:2bm qwen2:1.5b and finally mistral-small:22b. Ollama loads and runs generation for the smaller models in parallel, by the time it gets to the larger model, segmentation fault occurs.

I bit the bullet on downgrading to v0.3.6 and things are working again
ollama ps
NAME                        	ID          	SIZE 	PROCESSOR	UNTIL
command-r:35b-08-2024-q4_K_M	376304b5a505	21 GB	100% GPU 	4 minutes from now

me too
ollama start
2024/09/24 01:50:45 routes.go:1153: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://<IP_ADDRESS>:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/usr/share/ollama/.ollama/models OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://<IP_ADDRESS> https://<IP_ADDRESS> http://<IP_ADDRESS>:* https://<IP_ADDRESS>:* http://<IP_ADDRESS> https://<IP_ADDRESS> http://<IP_ADDRESS>:* https://<IP_ADDRESS>:* app://* file://* tauri://*] OLLAMA_SCHED_SPREAD:false OLLAMA_TMPDIR: ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]"
time=2024-09-24T01:50:45.820Z level=INFO source=images.go:753 msg="total blobs: 5"
time=2024-09-24T01:50:45.820Z level=INFO source=images.go:760 msg="total unused blobs removed: 0"
time=2024-09-24T01:50:45.821Z level=INFO source=routes.go:1200 msg="Listening on <IP_ADDRESS>:11434 (version 0.3.11)"
time=2024-09-24T01:50:45.821Z level=INFO source=common.go:135 msg="extracting embedded files" dir=/tmp/ollama4127840924/runners
time=2024-09-24T01:51:01.096Z level=INFO source=common.go:49 msg="Dynamic LLM libraries" runners="[cuda_v12 rocm_v60102 cpu cpu_avx cpu_avx2 cuda_v11]"
time=2024-09-24T01:51:01.096Z level=INFO source=gpu.go:199 msg="looking for compatible GPUs"
time=2024-09-24T01:51:01.156Z level=WARN source=amd_linux.go:60 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-09-24T01:51:01.159Z level=INFO source=amd_linux.go:346 msg="amdgpu is supported" gpu=0 gpu_type=gfx906
time=2024-09-24T01:51:01.159Z level=INFO source=types.go:107 msg="inference compute" id=0 library=rocm variant="" compute=gfx906 driver=0.0 name=1002:66a3 total="32.0 GiB" available="32.0 GiB"
[GIN] 2024/09/24 - 01:51:35 | 200 |      49.903µs |       <IP_ADDRESS> | HEAD     "/"
[GIN] 2024/09/24 - 01:51:35 | 200 |   38.215884ms |       <IP_ADDRESS> | POST     "/api/show"
time=2024-09-24T01:51:35.308Z level=INFO source=sched.go:714 msg="new model will fit in available VRAM in single GPU, loading" model=/usr/share/ollama/.ollama/models/blobs/sha256-ba1103315c449ad06c9f5fd94230bde5bcf977f794af70afb107d29153c3cd53 gpu=0 parallel=4 available=34332102656 required="28.6 GiB"
time=2024-09-24T01:51:35.308Z level=INFO source=server.go:103 msg="system memory" total="11.7 GiB" free="8.1 GiB" free_swap="4.0 GiB"
time=2024-09-24T01:51:35.312Z level=INFO source=memory.go:326 msg="offload to rocm" layers.requested=-1 layers.model=81 layers.offload=81 layers.split="" memory.available="[32.0 GiB]" memory.gpu_overhead="0 B" memory.required.full="28.6 GiB" memory.required.partial="28.6 GiB" memory.required.kv="2.5 GiB" memory.required.allocations="[28.6 GiB]" memory.weights.total="25.9 GiB" memory.weights.repeating="25.1 GiB" memory.weights.nonrepeating="822.0 MiB" memory.graph.full="1.1 GiB" memory.graph.partial="1.1 GiB"
time=2024-09-24T01:51:35.316Z level=INFO source=server.go:388 msg="starting llama server" cmd="/tmp/ollama4127840924/runners/rocm_v60102/ollama_llama_server --model /usr/share/ollama/.ollama/models/blobs/sha256-ba1103315c449ad06c9f5fd94230bde5bcf977f794af70afb107d29153c3cd53 --ctx-size 8192 --batch-size 512 --embedding --log-disable --n-gpu-layers 81 --no-mmap --parallel 4 --port 38195"
time=2024-09-24T01:51:35.317Z level=INFO source=sched.go:449 msg="loaded runners" count=1
time=2024-09-24T01:51:35.317Z level=INFO source=server.go:587 msg="waiting for llama runner to start responding"
time=2024-09-24T01:51:35.317Z level=INFO source=server.go:621 msg="waiting for server to become available" status="llm server error"
INFO [main] build info | build=10 commit="44ccccd" tid="134189328024384" timestamp=1727142695
INFO [main] system info | n_threads=4 n_threads_batch=4 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | " tid="134189328024384" timestamp=1727142695 total_threads=8
INFO [main] HTTP server listening | hostname="<IP_ADDRESS>" n_threads_http="7" port="38195" tid="134189328024384" timestamp=1727142695
llama_model_loader: loaded meta data with 29 key-value pairs and 724 tensors from /usr/share/ollama/.ollama/models/blobs/sha256-ba1103315c449ad06c9f5fd94230bde5bcf977f794af70afb107d29153c3cd53 (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Meta Llama 3.1 70B Instruct
llama_model_loader: - kv   3:                           general.finetune str              = Instruct
llama_model_loader: - kv   4:                           general.basename str              = Meta-Llama-3.1
llama_model_loader: - kv   5:                         general.size_label str              = 70B
llama_model_loader: - kv   6:                            general.license str              = llama3.1
llama_model_loader: - kv   7:                               general.tags arr[str,6]       = ["facebook", "meta", "pytorch", "llam...
llama_model_loader: - kv   8:                          general.languages arr[str,8]       = ["en", "de", "fr", "it", "pt", "hi", ...
llama_model_loader: - kv   9:                          llama.block_count u32              = 80
llama_model_loader: - kv  10:                       llama.context_length u32              = 131072
llama_model_loader: - kv  11:                     llama.embedding_length u32              = 8192
llama_model_loader: - kv  12:                  llama.feed_forward_length u32              = 28672
llama_model_loader: - kv  13:                 llama.attention.head_count u32              = 64
llama_model_loader: - kv  14:              llama.attention.head_count_kv u32              = 8
llama_model_loader: - kv  15:                       llama.rope.freq_base f32              =<PHONE_NUMBER>00
llama_model_loader: - kv  16:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  17:                          general.file_type u32              = 10
llama_model_loader: - kv  18:                           llama.vocab_size u32              = 128256
llama_model_loader: - kv  19:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv  20:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  21:                         tokenizer.ggml.pre str              = llama-bpe
llama_model_loader: - kv  22:                      tokenizer.ggml.tokens arr[str,128256]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  23:                  tokenizer.ggml.token_type arr[i32,128256]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
time=2024-09-24T01:51:35.570Z level=INFO source=server.go:621 msg="waiting for server to become available" status="llm server loading model"
llama_model_loader: - kv  24:                      tokenizer.ggml.merges arr[str,280147]  = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
llama_model_loader: - kv  25:                tokenizer.ggml.bos_token_id u32              = 128000
llama_model_loader: - kv  26:                tokenizer.ggml.eos_token_id u32              = 128009
llama_model_loader: - kv  27:                    tokenizer.chat_template str              = {{- bos_token }}\n{%- if custom_tools ...
llama_model_loader: - kv  28:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:  162 tensors
llama_model_loader: - type q2_K:  321 tensors
llama_model_loader: - type q3_K:  160 tensors
llama_model_loader: - type q5_K:   80 tensors
llama_model_loader: - type q6_K:    1 tensors
llm_load_vocab: special tokens cache size = 256
llm_load_vocab: token to piece cache size = 0.7999 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = llama
llm_load_print_meta: vocab type       = BPE
llm_load_print_meta: n_vocab          = 128256
llm_load_print_meta: n_merges         = 280147
llm_load_print_meta: vocab_only       = 0
llm_load_print_meta: n_ctx_train      = 131072
llm_load_print_meta: n_embd           = 8192
llm_load_print_meta: n_layer          = 80
llm_load_print_meta: n_head           = 64
llm_load_print_meta: n_head_kv        = 8
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_swa            = 0
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 8
llm_load_print_meta: n_embd_k_gqa     = 1024
llm_load_print_meta: n_embd_v_gqa     = 1024
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 28672
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 500000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn  = 131072
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: ssm_dt_b_c_rms   = 0
llm_load_print_meta: model type       = 70B
llm_load_print_meta: model ftype      = Q2_K - Medium
llm_load_print_meta: model params     = 70.55 B
llm_load_print_meta: model size       = 24.56 GiB (2.99 BPW)
llm_load_print_meta: general.name     = Meta Llama 3.1 70B Instruct
llm_load_print_meta: BOS token        = 128000 '<|begin_of_text|>'
llm_load_print_meta: EOS token        = 128009 '<|eot_id|>'
llm_load_print_meta: LF token         = 128 'Ä'
llm_load_print_meta: EOT token        = 128009 '<|eot_id|>'
llm_load_print_meta: max token length = 256
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD Radeon Graphics, compute capability 9.0, VMM: no
llm_load_tensors: ggml ctx size =    0.68 MiB
llm_load_tensors: offloading 80 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 81/81 layers to GPU
llm_load_tensors:      ROCm0 buffer size = 24817.00 MiB
llm_load_tensors:  ROCm_Host buffer size =   328.78 MiB
time=2024-09-24T01:51:38.990Z level=INFO source=server.go:621 msg="waiting for server to become available" status="llm server error"
time=2024-09-24T01:51:39.241Z level=ERROR source=sched.go:455 msg="error loading llama server" error="llama runner process has terminated: signal: segmentation fault (core dumped)"

@remon-nashid so it sounds like this is not actually a memory prediction error, but a crash due to some incompatibility between the driver, rocm, and llama.cpp code.  It sounds like you're on archlinux, so installing the AMD downstream latest driver isn't an option for you and you have to stick on the older older driver bundled into the linux kernel.  (We've found the AMD downstream driver is more reliable in our testing, but has limited distro support).  It looks like you're setting HSA_OVERRIDE_GFX_VERSION=11.0.0 but that IS the gfx of your GPU, so I'm a little confused why you're doing that, as it shouldn't be necessary.  That setting disables some of our safeguard checks to verify the ROCm library being used supports the detected GPU.

I have arch linux's package installed

I can't speak to how these are built, so given it looks like we're dealing with a runtime incompatibility/mismatch between driver/rocm/ollama, can you try using our official build with our bundled ROCm?   If that fixes it, then we can shift this issue over to the archlinux maintainers of the package.  If it still fails, then we know it's something else.

Does ollama update itself? I have to cp 0.3.6 over each time to get it to start without segfault. I have no idea why that would be. I'm copying it to /usr/share/ollama

I'm running a 7600 XT on NixOS and having the same issue on both 0.3.10 and 0.3.11. Downgrading to 0.3.9 temporarily fixed the issue for me...

@dhiltgen One interesting finding revealed by sending requests to ollama in parallel:
* Send a request to **two larger models**, eg `mistral-small:22b` and `internlm2:20b`. Ollama would load one model, generate, unload it, load the next one, generate. **Everything is fine**.

* Send a request to **one larger model and two smaller models** eg `gemma2:2b`m `qwen2:1.5b` and finally `mistral-small:22b`. Ollama loads and runs generation for the smaller models in parallel, by the time it gets to the larger model, `segmentation fault` occurs.

I've noticed behavior like this too. It will sometimes seemingly get confused about how it should load or unload models, and try to load a model into the GPU while the GPU is full. Only happens in parallel processing situations. I've had it with two larger models (14 GB+ VRAM) on my main AMD GPU while a 3rd smaller model is running on my secondary ancient NVidia GPU.

That might explain why I had to turn mmap back on for a bunch of models.

--no-mmap

--no-mmap allows loading big models on 3.9 ollama without OOM.
There is another recent issue but from Nvidia card that has this pattern no-mmap and cuda out of memory 6864

i'm running 70b without --no-map on 0.3.9

i'm running 70b without --no-map on 0.3.9

Because if fit on GPU VRAM
time=2024-09-24T01:51:35.312Z level=INFO source=memory.go:326 msg="offload to rocm" layers.requested=-1 layers.model=81 layers.offload=81 layers.split="" memory.available="[32.0 GiB]" memory.gpu_overhead="0 B" memory.required.full="28.6 GiB" memory.required.partial="28.6 GiB" memory.required.kv="2.5 GiB" memory.required.allocations="[28.6 GiB]" memory.weights.total="25.9 GiB" memory.weights.repeating="25.1 GiB" memory.weights.nonrepeating="822.0 MiB" memory.graph.full="1.1 GiB" memory.graph.partial="1.1 GiB"

Partial offload on memory constrain situations can make memory paging indefinitely on Linux. And there are other situations like for APU I am able to load to GTT memory mistral large q4 with --no-mmap flag, without it max is mistral large q2K on 80Gib GTT, 96Gib RAM. Without it CPU buffer often became of a size of whole model giving 2x memory requirements one ROCM buffer one CPU buffer. But this is a problem specific to APU shared memory (VRAM is on RAM).

Hi everyone,
The way I been avoiding the segmentation fault is using Ollama bundled libraries (LD_LIBRARY_PATH).
While not ideal, it works fine. https://github.com/ollama/ollama/releases/download/v0.3.12/ollama-linux-amd64-rocm.tgz
This is the way I was able to run my tests for PR 6282 https://github.com/ollama/ollama/pull/6282#issuecomment-2375832252

For me both system libraries and the ones bundled in the zip file results in the same segfault behavior, it crashes even with OLLAMA_GPU_OVERHEAD set to ridiculous amounts (3gb), with a dGPU with 12gb ram.  Previous releases do work fine for me.

i notice that the libraries definitely have something to do with it. If i compile from soruce it is ok, but i still have to restart ollama to get it to not use the packaged libraries from a previous install, i think
the ENV variables probably have something to do with this. also, the LDs are an extra 7GB so to not use them would be ideal.

I can't speak to how these are built, so given it looks like we're dealing with a runtime incompatibility/mismatch between driver/rocm/ollama, can you try using our official build with our bundled ROCm? If that fixes it, then we can shift this issue over to the archlinux maintainers of the package. If it still fails, then we know it's something else.

I installed the official build after uninstalling the arch package but unfortunately, the same error is still reproducible.

+1 for the segmentation fault. I could run the 8b model, but not the 70b.
OS: Fedora 40 (Sway)
GPU: AMD RX 6800
Model: Llama3.1:70b
Ollama version: 3.12
I first tried installing Ollama's packaged ROCm drivers. This didn't seem to make a difference.
I tried Ollama version 3.9, and that was broken as well. Downgrading to 3.6 fixed it.
curl -fsSL https://ollama.com/install.sh | OLLAMA_VERSION=0.3.6 sh

+1 here as well.
I'm using the Docker image. I can confirm that if I roll back to 3.6 everything seems to work fine.
Host OS: Gentoo
GPU: AMD RX 7900 XTX
Model: Mixtral:8x7b
Ollama version: ollama:rocm (3.12)
Using ollama:0.3.6-rocm seems to address the issue. Setting OLLAMA_GPU_OVERHEAD would occasionally work if I set it to ludicrously high values (20 GB), but the end result was the model would (unsurprisingly) run effectively entirely on the CPU. It would only work if I used really high values like that.

So obviously one of the changes here v0.3.6...v0.3.8 introduced that bug, but hard for my untrained eyes to spot it.

this could also be a rocm 6.2.X issue. i think the timeline matches up

This happens also on 6.0.2 so it might not be that

does anyone know if this exists on the most recent versions of llama.cpp?

i ran sudo apt install libclblast-dev and it is working. i recompiled from source. gfx906

It sounds like some people seeing the regression are able to run v0.3.6.  Does anyone have confirmation the regression was in v0.3.7?   If so, it may be this commit, which brought back a compile flag that fixed problems for some people with multiple radeon cards, but if that's causing regressions for a broader set of people, maybe we need to back that out.  https://github.com/ollama/ollama/commit/0b03b9c32f483be2d7a4e902d13a909b546ae6bf

It sounds like some people seeing the regression are able to run v0.3.6. Does anyone have confirmation the regression was in v0.3.7? If so, it may be this commit, which brought back a compile flag that fixed problems for some people with multiple radeon cards, but if that's causing regressions for a broader set of people, maybe we need to back that out. 0b03b9c

Can confirm. ollama 3.6 works, 3.7 segfaults

It sounds like some people seeing the regression are able to run v0.3.6. Does anyone have confirmation the regression was in v0.3.7? If so, it may be this commit, which brought back a compile flag that fixed problems for some people with multiple radeon cards, but if that's causing regressions for a broader set of people, maybe we need to back that out. 0b03b9c

I've notice that the normal install for Ollama AMD will install it's own version of lib/rocm. When deleted, I was able to use up to Ollama 0.3.9 when compiled from source.
I managed to get Ollama 0.3.12 to run, yesterday from source. it Ran 70B, but after i restarted my computer, i had to revert back to 0.3.9.
I mentioned similar issues above a couple of weeks ago where it seems like this is an environment variable issue. I don't understand how newer versions would be able to run otherwise. Perhaps during the build phase the environment is properly set, but upon rebuild, or environment restart, the variable gets lost?
I just want to confirm, i currently  have 0.3.9 (compiled from source and with the 0.3.6 lib folder deleted) working live on Ubuntu 24.04 with ROCm 6.2.1.
0.3.6 works out of the box, and I BELIEVE i managed to get 0.3.12 to work, but i cant replicate this.

There might be multiple issues lurking in here.  The fact that OLLAMA_GPU_OVERHEAD doesn't seem to help implies it's not a simple over-allocation, but a more subtle crash in llama.cpp or ROCm, possibly specific to a combination of ROCm versions and driver versions.
I haven't managed to reproduce so far.   What would be helpful to try to narrow this down - could folks who are seeing these crashes enumerate the following details so we can try to see if we can repro?

Linux vs. Windows - for linux, what distro and version
What GPU
AMD Driver version

for Linux, was it the upstream kernel bundled amdgpu and if so, what kernel version
or the downstream AMD newer driver, and if so, what amdgpu driver version

ROCm version - bundled from Ollama, or "bring your own" and if BYO, what version
Which model causes a crash, and did you use any custom parameters or is a simple ollama run somemodel sufficient to trigger the crash

The volume of logs is intense, but it may also be helpful to run the server with AMD_LOG_LEVEL=3 and share the logs around the time of the crash in case there's any interesting warnings/errors from the ROCm logging.

Didn't comment earlier, but I believe I have been getting this issue as well

Archlinux (was a problem already 2 weeks ago, but still valid with up to date packages), running Kernel 6.11 with the amdgpu driver. Not an option to run the AMD driver on Arch.
GPU RX6600, 8GB VRAM ( I am using HSA_OVERRIDE_GFX_VERSION=10.3.0, I know the GPU is technically not supported)
CPU is an old i7 2600K, only 8GB system ram
Tried various ROCM version: the one from arch (6.0.2), the one from aur (6.2.1?), and the one packaged by ollama with no luck in either case.
I tried rebuilding from source using the ollama-rocm-git AUR package with no better luck.
All models I have tried (llama3.2, llama3.2:1b, llama3.1, phi3:mini ) are causing this error.
-Environment variables in the systemd file:  HSA_OVERRIDE_GFX_VERSION=10.3.0 OLLAMA_DEBUG=1 AMD_LOG_LEVEL=3

Please find attached the log I get when trying to run llama3.2:1b
ollama.log
From what I understand in the log, the crash happens after just after the model is done loading in the GPU.
All these model run fine in CPU (albeit slowly) if I remove the HSA_OVERRIDE_GFX_VERSION variable to disable the GPU.

I have same issue.
My environment:

2x MI25
Ubuntu Server 24.04.1 LTS
rocm version <IP_ADDRESS>202-116

I tried some versions in my environment.

22b
27b

0.3.12-rocm
unavailable
unavailable

0.3.11-rocm
unavailable
unavailable

0.3.10-rocm
unavailable
unavailable

0.3.9-rocm
unavailable
unavailable

0.3.8-rocm
unavailable
unavailable

0.3.7-rocm
unavailable
unavailable

0.3.6-rocm
work on 1 gpu
work on 2 gpus

Using models:

22b: schroneko/calm3-22b-chat:q4_0
27b: gemma2:27b

I wish it run on latest version.

0.3.13 branch Ubuntu 24.04 ROCm 6.2.1 Failed
× ollama.service - Ollama Service
     Loaded: loaded (/etc/systemd/system/ollama.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Fri 2024-10-11 00:45:43 UTC; 1h 19min ago
   Duration: 1d 21h 22min 46.223s
    Process: 16029 ExecStart=/usr/local/bin/ollama0.3.13 serve (code=exited, status=0/SUCCESS)
    Process: 16030 ExecStartPost=/usr/local/bin/ollama0.3.13 run llama3.1:70b-instruct-q2_K < /dev/null (code=exited, status=1/FAILURE)
   Main PID: 16029 (code=exited, status=0/SUCCESS)
        CPU: 3.789s

Oct 11 00:45:42 kamala ollama0.3.13[16029]: llm_load_tensors:      ROCm0 buffer size = 24817.00 MiB
Oct 11 00:45:42 kamala ollama0.3.13[16029]: llm_load_tensors:  ROCm_Host buffer size =   328.78 MiB
Oct 11 00:45:42 kamala ollama0.3.13[16029]: time=2024-10-11T00:45:42.816Z level=INFO source=server.go:621 msg="waiting for server to become available" status="llm server error"
Oct 11 00:45:43 kamala ollama0.3.13[16029]: time=2024-10-11T00:45:43.066Z level=ERROR source=sched.go:455 msg="error loading llama server" error="llama runner process has terminated: signal: segmentation fault (core dumped)"
Oct 11 00:45:43 kamala ollama0.3.13[16029]: [GIN] 2024/10/11 - 00:45:43 | 500 |  3.028948339s |       <IP_ADDRESS> | POST     "/api/generate"
Oct 11 00:45:43 kamala ollama0.3.13[16030]: [836B blob data]
Oct 11 00:45:43 kamala systemd[1]: ollama.service: Control process exited, code=exited, status=1/FAILURE
Oct 11 00:45:43 kamala systemd[1]: ollama.service: Failed with result 'exit-code'.
Oct 11 00:45:43 kamala systemd[1]: Failed to start ollama.service - Ollama Service.
Oct 11 00:45:43 kamala systemd[1]: ollama.service: Consumed 3.789s CPU time.

Linux vs. Windows - for linux, what distro and version

Arch Linux

What GPU

RX 7900 XTX

How much system RAM

32 GB

AMD Driver version

for Linux, was it the upstream kernel bundled amdgpu and if so, what kernel version

Kernel bundled driver. Kernel v6.11.3.

ROCm version - bundled from Ollama, or "bring your own" and if BYO, what version

BYO, v6.0.2.

Which model causes a crash, and did you use any custom parameters or is a simple ollama run somemodel sufficient to trigger the crash

Just running this without custom parameters causes the crash ollama run command-r:35b-08-2024-q4_K_M

Any custom env vars that you're using

Nope.
Logs below
Oct 12 07:34:54 computer ollama[2247]: time=2024-10-12T07:34:54.430-06:00 level=INFO source=sched.go:714 msg="new model will fit in available VRAM in single GPU, loading" model=/var/lib/ollama/blobs/sha256-c264671c5e5347f66317f2d92db914e46dd242de0fe50f5a534991a0a15a2656 gpu=0 parallel=1 available=24361586688 required="20.1 GiB"
Oct 12 07:34:54 computer ollama[2247]: time=2024-10-12T07:34:54.430-06:00 level=INFO source=server.go:103 msg="system memory" total="30.5 GiB" free="19.2 GiB" free_swap="0 B"
Oct 12 07:34:54 computer ollama[2247]: time=2024-10-12T07:34:54.436-06:00 level=INFO source=memory.go:326 msg="offload to rocm" layers.requested=-1 layers.model=41 layers.offload=41 layers.split="" memory.available="[22.7 GiB]" memory.gpu_overhead="0 B" memory.required.full="20.1 GiB" memory.required.partial="20.1 GiB" memory.required.kv="320.0 MiB" memory.required.allocations="[20.1 GiB]" memory.weights.total="17.1 GiB" memory.weights.repeating="15.5 GiB" memory.weights.nonrepeating="1.6 GiB" memory.graph.full="516.0 MiB" memory.graph.partial="2.1 GiB"
Oct 12 07:34:54 computer ollama[2247]: time=2024-10-12T07:34:54.439-06:00 level=INFO source=server.go:388 msg="starting llama server" cmd="/tmp/ollama122407284/runners/rocm/ollama_llama_server --model /var/lib/ollama/blobs/sha256-c264671c5e5347f66317f2d92db914e46dd242de0fe50f5a534991a0a15a2656 --ctx-size 2048 --batch-size 512 --embedding --log-disable --n-gpu-layers 41 --no-mmap --parallel 1 --port 43025"
Oct 12 07:34:54 computer ollama[2247]: time=2024-10-12T07:34:54.439-06:00 level=INFO source=sched.go:449 msg="loaded runners" count=1
Oct 12 07:34:54 computer ollama[2247]: time=2024-10-12T07:34:54.439-06:00 level=INFO source=server.go:587 msg="waiting for llama runner to start responding"
Oct 12 07:34:54 computer ollama[2247]: time=2024-10-12T07:34:54.440-06:00 level=INFO source=server.go:621 msg="waiting for server to become available" status="llm server error"
Oct 12 07:34:57 computer ollama[8361]: INFO [main] build info | build=3670 commit="84935033d" tid="123905119087680" timestamp=1728740097
Oct 12 07:34:57 computer ollama[8361]: INFO [main] system info | n_threads=16 n_threads_batch=16 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | " tid="123905119087680" timestamp=1728740097 total_threads=32
Oct 12 07:34:57 computer ollama[8361]: INFO [main] HTTP server listening | hostname="<IP_ADDRESS>" n_threads_http="31" port="43025" tid="123905119087680" timestamp=1728740097
Oct 12 07:34:57 computer ollama[2247]: time=2024-10-12T07:34:57.207-06:00 level=INFO source=server.go:621 msg="waiting for server to become available" status="llm server loading model"
Oct 12 07:34:57 computer ollama[2247]: llama_model_loader: loaded meta data with 34 key-value pairs and 322 tensors from /var/lib/ollama/blobs/sha256-c264671c5e5347f66317f2d92db914e46dd242de0fe50f5a534991a0a15a2656 (version GGUF V3 (latest))
Oct 12 07:34:57 computer ollama[2247]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
Oct 12 07:34:57 computer ollama[2247]: llama_model_loader: - kv   0:                       general.architecture str              = command-r
Oct 12 07:34:57 computer ollama[2247]: llama_model_loader: - kv   1:                               general.type str              = model
Oct 12 07:34:57 computer ollama[2247]: llama_model_loader: - kv   2:                               general.name str              = C4Ai Command R 08 2024
Oct 12 07:34:57 computer ollama[2247]: llama_model_loader: - kv   3:                            general.version str              = 08-2024
Oct 12 07:34:57 computer ollama[2247]: llama_model_loader: - kv   4:                           general.basename str              = c4ai-command-r
Oct 12 07:34:57 computer ollama[2247]: llama_model_loader: - kv   5:                         general.size_label str              = 32B
Oct 12 07:34:57 computer ollama[2247]: llama_model_loader: - kv   6:                            general.license str              = cc-by-nc-4.0
Oct 12 07:34:57 computer ollama[2247]: llama_model_loader: - kv   7:                          general.languages arr[str,10]      = ["en", "fr", "de", "es", "it", "pt", ...
Oct 12 07:34:57 computer ollama[2247]: llama_model_loader: - kv   8:                      command-r.block_count u32              = 40
Oct 12 07:34:57 computer ollama[2247]: llama_model_loader: - kv   9:                   command-r.context_length u32              = 131072
Oct 12 07:34:57 computer ollama[2247]: llama_model_loader: - kv  10:                 command-r.embedding_length u32              = 8192
Oct 12 07:34:57 computer ollama[2247]: llama_model_loader: - kv  11:              command-r.feed_forward_length u32              = 24576
Oct 12 07:34:57 computer ollama[2247]: llama_model_loader: - kv  12:             command-r.attention.head_count u32              = 64
Oct 12 07:34:57 computer ollama[2247]: llama_model_loader: - kv  13:          command-r.attention.head_count_kv u32              = 8
Oct 12 07:34:57 computer ollama[2247]: llama_model_loader: - kv  14:                   command-r.rope.freq_base f32              = 4000000.000000
Oct 12 07:34:57 computer ollama[2247]: llama_model_loader: - kv  15:     command-r.attention.layer_norm_epsilon f32              = 0.000010
Oct 12 07:34:57 computer ollama[2247]: llama_model_loader: - kv  16:                          general.file_type u32              = 15
Oct 12 07:34:57 computer ollama[2247]: llama_model_loader: - kv  17:                      command-r.logit_scale f32              = 0.062500
Oct 12 07:34:57 computer ollama[2247]: llama_model_loader: - kv  18:                command-r.rope.scaling.type str              = none
Oct 12 07:34:57 computer ollama[2247]: llama_model_loader: - kv  19:                       tokenizer.ggml.model str              = gpt2
Oct 12 07:34:57 computer ollama[2247]: llama_model_loader: - kv  20:                         tokenizer.ggml.pre str              = command-r
Oct 12 07:34:57 computer ollama[2247]: llama_model_loader: - kv  21:                      tokenizer.ggml.tokens arr[str,256000]  = ["<PAD>", "<UNK>", "<CLS>", "<SEP>", ...
Oct 12 07:34:57 computer ollama[2247]: llama_model_loader: - kv  22:                  tokenizer.ggml.token_type arr[i32,256000]  = [3, 3, 3, 3, 3, 3, 3, 3, 1, 1, 1, 1, ...
Oct 12 07:34:57 computer ollama[2247]: llama_model_loader: - kv  23:                      tokenizer.ggml.merges arr[str,253333]  = ["Ġ Ġ", "Ġ t", "e r", "i n", "Ġ a...
Oct 12 07:34:57 computer ollama[2247]: llama_model_loader: - kv  24:                tokenizer.ggml.bos_token_id u32              = 5
Oct 12 07:34:57 computer ollama[2247]: llama_model_loader: - kv  25:                tokenizer.ggml.eos_token_id u32              = 255001
Oct 12 07:34:57 computer ollama[2247]: llama_model_loader: - kv  26:            tokenizer.ggml.padding_token_id u32              = 0
Oct 12 07:34:57 computer ollama[2247]: llama_model_loader: - kv  27:               tokenizer.ggml.add_bos_token bool             = true
Oct 12 07:34:57 computer ollama[2247]: llama_model_loader: - kv  28:               tokenizer.ggml.add_eos_token bool             = false
Oct 12 07:34:57 computer ollama[2247]: llama_model_loader: - kv  29:           tokenizer.chat_template.tool_use str              = {{ bos_token }}{% if messages[0]['rol...
Oct 12 07:34:57 computer ollama[2247]: llama_model_loader: - kv  30:                tokenizer.chat_template.rag str              = {{ bos_token }}{% if messages[0]['rol...
Oct 12 07:34:57 computer ollama[2247]: llama_model_loader: - kv  31:                   tokenizer.chat_templates arr[str,2]       = ["tool_use", "rag"]
Oct 12 07:34:57 computer ollama[2247]: llama_model_loader: - kv  32:                    tokenizer.chat_template str              = {{ bos_token }}{% if messages[0]['rol...
Oct 12 07:34:57 computer ollama[2247]: llama_model_loader: - kv  33:               general.quantization_version u32              = 2
Oct 12 07:34:57 computer ollama[2247]: llama_model_loader: - type  f32:   41 tensors
Oct 12 07:34:57 computer ollama[2247]: llama_model_loader: - type q4_K:  240 tensors
Oct 12 07:34:57 computer ollama[2247]: llama_model_loader: - type q6_K:   41 tensors
Oct 12 07:34:57 computer ollama[2247]: llm_load_vocab: special tokens cache size = 42
Oct 12 07:34:58 computer ollama[2247]: llm_load_vocab: token to piece cache size = 1.8428 MB
Oct 12 07:34:58 computer ollama[2247]: llm_load_print_meta: format           = GGUF V3 (latest)
Oct 12 07:34:58 computer ollama[2247]: llm_load_print_meta: arch             = command-r
Oct 12 07:34:58 computer ollama[2247]: llm_load_print_meta: vocab type       = BPE
Oct 12 07:34:58 computer ollama[2247]: llm_load_print_meta: n_vocab          = 256000
Oct 12 07:34:58 computer ollama[2247]: llm_load_print_meta: n_merges         = 253333
Oct 12 07:34:58 computer ollama[2247]: llm_load_print_meta: vocab_only       = 0
Oct 12 07:34:58 computer ollama[2247]: llm_load_print_meta: n_ctx_train      = 131072
Oct 12 07:34:58 computer ollama[2247]: llm_load_print_meta: n_embd           = 8192
Oct 12 07:34:58 computer ollama[2247]: llm_load_print_meta: n_layer          = 40
Oct 12 07:34:58 computer ollama[2247]: llm_load_print_meta: n_head           = 64
Oct 12 07:34:58 computer ollama[2247]: llm_load_print_meta: n_head_kv        = 8
Oct 12 07:34:58 computer ollama[2247]: llm_load_print_meta: n_rot            = 128
Oct 12 07:34:58 computer ollama[2247]: llm_load_print_meta: n_swa            = 0
Oct 12 07:34:58 computer ollama[2247]: llm_load_print_meta: n_embd_head_k    = 128
Oct 12 07:34:58 computer ollama[2247]: llm_load_print_meta: n_embd_head_v    = 128
Oct 12 07:34:58 computer ollama[2247]: llm_load_print_meta: n_gqa            = 8
Oct 12 07:34:58 computer ollama[2247]: llm_load_print_meta: n_embd_k_gqa     = 1024
Oct 12 07:34:58 computer ollama[2247]: llm_load_print_meta: n_embd_v_gqa     = 1024
Oct 12 07:34:58 computer ollama[2247]: llm_load_print_meta: f_norm_eps       = 1.0e-05
Oct 12 07:34:58 computer ollama[2247]: llm_load_print_meta: f_norm_rms_eps   = 0.0e+00
Oct 12 07:34:58 computer ollama[2247]: llm_load_print_meta: f_clamp_kqv      = 0.0e+00
Oct 12 07:34:58 computer ollama[2247]: llm_load_print_meta: f_max_alibi_bias = 0.0e+00
Oct 12 07:34:58 computer ollama[2247]: llm_load_print_meta: f_logit_scale    = 6.2e-02
Oct 12 07:34:58 computer ollama[2247]: llm_load_print_meta: n_ff             = 24576
Oct 12 07:34:58 computer ollama[2247]: llm_load_print_meta: n_expert         = 0
Oct 12 07:34:58 computer ollama[2247]: llm_load_print_meta: n_expert_used    = 0
Oct 12 07:34:58 computer ollama[2247]: llm_load_print_meta: causal attn      = 1
Oct 12 07:34:58 computer ollama[2247]: llm_load_print_meta: pooling type     = 0
Oct 12 07:34:58 computer ollama[2247]: llm_load_print_meta: rope type        = 0
Oct 12 07:34:58 computer ollama[2247]: llm_load_print_meta: rope scaling     = none
Oct 12 07:34:58 computer ollama[2247]: llm_load_print_meta: freq_base_train  = 4000000.0
Oct 12 07:34:58 computer ollama[2247]: llm_load_print_meta: freq_scale_train = 1
Oct 12 07:34:58 computer ollama[2247]: llm_load_print_meta: n_ctx_orig_yarn  = 131072
Oct 12 07:34:58 computer ollama[2247]: llm_load_print_meta: rope_finetuned   = unknown
Oct 12 07:34:58 computer ollama[2247]: llm_load_print_meta: ssm_d_conv       = 0
Oct 12 07:34:58 computer ollama[2247]: llm_load_print_meta: ssm_d_inner      = 0
Oct 12 07:34:58 computer ollama[2247]: llm_load_print_meta: ssm_d_state      = 0
Oct 12 07:34:58 computer ollama[2247]: llm_load_print_meta: ssm_dt_rank      = 0
Oct 12 07:34:58 computer ollama[2247]: llm_load_print_meta: ssm_dt_b_c_rms   = 0
Oct 12 07:34:58 computer ollama[2247]: llm_load_print_meta: model type       = 35B
Oct 12 07:34:58 computer ollama[2247]: llm_load_print_meta: model ftype      = Q4_K - Medium
Oct 12 07:34:58 computer ollama[2247]: llm_load_print_meta: model params     = 32.30 B
Oct 12 07:34:58 computer ollama[2247]: llm_load_print_meta: model size       = 18.43 GiB (4.90 BPW)
Oct 12 07:34:58 computer ollama[2247]: llm_load_print_meta: general.name     = C4Ai Command R 08 2024
Oct 12 07:34:58 computer ollama[2247]: llm_load_print_meta: BOS token        = 5 '<BOS_TOKEN>'
Oct 12 07:34:58 computer ollama[2247]: llm_load_print_meta: EOS token        = 255001 '<|END_OF_TURN_TOKEN|>'
Oct 12 07:34:58 computer ollama[2247]: llm_load_print_meta: PAD token        = 0 '<PAD>'
Oct 12 07:34:58 computer ollama[2247]: llm_load_print_meta: LF token         = 136 'Ä'
Oct 12 07:34:58 computer ollama[2247]: llm_load_print_meta: max token length = 1024
Oct 12 07:35:03 computer ollama[2247]: ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
Oct 12 07:35:03 computer ollama[2247]: ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
Oct 12 07:35:03 computer ollama[2247]: ggml_cuda_init: found 1 ROCm devices:
Oct 12 07:35:03 computer ollama[2247]:   Device 0: AMD Radeon RX 7900 XTX, compute capability 11.0, VMM: no
Oct 12 07:35:03 computer ollama[2247]: llm_load_tensors: ggml ctx size =    0.31 MiB
Oct 12 07:35:03 computer ollama[2247]: llm_load_tensors: offloading 40 repeating layers to GPU
Oct 12 07:35:03 computer ollama[2247]: llm_load_tensors: offloading non-repeating layers to GPU
Oct 12 07:35:03 computer ollama[2247]: llm_load_tensors: offloaded 41/41 layers to GPU
Oct 12 07:35:03 computer ollama[2247]: llm_load_tensors:      ROCm0 buffer size = 18873.16 MiB
Oct 12 07:35:03 computer ollama[2247]: llm_load_tensors:  ROCm_Host buffer size =  1640.62 MiB
Oct 12 07:35:04 computer ollama[2247]: time=2024-10-12T07:35:04.138-06:00 level=INFO source=server.go:621 msg="waiting for server to become available" status="llm server not responding"
Oct 12 07:35:12 computer ollama[2247]: time=2024-10-12T07:35:12.958-06:00 level=ERROR source=sched.go:455 msg="error loading llama server" error="llama runner process has terminated: signal: segmentation fault (core dumped)"
Oct 12 07:35:12 computer ollama[2247]: [GIN] 2024/10/12 - 07:35:12 | 500 |  18.62111287s |       <IP_ADDRESS> | POST     "/api/generate"
Oct 12 07:35:17 computer ollama[2247]: time=2024-10-12T07:35:17.959-06:00 level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.000688676 model=/var/lib/ollama/blobs/sha256-c264671c5e5347f66317f2d92db914e46dd242de0fe50f5a534991a0a15a2656
Oct 12 07:35:18 computer ollama[2247]: time=2024-10-12T07:35:18.208-06:00 level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.250212765 model=/var/lib/ollama/blobs/sha256-c264671c5e5347f66317f2d92db914e46dd242de0fe50f5a534991a0a15a2656
Oct 12 07:35:18 computer ollama[2247]: time=2024-10-12T07:35:18.459-06:00 level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.500758542 model=/var/lib/ollama/blobs/sha256-c264671c5e5347f66317f2d92db914e46dd242de0fe50f5a534991a0a15a2656

Is this solved for everyone? I am still getting the same error.
I just tried with the git version (by using AUR package ollama-rocm-git)
In parrallel, I updated to 32GB system ram so that I could run models on the CPU while doing other things and I am still getting the error.
I can still run many models on CPU.
ollama_2024_10_12.log

Is this solved for everyone? I am still getting the same error. I just tried with the git version (by using AUR package ollama-rocm-git)
In parrallel, I updated to 32GB system ram so that I could run models on the CPU while doing other things and I am still getting the error. I can still run many models on CPU.
ollama_2024_10_12.log

this is solved in the main branch currently when you git pull.

It's possible there might be a couple distinct root causes in here, but the main issue should be resolved in 0.3.14 when we release that in the coming days.   If anyone still hits their failures on that release let us know and we'll continue to investigate.

I am currently getting the same issues with the Instinct Mi60, I can run deepseek v2 16b, and many other models. But once it reaches 27b (tested, gemma 27b) it causes segmentation error.
I did not try old versions, but I will. I have also tried to set OLLAMA_GPU_OVERHEAD. No difference.

It's possible there might be a couple distinct root causes in here, but the main issue should be resolved in 0.3.14 when we release that in the coming days. If anyone still hits their failures on that release let us know and we'll continue to investigate.

Oops didnt see this. Hopefully it does fix it.

Ollama 0.3.14-rc0 works great! Thanks @dhiltgen !


fix: Meson install header files and pkgconfig

I'll take a look at setting up a ci job for installed components. Think that should be simple


Upgrade to winapi 0.3

Hi @Draphar, thanks for your PR :+1: I will have a look at this soon. Please note that my open-source time is limited, so I might not be as quick as expected. I will get to it though. Thanks again !

@Draphar Awesome ! Thanks a lot for your help ! I'll quickly compile locally and then push the new version to crates.io. Are you OK with me adding you to README.md as a contributor ? I like giving credit where credit is due :smiley:

No problem! And I'm fine with adding me :)

@Draphar Great, thanks ! Just published to crates.io, then noticed the README.md wasn't up to date so I re-published with version 4.0.1 with the updated README. Thanks again for your help !


[Docs] Update Common Gotchas for ShallowWrapper setProps

As discussed here: https://github.com/airbnb/enzyme/issues/34 setProps does not trigger the componentDidUpdate lifecycle method by default.
I've updated the Common Gotchas section of the setProps documentation with the solution to set lifecycleExperimental to true.

Hi. The bit about lifecycleExperimental: true would have been really helpful to me if it were documented. It would be nice to see this in the documentation sometime soon. Thanks.


Handling File Input Dialogs

Is there any way to handle File Input Dialogs?
When I click an element on the page, it opens up the upload file dialog.
I can not use elementHandle.uploadFile as it is not an input[type="file"]. Plus, #1376 will not help as, dragging and dropping in the file upload field does not work, instead, dragging and dropping opens up the file in the browser tab.
I am using Windows, so there won't be the problem where, file dialog behavior is different across operating systems.
So, can you provide some example code on how would I go about doing that? Thanks!

If there is something like which could keyboard.type into the file dialog and press enter, that would do the trick. We can accomplish this using different node packages that generate keyboard events at operating system level, but, that requires the browser to always be in focus.
So, is there something like this possible in Puppeteer?

So, is there something like this possible in Puppeteer?

@KJ1i Puppeteer generally operates on the web page level; we don't support the fileinput dialog and instead "override" it with elementHandle.uploadFile.
How's the file input dialog opened? Maybe I'll be able to come up with a workaround for your case.

@aslushnikov Thanks for replying!
Here is the image of the website upload button:-

When we click on Browser button, it opens up the file dialog and, then, after selecting the file in the file dialog, it fills the  which is at the left side of the button.
The following is the website:- http://www.createspace.com
If you want, I can create an account for you here and message you its login details.
I have asked this question on StackOverflow also:- https://stackoverflow.com/questions/51491928/uploading-a-file-on-non-input-type-file
It contains more info about the question in hand.
Please let me know if more details are needed.
Thanks a lot for replying!
Awaiting your reply.

Also having the same problem as @KJ1i where clicking on an element opens a native dialog (and it's not input[type="file"]).

@KJ1i @cloudZQY @HawkiesZA this is indeed poorly supported atm; I failed to come up with any reasonable workaround for you guys.
For this to work properly, we need to expose more events from DevTools procol, similarly how we did it with javascript dialogs.
Please upvote the issue to bump its priority. We rely on "likes" to figure what's most important to do next.

sounds like a must have feature, most sites these days tend to use alternative file upload controls instead of the common file input

I would really like to see this feature supported soon !

This support is something which I am in need too, any plan to provide a fix on this?

Seems that for my specific case, I was able to work it out:

Click on the button which triggers the native file input dialog.
Look for input[type="file"] and upload your file over there.

Here's a naive example:
  await page.waitForSelector('.file-dialog-trigger')
  await page.click('.file-dialog-trigger')
  const input = await page.$('input[type="file"]')
  await input.uploadFile('./content.csv')

need this feature badly!

@nuc
I also work it out using your way.
But problem is the file dialog is still opened even after upload file succeed.
How can you close file dialog?

Looking forward for this feature. Also have this issue!

Any progress on this?

I second the need for this. I am writing a fuzz tester for our app. Works great, but I'd love to be able to upload a generic image whenever a file dialog appears.
This would be a sweet API:
page.on('filedialog', async input => {
  await input.uploadFile('./foo.jpg')
})

I confirm this is much needed

Is there any progress on this? me and my team would love for this feature to be implemented.

Agreed, this would be nice for testing server google cloud server deployments of Drupal, Wordpress, and others just to make sure files can be uploaded properly.

Trying to make it instagram.
await this.core.bot.emulate(iPhone);
let selector = "#react-root section nav div div header div.b5itu div button";
await this.core.bot.waitForSelector(selector, {timeout: 5000});
let button = await this.core.bot.$(selector);
await button.click();
let select = "input[type=\"file\"]"; //input.tb_sK
await this.core.bot.waitForSelector(select, {timeout: 5000});
let input = await this.core.bot.$(select);
let filePath = path.relative(process.cwd(), __dirname + '/assets/ByaFCAwApsN.jpg');
await input.uploadFile(filePath);

Actions are performed, but the file is not loaded. Why not working?

@Ermolaev-Nikolay try using another selector for select variable, e.g.: #react-root input[type=\"file\"]". I saw an issue before when it didn't work with the input[type=file] selector even though it was a valid one.

@slmgc, thank you for the answer, but seems the problem in another. It seems the photo is loaded, but instead of the edit window goes to main page.

@Ermolaev-Nikolay I have the exact same problem with instagram.
I think Instagram write some hidden input after click on upload button
contact me in private for more info...

@r1si, contacts in profile. I wrote you. I'll hide this comment in a few hours.

Anybody solve it? Thanks

@aslushnikov is there any documentation on file chooser api?

Many thanks for this! Right on time.

@aslushnikov is there any documentation on file chooser api?

@OmarKhattab it just was just released in 1.19.0: https://pptr.dev/#?product=Puppeteer&version=v1.19.0&show=api-class-filechooser

Thank you @aslushnikov!
I'm trying to use the file chooser but my program hangs on await fileChooser.accept. For example, the code below never prints the last line! Am I missing something?
Thank you!
const [fileChooser] = await Promise.all([
page.waitForFileChooser(),
page.click('#upload-file-button'), // some button that triggers file selection
]);
await fileChooser.accept(['/tmp/myfile.pdf']);
console.log("never print this!!!")

The new file chooser does not work for me. I get the "File chooser handling does not work with multiple connections to the same page" error no matter what, despite only having a simple example with a single page connection. See #4783.

@sekreiner sekreinerI have the same problem.
Is it solved?

I'm having the same issue. I cannot send an image to a facebook input [type=file]. Did anyone figured it out ?

if you you youse some kind of upload library (react-dropzone in my case) you can tri to trigger a change event on the input after uploading the file.

@robinjoerke 's solution works like a charm for me. Thanks for sharing!

@robinjoerke , thanks! At last somebody found a solution that is working.
@aslushnikov maybe that should be fixed on puppeteer/playwrite API level? Because it's the same on both of libraries - it does attach file to element, but upload event isn't triggered. Same cases are working with the help of webdriver with no issues.

phpmyadmin form import not working.

if you you youse some kind of upload library (react-dropzone in my case) you can try to trigger a change event on the input after uploading the file.
await page.waitForSelector('input[type=file]');
const fileInput = await page.$(''input[type=file]');
await fileInput.uploadFile("./testfile.pdf");
await fileInput.evaluate(upload => upload.dispatchEvent(new Event('change', { bubbles: true })));
this did the trick for me

Thanks, it works.

if you you youse some kind of upload library (react-dropzone in my case) you can try to trigger a change event on the input after uploading the file.
await page.waitForSelector('input[type=file]');
const fileInput = await page.$(''input[type=file]');
await fileInput.uploadFile("./testfile.pdf");
await fileInput.evaluate(upload => upload.dispatchEvent(new Event('change', { bubbles: true })));
this did the trick for me

if you you youse some kind of upload library (react-dropzone in my case) you can try to trigger a change event on the input after uploading the file.
await page.waitForSelector('input[type=file]');
const fileInput = await page.$(''input[type=file]');
await fileInput.uploadFile("./testfile.pdf");
await fileInput.evaluate(upload => upload.dispatchEvent(new Event('change', { bubbles: true })));
this did the trick for me

This code works very well thanks, but I have a problem, event object received by my event listener doesn't have a file type when I trigger a change event. Like this event.target.files[0].type =  "".  Normally, event.target.files[0].type should have a value like event.target.files[0].type =  "image/png".
What should i do.

@robinjoerke's code looks a bit hackish but has anyone actually managed to use the fileChooser API (waitForFileChooser and accept) or is it broken?

This code works very well thanks, but I have a problem, event object received by my event listener doesn't have a file type when I trigger a change event. Like this event.target.files[0].type = "". Normally, event.target.files[0].type should have a value like event.target.files[0].type = "image/png".
What should i do.
As I said I was using an upload library, I was not noticing this issue (library somehow just ignored this). But backend always recieved contentType "application/octet-stream". Should have been something as "image/png" as well. This might be an effect of the issue you have. What I do to fix this is just guess the content type from the file extension part of the filename. Not a nice solution but works for my testing scenarios.

I think the biggest problem is that the file will be sent as application/octet-stream which would break the upload on the server side that wants to treat the upload depending on the content-type (I know it shouldn't rely on it, but I'm not at a liberty to modify the receiving end's server side code) being sent from the browser which is a very different behavior from a natural browser.
I have searched around a lot but I cannot figure out how to set content-type for the multipart header. Even tried to upload via XHR after setting the file with uploadFile but no luck and applying input.files[0].type = 'image/jpeg' won't change the content-type.

if you you youse some kind of upload library (react-dropzone in my case) you can try to trigger a change event on the input after uploading the file.
await page.waitForSelector('input[type=file]');
const fileInput = await page.$(''input[type=file]');
await fileInput.uploadFile("./testfile.pdf");
await fileInput.evaluate(upload => upload.dispatchEvent(new Event('change', { bubbles: true })));
this did the trick for me

Thanks a lot!!! It works for me.

    await page.waitForSelector('input[type=file]');
    await page.waitFor(1000);
    const input = await page.$('input[type="file"]');
    await input.uploadFile(filepath);

this works without click on the button and without a remaining open windows from the browser

Seems that for my specific case, I was able to work it out:

Click on the button which triggers the native file input dialog.
Look for input[type="file"] and upload your file over there.

Here's a naive example:
  await page.waitForSelector('.file-dialog-trigger')
  await page.click('.file-dialog-trigger')
  const input = await page.$('input[type="file"]')
  await input.uploadFile('./content.csv')

You saved my life man!!!!

This works for me:
const [fileChooser] = await Promise.all([ page.waitForFileChooser(), page.click('.file-dialog-triger'), ]); await fileChooser.accept([./content.csv']);

How do you close the dialog?

How do you close the dialog?

using that code, the dialog there is no pop up dialog appear. it automatically upload the file without open file dialog

Simplest example:
const fileInput = await page.$('#file-input')
await fileInput.uploadFile(filename)


Error: error sending request for url (http://localhost:8080/api/login): connection closed before message completed

Hello,
I am going through your great Axum course on Youtube and keep running up against this problem while trying to run the tests. It doesn't happen on the first test you go through (so the simple hello get request works as intended) but all the other ones after that have this issue. Using the old school curl approach to test the server works correctly as well.
I think it might have something to do with this hyper issue: https://github.com/hyperium/hyper/issues/2136 but that's just a guess as you say this library is built on top of reqwest (which uses hyper right?).
Any clue how to overcome this issue? I like this workflow and would like to use it while developing my own projects. My apologies if this is something related to my system and nothing to do with your library.
p.s. Crates.io has the wrong repository link. It tried to take me here: https://github.com/rust-httpc-test

Can you do a cargo update and make sure you use Axum 0.12.16 or above. There was a bug in the static routing that caused such issue when requesting a static file.
Otherwise, I am not sure what it is. Will try to look at it, but it has been working pretty well on my side.
Yes, it is using reqwest.

That worked! Thank you and thanks for the great material you are putting out on YT. 0.6.15 -> 0.6.16


⚠ Processors has degraded performance

In 5e01252, Processors ($STATUS_URL) experienced degraded performance:

HTTP code: 200
Response time: 41 ms

Resolved: Processors performance has improved in c4ef356.


Fix git changelog action when tag names need shell escaping

Tag names like v1.8.0(30) were causing trouble for the shell command because some characters need escaping (In this case, the parens).

@asfalcone Let me know what you think

:shell: :rocket: LGTM!


Limiting sharing outside an MS Team following the Microsoft 365 Roadmap ID 51230

This article doesn't really show how we can lock down content in a MS Teams: SharePoint Online site with AIP ( or Unified labelling) to say just members of the Team.  The roadmap feature allows for wider sharing which we would want to prevent by default but still retain the option to share in a case by case basis; after an approval stage.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 1f4971fc-00a7-9d15-fea6-320638d059ff
Version Independent ID: aa7f0c07-94e0-d86e-0f79-292d74435cb4
Content: Microsoft Teams and SharePoint Online sites for highly regulated data
Content Source: microsoft-365/enterprise/teams-sharepoint-online-sites-highly-regulated-data.md
Service: o365-solutions
GitHub Login: @JoeDavies-MSFT
Microsoft Alias: josephd

@officedocsbot assign @mypil

@westerdaled - Thank you for submitting feedback.
I will get this issue over to the Microsoft 365 writing team for investigation.
Thank you for reporting and making the docs better. Much appreciated.
I made a note to request the team to update this when the work is complete.

@JoeDavies-MSFT - Can you please share your insights on this issue?
Thank you.

@westerdale  To lock down content to members of the Team, include the Teams group in the appropriate access groups for the isolated SharePoint site underlying the team. See https://docs.microsoft.com/en-us/office365/securitycompliance/deploy-an-isolated-sharepoint-online-team-site for info on how to do this. I can revisit the Teams sharing option when it becomes generally available.

@JoeDavies-MSFT  . This looks very similar to RBAC I have I have previously implemented on-prem.  Ok, as long as a user can't circumvent this by the new sharing dialog then were are ok.

@westerdaled - From our understanding, the issue you raised has been answered by JoeDavies-MSFT. If you feel it hasn't been answered, please re-open this issue.
Thank you for your contribution to make the docs better! Much appreciated!

@officedocsbot close

Actually, I am not so sure there is a better way to restrict file sharing within MS Teams. At the SPC 19  keynote Navjot Virk, demonstrated restricting content to members of the Executive Team only for members of a MS Team,with  Now given we don't want create and manage 100s of Unified labels for each group within the org. My view is to have a default global unified label created for each Team with this preset to "Restrict to members for current team only" .  The difficulty being we  will want to aggregate multiple sensitivity labels e.g. "Highly confidential" + "Restrict to members of current team only".


Adds Browser.close()/1

Allows for closing the whole browser, not just a page using Brower.close()/1

@dbrody, thanks for contributing this. I've gone ahead with the merge, with the addition of a couple tweaks:

I added a test of Browser.close/1, so we have at least one test explicitly covering the behavior.
I updated the documentation for the function. It now more closely matches the docs from playwright.dev, with some small language tweaks.


docs: standardize how we refer to methods in docstrings

Current behavior:
Currently we use several different ways to refer to methods, objects and arguments in the Metadata API and ngclient docstrings. Examples:
:func:`~tuf.ngclient.updater.Updater.refresh()`    # creates a clickable link in RTD, is ugly in source
Updater.refresh()    # does not stand out from text in either RTD or source code
`Updater.refresh()`    # does not stand out from text in RTD
``Updater.refresh()``    # is rendered as "inline code" in RTD

This has been partially on purpose to see what these all look like (both in published docs and in code).
Expected behavior:
Ideally we should be consistent and follow the coding style guidelines (which somewhat reasonably argue against the sphinx links). The chosen method should be useful in the published docs and not ruin the docstring readability in source code. We'd want to apply the style in docstrings in tuf/api/ and tuf/ngclient/ (as those are used to produce the ReadTheDocs api docs).
After seeing all options I think I lean on using ``refresh()``: it creates a distinct rendering on ReadTheDocs and does not ruing the docstring readability in source code like the sphinx link format does. It does not create a clickable link but usually docstring text does not need that.
https://github.com/theupdateframework/python-tuf/pull/1590 has many examples of using this format.

+1 for double back-ticks. Note to myself: Add a corresponding line to https://github.com/secure-systems-lab/code-style-guidelines/blob/master/python.md#docstrings.

oops, so I did :) carry on then


Fix bad indentation and missing hyphen on vqd.py

This PR replicates the changes done in https://github.com/Qiskit/qiskit/pull/12162 (See for information) to qiskit/algorithms/eigensolvers/vqd.py to fix the docs of two attributes (optimizer and initial_point)

Here copyright dates are checked so that the last modified date should now been 2024. Ie line 3 should look like this now (qiskit does not have this check in that repo)

(C) Copyright IBM 2022, 2024.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 90.653%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
6488

Relevant Lines:
7157

💛  - Coveralls

Oh, good catch! I didn't see it when I checked the file. I fixed it in the last commit, and I'll create another PR to do the same in the Qiskit/qiskit repo for stable/0.46. Thanks @woodsp-ibm :+1:

Docs are now live with the fix https://qiskit-community.github.io/qiskit-algorithms/stubs/qiskit_algorithms.VQD.html


Adding css not clearly documented

Hey , so when trying to install I missed the part where I have to reference the css , which is not clearly mentioned in the docs so I suggest adding that to Readme and docs site.
...
//  Reference css in the app entry point
import 'noty/lib/noty.css'  ;
...

Hi I have created a service for noty.
The text shows without design. I don't konw how to import css. My code is
import { Injectable } from '@angular/core';
import 'noty/lib/noty.css';
import * as Noty from "noty";

@Injectable()
export class NotifyService {

  constructor(public ngNoty: Noty) { }

  openNgNoty() {
  	console.log('works');
  	new Noty({
		  text: 'Notification text',
		  type: 'success',
		  layout: 'topRight',
		  theme:'metroui'
		}).show();
  }

}

Regards,
Deepak

@tony-stark001 you also need to import the css for the theme you are using, in your case...
import 'noty/lib/noty.css';
import 'noty/lib/themes/metroui.css';

Hell yeah!


Provide flush method in producer for cpp client

We already have flush() method in java api: http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/Producer.html#flush--
It is better to provide a same method for cpp client.

pr #3020 is for this issue

pr merged


Admin user should not be able to delete themselves

User Story
As an Automate User, I will use this work when I accidentally try to delete my own user while logged in as that user, which would in cutting off their own access to Automate.
Definition of Done

[ ] the Local Administrator (username: admin) cannot delete her own user admin when she is logged in as that user. Only bob, a different user with user-delete permissions, can delete admin

Demo Script / Repro Steps

log in as the admin user
navigate to the User list page
attempt to delete the admin user
get blocked by a 403 error
grant new user bob admin permissions
log in as bob
navigate to the User list page
successfully delete the admin user

Is this one specific way to cut yourself off of Automate standing out, though? You can still create a policy that kills your access, or delete one that enables your access. You can remove yourself from a a group, or add yourself to a group, which could have the same effect (qua policies for that group). Before, we've sometimes discussed this, and I still think that helping the user unblock themselves is better than trying to foresee what they could do wrong, and forbid them doing it. (This is how they started with HAL, I'm sure.)

Looks like this functionality is already in place. I logged in as admin and tried to delete the admin user and got this error message: Could not delete user: unauthorized: members ["team:local:admins" "user:local:admin"] cannot perform action "iam:users:delete" on resource "iam:users:admin" filtered by projects []

whoa did it just start working? cool.


[BUGFIX] Remove online state enrichment in describe_devices() by default

Skip online state enrichment in describe_devices() by default

[ ] Enhancement
[ ] Major feature update
[x] Bug fixes
[ ] Breaking change
[ ] Updated unit tests
[ ] Documentation

Bandit analysis
Run started:2023-05-25 13:39:24.885780

Test results:
        No issues identified.

Code scanned:
        Total lines of code: 5937
        Total lines skipped (#nosec): 1

Run metrics:
        Total issues (by severity):
                Undefined: 0
                Low: 0
                Medium: 0
                High: 0
        Total issues (by confidence):
                Undefined: 0
                Low: 0
                Medium: 0
                High: 0
Files skipped (0):

Issues resolved

Bug fix: #99

Other

describe_devices() has a new optional parameter that defaults to False to specify if you want to enrich the results with the online state of the device. More testing is needed for large environments before setting that parameter to True.

@kenoel we just need a version bump in pyproject.toml and then I'll get this merged. Thank you! We'll figure out the large environment issue separately against a larger test group.


cherry-pick #910 to 2.4.x

Description of changes
Cherry-pick #910

Looks like CI is failing due to use of the is operator in the formatter tests. Can you switch is to a different operator?

Looks like CI is failing due to use of the is operator in the formatter tests. Can you switch is to a different operator?

Done.


"Import steam categories" button disappeared ?

Hello
Sorry if I just can't find the functionality, or if the functionality has been removed for some reason, but I can't find the "Steam import categories/hidden flag" any more.
Thanks in advance

It was removed because it no longer worked properly after recent Steam UI rework. It won't be coming back unless Valve add proper API for it.

Too bad... Thank you very much for your answer. Regards?


Can't Download rhcos 4.5.6 files

TASK [download coreos kernel] ***************************************************************************************************
Tuesday 25 August 2020  14:26:21 +0900 (0:00:00.051)       0:00:03.033 ********
FAILED - RETRYING: download coreos kernel (3 retries left).
FAILED - RETRYING: download coreos kernel (2 retries left).
FAILED - RETRYING: download coreos kernel (1 retries left).
fatal: [<IP_ADDRESS>]: FAILED! => {"attempts": 3, "changed": false, "msg": "Unable to find a checksum for file 'rhcos-4.5.6-x86_64-installer-kernel-x86_64' in 'https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.5/4.5.6/sha256sum.txt'"}

Make get_url fail the Hush comparison because https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.5/4.5.6/sha256sum.txt  has the same hash value with different filenames.
This problem has never existed before.
For example in 4.5.2 the hash value is unique.
https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.5/4.5.2/sha256sum.txt

checksum_map is overwritten with the file name corresponding to the checksum written after the file, so the file name written first does not seem to be compared...
https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/get_url.py#L505-L521

get_url is unable to find a checksum for file · Issue #71420 · ansible/ansible
https://github.com/ansible/ansible/issues/71420

Interim workaround:
diff --git a/vars/download.yml b/vars/download.yml
index 9a708a9..f21c63d 100644
--- a/vars/download.yml
+++ b/vars/download.yml
@@ -28,18 +28,21 @@ config:
   coreos_kernel: >-
     {%- if   openshift.dist == 'okd' -%}                                   fedora-coreos-{{ openshift.coreos_version }}-live-kernel-x86_64
     {%- elif openshift.coreos_version | regex_search("^4\.2\.") -%}        rhcos-{{ openshift.coreos_version }}-x86_64-installer-kernel
+    {%- elif openshift.coreos_version | regex_search("^4\.5\.6") -%}       rhcos-installer-kernel-x86_64
     {%- elif openshift.coreos_version | regex_search("^4\.+([3-5])\.") -%} rhcos-{{ openshift.coreos_version }}-x86_64-installer-kernel-x86_64
     {%- else -%}                                                           rhcos-{{ openshift.coreos_version }}-x86_64-installer-kernel-x86_64
     {%- endif -%}
   coreos_initramfs: >-
     {%- if   openshift.dist == 'okd' -%}                                   fedora-coreos-{{ openshift.coreos_version }}-live-initramfs.x86_64.img
     {%- elif openshift.coreos_version | regex_search("^4\.2\.") -%}        rhcos-{{ openshift.coreos_version }}-x86_64-installer-initramfs.img
+    {%- elif openshift.coreos_version | regex_search("^4\.5\.6") -%}       rhcos-installer-initramfs.x86_64.img
     {%- elif openshift.coreos_version | regex_search("^4\.+([4-5])\.") -%} rhcos-{{ openshift.coreos_version }}-x86_64-installer-initramfs.x86_64.img
     {%- else -%}                                                           rhcos-{{ openshift.coreos_version }}-x86_64-installer-initramfs.x86_64.img
     {%- endif -%}
   coreos_bios: >-
     {%- if   openshift.dist == 'okd' -%}                                   fedora-coreos-{{ openshift.coreos_version }}-metal.x86_64.raw.xz
     {%- elif openshift.coreos_version | regex_search("^4\.2\.") -%}        rhcos-{{ openshift.coreos_version }}-x86_64-metal-bios.raw.gz
+    {%- elif openshift.coreos_version | regex_search("^4\.5\.6") -%}       rhcos-metal.x86_64.raw.gz
     {%- elif openshift.coreos_version | regex_search("^4\.+([3-5])\.") -%} rhcos-{{ openshift.coreos_version }}-x86_64-metal.x86_64.raw.gz
     {%- else -%}                                                           rhcos-{{ openshift.coreos_version }}-x86_64-metal.x86_64.raw.gz
     {%- endif -%}

This issue was solved with upstream ansible.
We can wait for the backport.
https://github.com/ansible/ansible/commit/159544610e5a925d589198febaa5198778ea51c0

workaround:
This is a workaround until the fix(ansible/ansible@1595446) is incorporated into Fedora / RHEL.
$ git checkout fix-get_url
$ ./fix-handle_same_sha_for_dl_files.sh


Azure DevOps Search Result not displaying matches

Repro Steps

Opened an Azure DevOps repo in vscode.dev.
Searched for something in the repo

Expected Results

Search result UI shows the line where the match was found

Actual Results

The UI displays the character offsets

Interesting part is if i click on one of those charOffset lines, open the file, and close the file the search results UI updates to the expected result

@lszomoru I can reproduce this but only on certain repositories, it seems like even though we are passing includeSnippet: true in our request, the results that come back from the search/codesearchresults endpoint have a null codeSnippet, so we end up displaying the fallback values as shown above as placeholders.
The reason that the search results UI updates to the expected result is because we are able to use the open file's content to resolve the search results, but that will only work after you open the file.

@joyceerhl, I believe that we have seen this type of behaviour with very large repositories hosted in Azure Repos. @monil-patel, could you please reach out to me privately on Teams and provide the details of the repository where you are seeing this behaviour so that we can validate that everything is configured as expected in Azure DevOps. Thanks!

I am seeing a similar issue where the devops api response is not providing anything for the matches response. I am searching my whole devops Org for a certain string and want to retrieve the code snippet. Right now, I receive the file name and project and other information like that, but not the matches information
@lszomoru

Still a problem


Impossible to use AntiVPN with Java 17 (--illegal-access-permit flag deprecated)

Plugin version: 6.4.13
Server type: Paper
Server version: 1.17.1
Java version: 17
Issue: The only thing that kept AntiVPN working on Java 16 was the startup flag "--illegal-access-permit"
but this has been removed in Java 17, no way of using AntiVPN now, had to switch to another which I won't name here.
Is this the end of AntiVPN?
Expected Behavior: Plugin to work without special flag or old java version, others do it why can't this one?
I run all of my servers on Java 17 now, not a single issue/error with any other plugin or software in sight. Except for this of course.
I understand the project might be abandoned but I will leave this report here so others can know!

Minecraft 1.18 onwards will be requiring a minimum Java version of Java 17. Minecraft 1.18 releases on November 30th which will cause this plugin to break on any servers that are updating.
@Laarryy what are some problems with updating the plugin to support Java 17 and could there be anything potential contributors could assist in? If you could explain what the issues are with this plugins Java 17 support I'm willing to try to assist.


[WIP] Fix Windows wheel and add auditwheel.

@henryiii Since it looks like you're not working on this branch at the moment, I'm going to use it to try to fix the Windows part. That seems to be orthogonal to what you're doing. I'll be temporarily turning off all deployment builds except Windows, so we probably can't work on it at the same time. Let me know if this affects your plans.

I think I nearly had it, fixed the one mistake, so macOS and Linux should work when they come back on.
(Sorry, forgot to skip ci!)

Because you are using your own personal Azure instead of the Scikit-HEP one, I can't cancel the build. Feel free to do so.

@henryiii I'm done with WOWW. Anything more that you want to do with the PR is up to you; let me know when I should merge it.
I'd prefer to use the Scikit-HEP Azure. That's another thing that we should switch over to. Not today, though.

After tests pass (are they running?), if you could trigger a deploy just to make sure everything works and so I can see what the output of the auditwheel show is, that should be great.

I think the tests are waiting on Yana's "final cleanup" in PR #132 to finish. Just a minute or two...