Add support for older AMD GPU gfx803 (e.g. Radeon RX 580)

Officially ROCm no longer supports these cards, but it looks like other projects have found workarounds.  Let's explore if that's possible.  Best case, built-in to our binaries.  Fall-back if that's not plausible is document how to build from source with the appropriate older ROCm library and AMD drivers installed on your system and build a local binary that works.

One interesting observation.  I managed to get this gfx803 card not to crash with the invalid free by uninstalling the rocm libs on the host, and copying the exact libs from the build container over, however, when running models on the card, the responses were gibberish, so clearly it's more than just library dependencies and will require compile time changes.

I'm trying to get this working on an RX 580.
With the 6.0.0-2 rocm packages on arch, I was getting free(): invalid pointer from clinfo (maybe a related issue).
In the logs after sending a "prompt" (not sure of the lingo?).
rocBLAS error: Cannot read /opt/rocm/lib/rocblas/library/TensileLibrary.dat: Illegal seek for GPU arch : gfx803

I notice in the rocblas cmake file file that they removed support for gfx803 for the 6.0.X builds, so I downgraded to the 5.7.1 packages and rebuilt ollama using the PKGBUILD from #2473
Then when I sent the prompt I get this error:
Feb 19 19:43:16 tokyo ollama[130295]: /usr/lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/bits/random.tcc:2665: void std::discrete_distribution<>::param_type::_M_initialize() [_IntType = int]: Assertion '__sum > 0' failed.

The assertion is coming from stdlibc++ here, so maybe if I change the PKGBUILD to build a different version of ollama, that might get fixed, I'll try that next.
Not sure how much help I can be here, but I can test things out if needed.
This is the full output in the logs:
Feb 19 19:38:10 tokyo systemd[1]: Started Ollama Service.
Feb 19 19:38:10 tokyo ollama[130295]: time=2024-02-19T19:38:10.155-06:00 level=INFO source=images.go:863 msg="total blobs: 6"
Feb 19 19:38:10 tokyo ollama[130295]: time=2024-02-19T19:38:10.155-06:00 level=INFO source=images.go:870 msg="total unused blobs removed: 0"
Feb 19 19:38:10 tokyo ollama[130295]: [GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.
Feb 19 19:38:10 tokyo ollama[130295]: [GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.

Feb 19 19:38:10 tokyo ollama[130295]: time=2024-02-19T19:38:10.367-06:00 level=INFO source=payload_common.go:145 msg="Dynamic LLM libraries [rocm_v5 cpu cpu_avx cpu_avx2]"
Feb 19 19:38:10 tokyo ollama[130295]: time=2024-02-19T19:38:10.367-06:00 level=INFO source=gpu.go:94 msg="Detecting GPU type"
Feb 19 19:38:10 tokyo ollama[130295]: time=2024-02-19T19:38:10.367-06:00 level=INFO source=gpu.go:242 msg="Searching for GPU management library libnvidia-ml.so"
Feb 19 19:38:10 tokyo ollama[130295]: time=2024-02-19T19:38:10.374-06:00 level=INFO source=gpu.go:288 msg="Discovered GPU libraries: []"
Feb 19 19:38:10 tokyo ollama[130295]: time=2024-02-19T19:38:10.374-06:00 level=INFO source=gpu.go:242 msg="Searching for GPU management library librocm_smi64.so"
Feb 19 19:38:10 tokyo ollama[130295]: time=2024-02-19T19:38:10.374-06:00 level=INFO source=gpu.go:288 msg="Discovered GPU libraries: [/opt/rocm/lib/librocm_smi64.so.5.0]"
Feb 19 19:38:10 tokyo ollama[130295]: time=2024-02-19T19:38:10.378-06:00 level=INFO source=gpu.go:109 msg="Radeon GPU detected"
Feb 19 19:38:10 tokyo ollama[130295]: time=2024-02-19T19:38:10.378-06:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
Feb 19 19:43:06 tokyo ollama[130295]: [GIN] 2024/02/19 - 19:43:06 | 200 |      41.169µs |       <IP_ADDRESS> | HEAD     "/"
Feb 19 19:43:06 tokyo ollama[130295]: [GIN] 2024/02/19 - 19:43:06 | 200 |     498.618µs |       <IP_ADDRESS> | POST     "/api/show"
Feb 19 19:43:06 tokyo ollama[130295]: time=2024-02-19T19:43:06.221-06:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
Feb 19 19:43:06 tokyo ollama[130295]: time=2024-02-19T19:43:06.221-06:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
Feb 19 19:43:06 tokyo ollama[130295]: time=2024-02-19T19:43:06.221-06:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
Feb 19 19:43:06 tokyo ollama[130295]: time=2024-02-19T19:43:06.256-06:00 level=INFO source=dyn_ext_server.go:90 msg="Loading Dynamic llm server: /tmp/ollama460181430/rocm_v5/libext_server.so"
Feb 19 19:43:06 tokyo ollama[130295]: time=2024-02-19T19:43:06.256-06:00 level=INFO source=dyn_ext_server.go:145 msg="Initializing llama server"
Feb 19 19:43:06 tokyo ollama[130295]: ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no
Feb 19 19:43:06 tokyo ollama[130295]: ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes
Feb 19 19:43:06 tokyo ollama[130295]: ggml_init_cublas: found 1 ROCm devices:
Feb 19 19:43:06 tokyo ollama[130295]:   Device 0: AMD Radeon RX 580 Series, compute capability 8.0, VMM: no
Feb 19 19:43:06 tokyo ollama[130295]: llama_model_loader: loaded meta data with 20 key-value pairs and 291 tensors from /var/lib/ollama/.ollama/models/blobs/sha256:3a43f93b78ec50f7c4e4dc8bd1cb3fff5a900e7d574c51a6f7495e48486e0dac (version GGUF V2)
Feb 19 19:43:06 tokyo ollama[130295]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
Feb 19 19:43:06 tokyo ollama[130295]: llama_model_loader: - kv   0:                       general.architecture str              = llama
Feb 19 19:43:06 tokyo ollama[130295]: llama_model_loader: - kv   1:                               general.name str              = codellama
Feb 19 19:43:06 tokyo ollama[130295]: llama_model_loader: - kv   2:                       llama.context_length u32              = 16384
Feb 19 19:43:06 tokyo ollama[130295]: llama_model_loader: - kv   3:                     llama.embedding_length u32              = 4096
Feb 19 19:43:06 tokyo ollama[130295]: llama_model_loader: - kv   4:                          llama.block_count u32              = 32
Feb 19 19:43:06 tokyo ollama[130295]: llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 11008
Feb 19 19:43:06 tokyo ollama[130295]: llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128
Feb 19 19:43:06 tokyo ollama[130295]: llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 32
Feb 19 19:43:06 tokyo ollama[130295]: llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 32
Feb 19 19:43:06 tokyo ollama[130295]: llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
Feb 19 19:43:06 tokyo ollama[130295]: llama_model_loader: - kv  10:                       llama.rope.freq_base f32              = 1000000.000000
Feb 19 19:43:06 tokyo ollama[130295]: llama_model_loader: - kv  11:                          general.file_type u32              = 2
Feb 19 19:43:06 tokyo ollama[130295]: llama_model_loader: - kv  12:                       tokenizer.ggml.model str              = llama
Feb 19 19:43:06 tokyo ollama[130295]: llama_model_loader: - kv  13:                      tokenizer.ggml.tokens arr[str,32016]   = ["<unk>", "<s>", "</s>", "<0x00>", "<...
Feb 19 19:43:06 tokyo ollama[130295]: llama_model_loader: - kv  14:                      tokenizer.ggml.scores arr[f32,32016]   = [0.000000, 0.000000, 0.000000, 0.0000...
Feb 19 19:43:06 tokyo ollama[130295]: llama_model_loader: - kv  15:                  tokenizer.ggml.token_type arr[i32,32016]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
Feb 19 19:43:06 tokyo ollama[130295]: llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1
Feb 19 19:43:06 tokyo ollama[130295]: llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2
Feb 19 19:43:06 tokyo ollama[130295]: llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0
Feb 19 19:43:06 tokyo ollama[130295]: llama_model_loader: - kv  19:               general.quantization_version u32              = 2
Feb 19 19:43:06 tokyo ollama[130295]: llama_model_loader: - type  f32:   65 tensors
Feb 19 19:43:06 tokyo ollama[130295]: llama_model_loader: - type q4_0:  225 tensors
Feb 19 19:43:06 tokyo ollama[130295]: llama_model_loader: - type q6_K:    1 tensors
Feb 19 19:43:06 tokyo ollama[130295]: llm_load_vocab: mismatch in special tokens definition ( 264/32016 vs 259/32016 ).
Feb 19 19:43:06 tokyo ollama[130295]: llm_load_print_meta: format           = GGUF V2
Feb 19 19:43:06 tokyo ollama[130295]: llm_load_print_meta: arch             = llama
Feb 19 19:43:06 tokyo ollama[130295]: llm_load_print_meta: vocab type       = SPM
Feb 19 19:43:06 tokyo ollama[130295]: llm_load_print_meta: n_vocab          = 32016
Feb 19 19:43:06 tokyo ollama[130295]: llm_load_print_meta: n_merges         = 0
Feb 19 19:43:06 tokyo ollama[130295]: llm_load_print_meta: n_ctx_train      = 16384
Feb 19 19:43:06 tokyo ollama[130295]: llm_load_print_meta: n_embd           = 4096
Feb 19 19:43:06 tokyo ollama[130295]: llm_load_print_meta: n_head           = 32
Feb 19 19:43:06 tokyo ollama[130295]: llm_load_print_meta: n_head_kv        = 32
Feb 19 19:43:06 tokyo ollama[130295]: llm_load_print_meta: n_layer          = 32
Feb 19 19:43:06 tokyo ollama[130295]: llm_load_print_meta: n_rot            = 128
Feb 19 19:43:06 tokyo ollama[130295]: llm_load_print_meta: n_embd_head_k    = 128
Feb 19 19:43:06 tokyo ollama[130295]: llm_load_print_meta: n_embd_head_v    = 128
Feb 19 19:43:06 tokyo ollama[130295]: llm_load_print_meta: n_gqa            = 1
Feb 19 19:43:06 tokyo ollama[130295]: llm_load_print_meta: n_embd_k_gqa     = 4096
Feb 19 19:43:06 tokyo ollama[130295]: llm_load_print_meta: n_embd_v_gqa     = 4096
Feb 19 19:43:06 tokyo ollama[130295]: llm_load_print_meta: f_norm_eps       = 0.0e+00
Feb 19 19:43:06 tokyo ollama[130295]: llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
Feb 19 19:43:06 tokyo ollama[130295]: llm_load_print_meta: f_clamp_kqv      = 0.0e+00
Feb 19 19:43:06 tokyo ollama[130295]: llm_load_print_meta: f_max_alibi_bias = 0.0e+00
Feb 19 19:43:06 tokyo ollama[130295]: llm_load_print_meta: n_ff             = 11008
Feb 19 19:43:06 tokyo ollama[130295]: llm_load_print_meta: n_expert         = 0
Feb 19 19:43:06 tokyo ollama[130295]: llm_load_print_meta: n_expert_used    = 0
Feb 19 19:43:06 tokyo ollama[130295]: llm_load_print_meta: rope scaling     = linear
Feb 19 19:43:06 tokyo ollama[130295]: llm_load_print_meta: freq_base_train  = 1000000.0
Feb 19 19:43:06 tokyo ollama[130295]: llm_load_print_meta: freq_scale_train = 1
Feb 19 19:43:06 tokyo ollama[130295]: llm_load_print_meta: n_yarn_orig_ctx  = 16384
Feb 19 19:43:06 tokyo ollama[130295]: llm_load_print_meta: rope_finetuned   = unknown
Feb 19 19:43:06 tokyo ollama[130295]: llm_load_print_meta: model type       = 7B
Feb 19 19:43:06 tokyo ollama[130295]: llm_load_print_meta: model ftype      = Q4_0
Feb 19 19:43:06 tokyo ollama[130295]: llm_load_print_meta: model params     = 6.74 B
Feb 19 19:43:06 tokyo ollama[130295]: llm_load_print_meta: model size       = 3.56 GiB (4.54 BPW)
Feb 19 19:43:06 tokyo ollama[130295]: llm_load_print_meta: general.name     = codellama
Feb 19 19:43:06 tokyo ollama[130295]: llm_load_print_meta: BOS token        = 1 '<s>'
Feb 19 19:43:06 tokyo ollama[130295]: llm_load_print_meta: EOS token        = 2 '</s>'
Feb 19 19:43:06 tokyo ollama[130295]: llm_load_print_meta: UNK token        = 0 '<unk>'
Feb 19 19:43:06 tokyo ollama[130295]: llm_load_print_meta: LF token         = 13 '<0x0A>'
Feb 19 19:43:06 tokyo ollama[130295]: llm_load_tensors: ggml ctx size =    0.22 MiB
Feb 19 19:43:06 tokyo ollama[130295]: llm_load_tensors: offloading 32 repeating layers to GPU
Feb 19 19:43:06 tokyo ollama[130295]: llm_load_tensors: offloading non-repeating layers to GPU
Feb 19 19:43:06 tokyo ollama[130295]: llm_load_tensors: offloaded 33/33 layers to GPU
Feb 19 19:43:06 tokyo ollama[130295]: llm_load_tensors:      ROCm0 buffer size =  3577.61 MiB
Feb 19 19:43:06 tokyo ollama[130295]: llm_load_tensors:        CPU buffer size =    70.35 MiB
Feb 19 19:43:07 tokyo ollama[130295]: .................................................................................................
Feb 19 19:43:07 tokyo ollama[130295]: llama_new_context_with_model: n_ctx      = 2048
Feb 19 19:43:07 tokyo ollama[130295]: llama_new_context_with_model: freq_base  = 1000000.0
Feb 19 19:43:07 tokyo ollama[130295]: llama_new_context_with_model: freq_scale = 1
Feb 19 19:43:07 tokyo ollama[130295]: llama_kv_cache_init:      ROCm0 KV buffer size =  1024.00 MiB
Feb 19 19:43:07 tokyo ollama[130295]: llama_new_context_with_model: KV self size  = 1024.00 MiB, K (f16):  512.00 MiB, V (f16):  512.00 MiB
Feb 19 19:43:07 tokyo ollama[130295]: llama_new_context_with_model:  ROCm_Host input buffer size   =    12.01 MiB
Feb 19 19:43:07 tokyo ollama[130295]: llama_new_context_with_model:      ROCm0 compute buffer size =   171.60 MiB
Feb 19 19:43:07 tokyo ollama[130295]: llama_new_context_with_model:  ROCm_Host compute buffer size =     8.80 MiB
Feb 19 19:43:07 tokyo ollama[130295]: llama_new_context_with_model: graph splits (measure): 3
Feb 19 19:43:07 tokyo ollama[130295]: time=2024-02-19T19:43:07.868-06:00 level=INFO source=dyn_ext_server.go:156 msg="Starting llama main loop"
Feb 19 19:43:07 tokyo ollama[130295]: loading library /tmp/ollama460181430/rocm_v5/libext_server.so
Feb 19 19:43:07 tokyo ollama[130295]: {"timestamp":1708393387,"level":"VERBOSE","function":"start_loop","line":266,"message":"have new task"}
Feb 19 19:43:07 tokyo ollama[130295]: {"timestamp":1708393387,"level":"VERBOSE","function":"start_loop","line":281,"message":"callback_all_task_finished"}
Feb 19 19:43:07 tokyo ollama[130295]: {"timestamp":1708393387,"level":"VERBOSE","function":"start_loop","line":302,"message":"wait for new task"}
Feb 19 19:43:07 tokyo ollama[130295]: {"timestamp":1708393387,"level":"VERBOSE","function":"start_loop","line":266,"message":"have new task"}
Feb 19 19:43:07 tokyo ollama[130295]: {"timestamp":1708393387,"level":"VERBOSE","function":"start_loop","line":278,"message":"callback_new_task"}
Feb 19 19:43:07 tokyo ollama[130295]: {"timestamp":1708393387,"level":"VERBOSE","function":"start_loop","line":281,"message":"callback_all_task_finished"}
Feb 19 19:43:07 tokyo ollama[130295]: {"timestamp":1708393387,"level":"VERBOSE","function":"update_slots","line":1623,"message":"prompt ingested","n_past":0,"cached":"","to_eval":" [INST] <<SYS>><</SYS>>\n\nWrite me a function that outputs the fibonacci sequence in C. [/INST]\n"}
Feb 19 19:43:16 tokyo ollama[130295]: /usr/lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/bits/random.tcc:2665: void std::discrete_distribution<>::param_type::_M_initialize() [_IntType = int]: Assertion '__sum > 0' failed.
Feb 19 19:43:17 tokyo systemd[1]: ollama.service: Main process exited, code=dumped, status=6/ABRT
Feb 19 19:43:17 tokyo systemd[1]: ollama.service: Failed with result 'core-dump'.
Feb 19 19:43:17 tokyo systemd[1]: ollama.service: Consumed 5.707s CPU time.
Feb 19 19:43:20 tokyo systemd[1]: ollama.service: Scheduled restart job, restart counter is at 2.

I ended up disabling _GLIBCXX_ASSERTIONS in /etc/makepkg.conf and I am starting to get some responses, but they are gibberish, at least sometimes. I think the problem is in llama.cpp  perhaps some sort of UB in the use of std::discrete_distribution that was triggering the assert. This is the only place I could find it being used. And a discussion which seems to resemble what's going on.
This is where libstdc++ was asserting in c++/13.2.1/bits/random.tcc on line 2665:
  template<typename _IntType>
    void
    discrete_distribution<_IntType>::param_type::
    _M_initialize()
    {
      ...
      const double __sum = std::accumulate(_M_prob.begin(),
					   _M_prob.end(), 0.0);
      __glibcxx_assert(__sum > 0);
      // Now normalize the probabilites.
      ...
    }

So it seems like the sum should be greater than 0, idk what the implications are, but that seems to be one of preconditions of using this type which llama.cpp is violating. May have some impact on the maths involved (which I am totally oblivious to).
I tried this:
ollama run codellama "Write me a function that outputs the fibonacci sequence in C."
and it just output a bunch of
############################## forever until I ctrl-c
Running the llama2 model:
>>> Why is the sky blue?
OOOlatooOwnGootUNSIreetOreoooohatГ GovernGBUNootIadruo delegladred EderGAootOALOO rangrehojection byther sywn�OOOLmoootGC�regnhatiloonoOWooAINTegruophonOalkOreouтоSIO го nobody.

I don't know if it's just messing with me, or if the bug is random.
Next try using codellama example:
➜  ollama-rocm git:(rocm) ✗ ollama run codellama
>>> Write me a function that outputs the fibonacci sequence

  Solution

``
def fibonacci(n):
    if n <= 1:
        return n
    else:
        return fibonacci(n-1) + fibonacci(n-2)
``

This function uses a recursive approach to calculate the `n`th number in the Fibonacci sequence. The base case is when `n <= 1`, in which case the function returns `n` directly. Otherwise,
it calculates the result by adding the previous two values in the sequence.

For example:
``
fibonacci(0) -> 0
fibonacci(1) -> 1
fibonacci(2) -> fibonacci(1) + fibonacci(0) -> 1 + 0 -> 1
fibonacci(3) -> fibonacci(2) + fibonacci(1) -> (1 + 0) + 1 -> 2
fibonacci(4) -> fibonacci(3) + fibonacci(2) -> (2 + 1) + 1 -> 4
fibonacci(5) -> fibonacci(4) + fibonacci(3) -> (4 + 2) + 1 -> 7
``

>>> Now do the same in C++

  Solution

``
int fibonacci(int n) {
    if (n <= 1) {
        return n;
    } else {
        return fibonacci(n-1) + fibonacci(n-2);
    }
}
``
This function uses a recursive approach to calculate the `n`th number in the Fibonacci sequence. The base case is when `n <= 1`, in which case the function returns `n` directly. Otherwise,
it calculates the result by adding the previous two values in the sequence.

For example:
``
fibonacci(0) -> 0
fibonacci(1) -> 1
fibonacci(2) -> fibonacci(1) + fibonacci(0) -> 1 + 0 -> 1
fibonacci(3) -> fibonacci(2) + fibonacci(1) -> (1 + 0) + 1 -> 2
fibonacci(4) -> fibonacci(3) + fibonacci(2) -> (2 + 1) + 1 -> 4
fibonacci(5) -> fibonacci(4) + fibonacci(3) -> (4 + 2) + 1 -> 7
``

ollama run llama2 "Why is the sky blue?"
 pedkO NO NO  O-   a                          dark in  a  .     to  in   of’ in  gar  a  of     in aj            a                           .  for  dark    in  a   at a   in   a a in
a     a   of   to     the    IN of aj    very  gall    dark   in    and in     in     gar      in   dark         a  ap  for    the a  theaj  pro      .  in   in  gall IN  will  a of   and
’aj gar to  in O in  in  to a no    no of   ofaj a can  to  dark a by for the    in   pro gall of  ap a   dark in  and have   f of a  for are   ’  that   a in by  under only’ a. in  rock
the aaj a gar in  no due as in  to gar   in  friendly  a in     a   a  dark a  in     a in       for  aj to a  in dark in   have   a ofaj   a by a         a    were     a   in  a   at dark
 very the gall a gar  . ap ano a a CL a aj  a  the a  aj  to   as  gallaj     in     aj   aaj of  and   gar. of in f  a  in  at  dark    will    dark    a                for  in     of a
in    to     following the  a   to   in a   aj CL  dark    in a       aj     gall a    a  gar aj O a          aj for  of in  a   a  friendly   and  a    a the dark  in     to  a   dark
a in a   in    gar    to gar   as ajaj   a  that   for        the of  in   at       friendly  O     gall       previous  a in     a a  ’     dark  back        aj in                   the
 a   a  to    a     a    a         a              in     aj   a    at  in aj aj a  of   a  gar           a              in        the       for IN     aj         in  a           ^V     in a
 in   in^C

➜  ollama-rocm git:(rocm) ✗ ollama run codellama "Write me a function that outputs the fibonacci sequence in C."

Here's an example of how you could calculate the Fibonacci sequence in C:
``
#include <stdio.h>

int main() {
  int n, i;
  long long int fib[n];

  // Ask user for input
  printf("Enter a number: ");
  scanf("%d", &n);

  // Initialize first two elements of the sequence
  fib[0] = 0;
  fib[1] = 1;

  // Calculate remaining elements of the sequence
  for (i = 2; i < n; i++) {
    fib[i] = fib[i-1] + fib[i-2];
  }

  // Output the calculated sequence
  printf("The Fibonacci sequence is: ");
  for (i = 0; i < n; i++) {
    printf("%lld ", fib[i]);
  }

  return 0;
}
``
This program will ask the user to input a number `n`, and then calculate the first `n` elements of the Fibonacci sequence. The output will be the calculated sequence, with each element
separated by a space.

For example, if the user inputs `5`, the output will be:
``
The Fibonacci sequence is: 0 1 1 2 3 5
``
Note that this program uses an array to store the elements of the sequence, and loops through the elements to calculate them. The `long long int` type is used to avoid overflowing the
integer range when calculating larger Fibonacci numbers.

@wilkensgomes
for the error
rocBLAS error: Cannot read /opt/rocm/lib/rocblas/library/TensileLibrary.dat: Illegal seek for GPU arch : gfx803
I downgraded to 5.7.1 rocm packages using downgrade on arch linux and then added them to Ignore at the end of the installation so that they don't get upgraded to 6.X packages.
For the error:
Feb 19 19:43:16 tokyo ollama[130295]: /usr/lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/bits/random.tcc:2665: void std::discrete_distribution<>::param_type::_M_initialize() [_IntType = int]: Assertion '__sum > 0' failed.
I turned off _GLIBCXX_ASSERTIONS when building ollama, in /etc/makepkg.conf
# CXXFLAGS="$CFLAGS -Wp,-D_GLIBCXX_ASSERTIONS"
CXXFLAGS="$CFLAGS"

There might be a better way to disabling this in the PKGBUILD file just for building ollama/llama.cpp, but I haven't bothered with it, and just disabled it locally.
Reading over the discussion for the second error, the gibberish happens after disabling the asserts, as the initialize method for std::discrete_distribution<> requires that the sum of the probabilities are greater than 0, this make sense. AFAIK it doesn't make sense for a probability to be negative, or NAN, or all 0, which are the cases I can think of that would trigger the assertion after summing the probabilities.
So as far as I can tell the gibberish is a result from certain models and small input prompts as said in the conversation. Somewhere between the model and the calculation of the probabilities, either some of them are negative, all are zero, or there is a NaN in there. For example, if for some reason a probability is a result of dividing a float by 0.0 p =  x / y where y is 0.0 then p = NaN and then when llama.cpp calls llama_sample_token() and std::discrete_distribution calls std::accumulate then the result will be NaN, I can only imagine how that would mess up the LLM when trying to figure out the next word to use. At least this is as far as my understanding goes.
Apart from some of the smaller models and a small input prompts that produce gibberish, everything has been working for me since yesterday. I'm not even sure if the gibberish is particular to polaris gpus. I spent a few hours using llama2:13b as a Dungeon Master yesterday, was mind blowing.

I'm still getting familiar with these code bases, but I did some print debugging in llama_sample_softmax and llama_sample_token and sure enough, there are nans everywhere on short prompt, it's fairly reproducible on my end.
I built both ollama and llama.cpp from their respective main branches, but took out the check for AMD version > 9 in ollama.
In file llama.cpp, with the logging that I put in.
void llama_sample_softmax(struct llama_context * ctx, llama_token_data_array * candidates) {
    //...
    //...
    float max_l = candidates->data[0].logit;
    float cum_sum = 0.0f;
    std::stringstream plogs;
    for (size_t i = 0; i < candidates->size; ++i) {
        float p = expf(candidates->data[i].logit - max_l);
        candidates->data[i].p = p;
        cum_sum += p;
    }
    for (size_t i = 0; i < candidates->size - 1; ++i) {
        candidates->data[i].p /= cum_sum;
        plogs << "{ token: " << candidates->data[i].id
            << ", probability: " << candidates->data[i].p
            << ", logit: " << candidates->data[i].logit
            << "},\n";
    }
    candidates->data[candidates->size - 1].p /= cum_sum;
    plogs << "{ token: " << candidates->data[candidates->size - 1].id
        << ", probability: " << candidates->data[candidates->size - 1].p
        << ", logit: " << candidates->data[candidates->size - 1].logit
        << " }\n";

    std::string plogs_string = plogs.str();

    LLAMA_LOG_INFO("Probabilities: [%s]\n", plogs_string.data());
    //...
}

I'll do my best to track down where the nans are coming from, it might be the gpu, which I have little experience in. I might try building rocm6.x from source if I can find an option to enable gfx803 support in the cmake files, and then build against that in case it's a bug in rocm 5.7.1 that I have installed.
Short prompt, nans, nans everywhere:
➜  ollama-rocm ollama run llama2
>>> Why is the sky blue?
####################################################################################^C

[Server side]
{"cached":"","function":"update_slots","level":"VERB","line":1876,"msg":"prompt ingested","n_past":0,"tid":"139442878678720","timestamp":1708984989,
"to_eval":" [INST] <<SYS>><</SYS>>\n\nWhy is the sky blue? [/INST]\n"}
Probabilities: [{ token: 38, probability: nan, logit: nan},
{ token: 22, probability: nan, logit: nan},
{ token: 10, probability: nan, logit: nan},
{ token: 34, probability: nan, logit: nan},
{ token: 26, probability: nan, logit: nan},
{ token: 18, probability: nan, logit: nan},
{ token: 20, probability: nan, logit: nan},
{ token: 4, probability: nan, logit: nan},
{ token: 24, probability: nan, logit: nan},
{ token: 12, probability: nan, logit: nan},
{ token: 32, probability: nan, logit: nan},
{ token: 28, probability: nan, logit: nan},
{ token: 16, probability: nan, logit: nan},
{ token: 36, probability: nan, logit: nan},
{ token: 8, probability: nan, logit: nan},
{ token: 39, probability: nan, logit: nan},
{ token: 9, probability: nan, logit: nan},
{ token: 21, probability: nan, logit: nan},
{ token: 1, probability: nan, logit: nan},
{ token: 23, probability: nan, logit: nan},
{ token: 11, probability: nan, logit: nan},
{ token: 25, probability: nan, logit: nan},
{ token: 5, probability: nan, logit: nan},
{ token: 27, probability: nan, logit: nan},
{ token: 13, probability: nan, logit: nan},
{ token: 31, probability: nan, logit: nan},
{ token: 29, probability: nan, logit: nan},
{ token: 15, probability: nan, logit: nan},
{ token: 33, probability: nan, logit: nan},
{ token: 7, probability: nan, logit: nan},
{ token: 35, probability: nan, logit: nan},
{ token: 17, probability: nan, logit: nan},
{ token: 37, probability: nan, logit: nan},
{ token: 3, probability: nan, logit: nan},
{ token: 19, probability: nan, logit: nan},
{ token: 0, probability: nan, logit: nan},
{ token: 2, probability: nan, logit: nan},
{ token: 6, probability: nan, logit: nan},
{ token: 14, probability: nan, logit: nan},
{ token: 30, probability: nan, logit: nan }
]

A little bit longer prompt, the calculations look right here:
>>> Why is the sky blue? Please explain it like I'm 5 years old. Use colorful language, but try
... to also explain the science.

Oh my goodness, let me tell you a secret about the sky! *winks* It's so cool! *excited
tone* The sky is blue because of something called light. *giggles* You know how things can
look different colors when the light hits them from different angles? Like how a red apple
looks red when the sun shines on it, but green when it's in shadow? Well, the sky does
that too! *excited nod*

So, when the sun shines on the Earth, it sends out all sorts of different colored lights.
*giggles* Like, did you know that light can be red, orange, yellow, green, blue, and
purple? Yep! And when these colors hit the Earth's atmosphere, they bounce around and mix
together to make the sky look blue! It's like a big ol' party in the sky! *giggles*
....
....

[Server Side]
{"cached":"","function":"update_slots","level":"VERB","line":1876,"msg":"prompt ingested","n_past":0,"tid":"137691662386880","timestamp":1708985350,
"to_eval":" [INST] <<SYS>><</SYS>>\n\nWhy is the sky blue? Please explain it like I'm 5 years old.
 Use colorful language, but try to also explain the science. [/INST]\n"}
Probabilities: [{ token: 13, probability: 0.999558, logit: 24.8022},
{ token: 9048, probability: 0.000197175, logit: 16.2713},
{ token: 6439, probability: 0.000159607, logit: 16.0599},
{ token: 243, probability: 2.25609e-05, logit: 14.1034},
{ token: 23170, probability: 1.64754e-05, logit: 13.789},
{ token: 18527, probability: 9.54511e-06, logit: 13.2432},
{ token: 11284, probability: 5.65038e-06, logit: 12.7189},
{ token: 29956, probability: 4.85538e-06, logit: 12.5673},
{ token: 7030, probability: 4.57228e-06, logit: 12.5072},
{ token: 9360, probability: 3.22113e-06, logit: 12.1569},
{ token: 2, probability: 3.0808e-06, logit: 12.1124},
{ token: 17565, probability: 2.19157e-06, logit: 11.7718},
{ token: 18637, probability: 2.17322e-06, logit: 11.7634},
{ token: 5674, probability: 1.88993e-06, logit: 11.6237},
{ token: 3611, probability: 1.08913e-06, logit: 11.0725},
{ token: 3257, probability: 8.80308e-07, logit: 10.8597},
{ token: 29930, probability: 7.47415e-07, logit: 10.696},
{ token: 9070, probability: 6.84936e-07, logit: 10.6087},
{ token: 1148, probability: 6.06698e-07, logit: 10.4874},
{ token: 29979, probability: 5.32776e-07, logit: 10.3575},
{ token: 9806, probability: 4.99208e-07, logit: 10.2924},
{ token: 2776, probability: 4.0466e-07, logit: 10.0825},
{ token: 8187, probability: 3.9383e-07, logit: 10.0553},
{ token: 29949, probability: 3.60271e-07, logit: 9.96626},
{ token: 10994, probability: 3.32183e-07, logit: 9.88509},
{ token: 29898, probability: 2.46243e-07, logit: 9.58573},
{ token: 1068, probability: 2.04658e-07, logit: 9.40075},
{ token: 27269, probability: 1.98792e-07, logit: 9.37167},
{ token: 827, probability: 1.87496e-07, logit: 9.31317},
{ token: 5872, probability: 1.8679e-07, logit: 9.30939},
{ token: 5634, probability: 1.83938e-07, logit: 9.29401},
{ token: 22110, probability: 1.67096e-07, logit: 9.19798},
{ token: 1532, probability: 1.5426e-07, logit: 9.11805},
{ token: 229, probability: 1.44642e-07, logit: 9.05367},
{ token: 9800, probability: 1.24852e-07, logit: 8.90654},
{ token: 399, probability: 1.23796e-07, logit: 8.89804},
{ token: 14962, probability: 1.16326e-07, logit: 8.83581},
{ token: 8851, probability: 1.08058e-07, logit: 8.76208},
{ token: 29909, probability: 1.06776e-07, logit: 8.75014},
{ token: 8879, probability: 9.68724e-08, logit: 8.6528 }
]

More detailed logs:
llama.cpp.good.log
llama.cpp.nan.log

Is it not possible to create a docker that supports gfx803? It would be easier than doing trial and error.

This issue on llama.cpp seems to be the same bug.
I'm currently going through the Rocm stack and building it from source using the main branches and trying to find out if I can reintroduce rx580 "support" with patches if needed. I will put up a script and patches if I'm successful in that and it solves the problem. We could create a docker image from that script, or just use the script to create binary packages, or PKGBUILDS if it comes to that. Various parts of the stack still seem to "support" gfx803 (rx580), while other seem to have at least officially dropped it, like rocBLAS (though it might still work if I just patch up the build scripts).
I don't think this is a bug in ollama, but further down the stack. For example, clr introduced a free(): invalid pointer bug somewhere between 6.0.0 (unreleased) and 6.0.2 tags, that was the reason I downgraded to 5.7.1. So it's a matter of finding which commit introduced that bug.
As for the gibberish, I think that's a result of nans coming from somewhere. It seems to be specific to gfx803, otherwise a lot more users would be reporting it, and that bug also occurs in rocm 5.7.1.
It might be worth trying even older versions of rocm than 5.7.1 if ollama and llama.cpp are still compatible with those, at least in the meantime. Adding support for older gpus without requiring downgrading rocm doesn't seem possible if rocm isn't going to support older gpus in the first place, users would still have to install older versions, or at least would require re-implementing that functionality.
If the gibberish is coming from clBLAST, then that narrows that down and rocm support for older gpus is just a side issue, I think users will either have to work on support in the open source, or just use older packages.

Any progress on this... ROCm successfully detects my gfx803 and it should work but ollama is blocking the card :/

Could this also be applied to gfx804?

Support for Radeon RX 580/590 (I have a 590) would be super nice. Tried Ollama 0.1.30 update and is not possible yet.

Please add support Older GPU's like RX 580 as Llama.cpp already support those GPU's

@Todd-Fulton That's a regression with ROCm versions 6.0.* (see https://github.com/rocm-arch/rocm-arch/issues/981). Downgrading to 5.7.1 will enable support for, e.g., Polaris cards again.

RX 580

True, using CLBlast.

Please add support Older GPU's like RX 580 as Llama.cpp already support those GPU's

True, using CLBlast.

@6b6279 Can you give me detailed Instructions how to downgrade to 5.7.1 on Arch? I got an Rx 580

@DerRehberg Try downgrade rocm-opencl-runtime and choose 5.7.1 as the target version. Don't forget to add the package in IgnorePkg to pin that version until you manually update.

@6b6279 Which Packages do I need to downgrade with? I downgraded rocm-opencl-runtime but ollama and everything else doesn't seem to work, probably because plain rocm is still at 6.0

@6b6279 And now give me detailed instruction how to run Stable Diffusion on an RX 580

@DerRehberg Try downgrade rocm-opencl-runtime and choose 5.7.1 as the target version. Don't forget to add the package in IgnorePkg to pin that version until you manually update.
(downgrade is available on the AUR: https://aur.archlinux.org/packages/downgrade)
ollama won't use the GPU regardless, but it'll enable support for, e.g., the RX 580, while using darktable.

comgr, hsa-rocr, hsakmt-roct, rocm-cmake, rocm-device-libs, rocm-llvm, rocm-opencl-runtime

@DerRehberg No idea. I use rocm only for image processing.

Is there any update to this? I have a 580 and would like to use it in addition to another gpu.

Helo. I'm a user of an Radeon Rx580 8GB and the statement that

Officially ROCm no longer supports these cards

is not entirely true. While it is not officially supported anymore you don't really need any workarounds to make ROCm work with these GPUs. I've been using OpenCL through ROCm for quite some time in Blender without any issues at all. All I needed to do is set an environment variable: ROC_ENABLE_PRE_VEGA=1 and the GPU just worked.
I've tried dong so with Ollama but it seems that it disables the GPU manually as unsupported even if ROCm is able to run on it.
From ArchWiki

unofficial and partial support for Navi10 based cards. To support cards older than Vega, you need to set the runtime variable ROC_ENABLE_PRE_VEGA=1.

[!Note]
I haven't used blender for some time and I switched to NixOS so I didn't test it right now. But if someone wants me to I'll look into it and see whether I can run ROCm on tha card without any additional setup.

I think this is really exciting as a RX580 on ebay is much more affordable. If support could be added it should be possible to built an AI machine for under 200 USD.

闲鱼上的rx580/rx590 200-300元。如果ollama能支持rx580就太好了。或者有什么方案能让用户自己构建ollama也可以。

For example: GPT4All 3.0 (Vulkan support) works great with RX580 (8GB). Tested with Windows 10 and latest AMD driver.

For example: GPT4All 3.0 (Vulkan support) works great with RX580 (8GB). Tested with Windows 10 and latest AMD driver.

Also tested GPT4All 3.0 on an old PC with an RX580 (8GB) and it works nicely on Windows 10, with ROC_ENABLE_PRE_VEGA env variable set to 1, and HSA_OVERRIDE_GFX_VERSION set to "10.3.0". AMD HIP SDK version 5.5.1 only, not 5.7.1.

for ollama running on gfx803 on windows, just check https://github.com/likelovewant/ollama-for-amd

@KazeLiu  rx580 === rx 570 , they are both gfx803.

@yourchanges
Thank you, this is my first time installing a large model, and I still don't quite understand the process. My environment is an internal offline network. My current process is to first enter the ollama-for-amd project, then download ollama-windows-amd64.7z and OllamaSetup.exe from version 0.2.8. After transferring them to the internal network, I first install OllamaSetup and then extract ollama-windows-amd64.7z, replacing the files in the Ollama folder. Then, I enter the ROCmLibs-for-gfx1103-AMD780M-APU project and download version <IP_ADDRESS> of rocm.gfx803.optic.vega10.logic.hip.sdk.<IP_ADDRESS>z. I download it and replace the files in the rocm folder within the Ollama folder. Then, I run the glm4 that I downloaded before going offline. When I check using ollama ps, I find that the CPU is still at 100%. I want to know which step I didn't do correctly.
谢谢，我是第一次安装大模型，这边还是不太明白内容，我这边的环境是内网离线环境。我现在的流程就是先进入ollama-for-amd项目，然后下载V0.2.8版本中的ollama-windows-amd64.7z和OllamaSetup.exe。传输到内网后，我先安装了OllamaSetup然后把ollama-windows-amd64.7z解压并替换掉了Ollama文件夹内的文件。然后进入了ROCmLibs-for-gfx1103-AMD780M-APU项目，下载 v<IP_ADDRESS> 的 rocm.gfx803.optic.vega10.logic.hip.sdk.<IP_ADDRESS>z。下载并替换掉Ollama文件夹内rocm文件夹内的文件。然后我将断网之前下载的glm4运行，用ollama ps查看，发现CPU还是100%。我想知道我是哪一步没有做好。

You should check your logs of the ollama serve ,

may be try  this https://github.com/YellowRoseCx/koboldcpp-rocm/releases/tag/v1.71.1.yr0-ROCm,it's work fine for me.my gpu card is amd rx580,os is almalinux 9.if ollama not work,try koboldcpp-rocm instead.before install koboldcpp-rocm,rocm 5.7.1 enviorment must be ready.try this https://www.youtube.com/watch?v=ljXdEih2GOI&ab_channel=LinuxMadeEZ video step by step,finish the rocm 5.7.1 enviorment ready,good luck.

came across this as i am in the same situation with an old RX580 without any budget to get something new und unable to get it running, neither ollama (only cpu) nor comfyui.. trying koboldcpp but not all models are supported there...
Very sad, please let me know if you need any information, debug output etc...
Hope you can implement it some day 👍
cheers

came across this as i am in the same situation with an old RX580 without any budget to get something new und unable to get it running, neither ollama (only cpu) nor comfyui.. trying koboldcpp but not all models are supported there...
Very sad, please let me know if you need any information, debug output etc...
Hope you can implement it some day 👍
cheers

Hi
May be try https://discuss.linuxcontainers.org/t/llama-cpp-and-ollama-servers-plugins-for-vs-code-vs-codium-and-intellij-ai/19744 ，you can recompile ollama code for rx580.
You are right,koboldcpp is not all models are supported,even have stable issue.
May be the llamacpp or ollama is the good choice.
Good luck.
Jerome

As alternative, maybe pushing Vulkan support would be reasonable? It's here:
https://github.com/ollama/ollama/issues/2033

As alternative, maybe pushing Vulkan support would be reasonable? It's here: #2033

The LM Studio is support vulkan，but still have stable issue.I have previous experience using LM Studio. After running several LLM models on LM Studio, it suddenly stopped working properly, and Koboldcpp also started experiencing issues.
So, I am currently reprogramming with Docker, using Rocm to reinstall and compile Ollama, allowing Ollama to run LLM models on Docker with the RX580. Since running Stable Diffusion in the Rocm environment seems very stable—I’ve generated dozens of images over an entire day without any issues—I’m confident this setup will work well.
Perhaps these experiences might be useful as a reference.
Good Luck.
Jerome

@jeromechungmf have you tried alternative to LM Studio, called gpt4all? They're having AFAIK best Vulkan support for LLM acceleration. I'm using RX 5500M and it does wonders, where ROCm installation for this GPU, on Ubuntu 24.04, is a nightmare or hackfest, or boht.

@jeromechungmf have you tried alternative to LM Studio, called gpt4all? They're having AFAIK best Vulkan support for LLM acceleration. I'm using RX 5500M and it does wonders, where ROCm installation for this GPU, on Ubuntu 24.04, is a nightmare or hackfest, or boht.

Finally, i found the docker image can support the rx580 gpu card for ollama,  https://hub.docker.com/r/bergutman/ollama-rocm .
if you still use rx580 gpu card,docker pull the bergutman/ollama-rocm image,follow the overview ,use the docker-compose,run the bergutman/ollama-rocm image build to docker container.you can use 'docker logs containerID' command,see the ollama can get rx580 vram 8G.
I will likely continue testing the RX580 paired with Ollama to run various LLM large language models.
Hope these experiences might be useful as a reference.
Good Luck
Jerome

@jeromechungmf have you tried alternative to LM Studio, called gpt4all? They're having AFAIK best Vulkan support for LLM acceleration. I'm using RX 5500M and it does wonders, where ROCm installation for this GPU, on Ubuntu 24.04, is a nightmare or hackfest, or boht.

Finally, i found the docker image can support the rx580 gpu card for ollama, https://hub.docker.com/r/bergutman/ollama-rocm . if you still use rx580 gpu card,docker pull the bergutman/ollama-rocm image,follow the overview ,use the docker-compose,run the bergutman/ollama-rocm image build to docker container.you can use 'docker logs containerID' command,see the ollama can get rx580 vram 8G.
I will likely continue testing the RX580 paired with Ollama to run various LLM large language models.
Hope these experiences might be useful as a reference.
Good Luck
Jerome

After try download llama3 model for bergutman/ollama-rocm,i get error: template: :7:3: executing "" at <.response>: can't evaluate field response in type struct { first bool; system string; prompt string } #4057 ，the ollama version too old.
It's very sad.
Try llamacpp on docker or LM-Studio on Ubuntu 22.04 is my next step.
Jerome

Hello guys,
I am very interested in AI models for Ollama for text translation.
And I even managed to run Gemma 2 2B and Llama 3.1 8B on N100 CPU and Debian 12 OS.
The possibilities of their intelligence I liked very much.
Unfortunately the speed of these models on N100 is not very fast.
So I want to increase their speed with a graphics card.
I can only afford an old SAPHIRE NITRO RX480 8 GB graphics card.
But and unfortunately Ollama refuses to work with it.
Yes, I know there is still no official support for this graphics card: https://ollama.com/blog/amd-preview
There are a lot of discussions and attempts to implement this support on the web,  for example: 1, 2, 3
And maybe some real solution for RX580 has already been found?
Tell me about it, please!

I have some success running Ollama using the 5.7.1 Rocm libraries with Radeon RX 580 GPU, within a docker container. Only tried so far with the llama3.1 model but seems to work ok.
The process I went through was just to focus and fix errors I encountered while trying to get something working, so this is by no means a proper fix for this issue.
The changes are based on the main Ollama branch from 6th Sept 2024, it would be interesting if somebody else could try to see if it works them too.
Repo with the changes - https://github.com/mnccouk/ollama/tree/rx580_gpu
See the top of the readme for some basic instructions to build. I only focused on the docker build, so make sure to install Rocm libraries on the docker host machine first (5.7.1).

Hello @Tamila-2017
I have been trying to run AI large language models (LLMs) on my PC using an RX580 GPU. I've tried two main approaches so far:
My Experience:
Linux with Vulkan:
I install Ubuntu 24.04 or 22.04, configuring it to use the Vulkan drivers, and then installing LMStudio. The goal is to make this setup work with my RX580 GPU for optimal LLM performance. However, attempting to install Vulkan drivers directly from AMD's website (for Ubuntu 22.04) proved unsuccessful. Instead, I found success by downloading the corresponding Vulkan driver package from https://vulkan.lunarg.com/sdk/home#linux
This allowed LMStudio to detect my RX580 GPU and use it for LLM calculations.
My Installation:
Currently, I am running Ubuntu Release 1.3.290 Ubuntu 22.04.
Recommendations
Thorough Driver Research: Before installing anything, make sure you download the correct Vulkan driver based on your Ubuntu version from https://vulkan.lunarg.com/sdk/home#linux
Windows with Vulkan:
In Windows OS Enviroment，just download AMD Driver Installer ， follow the GUI step by step，Finish the gpu driver install.The Vulkan driver for windows os have been ready .

Hi guys,
I am very grateful to all of you for your detailed account of how to solve my problem with the RX580.
Now I appeared a great chance to solve this problem and use RX580 to communicate with AI.
Can you please tell me, what is the speed increase of working with RX580 with AI compared to СPU N100?
Is it 10x, 100x, or maybe 1000x? :-)

You can all try ollama-for-amd fork. I'm using it myself with my RX5500M card and ROCm 6.2. works wonders.

KhazAkar,
This is also very interesting information. Could you tell us more about this?
For example, what exactly is the name of this fork, where can I get it and how to use it.
And most importantly, you are using the RX 5500M.
Will this fork work with the RX580?

jeromechungmf
I chose your advice first:

Linux with Vulkan:
I install Ubuntu 24.04 or 22.04, configuring it to use the Vulkan drivers, and then installing LMStudio. The goal is to make this setup work with my RX580 GPU for optimal LLM performance. However, attempting to install Vulkan drivers directly from AMD's website (for Ubuntu 22.04) proved unsuccessful. Instead, I found success by downloading the corresponding Vulkan driver package from https://vulkan.lunarg.com/sdk/home#linux
This allowed LMStudio to detect my RX580 GPU and use it for LLM calculations.

And it ended in failure. What I did, in detail:

Install Ubuntu-24.04.1 LTS Server Minimal (without X's)
 wget -qO- https://packages.lunarg.com/lunarg-signing-key-pub.asc | sudo tee /etc/apt/trusted.gpg.d/lunarg.asc sudo wget -qO /etc/apt/sources.list.d/lunarg-vulkan-noble.list https://packages.lunarg.com/vulkan/lunarg-vulkan-noble.list sudo apt update sudo apt install vulkan-sdk
Download LM_Studio-0.3.2.AppImage to /home/ai/.appimage
Run LM Studio: needs FUSE
sudo add-apt-repository universe sudo apt install libfuse2t64
Run LM Studio: needs libnss3.so
sudo apt install libnss3
Run LM Studio: needs libasound
sudo apt install libasound2t64
Run LM Studio:

FATAL:setuid_sandbox_host.cc(158)] The SUID sandbox helper binary was found, but is not configured correctly.
Rather than run without sandboxing I'm aborting now.
You need to make sure that /tmp/.mount_LM_StusEylzO/chrome-sandbox is owned by root and has mode 4755.

But this file does not exist!
Next, I began to understand that a graphical environment is required to run LM Studio and I did not like it at all, because I want to using AI in a console environment, without X's.
That's why I stopped this experiment, and now I want to try KhazAkar advice.
But unfortunately he's not giving any details yet :-(

Hi guys,
I am very grateful to all of you for your detailed account of how to solve my problem with the RX580. Now I appeared a great chance to solve this problem and use RX580 to communicate with AI.
Can you please tell me, what is the speed increase of working with RX580 with AI compared to СPU N100? Is it 10x, 100x, or maybe 1000x? :-)

Here's my performance figures
CPU - AMD Ryzen 5 3400G with Radeon Vega Graphics (16GB RAM)
"eval_count":342,"eval_duration":66228944000 = 5.1639 tokens a second
GPU - Radeon Rx580 8GB vRAM
"eval_count":308,"eval_duration":11708779000} = 26.305 tokens a second

mnccouk, thank you so much for the convincing test! :-)
It turns out that the gain about speed is small, about 5 times.
At the same time, the TDP of:
AMD Ryzen 5 3400G = 65 Watts
Radeon Vega 16 = 75 Watts.
AMD RX580 = 185 Watts.

jeromechungmf
I chose your advice first:

Linux with Vulkan:
I install Ubuntu 24.04 or 22.04, configuring it to use the Vulkan drivers, and then installing LMStudio. The goal is to make this setup work with my RX580 GPU for optimal LLM performance. However, attempting to install Vulkan drivers directly from AMD's website (for Ubuntu 22.04) proved unsuccessful. Instead, I found success by downloading the corresponding Vulkan driver package from https://vulkan.lunarg.com/sdk/home#linux
This allowed LMStudio to detect my RX580 GPU and use it for LLM calculations.

And it ended in failure. What I did, in detail:

Install Ubuntu-24.04.1 LTS Server Minimal (without X's)
 wget -qO- https://packages.lunarg.com/lunarg-signing-key-pub.asc | sudo tee /etc/apt/trusted.gpg.d/lunarg.asc  sudo wget -qO /etc/apt/sources.list.d/lunarg-vulkan-noble.list https://packages.lunarg.com/vulkan/lunarg-vulkan-noble.list  sudo apt update  sudo apt install vulkan-sdk
Download LM_Studio-0.3.2.AppImage to /home/ai/.appimage
Run LM Studio: needs FUSE
sudo add-apt-repository universe sudo apt install libfuse2t64
Run LM Studio: needs libnss3.so
sudo apt install libnss3
Run LM Studio: needs libasound
sudo apt install libasound2t64
Run LM Studio:

FATAL:setuid_sandbox_host.cc(158)] The SUID sandbox helper binary was found, but is not configured correctly.
Rather than run without sandboxing I'm aborting now.
You need to make sure that /tmp/.mount_LM_StusEylzO/chrome-sandbox is owned by root and has mode 4755.

But this file does not exist!
Next, I began to understand that a graphical environment is required to run LM Studio and I did not like it at all, because I want to using AI in a console environment, without X's.
That's why I stopped this experiment, and now I want to try KhazAkar advice. But unfortunately he's not giving any details yet :-(

Hi Tamila-2017
Sorry，I should show my install process step by step:
1.downlaod the Ubuntu 22.04 Desktop iso file,my file name:ubuntu-22.04.4-desktop-amd64.iso .
2.make the iso file to usb boot Disk.In the linux you can use the dd command make it.ex:sudo dd bs=4M if=/path/to/ubuntu-22.04.4-desktop-amd64.iso of=/dev/sdX status=progress oflag=sync
3.I booted from a USB drive and installed Ubuntu 22.04 on my computer.You can choice minimal option，follw the gui guide step by step.
4.open the firefox paste the https://vulkan.lunarg.com/sdk/home#linux,you can browse the vulkan page，in the Linux content,click the Ubuntu Packages tab,then roll down until see the Release 1.3.290 link,click it.you can see the Ubuntu 22.04 (Jammy Jellyfish) and Ubuntu 24.04 (Noble Numbat).in my case,i click the Ubuntu 22.04 (Jammy Jellyfish) item.it's show the command like below:
wget -qO- https://packages.lunarg.com/lunarg-signing-key-pub.asc | sudo tee /etc/apt/trusted.gpg.d/lunarg.asc
sudo wget -qO /etc/apt/sources.list.d/lunarg-vulkan-1.3.290-jammy.list https://packages.lunarg.com/vulkan/1.3.290/lunarg-vulkan-1.3.290-jammy.list
sudo apt update
sudo apt install vulkan-sdk
5.open the terminal follow the command and execute.finish to install vulkan driver in my ubuntu 22.04 desktop enviroment.
6.reboot
7.download the lmstudio and follow the https://www.tecmint.com/lm-studio-run-llms-linux/ to run lmstudio.
In my experince,it's should be work.
PS:use xdrp to longin Ubuntu should cause by privilege issue of lmstudio,if you want to remote login ubuntu to run lmstudio.Using like RealVNC Viewer for remote access is a better option, as it can avoid permission problems and allow the lmstudio can detection of the RX580 GPU.
Good luck.
Jerome

jeromechungmf,
Thank you for your attention and detailed description of the installation.
And I'm sorry that I still have a little experience in Linux.
So, I'm starting to install the recommended one for me ubuntu-22.04.4-desktop-amd64

For vulkan, you don't need to downgrade to 22.04. 24.04 is also fine, even better. Ollama does not yet have vulkan support, but will gain it. For vulkan in CLI, you can either build Ollama with vulkan support from PR in this repo or build llama.cpp from ggreganov repository with vulkan support enabled. If you need GUI, then GPT4All is best one for GUI apps available, eventually LM Studio, but since I prefer open source, I'd pick GPT4ALL. In general I think we should move such discussion somewhere else and here focus on trying to get Ollama running with older AMD GPUs. So my proposition to try vulkan PR in Ollama marches the best.

So, I successfully installed Ubuntu-22.04.4-desktop-amd64 and followed steps "1-4".
However, unfortunately, action "5" is not executed, because there is no such command: execute.finish

@Tamila-2017
I happen to have a real time power meter on the server, The results for power consumption:-
CPU while processing - "Why is the sky blue?"
= Duration = 66.7 seconds with power consumption = 104 Watts
= 0.001926889 kWh
= 1.92 Wh
GPU while processing - "Why is the sky blue?"
Duration = 11.7 seconds with power consumption = 230 Watts
= 0.001661111 kWh
= 1.66 Wh
Server power while GPU\CPU idle is 40 Watts

mnccouk,
Measurements are very useful, thank you

jeromechungmf,
I don't understand  what the execute command execute.finish, which does not exist.
Therefore, I used this instruction - https://www.tecmint.com/lm-studio-run-llms-linux/
and I managed to launch the LM Studio.
Unfortunately, LM Studio also doesn't see the RX580 :-(
Although Ubuntu and nvtop sees it perfectly.

KhazAkar

If you need GUI, then ...

I don't need the GUI, I'm strongly against it because I need to work with Ollama in the console.

In general I think we should move such discussion somewhere else and here focus on trying to get Ollama running with older AMD GPUs.

I completely agree with you! But for some reason you're stubbornly not sharing the details of this success of yours -

You can all try ollama-for-amd fork. I'm using it myself with my RX5500M card and ROCm 6.2. works wonders.

even though I've asked you about it before.
Why don't you tell me about it? Is it your big secret or  it your was just a bad joke?
Unlike you jeromechungmf detailed his experience and I tried to use it, although I'm not thrilled with the graphical environment because I need a console server option that works with Ollama.

@Tamila-2017 it's not a joke. Fork is here: https://github.com/likelovewant/ollama-for-amd
But I can't assure you of anything with older card than I currently have. This one does not use vulkan but ROCm and I don't know if your card is supported by ROCm 6.x.
For Vulkan usage, if you're feeling brave, you can try building Ollama with vulkan support by cloning repository and applying this PR:
https://github.com/ollama/ollama/pull/5059
In both cases, building Ollama by hand is inevitable.

Plus, @Tamila-2017 - only Info in this fork I've found is in foreign language to me.
https://github.com/likelovewant/ollama-for-amd/issues/1
You can translate it if you want.
And it would be great idea that you join Ollama discord server :)

jeromechungmf,
I don't understand what the execute command execute.finish, which does not exist.
Therefore, I used this instruction - https://www.tecmint.com/lm-studio-run-llms-linux/ and I managed to launch the LM Studio.
Unfortunately, LM Studio also doesn't see the RX580 :-( Although Ubuntu and nvtop sees it perfectly.

Hi Tamila-2017

Attach my screenshot,it's contain execute the vulkan driver install commands,my be you can refer.
Good lucks.
Jerome

jeromechungmf,
I don't understand what the execute command execute.finish, which does not exist.
Therefore, I used this instruction - https://www.tecmint.com/lm-studio-run-llms-linux/ and I managed to launch the LM Studio.
Unfortunately, LM Studio also doesn't see the RX580 :-( Although Ubuntu and nvtop sees it perfectly.

Hi Tamila-2017
Attach the LMStudio Configuration sccreenshot,in the develper tab,click the LM Runtimes,you can check llama.cpp Vulkan should be v0.0.4.Then Click Settings tab you can see the VRAM 8 GB.
Jerome

jeromechungmf,
I don't understand what the execute command execute.finish, which does not exist.
Therefore, I used this instruction - https://www.tecmint.com/lm-studio-run-llms-linux/ and I managed to launch the LM Studio.
Unfortunately, LM Studio also doesn't see the RX580 :-( Although Ubuntu and nvtop sees it perfectly.

Hi Tamila-2017
Attach the LMStudio Configuration sccreenshot,in the developer tab,click the LM Runtimes,you can check llama.cpp Vulkan should be v0.0.4.Then Click Settings tab you can see the VRAM 8 GB.
Jerome

I tried LM Studio using your method (vulkan + lmstudio appimage) on my Linux Mint 22 with RX580.  It just works right away.     My RX580 is "refurbished" 16Gb from Aliexpress.

10.81 tok/sec  531 tokens 0.91s to first token
Hopefully Ollama will works with RX580 soon.

KhazAkar

And it would be great idea that you join Ollama discord server :)

I tried to register on Discord - https://discord.com
But it first demanded my phone number and then told me that the this number was incorrect.

I've added the docker image I'm using with the rx580 with Ollama to docker hub, Hopefully it might prove to be useful to someone.
https://hub.docker.com/r/mnccouk/ollama-gpu-rx580

Hello guys:
https://phdluffy.com/2024/08/08/ollama-rx580-egpu-amd/ this article is detail description how to install ollama in the Windows OS Enviroment.I follow it and work fine.
If your OS is Windows,may be can refer it.
PS:Ollama run in the Windwos OS,it have better performance than LMStudio.
Good Lucks.
Jerome

I've added the docker image I'm using with the rx580 with Ollama to docker hub, Hopefully it might prove to be useful to someone. https://hub.docker.com/r/mnccouk/ollama-gpu-rx580

I have tried your docker image and it runs successfully.
Eval rate = 15.25 tokes/s.  Very impressive compared to LM Studio above.
RX580 2048 with 16GB VRAM
Ryzen 5 4650G with 16GB Memory.
Thank you for your effort.

@bellmancity, Awesome! Glad it worked for you.

I've added the docker image I'm using with the rx580 with Ollama to docker hub, Hopefully it might prove to be useful to someone.
https://hub.docker.com/r/mnccouk/ollama-gpu-rx580

I can't install ROCm 5.7.0 because there are obscure errors when compiling.
mnccouk,
please give the exact download link of the distro you used here

Maybe this link will help:

Follow this documentation for rocm installation, just substitute the 5.7.0 references to 5.7.1 in the documentation. --https://rocm.docs.amd.com/en/docs-5.7.0/deploy/linux/os-native/install.html

From https://github.com/mnccouk/ollama/tree/rx580_gpu?tab=readme-ov-file#linux-with-rx580-radeon-gpu

eliot-akira,
Thanks, these links have been known to me for a long time.
But I am not aware of the exact distribution that uses mnccouk.
I'm waiting for the exact link from mnccouk to download the distro, because a lot depends on the distro.

Eventually just make use of Vulkan driver, either in ollama from PR in this repo, or llama.cpp from https://github.com/ggerganov/llama.cpp built from source....

I've tried using Vulkan before, but nothing good came out of it.
So I want to use the docker that mnccouk created and it worked for him and also at bellmancity

I've tried using Vulkan before, but nothing good came out of it.
So I want to use the docker that mnccouk created and it worked for him and also at bellmancity

Something feels off for me here. I'm not talking about gui apps like somebody here tried to give LMStudio here
Plus, I think fastest way would be to use vulkan based inference, even building ollama from source, which is well written in docs here.
ROCm 5.7.x is quite... old. 5.7.1 is shipped with ubuntu 24.04 and debian 12 I think, under slightly different name, search for hip. Current versions of Ollama work with 6.0.x at least and I don't know if they work right with 5.7.x series. Screenshot on top says otherwise, but currently RX 580, besides pletinful of VRAM, ended it's life after cryptomining boom.
Best bet for its usage for AI is to push vulkan implementation for Ollama to be merged, available as PR here:
https://github.com/ollama/ollama/pull/5059
Either way, signing out. This thread got quite derailed IMHO and 3/4 of it should be on official discord server.

KhazAkar ,
You narrate very much and interestingly.
Unfortunately, your thoughts do not lead to the final result, because they are based only on probabilistic guesses, but without their practical verification.
The only one who has a real  success here without using the DE graphical environment and Vulkan - is modest mnccouk.
I'm waiting for him to give me the exact link to download the distro he uses.
Once again: not the name of the distribution, but the EXACT LINK to download.

@Tamila-2017, I installed Ubuntu 20.04 then did a distro upgrade to Ubuntu 22.04.4. Only reason I didn't install 22.04 directly was due to usb boot issue using 22.04 installation media, but that's another story.
Direct link  (https://releases.ubuntu.com/22.04.4/) to 22.04.4 doesn't work for some reason. However, there is a link for 22.04.5, https://releases.ubuntu.com/jammy/ubuntu-22.04.5-desktop-amd64.iso.
Then install the Room libraries @eliot-akira  provided the link to (above).
Sorry can't provide 22.04.4 working direct link but can't see why 22.04.5 shouldn't work for you.
Obviously, also make sure to install docker, this can be done using the packages that come with Ubuntu 22.04.x

I've added the docker image I'm using with the rx580 with Ollama to docker hub, Hopefully it might prove to be useful to someone. https://hub.docker.com/r/mnccouk/ollama-gpu-rx580

This worked for me using an RX480

Dear mnccouk,
Here is where you can download any version of Ubuntu 22.04.04 -
https://old-releases.ubuntu.com/releases/jammy/
But there are 2 options here:

ubuntu-22.04.4-desktop-amd64.iso (Xorg)
ubuntu-22.04.4-live-server-amd64.iso (console)

Can I use the version without Xorg and Gnom?
I.e. ubuntu-22.04.4-live-server-amd64.iso ?

Ah nice, glad you found the link.
I installed the GUI version on my server as I have a monitor connected to it and wanted to use GUI tools to do stuff. If you have a monitor plugged into your server maybe this is a good option to go with in the beginning.
I'm sure  ubuntu-22.04.4-live-server-amd64.iso should also work fine but can't validate that personally.
Maybe get it working first with the desktop version, then once you know all the steps switch to the server version, of cause the choice is yours.

Has anyone been successful using multiple GPUs? I was trying to add a couple 8gb RX470 but I can't seem to get them to show up in rocminfo.

This nightmarishly unfinished manual -
https://rocm.docs.amd.com/en/docs-5.7.0/deploy/linux/os-native/install.html
this is not an instruction, but a mockery of common sense.
This is the third time I have installed Ubuntu-<IP_ADDRESS>-Server "Jammy", then followed these instructions, remembering to replace 5.7.0 with 5.7.1, but fail every time:

Only the CPU is visible, but the graphics card is not.

I don't understand why this is the case :-(
Dear mnccouk,
could you do me the courtesy of creating a linear bash script instead of this nightmarish instruction, which I could execute without having to think about unnecessary complications to get a working ROCm 5.7.1 ?

Just been through the docs, and back through my notes. This should get you close if not up and running. Haven't tested but hopefully this should get you on the right track
sudo mkdir --parents --mode=0755 /etc/apt/keyrings

wget https://repo.radeon.com/rocm/rocm.gpg.key -O - | \
    gpg --dearmor | sudo tee /etc/apt/keyrings/rocm.gpg > /dev/null

echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/rocm.gpg] https://repo.radeon.com/amdgpu/5.7.1/ubuntu jammy main" \
    | sudo tee /etc/apt/sources.list.d/amdgpu.list

sudo apt update

echo -e 'Package: *\nPin: release o=repo.radeon.com\nPin-Priority: 600' | sudo tee /etc/apt/preferences.d/rocm-pin-600

sudo apt install amdgpu-dkms

sudo reboot

for ver in 5.7.1; do
echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/rocm.gpg] https://repo.radeon.com/rocm/apt/$ver jammy main" \
    | sudo tee --append /etc/apt/sources.list.d/rocm.list
done

echo -e 'Package: *\nPin: release o=repo.radeon.com\nPin-Priority: 600' \
    | sudo tee /etc/apt/preferences.d/rocm-pin-600

sudo apt update

sudo apt install rocm-hip-sdk #if that didn't work try -> sudo apt install rocm-hip-sdk5.7.1

#Think these may already be installed through deps, but no harm in trying
sudo apt install rocm-utils #if that didn't work again try-> sudo apt install rocm-utils5.7.1

sudo tee --append /etc/ld.so.conf.d/rocm.conf <<EOF
/opt/rocm/lib
/opt/rocm/lib64
EOF

sudo ldconfig

#You'll need to add this in you user profile, this will work if you stay in the same terminal
export PATH=$PATH:/opt/rocm-5.7.1/bin:/opt/rocm-5.7.1/opencl/bin

sudo usermod -a -G render your_linux_username_here

#I still had permissions issue so brute forced access
sudo chmod 666 /dev/kfd

#to varify the libraries\driver are\is working
/opt/rocm/bin/rocminfo

Wow! Thank you so much you mnccouk,  for your quick and creative help!
I'm immediately proceeding with the 4th install  ubuntu-22.04.4-live-server-amd64 :-)

I did a fresh install of ubuntu 22.04.5 then updated and rebooted but getting stuck at the sudo apt install amdgpu-dkms part with this error
Setting up amdgpu-dkms-firmware (1:<IP_ADDRESS>701-1664922.22.04) ...
Setting up dctrl-tools (2.24-3build2) ...
Setting up dkms (2.8.7-2ubuntu2.2) ...
Setting up amdgpu-dkms (1:<IP_ADDRESS>701-1664922.22.04) ...
Loading new amdgpu-6.2.4-1664922.22.04 DKMS files...
Building for 6.8.0-45-generic
Building for architecture x86_64
Building initial module for 6.8.0-45-generic
ERROR: Cannot create report: [Errno 17] File exists: '/var/crash/amdgpu-dkms.0.crash'
Error! Bad return status for module build on kernel: 6.8.0-45-generic (x86_64)
Consult /var/lib/dkms/amdgpu/6.2.4-1664922.22.04/build/make.log for more information.
dpkg: error processing package amdgpu-dkms (--configure):
 installed amdgpu-dkms package post-installation script subprocess returned error exit status 10
Processing triggers for man-db (2.10.2-1) ...
Errors were encountered while processing:
 amdgpu-dkms
E: Sub-process /usr/bin/dpkg returned an error code (1)

Dear mnccouk,
Your wonderful script executed without errors, only I had to split it into two parts, where the
sudo reboot
However, I got the same result as before, in which I don't see a graphics card, only a Celeron CPU:
~$ /opt/rocm/bin/rocminfo
ROCk module is loaded
=====================
HSA System Attributes
=====================
Runtime Version:         1.1
System Timestamp Freq.:  1000.000000MHz
Sig. Max Wait Duration:<PHONE_NUMBER>3709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
Machine Model:           LARGE
System Endianness:       LITTLE
Mwaitx:                  DISABLED
DMAbuf Support:          YES

==========
HSA Agents
==========
*******
Agent 1
*******
  Name:                    Intel(R) Celeron(R) CPU G1620 @ 2.70GHz
  Uuid:                    CPU-XX
  Marketing Name:          Intel(R) Celeron(R) CPU G1620 @ 2.70GHz
  Vendor Name:             CPU
  Feature:                 None specified
  Profile:                 FULL_PROFILE
  Float Round Mode:        NEAR
  Max Queue Number:        0(0x0)
  Queue Min Size:          0(0x0)
  Queue Max Size:          0(0x0)
  Queue Type:              MULTI
  Node:                    0
  Device Type:             CPU
  Cache Info:
    L1:                      32768(0x8000) KB
  Chip ID:                 0(0x0)
  ASIC Revision:           0(0x0)
  Cacheline Size:          64(0x40)
  Max Clock Freq. (MHz):   2700
  BDFID:                   0
  Internal Node ID:        0
  Compute Unit:            2
  SIMDs per CU:            0
  Shader Engines:          0
  Shader Arrs. per Eng.:   0
  WatchPts on Addr. Ranges:1
  Features:                None
  Pool Info:
    Pool 1
      Segment:                 GLOBAL; FLAGS: FINE GRAINED
      Size:                    3970100(0x3c9434) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Alignment:         4KB
      Accessible by all:       TRUE
    Pool 2
      Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
      Size:                    3970100(0x3c9434) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Alignment:         4KB
      Accessible by all:       TRUE
    Pool 3
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED
      Size:                    3970100(0x3c9434) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Alignment:         4KB
      Accessible by all:       TRUE
  ISA Info:
*** Done ***

What could be the reason for this failure?

I did a fresh install of ubuntu 22.04.5 then updated and rebooted but getting stuck at the sudo apt install amdgpu-dkms part with this error
Setting up amdgpu-dkms-firmware (1:<IP_ADDRESS>701-1664922.22.04) ...
Setting up dctrl-tools (2.24-3build2) ...
Setting up dkms (2.8.7-2ubuntu2.2) ...
Setting up amdgpu-dkms (1:<IP_ADDRESS>701-1664922.22.04) ...
Loading new amdgpu-6.2.4-1664922.22.04 DKMS files...
Building for 6.8.0-45-generic
Building for architecture x86_64
Building initial module for 6.8.0-45-generic
ERROR: Cannot create report: [Errno 17] File exists: '/var/crash/amdgpu-dkms.0.crash'
Error! Bad return status for module build on kernel: 6.8.0-45-generic (x86_64)
Consult /var/lib/dkms/amdgpu/6.2.4-1664922.22.04/build/make.log for more information.
dpkg: error processing package amdgpu-dkms (--configure):
 installed amdgpu-dkms package post-installation script subprocess returned error exit status 10
Processing triggers for man-db (2.10.2-1) ...
Errors were encountered while processing:
 amdgpu-dkms
E: Sub-process /usr/bin/dpkg returned an error code (1)

from that make.log: https://termbin.com/hjob

I think i encountered a similar problem. I wound up deleting the 6.8.0-45 generic kernel and am using the older 5.19.0-32 generic kernel instead.

@Tamila-2017
Probably best to run that script step by step, one command at a time, After each command validate you have no errors, apologies for the confusion. You may have had similar issue to @kth8.but not noticed due to lack of error trapping when running all the commands at once.
@SirDubbins proposed a fix, by trying again after downgrading your kernel, or reinstall ubuntu again(sorry!).
Looking at the doc - https://rocm.docs.amd.com/en/docs-5.7.1/release/gpu_os_support.html - Ubuntu 22.04.3 has been validated with kernel 6.2. Try that version(22.04.3) of Ubuntu instead, also don't install any updates as part of the Ubuntu installation.
I went from an earlier of Ubuntu 20.04 then upgraded through to Ubuntu 22.04.4, the whole process wasn't smooth for me either. It's quite possible the kernel drivers could have compiled against an earlier kernel version.
Just for info - I'm actually on  kernel version 6.8.0-40-generic now.
Sorry, but you may have to target an install of (Ubuntu 22.04.3) or try downgrading you kernel, but that introduce other issues. I'd opt for the fresh install again if you have the luxury of being able to.

Dear mnccouk,
I've reinstalled again ubuntu-22.04.4-live-server-amd64.iso
uname -a
Linux ai-server 5.15.0-122-generic #132-Ubuntu SMP Thu Aug 29 13:45:52 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Then, carefully step by step, I executed each command of your magic script separately, watching for mistakes.
There were no mistakes.
However, I got the same result without the GPU -
/opt/rocm/bin/rocminfo
ROCk module is loaded
=====================
HSA System Attributes
=====================
Runtime Version:         1.1
System Timestamp Freq.:  1000.000000MHz
Sig. Max Wait Duration:<PHONE_NUMBER>3709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
Machine Model:           LARGE
System Endianness:       LITTLE
Mwaitx:                  DISABLED
DMAbuf Support:          YES

==========
HSA Agents
==========
*******
Agent 1
*******
  Name:                    Intel(R) Celeron(R) CPU G1620 @ 2.70GHz
  Uuid:                    CPU-XX
  Marketing Name:          Intel(R) Celeron(R) CPU G1620 @ 2.70GHz
  Vendor Name:             CPU
  Feature:                 None specified
  Profile:                 FULL_PROFILE
  Float Round Mode:        NEAR
  Max Queue Number:        0(0x0)
  Queue Min Size:          0(0x0)
  Queue Max Size:          0(0x0)
  Queue Type:              MULTI
  Node:                    0
  Device Type:             CPU
  Cache Info:
    L1:                      32768(0x8000) KB
  Chip ID:                 0(0x0)
  ASIC Revision:           0(0x0)
  Cacheline Size:          64(0x40)
  Max Clock Freq. (MHz):   2700
  BDFID:                   0
  Internal Node ID:        0
  Compute Unit:            2
  SIMDs per CU:            0
  Shader Engines:          0
  Shader Arrs. per Eng.:   0
  WatchPts on Addr. Ranges:1
  Features:                None
  Pool Info:
    Pool 1
      Segment:                 GLOBAL; FLAGS: FINE GRAINED
      Size:                    3970100(0x3c9434) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Alignment:         4KB
      Accessible by all:       TRUE
    Pool 2
      Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
      Size:                    3970100(0x3c9434) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Alignment:         4KB
      Accessible by all:       TRUE
    Pool 3
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED
      Size:                    3970100(0x3c9434) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Alignment:         4KB
      Accessible by all:       TRUE
  ISA Info:
*** Done ***

What do I do next?

# cat /var/log/dmesg | grep -e amdgpu -e drm
cat: /var/log/dmesg: No such file or directory

How about?
sudo dmesg | grep -e amdgpu -e drm

And...
lsmod | grep amdgpu

sudo dmesg | grep -e amdgpu -e drm
[    5.127309] [drm] amdgpu kernel modesetting enabled.
[    5.156647] [drm] amdgpu version: 6.2.4
[    5.185611] [drm] OS DRM version: 5.15.0
[    5.244324] amdgpu: CRAT table not found
[    5.305139] amdgpu: Virtual CRAT table created for CPU
[    5.368709] amdgpu: Topology: Add CPU node
[    5.448129] amdgpu: PeerDirect support was initialized successfully
[    5.480922] [drm] initializing kernel modesetting (POLARIS10 0x1002:0x67DF 0x1DA2:0xE366 0xE7).
[    5.513856] [drm] register mmio base: 0xF7E00000
[    5.546704] [drm] register mmio size: 262144
[    5.579527] [drm] add ip block number 0 <vi_common>
[    5.612338] [drm] add ip block number 1 <gmc_v8_0>
[    5.645021] [drm] add ip block number 2 <tonga_ih>
[    5.677367] [drm] add ip block number 3 <gfx_v8_0>
[    5.709378] [drm] add ip block number 4 <sdma_v3_0>
[    5.741213] [drm] add ip block number 5 <powerplay>
[    5.772408] [drm] add ip block number 6 <dm>
[    5.802951] [drm] add ip block number 7 <uvd_v6_0>
[    5.833281] [drm] add ip block number 8 <vce_v3_0>
[    5.925618] amdgpu 0000:01:00.0: No more image in the PCI ROM
[    5.956693] amdgpu 0000:01:00.0: amdgpu: Fetched VBIOS from ROM BAR
[    5.987774] amdgpu: ATOM BIOS: 113-1E3660U-O51
[    6.018348] [drm] UVD is enabled in VM mode
[    6.048418] [drm] UVD ENC is enabled in VM mode
[    6.078313] [drm] VCE enabled in VM mode
[    6.108367] amdgpu 0000:01:00.0: vgaarb: deactivate vga console
[    6.108372] amdgpu 0000:01:00.0: amdgpu: Trusted Memory Zone (TMZ) feature not supported
[    6.108376] amdgpu 0000:01:00.0: amdgpu: PCIE atomic ops is not supported
[    6.108593] [drm] vm size is 64 GB, 2 levels, block size is 10-bit, fragment size is 9-bit
[    6.108638] amdgpu 0000:01:00.0: amdgpu: VRAM: 8192M 0x000000F400000000 - 0x000000F5FFFFFFFF (8192M used)
[    6.108643] amdgpu 0000:01:00.0: amdgpu: GART: 256M 0x000000FF00000000 - 0x000000FF0FFFFFFF
[    6.108734] [drm] Detected VRAM RAM=8192M, BAR=256M
[    6.108738] [drm] RAM width 256bits GDDR5
[    6.108788] [drm] amdgpu: 8192M of VRAM memory ready
[    6.108792] [drm] amdgpu: 1938M of GTT memory ready.
[    6.108862] [drm] GART: num cpu pages 65536, num gpu pages 65536
[    6.109304] [drm] PCIE GART of 256M enabled (table at 0x000000F400800000).
[    6.109428] [drm] Chained IB support enabled!
[    6.110196] amdgpu: [powerplay] hwmgr_sw_init smu backed is polaris10_smu
[    6.110341] [drm] Found UVD firmware Version: 1.130 Family ID: 16
[    6.110955] [drm] Found VCE firmware Version: 53.26 Binary ID: 3
[    6.179076] [drm] Display Core v3.2.241 initialized on DCE 11.2
[    6.274702] [drm] UVD and UVD ENC initialized successfully.
[    6.384641] [drm] VCE initialized successfully.
[    6.385053] kfd kfd: amdgpu: skipped device 1002:67df, PCI rejects atomics 730<0
[    6.385072] amdgpu 0000:01:00.0: amdgpu: SE 4, SH per SE 1, CU per SH 9, active_cu_number 36
[    6.388350] amdgpu: legacy kernel without apple_gmux_detect()
[    6.388356] amdgpu 0000:01:00.0: amdgpu: Using BACO for runtime pm
[    6.389279] [drm] Initialized amdgpu 3.54.0 20150101 for 0000:01:00.0 on minor 0
[    6.403933] fbcon: amdgpudrmfb (fb0) is primary device
[    6.479039] amdgpu 0000:01:00.0: [drm] fb0: amdgpudrmfb frame buffer device
[    7.845748] systemd[1]: Starting Load Kernel Module drm...
[    9.367404] snd_hda_intel 0000:01:00.1: bound 0000:01:00.0 (ops amdgpu_dm_audio_component_bind_ops [amdgpu])

lsmod | grep amdgpu
amdgpu              13590528  0
amddrm_ttm_helper      16384  1 amdgpu
amdttm                 94208  2 amdgpu,amddrm_ttm_helper
amdxcp                 16384  1 amdgpu
iommu_v2               24576  1 amdgpu
amddrm_buddy           20480  1 amdgpu
amd_sched              49152  1 amdgpu
amdkcl                 32768  3 amd_sched,amdttm,amdgpu
i2c_algo_bit           16384  1 amdgpu
drm_kms_helper        311296  3 amdgpu
drm                   622592  9 drm_kms_helper,amd_sched,amdttm,amdgpu,amddrm_buddy,amdkcl,amddrm_ttm_helper,amdxcp

I'm running out of ideas here, but I don't like the look of:
[    6.385053] kfd kfd: amdgpu: skipped device 1002:67df, PCI rejects atomics 730<0

It's incredible! You are doing well, but I am not, even though I follow your instructions exactly
So I offer you remote access via ssh, and you personally can install this cranky ROCm 5.7.1

I can't be certain on this, but what I'm reading  is the ROCm requires PCIe 3.0 AtomicOp feature,  looking at the spec of your CPU - https://www.intel.com/content/www/us/en/products/sku/71073/intel-celeron-processor-g1620-2m-cache-2-70-ghz/specifications.html
It looks like it supports PCIe revision 2.0 which unfortunately is not compatible with the ROCm system. Also motherboard seems to be PCIe 2.x too.
I'm sorry to say I don't think ROCm will work for you with that CPU\Motherboard GPU combination, I maybe wrong but that's I think. Open to any other thoughts.

Dear mnccouk  💓
You have no need to apologize :-)  On the contrary, I am very grateful to you for your courtesy and your great work you have done to analyze my situation and create a magic script to build ROCm.
I think you are right about my outdated hardware.
I will consider buying a more modern CPU and Motherboard, and I would be grateful if you could tell me the system requirements for it.

@Tamila-2017
I can only forward the motherboard and CPU I'm using as I know that would work:
Motherboard - B450M PRO-VDH MAX - https://www.msi.com/Motherboard/B450M-PRO-VDH-MAX/support
CPU - AMD Ryzen 5 3400G with Radeon Vega Graphics
And of course the RX580
There is a discussion here - https://github.com/ROCm/ROCm/issues/237  about a similar issue to what you have, except in this instance it's just one of the PCIe ports that is failing when attempting to use two graphics cards, even though it has PCIe 3.0 specification.
Maybe others would share what hardware(motherboard\cpu combination) they have successfully used with ROCm to give you some alternatives?

Ok. Guys, how do you measure speed in tokens/second? I want to measure the speed of CPU N100.

/set verbose

So I need to run some model to measure the speed?
I ran the Llama 3.1 model and executed your command, but it did not report the speed:
>>> /set verbose
Set 'verbose' mode.

Then you make a prompt:-

Surprisingly, I got a different answer:

~$ ollama run llama3.1
/set verbose
Set 'verbose' mode.
why is the sky blue
The sky appears blue to us because of a phenomenon called scattering,
which occurs when sunlight interacts with the tiny molecules of gases in
the atmosphere. Here's a simplified explanation:

Sunlight enters Earth's atmosphere: When the sun shines, it emits a
wide range of electromagnetic radiation, including visible light,
ultraviolet (UV) light, and infrared (IR) radiation.
Light scatters off gas molecules: As sunlight travels through the
atmosphere, it encounters tiny molecules of gases like nitrogen (N2) and
oxygen (O2). These molecules are much smaller than the wavelength of
light, so they scatter the shorter (blue) wavelengths more efficiently
than the longer (red) wavelengths.
Blue light is scattered in all directions: The scattering of blue
light by gas molecules causes it to be distributed throughout the
atmosphere in all directions. This means that we see this scattered blue
light from almost everywhere around us.
Red light continues straight: On the other hand, the longer
wavelengths (like red and infrared) continue traveling in a more direct
path, with less scattering occurring. As a result, these colors are not as
prominently visible to our eyes.

Why doesn't the sky always appear blue?

Time of day: During sunrise and sunset, the sun's light travels
through more of the Earth's atmosphere, which scatters shorter wavelengths
like blue even further. This results in the appearance of reds and
oranges.
Clouds and dust: When clouds or atmospheric particles are
present, they can absorb or scatter light in different ways, affecting the
apparent color of the sky.
Atmospheric conditions: Weather events, pollution, or other factors
can alter the way light interacts with the atmosphere, making the sky
appear more hazy, gray, or even greenish.

Additional fun facts:

The sky appears blue to us because our eyes are more sensitive to the
shorter (blue) wavelengths of light.
If we were to live on a planet with a different atmospheric composition
or no atmosphere at all, the sky might appear differently colored or even
not blue at all!
Some people claim that the sky can appear violet under certain
conditions. However, this is relatively rare and usually requires specific
optical phenomena.

I hope you enjoyed learning about why the sky appears blue!

And the performance of Llama 3.1 on CPU N100 turned out like this:

The performance of Llama 3.1 on CPU [Intel-i3-4330-3 50GHz:

Dear mnccouk,
You have achieved a performance of ~26 tokens/sec.
Let me clarify: this is the total performance AMD Ryzen 5 3400G + Radeon Vega Graphics  + RX580 ?
Or is the performance of only one RX580 ?

docker exec -it ollama_gpu ollama run llama3.1

/set verbose
Set 'verbose' mode.
Fix grammar and polish message:
...
... @mnccouk Thank you! Works great for models which fit into GPU memory.
...
... For those struggling with installation on Ubuntu:
... just skip "2. Add the AMDGPU Repository and Install the Kernel-mode Driver" from this instruction: https://rocm.docs.amd.com/en/docs-5.7.0/deploy/linux/os-native/install.html.
... You do not need it: recent Ubuntu releases already contain amdgpu driver.
...
... I got it worked without any problems within less than half an hour.
... 22.04.5 LTS
... 6.8.0-40-generic
... RX590
Here is the revised message with grammar and polish:

"@mnccouk Thank you! Works great for models that fit into GPU memory.
For those struggling to install on Ubuntu, here's a tip:
You can skip step 2 ("Add the AMDGPU Repository and Install the Kernel-mode Driver") from this instruction set: https://rocm.docs.amd.com/en/docs-5.7.0/deploy/linux/os-native/install.html.
You don't need it, as recent Ubuntu releases already contain the amdgpu driver.
I was able to get it working without any issues within less than half an hour on:
22.04.5 LTS
6.8.0-40-generic
RX590"
Note: I made some minor changes for clarity and readability, such as changing "those struggling with installation" to "those struggling to install", and rephrasing the sentence about skipping step 2
to make it more concise. Let me know if you'd like any further adjustments!
total duration:       31.579130243s
load duration:        28.163993ms
prompt eval count:    137 token(s)
prompt eval duration: 24.679448s
prompt eval rate:     5.55 tokens/s
eval count:           202 token(s)
eval duration:        6.829664s
eval rate:            29.58 tokens/s

@kth8  @Tamila-2017 ^^^

Dear mnccouk,
You have achieved a performance of ~26 tokens/sec. Let me clarify: this is the total performance AMD Ryzen 5 3400G + Radeon Vega Graphics + RX580 ? Or is the performance of only one RX580 ?

This is just with 1 RX580 + CPU,  the CPU is still utilised to some degree but the GPU is performing all the heavy lifting.
As far as I'm aware the embedded Radeon Vega Graphics(embedded in the CPU) in not utilised.

@kth8
looks like you have similar problem to @Tamila-2017
from your log
[   24.636266] kfd kfd: amdgpu: skipped device 1002:67df, PCI rejects atomics 730<0

@kth8 looks like you have similar problem to @Tamila-2017 from your log
[   24.636266] kfd kfd: amdgpu: skipped device 1002:67df, PCI rejects atomics 730<0

Do you know what that means? Is there something wrong with my motherboard or graphics card? Do I need to change a BIOS setting or set a kernel parameter or use a different kernel or something else?

I believe it's to do with PCI specification - chat from earlier - https://github.com/ollama/ollama/issues/2453#issuecomment-2369372398

I believe it's to do with PCI specification - see chat from earlier - #2453 (comment)

If this is a hardware issue then I guess there is no way to solve this?

Guys, let me express my humble opinion, perhaps erroneous.
Yes, outdated hardware can cause and failure ROCm to  at it work.
But it should not affect ROCm assembly, i.e. common compilation, imho.

About Atomics and Motherboards: https://github.com/ROCm/ROCm/issues/1146#issuecomment-758624560
Very sad information.... It turns out that ROCm is a capricious and unreliable project :-(

Hello everyone,
I have read this topic carefully, and I want to buy a PRIME X370-PRO motherboard. Do you think it will be able to work with two AMD RX580 graphics cards?

It should work the big question is what kind of performance you will get. That will depend heavily on the CPU and motherboard. For optimal performance you want two x16 slots but the board you have is either a single x16 or dual x8. That might be fine for your needs but it is something you should keep in mind.
If you want something that can handle two cards you want a server board with a server CPU which can be pretty expensive. The other option would be to just build two systems. Either way this issue is not really the place for this so I would recommend that you open up a new discussion.

@Darin755, I'm sorry, but is AI performance dependent on the CPU or motherboard?
After all, graphics cards are used here, which provide the main performance of AI.
Take a look at this, please
Regarding the question of the other thread. The use of RX580 is closely related to the type of motherboard, and it was already discussed above. So I also have the right to discuss here the choice of motherboard for the RX580, Isn't that right?
However, I took your advice into account and asked my question here as well.

mnccouk,
I used Ubuntu 24.04 and your handy script: https://github.com/ollama/ollama/issues/2453#issuecomment-2366982275
However, when executing this script, several errors occurred and the script stopped:
<skip>
update-alternatives: using /usr/lib/shim/shimx64.efi.signed.latest to provide /usr/lib/shim/shimx64.efi.signed (shimx64.e
fi.signed) in auto mode
Generating a new Secure Boot signing key:
Can't load /var/lib/shim-signed/mok/.rnd into RNG
40474B4D84740000:error:12000079:random number generator:RAND_load_file:Cannot open file:../crypto/rand/randfile.c:106:Fil
ename=/var/lib/shim-signed/mok/.rnd
<skip>
-----
EFI variables are not supported on this system
/sys/firmware/efi/efivars not found, aborting.
Setting up libgprofng0:amd64 (2.42-4ubuntu2) ...
<skip>
Setting up dkms (3.0.11-1ubuntu13) ...
Setting up amdgpu-dkms (1:<IP_ADDRESS>701-1664922.22.04) ...
Loading new amdgpu-6.2.4-1664922.22.04 DKMS files...
Building for 6.8.0-45-generic
Building for architecture x86_64
Building initial module for 6.8.0-45-generic
Error! Bad return status for module build on kernel: 6.8.0-45-generic (x86_64)
Consult /var/lib/dkms/amdgpu/6.2.4-1664922.22.04/build/make.log for more information.
dpkg: error processing package amdgpu-dkms (--configure):
 installed amdgpu-dkms package post-installation script subprocess returned error exit status 10
Setting up g++-x86-64-linux-gnu (4:13.2.0-7ubuntu1) ...
Setting up g++-13 (13.2.0-23ubuntu4) ...
Setting up g++ (4:13.2.0-7ubuntu1) ...
update-alternatives: using /usr/bin/g++ to provide /usr/bin/c++ (c++) in auto mode
Setting up build-essential (12.10ubuntu1) ...
Processing triggers for libc-bin (2.39-0ubuntu8.3) ...
Processing triggers for man-db (2.12.0-4build2) ...
Processing triggers for install-info (7.1-3build2) ...
Errors were encountered while processing:
 amdgpu-dkms
E: Sub-process /usr/bin/dpkg returned an error code (1)

This topic was created on February 12. It's been 310 days already!
It contains a lot of questions and various tips on using AMD RX580. For example, there was created a handy docker for using Ollama.
However, I studied this topic carefully, and realized that the main problem here is a failed installation of ROCm 5.7.1.
After that I tried using the most promising tips for installing ROCm 5.7.1, but unfortunately none of them work.
Well, can someone please courage the trouble to summarize and create a how to for a RELIABLE installation of Rock 5.7.1?

Regarding your error:
Check inside your build log -
/var/lib/dkms/amdgpu/6.2.4-1664922.22.04/build/make.log
See if that gives you some extra information on why the build failed.
I noticed that you are using 24.04 of ubuntu, ROCm 5.7.1 drivers are
relatively old so there's a chance that something might not be compatible
between the two.
On Fri, 11 Oct 2024 at 15:34, T-Shilov @.***> wrote:

This topic was created on February 12. It's been 310 days already!
It contains a lot of questions and various tips on using AMD RX580. For
example, there was created a handy docker for using Ollama.
However, I studied this topic carefully, and realized that the main
problem here is a failed installation of ROCm 5.7.1.
After that I tried using the most promising tips for installing ROCm
5.7.1, but unfortunately none of them work.
Well, can someone please courage the trouble to summarize and create a how
to for a RELIABLE installation of Rock 5.7.1? :-)
—
Reply to this email directly, view it on GitHub
https://github.com/ollama/ollama/issues/2453#issuecomment-2407547842,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABOLJO4NJJQRC6BVXJDUMQDZ27OX7AVCNFSM6AAAAABDD3P55SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMBXGU2DOOBUGI
.
You are receiving this because you were mentioned.Message ID:
@.***>

mnccouk, thank you the quick response.
But since my attempts were unsuccessful, during this time I have already  to variant  to the option using Ubuntu 20.04.
A little later I will tell you what happened with it.

So, since in Ubuntu 22.04 I was not able to overcome this vicious error -
Errors were encountered while processing:
 amdgpu-dkms
E: Sub-process /usr/bin/dpkg returned an error code (1)

I had to switch to Ubuntu 20.04.
I modified your script a bit using this documentation
It turned out to be 2 parts:
# Install ROCm 5.7.1 to Ubuntu 20.04.xx
# https://github.com/ollama/ollama/issues/2453
# https://github.com/ollama/ollama/issues/2453#issuecomment-2366982275
# https://rocm.docs.amd.com/en/docs-5.7.0/deploy/linux/os-native/install.html
# Here change 5.7.0 на 5.7.1
#---------------------------

# Part 1

sudo apt install  curl dialog mc htop openssh-server
echo

# 1. Download and convert the package signing key
# -----------------------------------------------

sudo mkdir --parents --mode=0755 /etc/apt/keyrings
echo

wget https://repo.radeon.com/rocm/rocm.gpg.key -O - | \
    gpg --dearmor | sudo tee /etc/apt/keyrings/rocm.gpg > /dev/null
echo

# 2. Add the AMDGPU Repository and Install the Kernel-mode Driver
# ---------------------------------------------------------------

# version
ver=5.7.1

# amdgpu repository for focal
echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/rocm.gpg] https://repo.radeon.com/amdgpu/$ver/ubuntu focal main" \
    | sudo tee /etc/apt/sources.list.d/amdgpu.list
sudo apt update
echo

sudo apt install amdgpu-dkms
echo

sleep 24h	# Press Ctrl-C

sudo reboot

and
# Install ROCm 5.7.1 to Ubuntu 20.04.xx
# https://github.com/ollama/ollama/issues/2453
# https://github.com/ollama/ollama/issues/2453#issuecomment-2366982275
# https://rocm.docs.amd.com/en/docs-5.7.0/deploy/linux/os-native/install.html
# Here 5.7.0 replaced на 5.7.1
#-----------------------------

# Part 2

# 3. Add the ROCm Repository
# --------------------------

# ROCm repositories for focal
for ver in 5.3.3 5.4.6 5.5.3 5.6.1 5.7 5.7.1; do
echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/rocm.gpg] https://repo.radeon.com/rocm/apt/$ver focal main" \
    | sudo tee --append /etc/apt/sources.list.d/rocm.list
done
echo -e 'Package: *\nPin: release o=repo.radeon.com\nPin-Priority: 600' \
    | sudo tee /etc/apt/preferences.d/rocm-pin-600
echo

sudo apt update
echo

# 4. Install packages
# -------------------

sudo apt install rocm-hip-sdk   # if that didn't work try -> sudo apt install rocm-hip-sdk5.7.1
echo

# Post-install Actions
# --------------------
# Think these may already be installed through deps, but no harm in trying
# sudo apt install rocm-utils #if that didn't work again try-> sudo apt install rocm-utils5.7.1

sudo tee --append /etc/ld.so.conf.d/rocm.conf <<EOF
/opt/rocm/lib
/opt/rocm/lib64
EOF
echo

sudo ldconfig
echo

# You'll need to add this in you user profile, this will work if you stay in the same terminal
export PATH=$PATH:/opt/rocm-5.7.1/bin:/opt/rocm-5.7.1/opencl/bin
echo

# Verifying Kernel-mode Driver Installation
dkms status
echo

# Warning!
# Here are my login = ai
sudo usermod -a -G render  ai  # <--- My login
echo

# I still had permissions issue so brute forced access
sudo chmod 666 /dev/kfd
echo

# To verify the libraries/driver are/is working
/opt/rocm/bin/rocminfo
echo

echo "Completion of ROCm 5.7.1 building"
echo

The obtained results are in the attachment.
Install-ROCm-571-Ubuntu-20.04-Part-1.pdf
Install-ROCm-571-Ubuntu-20.04-Part-2.pdf
I marked the detected errors in red color.
mnccouk,  please take a look at them. My concern is why the RX580 is not detected.
Although when I run Ollama it reports:
Start-Ollama.pdf
>>> AMD GPU ready.

I've built  a docker image that you can try, see - https://github.com/ollama/ollama/issues/2453#issuecomment-2362217923
This was built to detect the older rx580 card, with the prerequisite of making sure the ROCm driver module is already installed on the host and working - This is the step you've been working on.
Also after a fresh reboot, execute the following command:-
sudo dmesg | grep -e amdgpu -e drm
Look through the output for any errors, if the amdgpu module looks to be loaded ok, try Ollama from the docker image.

Thanks for the advice, only I didn't understand the necessary sequence of steps I should follow.
Do I need to first uninstall my futile attempts to install ROCm 5.7.1 on Ubuntu 20.04, and then install your docker on a clean Ubuntu?
Or do these actions have to be performed in some other sequence?
Could you please explain in more detail the right sequence of my actions, step by step??

The sequence is:-

Install the ROCm drivers on your machine, which is what you have been
working on. It's still not verified if what you have done up to now is
working.

Seeing the output of the dmesg command will help verify this.
If the drivers are installed and working then move to the next step.

Then fire up the docker container(make sure docker is installed first).
Follow the link already provided for some instructions on how to start the
container.

On Sat, 12 Oct 2024 at 20:57, T-Shilov @.***> wrote:

Thanks for the advice, only I didn't understand the necessary sequence of
steps I should follow.
Do I need to first uninstall my futile attempts to install ROCm 5.7.1 on
Ubuntu 20.04, and then install your docker on a clean Ubuntu?
Or do these actions have to be performed in some other sequence?
Could you please explain in more detail the right sequence of my actions,
step by step??
—
Reply to this email directly, view it on GitHub
https://github.com/ollama/ollama/issues/2453#issuecomment-2408684165,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABOLJO4MZFZXSKHV7HHAK3LZ3F5JRAVCNFSM6AAAAABDD3P55SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMBYGY4DIMJWGU
.
You are receiving this because you were mentioned.Message ID:
@.***>

Thanks for the tip. Please, here is the output of your command:
$ sudo dmesg | grep -e amdgpu -e drm
[sudo] password for ai:
[    1.444548] [drm] amdgpu kernel modesetting enabled.
[    1.444551] [drm] amdgpu version: 6.2.4
[    1.444553] [drm] OS DRM version: 5.15.0
[    1.444656] amdgpu: CRAT table not found
[    1.444660] amdgpu: Virtual CRAT table created for CPU
[    1.444669] amdgpu: Topology: Add CPU node
[    1.477640] amdgpu: PeerDirect support was initialized successfully
[    1.477780] [drm] initializing kernel modesetting (POLARIS10 0x1002:0x67DF 0x1DA2:0xE366 0xE7).
[    1.477794] [drm] register mmio base: 0xFE600000
[    1.477795] [drm] register mmio size: 262144
[    1.477827] [drm] add ip block number 0 <vi_common>
[    1.477828] [drm] add ip block number 1 <gmc_v8_0>
[    1.477829] [drm] add ip block number 2 <tonga_ih>
[    1.477830] [drm] add ip block number 3 <gfx_v8_0>
[    1.477831] [drm] add ip block number 4 <sdma_v3_0>
[    1.477831] [drm] add ip block number 5 <powerplay>
[    1.477832] [drm] add ip block number 6 <dm>
[    1.477833] [drm] add ip block number 7 <uvd_v6_0>
[    1.477834] [drm] add ip block number 8 <vce_v3_0>
[    1.478008] amdgpu 0000:01:00.0: No more image in the PCI ROM
[    1.478024] amdgpu 0000:01:00.0: amdgpu: Fetched VBIOS from ROM BAR
[    1.478026] amdgpu: ATOM BIOS: 113-1E3660U-O51
[    1.478038] [drm] UVD is enabled in VM mode
[    1.478039] [drm] UVD ENC is enabled in VM mode
[    1.478040] [drm] VCE enabled in VM mode
[    1.478137] amdgpu 0000:01:00.0: vgaarb: deactivate vga console
[    1.478139] amdgpu 0000:01:00.0: amdgpu: Trusted Memory Zone (TMZ) feature not supported
[    1.478141] amdgpu 0000:01:00.0: amdgpu: PCIE atomic ops is not supported
[    1.478165] [drm] vm size is 64 GB, 2 levels, block size is 10-bit, fragment size is 9-bit
[    1.478205] amdgpu 0000:01:00.0: amdgpu: VRAM: 8192M 0x000000F400000000 - 0x000000F5FFFFFFFF (8192M used)
[    1.478209] amdgpu 0000:01:00.0: amdgpu: GART: 256M 0x000000FF00000000 - 0x000000FF0FFFFFFF
[    1.478216] [drm] Detected VRAM RAM=8192M, BAR=256M
[    1.478217] [drm] RAM width 256bits GDDR5
[    1.478253] [drm] amdgpu: 8192M of VRAM memory ready
[    1.478254] [drm] amdgpu: 7979M of GTT memory ready.
[    1.478268] [drm] GART: num cpu pages 65536, num gpu pages 65536
[    1.478650] [drm] PCIE GART of 256M enabled (table at 0x000000F400800000).
[    1.478739] [drm] Chained IB support enabled!
[    1.479263] amdgpu: [powerplay] hwmgr_sw_init smu backed is polaris10_smu
[    1.479346] [drm] Found UVD firmware Version: 1.130 Family ID: 16
[    1.479889] [drm] Found VCE firmware Version: 53.26 Binary ID: 3
[    1.542400] [drm] Display Core v3.2.241 initialized on DCE 11.2
[    1.610063] [drm] UVD and UVD ENC initialized successfully.
[    1.720002] [drm] VCE initialized successfully.
[    1.720582] kfd kfd: amdgpu: skipped device 1002:67df, PCI rejects atomics 730<0
[    1.720594] amdgpu 0000:01:00.0: amdgpu: SE 4, SH per SE 1, CU per SH 9, active_cu_number 36
[    1.723664] amdgpu: legacy kernel without apple_gmux_detect()
[    1.723667] amdgpu 0000:01:00.0: amdgpu: Using BACO for runtime pm
[    1.723958] [drm] Initialized amdgpu 3.54.0 20150101 for 0000:01:00.0 on minor 0
[    1.735126] fbcon: amdgpudrmfb (fb0) is primary device
[    1.823364] amdgpu 0000:01:00.0: [drm] fb0: amdgpudrmfb frame buffer device
[    2.432046] systemd[1]: Condition check resulted in Load Kernel Module drm being skipped.
[    3.023163] snd_hda_intel 0000:01:00.1: bound 0000:01:00.0 (ops amdgpu_dm_audio_component_bind_ops [amdgpu])
[ 9736.893119] amdgpu 0000:01:00.0: amdgpu: PCI CONFIG reset
[ 9737.231304] [drm] PCIE GART of 256M enabled (table at 0x000000F400800000).
[ 9737.923097] [drm] Fence fallback timer expired on ring sdma0
[ 9738.435083] [drm] Fence fallback timer expired on ring sdma0
[ 9738.947040] [drm] Fence fallback timer expired on ring sdma0
[ 9739.459030] [drm] Fence fallback timer expired on ring sdma0
[ 9739.970994] [drm] Fence fallback timer expired on ring sdma0
[ 9740.482986] [drm] Fence fallback timer expired on ring sdma0
[ 9740.530084] [drm] UVD and UVD ENC initialized successfully.
[ 9740.640068] [drm] VCE initialized successfully.

Is this normal or not? I'm very worried about it.

 Generating new private key.
Your new public key is:

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICronOwazaSIQqdU/Pblye5mYUCT5OVXdgZe33dG8c/1

2024/10/12 21:09:09 routes.go:1153: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://<IP_ADDRESS>:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/root/.ollama/models OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://<IP_ADDRESS> https://<IP_ADDRESS> http://<IP_ADDRESS>:* https://<IP_ADDRESS>:* http://<IP_ADDRESS> https://<IP_ADDRESS> http://<IP_ADDRESS>:* https://<IP_ADDRESS>:* app://* file://* tauri://*] OLLAMA_SCHED_SPREAD:false OLLAMA_TMPDIR: ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]"
time=2024-10-12T21:09:09.862Z level=INFO source=images.go:753 msg="total blobs: 0"
time=2024-10-12T21:09:09.862Z level=INFO source=images.go:760 msg="total unused blobs removed: 0"
time=2024-10-12T21:09:09.862Z level=INFO source=routes.go:1200 msg="Listening on [::]:11434 (version 0.0.0)"
time=2024-10-12T21:09:09.863Z level=INFO source=common.go:49 msg="Dynamic LLM libraries" runners="[cpu_avx cpu_avx2 rocm_v0 cpu]"
time=2024-10-12T21:09:09.863Z level=INFO source=gpu.go:199 msg="looking for compatible GPUs"
time=2024-10-12T21:09:09.864Z level=INFO source=amd_linux.go:361 msg="no compatible amdgpu devices detected"
time=2024-10-12T21:09:09.865Z level=INFO source=gpu.go:346 msg="no compatible GPUs were discovered"
time=2024-10-12T21:09:09.865Z level=INFO source=types.go:107 msg="inference compute" id=0 library=cpu variant=avx compute="" driver=0.0 name="" total="15.6 GiB" available="14.6 GiB"

ai@ai-server:~$ docker exec -it ollama_gpu ollama run llama3.1
pulling manifest
pulling 8eeb52dfb3bb... 100% ▕████████████████████████████████████████████████████████████████████████▏ 4.7 GB
pulling 948af2743fc7... 100% ▕████████████████████████████████████████████████████████████████████████▏ 1.5 KB
pulling 0ba8f0e314b4... 100% ▕████████████████████████████████████████████████████████████████████████▏  12 KB
pulling 56bb8bd477a5... 100% ▕████████████████████████████████████████████████████████████████████████▏   96 B
pulling 1a4c3c319823... 100% ▕████████████████████████████████████████████████████████████████████████▏  485 B
verifying sha256 digest
writing manifest
success
>>> /set verbose
Set 'verbose' mode.
>>>
>>> Why is the sky blue?

total duration:       1m18.881610351s
load duration:        31.987745ms
prompt eval count:    16 token(s)
prompt eval duration: 2.473824s
prompt eval rate:     6.47 tokens/s
eval count:           306 token(s)
eval duration:        1m16.331472s
eval rate:            4.01 tokens/s
>>> Send a message (/? for help)

The docker container is just using the CPU there. Referring back to the
output of the dmesg command it looks like there is an issue with the ROCm
drivers using the PCI bus
This message from your log -  kfd kfd: amdgpu: skipped device 1002:67df,
PCI rejects atomics 730<0
This is probably why your device is not being detected.
There is some discussion regarding this already in this thread, but I'm
afraid I don't know of a software solution around this issue. It seems as
if the ROCm driver is very particular about the PCI spec required.
 Generating new private key.
Your new public key is:
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICronOwazaSIQqdU/Pblye5mYUCT5OVXdgZe33dG8c/1
2024/10/12 21:09:09 routes.go:1153: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://<IP_ADDRESS>:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/root/.ollama/models OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://<IP_ADDRESS> https://<IP_ADDRESS> http://<IP_ADDRESS>:* https://<IP_ADDRESS>:* http://<IP_ADDRESS> https://<IP_ADDRESS> http://<IP_ADDRESS>:* https://<IP_ADDRESS>:* app://* file://* tauri://*] OLLAMA_SCHED_SPREAD:false OLLAMA_TMPDIR: ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]"
time=2024-10-12T21:09:09.862Z level=INFO source=images.go:753 msg="total blobs: 0"
time=2024-10-12T21:09:09.862Z level=INFO source=images.go:760 msg="total unused blobs removed: 0"
time=2024-10-12T21:09:09.862Z level=INFO source=routes.go:1200 msg="Listening on [::]:11434 (version 0.0.0)"
time=2024-10-12T21:09:09.863Z level=INFO source=common.go:49 msg="Dynamic LLM libraries" runners="[cpu_avx cpu_avx2 rocm_v0 cpu]"
time=2024-10-12T21:09:09.863Z level=INFO source=gpu.go:199 msg="looking for compatible GPUs"
time=2024-10-12T21:09:09.864Z level=INFO source=amd_linux.go:361 msg="no compatible amdgpu devices detected"
time=2024-10-12T21:09:09.865Z level=INFO source=gpu.go:346 msg="no compatible GPUs were discovered"
time=2024-10-12T21:09:09.865Z level=INFO source=types.go:107 msg="inference compute" id=0 library=cpu variant=avx compute="" driver=0.0 name="" total="15.6 GiB" available="14.6 GiB"
@.***:~$ docker exec -it ollama_gpu ollama run llama3.1
pulling manifest
pulling 8eeb52dfb3bb... 100% ▕████████████████████████████████████████████████████████████████████████▏ 4.7 GB
pulling 948af2743fc7... 100% ▕████████████████████████████████████████████████████████████████████████▏ 1.5 KB
pulling 0ba8f0e314b4... 100% ▕████████████████████████████████████████████████████████████████████████▏  12 KB
pulling 56bb8bd477a5... 100% ▕████████████████████████████████████████████████████████████████████████▏   96 B
pulling 1a4c3c319823... 100% ▕████████████████████████████████████████████████████████████████████████▏  485 B
verifying sha256 digest
writing manifest
success

/set verbose
Set 'verbose' mode.
Why is the sky blue?

total duration:       1m18.881610351s
load duration:        31.987745ms
prompt eval count:    16 token(s)
prompt eval duration: 2.473824s
prompt eval rate:     6.47 tokens/s
eval count:           306 token(s)
eval duration:        1m16.331472s
eval rate:            4.01 tokens/s

Send a message (/? for help)

—
Reply to this email directly, view it on GitHub
https://github.com/ollama/ollama/issues/2453#issuecomment-2408706026,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABOLJO45K7AUID23ELFKNETZ3GIYNAVCNFSM6AAAAABDD3P55SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMBYG4YDMMBSGY
.
You are receiving this because you were mentioned.Message ID:
@.***>

Yes, I've already realized that ROCm is a very demanding and capricious thing.
I don't understand why she's making claims about my S1200KP server motherboard, as I think it has bus PCI-3 and my CPU Xeon E3 1245 v2:

Moreover, CPU-Z confirms that in this motherboard PCI is worked in  mode  PCIe x 16 3.0 @ x16 1.1 -

In the output of the command you should see the GPU detailed in the
response if it has been correctly identified by the driver.
See
https://rocm.docs.amd.com/projects/rocminfo/en/latest/how-to/use-rocminfo.html
For an example, it's the GPU section that you're looking for not CPU.
On Mon, 21 Oct 2024, 22:57 T-Shilov, @.***> wrote:

mnccouk, you wrote this here
https://hub.docker.com/r/mnccouk/ollama-gpu-rx580:
To verify, use the following command. You should see details about your
graphics card.
rocminfo
Unfortunately, "see detail" it is very vague and unclear.
Please show the full output of the rocminfo command for RX580.
It is very important.
—
Reply to this email directly, view it on GitHub
https://github.com/ollama/ollama/issues/2453#issuecomment-2427817124,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABOLJO6PFZSNWENOUT3L2BTZ4V2GHAVCNFSM6AAAAABDD3P55SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMRXHAYTOMJSGQ
.
You are receiving this because you were mentioned.Message ID:
@.***>

Thank you for the answer. I got this result, please tell me, is it correct or not?
...................
Agent 2
*******
  Name:                    gfx803
  Uuid:                    GPU-XX
  Marketing Name:          Radeon RX 580 Series
  Vendor Name:             AMD
  Feature:                 KERNEL_DISPATCH
  Profile:                 BASE_PROFILE
  Float Round Mode:        NEAR
  Max Queue Number:        128(0x80)
  Queue Min Size:          64(0x40)
  Queue Max Size:          131072(0x20000)
  Queue Type:              MULTI
  Node:                    1
  Device Type:             GPU
  Cache Info:
    L1:                      16(0x10) KB
  Chip ID:                 26591(0x67df)
  ASIC Revision:           1(0x1)
  Cacheline Size:          64(0x40)
  Max Clock Freq. (MHz):   1340
  BDFID:                   256
  Internal Node ID:        1
  Compute Unit:            36
  SIMDs per CU:            4
  Shader Engines:          4
  Shader Arrs. per Eng.:   1
  WatchPts on Addr. Ranges:4
  Features:                KERNEL_DISPATCH
  Fast F16 Operation:      TRUE
  Wavefront Size:          64(0x40)
  Workgroup Max Size:      1024(0x400)
  Workgroup Max Size per Dimension:
    x                        1024(0x400)
    y                        1024(0x400)
    z                        1024(0x400)
  Max Waves Per CU:        40(0x28)
  Max Work-item Per CU:    2560(0xa00)
  Grid Max Size:<PHONE_NUMBER>(0xffffffff)
  Grid Max Size per Dimension:
    x<PHONE_NUMBER>(0xffffffff)
    y<PHONE_NUMBER>(0xffffffff)
    z<PHONE_NUMBER>(0xffffffff)
  Max fbarriers/Workgrp:   32
  Packet Processor uCode:: 730
  SDMA engine uCode::      58
  IOMMU Support::          None
  Pool Info:
    Pool 1
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED
      Size:                    8388608(0x800000) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Alignment:         4KB
      Accessible by all:       FALSE
    Pool 2
      Segment:                 GLOBAL; FLAGS:
      Size:                    8388608(0x800000) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Alignment:         4KB
      Accessible by all:       FALSE
    Pool 3
      Segment:                 GROUP
      Size:                    64(0x40) KB
      Allocatable:             FALSE
      Alloc Granule:           0KB
      Alloc Alignment:         0KB
      Accessible by all:       FALSE
  ISA Info:
    ISA 1
      Name:                    amdgcn-amd-amdhsa--gfx803
      Machine Models:          HSA_MACHINE_MODEL_LARGE
      Profiles:                HSA_PROFILE_BASE
      Default Rounding Mode:   NEAR
      Default Rounding Mode:   NEAR
      Fast f16:                TRUE
      Workgroup Max Size:      1024(0x400)
      Workgroup Max Size per Dimension:
        x                        1024(0x400)
        y                        1024(0x400)
        z                        1024(0x400)
      Grid Max Size:<PHONE_NUMBER>(0xffffffff)
      Grid Max Size per Dimension:
        x<PHONE_NUMBER>(0xffffffff)
        y<PHONE_NUMBER>(0xffffffff)
        z<PHONE_NUMBER>(0xffffffff)
      FBarrier Max Size:       32
*** Done ***

Hello mnccouk  and other friends,
I have read this topic carefully and have done a lot of work through numerous trials and errors to create a reliable installation procedure for the capricious ROCm 5.7.1 for the RX580.
I note that it is impossible to get the desired result on Ubuntu 22, 24 and Alma 9.2, because this malicious error invariably occurs every time you compile -
	Errors were encountered while processing:
	 amdgpu-dkms
	E: Sub-process /usr/bin/dpkg returned an error code (1)

The desired result is possible only on Ubuntu 20.04.
As a result, I created a two-part script that creates a workable ROCm 5.7.1 without much mental strain :-)
I am giving this script to our community, which is having difficulty installing ROCm 5.7.1.
I'm giving this script to our community, which is having difficulty installing ROCm 5.7.1.
Install_ROCm_5.7.1.zip

mnccouk,
unfortunately I bad don't understand how to use your instruction mnccouk/ollama-gpu-rx580 correctly
It is very brief, and therefore not very clear.
For example, I still can't figure out if it needs to use the sudo command, or if I can do without it.
Also, there are often conflicts between volumes and other confusion.
So I have to delete conflicting volumes frequently.
There is also a problem of delimiting access of several users to one LLM.
Therefore I very don't like docker!
mnccouk, could you tell me how to use ollama c RX580 without using docker?
It was much clearer than using an extra entity like docker.

Sorry you're having difficulty getting this up and running, I feel this is
not the place to be discussing this as it's veering off topic. However, in
the interest of getting you up and running here are a few pointers:-

Should you use sudo or not?
This depends on how docker has been set up - see
https://docs.docker.com/engine/install/linux-postinstall/ to give
permission to your user(that runs docker), so that you don't have to use
sudo. Sudo'ing is still ok though to start your docker container if the
post install steps have not been carried out.

I'm not sure why you have conflicts between volumes, the -v
ollama:/root/.ollama This mounts the local directory "ollama" into the
container as the /root/.ollama directory.

The docker container starts with the Ollama serve command, which
means the API is exposed, in the case of the docker run command, on port

Several client applications(users) can then access the same LLM from
the ${docker_host_ip}:11434. -
https://github.com/ollama/ollama/blob/main/docs/api.md

I've only focused on building a docker image, so I can't help at this stage
with the request to build without utilisation of docker. My forked source
is here - https://github.com/mnccouk/ollama/tree/rx580_gpu if you would
like to try, I believe there is another build script in the codebase to
achieve this - ./scripts/build_linux.sh but have not tried this myself.
On Wed, 23 Oct 2024 at 19:32, T-Shilov @.***> wrote:

mnccouk,
unfortunately I bad don't understand how to use your instruction
mnccouk/ollama-gpu-rx580
https://hub.docker.com/r/mnccouk/ollama-gpu-rx580 correctly
It is very brief, and therefore not very clear.
For example, I still can't figure out if it needs to use the sudo command,
or if I can do without it.
Also, there are often conflicts between volumes and other confusion.
So I have to delete conflicting volumes frequently.
There is also a problem of delimiting access of several users to one LLM.
Therefore I very don't like docker!
mnccouk, could you tell me how to use ollama c RX580 without using docker?
It was much clearer than using an extra entity like docker.
—
Reply to this email directly, view it on GitHub
https://github.com/ollama/ollama/issues/2453#issuecomment-2433128011,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABOLJO6ZKTY7BWXQDSN7WV3Z47TUDAVCNFSM6AAAAABDD3P55SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMZTGEZDQMBRGE
.
You are receiving this because you were mentioned.Message ID:
@.***>

mnccouk,
thanks for the advice, I understand you.
I agree with you that this topic is not the place to discuss docker, it is better to discuss it on discord.
But unfortunately, I can't register on discord because I don't have an actual invitation, and discord rejects my phones for unknown reasons.
So, if mysterious problems arise using docker, then maybe you will make your project on App image?
It consists of only one file, and it is much easier and more convenient to work with it.
By the way, I think Debian is much better and more stable compared to Ubuntu.
That's why I redesigned my scripts for Debian 11.
They are not perfect yet, but Ollama is already working with them.
And if earlier after the question "Why is the sky blue?" Lama 3.1 in Ubuntu first thinks, there is a pause of 5-10 seconds or more, then Debian answers this question without a pause, instantly.
Here is the result in Debian 11:

I will tell you more about the incomprehensible behavior of Llama 3.1 in your Docker.
On the 1st question "Why is the sky blue?" Llama answered without pause, instantly and showed a speed of 27 tokens/sec.
On the 2nd exactly the same question, Llama  thought for a long time, and showed a speed of 12 tokens / sec.
On the 3rd exactly the same question, Llama thought for 4 minutes and showed a speed of 12 tokens/sec.
Why??  This is not normal, and it is impossible to use such a thoughtful and unpredictable Ollama.
PS. When Ollama thinks, the RX580 consumes 105 watts
When Ollama responds, the RX580 consumes 155 watts.

Do you clear the conversation context between promoting the same question?
If not, processing subsequent questions also processes the previous
conversational context, which includes the responses from previous
questions.
This might be why each prompt is taking longer.
On Thu, 24 Oct 2024, 11:34 T-Shilov, @.***> wrote:

I will tell you more about the incomprehensible behavior of Llama 3.1 in
your Docker.
On the 1st question "Why is the sky blue?" Llama answered without pause,
instantly and showed a speed of 27 tokens/sec.
On the 2nd exactly the same question, Llama thought for a long time, and
showed a speed of 12 tokens / sec.
On the 3rd exactly the same question, Llama thought for 4 minutes and
showed a speed of 12 tokens/sec.
Why?? This is not normal, and it is impossible to use such a thoughtful
and unpredictable Ollama.
PS. When Ollama thinks, the RX580 consumes 105 watts
When Ollama responds, the RX580 consumes 155 watts.
—
Reply to this email directly, view it on GitHub
https://github.com/ollama/ollama/issues/2453#issuecomment-2434907495,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABOLJO4CEJATN5DBXT2FABLZ5DEJ7AVCNFSM6AAAAABDD3P55SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMZUHEYDONBZGU
.
You are receiving this because you were mentioned.Message ID:
@.***>

No, I didn't clean it, unaware of the need for cleaning.
After all, I asked only three identical short questions, and already on the 2nd question there was a problem with slowing down Ollma.
And what, even in this simple situation, me need to cleanse the context?
I didn't know about this because there is no such problem in ChatGPT.
OK, how exactly is the context of the conversation cleared?

Hi all,
I now use an NVIDIA graphics card, which is easy to configure and provides a good speed of 90 tokens per second.
But I still have the RX580, and I'm still interested in using it.
I remember that someone is here in this topic promised that its full support in ROCm will soon appear and it will also be easy to use.
Tell me please, has this promise already been fulfilled?

Hi all, I now use an NVIDIA graphics card, which is easy to configure and provides a good speed of 90 tokens per second. But I still have the RX580, and I'm still interested in using it. I remember that someone is here in this topic promised that its full support in ROCm will soon appear and it will also be easy to use. Tell me please, has this promise already been fulfilled?

Waiting on this too, will be immense

Ok, can you be more specific - 1 month, or 1 year, or more?