AMD integrated graphic on linux kernel 6.9.9+, GTT memory, loading freeze fix

This commit reflects changes in linux kernel 6.9.9+ on small APU. LLM load to GTT memory, which is set to 1/2 of RAM by default and can be changed. This allows to use bigger models with AMD APU without VRAM carveout and load models bigger than max VRAM carveout of 16GiB. No hacks like torch-apu-helper, force-host-alloction-APU, Rusticl, unlock VRAM allocation are needed
APU-s this applied to
"gfx1103" //890m, 780m, 760m, 740m GPU RDNA3
"gfx1037" //610M GPU RDNA2
"gfx1035" //680m, 660m GPU RDNA2
"gfx1033" //Van Gogh RDNA2
"gfx1036" //RDNA2 APU
"gfx1151" //RDNA3+ APU
"gfx1152" //RDNA3+ APU
"gfx940" //MI300A CDNA3
"gfx90c" //Radeon Vega 7 Ryzen 5600G
commit also address problems with ollama and APU on kernel 6.9.9+:

ollama server hang due to memory management issues
CPU tensor buffer causing OOM in linux

Changes are applied if only one graphic is preset, and it's from the list above (no discrete graphic card added) and kernel above 6.9.9 is used on the system.  Existing discrete graphic functionality is not changed.
Note:
APU are not officially supported. Can be enabled
for 680m by:
export HSA_OVERRIDE_GFX_VERSION=10.3.0
export OLLAMA_MAX_LOADED_MODELS=1
export OLLAMA_NUM_PARALLEL=1
for 780m by:
export HSA_OVERRIDE_GFX_VERSION=11.0.1
export OLLAMA_MAX_LOADED_MODELS=1
export OLLAMA_NUM_PARALLEL=1
To mitigate gpu hang on unsusported rocm gpu use OLLAMA_MAX_LOADED_MODELS=1, OLLAMA_NUM_PARALLEL=1
Memory available to APU can be adjusted by editing /etc/modprobe.d/ttm.conf (in number of 4k pages , for 48Go it will be):
options ttm pages_limit=12582912
options ttm page_pool_size=12582912
more info
fix issues https://github.com/ollama/ollama/issues/6362#issue-2466206599 https://github.com/ollama/ollama/issues/6572#issue-2498127452
Partially fix (ollama part) https://github.com/ollama/ollama/issues/2637#issue-2146959786
To test this:
git clone https://github.com/Maciej-Mogilany/ollama.git
cd ollama
git checkout AMD_APU_GTT_memory
go generate ./...
go build .
export HSA_OVERRIDE_GFX_VERSION=11.0.1 // for 780m
sudo systemctl stop ollama // stop original ollama for now
./ollama serve
in another terminal
./ollama run model name
if all work, you many replace original ollama bin file with generated form source and add HSA_OVERRIDE_GFX_VERSION=11.0.1 to ollama service for convenience
sudo systemctl start ollama // start original ollama

If you have any problem, please ask sonet 3.5 about it. This way, you will be able to solve 95% of problems.

Hi @MaciejMogilany, is this PR in a working state with models being loaded completely in RAM/GTT? I ran Gemma 27B but I cannot get it working only on the igpu. Instead, it is running on both cpu+igpu (like before the PR).
I cloned https://github.com/Maciej-Mogilany/ollama/, switched to the AMD_APU_GTT_memory branch, removed the arm64 build arch in scripts/build_docker.sh, the ran ./scripts/build_docker.sh. I did not see any errors in building the docker image. Maybe I am doing something wrong.
My setup has a modified ttm pages_limit, 64GB system RAM, and the BIOS option configured for 16GB VRAM reserved.
snuupy@um780:~/ollama-src$ cat /etc/modprobe.d/ttm.conf
options ttm pages_limit=12582912 # 48 GiB, each page is 4 KiB

In amdgpu_top, I see: VRAM: 16384 MiB            GTT: 49152 MiB so GTT should be activated. I'm on 6.9.10-zabbly+ kernel.

Hi @MaciejMogilany, is this PR in a working state with models being loaded completely in RAM/GTT? I ran Gemma 27B but I cannot get it working only on the igpu. Instead, it is running on both cpu+igpu (like before the PR).
I cloned https://github.com/Maciej-Mogilany/ollama/, switched to the AMD_APU_GTT_memory branch, removed the arm64 build arch in scripts/build_docker.sh, the ran ./scripts/build_docker.sh. I did not see any errors in building the docker image. Maybe I am doing something wrong.
My setup has a modified ttm pages_limit, 64GB system RAM, and the BIOS option configured for 16GB VRAM reserved.
snuupy@um780:~/ollama-src$ cat /etc/modprobe.d/ttm.conf
options ttm pages_limit=12582912 # 48 GiB, each page is 4 KiB

In amdgpu_top, I see: VRAM: 16384 MiB GTT: 49152 MiB so GTT should be activated. I'm on 6.9.10-zabbly+ kernel.

Did not test this in docker but GTT+VRAM is set by the kernel to half of RAM if you have 64 GB and reserve 16b to VRAM, GTT will have around 15 GB and the 27B model may not fit. From my testing on 6.10 kernel VRAM is not used at all it will be better to set UMA VRAM in bios to AUTO. This will set VRAM to 500MB or 1G and GTT to 31GB enough to fit the 27B model
you may run "sudo journalctl -u ollama.service | tail -n 400"  and check how many RAM ollama is allocating for GPU to use
should be the same as GTT
you may use amdgpu_top to check GTT size and how it is filled when you load the model

In amdgpu_top, I see: VRAM: 16384 MiB GTT: 49152 MiB so GTT should be activated. I'm on 6.9.10-zabbly+ kernel.

clone repository again and compile. Changed check to account 6.9.9+ kernels. APU diver backported to this kernel.
Can you load llama 3.1 70b or grater model? If I load model grater than 50GB to GTT ollama EOL (96RAM) but can easily do this on CPU

hi @MaciejMogilany, in your last commit there might be an accidental line deletion. After putting it back in and compiling it with build_linux.sh, I get the following:

uname -r outputs 6.9.9-060909-generic for me, seems to not handle the dash after the patch version.

hi @MaciejMogilany, in your last commit there might be an accidental line deletion. After putting it back in and compiling with build_linux.sh, I get the following:
uname -r outputs 6.9.9-060909-generic for me, seems to not handle the dash after the patch version.

should work now

Could you add gfx90c (Ryzen 5600G)? I had it working with HSA_OVERRIDE_GFX_VERSION=9.0.0 and 8GB VRAM allocation in BIOS. After upgrading to kernel 6.10, VRAM is not used anymore and the model goes into GTT. I have set VRAM to auto in BIOS to avoid wasting RAM, but now Ollama wouldn't use GTT either.
BTW, why is there a hardcoded list of iGPUs anyway? Are some iGPUs broken even though ROCm recognizes them? If that is so, there should be an environment variable that forces iGPU use, so that new models can be easily tested.

BTW, why is there a hardcoded list of iGPUs anyway? Are some iGPUs broken even though ROCm recognizes them? If that is so, there should be an environment variable that forces iGPU use, so that new models can be easily tested.

I think because he don't now an other way to know it is an iGPU ... If anyone has an idea how to do it differently I'm sure it may be of interest.
If I correct read amd_kfd driver, the change have been made for all APU.

BTW, why is there a hardcoded list of iGPUs anyway? Are some iGPUs broken even though ROCm recognizes them? If that is so, there should be an environment variable that forces iGPU use, so that new models can be easily tested.

I think because he don't now an other way to know it is an iGPU ... If anyone has an idea how to do it differently I'm sure it may be of interest. If I correct read amd_kfd driver, the change have been made for all APU.

1 hardcoded list of APU like in current implementation. With Ryzen 5600G we should have all usable APU-s including newest RDNA3+
2 ollama way, check against graphic memory if bellow 1G assume it's integrated. It's not reliable as user can change that
3 environmental variable, user need to remember that
4 maybe check against CPU name. APU always have Graphics in cpu name. My have "AMD Ryzen 7 7840HS w/ Radeon 780M Graphics"
Something like this should work.
`package main
import (
"bufio"
"fmt"
"os"
"strings"
)
func main() {
// Get CPU model name
cpuModel := getCPUModel()
// Check for "AMD" and "Graphics" in CPU model name
if strings.Contains(cpuModel, "AMD") && strings.Contains(cpuModel, "Graphics") {
	fmt.Println("AMD APU detected:")
	fmt.Println("CPU Model:", cpuModel)
} else if strings.Contains(cpuModel, "AMD") {
	fmt.Println("AMD CPU without integrated graphics detected:")
	fmt.Println("CPU Model:", cpuModel)
} else {
	fmt.Println("Non-AMD CPU detected:")
	fmt.Println("CPU Model:", cpuModel)
}

}
func getCPUModel() string {
file, err := os.Open("/proc/cpuinfo")
if err != nil {
fmt.Println("Error opening /proc/cpuinfo:", err)
return ""
}
defer file.Close()
scanner := bufio.NewScanner(file)
for scanner.Scan() {
	line := scanner.Text()
	if strings.HasPrefix(line, "model name") {
		return strings.TrimSpace(strings.Split(line, ":")[1])
	}
}
return ""

}`

How do we build this?
Trying on Linux 6.11.0, but memory is shown without GTT.
time=2024-09-16T16:37:02.878-04:00 level=INFO source=amd_linux.go:348 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=11.0.1 time=2024-09-16T16:37:02.879-04:00 level=INFO source=amd_linux.go:348 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=11.0.1 time=2024-09-16T16:37:02.879-04:00 level=INFO source=types.go:105 msg="inference compute" id=0 library=rocm compute=gfx1102 driver=0.0 name=1002:7480 total="8.0 GiB" available="8.0 GiB" time=2024-09-16T16:37:02.879-04:00 level=INFO source=types.go:105 msg="inference compute" id=1 library=rocm compute=gfx1103 driver=0.0 name=1002:15bf total="2.0 GiB" available="513.0 MiB"
`1 DevicePath { render: "/dev/dri/renderD129", card: "/dev/dri/card1", pci: "0000:c5:00.0", sysfs_path: "/sys/bus/pci/devices/0000:c5:00.0", device_id: Some(5567), revision_id: Some(194), device_name: "AMD Radeon 780M" }
drm version: 3.58.0
Device Name              : [AMD Radeon 780M]
PCI (domain:bus:dev.func): 0000:c5:00.0
DeviceID.RevID           : 0x15BF.0xC2
gfx_target_version       : gfx1103
GPU Type  : APU
Family    : GC 11.0.1
ASIC Name : GFX1103_R1
Chip Class: GFX11
Shader Engine (SE)         :   1
Shader Array (SA/SH) per SE:   2
CU per SA                  :   6
Total Compute Unit         :  12
RenderBackendPlus (RB+)    :   4 (32 ROPs)
Peak Pixel Fill-Rate       :  86 GP/s
GPU Clock: 800-2700 MHz
Peak FP32: 8294 GFLOPS
VRAM Type     : DDR5
VRAM Bit Width: 128-bit
Memory Clock  : 1000-2800 MHz
Peak Memory BW: 89 GB/s
ResizableBAR  : Enabled
VRAM              : usage  1400 MiB, total  2048 MiB (usable  1953 MiB)
CPU-Visible VRAM  : usage  1402 MiB, total  2048 MiB (usable  1953 MiB)
GTT               : usage   707 MiB, total 82000 MiB (usable 81986 MiB)
`

How do we build this?
Trying on Linux 6.11.0, but memory is shown without GTT.
time=2024-09-16T16:37:02.878-04:00 level=INFO source=amd_linux.go:348 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=11.0.1
time=2024-09-16T16:37:02.879-04:00 level=INFO source=amd_linux.go:348 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=11.0.1
time=2024-09-16T16:37:02.879-04:00 level=INFO source=types.go:105 msg="inference compute" id=0 library=rocm compute=gfx1102 driver=0.0 name=1002:7480 total="8.0 GiB" available="8.0 GiB"
time=2024-09-16T16:37:02.879-04:00 level=INFO source=types.go:105 msg="inference compute" id=1 library=rocm compute=gfx1103 driver=0.0 name=1002:15bf total="2.0 GiB" available="513.0 MiB"

drm version: 3.58.0

Device Name              : [AMD Radeon 780M]
PCI (domain:bus:dev.func): 0000:c5:00.0
DeviceID.RevID           : 0x15BF.0xC2
gfx_target_version       : gfx1103

GPU Type  : APU
Family    : GC 11.0.1
ASIC Name : GFX1103_R1
Chip Class: GFX11

Shader Engine (SE)         :   1
Shader Array (SA/SH) per SE:   2
CU per SA                  :   6
Total Compute Unit         :  12
RenderBackendPlus (RB+)    :   4 (32 ROPs)
Peak Pixel Fill-Rate       :  86 GP/s

GPU Clock: 800-2700 MHz
Peak FP32: 8294 GFLOPS

VRAM Type     : DDR5
VRAM Bit Width: 128-bit
Memory Clock  : 1000-2800 MHz
Peak Memory BW: 89 GB/s
ResizableBAR  : Enabled

VRAM              : usage  1400 MiB, total  2048 MiB (usable  1953 MiB)
CPU-Visible VRAM  : usage  1402 MiB, total  2048 MiB (usable  1953 MiB)
GTT               : usage   707 MiB, total 82000 MiB (usable 81986 MiB)

git clone https://github.com/MaciejMogilany/ollama.git
cd ollama
git checkout AMD_APU_GTT_memory
go generate ./...
go build .
export HSA_OVERRIDE_GFX_VERSION=11.0.1
./ollama serve
in another terminal
./ollama run model name
if all work, you many replace original ollama bin file with generated form source and add HSA_OVERRIDE_GFX_VERSION=11.0.1 to ollama service for convenience

You have a good point here, it works when using HIP_VISIBLE_DEVICES=1
Said that, it crashes for me, whatever I use 11.0.0 or 11.0.1.
server
time=2024-09-17T23:37:34.727-04:00 level=INFO source=server.go:626 msg="waiting for server to become available" status="llm server loading model"
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD Radeon 780M, compute capability 11.0, VMM: no
llm_load_tensors: ggml ctx size =    0.74 MiB
time=2024-09-17T23:37:41.198-04:00 level=INFO source=server.go:626 msg="waiting for server to become available" status="llm server not responding"
time=2024-09-17T23:37:43.704-04:00 level=INFO source=server.go:626 msg="waiting for server to become available" status="llm server loading model"
llm_load_tensors: offloading 60 repeating layers to GPU
llm_load_tensors: offloaded 60/89 layers to GPU
llm_load_tensors:      ROCm0 buffer size = 44555.62 MiB
llm_load_tensors:  ROCm_Host buffer size = 21323.67 MiB
time=2024-09-17T23:37:44.051-04:00 level=INFO source=server.go:626 msg="waiting for server to become available" status="llm server error"
time=2024-09-17T23:37:44.301-04:00 level=ERROR source=sched.go:456 msg="error loading llama server" error="llama runner process has terminated: signal: segmentation fault (core dumped)"
[GIN] 2024/09/17 - 23:37:44 | 500 |  9.852222901s |       <IP_ADDRESS> | POST     "/api/chat"
time=2024-09-17T23:37:44.301-04:00 level=WARN source=amd_linux.go:130 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-09-17T23:37:44.303-04:00 level=INFO source=amd_linux.go:415 msg="filtering out device per user request" id=0 visible_devices=[1]
time=2024-09-17T23:37:44.306-04:00 level=INFO source=amd_linux.go:451 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=11.0.0
time=2024-09-17T23:37:44.557-04:00 level=WARN source=amd_linux.go:130 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-09-17T23:37:44.558-04:00 level=INFO source=amd_linux.go:415 msg="filtering out device per user request" id=0 visible_devices=[1]
time=2024-09-17T23:37:44.563-04:00 level=INFO source=amd_linux.go:451 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=11.0.0

client
$ ./ollama run mistral-large:latest
Error: llama runner process has terminated: signal: segmentation fault (core dumped)

journald
Sep 17 23:37:43 MSICompu kernel: ollama_llama_se[1616538]: segfault at 0 ip 00007f0142330ef4 sp 00007ffe065f4490 error 4 in libggml.so[1a12fef4,7f01422dd000+26f000] likely on CPU 10 (core 5, socket 0)
Sep 17 23:37:43 MSICompu kernel: Code: 48 8b 00 48 8b 80 90 00 00 00 48 8b 7d f8 ff d0 48 83 c4 10 5d c3 f3 0f 1e fa 55 48 89 e5 48 83 ec 10 48 89 7d f8 48 8b 45 f8 <48> 8b 00 48 83 b8 a0 00 00 00 00 0f 85 21 00 00 00 48 8d 3d c4 ed
Sep 17 23:37:43 MSICompu systemd-coredump[1616573]: Process 1616538 (ollama_llama_se) of user 1000 terminated abnormally with signal 11/SEGV, processing...
Sep 17 23:37:43 MSICompu systemd[1]: Started Process Core Dump (PID 1616573/UID 0).
Sep 17 23:37:43 MSICompu systemd[1]: Started Pass systemd-coredump journal entries to relevant user for potential DrKonqi handling.
Sep 17 23:37:43 MSICompu systemd-coredump[1616574]: [🡕] Process 1616538 (ollama_llama_se) of user 1000 terminated abnormally without generating a coredump.
Sep 17 23:37:43 MSICompu systemd[1]<EMAIL_ADDRESS>Deactivated successfully.
Sep 17 23:37:43 MSICompu drkonqi-coredump-processor[1616575]: "/tmp/ollama406696529/runners/rocm/ollama_llama_server" 1616538 ""
Sep 17 23:37:43 MSICompu systemd[1378]: Started Launch DrKonqi for a systemd-coredump crash (PID 1616575/UID 0).
Sep 17 23:37:43 MSICompu drkonqi-coredump-launcher[1616580]: Unable to find file for pid 1616538 expected at "kcrash-metadata/ollama_llama_server.ae110f0170c6462aa93a204497e6f756.1616538.ini"
Sep 17 23:37:43 MSICompu drkonqi-coredump-launcher[1616580]: Nothing handled the dump :O

try small model
śr., 18 wrz 2024, 05:42 użytkownik WalterCool @.***>
napisał:

You have a good point here, it works when using HIP_VISIBLE_DEVICES=1
Said that, it crashes for me, whatever I use 11.0.0 or 11.0.1.
server
time=2024-09-17T23:37:34.727-04:00 level=INFO source=server.go:626 msg="waiting for server to become available" status="llm server loading model"
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
Device 0: AMD Radeon 780M, compute capability 11.0, VMM: no
llm_load_tensors: ggml ctx size =    0.74 MiB
time=2024-09-17T23:37:41.198-04:00 level=INFO source=server.go:626 msg="waiting for server to become available" status="llm server not responding"
time=2024-09-17T23:37:43.704-04:00 level=INFO source=server.go:626 msg="waiting for server to become available" status="llm server loading model"
llm_load_tensors: offloading 60 repeating layers to GPU
llm_load_tensors: offloaded 60/89 layers to GPU
llm_load_tensors:      ROCm0 buffer size = 44555.62 MiB
llm_load_tensors:  ROCm_Host buffer size = 21323.67 MiB
time=2024-09-17T23:37:44.051-04:00 level=INFO source=server.go:626 msg="waiting for server to become available" status="llm server error"
time=2024-09-17T23:37:44.301-04:00 level=ERROR source=sched.go:456 msg="error loading llama server" error="llama runner process has terminated: signal: segmentation fault (core dumped)"
[GIN] 2024/09/17 - 23:37:44 | 500 |  9.852222901s |       <IP_ADDRESS> | POST     "/api/chat"
time=2024-09-17T23:37:44.301-04:00 level=WARN source=amd_linux.go:130 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-09-17T23:37:44.303-04:00 level=INFO source=amd_linux.go:415 msg="filtering out device per user request" id=0 visible_devices=[1]
time=2024-09-17T23:37:44.306-04:00 level=INFO source=amd_linux.go:451 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=11.0.0
time=2024-09-17T23:37:44.557-04:00 level=WARN source=amd_linux.go:130 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-09-17T23:37:44.558-04:00 level=INFO source=amd_linux.go:415 msg="filtering out device per user request" id=0 visible_devices=[1]
time=2024-09-17T23:37:44.563-04:00 level=INFO source=amd_linux.go:451 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=11.0.0
client
$ ./ollama run mistral-large:latest
Error: llama runner process has terminated: signal: segmentation fault (core dumped)
journald
Sep 17 23:37:43 MSICompu kernel: ollama_llama_se[1616538]: segfault at 0 ip 00007f0142330ef4 sp 00007ffe065f4490 error 4 in libggml.so[1a12fef4,7f01422dd000+26f000] likely on CPU 10 (core 5, socket 0)
Sep 17 23:37:43 MSICompu kernel: Code: 48 8b 00 48 8b 80 90 00 00 00 48 8b 7d f8 ff d0 48 83 c4 10 5d c3 f3 0f 1e fa 55 48 89 e5 48 83 ec 10 48 89 7d f8 48 8b 45 f8 <48> 8b 00 48 83 b8 a0 00 00 00 00 0f 85 21 00 00 00 48 8d 3d c4 ed
Sep 17 23:37:43 MSICompu systemd-coredump[1616573]: Process 1616538 (ollama_llama_se) of user 1000 terminated abnormally with signal 11/SEGV, processing...
Sep 17 23:37:43 MSICompu systemd[1]: Started Process Core Dump (PID 1616573/UID 0).
Sep 17 23:37:43 MSICompu systemd[1]: Started Pass systemd-coredump journal entries to relevant user for potential DrKonqi handling.
Sep 17 23:37:43 MSICompu systemd-coredump[1616574]: [🡕] Process 1616538 (ollama_llama_se) of user 1000 terminated abnormally without generating a coredump.
Sep 17 23:37:43 MSICompu systemd[1]: @.***: Deactivated successfully.
Sep 17 23:37:43 MSICompu drkonqi-coredump-processor[1616575]: "/tmp/ollama406696529/runners/rocm/ollama_llama_server" 1616538 ""
Sep 17 23:37:43 MSICompu systemd[1378]: Started Launch DrKonqi for a systemd-coredump crash (PID 1616575/UID 0).
Sep 17 23:37:43 MSICompu drkonqi-coredump-launcher[1616580]: Unable to find file for pid 1616538 expected at "kcrash-metadata/ollama_llama_server.ae110f0170c6462aa93a204497e6f756.1616538.ini"
Sep 17 23:37:43 MSICompu drkonqi-coredump-launcher[1616580]: Nothing handled the dump :O
—
Reply to this email directly, view it on GitHub
https://github.com/ollama/ollama/pull/6282#issuecomment-2357431124, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ANORXNZYCBZHR4MRK4AJYZDZXDZCXAVCNFSM6AAAAABMIHDP52VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNJXGQZTCMJSGQ
.
You are receiving this because you were mentioned.Message ID:
@.***>

Using Meta Llama 3.1 8B. Same problem using iGPU + GTT.
No idea if this might be actually a problem of AMD itself. I been concerned if my dGPU is actually creating problems with GTT memory for iGPU.
output.log

Using Meta Llama 3.1 8B. Same problem using iGPU + GTT.
No idea if this might be actually a problem of AMD itself. I been concerned if my dGPU is actually creating problems with GTT memory for iGPU.
For this test, added extra UMA memory from 2GB to 4GB VRAM
output.log

You may try pt pull fresh copy from git and try again. I made some changes. I think that the problem lies that no-mmap is not set because APU is no a Primary GPU in Your system. I made changes that will set this flag if any of the available graphic card is APU.

Any reason not to include gfx900/gfx90c for the 5xxx series?

Any reason not to include gfx900/gfx90c for the 5xxx series?

it's included  "gfx90c" //Radeon Vega 7 Ryzen 5600G

Great to see this being implemented. However, it's not working yet for me in the current state. I'm using the following config: Ubuntu 24.04.1 LTS (GNU/Linux 6.10.6-061006-generic x86_64) with Ryzen 8700G, 64GB RAM, rocm v. 6.2.1.
When using llama cpp (via llamafile) directly, it works out of the box, for example: "./Meta-Llama-3.1-8B-Instruct.Q6_K.llamafile -ngl 999" I can see with radeontop that the model has been completely loaded into the GTT and the graphics pipe goes to 100% when inferencing.
I compiled this PR and ran it with the same model Llama-3.1-8B:
export HSA_OVERRIDE_GFX_VERSION=11.0.1
export AMD_LOG_LEVEL=3
./ollama serve

The log looks good at first (offloaded 33/33 layers to GPU) but in the end it crashes without a meaningful error message (seg fault):

error="llama runner process has terminated: signal: segmentation fault (core dumped)"

This happen after repo sync. Something around ollama tag 3.09-3.11 cause this. Looking at recent commits, I don't see immediate offender. Change this PR to draft for now.

https://github.com/ollama/ollama/pull/6282#issuecomment-2357431124
https://github.com/ollama/ollama/pull/6282#issuecomment-2371240145
I disabled no-mmap this fix seg fault but will reduce model size you will be able to load. Extensive models may get stuck on load.

Sorry my delay responding, been stuck at bug report 6857
This works for me on small models now, but crashes on large models with OOM. It works fine under dGPU+CPU
Running something like Codestral or Mistral Small works fine, but running something like Mistral Large makes a crash.
Running both iGPU+dGPU still experience a crash, seems like iGPU loads before dGPU causing dGPU to never be used.
I'm okay with this result so far, as iGPU works as long is smaller than GTT size, but eventually, it should try to fit as-much-of GTT then use RAM

Hey Maciej, I think your code have a huge regression for dGPUs when loading a big model. I can easily reproduce this when using your code (75b9e9df0d7e1661fac7e3daec33e8e056e704c5) and latest official.
Loading a huge model is impossible, while official makes it "bearable" depending how big is it.
Just adding some numbers here:
igpu.txt
dgpu.txt
dgpu-2nd_test.txt
cpu.txt
When testing latest ollama:
dgpu-officialbuild.txt
cpu-officialbuild.txt

Hey Maciej, I think your code have a huge regression for dGPUs when loading a big model. I can easily reproduce this when using your code (75b9e9d) and latest official.

This code may increase model load time, but regression you are saying probably are from your local libraries used to compile

Loading a huge model is impossible, while official makes it "bearable" depending how big is it.

Thanks for that. If official, don't give seq fault. Then problem are from our local setups. All seq fault reports I see are from linux on the edge. You have Gentoo other I have seen are archlinux users. Some last update, probably some cuda library cause seq fault. This comment may apply here

To add a data point to this discussion, this patch is working perfectly for me on my Framework 13 (AMD).

Ryzen 7 7840U
780M Graphics
Arch Linux

Models tested: Llama3.2, qwen2.5.

works great for me, thanks for that, I don't have a rocm driver installed.
os: ubuntu 24.04
cpu: AMD Ryzen™ 7 PRO 6850U
gpu: 680m
services:
  ollama:
    #image: ollama/ollama:rocm
    image: ollama-build:apu-rocm
    container_name: ollama
    environment:
      - HSA_OVERRIDE_GFX_VERSION=10.3.0
      - HSA_ENABLE_SDMA=0
      - HCC_AMDGPU_TARGET=gfx1035
      - OLLAMA_FLASH_ATTENTION=1
    devices:
      - /dev/kfd
      - /dev/dri
    ports:
      - 11434:11434
    volumes:
      - ./ollama:/root/.ollama
    security_opt:
      - seccomp:unconfined

i dont know it the environment variables necessary

@badsmoke  did you build that docker yourself?

yes
OLLAMA_SKIP_MANIFEST_CREATE=1 sudo OLLAMA_SKIP_MANIFEST_CREATE=1 ./scripts/build_docker.sh
i have pushed it to my public repo if you want to try it out.
image: docker.badcloud.eu/library/ollama-build:apu-rocm

I have a Ryzen 8700G / 780m with 64 GB RAM.
Running Kernel - 6.10.10.
Built the Ollama exactly as described above. It did not seem to run on APU:
time=2024-10-04T22:36:18.659+05:30 level=INFO source=common.go:49 msg="Dynamic LLM libraries" runners="[cpu_avx2 cpu cpu_avx]"

Just wanted to add a testimonial: this branch ("AMD_APU_GTT_memory") worked perfectly for my ASUS Vivobook S15  M5506NA laptop - i.e. an AMD Ryzen 5 7535HS, with rocminfo reporting a gfx1035.
Like many others, my BIOS does not allow changing the VRAM setting - which is pinned to 512MB.  This branch allowed me to run ollama with llama3.2, while my laptop's CPU fans remain completely silent... since all the work is done in the GPU.
Many thanks for this.
I was trying various things, so I am not sure if all of the following were necessary - but here they are anyway:
git clone https://github.com/Maciej-Mogilany/ollama.git
git checkout AMD_APU_GTT_memory
git submodule init
git submodule update --recursive
export AMDGPU_TARGETS="gfx1030"  # as shown by rocminfo
vim llm/generate/gen_linux.sh  # Added CMAKE_DEFS="-DLLAMA_HIP_UMA=ON ...   in front of the CMAKE_DEFS
export OLLAMA_SKIP_ONEAPI_GENERATE=1  # the build otherwise depended on intel oneapi which I don't want
go generate ./...
go build
./ollama serve

...and then, from another terminal
./ollama pull llama3.2
./ollama run llama3.2

I then saw in amdgpu_top  that as I was interacting with the LLM, half of my GTT (which is 8GB) was used; as well as a variable part of the VRAM (I am guessing data were being "swapped" into and out of the VRAM from the GTT; but I may be talking nonsense here. All I know is that I can now talk to an LLM on my laptop, running really fast (feels faster than my CPU, but TBH all I care is that I don't hear the fans blowing at max speed anymore while all 6(real) and 6(smt) cores were used to 100%. No noise, much better battery use...
...again, many thanks.

git clone https://github.com/Maciej-Mogilany/ollama.git
git checkout AMD_APU_GTT_memory
git submodule init
git submodule update --recursive
export AMDGPU_TARGETS="gfx1030"  # as shown by rocminfo
export HSA_OVERRIDE_GFX_VERSION=10.3.0
vim llm/generate/gen_linux.sh  # Added CMAKE_DEFS="-DLLAMA_HIP_UMA=ON ...   in front of the CMAKE_DEFS
export OLLAMA_SKIP_ONEAPI_GENERATE=1  # the build otherwise depended on intel oneapi which I don't want
go generate ./...
go build
./ollama serve

...and then, from another terminal
./ollama pull llama3.2
./ollama run llama3.2

here are simpler instructions https://github.com/ollama/ollama/pull/6282#issuecomment-2354642416

@MaciejMogilany

EDIT:
ouch sorry didn't read your message with an understanding patch search for 6.10+ kernel, you have 6.9.10

Zabbly 6.9.9/10 or newer have the GTT patches applied (backported), but it doesn't matter as I am now on 6.10.12-zabbly+
I followed your latest instructions to build the binary and was able to get it working with gemma2:27b-instruct-q6_K which uses 22-23GB VRAM when loaded. I specifically picked this one because if it's not working, it would go over the 16GB VRAM limit I have set in BIOS. amdgpu_top shows 24GB GTT available (as half of 48GB = 24GB) and this setup has a total system memory of 64GB.
ollama       | time=2024-10-06T03:13:32.656Z level=DEBUG source=amd_linux.go:386 msg="amdgpu memory" gpu=0 total="23.4 GiB"
ollama       | time=2024-10-06T03:13:32.656Z level=DEBUG source=amd_linux.go:387 msg="amdgpu memory" gpu=0 available="23.4 GiB"

and it uses GTT no problem, with 22-23GB used out of 24GB GTT.
I have not yet tested a custom ttm but will do so soon
I am now trying to build the docker image, but the latest commit in https://github.com/ollama/ollama/blob/main/scripts/build_docker.sh (and subsequently https://github.com/MaciejMogilany/ollama/blob/main/scripts/build_docker.sh) both don't work when running OLLAMA_SKIP_MANIFEST_CREATE=1 sudo OLLAMA_SKIP_MANIFEST_CREATE=1 ./scripts/build_docker.sh @badsmoke nor just ./scripts/build_docker.sh since it was refactored to the Dockerfile
snuupy@snuum780:~/ollama-GTT$ OLLAMA_SKIP_MANIFEST_CREATE=1 sudo OLLAMA_SKIP_MANIFEST_CREATE=1 ./scripts/build_docker.sh
Building Ollama
VERSION=a7e070e-dirty
PLATFORM=linux/arm64,linux/amd64
Building ollama/ollama:a7e070e-dirty locally.  set PUSH=1 to push
[+] Building 0.0s (0/0)                                                                                docker:default
ERROR: docker exporter does not currently support exporting manifest lists

Looks like I need a docker registry? Can I not just build locally and commit to the local image repository? What is the command to build only the amd64-rocm image?
Either way I reverted to the previous build_docker.sh
and got
ollama  | time=2024-10-06T09:03:47.792Z level=DEBUG source=amd_linux.go:332 msg="AMD APU valid to use GTT memory"
ollama  | time=2024-10-06T09:03:47.792Z level=INFO source=amd_linux.go:366 msg="For best performance set APU VRAM carveout to <=2GiB"
ollama  | time=2024-10-06T09:03:47.792Z level=DEBUG source=amd_linux.go:386 msg="amdgpu memory" gpu=0 total="23.4 GiB"
ollama  | time=2024-10-06T09:03:47.792Z level=DEBUG source=amd_linux.go:387 msg="amdgpu memory" gpu=0 available="23.4 GiB"
ollama  | time=2024-10-06T09:03:47.792Z level=DEBUG source=amd_common.go:18 msg="evaluating potential rocm lib dir /usr/lib/ollama"
ollama  | time=2024-10-06T09:03:47.792Z level=DEBUG source=amd_common.go:61 msg="detected ROCM next to ollama executable /usr/lib/ollama"

but when I run the docker container with docker exec -it ollama /bin/bash and ollama run gemma2:27b-instruct-q6_K it uses CPU instead of GTT.
I did not see any error logs.

docker build log
snuupy@snuum780:~/ollama-GTT$ OLLAMA_SKIP_MANIFEST_CREATE=1 sudo OLLAMA_SKIP_MANIFEST_CREATE=1 ./scripts/build_docker.sh
[+] Building 16.9s (43/43) FINISHED                                                                                docker:default
 => [internal] load build definition from Dockerfile                                                                                0.0s
 => => transferring dockerfile: 11.35kB                                                                                0.0s
 => [internal] load .dockerignore                                                                                0.0s
 => => transferring context: 129B                                                                                0.0s
 => [internal] load metadata for docker.io/nvidia/cuda:11.3.1-devel-centos7                                                                                0.3s
 => [internal] load metadata for docker.io/nvidia/cuda:12.4.0-devel-centos7                                                                                0.3s
 => [internal] load metadata for docker.io/library/ubuntu:22.04                                                                                0.3s
 => [internal] load metadata for docker.io/library/centos:7                                                                                0.4s
 => [cuda-12-build-amd64 1/6] FROM docker.io/nvidia/cuda:12.4.0-devel-centos7@sha256:9538c42b4381ed8844210d0069e5bf848e2b97a91281ca03ad2ec1cd19e618cc                                      0.0s
 => [internal] load build context                                                                                0.0s
 => => transferring context: 40.75kB                                                                                0.0s
 => [cuda-11-build-amd64 1/6] FROM docker.io/nvidia/cuda:11.3.1-devel-centos7@sha256:47c328a6ecf7760a65d6eee834be8040054ada756d14fe848fb5842aa0fb96d6                                      0.0s
 => [runtime-amd64 1/8] FROM docker.io/library/ubuntu:22.04@sha256:58b87898e82351c6cf9cf5b9f3c20257bb9e2dcf33af051e12ce532d7f94e3fe                                                        0.0s
 => [cpu-builder-amd64 1/5] FROM docker.io/library/centos:7@sha256:be65f488b7764ad3638f236b7b515b3678369a5124c47b8d32916d6487418ea4                                                        0.0s
 => CACHED [cpu-builder-amd64 2/5] COPY ./scripts/rh_linux_deps.sh /                                                                                0.0s
 => CACHED [cpu-builder-amd64 3/5] RUN CMAKE_VERSION=3.22.1 GOLANG_VERSION=1.22.5 sh /rh_linux_deps.sh                                                                                0.0s
 => CACHED [llm-code 1/3] COPY .git .git                                                                                0.0s
 => CACHED [llm-code 2/3] COPY .gitmodules .gitmodules                                                                                0.0s
 => CACHED [llm-code 3/3] COPY llm llm                                                                                0.0s
 => CACHED [cpu-builder-amd64 4/5] COPY --from=llm-code / /go/src/github.com/ollama/ollama/                                                                                0.0s
 => CACHED [cpu-builder-amd64 5/5] WORKDIR /go/src/github.com/ollama/ollama/llm/generate                                                                                0.0s
 => CACHED [static-build-amd64 1/1] RUN --mount=type=cache,target=/root/.ccache     OLLAMA_CPU_TARGET="static" bash gen_linux.sh                                                           0.0s
 => CACHED [container-build-amd64 1/3] WORKDIR /go/src/github.com/ollama/ollama                                                                                0.0s
 => [container-build-amd64 2/3] COPY . .                                                                                0.4s
 => [container-build-amd64 3/3] RUN --mount=type=cache,target=/root/.ccache     go build -trimpath -o dist/linux-amd64/bin/ollama .                                                       15.9s
 => CACHED [runtime-amd64 2/8] RUN apt-get update &&     apt-get install -y ca-certificates &&     apt-get clean && rm -rf /var/lib/apt/lists/*                                            0.0s
 => CACHED [runtime-amd64 3/8] COPY --from=container-build-amd64 /go/src/github.com/ollama/ollama/dist/linux-amd64/bin/ /bin/                                                              0.0s
 => CACHED [cpu-build-amd64 1/1] RUN --mount=type=cache,target=/root/.ccache     OLLAMA_SKIP_STATIC_GENERATE=1 OLLAMA_CPU_TARGET="cpu" bash gen_linux.sh                                   0.0s
 => CACHED [runtime-amd64 4/8] COPY --from=cpu-build-amd64 /go/src/github.com/ollama/ollama/dist/linux-amd64/lib/ /lib/                                                                    0.0s
 => CACHED [cpu_avx-build-amd64 1/1] RUN --mount=type=cache,target=/root/.ccache     OLLAMA_SKIP_STATIC_GENERATE=1 OLLAMA_CPU_TARGET="cpu_avx" bash gen_linux.sh                           0.0s
 => CACHED [runtime-amd64 5/8] COPY --from=cpu_avx-build-amd64 /go/src/github.com/ollama/ollama/dist/linux-amd64/lib/ /lib/                                                                0.0s
 => CACHED [cpu_avx2-build-amd64 1/1] RUN --mount=type=cache,target=/root/.ccache     OLLAMA_SKIP_STATIC_GENERATE=1 OLLAMA_CPU_TARGET="cpu_avx2" bash gen_linux.sh                         0.0s
 => CACHED [runtime-amd64 6/8] COPY --from=cpu_avx2-build-amd64 /go/src/github.com/ollama/ollama/dist/linux-amd64/lib/ /lib/                                                               0.0s
 => CACHED [cuda-11-build-amd64 2/6] COPY ./scripts/rh_linux_deps.sh /                                                                                0.0s
 => CACHED [cuda-11-build-amd64 3/6] RUN CMAKE_VERSION=3.22.1 sh /rh_linux_deps.sh                                                                                0.0s
 => CACHED [cuda-11-build-amd64 4/6] COPY --from=llm-code / /go/src/github.com/ollama/ollama/                                                                                0.0s
 => CACHED [cuda-11-build-amd64 5/6] WORKDIR /go/src/github.com/ollama/ollama/llm/generate                                                                                0.0s
 => CACHED [cuda-11-build-amd64 6/6] RUN --mount=type=cache,target=/root/.ccache     OLLAMA_SKIP_STATIC_GENERATE=1     OLLAMA_SKIP_CPU_GENERATE=1     CMAKE_CUDA_ARCHITECTURES="50;52;53;  0.0s
 => CACHED [runtime-amd64 7/8] COPY --from=cuda-11-build-amd64 /go/src/github.com/ollama/ollama/dist/linux-amd64/lib/ /lib/                                                                0.0s
 => CACHED [cuda-12-build-amd64 2/6] COPY ./scripts/rh_linux_deps.sh /                                                                                0.0s
 => CACHED [cuda-12-build-amd64 3/6] RUN CMAKE_VERSION=3.22.1 sh /rh_linux_deps.sh                                                                                0.0s
 => CACHED [cuda-12-build-amd64 4/6] COPY --from=llm-code / /go/src/github.com/ollama/ollama/                                                                                0.0s
 => CACHED [cuda-12-build-amd64 5/6] WORKDIR /go/src/github.com/ollama/ollama/llm/generate                                                                                0.0s
 => CACHED [cuda-12-build-amd64 6/6] RUN --mount=type=cache,target=/root/.ccache     OLLAMA_SKIP_STATIC_GENERATE=1     OLLAMA_SKIP_CPU_GENERATE=1     CMAKE_CUDA_ARCHITECTURES="60;61;62;  0.0s
 => CACHED [runtime-amd64 8/8] COPY --from=cuda-12-build-amd64 /go/src/github.com/ollama/ollama/dist/linux-amd64/lib/ /lib/                                                                0.0s
 => exporting to image                                                                                0.0s
 => => exporting layers                                                                                0.0s
 => => writing image sha256:4b702dc2bc635f1d557227e89790e0f24f34d1710786dbff78889a8636e4f658                                                                                0.0s
 => => naming to docker.io/ollama/release:a7e070e-dirty-amd64                                                                                0.0s
[+] Building 102.9s (27/36)                                                                                docker:default
 => [internal] load build context                                                                                0.0s
 => => transferring context: 37.86kB                                                                                0.0s
 => CACHED [rocm-build-amd64 1/7] FROM docker.io/rocm/dev-centos-7:6.1.2-complete@sha256:d32024d55afc71bd8e9b9eb20835a369b9df69c1f1b0a386fe06d3d9d5e19293                                  0.0s
 => CACHED [runtime-rocm 1/8] FROM docker.io/library/ubuntu:22.04@sha256:58b87898e82351c6cf9cf5b9f3c20257bb9e2dcf33af051e12ce532d7f94e3fe                                                  0.0s
 => [cpu-builder-amd64 1/5] FROM docker.io/library/centos:7@sha256:be65f488b7764ad3638f236b7b515b3678369a5124c47b8d32916d6487418ea4                                                        0.0s
 => CACHED [llm-code 1/3] COPY .git .git                                                                                0.0s
 => CACHED [llm-code 2/3] COPY .gitmodules .gitmodules                                                                                0.0s
 => CACHED [llm-code 3/3] COPY llm llm                                                                                0.0s
 => CACHED [cpu-builder-amd64 2/5] COPY ./scripts/rh_linux_deps.sh /                                                                                0.0s
 => CACHED [cpu-builder-amd64 3/5] RUN CMAKE_VERSION=3.22.1 GOLANG_VERSION=1.22.5 sh /rh_linux_deps.sh                                                                                0.0s
 => CACHED [cpu-builder-amd64 4/5] COPY --from=llm-code / /go/src/github.com/ollama/ollama/                                                                                0.0s
 => CACHED [cpu-builder-amd64 5/5] WORKDIR /go/src/github.com/ollama/ollama/llm/generate                                                                                0.0s
 => CACHED [static-build-amd64 1/1] RUN --mount=type=cache,target=/root/.ccache     OLLAMA_CPU_TARGET="static" bash gen_linux.sh                                                           0.0s
 => CACHED [container-build-amd64 1/3] WORKDIR /go/src/github.com/ollama/ollama                                                                                0.0s
 => CACHED [container-build-amd64 2/3] COPY . .                                                                                [+] Building 103.1s (27/36)                                                                                docker:defaultHED [container-build-amd64 3/3] RUN --mount=type=cache,target=/root/.ccache     go build -trimpath -o dist/linux-amd64/bin/ollama .
 => [internal] load build context                                                                                0.0sHED [cpu-build-amd64 1/1] RUN --mount=type=cache,target=/root/.ccache     OLLAMA_SKIP_STATIC_GENERATE=1 OLLAMA_CPU_TARGET="cpu" bash gen_linux.sh
 => => transferring context: 37.86kB                                                                                0.0scm-build-amd64 2/7] COPY ./scripts/rh_linux_deps.sh /
 => CACHED [rocm-build-amd64 1/7] FROM docker.io/rocm/dev-centos-7:6.1.2-complete@sha256:d32024d55afc71bd8e9b9eb20835a369b9df69c1f1b0a386fe06d3d9d5e19293                                  0.0sHED [cpu_avx-build-amd64 1/1] RUN --mount=type=cache,target=/root/.ccache     OLLAMA_SKIP_STATIC_GENERATE=1 OLLAMA_CPU_TARGET="cpu_avx" bash gen_linux.sh
 => CACHED [runtime-rocm 1/8] FROM docker.io/library/ubuntu:22.04@sha256:58b87898e82351c6cf9cf5b9f3c20257bb9e2dcf33af051e12ce532d7f94e3fe                                                  0.0sHED [cpu_avx2-build-amd64 1/1] RUN --mount=type=cache,target=/root/.ccache     OLLAMA_SKIP_STATIC_GENERATE=1 OLLAMA_CPU_TARGET="cpu_avx2" bash gen_linux.sh
 => [cpu-builder-amd64 1/5] FROM docker.io/library/centos:7@sha256:be65f488b7764ad3638f236b7b515b3678369a5124c47b8d32916d6487418ea4                                                        0.0scm-build-amd64 3/7] RUN CMAKE_VERSION=3.22.1 sh /rh_linux_deps.sh
 => CACHED [llm-code 1/3] COPY .git .git                                                                                0.0scm-build-amd64 4/7] COPY --from=llm-code / /go/src/github.com/ollama/ollama/
 => CACHED [llm-code 2/3] COPY .gitmodules .gitmodules                                                                                0.0scm-build-amd64 5/7] WORKDIR /go/src/github.com/ollama/ollama/llm/generate
 => CACHED [llm-code 3/3] COPY llm llm                                                                                0.0scm-build-amd64 6/7] RUN --mount=type=cache,target=/root/.ccache     OLLAMA_SKIP_STATIC_GENERATE=1 OLLAMA_SKIP_CPU_GENERATE=1 bash gen_linux.sh
 => CACHED [cpu-builder-amd64 2/5] COPY ./scripts/rh_linux_deps.sh /                                                                                0.0s# [  7%] Building HIP object ggml/src/CMakeFiles/ggml.dir/ggml-cuda/arange.cu.o
 => CACHED [cpu-builder-amd64 3/5] RUN CMAKE_VERSION=3.22.1 GOLANG_VERSION=1.22.5 sh /rh_linux_deps.sh                                                                                0.0s# [  7%] Building CXX object common/CMakeFiles/build_info.dir/build-info.cpp.o
 => CACHED [cpu-builder-amd64 4/5] COPY --from=llm-code / /go/src/github.com/ollama/ollama/                                                                                0.0s# [  7%] Built target build_info
 => CACHED [cpu-builder-amd64 5/5] WORKDIR /go/src/github.com/ollama/ollama/llm/generate                                                                                0.0s# [  7%] Building HIP object ggml/src/CMakeFiles/ggml.dir/ggml-cuda/binbcast.cu.o
 => CACHED [static-build-amd64 1/1] RUN --mount=type=cache,target=/root/.ccache     OLLAMA_CPU_TARGET="static" bash gen_linux.sh                                                           0.0s# [ 10%] Building HIP object ggml/src/CMakeFiles/ggml.dir/ggml-cuda/clamp.cu.o
 => CACHED [container-build-amd64 1/3] WORKDIR /go/src/github.com/ollama/ollama                                                                                0.0s# [ 10%] Building HIP object ggml/src/CMakeFiles/ggml.dir/ggml-cuda/concat.cu.o
 => CACHED [container-build-amd64 2/3] COPY . .                                                                                [+] Building 103.2s (27/36)                                                                                docker:defaultHED [container-build-amd64 3/3] RUN --mount=type=cache,target=/root/.ccache     go build -trimpath -o dist/linux-amd64/bin/ollama .
 => [internal] load build context                                                                                0.0sHED [cpu-build-amd64 1/1] RUN --mount=type=cache,target=/root/.ccache     OLLAMA_SKIP_STATIC_GENERATE=1 OLLAMA_CPU_TARGET="cpu" bash gen_linux.sh
 => => transferring context: 37.86kB                                                                                0.0scm-build-amd64 2/7] COPY ./scripts/rh_linux_deps.sh /
 => CACHED [rocm-build-amd64 1/7] FROM docker.io/rocm/dev-centos-7:6.1.2-complete@sha256:d32024d55afc71bd8e9b9eb20835a369b9df69c1f1b0a386fe06d3d9d5e19293                                  0.0sHED [cpu_avx-build-amd64 1/1] RUN --mount=type=cache,target=/root/.ccache     OLLAMA_SKIP_STATIC_GENERATE=1 OLLAMA_CPU_TARGET="cpu_avx" bash gen_linux.sh
 => CACHED [runtime-rocm 1/8] FROM docker.io/library/ubuntu:22.04@sha256:58b87898e82351c6cf9cf5b9f3c20257bb9e2dcf33af051e12ce532d7f94e3fe                                                  0.0sHED [cpu_avx2-build-amd64 1/1] RUN --mount=type=cache,target=/root/.ccache     OLLAMA_SKIP_STATIC_GENERATE=1 OLLAMA_CPU_TARGET="cpu_avx2" bash gen_linux.sh
 => [cpu-builder-amd64 1/5] FROM docker.io/library/centos:7@sha256:be65f488b7764ad3638f236b7b515b3678369a5124c47b8d32916d6487418ea4                                                        0.0scm-build-amd64 3/7] RUN CMAKE_VERSION=3.22.1 sh /rh_linux_deps.sh
 => CACHED [llm-code 1/3] COPY .git .git                                                                                0.0scm-build-amd64 4/7] COPY --from=llm-code / /go/src/github.com/ollama/ollama/
 => CACHED [llm-code 2/3] COPY .gitmodules .gitmodules                                                                                0.0scm-build-amd64 5/7] WORKDIR /go/src/github.com/ollama/ollama/llm/generate
 => CACHED [llm-code 3/3] COPY llm llm                                                                                0.0scm-build-amd64 6/7] RUN --mount=type=cache,target=/root/.ccache     OLLAMA_SKIP_STATIC_GENERATE=1 OLLAMA_SKIP_CPU_GENERATE=1 bash gen_linux.sh
 => CACHED [cpu-builder-amd64 2/5] COPY ./scripts/rh_linux_deps.sh /                                                                                0.0s# [  7%] Building HIP object ggml/src/CMakeFiles/ggml.dir/ggml-cuda/arange.cu.o
 => CACHED [cpu-builder-amd64 3/5] RUN CMAKE_VERSION=3.22.1 GOLANG_VERSION=1.22.5 sh /rh_linux_deps.sh                                                                                0.0s# [  7%] Building CXX object common/CMakeFiles/build_info.dir/build-info.cpp.o
 => CACHED [cpu-builder-amd64 4/5] COPY --from=llm-code / /go/src/github.com/ollama/ollama/                                                                                0.0s# [  7%] Built target build_info
 => CACHED [cpu-builder-amd64 5/5] WORKDIR /go/src/github.com/ollama/ollama/llm/generate                                                                                0.0s# [  7%] Building HIP object ggml/src/CMakeFiles/ggml.dir/ggml-cuda/binbcast.cu.o
 => CACHED [static-build-amd64 1/1] RUN --mount=type=cache,target=/root/.ccache     OLLAMA_CPU_TARGET="static" bash gen_linux.sh                                                           0.0s# [ 10%] Building HIP object ggml/src/CMakeFiles/ggml.dir/ggml-cuda/clamp.cu.o
 => CACHED [container-build-amd64 1/3] WORKDIR /go/src/github.com/ollama/ollama                                                                                0.0s# [ 10%] Building HIP object ggml/src/CMakeFiles/ggml.dir/ggml-cuda/concat.cu.o
 => CACHED [container-build-amd64 2/3] COPY . .                                                                                [+] Building 1385.5s (37/37) FINISHED                                                                                docker:default
 => [internal] load .dockerignore                                                                                0.0s
 => => transferring context: 129B                                                                                0.0s
 => [internal] load build definition from Dockerfile                                                                                0.0s
 => => transferring dockerfile: 11.35kB                                                                                0.0s
 => [internal] load metadata for docker.io/library/ubuntu:22.04                                                                                0.3s
 => [internal] load metadata for docker.io/library/centos:7                                                                                0.3s
 => [internal] load metadata for docker.io/rocm/dev-centos-7:6.1.2-complete                                                                                0.9s
 => [internal] load build context                                                                                0.0s
 => => transferring context: 37.86kB                                                                                0.0s
 => CACHED [rocm-build-amd64 1/7] FROM docker.io/rocm/dev-centos-7:6.1.2-complete@sha256:d32024d55afc71bd8e9b9eb20835a369b9df69c1f1b0a386fe06d3d9d5e19293                          0.0s
 => CACHED [runtime-rocm 1/8] FROM docker.io/library/ubuntu:22.04@sha256:58b87898e82351c6cf9cf5b9f3c20257bb9e2dcf33af051e12ce532d7f94e3fe                                          0.0s
 => [cpu-builder-amd64 1/5] FROM docker.io/library/centos:7@sha256:be65f488b7764ad3638f236b7b515b3678369a5124c47b8d32916d6487418ea4                                                0.0s
 => CACHED [llm-code 1/3] COPY .git .git                                                                                0.0s
 => CACHED [llm-code 2/3] COPY .gitmodules .gitmodules                                                                                0.0s
 => CACHED [llm-code 3/3] COPY llm llm                                                                                0.0s
 => CACHED [cpu-builder-amd64 2/5] COPY ./scripts/rh_linux_deps.sh /                                                                                0.0s
 => CACHED [cpu-builder-amd64 3/5] RUN CMAKE_VERSION=3.22.1 GOLANG_VERSION=1.22.5 sh /rh_linux_deps.sh                                                                             0.0s
 => CACHED [cpu-builder-amd64 4/5] COPY --from=llm-code / /go/src/github.com/ollama/ollama/                                                                                0.0s
 => CACHED [cpu-builder-amd64 5/5] WORKDIR /go/src/github.com/ollama/ollama/llm/generate                                                                                0.0s
 => CACHED [static-build-amd64 1/1] RUN --mount=type=cache,target=/root/.ccache     OLLAMA_CPU_TARGET="static" bash gen_linux.sh                                                   0.0s
 => CACHED [container-build-amd64 1/3] WORKDIR /go/src/github.com/ollama/ollama                                                                                0.0s
 => CACHED [container-build-amd64 2/3] COPY . .                                                                                0.0s
 => CACHED [container-build-amd64 3/3] RUN --mount=type=cache,target=/root/.ccache     go build -trimpath -o dist/linux-amd64/bin/ollama .                                         0.0s
 => CACHED [cpu-build-amd64 1/1] RUN --mount=type=cache,target=/root/.ccache     OLLAMA_SKIP_STATIC_GENERATE=1 OLLAMA_CPU_TARGET="cpu" bash gen_linux.sh                           0.0s
 => [rocm-build-amd64 2/7] COPY ./scripts/rh_linux_deps.sh /                                                                                0.0s
 => CACHED [cpu_avx-build-amd64 1/1] RUN --mount=type=cache,target=/root/.ccache     OLLAMA_SKIP_STATIC_GENERATE=1 OLLAMA_CPU_TARGET="cpu_avx" bash gen_linux.sh                   0.0s
 => CACHED [cpu_avx2-build-amd64 1/1] RUN --mount=type=cache,target=/root/.ccache     OLLAMA_SKIP_STATIC_GENERATE=1 OLLAMA_CPU_TARGET="cpu_avx2" bash gen_linux.sh                 0.0s
 => [rocm-build-amd64 3/7] RUN CMAKE_VERSION=3.22.1 sh /rh_linux_deps.sh                                                                                97.0s
 => [rocm-build-amd64 4/7] COPY --from=llm-code / /go/src/github.com/ollama/ollama/                                                                                0.1s
 => [rocm-build-amd64 5/7] WORKDIR /go/src/github.com/ollama/ollama/llm/generate                                                                                0.0s
 => [rocm-build-amd64 6/7] RUN --mount=type=cache,target=/root/.ccache     OLLAMA_SKIP_STATIC_GENERATE=1 OLLAMA_SKIP_CPU_GENERATE=1 bash gen_linux.sh                           1237.3s
 => [rocm-build-amd64 7/7] RUN mkdir -p ../../dist/linux-amd64-rocm/lib/ollama &&     (cd /opt/rocm/lib && tar cf - rocblas/library) | (cd ../../dist/linux-amd64-rocm/lib/ollama  5.0s
 => [runtime-rocm 2/8] COPY --from=rocm-build-amd64 /go/src/github.com/ollama/ollama/dist/linux-amd64-rocm/lib/ /lib/                                                              6.9s
 => [runtime-rocm 3/8] RUN apt-get update &&     apt-get install -y ca-certificates &&     apt-get clean && rm -rf /var/lib/apt/lists/*                                            8.1s
 => [runtime-rocm 4/8] COPY --from=container-build-amd64 /go/src/github.com/ollama/ollama/dist/linux-amd64/bin/ /bin/                                                              0.3s
 => [runtime-rocm 5/8] COPY --from=cpu-build-amd64 /go/src/github.com/ollama/ollama/dist/linux-amd64/lib/ /lib/                                                                    0.0s
 => [runtime-rocm 6/8] COPY --from=cpu_avx-build-amd64 /go/src/github.com/ollama/ollama/dist/linux-amd64/lib/ /lib/                                                                0.0s
 => [runtime-rocm 7/8] COPY --from=cpu_avx2-build-amd64 /go/src/github.com/ollama/ollama/dist/linux-amd64/lib/ /lib/                                                               0.0s
 => [runtime-rocm 8/8] COPY --from=rocm-build-amd64 /go/src/github.com/ollama/ollama/dist/linux-amd64/lib/ /lib/                                                                   0.4s
 => exporting to image                                                                                16.3s
 => => exporting layers                                                                                16.3s
 => => writing image sha256:190ba0d38115356616090b4636436b16f82fb7310ed3c96e13b5fe0f0922ad9f                                                                                0.0s
 => => naming to docker.io/ollama/release:a7e070e-dirty-rocm                                                                                0.0s
snuupy@snuum780:~/ollama-GTT$ docker images
REPOSITORY                        TAG                   IMAGE ID       CREATED             SIZE
ollama/release                    a7e070e-dirty-rocm    190ba0d38115   43 seconds ago      8.41GB
ollama/release                    a7e070e-dirty-amd64   4b702dc2bc63   About an hour ago   3.41GB

@MaciejMogilany

EDIT:
ouch sorry didn't read your message with an understanding patch search for 6.10+ kernel, you have 6.9.10

Zabbly 6.9.9/10 or newer have the GTT patches applied (backported), but it doesn't matter as I am now on 6.10.12-zabbly+
I followed your latest instructions to build the binary and was able to get it working with gemma2:27b-instruct-q6_K which uses 22-23GB VRAM when loaded. I specifically picked this one because if it's not working, it would go over the 16GB VRAM limit I have set in BIOS. amdgpu_top shows 24GB GTT available (as half of 48GB = 24GB) and this setup has a total system memory of 64GB.
ollama       | time=2024-10-06T03:13:32.656Z level=DEBUG source=amd_linux.go:386 msg="amdgpu memory" gpu=0 total="23.4 GiB"
ollama       | time=2024-10-06T03:13:32.656Z level=DEBUG source=amd_linux.go:387 msg="amdgpu memory" gpu=0 available="23.4 GiB"

and it uses GTT no problem, with 22-23GB used out of 24GB GTT.
I have not yet tested a custom ttm but will do so soon
I am now trying to build the docker image, but the latest commit in https://github.com/ollama/ollama/blob/main/scripts/build_docker.sh (and subsequently https://github.com/MaciejMogilany/ollama/blob/main/scripts/build_docker.sh) both don't work when running OLLAMA_SKIP_MANIFEST_CREATE=1 sudo OLLAMA_SKIP_MANIFEST_CREATE=1 ./scripts/build_docker.sh @badsmoke nor just ./scripts/build_docker.sh since it was refactored to the Dockerfile
snuupy@snuum780:~/ollama-GTT$ OLLAMA_SKIP_MANIFEST_CREATE=1 sudo OLLAMA_SKIP_MANIFEST_CREATE=1 ./scripts/build_docker.sh
Building Ollama
VERSION=a7e070e-dirty
PLATFORM=linux/arm64,linux/amd64
Building ollama/ollama:a7e070e-dirty locally.  set PUSH=1 to push
[+] Building 0.0s (0/0)                                                                                docker:default
ERROR: docker exporter does not currently support exporting manifest lists

Looks like I need a docker registry? Can I not just build locally and commit to the local image repository? What is the command to build only the amd64-rocm image?
Either way I reverted to the previous build_docker.sh
and got
ollama  | time=2024-10-06T09:03:47.792Z level=DEBUG source=amd_linux.go:332 msg="AMD APU valid to use GTT memory"
ollama  | time=2024-10-06T09:03:47.792Z level=INFO source=amd_linux.go:366 msg="For best performance set APU VRAM carveout to <=2GiB"
ollama  | time=2024-10-06T09:03:47.792Z level=DEBUG source=amd_linux.go:386 msg="amdgpu memory" gpu=0 total="23.4 GiB"
ollama  | time=2024-10-06T09:03:47.792Z level=DEBUG source=amd_linux.go:387 msg="amdgpu memory" gpu=0 available="23.4 GiB"
ollama  | time=2024-10-06T09:03:47.792Z level=DEBUG source=amd_common.go:18 msg="evaluating potential rocm lib dir /usr/lib/ollama"
ollama  | time=2024-10-06T09:03:47.792Z level=DEBUG source=amd_common.go:61 msg="detected ROCM next to ollama executable /usr/lib/ollama"

and now ollama (and open web UI) both use GTT! :D :tada:

Having some trouble on my 8845hs (780m, 32 GB RAM) getting more than >2 GB of GTT memory. My BIOS has video memory set to auto and /etc/modprobe.d/ttm.conf is:
options ttm pages_limit=3145728
options ttm page_pool_size=3145728

which should give ~12 GG but admgpu_top says GTT: 1941 MiB.
Any ideas?

CMAKE_DEFS="-DLLAMA_HIP_UMA=ON

Where exactly do you add this. CMAKE_DEFS is at various places in the gen_linux.sh script.

If anyone can explain any benefits of -DLLAMA_HIP_UMA=ON on linux kernel 6.9.9+ please share. From my testing on recent kernel, only difference for end user is that GPU memory usage is not properly reported. This flag was a workaround to use managed memory like other hacks similar to force-host-alloction-APU. This was made when amd driver did not allow to load models to GTT.
https://github.com/ollama/ollama/issues/2637#issuecomment-2288238339

                                                  ,vvnnmmnn

I am now trying to build the docker image, but the latest commit in https://github.com/ollama/ollama/blob/main/scripts/build_docker.sh (and subsequently https://github.com/MaciejMogilany/ollama/blob/main/scripts/build_docker.sh) both

please use https://github.com/Maciej-Mogilany/ollama/blob/main/scripts/build_docker.sh instead https://github.com/MaciejMogilany/ollama/blob/main/scripts/build_docker.sh
When creating this PR somehow i get two repositories. One with dash is up to date (this PR)
I added extra logging that you can see here
'''
ollama  | time=2024-10-06T09:03:47.792Z level=INFO source=amd_linux.go:366 msg="For best performance set APU VRAM carveout to <=2GiB"
'''
setting 16 will cut that memory from general use if you do not need this for some other use than LLM it is better to set UMA to auto  (I use 2GiB) this will free 14 GiB RAM for general use and add 7GiB for GTT by default.

Sorry. Just tried to compile but failed.
$ git clone https://github.com/Maciej-Mogilany/ollama.git
...
$ git checkout AMD_APU_GTT_memory
...
$ git submodule init
$ git submodule update --recursive
fatal: transport 'file' not allowed
fatal: Fetched in submodule path 'llm/llama.cpp', but it did not contain 8962422b1c6f9b8b15f5aeaea42600bcc2d44177. Direct fetching of that commit failed.

 This is for those who have the same problem.
Problem

I'm cloning the upstream ollama repository then add this as additional repository.

The git submodule update step would fail:
Cloning into '/usr/local/src/ollama/llm/llama.cpp'...
fatal: transport 'file' not allowed
fatal: Fetched in submodule path 'llm/llama.cpp', but it did not contain 8962422b1c6f9b8b15f5aeaea42600bcc2d44177. Direct fetching of that commit failed.

The problem is that in .gitmodules, the submodule is set to be shallow cloing:
[submodule "llama.cpp"]
        path = llm/llama.cpp
        url = https://github.com/ggerganov/llama.cpp.git
        shallow = true

which prohibit the patching script from working properly.

 This is for those who have the same problem.
Problem
1. I'm cloning the upstream ollama repository then add this as additional repository.

2. The `git submodule update` step would fail:
   ```
   Cloning into '/usr/local/src/ollama/llm/llama.cpp'...
   fatal: transport 'file' not allowed
   fatal: Fetched in submodule path 'llm/llama.cpp', but it did not contain 8962422b1c6f9b8b15f5aeaea42600bcc2d44177.  The problem is that in .gitmodules, the submodule is set to be shallow cloing:
   ```ini
   [submodule "llama.cpp"]
           path = llm/llama.cpp
           url = https://github.com/ggerganov/llama.cpp.git
           shallow = true
   ```

   which prohibit switching branch from working properly.

Solution
Instead, clone the actual feature branch recursively:
git clone -b AMD_APU_GTT_memory --recurse-submodules https://github.com/Maciej-Mogilany/ollama.git

Then do the remaining setup:
export OLLAMA_CUSTOM_CPU_DEFS="-DLLAMA_HIP_UMA=ON"
export AMDGPU_TARGETS="gfx1030"  # as shown by rocminfo
export HSA_OVERRIDE_GFX_VERSION=10.3.0
export OLLAMA_SKIP_ONEAPI_GENERATE=1  # the build otherwise depended on intel oneapi which I don't want
go generate ./...
go build

And run the ollama binary just built:
./ollama serve

For any one wanting to test, please use build instructions from first post.

@Djip007: I've tested this both on a Ryzen 7840U + Raedon 780m laptop and a Ryzen 8700g (builtin Raedon 780m) system. Both works perfectly fine with allocating GTT memory. Would you help to please review again to see what needs to be done before merging to main?

@robertvazan @yookoala @Snuupy @ttsiodras @JenaMischk
have been testing for a few days with flags below, without gpu hang. If you find any workflow that hang APU please try these and report if it gives any improvement. You must be on in sync sith this repot to incorporate commit https://github.com/ollama/ollama/pull/7174 that affect APU
export OLLAMA_MAX_LOADED_MODELS=1
export OLLAMA_NUM_PARALLEL=1
For convenience, I replaced ollama bin file with compiled from this branch and added these flags to ollama service here's my file
To edit
sudo systemctl edit ollama.service
sudo systemctl daemon-reload
sudo systemctl restart ollama

my ollama.service file this is for 780m
### Editing /etc/systemd/system/ollama.service.d/override.conf
### Anything between here and the comment below will become the contents of the drop-in file

[Service]
Environment="HSA_OVERRIDE_GFX_VERSION=11.0.1"
Environment="OLLAMA_HOST=<IP_ADDRESS>"
Environment="OLLAMA_MAX_LOADED_MODELS=1"
Environment="OLLAMA_NUM_PARALLEL=1"

### Edits below this comment will be discarded

### /etc/systemd/system/ollama.service
# [Unit]
# Description=Ollama Service
# After=network-online.target
#
# [Service]
# ExecStart=/usr/local/bin/ollama serve
# User=ollama
# Group=ollama
# Restart=always
# RestartSec=3
# Environment="PATH=/usr/local/bin:/usr/bin:/usr/local/sbin:/opt/cuda/bin:/opt/cuda/nsight_compute:/opt/cuda/nsight_systems/bin:/>
#
# [Install]
# WantedBy=default.target

ollama 3.14 require rewriting of this path, so this branch is 3.13+ granite dense and MOE LLM don't work on it jet.

@robertvazan @yookoala @Snuupy @ttsiodras @JenaMischk
have been testing for a few days with flags below, without gpu hang. If you find any workflow that hang APU please try these and report if it gives any improvement. You must be on in sync sith this repot to incorporate commit https://github.com/ollama/ollama/pull/7174 that affect APU
export OLLAMA_MAX_LOADED_MODELS=1
export OLLAMA_NUM_PARALLEL=1
For convenience, I replaced ollama bin file with compiled from this branch and added these flags to ollama service here's my file
To edit
sudo systemctl edit ollama.service
sudo systemctl daemon-reload
sudo systemctl restart ollama

my ollama.service file this is for 780m
### Editing /etc/systemd/system/ollama.service.d/override.conf
### Anything between here and the comment below will become the contents of the drop-in file

[Service]
Environment="HSA_OVERRIDE_GFX_VERSION=11.0.1"
Environment="OLLAMA_HOST=<IP_ADDRESS>"
Environment="OLLAMA_MAX_LOADED_MODELS=1"
Environment="OLLAMA_NUM_PARALLEL=1"

### Edits below this comment will be discarded

### /etc/systemd/system/ollama.service
# [Unit]
# Description=Ollama Service
# After=network-online.target
#
# [Service]
# ExecStart=/usr/local/bin/ollama serve
# User=ollama
# Group=ollama
# Restart=always
# RestartSec=3
# Environment="PATH=/usr/local/bin:/usr/bin:/usr/local/sbin:/opt/cuda/bin:/opt/cuda/nsight_compute:/opt/cuda/nsight_systems/bin:/>
#
# [Install]
# WantedBy=default.target

ollama 3.14 require rewriting of this path, so this branch is 3.13+ granite dense and MOE LLM don't work on it jet.

Thanks.
For the record, I only have this line in my service file and I have not experienced any hanging issue.
Environment="HSA_OVERRIDE_GFX_VERSION=11.0.1"

i am running your branch in a docker container.  it works well with text only models ( llama3.2:3b,  codestral:22b) but i experience hanging APU when using multimodal model llava:13b
AMD Ryzen 7 7840U w/ Radeon  780M Graphics

I was able to load dolphin-mixtral:8x7b (26gb) and it seems to be running well. The GPU gets quite toasty while it's running but then cools down quickly. On the charger, I'm seeing it pull around 34W.
amdgpu_top shows:

This is very exciting work. 7640u, 760M APU with 96Gb of DDR5 5600

I was able to load dolphin-mixtral:8x7b (26gb) and it seems to be running well. The GPU gets quite toasty while it's running but then cools down quickly. On the charger, I'm seeing it pull around 34W.

OK, I managed to freeze my entire UI and had to cold start the system. Looks like plasmashell crashed. The system was actually still running because I was able to sleep by shutting the lid, but without a working UI the system was unusable.
Oct 25 23:24:07 framework13-amd systemd-coredump[188327]: [🡕] Process 1284 (plasmashell) of user 1000 dumped core.

                                                          Stack trace of thread 1344:
                                                          #0  0x0000777e2fea53f4 n/a (libc.so.6 + 0x963f4)
                                                          #1  0x0000777e2fe4c120 raise (libc.so.6 + 0x3d120)
                                                          #2  0x0000777e32e2a2a1 _ZN6KCrash19defaultCrashHandlerEi (libKF6Crash.so.6 + 0x62a1)
                                                          #3  0x0000777e2fe4c1d0 n/a (libc.so.6 + 0x3d1d0)
                                                          #4  0x0000777e2fea53f4 n/a (libc.so.6 + 0x963f4)
                                                          #5  0x0000777e2fe4c120 raise (libc.so.6 + 0x3d120)
                                                          #6  0x0000777e2fe334c3 abort (libc.so.6 + 0x244c3)
                                                          #7  0x0000777e217b0083 n/a (libgallium-24.2.5-arch1.1.so + 0x9b0083)
                                                          #8  0x0000777e217b3573 n/a (libgallium-24.2.5-arch1.1.so + 0x9b3573)
                                                          #9  0x0000777e20eab854 n/a (libgallium-24.2.5-arch1.1.so + 0xab854)
                                                          #10 0x0000777e20ecec9d n/a (libgallium-24.2.5-arch1.1.so + 0xcec9d)
                                                          #11 0x0000777e2fea339d n/a (libc.so.6 + 0x9439d)
                                                          #12 0x0000777e2ff2849c n/a (libc.so.6 + 0x11949c)

                                                          Stack trace of thread 1284:
                                                          #0  0x0000777e2fe9fa19 n/a (libc.so.6 + 0x90a19)
                                                          #1  0x0000777e2fea2479 pthread_cond_wait (libc.so.6 + 0x93479)
                                                          #2  0x0000777e306dd030 _ZN14QWaitCondition4waitEP6QMutex14QDeadlineTimer (libQt6Core.so.6 + 0x2dd030)
                                                          #3  0x0000777e32048bbd n/a (libQt6Quick.so.6 + 0x448bbd)
                                                          #4  0x0000777e31e5e8da _ZN12QQuickWindow5eventEP6QEvent (libQt6Quick.so.6 + 0x25e8da)
                                                          #5  0x0000777e324fe31a _ZN19QApplicationPrivate13notify_helperEP7QObjectP6QEvent (libQt6Widgets.so.6 + 0xfe31a)
                                                          #6  0x0000777e305585a8 _ZN16QCoreApplication15notifyInternal2EP7QObjectP6QEvent (libQt6Core.so.6 + 0x1585a8)
                                                          #7  0x0000777e30bcef8e _ZN15QPlatformWindow20deliverUpdateRequestEv (libQt6Gui.so.6 + 0x1cef8e)
                                                          #8  0x0000777e305a348a _ZN7QObject5eventEP6QEvent (libQt6Core.so.6 + 0x1a348a)
                                                          #9  0x0000777e324fe31a _ZN19QApplicationPrivate13notify_helperEP7QObjectP6QEvent (libQt6Widgets.so.6 + 0xfe31a)
                                                          #10 0x0000777e305585a8 _ZN16QCoreApplication15notifyInternal2EP7QObjectP6QEvent (libQt6Core.so.6 + 0x1585a8)
                                                          #11 0x0000777e30559035 _ZN23QCoreApplicationPrivate16sendPostedEventsEP7QObjectiP11QThreadData (libQt6Core.so.6 >
                                                          #12 0x0000777e307c23fc n/a (libQt6Core.so.6 + 0x3c23fc)
                                                          #13 0x0000777e2efb2559 n/a (libglib-2.0.so.0 + 0x5d559)
                                                          #14 0x0000777e2f015157 n/a (libglib-2.0.so.0 + 0xc0157)
                                                          #15 0x0000777e2efb1a55 g_main_context_iteration (libglib-2.0.so.0 + 0x5ca55)
                                                          #16 0x0000777e307bf71d _ZN20QEventDispatcherGlib13processEventsE6QFlagsIN10QEventLoop17ProcessEventsFlagEE (libQ>
                                                          #17 0x0000777e30564566 _ZN10QEventLoop4execE6QFlagsINS_17ProcessEventsFlagEE (libQt6Core.so.6 + 0x164566)
                                                          #18 0x0000777e30559a2f _ZN16QCoreApplication4execEv (libQt6Core.so.6 + 0x159a2f)
                                                          #19 0x00005e0104b4117f n/a (plasmashell + 0x2817f)
                                                          #20 0x0000777e2fe34e08 n/a (libc.so.6 + 0x25e08)
                                                          #21 0x0000777e2fe34ecc __libc_start_main (libc.so.6 + 0x25ecc)
                                                          #22 0x00005e0104b41695 n/a (plasmashell + 0x28695)

@MaciejMogilany where can i find APUs for all AMG GPU? I have radeon rx570

BTW, why is there a hardcoded list of iGPUs anyway? Are some iGPUs broken even though ROCm recognizes them? If that is so, there should be an environment variable that forces iGPU use, so that new models can be easily tested.

I think because he don't now an other way to know it is an iGPU ... If anyone has an idea how to do it differently I'm sure it may be of interest. If I correct read amd_kfd driver, the change have been made for all APU.

hardcoded list of APU like in current implementation. With Ryzen 5600G we should have all usable APU-s including newest RDNA3+
ollama way, check against graphic memory if bellow 1G assume it's integrated. It's not reliable as user can change that
environmental variable, user need to remember that. Ollama docs discourage adding new variable.
maybe check against CPU name. APU always have Graphics in cpu name. I have "AMD Ryzen 7 7840HS w/ Radeon 780M Graphics"

Something like this should work.
package main

import (
	"bufio"
	"fmt"
	"os"
	"strings"
)

func main() {
	// Get CPU model name
	cpuModel := getCPUModel()

	// Check for "AMD" and "Graphics" in CPU model name
	if strings.Contains(cpuModel, "AMD") && strings.Contains(cpuModel, "Graphics") {
		fmt.Println("AMD APU detected:")
		fmt.Println("CPU Model:", cpuModel)
	} else if strings.Contains(cpuModel, "AMD") {
		fmt.Println("AMD CPU without integrated graphics detected:")
		fmt.Println("CPU Model:", cpuModel)
	} else {
		fmt.Println("Non-AMD CPU detected:")
		fmt.Println("CPU Model:", cpuModel)
	}
}

func getCPUModel() string {
	file, err := os.Open("/proc/cpuinfo")
	if err != nil {
		fmt.Println("Error opening /proc/cpuinfo:", err)
		return ""
	}
	defer file.Close()

	scanner := bufio.NewScanner(file)
	for scanner.Scan() {
		line := scanner.Text()
		if strings.HasPrefix(line, "model name") {
			return strings.TrimSpace(strings.Split(line, ":")[1])
		}
	}
	return ""
}

If the Vega iGPUs from Ryzen 5000G are supported, would it be possible to also get the first-gen Vega iGPUs (Raven Ridge, Ryzen 3 2200G and Ryzen 5 2400G) in the list as well? They should be identical to the Cezanne parts other than clock speed and SM count.

How do I run this locally to test?

I built the docker container, but it doesn't seem like it's working properly?
I ran ./scripts/build_docker.sh, I'm running the container with this command:
docker run --device /dev/kfd --device /dev/dri -e HSA_OVERRIDE_GFX_VERSION="11.0.1" -e OLLAMA_MAX_LOADED_MODELS=1 -e OLLAMA_NUM_PARALLEL=1 -v ~/.ollama:/root/.ollama -p 11434:11434 ollama/ollama:b94b77e-dirty-rocm

I do see this message which seems promising:
time=2024-11-05T15:46:38.955Z level=INFO source=routes.go:1205 msg="Listening on [::]:11434 (version 0.0.0)"
time=2024-11-05T15:46:38.957Z level=INFO source=common.go:135 msg="extracting embedded files" dir=/tmp/ollama3747493948/runners
time=2024-11-05T15:46:38.984Z level=INFO source=common.go:49 msg="Dynamic LLM libraries" runners="[cpu cpu_avx cpu_avx2]"
time=2024-11-05T15:46:38.985Z level=INFO source=gpu.go:221 msg="looking for compatible GPUs"
time=2024-11-05T15:46:38.986Z level=WARN source=amd_linux.go:132 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-11-05T15:46:38.989Z level=INFO source=amd_linux.go:489 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=11.0.1
time=2024-11-05T15:46:38.992Z level=WARN source=amd_linux.go:132 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-11-05T15:46:38.994Z level=INFO source=amd_linux.go:489 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=11.0.1
time=2024-11-05T15:46:38.994Z level=INFO source=types.go:113 msg="inference compute" id=0 library=rocm variant="" compute=gfx1103 driver=0.0 name=1002:15bf total="30.3 GiB" available="29.9 GiB"

And when I load a model it seems to be loading into GTT:
time=2024-11-05T15:47:46.273Z level=WARN source=amd_linux.go:132 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/
module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-11-05T15:47:46.276Z level=INFO source=amd_linux.go:489 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=11.0.1
time=2024-11-05T15:47:46.289Z level=INFO source=sched.go:714 msg="new model will fit in available VRAM in single GPU, loading" model=/root/.ollama/models/blobs/sha256-32ea3525913b548cb473
a2db319c2fcdb70fe552dd1f8a52d6cb0e76be12a60d gpu=0 parallel=1 available=32143392768 required="15.5 GiB"
time=2024-11-05T15:47:46.289Z level=WARN source=amd_linux.go:132 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/
module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-11-05T15:47:46.292Z level=INFO source=amd_linux.go:489 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=11.0.1
time=2024-11-05T15:47:46.292Z level=INFO source=server.go:108 msg="system memory" total="30.3 GiB" free="14.1 GiB" free_swap="5.3 GiB"
time=2024-11-05T15:47:46.293Z level=INFO source=memory.go:326 msg="offload to rocm" layers.requested=-1 layers.model=57 layers.offload=57 layers.split="" memory.available="[29.9 GiB]" mem
ory.gpu_overhead="0 B" memory.required.full="15.5 GiB" memory.required.partial="15.5 GiB" memory.required.kv="448.0 MiB" memory.required.allocations="[15.5 GiB]" memory.weights.total="14.
4 GiB" memory.weights.repeating="14.3 GiB" memory.weights.nonrepeating="157.5 MiB" memory.graph.full="244.0 MiB" memory.graph.partial="256.3 MiB"
time=2024-11-05T15:47:46.297Z level=INFO source=server.go:401 msg="starting llama server" cmd="/tmp/ollama3747493948/runners/cpu_avx2/ollama_llama_server --model /root/.ollama/models/blob
s/sha256-32ea3525913b548cb473a2db319c2fcdb70fe552dd1f8a52d6cb0e76be12a60d --ctx-size 2048 --batch-size 512 --embedding --log-disable --n-gpu-layers 57 --no-mmap --parallel 1 --port 42065"
time=2024-11-05T15:47:46.298Z level=INFO source=sched.go:449 msg="loaded runners" count=1
time=2024-11-05T15:47:46.298Z level=INFO source=server.go:600 msg="waiting for llama runner to start responding"
time=2024-11-05T15:47:46.299Z level=INFO source=server.go:634 msg="waiting for server to become available" status="llm server error"
WARN [server_params_parse] Not compiled with GPU offload support, --n-gpu-layers option will be ignored. See main README.md for information on enabling GPU BLAS support | n_gpu_layers=-1
tid="139637444553664" timestamp=1730821666
INFO [main] starting c++ runner | tid="139637444553664" timestamp=1730821666
INFO [main] build info | build=3670 commit="cb08e0e5" tid="139637444553664" timestamp=1730821666
INFO [main] system info | n_threads=8 n_threads_batch=8 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 1 | NEON
= 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | " tid="139637444553664" timestamp=1730
821666 total_threads=16
INFO [main] HTTP server listening | hostname="<IP_ADDRESS>" n_threads_http="15" port="42065" tid="139637444553664" timestamp=1730821666
llama_model_loader: loaded meta data with 25 key-value pairs and 507 tensors from /root/.ollama/models/blobs/sha256-32ea3525913b548cb473a2db319c2fcdb70fe552dd1f8a52d6cb0e76be12a60d (versi
on GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.name str              = Codestral-22B-v0.1
llama_model_loader: - kv   2:                          llama.block_count u32              = 56
llama_model_loader: - kv   3:                       llama.context_length u32              = 32768
llama_model_loader: - kv   4:                     llama.embedding_length u32              = 6144
llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 16384
llama_model_loader: - kv   6:                 llama.attention.head_count u32              = 48
llama_model_loader: - kv   7:              llama.attention.head_count_kv u32              = 8
llama_model_loader: - kv   8:                       llama.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  10:                          general.file_type u32              = 16
llama_model_loader: - kv  11:                           llama.vocab_size u32              = 32768
llama_model_loader: - kv  12:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv  13:            tokenizer.ggml.add_space_prefix bool             = true
llama_model_loader: - kv  14:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  15:                         tokenizer.ggml.pre str              = default
llama_model_loader: - kv  16:                      tokenizer.ggml.tokens arr[str,32768]   = ["<unk>", "<s>", "</s>", "[INST]", "[...
llama_model_loader: - kv  17:                      tokenizer.ggml.scores arr[f32,32768]   = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv  18:                  tokenizer.ggml.token_type arr[i32,32768]   = [2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ...
llama_model_loader: - kv  19:                tokenizer.ggml.bos_token_id u32              = 1
llama_model_loader: - kv  20:                tokenizer.ggml.eos_token_id u32              = 2
llama_model_loader: - kv  21:            tokenizer.ggml.unknown_token_id u32              = 0
llama_model_loader: - kv  22:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  23:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  24:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:  113 tensors
llama_model_loader: - type q5_K:  393 tensors
llama_model_loader: - type q6_K:    1 tensors
llm_load_vocab: special tokens cache size = 771
llm_load_vocab: token to piece cache size = 0.1731 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = llama
llm_load_print_meta: vocab type       = SPM
llm_load_print_meta: n_vocab          = 32768
llm_load_print_meta: n_merges         = 0
llm_load_print_meta: vocab_only       = 0
llm_load_print_meta: n_ctx_train      = 32768
llm_load_print_meta: n_embd           = 6144
llm_load_print_meta: n_layer          = 56
llm_load_print_meta: n_head           = 48
llm_load_print_meta: n_head_kv        = 8
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_swa            = 0
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 6
llm_load_print_meta: n_embd_k_gqa     = 1024
llm_load_print_meta: n_embd_v_gqa     = 1024
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 16384
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 1000000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn  = 32768
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: ssm_dt_b_c_rms   = 0
llm_load_print_meta: model type       = ?B
llm_load_print_meta: model ftype      = Q5_K - Small
llm_load_print_meta: model params     = 22.25 B
llm_load_print_meta: model size       = 14.27 GiB (5.51 BPW)
llm_load_print_meta: general.name     = Codestral-22B-v0.1
llm_load_print_meta: BOS token        = 1 '<s>'
llm_load_print_meta: EOS token        = 2 '</s>'
llm_load_print_meta: UNK token        = 0 '<unk>'
llm_load_print_meta: LF token         = 781 '<0x0A>'
llm_load_print_meta: max token length = 48
llm_load_tensors: ggml ctx size =    0.24 MiB
llm_load_tensors:        CPU buffer size = 14614.15 MiB
time=2024-11-05T15:47:46.550Z level=INFO source=server.go:634 msg="waiting for server to become available" status="llm server loading model"
llama_new_context_with_model: n_ctx      = 2048
llama_new_context_with_model: n_batch    = 512
llama_new_context_with_model: n_ubatch   = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base  = 1000000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init:        CPU KV buffer size =   448.00 MiB
llama_new_context_with_model: KV self size  =  448.00 MiB, K (f16):  224.00 MiB, V (f16):  224.00 MiB
llama_new_context_with_model:        CPU  output buffer size =     0.15 MiB
llama_new_context_with_model:        CPU compute buffer size =   244.01 MiB
llama_new_context_with_model: graph nodes  = 1798
llama_new_context_with_model: graph splits = 1
INFO [main] model loaded | tid="139637444553664" timestamp=1730821680
time=2024-11-05T15:48:00.364Z level=INFO source=server.go:639 msg="llama runner started in 14.07 seconds"

But when I interact with it, it just runs on my CPU. Never once touches my GPU:

Is there something obvious I'm doing wrong?

Arch just updated all the rocm stuff and it no longer works :(
[2024-11-07T14:14:41-0800] [ALPM] upgraded rocm-core (6.0.2-2 -> 6.2.2-1)
[2024-11-07T14:14:52-0800] [ALPM] upgraded rocm-llvm (6.0.2-1 -> 6.2.2-2)
[2024-11-07T14:14:52-0800] [ALPM] upgraded rocm-device-libs (6.0.2-1 -> 6.2.2-2)
[2024-11-07T14:14:52-0800] [ALPM] upgraded rocminfo (6.0.2-1 -> 6.2.2-1)
[2024-11-07T14:15:11-0800] [ALPM] upgraded rocm-opencl-runtime (6.0.2-1 -> 6.2.2-1)
[2024-11-07T14:15:11-0800] [ALPM] upgraded rocm-clang-ocl (6.0.2-1 -> 6.1.2-1)
[2024-11-07T14:15:14-0800] [ALPM] upgraded ollama-rocm (0.3.12-4 -> 0.3.12-5)
[2024-11-07T14:15:15-0800] [ALPM] upgraded rocm-smi-lib (6.0.2-1 -> 6.2.2-1)
[2024-11-07T14:15:16-0800] [ALPM] upgraded rocm-cmake (6.0.2-1 -> 6.2.2-1)
[2024-11-07T14:15:16-0800] [ALPM] upgraded rocm-language-runtime (6.0.2-1 -> 6.2.2-1)
[2024-11-07T14:15:16-0800] [ALPM] upgraded rocm-hip-runtime (6.0.2-1 -> 6.2.2-1)
[2024-11-07T14:15:16-0800] [ALPM] upgraded rocm-hip-libraries (6.0.2-1 -> 6.2.2-1)
[2024-11-07T14:15:16-0800] [ALPM] upgraded rocm-ml-libraries (6.0.2-1 -> 6.2.2-1)

I'm now seeing this error message, even after rebuilding the code
Error: llama runner process has terminated: CUDA error: CUBLAS_STATUS_INTERNAL_ERROR
  current device: 0, in function ggml_cuda_mul_mat_batched_cublas at /build/ollama-rocm/src/ollama/llm/llama.cpp/ggml/src/ggml-cuda.cu:1890
  hipblasGemmBatchedEx(ctx.cublas_handle(), HIPBLAS_OP_T, HIPBLAS_OP_N, ne01, ne11, ne10, alpha, (const void **) (ptrs_src.get() + 0*ne23), HIPBLAS_R_16F, nb01/nb00, (const void **) (ptrs_src.get() + 1*ne23), HIPBLAS_R_16F, nb11/nb10, beta, ( void **) (ptrs_dst.get() + 0*ne23), cu_data_type, ne01, ne23, cu_compute_type, HIPBLAS_GEMM_DEFAULT)
/build/ollama-rocm/src/ollama/llm/llama.cpp/ggml/src/ggml-cuda.cu:102: CUDA error

I guess cuda changed too much

Arch just updated all the rocm stuff and it no longer works :(
[2024-11-07T14:14:41-0800] [ALPM] upgraded rocm-core (6.0.2-2 -> 6.2.2-1)
[2024-11-07T14:14:52-0800] [ALPM] upgraded rocm-llvm (6.0.2-1 -> 6.2.2-2)
[2024-11-07T14:14:52-0800] [ALPM] upgraded rocm-device-libs (6.0.2-1 -> 6.2.2-2)
[2024-11-07T14:14:52-0800] [ALPM] upgraded rocminfo (6.0.2-1 -> 6.2.2-1)
[2024-11-07T14:15:11-0800] [ALPM] upgraded rocm-opencl-runtime (6.0.2-1 -> 6.2.2-1)
[2024-11-07T14:15:11-0800] [ALPM] upgraded rocm-clang-ocl (6.0.2-1 -> 6.1.2-1)
[2024-11-07T14:15:14-0800] [ALPM] upgraded ollama-rocm (0.3.12-4 -> 0.3.12-5)
[2024-11-07T14:15:15-0800] [ALPM] upgraded rocm-smi-lib (6.0.2-1 -> 6.2.2-1)
[2024-11-07T14:15:16-0800] [ALPM] upgraded rocm-cmake (6.0.2-1 -> 6.2.2-1)
[2024-11-07T14:15:16-0800] [ALPM] upgraded rocm-language-runtime (6.0.2-1 -> 6.2.2-1)
[2024-11-07T14:15:16-0800] [ALPM] upgraded rocm-hip-runtime (6.0.2-1 -> 6.2.2-1)
[2024-11-07T14:15:16-0800] [ALPM] upgraded rocm-hip-libraries (6.0.2-1 -> 6.2.2-1)
[2024-11-07T14:15:16-0800] [ALPM] upgraded rocm-ml-libraries (6.0.2-1 -> 6.2.2-1)

I'm now seeing this error message, even after rebuilding the code
Error: llama runner process has terminated: CUDA error: CUBLAS_STATUS_INTERNAL_ERROR
  current device: 0, in function ggml_cuda_mul_mat_batched_cublas at /build/ollama-rocm/src/ollama/llm/llama.cpp/ggml/src/ggml-cuda.cu:1890
  hipblasGemmBatchedEx(ctx.cublas_handle(), HIPBLAS_OP_T, HIPBLAS_OP_N, ne01, ne11, ne10, alpha, (const void **) (ptrs_src.get() + 0*ne23), HIPBLAS_R_16F, nb01/nb00, (const void **) (ptrs_src.get() + 1*ne23), HIPBLAS_R_16F, nb11/nb10, beta, ( void **) (ptrs_dst.get() + 0*ne23), cu_data_type, ne01, ne23, cu_compute_type, HIPBLAS_GEMM_DEFAULT)
/build/ollama-rocm/src/ollama/llm/llama.cpp/ggml/src/ggml-cuda.cu:102: CUDA error

I guess cuda changed too much

use opencl-amd opencl-amd-dev packages they are newer and works

Arch just updated all the rocm stuff and it no longer works :(
[2024-11-07T14:14:41-0800] [ALPM] upgraded rocm-core (6.0.2-2 -> 6.2.2-1)
[2024-11-07T14:14:52-0800] [ALPM] upgraded rocm-llvm (6.0.2-1 -> 6.2.2-2)
[2024-11-07T14:14:52-0800] [ALPM] upgraded rocm-device-libs (6.0.2-1 -> 6.2.2-2)
[2024-11-07T14:14:52-0800] [ALPM] upgraded rocminfo (6.0.2-1 -> 6.2.2-1)
[2024-11-07T14:15:11-0800] [ALPM] upgraded rocm-opencl-runtime (6.0.2-1 -> 6.2.2-1)
[2024-11-07T14:15:11-0800] [ALPM] upgraded rocm-clang-ocl (6.0.2-1 -> 6.1.2-1)
[2024-11-07T14:15:14-0800] [ALPM] upgraded ollama-rocm (0.3.12-4 -> 0.3.12-5)
[2024-11-07T14:15:15-0800] [ALPM] upgraded rocm-smi-lib (6.0.2-1 -> 6.2.2-1)
[2024-11-07T14:15:16-0800] [ALPM] upgraded rocm-cmake (6.0.2-1 -> 6.2.2-1)
[2024-11-07T14:15:16-0800] [ALPM] upgraded rocm-language-runtime (6.0.2-1 -> 6.2.2-1)
[2024-11-07T14:15:16-0800] [ALPM] upgraded rocm-hip-runtime (6.0.2-1 -> 6.2.2-1)
[2024-11-07T14:15:16-0800] [ALPM] upgraded rocm-hip-libraries (6.0.2-1 -> 6.2.2-1)
[2024-11-07T14:15:16-0800] [ALPM] upgraded rocm-ml-libraries (6.0.2-1 -> 6.2.2-1)

I'm now seeing this error message, even after rebuilding the code
Error: llama runner process has terminated: CUDA error: CUBLAS_STATUS_INTERNAL_ERROR
  current device: 0, in function ggml_cuda_mul_mat_batched_cublas at /build/ollama-rocm/src/ollama/llm/llama.cpp/ggml/src/ggml-cuda.cu:1890
  hipblasGemmBatchedEx(ctx.cublas_handle(), HIPBLAS_OP_T, HIPBLAS_OP_N, ne01, ne11, ne10, alpha, (const void **) (ptrs_src.get() + 0*ne23), HIPBLAS_R_16F, nb01/nb00, (const void **) (ptrs_src.get() + 1*ne23), HIPBLAS_R_16F, nb11/nb10, beta, ( void **) (ptrs_dst.get() + 0*ne23), cu_data_type, ne01, ne23, cu_compute_type, HIPBLAS_GEMM_DEFAULT)
/build/ollama-rocm/src/ollama/llm/llama.cpp/ggml/src/ggml-cuda.cu:102: CUDA error

I guess cuda changed too much

use opencl-amd opencl-amd-dev packages they are newer and works

Hi,
I have the following installed:
extra/rocm-clang-ocl 6.1.2-1 [installed]
OpenCL compilation with clang compiler
extra/rocm-opencl-runtime 6.2.2-1 [installed]
OpenCL implementation for AMD
extra/rocm-opencl-sdk 6.2.2-1 [installed]
Develop OpenCL-based applications for AMD platforms
What do I need to do to make ollama use opencl instead of cuda? I've been hunting through the documentation but it's not clear
time=2024-11-08T09:47:24.204-08:00 level=INFO source=types.go:113 msg="inference compute" id=0 library=rocm variant="" compute=gfx1103 driver=0.0 name=1002:15bf total="45.1 GiB" available="44.7 GiB"

It seems to find the APU and the GTT RAM. Do I need to rebuild the binaries?
 You will have to adjust AMDGPU_TARGETS and HSA_OVERRIDE_GFX_VERSION. Not sure whether those OLLAMA_SKIP_* variables actually do anything as I saw CUDA compiling anyway.
The good news is that Ollama built in this way works, allocates models in GTT memory, and uses the iGPU for compute, which I have checked in radeontop. The bad news is that it is slower than CPU :-( I have used Ollama with BIOS-allocated VRAM before Linux 6.10 and back then iGPU was clearly faster than the CPU. It's interesting that performance drops sharply as context size increases, something I do not observe with CPU and I don't remember seeing it with BIOS-allocated VRAM, which makes me think there must be some unhealthy interaction between attention kernels and GTT memory. Maybe the context is not read sequentially and the iGPU runs out of some sort of page table cache? Or maybe I just have an old iGPU with poor GTT performance since nobody else has mentioned this.

Here are the complete test results on Ryzen 5600G (RX Vega 7):

With 8B llama3.1, GTT ROCm runs at 33/3.7 t/s (prompt/generate, tested with 1.5K prompt tokens and 1K generated tokens)  while CPU runs at 19/7.3 t/s. That means there's a bug killing generation speed with ROCm.
With 3B llama3.2, GTT ROCm runs at 16/6.6 t/s while CPU runs at 36/14 t/s. Prompt processing with ROCm is bizarrely 2x slower instead of 2x faster compared to the 8B model. This is not a benchmark glitch. It's reproducible.
I get about the same results with official container images when I set up dummy VRAM allocation in BIOS, so these results cannot be caused by misconfigured build.

Before kernel 6.10, I was getting up to 40/10 t/s. Now the performance is worse than CPU and weirdly inconsistent. It might be just bitrot in RX Vega 7 support though. Maybe other iGPUs perform better.

9.0.0

I get a CUDA error when I try running after doing that build:
  current device: 0, in function stream at /go/src/github.com/ollama/ollama/llm/llama.cpp/ggml/src/ggml-cuda/common.cuh:632
  hipStreamCreateWithFlags(&streams[device][stream], 0x01)
/go/src/github.com/ollama/ollama/llm/llama.cpp/ggml/src/ggml-cuda.cu:102: CUDA error```

@robertvazan you should be comparing between rocm GPU without this patch and rocm GPU with GTT patch running the same model, not between rocm GTT and cpu.

9.0.0

I get a CUDA error when I try running after doing that build:
  current device: 0, in function stream at /go/src/github.com/ollama/ollama/llm/llama.cpp/ggml/src/ggml-cuda/common.cuh:632
  hipStreamCreateWithFlags(&streams[device][stream], 0x01)
/go/src/github.com/ollama/ollama/llm/llama.cpp/ggml/src/ggml-cuda.cu:102: CUDA error```

You don't mention what hardware you have.

@robertvazan you should be comparing between rocm GPU without this patch and rocm GPU with GTT patch running the same model, not between rocm GTT and cpu.

I did. Performance is identical to current master. To test it, I have allocated dummy unused VRAM in BIOS to persuade master to run on iGPU. This is expected, because this branch merely enables use of iGPUs by not requiring preallocated VRAM. I also got a freeze with master that I did not get with this branch, but that could have been a one-off glitch or memory pressure.
The problem that I have now is that the performance is way worse than what it used to be before kernel 6.10 with preallocated VRAM. Not only is the performance poor, it is weirdly unreliable and drops rapidly with context size, which are issues I did not have before kernel 6.10. That's not a problem with the patch itself, of course, but between this and random freezes reported above, I can now understand why iGPUs might be a low priority for the maintainer. At least on my APU, it's best to default to CPU inference for now.
IMO, to make this PR mergeable, iGPU inference must be disabled by default and there must be a Markdown file with instructions, warnings, and a list of tested hardware, so that people don't end up frustrated after enabling this without due consideration.

You don't mention what hardware you have.

I have a Framework 13 - Ryzen 7 7840U w/ Radeon  780M Graphics

@m0ngr31

Radeon 780M Graphics

But that's a much newer GPU, probably gfx1103. No wonder gfx900 does not work for you. Adjust AMDGPU_TARGETS and HSA_OVERRIDE_GFX_VERSION.

@m0ngr31

Radeon 780M Graphics

But that's a much newer GPU, probably gfx1103. No wonder gfx900 does not work for you. Adjust AMDGPU_TARGETS and HSA_OVERRIDE_GFX_VERSION.

Yeah, I modified the commands...

 New problem emerges, linux  6.12 rc kernels give sig term.

I updated to your newest version and rebuilt the docker container. It seems to be working now! Not super speedy, but it's nice that I can run bigger models now

I updated to your newest version and rebuilt the docker container. It seems to be working now! Not super speedy, but it's nice that I can run bigger models now

Edit - Some performance stats:
mistral:7b-instruct-v0.3-q6_K:
total duration:       1m25.357598515s
load duration:        3.17605ms
prompt eval count:    41 token(s)
prompt eval duration: 30ms
prompt eval rate:     1366.67 tokens/s
eval count:           938 token(s)
eval duration:        1m25.323s
eval rate:            10.99 tokens/s

codestral:22b-v0.1-q5_K_S:
total duration:       4m9.005800646s
load duration:        15.532368ms
prompt eval count:    41 token(s)
prompt eval duration: 30ms
prompt eval rate:     1366.67 tokens/s
eval count:           894 token(s)
eval duration:        4m8.959s
eval rate:            3.59 tokens/s

Performance is limited by memory bandwidth, roughly can be calculated by memory bandwidth/model size
Zen4 CPU have 2 channels DDR5 official max supported speed is 5600MHz it gives max theoretical memory speed 89Gb/s.
codestral:22b-v0.1-q5_K_M takes 17Gib of memory and should give max 89/17 = 5.2 tokens/s
My Ryzen 7840HS with two 48 GB DDR5@5600MHz stick for codestral:22b-v0.1-q5_K_M  got 4.65 tokens/s eval rate
for comparison NVIDIA RTX 4090 has 1008GB/s memory bandwidth so it could give 59.2 eval tokens for this model

Performance is limited by memory bandwidth, roughly can be calculated by memory bandwidth/model size Zen4 CPU have 2 channels DDR5 official max supported speed is 5600MHz it gives max theoretical memory speed 89Gb/s.
codestral:22b-v0.1-q5_K_M takes 17Gib of memory and should give max 89/17 = 5.2 tokens/s
My Ryzen 7840HS with two 48 GB DDR5@5600MHz stick for codestral:22b-v0.1-q5_K_M got 4.65 tokens/s eval rate
for comparison NVIDIA RTX 4090 has 1008GB/s memory bandwidth so it could give 59.2 eval tokens for this model

Great explanation thanks. Sounds like I'm right there at the limit of performance

Performance is limited by memory bandwidth, roughly can be calculated by memory bandwidth/model size Zen4 CPU have 2 channels DDR5 official max supported speed is 5600MHz it gives max theoretical memory speed 89Gb/s.
codestral:22b-v0.1-q5_K_M takes 17Gib of memory and should give max 89/17 = 5.2 tokens/s
My Ryzen 7840HS with two 48 GB DDR5@5600MHz stick for codestral:22b-v0.1-q5_K_M got 4.65 tokens/s eval rate
for comparison NVIDIA RTX 4090 has 1008GB/s memory bandwidth so it could give 59.2 eval tokens for this model

Great explanation thanks. Sounds like I'm right there at the limit of performance

To not be so negative. Benefits for APU are not only in generation tokens. Try for example to summarize one page of text using CPU vs APU there can be up to 10x ingestion speed increase (time before token generation is substantial decreased). GTT allow using large context window. Default for ollama is tiny 2k ctx (depends on the model) with APU you can use much bigger ctx because you can use almost all available RAM as VRAM memory
extra memory needed for ctx
8K context ≈ 4-6GB VRAM
16K context ≈ 8-10GB VRAM
32K context ≈ 16-20GB VRAM
So there are combination that will be available to run on APU but not RTX 4090 like cadastral with 32K ctx

Hi Joe,
I'm trying to use docker to run ollama gtt using your docker instructions
but whenever I try to run a model I get this error:
time=2024-11-14T10:20:13.826Z level=WARN source=amd_linux.go:135
msg="ollama recommends running the
https://www.amd.com/en/support/linux-drivers" error="amdgpu version file
missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no
such file or directory"
Any idea what flags I have to set to get the version file to appear in /sys
?
Thanks
Yan
On Mon, Nov 11, 2024 at 9:02 AM Joe Ipson @.***> wrote:

@m0ngr31 https://github.com/m0ngr31
Radeon 780M Graphics
But that's a much newer GPU, probably gfx1103. No wonder gfx900 does not
work for you. Adjust AMDGPU_TARGETS and HSA_OVERRIDE_GFX_VERSION.
Yeah, I modified the commands...
To build:
docker build
-f Dockerfile
--platform linux/amd64
--target runtime-rocm
--build-arg AMDGPU_TARGETS=gfx1103
--build-arg OLLAMA_CUSTOM_CPU_DEFS="-DLLAMA_HIP_UMA=ON"
--build-arg OLLAMA_SKIP_CUDA_GENERATE=1
--build-arg OLLAMA_SKIP_CUDA_11_GENERATE=1
--build-arg OLLAMA_SKIP_CUDA_12_GENERATE=1
--build-arg OLLAMA_SKIP_ONEAPI_GENERATE=1
-t ollama-gtt .
And to run:
docker run
--stop-signal=SIGKILL
-p 11434:11434
-v ~/.ollama-new:/root/.ollama
--device /dev/dri
--device /dev/kfd
-e HSA_OVERRIDE_GFX_VERSION=11.0.1
-e OLLAMA_DEBUG=1
-e OLLAMA_MAX_LOADED_MODELS=1
-e OLLAMA_NUM_PARALLEL=1
ollama-gtt
—
Reply to this email directly, view it on GitHub
https://github.com/ollama/ollama/pull/6282#issuecomment-2468645065, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AARLSU73SSMMRF2AILDWWV32ADPLFAVCNFSM6AAAAABMIHDP52VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINRYGY2DKMBWGU
.
You are receiving this because you commented.Message ID:
@.***>

--
"We do not learn so much by our successes as we learn by failures -- our
own and others'.  Especially if we see the failures properly corrected."
-- Frank Lloyd Wright

don't try to use docker but this error for me was from begging and can be
ignored. I think that the official rocm package on supposed os is needed to
get rid of this warning
czw., 14 lis 2024, 11:23 użytkownik Yan-Fa Li @.***>
napisał:

Hi Joe,
I'm trying to use docker to run ollama gtt using your docker instructions
but whenever I try to run a model I get this error:
time=2024-11-14T10:20:13.826Z level=WARN source=amd_linux.go:135
msg="ollama recommends running the
https://www.amd.com/en/support/linux-drivers" error="amdgpu version file
missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no
such file or directory"
Any idea what flags I have to set to get the version file to appear in
/sys
?
Thanks
Yan
On Mon, Nov 11, 2024 at 9:02 AM Joe Ipson @.***> wrote:

@m0ngr31 https://github.com/m0ngr31
Radeon 780M Graphics
But that's a much newer GPU, probably gfx1103. No wonder gfx900 does not
work for you. Adjust AMDGPU_TARGETS and HSA_OVERRIDE_GFX_VERSION.
Yeah, I modified the commands...
To build:
docker build
-f Dockerfile
--platform linux/amd64
--target runtime-rocm
--build-arg AMDGPU_TARGETS=gfx1103
--build-arg OLLAMA_CUSTOM_CPU_DEFS="-DLLAMA_HIP_UMA=ON"
--build-arg OLLAMA_SKIP_CUDA_GENERATE=1
--build-arg OLLAMA_SKIP_CUDA_11_GENERATE=1
--build-arg OLLAMA_SKIP_CUDA_12_GENERATE=1
--build-arg OLLAMA_SKIP_ONEAPI_GENERATE=1
-t ollama-gtt .
And to run:
docker run
--stop-signal=SIGKILL
-p 11434:11434
-v ~/.ollama-new:/root/.ollama
--device /dev/dri
--device /dev/kfd
-e HSA_OVERRIDE_GFX_VERSION=11.0.1
-e OLLAMA_DEBUG=1
-e OLLAMA_MAX_LOADED_MODELS=1
-e OLLAMA_NUM_PARALLEL=1
ollama-gtt
—
Reply to this email directly, view it on GitHub
https://github.com/ollama/ollama/pull/6282#issuecomment-2468645065,
or
unsubscribe
<
https://github.com/notifications/unsubscribe-auth/AARLSU73SSMMRF2AILDWWV32ADPLFAVCNFSM6AAAAABMIHDP52VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINRYGY2DKMBWGU>

.
You are receiving this because you commented.Message ID:
@.***>

--
"We do not learn so much by our successes as we learn by failures -- our
own and others'. Especially if we see the failures properly corrected."
-- Frank Lloyd Wright
—
Reply to this email directly, view it on GitHub
https://github.com/ollama/ollama/pull/6282#issuecomment-2475963377, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ANORXN4TJ3A6J3WHLBRXAAD2AR2YXAVCNFSM6AAAAABMIHDP52VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINZVHE3DGMZXG4
.
You are receiving this because you were mentioned.Message ID:
@.***>

Even more confusing if I turn off debug:
[GIN] 2024/11/14 - 10:28:10 | 200 |      17.513µs |       <IP_ADDRESS> | HEAD
"/"
[GIN] 2024/11/14 - 10:28:10 | 200 |   11.776684ms |       <IP_ADDRESS> | POST
"/api/show"
time=2024-11-14T10:28:10.566Z level=WARN source=amd_linux.go:135
msg="ollama recommends running the
https://www.amd.com/en/support/linux-drivers" error="amdgpu version file
missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no
such file or directory"
time=2024-11-14T10:28:10.568Z level=INFO source=amd_linux.go:475
msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=11.0.1
time=2024-11-14T10:28:10.591Z level=INFO source=sched.go:714 msg="new model
will fit in available VRAM in single GPU, loading"
model=/root/.ollama/models/blobs/sha256-dde5aa3fc5ffc17176b5e8bdc82f587b24b2678c6c66101bf7da77af9f7ccdff
gpu=0 parallel=1 available=48097943552 required="2.9 GiB"
time=2024-11-14T10:28:10.591Z level=WARN source=amd_linux.go:135
msg="ollama recommends running the
https://www.amd.com/en/support/linux-drivers" error="amdgpu version file
missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no
such file or directory"
time=2024-11-14T10:28:10.592Z level=INFO source=amd_linux.go:475
msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=11.0.1
time=2024-11-14T10:28:10.592Z level=INFO source=server.go:105 msg="system
memory" total="45.1 GiB" free="39.0 GiB" free_swap="4.0 GiB"
time=2024-11-14T10:28:10.592Z level=INFO source=memory.go:343 msg="offload
to rocm" layers.requested=-1 layers.model=29 layers.offload=29
layers.split="" memory.available="[44.8 GiB]" memory.gpu_overhead="0 B"
memory.required.full="2.9 GiB" memory.required.partial="2.9 GiB"
memory.required.kv="224.0 MiB" memory.required.allocations="[2.9 GiB]"
memory.weights.total="1.8 GiB" memory.weights.repeating="1.5 GiB"
memory.weights.nonrepeating="308.2 MiB" memory.graph.full="256.5 MiB"
memory.graph.partial="570.7 MiB"
time=2024-11-14T10:28:10.600Z level=INFO source=server.go:384 msg="starting
llama server" cmd="/tmp/ollama3977538518/runners/rocm/ollama_llama_server
--model
/root/.ollama/models/blobs/sha256-dde5aa3fc5ffc17176b5e8bdc82f587b24b2678c6c66101bf7da77af9f7ccdff
--ctx-size 2048 --batch-size 512 --n-gpu-layers 29 --threads 6 --parallel 1
--port 41357"
time=2024-11-14T10:28:10.600Z level=INFO source=sched.go:449 msg="loaded
runners" count=1
time=2024-11-14T10:28:10.600Z level=INFO source=server.go:563 msg="waiting
for llama runner to start responding"
time=2024-11-14T10:28:10.600Z level=INFO source=server.go:597 msg="waiting
for server to become available" status="llm server error"
/tmp/ollama3977538518/runners/rocm/ollama_llama_server:
/lib/x86_64-linux-gnu/libstdc++.so.6: version GLIBCXX_3.4.32' not found (required by /tmp/ollama3977538518/runners/rocm/ollama_llama_server) /tmp/ollama3977538518/runners/rocm/ollama_llama_server: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.38' not found (required
by /tmp/ollama3977538518/runners/rocm/ollama_llama_server)
/tmp/ollama3977538518/runners/rocm/ollama_llama_server:
/lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.38' not found (required
by /tmp/ollama3977538518/runners/rocm/libllama.so)
time=2024-11-14T10:28:10.851Z level=ERROR source=sched.go:455 msg="error
loading llama server" error="llama runner process has terminated: exit
status 1"
It's complaining about GLIBC which is weird as it's in a docker container.
This happens even when I run the ollama binary from within the container
using exec -it
What am I doing wrong?
On Thu, Nov 14, 2024 at 2:22 AM Yan-Fa Li @.***> wrote:

Hi Joe,
I'm trying to use docker to run ollama gtt using your docker instructions
but whenever I try to run a model I get this error:
time=2024-11-14T10:20:13.826Z level=WARN source=amd_linux.go:135
msg="ollama recommends running the
https://www.amd.com/en/support/linux-drivers" error="amdgpu version file
missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no
such file or directory"
Any idea what flags I have to set to get the version file to appear in
/sys ?
Thanks
Yan
On Mon, Nov 11, 2024 at 9:02 AM Joe Ipson @.***>
wrote:

@m0ngr31 https://github.com/m0ngr31
Radeon 780M Graphics
But that's a much newer GPU, probably gfx1103. No wonder gfx900 does not
work for you. Adjust AMDGPU_TARGETS and HSA_OVERRIDE_GFX_VERSION.
Yeah, I modified the commands...
To build:
docker build
-f Dockerfile
--platform linux/amd64
--target runtime-rocm
--build-arg AMDGPU_TARGETS=gfx1103
--build-arg OLLAMA_CUSTOM_CPU_DEFS="-DLLAMA_HIP_UMA=ON"
--build-arg OLLAMA_SKIP_CUDA_GENERATE=1
--build-arg OLLAMA_SKIP_CUDA_11_GENERATE=1
--build-arg OLLAMA_SKIP_CUDA_12_GENERATE=1
--build-arg OLLAMA_SKIP_ONEAPI_GENERATE=1
-t ollama-gtt .
And to run:
docker run
--stop-signal=SIGKILL
-p 11434:11434
-v ~/.ollama-new:/root/.ollama
--device /dev/dri
--device /dev/kfd
-e HSA_OVERRIDE_GFX_VERSION=11.0.1
-e OLLAMA_DEBUG=1
-e OLLAMA_MAX_LOADED_MODELS=1
-e OLLAMA_NUM_PARALLEL=1
ollama-gtt
—
Reply to this email directly, view it on GitHub
https://github.com/ollama/ollama/pull/6282#issuecomment-2468645065, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AARLSU73SSMMRF2AILDWWV32ADPLFAVCNFSM6AAAAABMIHDP52VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINRYGY2DKMBWGU
.
You are receiving this because you commented.Message ID:
@.***>

--
"We do not learn so much by our successes as we learn by failures -- our
own and others'.  Especially if we see the failures properly corrected."
-- Frank Lloyd Wright

--
"We do not learn so much by our successes as we learn by failures -- our
own and others'.  Especially if we see the failures properly corrected."
-- Frank Lloyd Wright

With the latest update, this GTT patch stopped working for me. I use containers build and target 680M (from Ryzen 7000 series) and Vega (from Ryzen 5000 series)
podman build --target runtime-rocm --platform linux/amd64 --build-arg=AMDGPU_TARGETS=gfx1030,gfx900 -t localhost/ollama-rocm:latest
On the container log
time=2024-11-14T16:05:08.027Z level=INFO source=amd_linux.go:296 msg="unsupported Radeon iGPU detected skipping" id=0 total="512.0 MiB" time=2024-11-14T16:05:08.027Z level=INFO source=amd_linux.go:399 msg="no compatible amdgpu devices detected" time=2024-11-14T16:05:08.027Z level=INFO source=gpu.go:386 msg="no compatible GPUs were discovered"
I run the container image with these parameters
podman run -d -v /some_path/:/root/.ollama:z -e OLLAMA_MAX_LOADED_MODELS=1 -e OLLAMA_KEEP_ALIVE="120m" -e HSA_OVERRIDE_GFX_VERSION="10.3.0" --device /dev/kfd --device /dev/dri -e localhost/ollama-rocm:latest serve

do you check-out to the proper branch before building?
czw., 14 lis 2024, 17:41 użytkownik Ricardo Jesus Malagon Jerez <
@.***> napisał:

With the latest update, this GTT patch stopped working for me. I use
containers build and target 680M (from Ryzen 7000 series) and Vega (from
Ryzen 5000 series)
podman build --target runtime-rocm --platform linux/amd64
--build-arg=AMDGPU_TARGETS=gfx1030,gfx900 -t localhost/ollama-rocm:latest
On the container log
time=2024-11-14T16:05:08.027Z level=INFO source=amd_linux.go:296
msg="unsupported Radeon iGPU detected skipping" id=0 total="512.0 MiB"
time=2024-11-14T16:05:08.027Z level=INFO source=amd_linux.go:399 msg="no
compatible amdgpu devices detected" time=2024-11-14T16:05:08.027Z
level=INFO source=gpu.go:386 msg="no compatible GPUs were discovered"
I run the container image with these parameters
podman run -d -v /some_path/:/root/.ollama:z -e OLLAMA_MAX_LOADED_MODELS=1
-e OLLAMA_KEEP_ALIVE="120m" -e HSA_OVERRIDE_GFX_VERSION="10.3.0" --device
/dev/kfd --device /dev/dri -e localhost/ollama-rocm:latest serve
—
Reply to this email directly, view it on GitHub
https://github.com/ollama/ollama/pull/6282#issuecomment-2476910888, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ANORXNZ26DL7OH2XKSW2IB32ATHERAVCNFSM6AAAAABMIHDP52VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINZWHEYTAOBYHA
.
 AMDGPU_TARGETS is ignored by the latest version. All AMD GPU targets are always built, which can take some time. You only have to customize HSA_OVERRIDE_GFX_VERSION.  OLLAMA_CUSTOM_CPU_DEFS seems unnecessary since kernel 6.10+ allocates in GTT anyway.

podman build
-f Dockerfile
--no-cache
--platform=linux/amd64
--target runtime-rocm
--build-arg=OLLAMA_SKIP_CUDA_GENERATE=1
-t

Yup, you were right, build cache was the problem, and thanks for the info, I cleaned my build arguments a bit too.

A note, something changed on Fedora 41 current (Linux 6.11.6-300.fc41.x86_64) that it needs "--privileged" on podman to detect GPU, something is missing now for the correct device mapping on the container.

OK. Not sure what changed here, but I rebuilt the entire thing locally last night and then I noticed it couldn't find the libggml_rocm.so library in the logs. So what worked for me:
LD_LIBRARY_PATH=~/projects/ollama/dist/linux-amd64/lib/ollama HSA_OVERRIDE_GFX_VERSION=11.0.1 ./ollama serve
```

Giving it the path of the missing libraries allows the /tmp runtime to work now.

> A note, something changed on Fedora 41 current (Linux 6.11.6-300.fc41.x86_64) that it needs "--privileged" on podman to detect GPU, something is missing now for the correct device mapping on the container.

Thanks for the tip. Building docker from a complete clean checkout seems to be working now. Thank you

A note, something changed on Fedora 41 current (Linux 6.11.6-300.fc41.x86_64) that it needs "--privileged" on podman to detect GPU, something is missing now for the correct device mapping on the container.

Thanks for the tip. Building from a clean checkout fixed my stuff too.

So my biggest issue right now is the GPU appears to hang, which causes it to reset and the entire UI restarts (KDE wayland). I'm not sure what the right solution to that issue is, but I loaded dolphin mistral 26GiB model and just did some simple queries and the entire UI restarted. I don't remember this happening with the earlier version nearly as often. I was able to hang it fairly reliably by just sending it too many tokens.