Use the Install script
Visit https://ollama.com/download and use the installer shell script. IOW,
curl -fsSL https://ollama.com/install.sh | sh
The script describes what it is doing:
>>> Installing ollama to /usr/local >>> Downloading Linux amd64 bundle ######################################################################## 100.0% >>> Creating ollama user... >>> Adding ollama user to render group... >>> Adding ollama user to video group... >>> Adding current user to ollama group... >>> Creating ollama systemd service... >>> Enabling and starting ollama service... Created symlink /etc/systemd/system/default.target.wants/ollama.service → /etc/systemd/system/ollama.service. >>> NVIDIA GPU installed.
See Ollama/install/script for the actual script used.
ollama -v
shows you that it's running by printing the version.
ollama version is 0.9.1
Simply running it as a Docker Image
Although you can download or install it from the repo on GitHub https://github.com/ollama/ollama, you can also run it as a docker image ollama/ollama[1]
However, I ran into multiple issues, and decided to go the straight install route instead.
First Issue: accommodate NVidia GPU
Because I have a GeForce RTX 4060 NVidia GPU, I had to install the NVidia Container Toolkit, and configure Docker to use the NVidia driver
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
| sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
| sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
| sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
Super User Problems
The docs advise to
(sudo) docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama
which clearly runs as root, however my Docker (Desktop) is running as non-root user, so although I previously fetched the image through Docker Desktop, the CLI command couldn't find it and downloaded another copy. And spit out additional errors:
Unable to find image 'ollama/ollama:latest' locally latest: Pulling from ollama/ollama 13b7e930469f: Pull complete 97ca0261c313: Pull complete 2ace2f9dde9e: Pull complete 41ea4d361810: Pull complete Digest: sha256:50ab2378567a62b811a2967759dd91f254864c3495cbe50576bd8a85bc6edd56 Status: Downloaded newer image for ollama/ollama:latest 40be284dab1709b74fa68d513f75c10239d7234a21d65aac1e80cbd743515498 docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running prestart hook #0: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy' nvidia-container-cli: initialization error: nvml error: driver/library version mismatch: unknown
The important part seems to be 'Auto-detected mode as legacy' and the nvml driver/library mismatch error is certainly a problem. (Is it the same problem; or two separate problems?)
Running the image from Docker Desktop, with setting options for ports and volumes, and copying the 'run' command spits out:
docker run --hostname=3f50cd4183bd --mac-address=02:42:ac:11:00:02 --env=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin --env=LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 --env=NVIDIA_DRIVER_CAPABILITIES=compute,utility --env=NVIDIA_VISIBLE_DEVICES=all --env=OLLAMA_HOST=0.0.0.0:11434 --volume=ollama:/root/.ollama --network=bridge -p 11434:11434 --restart=no --label='org.opencontainers.image.ref.name=ubuntu' --label='org.opencontainers.image.version=20.04' --runtime=runc -d ollama/ollama:latest
http://localhost:11434/ just reveals 'Ollama is running' in 'hello world' style.
In the docker-compose file there is supposed to be another container image accessible at port 3000 providing a webUI. This didn't happen.
Clearly, the full ollama setup is supposed to be run as 'root'. It is not designed to be run as a regular user who has docker or sudo / adm group membership.
Problems with GPU
Just like in the Docker-based attempt to get up and running, I ran into challenges getting the right software for my GeForce RTX 4060 video card by NVidia. The docs tell you to check with
nvidia-smi
Failed to initialize NVML: Driver/library version mismatch NVML library version: 550.144
So, apparently I'm supposed to install the CUDA Toolkit? and the Driver?[2]
Simply installing the toolkit and rebooting managed to resolve my driver issue. Temporarily
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb sudo dpkg -i cuda-keyring_1.1-1_all.deb sudo apt-get update sudo apt-get install -y cuda-toolkit-12-9 sudo shutdown -r now ollama -v nvidia-smi journalctl -e -u ollama
I say 'temporarily' because things shortly went badly and I spent hours installing, removing, re-installing various NVidia drivers. The Internet is full of problem reports trying to get NVidia drivers working properly on Ubuntu.
Errors installing NVidia driver
Error! Installation aborted.
dpkg: error processing package nvidia-dkms-550 (--configure):
installed nvidia-dkms-550 package post-installation script subprocess returned error exit status 6
Setting up libnvidia-encode-550:amd64 (550.144.03-0ubuntu0.24.04.1) ...
Setting up libnvidia-encode-550:i386 (550.144.03-0ubuntu0.24.04.1) ...
dpkg: dependency problems prevent configuration of nvidia-driver-550:
nvidia-driver-550 depends on nvidia-dkms-550 (<= 550.144.03-1); however:
Package nvidia-dkms-550 is not configured yet.
nvidia-driver-550 depends on nvidia-dkms-550 (>= 550.144.03); however:
Package nvidia-dkms-550 is not configured yet.
dpkg: error processing package nvidia-driver-550 (--configure):
dependency problems - leaving unconfigured
No apport report written because the error message indicates its a followup error from a previous failure.
Processing triggers for libc-bin (2.39-0ubuntu8.4) ...
Processing triggers for man-db (2.12.0-4build2) ...
Processing triggers for initramfs-tools (0.142ubuntu25.5) ...
update-initramfs: Generating /boot/initrd.img-6.8.0-60-generic
Errors were encountered while processing:
nvidia-dkms-550
nvidia-driver-550
E: Sub-process /usr/bin/dpkg returned an error code (1)
Eventually, after totally purging everything NVidia-related, and starting over, I seem to have a working system that survives reboots. (I saved a transcript of the misery in email under 'NVidia driver F*ckup'.) Supposedly, there is a newer driver available as 'production' but I didn't want to try upgrading.
$ nvidia-smi
Thu Jun 19 08:50:34 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.144.03 Driver Version: 550.144.03 CUDA Version: 12.4 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 4060 Off | 00000000:01:00.0 On | N/A |
| 0% 49C P5 N/A / 115W | 1567MiB / 8188MiB | 22% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 2579 G /usr/lib/xorg/Xorg 912MiB |
| 0 N/A N/A 3625 G /usr/bin/kwalletd5 2MiB |
| 0 N/A N/A 3821 G ...-gnu/libexec/xdg-desktop-portal-kde 2MiB |
| 0 N/A N/A 3939 G /usr/bin/ksmserver 2MiB |
| 0 N/A N/A 3941 G /usr/bin/kded5 2MiB |
| 0 N/A N/A 3942 G /usr/bin/kwin_x11 167MiB |
| 0 N/A N/A 3992 G /usr/bin/plasmashell 170MiB |
| 0 N/A N/A 4039 G ...c/polkit-kde-authentication-agent-1 2MiB |
| 0 N/A N/A 4152 G ...86_64-linux-gnu/libexec/kdeconnectd 2MiB |
| 0 N/A N/A 4191 G /usr/bin/kaccess 2MiB |
| 0 N/A N/A 4203 G ...-linux-gnu/libexec/DiscoverNotifier 2MiB |
| 0 N/A N/A 4974 G /opt/google/chrome/chrome 2MiB |
| 0 N/A N/A 5080 G ...seed-version=20250618-050053.671000 181MiB |
| 0 N/A N/A 5154 G ...bin/plasma-browser-integration-host 2MiB |
| 0 N/A N/A 9507 G ..._64-linux-gnu/libexec/kf5/klauncher 2MiB |
| 0 N/A N/A 12144 G /usr/bin/konsole 2MiB |
| 0 N/A N/A 12337 G ...irefox/6316/usr/lib/firefox/firefox 59MiB |
+-----------------------------------------------------------------------------------------+
Resources
No animals were harmed during the creation of this web page. But my eyes bugged out from staring at countless web resources including the following:
- https://docs.nvidia.com/cuda/cuda-installation-guide-linux/#meta-packages
- https://documentation.ubuntu.com/server/how-to/graphics/install-nvidia-drivers/
- https://askubuntu.com/questions/1516250/installing-nvidia-drivers-on-24-04-is-an-impossible-task
- https://wiki.ubuntu.com/Kernel/MainlineBuilds
- https://github.com/oddmario/NVIDIA-Ubuntu-Driver-Guide
- https://launchpad.net/~graphics-drivers/+archive/ubuntu/ppa
- https://forums.developer.nvidia.com/t/error-installing-nvidia-drivers-on-ubuntu/231193/6
- https://forums.developer.nvidia.com/t/nvidia-smi-has-failed-because-it-couldnt-communicate-with-the-nvidia-driver-make-sure-that-the-latest-nvidia-driver-is-installed-and-running/197141/5
- https://forums.developer.nvidia.com/t/errors-were-encountered-while-processing-dkms/236521
