Ollama Chatbot on Linux: 7 Easy Steps and Mistakes to Avoid
An Ollama chatbot on Linux runs a language model on your own machine: no account, no API key and no cloud. The old version of this post got that wrong. It told you to download a script from get.ollama.io and to add an OpenAI API key to a configuration file. I tested both claims, and neither works. This rewrite uses a real lab: the official install script in a Debian 13 container and Ollama 0.40.13 in a container on a 20-thread CPU with no GPU. All output below is from that lab.
Contents
Step 1: install the Ollama chatbot on Linux
The domain from the old post does not exist, so that first step of the old Ollama chatbot on Linux guide fails at once:
$ curl -fsSL -m 10 https://get.ollama.io | head -3
curl: (6) Could not resolve host: get.ollama.io
The Ollama Linux documentation gives the real command:
$ curl -fsSL https://ollama.com/install.sh | sh
Read a script before you pipe it to a shell. This one had 455 lines. On a normal system it creates an ollama user with useradd, writes /etc/systemd/system/ollama.service and runs systemctl enable ollama. In my container, which has no systemd, it installed the program and skipped the service:
WARNING: Unable to detect NVIDIA/AMD GPU. Install lspci or lshw to automatically detect and install GPU dependencies.
>>> The Ollama API is now available at 127.0.0.1:11434.
>>> Install complete. Run "ollama" from the command line.
$ du -sh /usr/local/lib/ollama
2.3G /usr/local/lib/ollama
$ id ollama
id: 'ollama': no such user
Note the size: 2.3 GB before you download a single model. That is the base cost of an Ollama chatbot on Linux.
Step 2: start the server and check it
With systemd, the service starts by itself. In a container, run ollama serve, or use the ollama/ollama image, as I did for the rest of the lab. The server answers on port 11434:
$ curl -s http://127.0.0.1:21434/; echo; curl -s http://127.0.0.1:21434/api/version
Ollama is running
{"version":"0.40.13"}
(I mapped the container to port 21434 on the host.) If the client cannot reach the server, every command fails the same way:
$ OLLAMA_HOST=127.0.0.1:1 ollama list
Error: could not connect to ollama server, run 'ollama serve' to start it
Step 3: pull a model for your Ollama chatbot on Linux
Ollama runs models from its own library, not GPT-3.5. A small model is enough to test the setup. TinyLlama is 637 MB:
$ ollama pull tinyllama
writing manifest
success
$ ollama show tinyllama
Model
architecture llama
parameters 1B
context length 2048
embedding length 2048
quantization Q4_0
The two commands from the old post fail with clear messages. The first is not a command at all, and the second looks for a model with that name in the library:
$ ollama gpt-3.5-turbo
Error: unknown command "gpt-3.5-turbo" for "ollama"
$ ollama run gpt-3.5-turbo "hi"
Error: pull model manifest: file does not exist

Step 4: chat in the terminal
This is the part most people mean by an Ollama chatbot on Linux: ollama run tinyllama opens an interactive chat, and /bye ends it. You can also pass one prompt and get one answer, which took 4.3 seconds the first time, including loading the model. While it was loaded, ollama ps showed where it ran:
$ ollama ps
NAME ID SIZE PROCESSOR CONTEXT RUNNER UNTIL
tinyllama:latest 2644915ede35 739 MB 100% CPU 2048 llamacpp 4 minutes from now
"100% CPU" means no GPU was used. The server log said the same with library=cpu. The model stays in memory for a few minutes after the last request; ollama stop tinyllama unloads it at once.
Step 5: use the REST API
The API is what turns an Ollama chatbot on Linux into something your scripts can call. With "stream":false you get one JSON object:
$ curl -s http://127.0.0.1:21434/api/generate -d '{"model":"tinyllama","prompt":"Say hi in 3 words","stream":false,"options":{"num_predict":12,"temperature":0}}'
{'model': 'tinyllama', 'response': 'Hey!', 'done': True, 'done_reason': 'stop', 'prompt_eval_count': 41, 'eval_count': 4}
eval/s 63.1
I printed selected fields with Python; the last line is tokens per second from eval_count and eval_duration. Without "stream":false, the answer arrives as one JSON line per piece of text:
{"model":"tinyllama","created_at":"2026-11-02T07:05:03.259305728Z","response":"Ab","done":false}
{"model":"tinyllama","created_at":"2026-11-02T07:05:03.280374872Z","response":"sol","done":false}
{"model":"tinyllama","created_at":"2026-11-02T07:05:03.301745905Z","response":"utely","done":false}
There is also an OpenAI-style endpoint at /v1/chat/completions. My request returned chat.completion as the object type, so tools written for that format can point at Ollama instead. Small models do not always follow instructions, though. With the system message "You answer in one word" and the question "Capital of France?", TinyLlama answered:
{'role': 'assistant', 'content': 'Yes, Paris is the capital of France.'}
Step 6: build your own Ollama chatbot on Linux with a Modelfile
A Modelfile sets a system prompt and parameters on top of a model:
FROM tinyllama
PARAMETER temperature 0.2
PARAMETER num_ctx 1024
SYSTEM """You are a Linux helper. Answer in one short sentence."""
$ ollama create linuxbot -f /tmp/Modelfile
success
$ ollama show linuxbot --parameters
temperature 0.2
num_ctx 1024
The new model used no extra disk: ollama list showed both at 637 MB, because they share the same weights. Asked "How do I list files?", linuxbot answered with ls and ls -a, but in several paragraphs, not one short sentence. A 1B model ignores a lot of what you ask. For real work, try a larger model if your RAM allows it.
Step 7: update or remove Ollama
Keeping an Ollama chatbot on Linux current is simple. The documentation says to update by running the install script again, and to set OLLAMA_VERSION for a specific version:
$ curl -fsSL https://ollama.com/install.sh | sh
$ curl -fsSL https://ollama.com/install.sh | OLLAMA_VERSION=0.5.7 sh
The version in the second line is the example from the docs. To see the logs of the service, the docs give journalctl -e -u ollama. Models take space fast: the lab's model folder was 3.2 GB with the six model names listed at that point. Remove one with ollama rm.
Mistakes to avoid
- Using install commands from copied tutorials. Check the domain first;
get.ollama.iodoes not resolve. - Expecting OpenAI models. An Ollama chatbot on Linux runs local models only, and needs no API key.
- Exposing port 11434 to the network. The API in this lab had no authentication: anyone who reaches it can use your models.
- Judging the tool by a tiny model. TinyLlama ignored instructions in this lab.
- Forgetting disk space: 2.3 GB for the program plus every model you pull.
To keep the server private, run it behind a firewall. The iptables post shows how to limit a port.
FAQ
Does an Ollama chatbot on Linux need a GPU? No. This whole lab ran on CPU at about 63 tokens per second with a 1B model on 20 threads. Larger models are slower.
Is there an apt package? The documentation uses the install script or a manual tar download; I did not test any distribution package.
How do I talk to it from Python or another app? Send JSON to /api/chat or /api/generate on port 11434, as shown above.
Comments
Post a Comment