Ollama Chatbot on Linux: 7 Easy Steps and Mistakes to Avoid

An Ollama chatbot on Linux runs a language model on your own machine: no account, no API key and no cloud. The old version of this post got that wrong. It told you to download a script from get.ollama.io and to add an OpenAI API key to a configuration file. I tested both claims, and neither works. This rewrite uses a real lab: the official install script in a Debian 13 container and Ollama 0.40.13 in a container on a 20-thread CPU with no GPU. All output below is from that lab.

Step 1: install the Ollama chatbot on Linux

The domain from the old post does not exist, so that first step of the old Ollama chatbot on Linux guide fails at once:

$ curl -fsSL -m 10 https://get.ollama.io | head -3
curl: (6) Could not resolve host: get.ollama.io

The Ollama Linux documentation gives the real command:

$ curl -fsSL https://ollama.com/install.sh | sh

Read a script before you pipe it to a shell. This one had 455 lines. On a normal system it creates an ollama user with useradd, writes /etc/systemd/system/ollama.service and runs systemctl enable ollama. In my container, which has no systemd, it installed the program and skipped the service:

WARNING: Unable to detect NVIDIA/AMD GPU. Install lspci or lshw to automatically detect and install GPU dependencies.
>>> The Ollama API is now available at 127.0.0.1:11434.
>>> Install complete. Run "ollama" from the command line.

$ du -sh /usr/local/lib/ollama
2.3G	/usr/local/lib/ollama
$ id ollama
id: 'ollama': no such user

Note the size: 2.3 GB before you download a single model. That is the base cost of an Ollama chatbot on Linux.

Step 2: start the server and check it

With systemd, the service starts by itself. In a container, run ollama serve, or use the ollama/ollama image, as I did for the rest of the lab. The server answers on port 11434:

$ curl -s http://127.0.0.1:21434/; echo; curl -s http://127.0.0.1:21434/api/version
Ollama is running
{"version":"0.40.13"}

(I mapped the container to port 21434 on the host.) If the client cannot reach the server, every command fails the same way:

$ OLLAMA_HOST=127.0.0.1:1 ollama list
Error: could not connect to ollama server, run 'ollama serve' to start it

Step 3: pull a model for your Ollama chatbot on Linux

Ollama runs models from its own library, not GPT-3.5. A small model is enough to test the setup. TinyLlama is 637 MB:

$ ollama pull tinyllama
writing manifest 
success 

$ ollama show tinyllama
  Model
    architecture        llama          
    parameters          1B             
    context length      2048           
    embedding length    2048           
    quantization        Q4_0           

The two commands from the old post fail with clear messages. The first is not a command at all, and the second looks for a model with that name in the library:

$ ollama gpt-3.5-turbo
Error: unknown command "gpt-3.5-turbo" for "ollama"

$ ollama run gpt-3.5-turbo "hi"
Error: pull model manifest: file does not exist
Ollama chatbot on Linux: the old gpt-3.5-turbo command fails, tinyllama runs on 100% CPU and the API answers
The real commands, and the old one that never worked.

Step 4: chat in the terminal

This is the part most people mean by an Ollama chatbot on Linux: ollama run tinyllama opens an interactive chat, and /bye ends it. You can also pass one prompt and get one answer, which took 4.3 seconds the first time, including loading the model. While it was loaded, ollama ps showed where it ran:

$ ollama ps
NAME                ID              SIZE      PROCESSOR    CONTEXT    RUNNER      UNTIL              
tinyllama:latest    2644915ede35    739 MB    100% CPU     2048       llamacpp    4 minutes from now    

"100% CPU" means no GPU was used. The server log said the same with library=cpu. The model stays in memory for a few minutes after the last request; ollama stop tinyllama unloads it at once.

Step 5: use the REST API

The API is what turns an Ollama chatbot on Linux into something your scripts can call. With "stream":false you get one JSON object:

$ curl -s http://127.0.0.1:21434/api/generate -d '{"model":"tinyllama","prompt":"Say hi in 3 words","stream":false,"options":{"num_predict":12,"temperature":0}}'
{'model': 'tinyllama', 'response': 'Hey!', 'done': True, 'done_reason': 'stop', 'prompt_eval_count': 41, 'eval_count': 4}
eval/s 63.1

I printed selected fields with Python; the last line is tokens per second from eval_count and eval_duration. Without "stream":false, the answer arrives as one JSON line per piece of text:

{"model":"tinyllama","created_at":"2026-11-02T07:05:03.259305728Z","response":"Ab","done":false}
{"model":"tinyllama","created_at":"2026-11-02T07:05:03.280374872Z","response":"sol","done":false}
{"model":"tinyllama","created_at":"2026-11-02T07:05:03.301745905Z","response":"utely","done":false}

There is also an OpenAI-style endpoint at /v1/chat/completions. My request returned chat.completion as the object type, so tools written for that format can point at Ollama instead. Small models do not always follow instructions, though. With the system message "You answer in one word" and the question "Capital of France?", TinyLlama answered:

{'role': 'assistant', 'content': 'Yes, Paris is the capital of France.'}

Step 6: build your own Ollama chatbot on Linux with a Modelfile

A Modelfile sets a system prompt and parameters on top of a model:

FROM tinyllama
PARAMETER temperature 0.2
PARAMETER num_ctx 1024
SYSTEM """You are a Linux helper. Answer in one short sentence."""
$ ollama create linuxbot -f /tmp/Modelfile
success 
$ ollama show linuxbot --parameters
temperature                    0.2
num_ctx                        1024

The new model used no extra disk: ollama list showed both at 637 MB, because they share the same weights. Asked "How do I list files?", linuxbot answered with ls and ls -a, but in several paragraphs, not one short sentence. A 1B model ignores a lot of what you ask. For real work, try a larger model if your RAM allows it.

Step 7: update or remove Ollama

Keeping an Ollama chatbot on Linux current is simple. The documentation says to update by running the install script again, and to set OLLAMA_VERSION for a specific version:

$ curl -fsSL https://ollama.com/install.sh | sh
$ curl -fsSL https://ollama.com/install.sh | OLLAMA_VERSION=0.5.7 sh

The version in the second line is the example from the docs. To see the logs of the service, the docs give journalctl -e -u ollama. Models take space fast: the lab's model folder was 3.2 GB with the six model names listed at that point. Remove one with ollama rm.

Mistakes to avoid

  • Using install commands from copied tutorials. Check the domain first; get.ollama.io does not resolve.
  • Expecting OpenAI models. An Ollama chatbot on Linux runs local models only, and needs no API key.
  • Exposing port 11434 to the network. The API in this lab had no authentication: anyone who reaches it can use your models.
  • Judging the tool by a tiny model. TinyLlama ignored instructions in this lab.
  • Forgetting disk space: 2.3 GB for the program plus every model you pull.

To keep the server private, run it behind a firewall. The iptables post shows how to limit a port.

FAQ

Does an Ollama chatbot on Linux need a GPU? No. This whole lab ran on CPU at about 63 tokens per second with a 1B model on 20 threads. Larger models are slower.

Is there an apt package? The documentation uses the install script or a manual tar download; I did not test any distribution package.

How do I talk to it from Python or another app? Send JSON to /api/chat or /api/generate on port 11434, as shown above.

Comments

Popular posts from this blog

How to Play Minecraft Bedrock Edition on Linux: A Comprehensive Guide for Tech Professionals

Linux desktop frozen? My 5 go-to tricks to try - before forcing a hard reboot

The Ultimate Guide: How to Set Up DXVK in Wine on Linux for Enhanced Gaming Performance