How to run Ollama on a VPS: a local LLM with API access

Ollama is a lightweight server for running language models on your own hardware. It installs with one command, exposes an HTTP API that is compatible with the OpenAI format, and works even on a plain VPS with no GPU - if you match the model to the amount of memory. Here is how to install Ollama on a server, query a model through the API, and, most importantly, how to expose it without handing your resources to the whole internet.

In short. Install with curl -fsSL https://ollama.com/install.sh | sh; the ollama service then listens on 127.0.0.1:11434. Pull a model with ollama pull; send requests to /api/generate or the OpenAI-compatible /v1/chat/completions. Memory rule: you need RAM of at least the model file size plus 1-2 GB. Ollama has no built-in auth - expose it only behind a reverse proxy with a password, an IP-scoped firewall, or an SSH tunnel.

What Ollama is and why run it on a server

Ollama runs GGUF-format models (Llama, Qwen, Gemma, Phi, Mistral and others), manages loading them into memory and unloading them, and serves a single HTTP API. On your own server this matters when:

• data cannot be sent to an external service (personal data, trade secrets);
• you want a predictable bill - a fixed VPS rental instead of paying per token;
• the model is called from your own scripts, a bot, a processing pipeline, and you want the endpoint close by;
• you need embeddings for document search without hitting a paid API.

How much it needs

The main constraint is RAM. The whole model loads into memory (or into VRAM if there is a GPU). You need: the model file size + 1-2 GB for context and bookkeeping.

llama3.2:1b

nomic-embed-text

About speed. With no GPU, inference runs on the CPU. On a modern VPS core, 1-3B models produce a few tokens per second - fine for scripts and background processing, already slow for a live chat. 7-8B models on CPU are single-digit tokens per second, usable for batch jobs, not for interactive use. A GPU changes this by an order of magnitude, but a VPS with a GPU costs noticeably more.

Step 1. Install

Log in to the server over SSH as root (or via sudo) and run the official installer:

curl -fsSL https://ollama.com/install.sh | sh

The script downloads the binary, creates a system user ollama and a service ollama.service that starts now and on boot. If the server has a supported NVIDIA or AMD GPU, the installer picks it up; otherwise it runs in CPU mode.

Check:

ollama --version
systemctl status ollama --no-pager

By default the service listens only on 127.0.0.1:11434 - it is not reachable from outside yet, which is correct (see step 4).

Step 2. Your first model

Pull a model that fits your memory. Start with a light one:

ollama pull llama3.2:1b

List what is already downloaded:

ollama list

A quick terminal test (exit with /bye):

ollama run llama3.2:1b "In one sentence: what is a reverse proxy?"

What is currently in memory and how much it takes:

ollama ps

Step 3. Query through the API

Ollama serves an HTTP API on the same port 11434. A single request:

curl http://localhost:11434/api/generate -d '{
"model": "llama3.2:1b",
"prompt": "List three upsides of running an LLM on your own VPS",
"stream": false
}'

The reply has a response field with the text. "stream": false returns the answer whole; without it Ollama sends a stream of JSON chunks.

For a conversation with history use the /api/chat endpoint with a messages array. For ready-made clients (the OpenAI SDK, LangChain, n8n) there is an OpenAI-compatible endpoint:

curl http://localhost:11434/v1/chat/completions -d '{
"model": "llama3.2:1b",
"messages": [{"role": "user", "content": "Hello"}]
}'

In the OpenAI SDK it is enough to set a base_url like http://YOUR_SERVER:11434/v1 and any non-empty key.

Step 4. Exposing it - safely only

Ollama has no auth. Anyone who can reach port 11434 can run your models and load the server. So simply "opening the port to the world" is a bad option. Pick one of three.

Option A. SSH tunnel (nothing is exposed)

Keep Ollama on localhost and forward the port from the client:

ssh -N -L 11434:localhost:11434 user@YOUR_SERVER

Requests to http://localhost:11434 from your machine now reach Ollama on the server. The simplest option if only you need access.

Option B. Reverse proxy with a password

Put a web server in front of Ollama to add HTTPS and a password check. In Caddy it is a few lines in the Caddyfile:

llm.example.com {
basic_auth {
apiuser $2a$14$...password_hash...
}
reverse_proxy 127.0.0.1:11434
}

Generate the password hash with caddy hash-password. How to install Caddy and get a certificate - see Caddy: automatic HTTPS in 5 minutes.

Option C. IP-scoped firewall

If Ollama must listen externally, open the port to your address only. First let the service listen beyond localhost - systemctl edit ollama, add:

[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"

systemctl daemon-reload && systemctl restart ollama

And immediately restrict access with the firewall:

ufw allow from YOUR_IP to any port 11434 proto tcp
ufw deny 11434

Service settings

Options are set as environment variables via systemctl edit ollama:

• OLLAMA_MODELS - where to store models (default /usr/share/ollama/.ollama/models); move it to a larger disk if you have many.
• OLLAMA_KEEP_ALIVE - how long to keep a model in memory after a request (default 5m); -1 is forever, 0 unloads immediately.
• OLLAMA_NUM_PARALLEL - how many requests to handle at once.
• OLLAMA_MAX_LOADED_MODELS - how many different models to keep in memory together.

Which model to pick for the task

• Classification, tagging, short extractions from text - llama3.2:1b or gemma2:2b, fast and cheap on memory.
• Substantive answers, summarizing, help with code - qwen2.5:7b or llama3.1:8b, if you have 8 GB of RAM.
• Search over your own documents (RAG) - nomic-embed-text for embeddings plus a small generative model for the answer.
• The full catalogue is at ollama.com/library; each model has variants by parameter count and by quantization (q4, q8) - lower quantization means less memory and less quality.

Common errors

• model "X" not found. The model is not downloaded. ollama pull X.
• "connection refused" from outside, works locally. The service listens only on 127.0.0.1. Either a tunnel, or OLLAMA_HOST=0.0.0.0 plus a proxy or firewall (step 4).
• The OOM killer kills the process while loading a model. The model is bigger than free memory. Take a smaller variant or a more aggressive quantization, or bump the plan.
• Answers are very slow. CPU inference. Shrink the model, cut the context, or move to a GPU server for heavy models.
• The first request is slow, then faster. Normal: the model loads into memory on the first call. Keep it loaded with OLLAMA_KEEP_ALIVE.

Next steps

• Caddy: automatic HTTPS in 5 minutes - a reverse proxy with a password in front of Ollama.
• Secure a fresh VPS - a firewall and key login before you publish anything.
• How to deploy a service on a server 24/7 - if Ollama serves production load.