AnythingLLM on a VPS: a web UI and document RAG on top of Ollama

An Ollama box gives you an engine and an HTTP API. The model answers, curl works, and then you hit the wall: there is no chat window, nothing remembers the previous message, there is no way to hand the model a folder of documents, and there is no login. Running AnythingLLM on a VPS is the shortest path past all four, because it is a browser UI with workspaces, file ingestion and an embedded vector database that uses your Ollama as the executor. What follows is the compose deployment, the wiring to an engine you already run, data persistence, the auth switch you should not postpone, and the perimeter this belongs behind.

Short version. AnythingLLM runs from one compose file off the mintplexlabs/anythingllm image, listens on port 3001, and keeps documents, LanceDB vector files, chats and settings under /app/server/storage, which has to be mounted to the host. Reaching a natively installed Ollama on the same box needs two things together: a host-gateway entry in extra_hosts, and an Ollama that listens beyond loopback. Resource-wise the project asks for 2 GB RAM, two cores with AVX2 and 5 GB of disk, with the model priced on top of that. Turn on password protection or multi-user mode right after the first login, because a fresh instance is open to anyone who knows the address. Do not publish the port: use 127.0.0.1:3001:3001, put nginx in front, keep ufw to 80 and 443. A plain ufw deny 3001 does nothing against Docker, and that gets its own section below.

What AnythingLLM adds that the raw Ollama API does not have

These are two layers, not two options. Ollama is the inference engine: weights in RAM, HTTP in, tokens out. AnythingLLM sits above it and owns everything else, starting with what actually goes into each request.

The substantial part is RAG (retrieval-augmented generation). You drop documents into a workspace; AnythingLLM splits them into chunks, runs each chunk through an embedder (a small model that turns text into a numeric vector) and stores the result in a vector database. Ask a question and it first searches that store for chunks close to your question, then attaches them to the prompt. Nothing is fine-tuned and the model learns nothing permanently: it simply receives the relevant excerpt alongside the question.

The reverse holds too. AnythingLLM computes nothing. Generation stays with whatever provider you point it at, so if answers crawl, the fix is the engine or the plan, not the interface.

Starting fork: you already run Ollama, or you do not

This piece continues the Ollama on a VPS walkthrough and assumes the engine is up, a model is pulled and curl against 127.0.0.1:11434 answers. If that is not your situation, go install it there and come back, because the rest of this needs a working engine address.

There is a second route. AnythingLLM also speaks to hosted providers, and the first-run wizard lets you pick one instead of a local model. Your server then carries only the UI, the vector store and the embedder, with no RAM reserved for weights. The trade is obvious: your document text leaves the machine. Privacy is usually the whole reason this stack exists on your own VPS, so treat the hosted-provider route as a stopgap while you decide which model to run locally.

What AnythingLLM on a VPS actually needs: RAM, disk, AVX2

The documented minimum is 2 GB of RAM, a 2-core CPU with AVX2 support and 5 GB of storage. Read that as the cost of AnythingLLM plus LanceDB plus the built-in embedder. The language model is not in that number and almost always outweighs everything else combined.

One expense is easy to overlook. The native embedder (EMBEDDING_ENGINE=native) is a small neural network loaded inside the same container. It is idle while you chat and busy while you index, which means on a small plan a heavy PDF can spike memory at the exact moment the model is producing an answer.

Those are estimates derived from the documented minimum and the "model file size plus headroom" rule from the Ollama article, not measurements on a live box. Check your own numbers with docker stats. Size the disk the same honest way: 5 GB for the application, plus weights, plus your documents and their vectors.

Now the check that takes one command and saves an evening. LanceDB is compiled against the AVX2 instruction set, and on a CPU without it the container dies with Illegal instruction and no useful hint. On KVM virtualisation this depends on the CPU model the hypervisor exposes to the guest, so look before you build:

grep -o -m1 avx2 /proc/cpuinfo

Output of avx2 means you are fine. Empty output means the stock LanceDB build will not start here, and your options are a different plan or location, or switching VECTOR_DB to an external database that you then have to run as its own service.

Step 1. Docker and a directory for the data

Install Docker from Docker's own repository rather than the distribution one. The docker.io package lags, and docker compose is packaged differently there. The sequence below is the official one and is identical on Ubuntu 22.04, 24.04 and 26.04.

sudo apt-get update
sudo apt-get install -y ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin

• $(. /etc/os-release && echo "$VERSION_CODENAME") fills in your release codename, so there is no line to hand-edit per version.
• docker-compose-plugin is the two-word docker compose. The hyphenated docker-compose is the older, separate tool and can behave differently with the file below.

Verify with docker compose version, which should print something like Docker Compose version v2.x.x. If you get docker: 'compose' is not a docker command instead, the plugin did not install and you should rerun the last line.

Next, the directory the service will live in. Documents, vectors and chats belong on the host, because otherwise a single docker compose down removes your work along with the container.

sudo mkdir -p /opt/anythingllm/storage
cd /opt/anythingllm
sudo touch .env
sudo chown -R 1000:1000 /opt/anythingllm

• storage is where /app/server/storage gets mounted. All persistence happens here.
• The empty .env is created deliberately: if the file is missing, Docker creates a directory in its place at mount time and the container will not start.
• chown 1000:1000 matters because the process inside the image does not run as root and otherwise cannot write to its own volume.

Step 2. The compose file and the variables

Write /opt/anythingllm/docker-compose.yml:

services:
anythingllm:
image: mintplexlabs/anythingllm:latest
container_name: anythingllm
restart: unless-stopped
cap_add:
- SYS_ADMIN
ports:
- "127.0.0.1:3001:3001"
volumes:
- ./storage:/app/server/storage
- ./.env:/app/server/.env
extra_hosts:
- "host.docker.internal:host-gateway"

• 127.0.0.1:3001:3001 publishes the port on loopback only. From outside the server it does not exist; nginx will be the only thing facing the internet. This single line is the subject of the perimeter section.
• ./.env:/app/server/.env matters because the application writes to that file as well as reads it: settings you pick in the UI are persisted there. Skip the mount and your provider and model choice disappear the first time the container is recreated.
• extra_hosts with host-gateway makes host.docker.internal resolve to the host from inside the container. Unlike Docker Desktop, plain Linux does not provide that name for free.
• cap_add: SYS_ADMIN is for the document collector, which starts a headless browser when you feed it a URL. If you only upload local files, try without it first: extra capabilities are not free.
• restart: unless-stopped brings the service back after a reboot. Why a restart policy matters at all, and how it differs from starting things by hand, is covered in the piece on deploying a service on a server.

Then /opt/anythingllm/.env. A minimal working set for the Ollama pairing:

SERVER_PORT=3001
STORAGE_DIR="/app/server/storage"
JWT_SECRET="replace-with-a-random-string-of-32-chars-or-more"
SIG_KEY="replace-with-a-random-string-of-32-chars-or-more"
SIG_SALT="replace-with-a-random-string-of-32-chars-or-more"
DISABLE_TELEMETRY="true"

LLM_PROVIDER="ollama"
OLLAMA_BASE_PATH="http://host.docker.internal:11434"
OLLAMA_MODEL_PREF="llama3.1:8b"
OLLAMA_MODEL_TOKEN_LIMIT=4096

EMBEDDING_ENGINE="native"
VECTOR_DB="lancedb"

• Generate all three secrets separately with openssl rand -hex 32. They sign sessions, so leaving documentation placeholders in place means anyone who read the docs can forge a login.
• OLLAMA_MODEL_PREF takes the tag exactly as ollama list prints it on your server. Substitute yours; the example may not exist on your box.
• OLLAMA_MODEL_TOKEN_LIMIT is the context window AnythingLLM assumes while assembling your question plus retrieved chunks. Set it above what the model supports and answers come back truncated.
• VECTOR_DB="lancedb" is the embedded option: no extra server, but it is the one that needs AVX2.

Step 3. Let the container reach Ollama

This is the usual stumbling point. On Linux, Ollama listens on 127.0.0.1:11434, which is the host's loopback only. The container has its own network stack, so its 127.0.0.1 is itself and the request never arrives. In the UI this shows up as a provider connection failure against a perfectly healthy Ollama, and half an hour goes into re-testing the part that works.

The fix is to have Ollama listen on all interfaces, via the standard systemd override:

sudo systemctl edit ollama

In the editor that opens, inside the block reserved for your own lines:

[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"

Apply and confirm:

sudo systemctl daemon-reload
sudo systemctl restart ollama
ss -ltnp | grep 11434

What you should see: ss reporting 0.0.0.0:11434 or *:11434. Still 127.0.0.1:11434 means the override was not picked up, so check that the file was saved and repeat the daemon-reload.

The consequence. Ollama now also listens on the server's public interface, and it has no authentication of its own. That is safe exactly as long as the firewall keeps 11434 shut. Confirm it:

sudo ufw status verbose

Nothing in the rule list should mention 11434. Unlike a published Docker port, this one is genuinely filtered: Ollama runs natively, so its traffic goes through the normal chains.

When the other way is simpler. If Ollama is not installed yet and you are happy to containerise it, run it as a second service in the same compose file. The host network then plays no part and nothing has to be opened anywhere:

ollama:
image: ollama/ollama:latest
container_name: ollama
restart: unless-stopped
volumes:
- ./ollama:/root/.ollama

No port is published for it at all, and .env takes the service name instead of the host alias: OLLAMA_BASE_PATH="http://ollama:11434". Compose puts both containers on a shared network where names resolve automatically. Pull a model afterwards with docker compose exec ollama ollama pull llama3.1:8b.

Bring it all up:

cd /opt/anythingllm
sudo docker compose up -d
sudo docker compose ps

ps should show status running and a ports column reading 127.0.0.1:3001->3001/tcp. A status of restarting sends you to sudo docker compose logs -n 50 anythingllm, where at this stage the answer is nearly always volume permissions or AVX2.

Step 4. Turn on auth before anything is reachable

A fresh AnythingLLM asks for no password. Whoever opens the address is an administrator: they can read every workspace, download the uploaded documents and change the model settings. While the port sits on loopback that is academic, but the switch belongs before nginx and a domain enter the picture.

Reach the UI without exposing anything, using an SSH tunnel from your own machine:

ssh -L 3001:127.0.0.1:3001 root@YOUR_IP

For as long as that session is open, http://127.0.0.1:3001 in your browser is the port on the server. Run the first-time setup, pick the provider, and wait until the UI confirms it can see the model. Then open the instance settings and find the security section, which offers two choices:

• Password Protect Instance: one shared password for the whole deployment. Fine when you are the only user, and it can be turned off or reset later.
• Enable multi-user mode: you set a username and password for the first admin account, then create users with roles. admin has everything, manager sees all workspaces but not the LLM, embedder and vector database settings, default gets only what it is granted.

One detail to know before clicking: per the project documentation, multi-user mode is one-way, and you cannot go back to a single-user instance afterwards. If anyone besides you will use this, switch to multi-user immediately, because converting one shared account into proper roles later means moving data by hand.

Step 5. nginx and ufw in front

Here is why the compose file says 127.0.0.1:3001:3001 rather than 3001:3001. When Docker publishes a port it writes iptables rules into chains evaluated before the ones ufw manages. The practical result: ufw default deny incoming is in place, the port is open to the internet anyway, and ufw status reports nothing unusual. It is not a bug, just chain ordering, and people walk into it constantly.

The dependable answer is to never let Docker publish outward in the first place. Binding to 127.0.0.1 does exactly that: the container is reachable from the server itself and invisible from outside, whatever the firewall is doing. From there a single nginx faces the world, and nginx is something ufw filters properly.

The server block is an ordinary reverse proxy plus three additions, without which the UI behaves oddly:

server {
listen 80;
server_name llm.example.com;

client_max_body_size 128M;

location / {
proxy_pass http://127.0.0.1:3001;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;

proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";

proxy_buffering off;
proxy_read_timeout 600s;
}
}

• proxy_buffering off: otherwise nginx accumulates the model's output and delivers it in one lump at the end. Token streaming stops being visible and a slow model looks hung.
• proxy_read_timeout 600s: CPU inference is slow. The 60-second default cuts long answers off with a 504.
• client_max_body_size 128M: the upload ceiling. The nginx default is 1 MB, which almost any real PDF turns into a 413.
• Upgrade and Connection: needed for the websocket connections that agent features use.

Then the usual order: sudo nginx -t first, and only on a clean result sudo systemctl reload nginx. ufw needs rules for SSH, 80 and 443 and nothing more. All of that, including why it is reload and not restart and why the certificate comes as a separate step once plain HTTP works, is covered in nginx and ufw in front of your app.

When it does not work

• The container restarts in a loop. Start with sudo docker compose logs -n 50 anythingllm. A crash mentioning Illegal instruction is the AVX2 problem, so run the CPU check from the resources section. File access errors mean volume ownership, so repeat the chown.
• The UI cannot see Ollama. Check in order: ss -ltnp | grep 11434 shows 0.0.0.0 rather than 127.0.0.1; extra_hosts is present in the compose file; .env holds http://host.docker.internal:11434 with no typo and no trailing /api.
• The model is listed but no answer arrives. Most likely the model is still loading into memory on first use. Check ollama ps on the host, and if the model is loaded with still no response, look at whether the nginx timeout is cutting the request.
• Settings reset after a container update. The .env mount is missing or wrong. Confirm the host path is a file and not a directory Docker created for you.
• Indexing a document takes the server down. The embedder and the model share one pool of RAM. Watch docker stats during an upload, then either index when nobody is chatting or move to a plan with headroom.

FAQ

What is the difference between AnythingLLM and Ollama?

Ollama is the inference engine: weights in RAM, HTTP API out, with no interface, no chat history, no file handling and no accounts. AnythingLLM is the layer above it, providing a browser UI, workspaces that each carry their own documents, ingestion and an embedded vector database for searching those documents, plus access control. Ollama still generates every token; AnythingLLM decides what it receives.

How do you connect AnythingLLM in Docker to Ollama on the host?

Two settings are required together. The compose file needs extra_hosts with host.docker.internal:host-gateway, because that name does not resolve inside a Linux container by default. And Ollama has to listen past loopback, which you set with Environment="OLLAMA_HOST=0.0.0.0:11434" through sudo systemctl edit ollama. Then point OLLAMA_BASE_PATH at http://host.docker.internal:11434 and keep port 11434 closed at the firewall.

How much RAM does AnythingLLM need?

The documented minimum is 2 GB of RAM, a 2-core CPU with AVX2 and 5 GB of storage. That covers AnythingLLM with its vector database and built-in embedder, and excludes the language model, which normally costs more than everything else. Pairing it with a 7-8B model realistically starts at an 8 GB plan.

Where does AnythingLLM store documents and vectors?

Under /app/server/storage inside the container: uploaded documents, LanceDB vector files, chat history and settings all live there. Mount it to a host directory, or the first docker compose down removes every workspace and indexed document. No separate vector database server is needed, since LanceDB is embedded and runs off files on disk.

Why does ufw not block a Docker published port?

Docker writes its own iptables rules when publishing a port, into chains evaluated before the ones ufw controls, so a deny rule there has no effect while ufw status still looks clean. For a service behind a reverse proxy the fix is not to publish on all interfaces at all: write 127.0.0.1:3001:3001 in the compose file. The port then does not exist externally, and every external request comes through nginx, which ufw filters normally.

Can multi-user mode be switched off again?

No. The project documentation states the move to multi-user mode is irreversible, with no path back to a single-user instance. Decide up front: if anyone other than you will use the deployment, enable multi-user before you start creating workspaces rather than after.

Takeaways

• AnythingLLM is an interface and a RAG layer, not an engine. Ollama still does the computing, so response speed follows the engine and the plan rather than the UI.
• Persistence is /app/server/storage plus the mounted .env. Leave either off the host and you lose workspaces and indexed documents the first time the container is recreated.
• Pairing with a native Ollama needs both host.docker.internal:host-gateway in extra_hosts and OLLAMA_HOST=0.0.0.0:11434 on the engine side. Once Ollama listens on all interfaces, verify that 11434 is firewalled, because it has no authentication of its own.
• 2 GB of RAM, two cores with AVX2 and 5 GB of disk is what AnythingLLM asks for. The model is extra, and without AVX2 the embedded LanceDB will not start at all.
• Enable auth before the service is reachable by domain. Instance password for solo use, multi-user for a team, and the second choice cannot be undone.
• Publish on 127.0.0.1. A ufw rule against a published Docker port will not fire, while a loopback binding settles the question without fighting iptables chains.

Where to go next

Engine plus interface is the foundation that the rest of a self-hosted AI stack hangs on. In order:

• Ollama on a VPS: a local LLM with API access is the prerequisite here: installing the engine, choosing a model for the job, sizing RAM against the weights.
• Deploy a service on a server (and keep it running 24/7) covers restart policies, systemd and surviving reboots and crashes.
• nginx and ufw in front of your app has the complete server block, the minimal firewall rule set, and TLS as its own step.