18 November 2026

RAG on a VPS without a GPU: Qdrant and Ollama embeddings for questions about your own documents

RAG on a 2 vCPU VPS with no GPU: Qdrant stores the index, Ollama writes the answers. Scripts, timings and three real failures.

RAG is how you make a language model answer from your own documents instead of from memory. We built one on a plain VPS with 2 vCPU and 4 GB of RAM and no GPU: Qdrant stores the search index, Ollama computes the embeddings and writes the answer. Below is everything as it ran: commands, code, timings, and three places where the system went wrong. The failures are more useful than the lucky answers.

In short. Cut the documents into ~800-character chunks, turn each into a 768-number vector with nomic-embed-text, store them in Qdrant. For a question, fetch the 4 to 6 nearest chunks and hand them to a chat model together with the question. On our server: 223 chunks (135 thousand characters) indexed in 121 seconds, a search takes 4 to 6 ms, and a 1B to 3B model needs 25 to 60 seconds per answer. Search is fast; generation is the bottleneck. That is fine for a background helper over your documents and too slow for live chat.

What RAG is and why run it on your own server

RAG (retrieval-augmented generation) has two steps. First the system finds a few fragments of your documents that are close in meaning to the question. Then it passes them to a language model with the instruction "answer from this only". The model learns nothing new: fresh documents are simply pasted into its prompt.

Why not plain text search? A keyword search won't find "how do I keep a service from running without a password" if the document says "authentication through a reverse proxy". A meaning-based search will. And a VPS of your own makes sense when the documents can't go to an outside service: internal instructions, contracts, customer emails.

Two terms you'll need. An embedding is a vector of hundreds of numbers that a model builds for a piece of text so that texts with similar meaning get similar vectors. A vector database (Qdrant here) stores those vectors and quickly finds the ones nearest to the vector of your question.

What you need

  • A VPS with Ubuntu 24.04. We used 2 vCPU, 4 GB of RAM and 40 GB of SSD (the Standard line). No GPU.
  • Ollama. If you don't have it yet, start with our guide Ollama on a VPS: install, service and port protection.
  • Docker for Qdrant. Qdrant also runs as a plain binary, but a container is easier to update and to throw away.
  • Python 3. The scripts below use only the standard library, so nothing to install.

A word on memory up front. In our run Qdrant with 223 points took 26 MB, the embedding model 389 MB in memory, and a 3-billion-parameter chat model 2.2 GB. Together they nearly filled the 4 GB, leaving about 700 MB of headroom. With 8 GB life is much calmer.

Step 1. Two models in Ollama

We need two different models. The first, nomic-embed-text, doesn't write text; it only turns text into a vector, and it is small and fast. The second is an ordinary chat model that phrases the answer. To start we take a very light one, llama3.2:1b.

SIZE is the file size on disk. In memory a model takes a bit more: the embedding model showed 389 MB for a 274 MB file.

Step 2. Qdrant in Docker, for your eyes only

Start Qdrant and bind the port to 127.0.0.1 right away. This matters. Docker publishes ports around the ufw rules, so without an explicit address the database ends up open to the whole internet, with no password. Anyone could read and wipe your documents.

The -v flag keeps the data on the server's disk, so the index survives if you recreate the container. A reply with the version means the database is alive. Empty, it takes about 22 MB.

Step 3. Indexing: cut, compute vectors, store

The script below reads every .txt from a folder, cuts it into chunks, computes a vector for each and sends it to Qdrant. What to understand about it:

  • Chunk size is 800 characters with a 120-character overlap. Chunks that are too small lose their meaning; chunks that are too big blur the vector and crowd the model's prompt. We cut at paragraph boundaries, not mid-sentence.
  • Dimension 768 is how many numbers nomic-embed-text produces. The collection must be created with the same size.
  • The search_document: prefix. According to the model's description it is trained to tell a document from a question by such a prefix. The question gets search_query: later. Without them, according to the model's description, search works worse.
  • Cosine distance (Cosine) compares the direction of vectors, not their length. It is the usual choice for text embeddings.
#!/usr/bin/env python3# index.py: index a folder of .txt files into Qdrant with Ollama embeddingsimport json, os, sys, time, urllib.request, globOLLAMA = "http://127.0.0.1:11434"QDRANT = "http://127.0.0.1:6333"COLL = "docs"DOCS = sys.argv[1] if len(sys.argv) > 1 else "/root/ragdemo_docs"SIZE, OVERLAP = 800, 120def post(url, body, method="POST"):    req = urllib.request.Request(url, data=json.dumps(body).encode(), method=method,                                 headers={"Content-Type": "application/json"})    return json.load(urllib.request.urlopen(req, timeout=600))def chunks(text):    parts, cur = [], ""    for p in text.split("\n\n"):        p = p.strip()        if not p:            continue        if len(cur) + len(p) + 2 > SIZE and cur:            parts.append(cur)            cur = cur[-OVERLAP:] + "\n\n" + p        else:            cur = (cur + "\n\n" + p) if cur else p    if cur:        parts.append(cur)    return partsitems = []for path in sorted(glob.glob(os.path.join(DOCS, "*.txt"))):    name = os.path.basename(path)[:-4]    for i, c in enumerate(chunks(open(path, encoding="utf-8").read())):        items.append((name, i, c))print("files:", len(set(i[0] for i in items)), "chunks:", len(items))try:    post(f"{QDRANT}/collections/{COLL}", {}, method="DELETE")except Exception:    passpost(f"{QDRANT}/collections/{COLL}", {"vectors": {"size": 768, "distance": "Cosine"}}, method="PUT")t0 = time.time()B = 16for s in range(0, len(items), B):    batch = items[s:s + B]    emb = post(f"{OLLAMA}/api/embed", {"model": "nomic-embed-text",                                        "input": ["search_document: " + b[2] for b in batch]})["embeddings"]    pts = [{"id": s + j, "vector": emb[j], "payload": {"doc": b[0], "n": b[1], "text": b[2]}}           for j, b in enumerate(batch)]    post(f"{QDRANT}/collections/{COLL}/points?wait=true", {"points": pts}, method="PUT")    print(f"  {min(s + B, len(items))}/{len(items)}", flush=True)dt = time.time() - t0print(f"done in {dt:.1f} s, {len(items) / dt:.1f} chunks/s")

Note the payload: next to each vector we store the chunk's text and the file name. Qdrant returns closeness scores, but the model needs the actual text. The script recreates the collection on every run, so updating the base just means running it again.

As documents we used six of our own blog articles: on Ollama, the OOM killer, zero-downtime deploys, VPS versus VDS, LiteLLM and AnythingLLM.

Here is the first number to remember: 1.8 chunks per second. Two minutes for 135 thousand characters is fine for a hundred pages and not for a gigabyte of documents, so do the math before you start. Language matters too. The Russian versions of the same six articles (234 chunks) took 322 seconds on the same server, 0.7 chunks a second: Russian apparently splits into noticeably more tokens, and the same model chews through it about 2.5 times slower.

Step 4. A question: search, then answer

The second script does three things: it turns the question into a vector (with the search_query: prefix), asks Qdrant for the nearest chunks, and builds a prompt for the chat model from them. The key line is the instruction "answer only from the fragments; if the answer isn't there, say so". Without it the model happily answers from memory.

#!/usr/bin/env python3# ask.py: question -> search in Qdrant -> answer from the retrieved chunksimport json, os, sys, time, urllib.requestOLLAMA = "http://127.0.0.1:11434"QDRANT = "http://127.0.0.1:6333"q = sys.argv[1]MODEL = sys.argv[2] if len(sys.argv) > 2 else "llama3.2:1b"TOP = int(os.environ.get("TOP", "4"))def post(url, body):    req = urllib.request.Request(url, data=json.dumps(body).encode(), headers={"Content-Type": "application/json"})    return json.load(urllib.request.urlopen(req, timeout=900))t = time.time()v = post(f"{OLLAMA}/api/embed", {"model": "nomic-embed-text", "input": ["search_query: " + q]})["embeddings"][0]t_emb = time.time() - tt = time.time()hits = post(f"{QDRANT}/collections/docs/points/search", {"vector": v, "limit": TOP, "with_payload": True})["result"]t_search = time.time() - tprint(f"question embedding: {t_emb * 1000:.0f} ms, Qdrant search: {t_search * 1000:.0f} ms")for h in hits:    p = h["payload"]    print(f"  {h['score']:.3f}  {p['doc']} #{p['n']}")context = "\n\n---\n\n".join(h["payload"]["text"] for h in hits)prompt = ("Answer only from the fragments below. If the answer is not in them, say so.\n\n"          f"Fragments:\n{context}\n\nQuestion: {q}\nAnswer:")t = time.time()r = post(f"{OLLAMA}/api/generate", {"model": MODEL, "prompt": prompt, "stream": False,                                    "options": {"temperature": 0.2, "num_ctx": 4096}})print(f"\nANSWER ({MODEL}, {time.time() - t:.1f} s, {r['eval_count']} tokens, "      f"{r['eval_count'] / (r['eval_duration'] / 1e9):.1f} tok/s):")print(r["response"].strip())

A temperature of 0.2 makes answers less "creative", and num_ctx sets how much text the model can read in one go. Now the first question, about the OOM killer, whose answer is definitely in the documents:

Search did its job: all four chunks come from the article on the OOM killer. The answer is a refusal, and a wrong one. The text was in the prompt; the 1-billion-parameter model failed to use it. That is failure number one.

Timings on 2 vCPU

Stage

What we got

Embedding of one question

103 to 134 ms (once 885 ms with a cold model)

Qdrant search over 223 chunks

4 to 6 ms (once 141 ms)

Indexing

1.8 chunks/s (121 s for 223 chunks)

Answer from llama3.2:1b

25 to 27 s, 7 to 8 tokens/s

Answer from qwen2.5:3b

46 to 59 s, 4.8 to 5.1 tokens/s

Memory, 3B + embedding + Qdrant

about 3.1 of 3.8 GB, about 700 MB left

The conclusion is simple: search is the cheapest part. You pay for generation, and the bigger the model and the more chunks you hand it, the longer you wait. A 3-billion-parameter model answers in about a minute. That is a "ask and go make tea" mode, not a chat.

Where RAG breaks: three failures we hit

Failure 1. The answer is in the prompt and the model ignores it

You have just seen it: right chunks, wrong refusal from a 1B model. The next question went to a 3B model, which handled it:

This is a correct answer, taken from our Ollama article, and the command is a real, working SSH tunnel. If you can afford the extra seconds, a bigger model is the first thing to try when answers look dumb. Even so, show the user the source chunks next to the answer: the script already prints file names and chunk numbers.

Failure 2. A question with no answer in the documents

We asked something the documents can't know: "What colour is the company logo?" The instruction says to admit it when the answer isn't there:

The model invented an answer. In the Russian run the same model answered "no answer in the fragments" to the same question, so this is luck, not a guarantee. The useful signal is in the numbers: the best score here is 0.653, while questions that do have answers scored 0.75 to 0.80 in the top four. A cut-off around 0.70 would have stopped this one before it reached the model:

The script probe.py prints the top results with a marker on the chunk that contains the phrase you're looking for. Use it to see where the right chunk lands for your own questions, and set the threshold from your numbers, not ours: scores depend on the model, the language and the documents. Treat the threshold as a coarse filter, not as proof. In our Russian run the right and wrong chunks sat in the same narrow 0.75 to 0.79 band, and there the cut-off would not have helped.

Language changed the picture in another way. On the Russian articles the question about Ollama missed: the right chunk was 6th and the 3B model improvised a Caddy config that Caddy rejected. On the English articles the same question hit on the first try. If your documents are not in English, test with more chunks and with your own questions.

Failure 3. The kernel killed the model

When we ran a series of questions against different models in a row, one request broke with Remote end closed connection without response, and the requests right after it got Connection refused while the service came back up. The kernel log had the reason:

The OOM killer is the kernel mechanism that kills the heaviest process when memory runs out. It killed llama-server, the process that runs the model. systemd restarted the Ollama service by itself a few seconds later, but the request was lost. What happened: Ollama keeps a loaded model in memory for about 5 minutes after the last request. We queried a small model and then a bigger one straight away, and both ended up sharing 4 GB with the embedding model and Qdrant.

The cure is discipline. Before switching models, unload the previous one with ollama stop model_name (that is what we did between runs) and check what is loaded with ollama ps. The /api/generate request also has a keep_alive parameter: 0 unloads the model right after the answer. Our guide on the OOM killer and load average shows how to catch it in the act.

When a VPS without a GPU is enough, and when it isn't

  • Enough for a personal or team knowledge base of a few hundred pages, for background jobs ("sort incoming mail against the documentation", "draft a reply"), and for a prototype before you buy a bigger server.
  • Not enough for live chat with several people: one generation keeps both cores busy for a minute, so a queue grows. Also not for big corpora: at 1.8 chunks a second, a gigabyte takes days.
  • What to change first: a smaller model (faster but dumber), fewer chunks (faster but more misses), or a bigger plan with more cores and memory.

If you need a single gateway in front of several models, see our article on LiteLLM on a VPS: the RAG script can talk to it instead of calling Ollama directly.

Security: what not to leave open

Neither Qdrant nor Ollama has a password by default. Keep both on 127.0.0.1, as in the steps above. If the script has to run from another computer, forward the port over an SSH tunnel or put a reverse proxy with a password in front. And don't store anything in the index that you wouldn't show everyone with access to the server: it keeps the texts in full.

FAQ

Do I need a GPU for RAG?

No, our measurements were made without one. A GPU speeds up generation, not search, and you need it if answers must arrive in seconds instead of a minute.

Why Qdrant instead of just keeping vectors in a file?

For a few hundred chunks a file of vectors compared in memory is enough. Qdrant earns its place when there are many documents and they change: it keeps the index on disk, searches fast and supports filters on fields such as the file name.

Which embedding model should I use?

We used nomic-embed-text. It worked well on the English articles and less well on the Russian ones, but two or three questions are not a benchmark. Take 20 to 30 typical questions of your own and see at which rank the right chunk appears.

How do I refresh the index when documents change?

In our version, run index.py again: it recreates the collection from scratch. For big bases it is better to update only the changed files and keep a hash for each.

In short

  • RAG on a VPS without a GPU works: search in 4 to 6 ms, indexing 1.8 chunks a second, an answer in 25 to 60 seconds.
  • Bind Qdrant and Ollama to 127.0.0.1: Docker publishes ports around ufw, and neither has a password.
  • The main risk is quality, not speed: a small model can refuse with the answer in front of it, or invent one when there is none.
  • Use the search score as a coarse filter, show the source chunks next to every answer, and verify commands in answers against the original documentation.
  • Two models don't coexist in 4 GB: unload the old one with ollama stop before loading another.
PUBLISHED
18 November 2026
AUTHOR
HIP-HOSTING
LANGUAGES
EN · RU