LiteLLM on a VPS: one OpenAI-style API in front of Ollama and hosted models

An LLM gateway is a small server that speaks one API, the OpenAI chat format, and forwards each request to whichever model you named: a local Ollama model, or a hosted one. Your apps keep one address and one key; swapping a model means editing a YAML file. We ran the LiteLLM proxy on a 2 vCPU VPS with no GPU, put two local models behind it, and tested the key check, a fallback, streaming and the Python client.

In short. Install litellm[proxy] into a virtualenv, write a config.yaml with a model_list of aliases, keep the master key in an environment file, and run the proxy from systemd on 127.0.0.1:4000. Apps then call /v1/chat/completions with the alias. It costs about 370 MiB of RAM and roughly 30 ms per request on our box.

Everything below was run on a HIP-HOSTING VPS: Ubuntu 24.04.5, 2 vCPU, 3.8 GB of RAM, no GPU. Ollama 0.40.16 served two small models, qwen2.5:0.5b and llama3.2:1b, and LiteLLM 1.109.1 came from pip. The outputs are copied from the terminal; the master key is replaced with sk-xxxxxxxx.

What an LLM gateway gives you, and when you can skip it

One app talking to one model needs no gateway: point the app at Ollama and stop. A gateway starts to pay off when something changes often or when more than one program asks for answers.

• One address, one key. Every script, bot and n8n workflow calls the same URL with the same header, whatever model sits behind it.
• Aliases. Apps ask for fast or smart. When you replace the model behind smart, no app changes.
• Fallbacks. If the first model fails, the gateway retries on another one and the caller still gets an answer.
• A place for rules. Limits, logging and keys live in one service instead of every script.

If you have not run a local model yet, start with our Ollama on a VPS guide; the gateway sits in front of it.

Install the LiteLLM proxy in a virtualenv

The proxy is a Python program with a lot of dependencies, so give it its own virtual environment: a private folder of packages that cannot break the system Python. The [proxy] part of the name pulls in the web server pieces; openai is the client library we use later for a test.

It took 50 seconds and 753 MB of disk. That is the first number to remember: this is not a tiny tool. On a small disk, plan for it.

Write config.yaml: aliases, a fallback and the master key

Create the folder and the service user first: mkdir -p /etc/litellm and useradd --system --no-create-home --shell /usr/sbin/nologin litellm. Save the config as /etc/litellm/config.yaml. It has three parts. model_list maps the names your apps use (model_name) to the real model behind them (litellm_params). router_settings holds retry and fallback rules. general_settings holds the master key. We create three aliases: fast and smart point to our two local models, and flaky points to a model that does not exist, to test the fallback on purpose.

model_list:  - model_name: fast    litellm_params:      model: ollama/qwen2.5:0.5b      api_base: http://127.0.0.1:11434  - model_name: smart    litellm_params:      model: ollama/llama3.2:1b      api_base: http://127.0.0.1:11434  - model_name: flaky    litellm_params:      model: ollama/no-such-model      api_base: http://127.0.0.1:11434router_settings:  num_retries: 0  fallbacks: [{"flaky": ["smart"]}]general_settings:  master_key: os.environ/LITELLM_MASTER_KEY

Line by line: ollama/qwen2.5:0.5b is "the Ollama provider, this model tag"; api_base is where Ollama listens. num_retries: 0 turns off repeated tries on the same model, so the fallback shows up immediately. The line os.environ/LITELLM_MASTER_KEY tells LiteLLM to read the key from an environment variable, so the secret never sits in the YAML you might paste into a chat or commit to git.

The master key goes into its own file, readable only by root and the service group:

Treat that key as the root password of the gateway: whoever has it can use every model you put behind it.

Run it as a systemd service on localhost

A service started by hand dies with your SSH session. systemd is the system's service manager: it starts the proxy at boot and restarts it if it crashes. We run it as a dedicated user with no shell and bind it to 127.0.0.1, so nothing outside the server can reach it yet.

[Unit]Description=LiteLLM proxyAfter=network-online.target ollama.service[Service]User=litellmEnvironmentFile=/etc/litellm/envExecStart=/opt/litellm/venv/bin/litellm --config /etc/litellm/config.yaml --host 127.0.0.1 --port 4000Restart=on-failureRestartSec=5[Install]WantedBy=multi-user.target

Save it as /etc/systemd/system/litellm.service, then start it and check that it listens where you expect:

Startup took about ten seconds. The log lists your three aliases, which is the quickest way to confirm the config was read.

First requests: the key check and the model list

The proxy has a liveness address that needs no key, and a model list that does. Calling the list without a key should fail, and it does:

The last line is our own one-line Python filter over the JSON reply, trimmed to the model names. Now a real request. The body is the standard OpenAI chat format: a model name and a list of messages. We shorten the JSON reply with a one-line script to the fields that matter, here and in the fallback test below.

It works. This first request took 5 seconds on two CPU cores, and that includes loading the model into memory. Look at the answer, though: it is wrong. A VPS is not "software where multiple physical servers are managed". A 0.5-billion-parameter model is fluent and often mistaken, and the gateway cannot fix that. The smart alias, the 1B model, answered the same question correctly (7.5 seconds, again a first request that loaded the model): "a virtualized computing environment that provides a dedicated server space for users". Pick the model for the job; the gateway only makes switching cheap.

Call it from Python by changing one line

Because the gateway speaks the OpenAI format, the official openai client works against it. You change base_url and the key; the rest of your code stays as it was.

import osfrom openai import OpenAIc = OpenAI(base_url="http://127.0.0.1:4000/v1", api_key=os.environ["LITELLM_MASTER_KEY"])r = c.chat.completions.create(    model="fast",    messages=[{"role": "user", "content": "Name one Linux command that shows free disk space. One line."}],    max_tokens=40,)print(r.choices[0].message.content)

The right command is df, so the model got the main point. The --format flag is invented (the real one is -h). Same lesson as above: never run what a small model suggests without reading it. Our AI agent guide shows how to keep a model's output away from a shell.

Fallback: when one model fails, another answers

This is the feature that justifies the extra moving part. Our flaky alias points at a model Ollama does not have. Called directly, Ollama refuses. Called through the gateway, the request is retried on smart and succeeds:

The caller asked for flaky and got a normal 200 reply, and the model field in the answer tells you who really responded. Note that a normal reply shows the alias you asked for (fast), while this fallback reply showed the real model name. That field is worth logging in your app: a silent fallback is convenient and easy to forget. The same rule works for a real outage, for example a hosted API that is down while a local model is up.

Streaming works too

Chat apps print the answer word by word instead of waiting for the whole text. With "stream": true the gateway passes the chunks through as they arrive. We shortened the lines below: real chunks carry more fields.

What it costs in memory and time

A gateway is extra work on every request, so we measured it. Memory first: the proxy process held about 370 MiB resident, and systemd reported 359 MiB for the service. With both Ollama models loaded, the server showed 1.0 GiB available out of 3.8 GiB. For speed we sent a one-token request 5 times straight to Ollama and 5 times through the gateway:

After the first, slower request, the gateway added about 20 to 35 ms each time. Against a few seconds per answer that is noise; against a tiny request it is visible. The honest summary: expect about 400 MB of RAM and 750 MB of disk for the proxy alone, so we would not put it on a 1 GB server, and we did not test 1 GB or 2 GB. For the models themselves, our measured speed was 22.5 tokens per second for the 0.5B model and 10.0 for the 1B model on these two cores.

Three things that surprised us

1. A wrong key gives HTTP 400, not 401. A missing key returns the clean 401 shown above, but a key that does not match the master key returned {"error":{"message":"No connected db.","type":"no_db_connection",...}} [400]. LiteLLM looks up unknown keys in a database as "virtual keys", and we did not connect one. If your app treats only 401 as "bad key", handle this case too.
2. An unknown alias is a clear 400. model=nope returned "Invalid model name passed in model=nope. Call /v1/models to view available models for your key." Good: typos in model names are easy to spot.
3. The restart is clean. After systemctl restart litellm the health check answered again within 20 seconds, and the unit is enabled for the next boot.

Adding a hosted model, and what we did not test

A hosted model is one more entry in model_list, under a new alias such as hosted. Reusing an existing alias would make LiteLLM spread requests between the two models. The sketch below follows the project's documented format; we did not run it, because it needs your own account key, so check the model name against your provider before you rely on it:

  - model_name: hosted    litellm_params:      model: openai/gpt-4o-mini            # not run here: needs your key      api_key: os.environ/OPENAI_API_KEY

With two entries sharing one model_name, LiteLLM can spread requests across them, but we did not test that either. We also did not set up a database, so per-user keys and spending limits are untested, and we did not use the Admin UI or Docker. Everything in the sections above is what we actually ran.

To make the gateway reachable from other machines, do not just open port 4000. Put a web server with HTTPS in front of it, as in our nginx and ufw guide, and allow only the addresses that need it.

FAQ

What is an LLM gateway or LLM router?

A server between your apps and your language models. Apps send every request in one format to one address; the gateway picks the real model by name, applies rules such as keys and fallbacks, and returns the answer in the same format. LiteLLM's proxy is one open-source way to do it.

Do I need a GPU to run LiteLLM?

No. LiteLLM itself only forwards requests and uses CPU and memory, about 370 MiB of RAM in our test. A GPU matters only for the models behind it, and our two small ones ran on CPU.

Is the LiteLLM proxy free?

We installed it with pip and used it without paying anything. The project also sells some features separately, so read its current licence page before you build a business on a specific feature.

How do I connect an app to the gateway?

Set the app's API base URL to http://127.0.0.1:4000/v1, use the master key as the API key and use an alias such as fast as the model name. Any client that can talk to the OpenAI API format works, as the Python example above shows.

Can the gateway use a hosted API and a local model together?

Yes, that is the usual setup: a local model for private, cheap requests and a hosted one for hard questions, with a fallback between them. We tested the local models and the fallback; the hosted entry above is untested.

Takeaways

• A gateway gives every app one address, one key and stable model names. It is worth it when you have several callers or switch models.
• Run it from systemd, bound to 127.0.0.1, with the master key in a root-owned environment file and not in the YAML.
• A fallback worked as intended: a broken alias answered with HTTP 200 from another model. Log the model field so you notice it.
• It cost 370 MiB of RAM, 753 MB of disk and about 20 to 35 ms per warm request on 2 vCPU.
• A wrong key returned 400 "No connected db", not 401, because we had no database for virtual keys.

Not tested here: hosted APIs, Postgres and virtual keys, balancing across several deployments, 1 GB and 2 GB servers, other systems than Ubuntu 24.04. We ran all of it on a HIP-HOSTING VPS with 2 vCPU and 4 GB of RAM; if you want one for this, see hip.hosting and check the order form for current availability.