HAProxy load balancer on Ubuntu 24.04: round robin, health checks and failover

A load balancer is a program that stands in front of several copies of your application and hands each incoming request to one of them. If one copy dies, it stops sending traffic there and your users notice nothing. HAProxy is a free, open-source balancer that does this job well and fits on the smallest VPS. Below we install it, point it at three backends, kill one on purpose and watch it route around the dead one.

TL;DR. sudo apt install haproxy gives you HAProxy 2.8 on Ubuntu 24.04. Add a frontend (where requests arrive) and a backend (the servers behind it) to /etc/haproxy/haproxy.cfg. Turn on option httpchk and check on each server so dead ones are removed automatically. Always run haproxy -c -f /etc/haproxy/haproxy.cfg before systemctl reload haproxy. One balancer is itself a single point of failure, so decide how much that matters to you.

We ran everything on a test VPS: Ubuntu 24.04.5, HAProxy 2.8.30 from the standard apt repository. To keep the lab on one machine, the three "application servers" are tiny Python scripts listening on ports 9001, 9002 and 9003 of 127.0.0.1. In real life they are separate servers or containers. Each stub answers every path with its own name and returns 200 on /health (the plain python3 -m http.server would not do: it answers 404 there and the server would be marked DOWN). Outputs below are from that run. TLS and the load-balancing methods other than round robin we did not test here, and we say so where it matters.

What a load balancer is for

Two reasons, and they are different problems.

• Capacity. One process can only handle so many requests. Several copies behind a balancer share the load.
• Survival. Servers crash, run out of memory, get rebooted for updates. With a balancer in front, a dead copy is skipped and you update the copies one at a time.

If you only want "my site is down for a minute when I deploy", you may not need a balancer at all; a reverse proxy with a restart script is enough. Start with a balancer when you really have, or soon will have, more than one copy of the application.

Install HAProxy

Ubuntu ships HAProxy in its own repository, so no extra sources are needed. The middle command installs it, the last prints the version so you know what you are reading the documentation for:

sudo apt update

sudo apt install -y haproxy

haproxy -v

On our server the first line of output began with HAProxy version 2.8.30-0ubuntu0.24.04.2. The package starts the service right away, but its default config listens on nothing, so at this point it does no harm. systemctl is-active haproxy printed active.

The config: frontend, backend, health check

HAProxy reads one file, /etc/haproxy/haproxy.cfg. The package puts global and defaults sections at the top; leave them alone and append your own sections below. Here is what we added:

frontend web

    bind 127.0.0.1:8080

    mode http

    option forwardfor

    default_backend app

backend app

    mode http

    balance roundrobin

    option httpchk GET /health

    http-check expect status 200

    default-server inter 2s fall 2 rise 2

    server app1 127.0.0.1:9001 check

    server app2 127.0.0.1:9002 check

    server app3 127.0.0.1:9003 check

Line by line:

• frontend web: the door. bind 127.0.0.1:8080 is the address and port it listens on. On a real server you would write bind *:80. We bound to localhost so the lab is not reachable from outside.
• mode http: HAProxy understands HTTP requests, not just raw TCP streams. Needed for forwardfor, headers and HTTP routing.
• option forwardfor: adds an X-Forwarded-For header with the client address, because otherwise every backend sees only the balancer's IP.
• default_backend app: send everything to the group named app.
• balance roundrobin: take servers in turn, 1, 2, 3, 1, 2, 3.
• option httpchk GET /health with http-check expect status 200: every few seconds HAProxy requests /health from each server. Anything other than a 200 answer counts as a failure. If your app uses virtual hosts, the check may need a Host header; see http-check send in the documentation. Give your app a cheap /health page that checks what really matters, for example that the database connection is alive.
• default-server inter 2s fall 2 rise 2: check every 2 seconds; two failures in a row take a server out; two successes put it back. inter 2s and rise 2 are the defaults; the default for fall is 3, and we set 2 to speed up the demo. For a busy app you may raise the interval.
• check at the end of each server line: without this word the server is never checked, and HAProxy keeps sending requests to a dead one. This is a common omission.

Before restarting anything, ask HAProxy to validate the file. It does not touch the running service:

haproxy -c -f /etc/haproxy/haproxy.cfg

You want Configuration file is valid. Then apply it:

sudo systemctl restart haproxy

ss -ltnp | grep haproxy

The ss output listed 127.0.0.1:8080 owned by haproxy. The balancer is up.

Round robin in action

Each backend answers with its own name. We sent six requests:

for i in 1 2 3 4 5 6; do curl -s http://127.0.0.1:8080/; done

The replies came in the exact order app1, app2, app3, app1, app2, app3. Round robin is that simple: no memory of who the client is, just the next server in the list. If your app keeps a login session in the memory of a single process, that is a problem, because the next request may land on a different copy. Either store sessions in a shared place (a database, Redis or Valkey) or look at the balance source and cookie-based stickiness options in the HAProxy documentation. We did not test those.

Kill one backend and watch

We stopped the second application process and waited six seconds, three times the check interval. Then six more requests (we trimmed the tail , X-Forwarded-For=… from each line for brevity):

hello from app1

hello from app3

hello from app1

hello from app3

hello from app1

hello from app3

Not one failed request, not one reply from app2. HAProxy took it out of rotation by itself. Its statistics show why. We pulled them in CSV form with curl -s 'http://127.0.0.1:8404/stats;csv' (the listen stats section in our config, see below) and kept five columns (we dropped the row for the backend itself):

app app1 UP L7OK 0

app app2 DOWN L4CON 2

app app3 UP L7OK 0

• UP / DOWN: current state.
• L7OK: the last check passed at application level (HTTP 200). L4CON: the check could not even open a connection; here, "connection refused".
• the last number: failed checks counted while the server was up. It is 2 because of fall 2: after the second failure the server went DOWN.

The same story is in the system log. journalctl -u haproxy contains a line such as Server app/app2 is DOWN, reason: Layer4 connection problem, info: "Connection refused". Put that line, or the stats page, behind an alert.

Then we started app2 again. After two successful checks it returned to the rotation, and the six replies again contained all three names.

The stats page

Append this section to the same file, then run the check and reload described below:

listen stats

    bind 127.0.0.1:8404

    mode http

    stats enable

    stats uri /stats

    stats refresh 5s

It serves a live table of every server: state, requests, errors, response times. We only read its CSV with curl. The HTML version opens in a browser at /stats; since we bound it to localhost, reach it from your computer through an SSH tunnel (ssh -L 8404:127.0.0.1:8404 user@your-server). Do not bind it to a public address without a password: the page tells strangers how your backend is built.

Weights: a stronger server takes more

Servers are rarely equal. We gave app1 weight 3 (the default is 1) and reloaded. Out of ten requests the counts were:

      6 hello from app1

      2 hello from app2

      2 hello from app3

That is 3 : 1 : 1, as asked. The reload command matters here:

haproxy -c -f /etc/haproxy/haproxy.cfg && sudo systemctl reload haproxy

reload starts a new process with the new config and lets the old one finish its open connections, so users are not cut off. The && means the reload happens only if the check passed. Another method, balance leastconn (send to the server with the fewest active connections), exists for long-lived connections; it is described in the HAProxy documentation, and we did not test it.

What happens when the config is wrong

We changed roundrobin to roundrobinn in a copy of the file and checked it:

[ALERT] (36471) : config : parsing [/tmp/hap_bad.cfg:44] : balance only supports 'roundrobin', 'static-rr', 'leastconn', 'source', 'uri', 'url_param', 'hdr(name)' and 'rdp-cookie(name)' options.

[ALERT] (36471) : config : Error(s) found in configuration file : /tmp/hap_bad.cfg

[ALERT] (36471) : config : Fatal errors found in configuration.

The exit code was 1, the file name and line number are in the message, and the live service was untouched because we only checked a copy. That is why we run -c first.

When every backend is down

We stopped all three. The balancer has nobody to ask, so the client receives:

HTTP/1.0 503 Service Unavailable

and the log says backend 'app' has no server available!. HAProxy itself is alive, but the answer is an error. You can show a friendlier page with errorfile 503 (see the documentation), or add a backup keyword to a server that should take traffic only when the others are gone.

The client address and a caution

With option forwardfor the backend gets the client address in X-Forwarded-For. In our lab every client was localhost, so the header read 127.0.0.1. Then we sent our own header, X-Forwarded-For: 203.0.113.7, from curl, and the application printed the forged value first. So a client can invent that header. Make your application, or the web server in front of it, trust the address added by your own balancer, not the first value in the list. On an outer balancer you can drop the client's own header first with http-request del-header X-Forwarded-For.

HTTPS

The documentation describes TLS termination with a line like bind *:443 ssl crt /etc/haproxy/certs/site.pem, where the file holds the certificate and the key together. We did not set up a certificate in this lab, so we do not claim a tested recipe. If you want to avoid manual certificate handling, a Caddy or Nginx front may be simpler, with HAProxy behind it or instead of it.

The balancer is a single point of failure

If the VPS with HAProxy goes down, nothing reaches the healthy backends. Three ways to live with it: accept it (a small project), keep the balancer config trivial so you can rebuild it in ten minutes on a fresh server, or run two balancers and switch the address in DNS or by a shared IP. We tested none of the last option here. For most one-person projects, a balancer that is just a small, rarely changed server is good enough, and it is usually more reliable than the application behind it, because it has few moving parts.

A sensible size for the balancer server

HAProxy is frugal. For plain HTTP forwarding the smallest plan is usually enough (we did not measure the load); the load grows with the number of connections and with TLS work, so watch CPU when you add HTTPS. Keep the balancer and the backends in the same location, so a request does not cross a continent twice on its way. We did not measure the extra hop.

FAQ

HAProxy or Nginx as a load balancer?

Both can do it. Nginx is a web server first and a balancer second; the open-source version checks servers only passively, by noticing failed requests. HAProxy is built for balancing, checks servers actively (as we showed) and can serve a stats page. If you already run Nginx and only need simple balancing, Nginx is fine.

How do I update the application without downtime?

Deploy to one backend while the others serve traffic, check that its /health answers 200, then move on to the next. With the checks on, HAProxy puts a server back by itself after the rise number of successes. A related recipe is in running several services together with Docker Compose.

Does the balancer need a lot of memory?

No. Memory depends on the number of simultaneous connections; test with your own traffic before settling on a size.

Can HAProxy balance something other than websites?

Yes, with mode tcp it forwards any TCP traffic, for instance a database or any other TCP service. We covered only mode http here and tested nothing else.

Why not just use DNS with several addresses?

DNS round robin hands out addresses without knowing which servers are alive, and clients cache the answer. A dead server keeps receiving a share of requests until you edit DNS and the old records expire. A balancer with checks, with our test settings, reacts in seconds.

Takeaways

• Install with apt install haproxy; on Ubuntu 24.04 that is version 2.8.
• Minimum config: a frontend, a backend, balance roundrobin and check on every server.
• Add option httpchk GET /health so a server that answers badly is removed, not only one that is switched off. In our test a stopped backend left the rotation within seconds and came back by itself.
• Run haproxy -c -f ... before every reload; use reload, not restart, on a live service.
• Do not trust X-Forwarded-For blindly, and do not expose the stats page.
• One balancer is one point of failure. Decide on purpose.

What's next

If your backends are containers, Docker Compose shows how to run several services together, and this guide helps when images and logs start eating the disk of a backend.