Diagnosing a slow Linux server: what to check when everything drags

When a Linux server is slow, the fastest path to an answer is measurement, not guessing. This is a triage order for Ubuntu 22.04, 24.04, 26.04 and Debian: get the whole picture with uptime and top first, then split by symptom (CPU, memory and swap, disk I/O, network). It also covers how to read load average, what CPU steal and inodes are, and what to attach to a support ticket if the problem turns out to be the host rather than your code.

TL;DR. Start with uptime (load average) and top or htop. High load with a busy CPU: check ps aux --sort=-%cpu | head and pidstat 1 5. High load with an idle CPU: look for processes in the D state (ps -eo state,pid,comm,wchan:32 | grep '^D') and check the disk with iostat -xz 1 (columns %util, r_await). Memory: free -h (the available column), vmstat 1 (si/so for active swap), sudo dmesg -T | grep -i oom to see whether the OOM killer took a process. Network: ss -s and sudo ss -tlnp. Disk shows free space in df -h but writes fail: check inodes with df -i. Several tools are not installed by default: sudo apt install sysstat iotop htop.

Linux server slow: how to diagnose it in the first two minutes

uptime prints three load-average numbers and how long the box has run without a reboot. top is a live process table plus summary lines for CPU and memory. htop, if installed, is the same thing with colour, a process tree and mouse support; a fresh server usually does not have it.

uptime
nproc
top

• uptime: three load-average values (1, 5 and 15 minutes) and system uptime.
• nproc: the number of available cores (vCPUs). This is the number you compare load against.
• top: interactive, quit with q. Useful keys: P sorts by CPU, M by memory, 1 breaks the load out per core.

What you should see. The top header has a %Cpu(s) line with fields us, sy, id, wa, st, plus MiB Mem and MiB Swap lines. Read the %Cpu(s) line first: it tells you straight away whether you are bound on computation (us), on the kernel (sy), on disk (wa), or losing CPU to the hypervisor (st).

Which tools are missing on a fresh system

ss, journalctl, top, free, vmstat, ps, df, du, dmesg and swapon are usually already there in a standard Ubuntu or Debian image (a heavily stripped cloud or container image may lack even these). But iostat, pidstat and mpstat ship in the sysstat package, and iotop, htop and (for a traffic graph) nload or iftop are separate packages.

sudo apt update
sudo apt install sysstat iotop htop

nload and iftop install the same way, as needed. You do not have to pull the whole list up front: install what the symptom calls for.

Load average: why load 20 with an idle CPU is not a paradox

Load average is a moving average, over 1, 5 and 15 minutes, of the number of threads that are either running on a CPU, waiting in the run queue for one, or in uninterruptible sleep. Uninterruptible sleep (the D state) means a process is stuck in a system call, usually on disk or network I/O, and will not respond even to kill -9 until that operation finishes. The key point: on Linux, load counts the D state as well, not just CPU demand. So load 20 with a nearly idle processor is not a broken gauge. It is twenty processes stuck waiting on a slow disk or a slow network filesystem.

Reading the three numbers: the 1-minute value is now, the 15-minute value is the trend. If the 1-minute figure is well above the 15-minute one, load is rising; below it, load is falling. As a rough scale: a sustained 1-minute load near your core count (nproc), when that load is mostly runnable CPU tasks, means every core is busy with no large queue (cross-check the %Cpu(s) line in top: a large wa means the CPU is not the issue), while a load well above the core count means processes are queuing, and you need to work out what they are queuing for: CPU (large us or sy in top) or I/O (large wa, processes in the D state).

An illustrative line (numbers for illustration only): %Cpu(s): 2.0 us, 1.3 sy, 0.0 ni, 8.0 id, 88.0 wa, 0.0 hi, 0.7 si, 0.0 st. The CPU sat idle almost the whole interval with I/O outstanding. Load is high, and the CPU is not the cause. A large wa is a strong reason to check the disk, but not proof it is the disk: the D state also covers a network filesystem. List the stuck processes with ps -eo state,pid,comm,wchan:32 | grep '^D'.

CPU: what is eating the processor, and what steal means

top has already told you where CPU time goes. The %Cpu(s) fields:

• us (user): application code in user space. High us means some program is genuinely computing.
• sy (system): kernel work such as system calls, context switches and network handling. High sy with low us points at an unusual number of system calls (sometimes bad code, sometimes a packet storm).
• wa (I/O wait): the share of time the CPU sat idle with an I/O operation outstanding. This is forced idle, not CPU load, and on a multi-core box it is a rough indicator, not a precise "fraction of time waiting on disk". A large wa sends you to the disk section.
• st (steal): time your VM was ready to run but the hypervisor did not schedule it, spending that time on other work on the host. Short spikes of st are not a problem in themselves; sustained st that lines up with a performance dip means the host is oversubscribed. Values around 10% and up are especially noticeable for CPU-bound work, but there is no universal threshold.

ps aux --sort=-%cpu | head
pidstat 1 5

• ps aux --sort=-%cpu | head: top processes by CPU. Caveat: the %CPU figure here is an average over the whole lifetime of the process, so it understates a recent spike. For a live view, top or pidstat is more accurate.
• pidstat 1 5: five one-second samples, per-process CPU over time. %usr versus %system is the same split as us/sy, but per process.

If it is st that is high rather than us, the problem is not your code: the hypervisor is not scheduling your vCPU often enough, usually because the host is oversubscribed, and that is a question for the host or the plan (dedicated cores, a guaranteed-CPU tier). Capture mpstat 1 over a few minutes and attach it to the ticket.

Memory and swap: when "everything is slow" is swap

free -h is the first check for memory. The column that matters is available: how much memory applications can take without paging to swap. The buff/cache column should not alarm you; it is the filesystem cache, and the kernel hands it back when applications need it.

Swap (a swap partition or file) is not bad in itself. The problem is swap thrashing (constant paging): RAM runs short and the kernel continuously moves pages between memory and disk. Memory technically exists, but every access to it goes through the disk, and the server slows down by a large factor. From the outside this looks like "the whole box is slow".

free -h
vmstat 1 5
swapon --show

• free -h: read available, not free.
• vmstat 1 5: the si and so columns (pages swapped in/out, KiB/s). Zeros mean no swap I/O is happening right now; swap itself can still be full - check the amount in the swpd column, in free -h or with swapon --show. Steady non-zero si/so means active paging, which is your slowdown. Other columns: r is the CPU run queue, b is processes in uninterruptible sleep. The first vmstat report is an average since boot; read from the second, or run vmstat -y 1 5 to skip it.
• swapon --show: whether swap exists and how much is used. Empty output means no swap is configured; a memory shortage then first makes the kernel reclaim page cache and other reclaimable memory, and only invokes the OOM killer if that is not enough.

If a process vanished with nothing in its own logs and no stack trace, the OOM killer probably took it: a kernel mechanism that fires when a memory request cannot be met within the limits available to the process. That is either a global RAM-and-swap shortage or a cgroup / container memory limit (memory.max) - in which case the OOM happens inside the container even if the host still has free RAM. The kernel picks the victim by a "badness" score, which oom_score_adj also influences.

sudo dmesg -T | grep -i -E 'oom|killed process'
journalctl -k -b | grep -i 'out of memory'

• dmesg -T: the kernel ring buffer; -T renders timestamps in a readable form. It needs sudo: reading dmesg is restricted by default on Ubuntu.
• journalctl -k -b: kernel messages for the current boot. A line like Out of memory: Killed process 1234 (mysqld) is your answer: which process and when.

Disk: I/O, a full partition and exhausted inodes

Two separate questions here: the disk is too slow, or the disk is out of space (or out of inodes).

Speed. iostat -xz 1 gives extended per-device stats; -z hides idle devices. The first report is an average since boot, so read the second one and later.

iostat -xz 1
sudo iotop -o
pidstat -d 1 5

• iostat -xz 1: the %util column is the share of time the device had at least one operation in flight. For a plain disk, close to 100% means saturation; for NVMe, %util is misleading (the device handles hundreds of parallel operations and shows 100% well short of its limit), so look at r_await / w_await (average read/write time in milliseconds, including queue time) and aqu-sz (average queue length). Tens of milliseconds on an SSD is suspicious.
• iotop -o (the iotop package, run as root): -o shows only processes that are actually reading or writing right now. It shows per-process read and write throughput with no extra setup; the I/O-wait percentage column and SWAPIN, however, rely on delay accounting, which is off by default on kernel 5.14+ (every current Ubuntu and Debian). If you need those columns, enable it temporarily with sudo sysctl kernel.task_delayacct=1, then set it back to =0 after - there is no point leaving it on. A simple no-setup alternative is pidstat -d below.
• pidstat -d 1 5: a sysstat alternative to iotop with kB_rd/s and kB_wr/s per process.

Space. df -h shows partition usage in human units. If a partition is near 100%, find what grew:

df -h
sudo du -xh --max-depth=1 / | sort -h

• du -xh --max-depth=1 /: the size of each top-level directory; -x stays on one filesystem (otherwise you count /proc and mount points). Repeat deeper into the largest directory. Usual culprits: /var/log, caches, core dumps, forgotten backups.

Inodes. Sometimes df -h shows free gigabytes yet a write fails with No space left on device. The most common cause is inode exhaustion: inodes are filesystem records, one per file or directory (they hold permissions, owner, timestamps, pointers to data). Less often it is a disk quota or a container limit. On ext4 the inode count is fixed when the filesystem is created and does not grow; XFS allocates them dynamically and rarely hits this.

df -i

• df -i: like df -h but for inodes. IUse% at 100% with free space is the classic picture: millions of tiny files (sessions, mail, cache, a stuck job queue). From there use du --inodes or find per directory to locate the source.

Network and connections: what is listening, and where the flood comes from

ss is the standard netstat replacement and needs no install.

ss -s
sudo ss -tlnp
ss -tn state established | wc -l
ss -tn state time-wait | wc -l

• ss -s: a summary of total sockets and TCP counts per state.
• ss -tlnp: which TCP ports are listening and which process owns them (t TCP, l listening, n numeric ports, p process; p needs root). This is the first check when a service does not respond: is it listening at all, and on the right address (127.0.0.1 versus 0.0.0.0).
• ss -tn state established | wc -l: the number of established connections.
• ss -tn state time-wait | wc -l: connections in TIME-WAIT (a normal state for about a minute after close). Tens of thousands means high connection churn; on a busy public service that can be normal. It is a concern if you are also running out of ephemeral ports or file descriptors, or latency is rising - then look at keep-alive, database connection pools and the traffic pattern.

For real-time throughput, nload and iftop do the job (both are separate packages; iftop runs as root).

Logs: journalctl, dmesg and where to find a service that died

When a service falls over, the answer is almost always in the logs.

journalctl -p err -b
journalctl -u service-name -e
journalctl -f
sudo dmesg -T | tail -n 50

• journalctl -p err -b: messages at error priority and worse for the current boot (-b). A quick way to see what complained.
• journalctl -u service-name -e: the log for one service, -e jumps to the end. The name matches systemctl status (for example nginx, ssh, docker).
• journalctl -f: a live tail; reproduce the problem and watch what appears.
• dmesg -T: for hardware, disk or network issues at the kernel level, such as filesystem errors, a NIC reset, or the OOM killer.

You can also see systemctl and journalctl in use in our guide to setting up the Caddy web server.

Symptom, first command, what the number means, likely cause

sudo dmesg -T | grep -i oom

df -i

sudo ss -tlnp

What to attach to a support ticket

If you have narrowed the problem down to the host (high st, odd disk errors in dmesg, network dropping without a traffic increase), gather the facts once. It saves a round trip:

• the time window, and what "slow" means: a specific request, an SSH login, or the whole machine;
• uptime and nproc;
• top -b -n1 | head -20, a non-interactive snapshot;
• free -h, plus df -h and df -i;
• iostat -xz 1 5, five samples;
• ss -s;
• the relevant lines from journalctl -p err -b and sudo dmesg -T | tail;
• if st is high, mpstat 1 over a few minutes.

CPU steal, disk-controller errors, packet loss at the gateway: none of that is fixable from inside the VM. It is the host's side.

FAQ

Why is load average high while the CPU is idle?

On Linux, load average counts not only processes on the CPU and in its run queue but also those stuck in uninterruptible sleep (the D state), usually waiting on disk or a network filesystem. So load 20 with an idle CPU often means not a CPU shortage but I/O wait - most often disk, sometimes a network filesystem. Find the stuck processes (ps -eo state,pid,comm,wchan:32 | grep '^D') and confirm with the wa column in top and iostat -xz 1.

How do I see which process is using the CPU right now?

Open top and press P (sort by CPU), or run pidstat 1 5 for several samples in a row. ps aux --sort=-%cpu | head also gives a list, but its %CPU is a lifetime average, so it understates a recent spike.

What is CPU steal on a VPS, and can I do anything about it?

Steal (the st column in top) is time your VM was ready to run but the hypervisor gave the physical core to another tenant. Short spikes are fine; sustained st that lines up with a performance dip means the host is oversubscribed (values around 10% and up are especially noticeable for CPU-bound work). It cannot be fixed from inside the VM: capture mpstat 1 and take it to the host for dedicated cores or a different tier.

df shows free space but files will not create. Why?

Most likely inodes are exhausted: they are filesystem records, one per file or directory. Check df -i: if IUse% is 100 with free gigabytes, that is it (less often it is a disk quota or a container limit). The usual cause is millions of tiny files: sessions, cache, mail, a stuck job queue.

How do I tell whether the OOM killer took a process?

Run sudo dmesg -T | grep -i 'killed process' or journalctl -k -b | grep -i 'out of memory'. A line like Out of memory: Killed process 1234 (name) with a timestamp is it. The cause is the kernel not being able to satisfy a memory request within the process's limits: globally (RAM and swap) or by a cgroup / container limit. Check free -h and what grew in top; for a container, its limit and memory.current.

How do I distinguish a swap slowdown from plain CPU shortage?

Run vmstat 1. Non-zero, non-decaying si/so columns mean active paging through disk, which is the sluggishness. With a CPU shortage, si/so stay at zero, while the r column (CPU run queue) and us/sy in top rise.

Which tools are missing on a fresh Ubuntu or Debian?

By default there is no iostat, pidstat or mpstat (the sysstat package), nor iotop, htop, nload or iftop, each installed separately. Usually already there in a standard image: top, free, vmstat, ps, ss, journalctl, dmesg, df, du, swapon.

top or htop?

For a live look, there is little difference. htop is easier on the eyes: a process tree, colour, scrolling, kill by F9. top is always there and has a batch mode (top -b -n1) for a snapshot to a file or a ticket. Install htop for daily use, keep top as the fallback.

Takeaways

• Order of work: uptime and top for the overall picture, then split by symptom.
• Linux load average includes processes waiting on I/O (the D state), so high load with an idle CPU is I/O wait (disk or a network filesystem), not a processor shortage.
• In the %Cpu(s) line: us is applications, sy is the kernel, wa is waiting on I/O, st is time taken by the hypervisor.
• Read memory from the available column in free -h; active swap shows in si/so in vmstat 1; killed processes show in dmesg under oom.
• Disk: iostat -xz 1 (r_await, %util) for speed, df -h for space, df -i for inodes.
• Network: ss -s for the summary, ss -tlnp for what is listening, the TIME-WAIT count for connection churn.
• iostat, pidstat, iotop, htop, nload install separately; ss and journalctl are usually already there.
• Sustained CPU steal and disk errors in dmesg are the host's side: capture the readings and open a ticket.