Docker filled the disk: how to find what takes the space and clean it safely

The server runs on Docker, and one day a build fails with "no space left on device" while df -h shows 100%. Docker does not lose space: it keeps images, stopped containers, volumes, build cache and logs, and deletes none of it until you say so. Below is the order of work: how to see in a couple of minutes what has grown, what is safe to clean, where it is easy to lose data and how to keep the disk from filling up again.

In short. See the picture with docker system df, and check log size separately: sudo sh -c 'du -h /var/lib/docker/containers/*/*-json.log | sort -h | tail'. Clean in order: docker container prune, then docker image prune, and only then think about volumes. A plain docker volume prune did not touch named volumes in our test (Docker 29.1.3), but docker volume prune -a deletes every volume no container is attached to, together with its data. To stop logs from growing forever, set rotation in /etc/docker/daemon.json.

Where the space actually goes

Docker has five sources of growth. Four show up in docker system df, and the fifth does not show up at all - and it is the one that often turns out to be guilty.

Step 1. Make sure Docker is what ate the space

First, see how much of the disk is used overall and compare it with the Docker directories. If the containerd image store is on (as on our test server), images live not in /var/lib/docker but in /var/lib/containerd, so we check both. du -sh prints the total size of a directory, and sudo is needed because the files inside belong to root.

How we got the output. To avoid touching real data we started a separate test Docker daemon with an empty store on a HIP-HOSTING VPS running Ubuntu 24.04 and built a "cluttered" server in it: a few images, two builds under one tag, three stopped containers with data, two volumes and a container that writes to its log without stopping. Your sizes will differ, the order of work is the same. One note on units: du -h counts in multiples of 1024 while docker uses decimal MB and GB, so numbers from the two commands differ a little. Long identifiers in the outputs are shortened. The docker commands here and below are written without sudo: we assume you are root or in the docker group.

8.9 GB of 38 are used, and Docker holds about 2.7 GB of it: 1.3 GB in /var/lib/docker and 1.4 GB in /var/lib/containerd. In our case the images ended up in containerd; with the classic store they usually sit in /var/lib/docker/overlay2. If both directories are small on your server and the disk is full, Docker is not to blame - see the last section.

Step 2. Show who ate the disk: docker system df

The docker system df command splits Docker's space into categories. The SIZE column shows how much a category takes, and RECLAIMABLE shows how much of it can be returned because nothing uses it.

Row by row. Images has six images and 1.475 GB; in the RECLAIMABLE column Docker estimated 256.7 MB as returnable (we did not dig into how it counts that; below, image prune actually freed 157.3 MB and image prune -a 313.8 MB). Containers has four stopped containers, 251.7 MB, all of it reclaimable (100%). Local Volumes has two volumes, 335.5 MB together, both "free" because no container is attached to them. The Build Cache row is empty for us: the images were built with the legacy builder, and no cache showed up in this row on the test server. With BuildKit you will most likely see a non-zero value.

The -v flag shows every image, container and volume separately, but its output is very wide, so we do not print it. Let us look at the images with the ordinary command. Note myapp:1.0 and the image without a name of the same 327 MB: we built the image under the tag myapp:1.0 twice with a changed file. The tag went to the new build and the old one lost its name. Such images are called "dangling": nothing uses them, but they take space. In Docker 29 the plain docker images did not list that image, while the filter -f dangling=true shows it as <untagged> (other versions may show <none>):

The SIZE column of docker ps -a --size shows how much space stopped containers hold (we cut the output to four columns with --format). Here are three containers of 83.9 MB each: we ran them so that each wrote 80 MB of data, and they finished but were not deleted.

What docker system df does not show: container logs

Remember this line from the output above: Containers ... 251.7MB. It is the size of the writable layers of stopped containers, and logs are not part of it. A log is a file where Docker puts everything a program prints to stdout and stderr. By default it is named <id>-json.log and grows without limit.

The biggest logs can be found with du and sorting. The asterisks in the path stand for all containers at once, sort -h orders by size taking units (M, G) into account, and tail -3 keeps the three biggest. All of it is wrapped in sudo sh -c '...' because the directory is closed to a normal user and the shell expands the asterisks: without the wrapper it expands them as you and finds nothing.

The chatty container is a test program that prints a "request handled" line in a loop. In 25 seconds it produced a 947 MB log. A real application writes slower, but the principle is the same: a chatty service without rotation collects gigabytes over weeks, and docker system df will not show it. In our test 947 MB of the 1.3 GB in /var/lib/docker were that one log:

How to read it: containers in this directory holds container logs and service files, volumes holds volumes. If you have hundreds of megabytes or gigabytes in containers, look for a big *-json.log.

Clean in order: from safe to dangerous

We go from what is almost risk-free to where data can be lost. After each step it is useful to look at df -h /.

1. Stopped containers

docker container prune deletes all stopped containers. Before that, review the list with docker ps -a: everything a program wrote inside a container (not into a volume) disappears with it. The -f flag removes the "are you sure?" question. We use it in the examples so the output comes without a dialogue, but on a production server, until you are used to it, run without it and answer the question by hand.

The command reported 251.7 MB freed, yet the disk got 1.1 GB lighter at once: together with the chatty container its 947 MB log was deleted. Those 947 MB never made it into Docker's report.

2. Images without a tag

docker image prune without flags deletes only "dangling" images, the very ones without a name. It is safe unless you need to roll back to the previous build: an image without a tag can only be run by its ID, and it may be the only copy of the earlier version.

3. Build cache

docker builder prune deletes the cache of intermediate layers. It is safe, but the next build will be slower because the cache has to warm up again. We had no cache, and the command returned 0B.

4. All images nobody uses

docker image prune -a goes further: it deletes every image that no container uses, tagged ones included. There is no danger to data, but the images will have to be downloaded again, and if the registry is unavailable or you built the image locally without the sources, there will be nothing to restore it from. So for local builds make sure you still have the sources or the Dockerfile.

5. Volumes: the main trap

Volumes hold what Docker was installed for: databases, files uploaded by users, certificates. A container can be recreated, the data of a volume cannot. There are two rules.

First. According to the Docker documentation a plain docker volume prune deletes only anonymous volumes, the ones created automatically without a name. In our test (Docker 29.1.3) named volumes stayed in place after it and it freed nothing (the first command in the output below); we did not test anonymous volumes.

Second. The -a flag changes everything: docker volume prune -a deletes every volume no container is attached to right now. If the database container is already deleted (for example you ran container prune or compose down earlier, and the order in this article is exactly that), its volume counts as "unused" and is deleted. We created the volumes pgdata and old-uploads, and with -a both disappeared, although the first one "played" a database:

So before -a always look at docker volume ls and ask yourself about every volume whose data it is. If in doubt, copy the volume first and clean afterwards. On older Docker versions the volume commands may behave differently, so on an unfamiliar server check docker --version and the volume list first.

docker system prune: what it does and does not do

docker system prune is "everything at once": it deletes stopped containers, unused networks, images without a tag and build cache. The -a flag adds all unused images, and --volumes adds volumes. We checked on the test daemon what exactly happens. We created a volume keepme with data, a stopped container and an image:

Without flags the command deleted only the stopped container. With -a --volumes the image went away, while the named volume keepme stayed in place: in our Docker version --volumes did not touch the named volume. But do not count on this as protection. On another version or in a script with docker volume prune -a the data will be gone, so on servers with databases we advise cleaning with separate commands, as described above.

Logs: empty them now and limit them for good

If a log ate the space, there are two actions: remove what has piled up and keep it from piling up again.

To empty a log without deleting the container, use truncate -s 0. The command makes the file empty while the file itself stays in place, so Docker usually keeps writing to it. We checked this on a stopped container: before the command the file weighed 766 MB, after it 0. For a running container the method usually works too, but we did not check that. If the log holds something you need to read, save it elsewhere first.

To keep logs from growing, set rotation. Docker stores a log in files, and you can limit the size of one file and the number of files. Create or extend /etc/docker/daemon.json (on the test daemon the file lay in a separate folder, the path in the output below is replaced with the usual one): max-size is the maximum size of one file, max-file is how many files to keep. These settings apply to new containers: already created ones keep writing by the old rules until you recreate them. After editing, restart the Docker service with sudo systemctl restart docker, but note that it stops running containers unless live-restore is on.

We tested the setting: we restarted the test daemon with such a file and started a new "chatty" container for 25 seconds. The log did not grow without limit but split into three files, 23 MB in total:

The same options can be set for one container with the keys --log-opt max-size=10m --log-opt max-file=3 in docker run, or with a logging: block in a Compose file (more on Compose in our article on several services in Docker Compose). Those two ways come from the Docker documentation; on the test server we checked only daemon.json.

How not to get here again

• Log rotation in daemon.json - once and for a long time. It is the main measure because a log grows unnoticed.
• Run one-off containers with --rm so they delete themselves when they finish.
• Keep data in named volumes and write down what lives in which. Nobody will recognise an unnamed volume in a month.
• Run docker system df regularly and keep it in your usual server checklist. Growth of the RECLAIMABLE value is a signal to remove the extra.
• Cleaning images without a tag and stopped containers can be put on a schedule. How to run commands on a timer we covered in the article on systemd timers instead of cron. Do not clean volumes on a schedule.

If it is not Docker that takes the space

If /var/lib/docker and /var/lib/containerd are small while the disk is full, look elsewhere. This command shows the biggest first-level directories without crossing into other file systems: sudo du -xh --max-depth=1 / | sort -h | tail. The -x flag stops du from leaving one file system, and --max-depth=1 shows only the top level. If bytes are free but files cannot be created, check inodes: see our article on "No space left on device" when there is free space. And if the server is slow along with the disk, start with diagnosing a slow server.

FAQ

Is it safe to run docker system prune?

Without flags, according to the documentation, it deletes stopped containers, unused networks, images without a tag and build cache; in our test volumes stayed in place after it. But deleting stopped containers erases what was written in them, so look at docker ps -a first.

Why does df say the disk is full while docker system df does not?

Most often because docker system df does not count container logs. Check the size of the *-json.log files in /var/lib/docker/containers.

Will docker system prune delete my volumes with data?

In our test (Docker 29.1.3) the named volume with data stayed in place after the command with --volumes. But docker volume prune -a deletes all unused volumes, named ones included, so run it only after checking the list.

How do I empty a container log without deleting the container?

With sudo truncate -s 0 path_to_the_log. The path is shown by docker inspect -f '{{.LogPath}}' container_name.

Does log rotation apply to containers that are already running?

No, the settings in daemon.json apply to containers created after it. Old ones have to be recreated.

Summary

• Docker's disk is eaten by images, stopped containers, volumes, build cache and logs. The first four are visible in docker system df, logs are not.
• Look for big logs with sudo sh -c 'du -h /var/lib/docker/containers/*/*-json.log | sort -h | tail'; they often turn out to be the culprit.
• Clean in order: containers, images without a tag, cache, all images. Come to volumes last and with a check.
• docker volume prune -a deletes all unused volumes together with their data, including the volume of a database whose container is already deleted.
• Limit logs in /etc/docker/daemon.json (max-size, max-file); the setting applies to new containers.
• If Docker is innocent, look for big directories with du -xh --max-depth=1 / and check inodes.