"No space left on device" but there is free space: how to find and free inodes

An application crashes with "No space left on device" while df -h honestly reports the disk is 3% full. Neither the disk nor the command is broken: what ran out is not bytes but inodes. Here is what that means, how to see it with a single command and how to clean up the server without breaking anything.

In short. An inode is a file system record about a file: every file or directory takes one inode, whatever its size. Their number is limited. Check with df -i, column IUse%. Find the culprit: sudo du --inodes -x -d1 /var | sort -rn | head. Clean up: find DIRECTORY -type f -delete. The cause is almost always the same: millions of tiny files (sessions, cache, mail queue).

What an inode is and why it runs out

Think of a library: there may be few books, but each needs a card in the catalogue. When the cards run out you cannot accept a new book even though the shelves are empty. Same here: an inode stores the owner, permissions and the location of a file's data, while the content lies elsewhere. A 1 byte file and a 1 GB file take one inode each.

In ext4, which most servers use, the number of inodes is fixed when the file system is created. It is usually derived from the disk size - roughly one inode per 16 KiB - so ordinary workloads have plenty. But when an application produces millions of tiny files, the limit is reached before the space is.

What it looks like

To show the symptom without breaking a real disk, we created a temporary in-memory disk (tmpfs) of 64 MB and limited it to two thousand inodes ourselves. It is an imitation: on real ext4 the limit is set at creation, we just set it by hand. This is the server before the problem:

df -h counts bytes, df -i counts inodes. These are two separate scales, and almost nobody looks at the second until trouble hits. Now we fill the disk with empty files, the way an application does when it does not clean up its temporary files. 1998 were created. The next file cannot be created:

Now look at the disk in both modes:

Avail still says 64 MB, while IFree is zero. This is the main sign: df -h is empty, df -i is at 100%. There is no point hunting for big files - they are not to blame.

How to find the culprit directory

You need to see where the many small files are. du has the --inodes option for this: it counts objects, not size. The other options: -x - do not cross into other file systems, -d1 - one level deep. sort -rn puts the biggest numbers first and head keeps the top of the list. Check the mount point you need (on a real server usually /, /var or /home):

The first line is the total for the directory, and the second shows who ate almost everything: the sessions directory. Then go deeper the same way until you find the exact place. On production servers the usual suspects are PHP session directories, application caches, the mail queue in /var/spool and temporary files in /tmp.

How to clean up safely

Before deleting, make sure the files really are temporary: look at which application creates them and which ones can go. If in doubt, move or archive them first.

For millions of files rm DIRECTORY/* does not work: the shell expands the asterisk into a huge list and rm answers "Argument list too long". Instead let find walk the directory and delete files one by one:

find /mnt/inode-demo/sessions -xdev -type f -name 'sess_1*' -delete

The options. -xdev - stay within one file system. -type f - regular files only, directories stay. -name 'sess_1*' - only files whose name starts with sess_1 (in real life this could be -mtime +7: older than seven days). -delete - remove what was found. If in doubt, first run the command without -delete: it will just list what it found.

45% used, files can be created, the error is gone (0 means the command succeeded). We deleted only part of the files and that was enough. After this you still have to cure the cause, otherwise it comes back in a week.

How to keep it from coming back

• Turn on cleanup of old files at the source: for PHP it is session garbage collection, for applications the cache lifetime, for mail the queue limit.
• For temporary files add a cleanup rule, for example find /var/tmp -type f -mtime +7 -delete once a day from cron.
• Watch IUse% as you watch free space. A df -i in your server health check is enough.
• If there are still many small files, pick a file system for that workload or keep the data in a database. The inode count of ext4 cannot be raised later: you would have to create the file system again with a different number of inodes.

If the server is also slow while it is short of space, see diagnosing a slow server, and keep an eye on memory with the guide to the OOM killer.

FAQ

Why "No space left on device" if df shows free space?

Most likely you ran out of inodes. Run df -i: if IUse% is 100%, that is it. If it is fine there, check quotas and volumes, but check inodes first.

What is an inode?

A record about a file or directory: owner, permissions and where the data lies. One file is one inode, whatever its size.

How do I find the directory with a huge number of files?

With sudo du --inodes -x -d1 /var | sort -rn | head. Repeat it for the directory you find, going deeper until you reach the source.

How do I delete a million files?

With find DIRECTORY -type f -delete. A plain rm * at that scale answers "Argument list too long".

Summary

• "No space left on device" with free bytes is almost always a lack of inodes.
• One command to check: df -i, look at IUse%.
• Find the culprit with sudo du --inodes -x -d1 /var | sort -rn | head.
• Clean with find DIRECTORY -type f -delete, not rm *; run it without -delete first.
• Then cure the cause: cleanup of sessions and cache at the source, and df -i in your regular checks.