The root file system is full — diagnose it in five commands

On this page

1. One: the honest numbers
2. Two: the biggest directories, without crossing file systems
3. Three: the deleted files nothing can see
4. Four: the journal
5. Five: the package manager’s cache
6. After those five

One: the honest numbers

df -h
df -i

Two different failures produce the same “No space left on device” message. The first df shows blocks. The second shows inodes, and a file system can be at 12% of its blocks and 100% of its inodes if something has created millions of tiny files — a mail queue, a session directory, a misconfigured cache. If IUse% is 100, do not go looking for large files; go looking for large counts of files.

Two: the biggest directories, without crossing file systems

sudo du -xh --max-depth=1 / 2>/dev/null | sort -h | tail -15

-x is essential. Without it, du descends into /proc, /sys, network mounts and any other file system mounted below /, and the answer is meaningless. Repeat it one level deeper on whichever directory is the largest:

sudo du -xh --max-depth=1 /var 2>/dev/null | sort -h | tail -15

On most servers the answer is now visible: /var/log, /var/lib/docker, /var/cache, or a home directory.

Three: the deleted files nothing can see

This is the one that wastes afternoons. A process holding a file open keeps its blocks allocated even after the file is unlinked. du cannot see it, because there is no directory entry; df counts it, because the space is still allocated. The symptom is df and du disagreeing by many gigabytes.

sudo lsof +L1 | sort -k7 -n | tail -20

+L1 lists open files with a link count below one, which is exactly the set of deleted-but-open files. The usual culprit is a log file that was removed by hand — or by a naive cleanup script — while the service writing to it kept running. The fix is to restart that service, not to delete anything further. To avoid recreating the problem, truncate rather than delete:

sudo truncate -s 0 /var/log/some-huge.log

Four: the journal

systemd’s journal has a default cap that is a percentage of the file system, which on a large disk means gigabytes. It is almost always among the top three on a server that has been running for a year.

journalctl --disk-usage
sudo journalctl --vacuum-size=200M

Set a permanent bound rather than doing this repeatedly — bounding the systemd journal properly covers SystemMaxUse and the rotation settings.

Five: the package manager’s cache

Every distribution keeps downloaded packages after installing them, and every one of them will clear it on request.

sudo apt clean && sudo apt autoremove --purge        # Debian, Ubuntu
sudo dnf clean all && sudo dnf autoremove            # Fedora, RHEL
sudo paccache -rk1 && sudo pacman -Rns $(pacman -Qtdq)  # Arch
sudo zypper clean --all                              # openSUSE

The autoremove half is usually the larger win, because it takes old kernels with it, and each kernel plus its initramfs and modules is 200 to 400 MB. Distribution specifics are in apt caches and old kernels, paccache on Arch and dnf on Fedora.

After those five

If the space is still missing, work down this list. Each one is a common single cause.

Keep a few hundred megabytes free at all times on /. Below that, apt cannot unpack, journald cannot rotate, and the tools you need to fix the problem are the ones that stop working first.