The Alert That Wakes You Up
The alert fires at 3am: disk usage at 98%. The server is still running. For now.
df -h confirms it. Root filesystem, nearly full. No recent deploy. Nothing obvious in the monitoring dashboard. You have maybe 200MB of headroom before writes start failing.
Here's the sequence I run. It took a few incidents to get the order right.
Step 1: Which Partition, and Is It Inodes?
df -h
df -hi
Note which mount point is full: /, /var, /home, /data. Everything that follows is scoped to that partition.
The -i flag shows inode usage separately. It's possible to have gigabytes free in bytes but zero free inodes. When inodes are exhausted, the kernel returns ENOSPC ("no space left on device") even though Avail shows plenty. Processes can't create new files. The symptom looks identical to a full disk.
If inodes are at 100% and bytes aren't, skip to finding directories with millions of small files. A misconfigured log shipper, a PHP session directory, a job queue that never cleans up. Those are the usual suspects. find /var -maxdepth 3 -type d | xargs -I{} sh -c 'echo "$(ls {} | wc -l) {}"' | sort -rn | head -10 gives you directory file counts.
If bytes are the problem, continue.
Step 2: Find the Largest Directories
du -sh /* 2>/dev/null | sort -rh | head -20
Top-level view. /var filling up is almost always logs or a database. /home is usually a user's directory or a build artifact cache. /tmp filling up is typically a long-running process that wrote to a temp file and never cleaned up.
Drill into whichever is biggest:
du -sh /var/* 2>/dev/null | sort -rh | head -20
du -sh /var/log/* 2>/dev/null | sort -rh | head -20
Keep drilling. Stop when you have a specific file or directory name.
Step 3: Look for Deleted Files Still Held Open
This is the check that took me longest to learn, and it explains more incidents than du does.
On Linux, deleting a file removes its directory entry. The actual disk blocks are not freed until every process holding the file open closes it. A running process can hold a file open indefinitely — and if that file has been rm'd, du won't show it. You see a full disk but du accounts for only 60% of the usage. The rest is invisible.
lsof +L1
+L1 lists files with a link count below 1: deleted from the directory tree but still open. The SIZE column shows how many bytes are locked up.
Example output:
COMMAND PID FD TYPE SIZE/OFF NAME
gunicorn 2847 7u REG 2.1G /var/log/app/app.log (deleted)
nginx 1103 22w REG 800M /tmp/nginx_temp_1234 (deleted)
The fix: restart the process. When it closes the file descriptor, the blocks are freed immediately.
If restarting isn't safe mid-operation, truncate via the proc filesystem:
> /proc/2847/fd/7
/proc/<pid>/fd/<fd> is a live reference to the open file descriptor, even after the directory entry is gone. Truncating it to zero frees the space without killing the process.
Step 4: Check Log Rotation
If du found a large log file and lsof didn't explain it, rotation isn't running.
logrotate --debug /etc/logrotate.conf
--debug shows what logrotate would do without touching anything. Look for which files it manages and when they last rotated.
Application logs (Flask, gunicorn, a background worker) are often missing from logrotate entirely. The developer added log output to /var/log/myapp/, assumed the OS would handle it, and it never did.
Add a config in /etc/logrotate.d/myapp:
/var/log/myapp/*.log {
daily
rotate 7
compress
delaycompress
missingok
notifempty
create 0640 www-data adm
postrotate
systemctl kill -s HUP gunicorn.service
endscript
}
The postrotate block is critical. Without it, the process keeps writing to the old file descriptor after the log file is rotated. The new file fills up, the old file gets compressed, and the running process is still writing to the unrotated location. You'll see this if lsof shows a large deleted file right after rotation ran.
The HUP signal tells gunicorn to reopen its log files. Check your application's documentation for the correct signal: HUP for gunicorn, SIGUSR1 for nginx.
Step 5: PostgreSQL WAL and Replication Slots
If /var/lib/postgresql is large, check the WAL directory:
du -sh /var/lib/postgresql/*/main/pg_wal/
WAL accumulates when a replication slot exists and the replica it was created for is gone. Postgres holds onto WAL indefinitely because it thinks something still needs it.
SELECT slot_name, active, pg_size_pretty(
pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)
) AS lag
FROM pg_replication_slots;
An inactive slot with a lag in gigabytes is the problem. If that replica is decommissioned, drop the slot:
SELECT pg_drop_replication_slot('slot_name');
Postgres starts cleaning up WAL immediately. The directory shrinks over the next few minutes.
This one has a way of appearing suddenly on otherwise stable servers — a replica goes down, nobody notices, WAL accumulates for weeks, then the disk fills.
Step 6: Core Dumps
Crashed processes leave core dump files, usually in /var/crash/, /tmp/, or the working directory of the process. A core dump for a Python or Java process can be multiple gigabytes.
find / -name "core" -o -name "core.[0-9]*" 2>/dev/null | xargs ls -lh 2>/dev/null
Check where they're configured to go:
cat /proc/sys/kernel/core_pattern
If you're not using core dumps for debugging, disable them for your services via the systemd unit file:
[Service]
LimitCORE=0
LimitCORE=0 sets the core dump size limit to zero for that process. Crashes still get logged in the journal; the multi-gigabyte file just doesn't appear.
After You've Found It
Write down what caused it before you fix it. One line:
2026-08-10: disk full on /var — gunicorn was holding a 2.1GB deleted log
file open. Restarted gunicorn. Added logrotate config with postrotate HUP.
Then address the underlying behavior. Set up logrotate if you haven't. Drop inactive replication slots. Add a disk usage alert at 80% so you're not debugging at 98% with minutes to spare.
The immediate fix takes five minutes. The repeat incident happens when you restart the process and move on without figuring out why the file got that large in the first place.
The sequence matters because lsof +L1 explains incidents that du can't. du only sees the directory tree. Deleted-but-open files are invisible to it. If you du everything and the numbers don't add up, skip straight to lsof — that's almost certainly where the missing space is.