du and df don’t match: deleted files still held open

You’ll do all the filling on a 512 MiB test file system kept in a file, /var/tmp/lab66.img, mounted through a loop device, so nothing in vg_data or on the root file system gets near full. Work on rocky01 as student, and take a snapshot first (VBoxManage snapshot rocky01 take before-l66 on the laptop). Outputs that came from running these steps for real were only adjusted for user, PID and device numbers; yours will differ in those.

1Build a test file system you can fill

Run this on rocky01:

mkdir -p ~/lab66
truncate -s 512M /var/tmp/lab66.img
mkfs.ext4 -q -L lab66 /var/tmp/lab66.img
sudo mkdir -p /mnt/lab66
sudo mount -o loop /var/tmp/lab66.img /mnt/lab66
sudo chown student:student /mnt/lab66
df -h /mnt/lab66

You should see something like this:

Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      488M   24K  452M   1% /mnt/lab66

truncate -s 512M makes an empty sparse file, mkfs.ext4 formats it, and -o loop tells mount to attach it to the first free loop device, /dev/loop0 here. Keep test images in /var/tmp. On Ubuntu 26.04 /tmp is a tmpfs in RAM, so a 512 MiB image there would eat half your VM’s memory.

✅ Checkpoint: df shows /dev/loop0 mounted on /mnt/lab66, about 488M in size.

2Start an app that keeps its log open

Create the program (paste this whole block):

cat > ~/lab66/writer66.py <<'EOF'
#!/usr/bin/env python3
# writer66.py: a small "app" that keeps its log file open, like a real service
import os, sys, time
path = sys.argv[1]
mode = sys.argv[2] if len(sys.argv) > 2 else "a"
log = open(path, mode)
line = "2026-10-27 11:05:00 INFO batch 0042 settled, filler " + "x" * 971 + "\n"
for i in range(200 * 1024):
    log.write(line)
log.flush()
print(f"writer66: PID {os.getpid()} wrote 200 MiB to {path}", flush=True)
n = 0
while True:
    n += 1
    log.write(time.strftime("%Y-%m-%d %H:%M:%S") + f" INFO heartbeat {n}\n")
    log.flush()
    time.sleep(5)
EOF
python3 -m py_compile ~/lab66/writer66.py && echo syntax ok

Start it in the background:

mkdir -p /mnt/lab66/app
python3 ~/lab66/writer66.py /mnt/lab66/app/app.log &

You should see something like this:

[1] 2481
writer66: PID 2481 wrote 200 MiB to /mnt/lab66/app/app.log

The program opens app.log in append mode, writes 200 MiB of log lines, and then adds a heartbeat line every five seconds, holding the file open the whole time, like a Java app or a database with a debug log switched on. Write down your PID; the rest of the lab uses 2481.

✅ Checkpoint: writer66 printed its PID and “wrote 200 MiB”.

3Check that du and df agree, and read df properly

Run this:

ls -lh /mnt/lab66/app/app.log
du -sh /mnt/lab66
df -h /mnt/lab66
sudo tune2fs -l $(findmnt -no SOURCE /mnt/lab66) | grep -E 'Block count|Reserved block count|Block size'

You should see something like this:

-rw-r--r--. 1 student student 201M Oct 27 14:53 /mnt/lab66/app/app.log
201M	/mnt/lab66
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      488M  201M  252M  45% /mnt/lab66

Block count:              131072
Reserved block count:     6553
Block size:               4096

du and df both say 201M, so right now there’s no mystery. Look at df’s own numbers though: 488M minus 201M is 287M, yet Avail is 252M. The difference is the reserved blocks, 6553 blocks of 4 KiB, about 25.6 MiB (5%), which only root can use.

That gap is normal on every ext4 file system and isn’t a du/df problem.

Try it yourself: find the biggest folders on any file system in the Command Builder.

✅ Checkpoint: du and df both show about 201M, and you can explain why Avail is less than Size minus Used.

4Delete the log the wrong way

Run this:

rm /mnt/lab66/app/app.log
ls -l /mnt/lab66/app
du -sh /mnt/lab66
df -h /mnt/lab66

You should see this:

total 0
24K	/mnt/lab66
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      488M  201M  252M  45% /mnt/lab66

The folder is empty and du counts 24K, but df still shows 201M used. This is the ticket you’ll get at work: “I deleted the big log and the disk is still full.” The writer is still running and still writing heartbeats into a file nobody can see.

✅ Checkpoint: du says 24K and df still says 201M used.

5Find the ghost with lsof +L1

Run this:

sudo lsof -a +L1 /mnt/lab66

You should see something like this:

COMMAND  PID    USER   FD   TYPE DEVICE  SIZE/OFF NLINK NODE NAME
python3 2481 student    3w   REG    7,0 209715348     0   13 /mnt/lab66/app/app.log (deleted)

+L1 selects open files with a link count below 1, -a means “and”, and the path limits it to this file system. NLINK 0 and “(deleted)” confirm it. FD 3w is file descriptor 3, open for writing, and SIZE/OFF is the hidden file’s size, 200 MiB plus a few heartbeat lines.

Run plain sudo lsof +L1 on a whole server; on rocky01 you may also see a few lines like /memfd:... (deleted), which are in-memory files used by systemd and others, not disk space.

Try it yourself: build the deleted-but-open files command in the Command Builder.

✅ Checkpoint: lsof shows python3 with PID 2481 (yours), FD 3w and NLINK 0.

6Look inside /proc and save a copy

Run this (use your PID):

ls -l /proc/2481/fd
tail -n 1 /proc/2481/fd/3
find /proc/[0-9]*/fd -lname '*(deleted)' -printf '%p -> %l\n' 2>/dev/null | grep lab66

You should see something like this:

lrwx------. 1 student student 64 Oct 27 14:53 0 -> /dev/pts/0
lrwx------. 1 student student 64 Oct 27 14:53 1 -> /dev/pts/0
lrwx------. 1 student student 64 Oct 27 14:53 2 -> /dev/pts/0
l-wx------. 1 student student 64 Oct 27 14:53 3 -> /mnt/lab66/app/app.log (deleted)
2026-10-27 14:53:44 INFO heartbeat 4
/proc/2481/fd/3 -> /mnt/lab66/app/app.log (deleted)

fd 0, 1 and 2 are the terminal, and fd 3 points to the deleted file. You can read it through that link as if it still had a name: tail printed the latest heartbeat. The find line is the “no lsof” way, the same command the Command Builder task gives you.

Because the data is still there, you can rescue it before you throw it away:

cp /proc/2481/fd/3 /var/tmp/app.log.saved
ls -lh /var/tmp/app.log.saved

You should see something like this:

-rw-r--r--. 1 student student 201M Oct 27 14:55 /var/tmp/app.log.saved

Copy to a different file system, never to the one that’s full. The same trick saves an important file that someone deleted by mistake, as long as a process still has it open.

✅ Checkpoint: /var/tmp/app.log.saved is 201M and ends with a heartbeat line.

7Free the space without stopping the app

Pretend this app can’t be restarted until the weekend. Empty the deleted file through the open handle:

Run this:

: > /proc/2481/fd/3
df -h /mnt/lab66
sudo lsof -a +L1 /mnt/lab66
sleep 6; sudo lsof -a +L1 /mnt/lab66

You should see something like this:

Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      488M   28K  452M   1% /mnt/lab66
COMMAND  PID    USER   FD   TYPE DEVICE SIZE/OFF NLINK NODE NAME
python3 2481 student    3w   REG    7,0        0     0   13 /mnt/lab66/app/app.log (deleted)
COMMAND  PID    USER   FD   TYPE DEVICE SIZE/OFF NLINK NODE NAME
python3 2481 student    3w   REG    7,0       37     0   13 /mnt/lab66/app/app.log (deleted)

: is the shell’s “do nothing” command, so : > /proc/2481/fd/3 opens the file for writing with truncation and closes it, which sets its size to 0. df dropped from 201M to 28K at once, and the app carried on: six seconds later the hidden file holds 37 bytes, one new heartbeat at the start. sudo truncate -s 0 /proc/2481/fd/3 does the same. You need sudo when the process belongs to another user.

Only truncate a file you are sure can be lost, such as a debug log, and only when the program writes in append mode. Truncating a database file, a queue file or anything the program reads back will corrupt it.

✅ Checkpoint: df shows 28K used while writer66 is still running.

8Get space back from a real service: rsyslog

Now the same mistake on a real daemon, with the local3 log from Lesson 29.

Run this on rocky01:

logger -p local3.info -t saatapp "order 1050 placed"
sudo ls -l /var/log/saatapp.log
sudo rm /var/log/saatapp.log
logger -p local3.info -t saatapp "order 1051 placed"
sudo lsof +L1 | grep saatapp

You should see something like this:

-rw-------. 1 root root 1843 Oct 27 15:01 /var/log/saatapp.log
rsyslogd 1108 root    9w   REG  253,0     1843     0 17305822 /var/log/saatapp.log (deleted)

rsyslogd still has the deleted file open on fd 9 (DEVICE 253,0 is rl-root), and “order 1051” was written into that hidden file, where nobody will ever read it. Tell rsyslog to reopen its files with SIGHUP, which is what the postrotate line in your Lesson 29 logrotate file does:

sudo systemctl kill -s HUP rsyslog
logger -p local3.info -t saatapp "order 1052 placed"
sudo ls -l /var/log/saatapp.log
sudo tail -n 1 /var/log/saatapp.log
sudo lsof +L1 | grep -c saatapp

You should see something like this:

-rw-r--r--. 1 root root 62 Oct 27 15:02 /var/log/saatapp.log
Oct 27 15:02:10 rocky01 saatapp[3120]: order 1052 placed
0

rsyslog closed the deleted file (freeing its blocks) and created a new saatapp.log with the next message. The new file gets rsyslog’s own default permissions rather than the 0600 that logrotate used, so check them with ls -l if the log is sensitive. sudo systemctl restart rsyslog works too and is what most runbooks say; the HUP is lighter because rsyslog keeps running and its forwarding connection to ubu01 stays up. The message logged between the rm and the HUP is gone for good, which is one more reason to never rm a live log.

✅ Checkpoint: lsof +L1 no longer shows saatapp.log, and the new /var/log/saatapp.log contains order 1052.

9Stop the app and confirm nothing is held

Run this:

kill %1
sudo lsof -a +L1 /mnt/lab66
df -h /mnt/lab66

You should see something like this:

[1]+  Terminated              python3 ~/lab66/writer66.py /mnt/lab66/app/app.log
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      488M   28K  452M   1% /mnt/lab66

When a process ends, the kernel closes all its files, so stopping or restarting the holder is the cleanest fix of all. lsof printed nothing because no deleted file is open on /mnt/lab66 any more. Keep the test file system mounted for the Break-It section, and delete the saved copy with rm /var/tmp/app.log.saved.

✅ Checkpoint: lsof prints nothing for /mnt/lab66 and df shows it nearly empty.