“No space left on device” with free space: inodes and other causes

Everything that fills up in this lab is a 256 MiB file system kept in /var/tmp/lab67.img and mounted through a loop device, so vg_data, lv_app, lv_logs and the root file system stay safe. Work on rocky01 as student and take a snapshot first (VBoxManage snapshot rocky01 take before-l67 on the laptop). The outputs on the test file system came from running these exact steps; only the loop device number and times will differ on yours.

1Build a small file system with very few inodes

Run this on rocky01:

truncate -s 256M /var/tmp/lab67.img
mkfs.ext4 -q -L lab67 -b 4096 -N 2000 /var/tmp/lab67.img
sudo mkdir -p /mnt/lab67
sudo mount -o loop /var/tmp/lab67.img /mnt/lab67
sudo chown student:student /mnt/lab67
df -h /mnt/lab67
df -i /mnt/lab67

You should see this:

Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      240M   24K  222M   1% /mnt/lab67
Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/loop0       2016    11  2005    1% /mnt/lab67

-N 2000 asks mkfs for about 2000 inodes instead of the usual one per 16 KiB (which would be about 16,000 here). ext4 rounds the number up to fill its block groups, so you get 2016, and 11 are taken straight away by the root folder, lost+found and the file system’s own reserved inodes. A real server volume has millions, but it runs out the same way.

✅ Checkpoint: df -h shows 240M with 222M available and df -i shows 2016 inodes.

2Fill it the way a busy app does

Create three big reports and a lot of tiny session files:

mkdir -p /mnt/lab67/app/sessions /mnt/lab67/app/reports
cd /mnt/lab67/app/reports
for m in 07 08 09; do dd if=/dev/zero of=gstr-2026-$m.pdf bs=1M count=20 status=none; done
cd /mnt/lab67/app/sessions
touch -d '5 days ago' sess_{0001..1500}
touch sess_{1501..2500} 2>&1 | tail -n 2
ls | wc -l
echo "invoice 1044" > /mnt/lab67/app/reports/inv-1044.txt

You’ll get this:

touch: cannot touch 'sess_2499': No space left on device
touch: cannot touch 'sess_2500': No space left on device
1999
-bash: /mnt/lab67/app/reports/inv-1044.txt: No space left on device

The first 1500 session files get a time five days ago (old sessions an app forgot to clean), the rest are new. Somewhere after 1999 files touch starts failing, and then the important write, a new invoice, fails too with the same error. 2>&1 | tail -n 2 keeps the screen from filling with 500 identical error lines.

✅ Checkpoint: touch fails with “No space left on device” and the invoice file can’t be created.

3Compare df -h with df -i

Run this:

df -h /mnt/lab67
df -i /mnt/lab67

You should see this:

Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      240M   61M  162M  28% /mnt/lab67
Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/loop0       2016  2016     0  100% /mnt/lab67

Blocks: 28% used, 162M available. Inodes: 2016 of 2016 used, 0 free, IUse% 100%. This pair of commands is the whole diagnosis for an inode problem, and the second one is the one people forget.

Always run both on the path that failed, because / can be fine while /var/lib/app is full.

✅ Checkpoint: You can point at IUse% 100% as the reason the invoice couldn’t be written.

4Find the folder with the most files

Run this:

sudo du --inodes -x /mnt/lab67 | sort -rn | head -n 5
find /mnt/lab67 -xdev -type f -printf '%h\n' | sort | uniq -c | sort -rn | head -n 3

You should see this:

2007	/mnt/lab67
2005	/mnt/lab67/app
2000	/mnt/lab67/app/sessions
4	/mnt/lab67/app/reports
1	/mnt/lab67/lost+found
   1999 /mnt/lab67/app/sessions
      3 /mnt/lab67/app/reports

du --inodes counts inodes (files and folders) under each folder, and the numbers add up the tree, so read from the deepest folder with a big count: app/sessions holds 2000. The find line prints the folder of every file and counts them, which works the same with any find and helps when du is an older version. sudo lets du read lost+found, which only root may enter. On Ubuntu 26.04 du comes from uutils; if it doesn’t accept --inodes on your version, use the find line.

Try it yourself: find the biggest folders by size in the Command Builder.

✅ Checkpoint: Both commands point at /mnt/lab67/app/sessions.

5Clean old files: dry run first, then delete

See what would go, and how many:

find /mnt/lab67/app/sessions -xdev -type f -name 'sess_*' -mtime +2 -print | head -n 3
find /mnt/lab67/app/sessions -xdev -type f -name 'sess_*' -mtime +2 -print | wc -l

You should see something like this:

/mnt/lab67/app/sessions/sess_1339
/mnt/lab67/app/sessions/sess_0058
/mnt/lab67/app/sessions/sess_0731
1500

-mtime +2 means “last modified more than two full days ago”, so only the 1500 old sessions match and the 499 new ones are safe. find lists files in directory order, not by name, which is why the sample looks random. When the count matches what you expect, run the same command with -delete at the end:

find /mnt/lab67/app/sessions -xdev -type f -name 'sess_*' -mtime +2 -delete
df -i /mnt/lab67
echo "invoice 1044" > /mnt/lab67/app/reports/inv-1044.txt && cat /mnt/lab67/app/reports/inv-1044.txt

You should see this:

Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/loop0       2016   516  1500   26% /mnt/lab67
invoice 1044

1500 inodes are free again and the invoice is written. On a real server, the next step is to find out why the app stopped cleaning its sessions, and to add a timer (Lesson 28) that runs this find every night.

Try it yourself: find the biggest files under a folder in the Command Builder.

✅ Checkpoint: df -i shows 516 used (26%) and inv-1044.txt contains “invoice 1044”.

6Hit the reserved-block limit as a normal user

Fill the space as student, then try the same as root:

mkdir /mnt/lab67/scratch
dd if=/dev/zero of=/mnt/lab67/scratch/fill.bin bs=1M
df -h /mnt/lab67

You’ll get this:

dd: error writing '/mnt/lab67/scratch/fill.bin': No space left on device
162+0 records in
161+0 records out
168820736 bytes (169 MB, 161 MiB) copied, 0.385302 s, 438 MB/s
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      240M  222M  220K 100% /mnt/lab67

Student’s dd stopped with ENOSPC, and df shows 100% with 220K available, even though Size minus Used is still 18M. Now write 5 MiB as root and look at the reserve:

sudo dd if=/dev/zero of=/mnt/lab67/scratch/root.bin bs=1M count=5 status=none; echo "root wrote 5 MiB"
df -h /mnt/lab67
sudo tune2fs -l $(findmnt -no SOURCE /mnt/lab67) | grep -E 'Reserved block count|Reserved blocks uid'

You should see this:

root wrote 5 MiB
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      240M  227M     0 100% /mnt/lab67
Reserved block count:     3276
Reserved blocks uid:      0 (user root)

Root could still write because 3276 blocks (5% of 65536, about 12.8 MiB) are reserved for UID 0. This is why an app running as a normal user can fail on a “95% full” disk while root’s cron jobs carry on. On data volumes that only an app uses, admins often lower the reserve with sudo tune2fs -m 1 DEVICE; on / and /var keep the default, because that reserve is what lets you log in and clean up.

Remove the fill files with sudo rm /mnt/lab67/scratch/*.bin.

✅ Checkpoint: student got ENOSPC, root wrote 5 MiB afterwards, and tune2fs shows 3276 reserved blocks for root.

7See how XFS and the real volumes count inodes

Read-only, on rocky01’s real file systems:

df -hT / /srv/app /srv/logs
df -i / /srv/app /srv/logs
xfs_info /srv/app | grep -o 'imaxpct=[0-9]*'

You should see something like this:

Filesystem                  Type  Size  Used Avail Use% Mounted on
/dev/mapper/rl-root         xfs    17G  3.9G   13G  23% /
/dev/mapper/vg_data-lv_app  xfs   4.2G  148M  4.1G   4% /srv/app
/dev/mapper/vg_data-lv_logs ext4  1.5G  1.2M  1.4G   1% /srv/logs
Filesystem                   Inodes IUsed   IFree IUse% Mounted on
/dev/mapper/rl-root         8597504 61842 8535662    1% /
/dev/mapper/vg_data-lv_app  2190336    52 2190284    1% /srv/app
/dev/mapper/vg_data-lv_logs   98304    14   98290    1% /srv/logs
imaxpct=25

lv_logs is ext4 and has a fixed 98,304 inodes: one per 16 KiB of its 1.5 GiB, including the ones added when you grew it in Lesson 45. The two XFS file systems show millions, because XFS reports how many inodes it could still create from free space, up to 25% of the space (imaxpct). On XFS the inode total shrinks as you fill the disk with data: on a 512 MiB XFS test, writing 300 MiB took it from 262,144 to 230,056 with only 4 inodes used.

That’s why inode exhaustion is mostly an ext4 problem.

✅ Checkpoint: You can say which of the three file systems has a fixed inode count.

8Check the other usual suspects

On ubu01, run this:

findmnt /tmp
df -h /tmp /boot

You should see something like this:

TARGET SOURCE FSTYPE OPTIONS
/tmp   tmpfs  tmpfs  rw,nosuid,nodev,size=1007616k,nr_inodes=1048576,inode64
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           984M  4.0K  984M   1% /tmp
/dev/sda2       1.9G  186M  1.6G  11% /boot

On Ubuntu 26.04 /tmp is a tmpfs with size set to 50% of RAM, under 1 GB on ubu01, plus a limit of about a million inodes. A 2 GB ISO downloaded to /tmp or a big tar extracted there will fail with ENOSPC while / has plenty of room. Then on rocky01:

df -h /boot
podman system df

You should see something like this:

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2       1.9G  418M  1.5G  22% /boot
TYPE           TOTAL       ACTIVE      SIZE        RECLAIMABLE
Images         2           1           195.8MB     45.3MB (23%)
Containers     1           1           1.093kB     0B (0%)
Local Volumes  0           0           0B          0B (0%)

/boot on rocky01 is its own partition of about 2 GB, and each kernel with its initramfs and rescue image takes room (Lesson 49 showed what happens when it fills). podman system df shows what your rootless Podman from Lesson 57 keeps under ~/.local/share/containers, on the root file system; podman image prune removes unused images. Together with sudo lsof +L1 from Lesson 66, that’s the full checklist.

✅ Checkpoint: You know the /tmp size limit on ubu01 and how full /boot is on both VMs.

9Leave the test file system for Break-It, or clean up now

Break-It below uses a separate small tmpfs, so you can remove the lab file system now:

cd ~
sudo umount /mnt/lab67
sudo rmdir /mnt/lab67
rm /var/tmp/lab67.img
losetup -a

losetup -a should print nothing: a loop device set up by mount -o loop is released automatically on unmount.

✅ Checkpoint: /mnt/lab67 is gone and losetup -a prints nothing.