Read-only file system and file system errors

Everything runs on rocky01 as student with sudo, in one terminal, because steps 2 to 9 use two shell variables, LOOP and LOOPX. The test disks are image files in /var/tmp/lab78 attached as loop devices, with a device-mapper device on top that you can switch between “working disk” and “disk that fails writes”. lv_app, lv_logs and lv_db stay untouched; check every command names lab78 or lab78x before you press Enter. Rocky outputs are examples from our lab VM; your dm and loop numbers can differ.

1Check the server for read-only or failing file systems

Run this:

findmnt -O ro -t xfs,ext4
awk '$3 ~ /^(xfs|ext4)$/ && ($4 ~ /^ro,/ || $4 ~ /emergency_ro/)' /proc/mounts
sudo journalctl -k -b -p warning --no-pager | grep -iE 'ext4-fs|xfs \(|i/o error' | tail -n 5

On a healthy rocky01 you should see:

(no output)

All three print nothing. The first two find ext4 or XFS mounts that are read-only, including the emergency_ro marker newer kernels use; the third looks for file system and I/O errors in this boot’s kernel log. These three lines are worth a place in your health-check script (Lesson 39), because a read-only data volume often goes unnoticed until someone tries to save.

✅ Checkpoint: No read-only xfs or ext4 mounts and no file system or I/O errors in the kernel log since boot.

2Build a test disk you can break

Run this:

sudo mkdir -p /var/tmp/lab78
sudo truncate -s 512M /var/tmp/lab78/ext4.img
LOOP=$(sudo losetup --find --show /var/tmp/lab78/ext4.img); echo "$LOOP"
echo "0 1048576 linear $LOOP 0" | sudo dmsetup create lab78
sudo dmsetup table lab78
lsblk "$LOOP"

You should see something like this:

/dev/loop0
0 1048576 linear 7:0 0
NAME      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
loop0       7:0    0  512M  0 loop
└─lab78   253:5    0  512M  0 dm

The image file is sparse, so it uses almost no space until you write to it. losetup turns it into a block device, and dmsetup puts a device called lab78 on top with one “linear” segment: sectors 0 to 1048575 (512 MiB in 512-byte sectors) map straight onto the loop device. LVM builds its logical volumes the same way, which is why lab78 appears as major 253 next to rl-root and lv_app.

If your SSH session drops, get the variables back with LOOP=$(losetup -j /var/tmp/lab78/ext4.img | cut -d: -f1) and, from step 7 on, LOOPX=$(losetup -j /var/tmp/lab78/xfs.img | cut -d: -f1).

✅ Checkpoint: sudo dmsetup table lab78 shows one linear line and lsblk shows lab78 under your loop device.

3Put ext4 on it with errors=remount-ro

Run this:

sudo mkfs.ext4 -q -L lab78 /dev/mapper/lab78
sudo mkdir -p /mnt/lab78
sudo mount -o errors=remount-ro /dev/mapper/lab78 /mnt/lab78
printf 'order 1041\norder 1042\n' | sudo tee /mnt/lab78/orders.txt > /dev/null
sync
sudo tune2fs -l /dev/mapper/lab78 | grep -E '^(Filesystem state|Errors behavior)'
grep lab78 /proc/mounts

You should see something like this:

Filesystem state:         clean
Errors behavior:          Continue
/dev/mapper/lab78 /mnt/lab78 ext4 rw,seclabel,relatime,errors=remount-ro 0 0

The superblock default is Continue, which is what mke2fs writes unless told otherwise. The mount option errors=remount-ro overrides it for this mount, and that’s the setting you want on a data volume: stop writing at the first sign of trouble. orders.txt is your “important data”, written and synced while the disk still works.

✅ Checkpoint: /proc/mounts shows lab78 mounted rw with errors=remount-ro, and the state is clean.

4Make the disk fail under the mounted file system

Swap the table for one where everything above 240 MiB returns an error, then keep writing:

sudo dmsetup suspend --nolockfs lab78
printf '0 491520 linear %s 0\n491520 557056 error\n' "$LOOP" | sudo dmsetup reload lab78
sudo dmsetup resume lab78
sudo dmsetup table lab78
sudo cp /etc/services /mnt/lab78/report.txt; echo "cp exit=$?"
sync; echo "sync exit=$?"
sleep 10
sudo touch /mnt/lab78/new.txt

You’ll get this:

0 491520 linear 7:0 0
491520 557056 error
cp exit=0
sync exit=0
touch: cannot touch '/mnt/lab78/new.txt': Read-only file system

The error target fails every read and write in its range, the way a dead area of disk or a lost LUN does. The ext4 journal on a 512 MiB file system sits in the middle, at 256 MiB, so the next journal commit failed. Look at the two exit codes: cp and sync both said success, yet report.txt was never safely written.

The first sign anything was wrong came from the next write, a few seconds later.

dmsetup suspend pauses I/O to the device, reload loads the new table and resume switches to it, which is how LVM changes a live volume too. --nolockfs skips freezing the file system first, because a freeze would fail on a file system that is already broken.

✅ Checkpoint: touch fails with “Read-only file system” even though cp and sync returned 0.

5Read what the kernel and the superblock say

Run this:

sudo journalctl -k -n 5 --no-pager
grep lab78 /proc/mounts
sudo tune2fs -l /dev/mapper/lab78 | grep -E '^(Filesystem state|FS Error count|First error function|First error err)'

You should see something like this:

Oct 27 13:12:41 rocky01 kernel: Aborting journal on device dm-5-8.
Oct 27 13:12:41 rocky01 kernel: Buffer I/O error on dev dm-5, logical block 65536, lost sync page write
Oct 27 13:12:41 rocky01 kernel: JBD2: I/O error when updating journal superblock for dm-5-8.
Oct 27 13:12:47 rocky01 kernel: EXT4-fs error (device dm-5): ext4_journal_check_start:84: comm touch: Detected aborted journal
Oct 27 13:12:47 rocky01 kernel: EXT4-fs (dm-5): Remounting filesystem read-only
/dev/mapper/lab78 /mnt/lab78 ext4 ro,seclabel,relatime,errors=remount-ro 0 0
Filesystem state:         clean with errors
FS Error count:           1
First error function:     ext4_journal_check_start
First error err:          EIO

Read the log top down: the journal hit a write error (“lost sync page write” for block 65536, where the journal starts), jbd2 aborted it, and on the next write ext4 noticed and remounted read-only. /proc/mounts now says ro. The superblock keeps the evidence, including the function and error code, so even after a reboot e2fsck and tune2fs know this file system had an error.

On Ubuntu 26.04 (kernel 7.0) the same failure leaves the mount listed as rw,...,emergency_ro instead of ro. That’s why the awk line in step 1 checks for both.

✅ Checkpoint: The kernel log shows “Aborting journal” and “Remounting filesystem read-only” for dm-5 (or your number), and tune2fs says “clean with errors”.

6Fix the cause first, then check and repair ext4

Put the working table back (the storage team has fixed the path), unmount, and check without changing anything:

sudo dmsetup suspend --nolockfs lab78
printf '0 1048576 linear %s 0\n' "$LOOP" | sudo dmsetup reload lab78
sudo dmsetup resume lab78
sudo umount /mnt/lab78
sudo e2fsck -n /dev/mapper/lab78; echo "e2fsck -n exit=$?"

You should see something like this:

e2fsck 1.47.1 (20-May-2024)
Warning: skipping journal recovery because doing a read-only filesystem check.
lab78 contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Free blocks count wrong (124714, counted=124718).
Fix? no

Free inodes count wrong (32755, counted=32756).
Fix? no

lab78: 13/32768 files (0.0% non-contiguous), 6358/131072 blocks
e2fsck -n exit=0

The error flag forced a full check. With -n e2fsck opens the device read-only, answers no to everything and skips the journal replay, so the counts it complains about may change once the journal is replayed. It’s your preview of how bad things are, and here it’s mild.

Now repair for real and look at your data:

sudo e2fsck -f -y /dev/mapper/lab78; echo "e2fsck exit=$?"
sudo mount /dev/mapper/lab78 /mnt/lab78
ls /mnt/lab78; cat /mnt/lab78/orders.txt

You should see something like this:

e2fsck 1.47.1 (20-May-2024)
lab78: recovering journal
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Free blocks count wrong (124714, counted=124718).
Fix? yes

Free inodes count wrong (32755, counted=32756).
Fix? yes


lab78: ***** FILE SYSTEM WAS MODIFIED *****
lab78: 12/32768 files (0.0% non-contiguous), 6354/131072 blocks
e2fsck exit=1
lost+found  orders.txt
order 1041
order 1042

e2fsck replayed the journal, fixed the summary counters and exited with 1, which means “errors corrected”. orders.txt survived because it was committed before the failure. report.txt is gone, along with its inode, and no fsck can bring it back; that data has to come from the application or a backup. That’s the real cost of a read-only incident, and why you tell the app team which time window to check.

✅ Checkpoint: e2fsck reports FILE SYSTEM WAS MODIFIED with exit code 1, lab78 mounts again, orders.txt has both lines and report.txt is missing.

7Watch XFS shut down instead

Build a second test disk with XFS and fail it the same way:

sudo truncate -s 512M /var/tmp/lab78/xfs.img
LOOPX=$(sudo losetup --find --show /var/tmp/lab78/xfs.img)
echo "0 1048576 linear $LOOPX 0" | sudo dmsetup create lab78x
sudo mkfs.xfs -q -L lab78x /dev/mapper/lab78x
sudo mkdir -p /mnt/lab78x && sudo mount /dev/mapper/lab78x /mnt/lab78x
printf 'order 1041\n' | sudo tee /mnt/lab78x/orders.txt > /dev/null; sync
sudo dmsetup suspend --nolockfs lab78x
printf '0 491520 linear %s 0\n491520 557056 error\n' "$LOOPX" | sudo dmsetup reload lab78x
sudo dmsetup resume lab78x
sudo cp /etc/services /mnt/lab78x/report.txt; sync; sleep 30
ls /mnt/lab78x
grep lab78x /proc/mounts
sudo journalctl -k -n 3 --no-pager

You should see something like this:

ls: cannot access '/mnt/lab78x': Input/output error
/dev/mapper/lab78x /mnt/lab78x xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0
Oct 27 13:20:05 rocky01 kernel: XFS (dm-6): log I/O error -5
Oct 27 13:20:05 rocky01 kernel: XFS (dm-6): Filesystem has been shut down due to log error (0x2).
Oct 27 13:20:05 rocky01 kernel: XFS (dm-6): Please unmount the filesystem and rectify the problem(s).

XFS keeps its log in the middle of a file system this size too, so the log write failed and XFS shut down. Even ls of the mount point fails with “Input/output error”, yet /proc/mounts still says rw. On an XFS server, “everything gives I/O errors” with nothing read-only in the mount table is this situation.

✅ Checkpoint: ls /mnt/lab78x fails with Input/output error, /proc/mounts shows rw, and the kernel log says the file system has been shut down.

8Bring XFS back: replay the log, then check

Fix the “storage”, unmount, and look before you touch anything:

sudo dmsetup suspend --nolockfs lab78x
printf '0 1048576 linear %s 0\n' "$LOOPX" | sudo dmsetup reload lab78x
sudo dmsetup resume lab78x
sudo umount /mnt/lab78x
sudo xfs_repair -n /dev/mapper/lab78x; echo "xfs_repair -n exit=$?"

You should see something like this (shortened):

Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - zero log...
ALERT: The filesystem has valuable metadata changes in a log which is being
ignored because the -n option was used.  Expect spurious inconsistencies
which may be resolved by first mounting the filesystem to replay the log.
        - scan filesystem freespace and inode maps...
        - found root inode chunk
Phase 3 - for each AG...
...
Phase 7 - verify link counts...
No modify flag set, skipping filesystem flush and exiting.
xfs_repair -n exit=1

The ALERT is the important line. The log still holds metadata changes that never reached their final place, and the check result means little until they are applied. XFS applies its log when it mounts, so mount once, unmount, and check again:

sudo mount /dev/mapper/lab78x /mnt/lab78x
sudo journalctl -k -n 3 --no-pager -o cat
ls /mnt/lab78x
sudo umount /mnt/lab78x
sudo xfs_repair -n /dev/mapper/lab78x > /dev/null; echo "xfs_repair -n exit=$?"

You should see something like this:

XFS (dm-6): Mounting V5 Filesystem 4b8e2d51-9c07-4f3a-a6e2-71d05c3f9b84
XFS (dm-6): Starting recovery (logdev: internal)
XFS (dm-6): Ending recovery (logdev: internal)
orders.txt
xfs_repair -n exit=0

“Starting recovery” and “Ending recovery” mean the log replay worked, and the second xfs_repair -n exits 0: no corruption at all. As with ext4, the file written after the failure is gone. Often a log replay is the whole repair on XFS.

✅ Checkpoint: The mount logged “Ending recovery”, orders.txt is there, and xfs_repair -n now exits 0.

9Corrupt XFS metadata on purpose, then repair it

Make a folder with three files, unmount, and use xfs_db to mark the folder’s inode as a plain file:

sudo mount /dev/mapper/lab78x /mnt/lab78x
sudo mkdir /mnt/lab78x/invoices
for i in 1 2 3; do echo "invoice $i" | sudo tee /mnt/lab78x/invoices/inv$i.txt > /dev/null; done
INO=$(stat -c %i /mnt/lab78x/invoices); echo "invoices inode: $INO"
sudo umount /mnt/lab78x
sudo xfs_db -x -c "inode $INO" -c 'write core.mode 0100644' /dev/mapper/lab78x
sudo mount /dev/mapper/lab78x /mnt/lab78x
ls /mnt/lab78x/invoices
sudo journalctl -k -n 4 --no-pager -o cat

You’ll get something like this (your inode may differ from 132):

invoices inode: 132
core.mode = 0100644
ls: cannot access '/mnt/lab78x/invoices': Structure needs cleaning
XFS (dm-6): Metadata corruption detected at xfs_dinode_verify_fork+0x3ae/0x440, inode 0x84 dinode
XFS (dm-6): Unmount and run xfs_repair
XFS (dm-6): First 128 bytes of corrupted metadata buffer:

The inode now says “regular file” but its data fork is in directory format, and XFS’s verifier refused it. “Structure needs cleaning” is how EFSCORRUPTED reaches the user. The file system is still mounted and the rest works; only that inode is refused.

Unmount, check, then repair:

sudo umount /mnt/lab78x
sudo xfs_repair -n /dev/mapper/lab78x; echo "exit=$?"

You should see something like this (shortened):

Phase 1 - find and verify superblock...
...
Phase 3 - for each AG...
        - scan (but don't clear) agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
bad inode format in inode 132
bad inode format in inode 132
would have cleared inode 132
...
entry "invoices" in shortform directory 128 references free inode 132
would have junked entry "invoices" in directory inode 128
...
Phase 6 - check inode connectivity...
        - traversing filesystem ...
entry "invoices" in shortform directory inode 128 points to free inode 132, would junk entry
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
disconnected inode 133, would move to lost+found
disconnected inode 134, would move to lost+found
disconnected inode 135, would move to lost+found
Phase 7 - verify link counts...
would have reset inode 128 nlinks from 3 to 2
No modify flag set, skipping filesystem flush and exiting.
exit=1

sudo xfs_repair /dev/mapper/lab78x; echo "exit=$?"
sudo mount /dev/mapper/lab78x /mnt/lab78x
sudo ls -l /mnt/lab78x/lost+found; sudo cat /mnt/lab78x/lost+found/*

You should see something like this (shortened):

...
bad inode format in inode 132
cleared inode 132
...
entry "invoices" in shortform directory 128 references free inode 132
junking entry "invoices" in directory inode 128
...
Phase 6 - check inode connectivity...
        - resetting contents of realtime bitmap and summary inodes
        - traversing filesystem ...
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
disconnected inode 133, moving to lost+found
disconnected inode 134, moving to lost+found
disconnected inode 135, moving to lost+found
Phase 7 - verify and correct link counts...
resetting inode 128 nlinks from 4 to 3
done
exit=0
total 12
-rw-r--r--. 1 root root 10 Oct 27 13:31 133
-rw-r--r--. 1 root root 10 Oct 27 13:31 134
-rw-r--r--. 1 root root 10 Oct 27 13:31 135
invoice 1
invoice 2
invoice 3

xfs_repair couldn’t trust inode 132, so it cleared it and removed the “invoices” entry that pointed at it. The three files inside were still fine but had lost their folder, so it moved them to lost+found under their inode numbers. Your data survived; the names didn’t.

Reading lost+found after a repair, and telling the app team, is part of the job.

Keep both test disks for the Break-It part. If you skip it, run the clean-up block at the end of the Break-It section now.

✅ Checkpoint: xfs_repair -n exited 1 and listed the problems, xfs_repair exited 0, and lost+found holds three files with “invoice 1”, “invoice 2” and “invoice 3”.