Server stuck in emergency mode: fix fstab on Rocky and Ubuntu

You’ll work on rocky01 first and ubu01 at the end, mostly from the VirtualBox console windows, because SSH isn’t available in emergency mode. Nothing in this lab leaves a change behind: the disk goes back, and the boot overrides are typed at the GRUB menu for one boot only. rocky01’s fstab is the one Lessons 43, 44, 47 and 56 built, so your UUIDs will differ from the ones shown.

1Take snapshots of both VMs

On the laptop, run this:

VBoxManage snapshot rocky01 take before-l70
VBoxManage snapshot ubu01 take before-l70

You should see something like this, twice:

0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Snapshot taken. UUID: 4f1d2c9a-7b3e-4e05-9a61-2c8d0f5b7e13

A snapshot of a running VM saves its disks and memory, so if anything in this lesson goes wrong you can restore it in a minute. At work the equivalent is a recent backup and a tested way to reach the console (iLO, iDRAC, the hypervisor) before you touch fstab.

✅ Checkpoint: Both snapshots are listed by VBoxManage snapshot rocky01 list and VBoxManage snapshot ubu01 list.

2List which fstab lines can stop a boot

On rocky01, run this:

awk '!/^#/ && NF && $3 != "swap" {print $2, ($4 ~ /nofail/ ? "optional" : "REQUIRED"), $4}' /etc/fstab | column -t
sudo findmnt --verify

You should see something like this:

/               REQUIRED  defaults
/boot           REQUIRED  defaults
/boot/efi       REQUIRED  umask=0077,shortname=winnt
/srv/data1      optional  defaults,nofail
/srv/data2      optional  defaults,nofail,noatime,x-systemd.device-timeout=10s
/srv/app        REQUIRED  defaults
/srv/logs       REQUIRED  defaults
/mnt/share      optional  _netdev,nofail
/var/lib/mysql  REQUIRED  defaults
Success, no errors or warnings detected

awk skips comments, empty lines and swap, and marks every line without nofail as REQUIRED. Six lines on rocky01 can stop the boot: the three from the installer, lv_app and lv_logs from Lesson 44, and lv_db from Lesson 56. The data disk lines from Lesson 43 and the NFS share from Lesson 47 can’t, because they have nofail.

At work, run this on any server before patching, and ask about every REQUIRED line that isn’t on the boot disk.

✅ Checkpoint: You can name the REQUIRED lines on your rocky01 and findmnt reports Success.

3See the units systemd built from fstab

Run this:

systemctl cat srv-data2.mount srv-app.mount
ls /run/systemd/generator/local-fs.target.requires/ /run/systemd/generator/local-fs.target.wants/
cat /run/systemd/generator/dev-disk-by*.device.d/50-device-timeout.conf

You should see something like this:

# /run/systemd/generator/srv-data2.mount
# Automatically generated by systemd-fstab-generator

[Unit]
Documentation=man:fstab(5) man:systemd-fstab-generator(8)
SourcePath=/etc/fstab
Requires=systemd-fsck@dev-disk-by\x2duuid-fcb309a2\x2de657\x2d4e8f\x2db20d\x2dbf5aa8cce041.service
After=systemd-fsck@dev-disk-by\x2duuid-fcb309a2\x2de657\x2d4e8f\x2db20d\x2dbf5aa8cce041.service
After=blockdev@dev-disk-by\x2duuid-fcb309a2\x2de657\x2d4e8f\x2db20d\x2dbf5aa8cce041.target

[Mount]
What=/dev/disk/by-uuid/fcb309a2-e657-4e8f-b20d-bf5aa8cce041
Where=/srv/data2
Type=ext4
Options=defaults,nofail,noatime

# /run/systemd/generator/srv-app.mount
# Automatically generated by systemd-fstab-generator

[Unit]
Documentation=man:fstab(5) man:systemd-fstab-generator(8)
SourcePath=/etc/fstab
Before=local-fs.target
After=blockdev@dev-disk-by\x2duuid-6b1f3c2e\x2d9a4d\x2d4e57\x2d8c1b\x2d2f7d0a9e5c31.target

[Mount]
What=/dev/disk/by-uuid/6b1f3c2e-9a4d-4e57-8c1b-2f7d0a9e5c31
Where=/srv/app
Type=xfs
/run/systemd/generator/local-fs.target.requires/:
-.mount  boot-efi.mount  boot.mount  srv-app.mount  srv-logs.mount  var-lib-mysql.mount

/run/systemd/generator/local-fs.target.wants/:
srv-data1.mount  srv-data2.mount  systemd-remount-fs.service
# Automatically generated by systemd-fstab-generator
# from supplied options "defaults,nofail,noatime,x-systemd.device-timeout=10s"

[Unit]
JobRunningTimeoutSec=10s

Compare the two. srv-app.mount has Before=local-fs.target and sits in the .requires folder, so local-fs.target can’t finish without it. srv-data2.mount has neither; it’s only wanted. Its x-systemd.device-timeout=10s became a drop-in for the .device unit, so systemd waits 10 seconds for that disk instead of 90. nofail and the timeout don’t appear in Options= because they are instructions for systemd, not for the kernel.

✅ Checkpoint: srv-app.mount is in local-fs.target.requires and srv-data2.mount in local-fs.target.wants.

4Pull a disk and watch nofail save the boot

Shut rocky01 down, take away its sdc disk (the one with data1 and data2) on the laptop, and start it again:

sudo systemctl poweroff

VBoxManage storageattach rocky01 --storagectl=SATA --port=2 --device=0 --medium=none
VBoxManage startvm rocky01

About 30 seconds after it starts, log in over SSH and run this:

systemd-analyze
systemctl list-jobs --no-pager
findmnt /srv/data1 /srv/data2; echo "findmnt exit=$?"
lsblk -d -o NAME,SIZE,FSTYPE

You should see something like this:

Bootup is not yet finished (org.freedesktop.systemd1.Manager.FinishTimestampMonotonic=0).
Please try again later.
Hint: Use 'systemctl list-jobs' to see active jobs
JOB UNIT                                                                       TYPE  STATE
 97 srv-data1.mount                                                            start waiting
 98 dev-disk-by\x2duuid-f7624e63\x2d8b7a\x2d4eaf\x2db92e\x2d2ddf3b10c1f7.device start running

2 jobs listed.
findmnt exit=1
NAME  SIZE FSTYPE
sda    20G
sdb     5G LVM2_member
sdc     5G LVM2_member
sr0  1024M

SSH worked and every service is up, yet systemd says the boot isn’t finished: it’s still waiting the full 90 seconds for the data1 disk in the background, because that line has nofail but no timeout. data2 already gave up after 10 seconds. Note the disk names too: with port 2 empty, the Lesson 45 disk that used to be sdd is now sdc, and nothing broke because LVM and fstab use UUIDs.

Wait two minutes, then read what happened:

journalctl -b --no-pager | grep -E 'Timed out waiting|Dependency failed for srv'
systemctl is-active srv-data1.mount srv-data2.mount

You should see something like this:

Oct 27 16:05:24 rocky01 systemd[1]: Timed out waiting for device dev-disk-by\x2duuid-fcb309a2\x2de657\x2d4e8f\x2db20d\x2dbf5aa8cce041.device - /dev/disk/by-uuid/fcb309a2-e657-4e8f-b20d-bf5aa8cce041.
Oct 27 16:05:24 rocky01 systemd[1]: Dependency failed for srv-data2.mount - /srv/data2.
Oct 27 16:06:44 rocky01 systemd[1]: Timed out waiting for device dev-disk-by\x2duuid-f7624e63\x2d8b7a\x2d4eaf\x2db92e\x2d2ddf3b10c1f7.device - /dev/disk/by-uuid/f7624e63-8b7a-4eaf-b92e-2ddf3b10c1f7.
Oct 27 16:06:44 rocky01 systemd[1]: Dependency failed for srv-data1.mount - /srv/data1.
inactive
inactive

The two timeouts are 80 seconds apart: 10 seconds for data2, 90 for data1. The mounts are inactive, not failed, so systemctl --failed shows nothing and only your monitoring of mount points would notice. That’s the price of nofail, and why Lesson 43 pairs it with a check that the mount is there.

✅ Checkpoint: rocky01 booted without sdc, /srv/data1 and /srv/data2 are empty, and the journal shows both timeouts.

5Put the disk back

Shut down, re-attach sdc on the laptop and start again:

sudo systemctl poweroff

VBoxManage storageattach rocky01 --storagectl=SATA --port=2 --device=0 --type=hdd --medium="C:\Lab\Linux\rocky01\rocky01-data2.vdi" --hotpluggable=on
VBoxManage startvm rocky01

Log in and check:

findmnt -n -o TARGET,SOURCE /srv/data1 /srv/data2
lsblk -d -o NAME,SIZE,FSTYPE | grep -c LVM2_member

You should see something like this:

/srv/data1 /dev/sdc1
/srv/data2 /dev/sdc2
2

On macOS or Linux the path is ~/Lab/Linux/rocky01/rocky01-data2.vdi. The disk is back on port 2, so it’s sdc again and the Lesson 45 disk is sdd.

✅ Checkpoint: /srv/data1 and /srv/data2 are mounted from sdc1 and sdc2 again.

6Practise the emergency shell on rocky01

Restart rocky01 from the console (Machine, Reset). When the boot menu appears, highlight the top entry, press e, go to the end of the line that starts with linux and add:

systemd.unit=emergency.target

Press Ctrl+X to boot. You’ll see this on the console:

You are in emergency mode. After logging in, type "journalctl -xb" to view
system logs, "systemctl reboot" to reboot, or "exit"
to continue bootup.

Give root password for maintenance
(or press Control-D to continue):

Type the root password you set in Lesson 2, then run this:

findmnt -no OPTIONS /
mount -o remount,rw /
findmnt -no OPTIONS /
findmnt /boot /srv/app; echo "findmnt exit=$?"

You should see something like this:

ro,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota
rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota
findmnt exit=1

Booted straight into emergency.target, the root file system is still read-only as the initramfs left it, because nothing ran the service that remounts it. That’s why editing fstab here fails with “Read-only file system” until you remount. Only / and the kernel’s own file systems are mounted; findmnt finds neither /boot nor /srv/app.

When you land in emergency mode because of a failed mount, / is usually already read-write, so always check before you remount.

Carry on to the normal boot:

systemctl default

✅ Checkpoint: You got a root prompt with the root password, saw ro change to rw, and rocky01 then booted normally with SSH working.

7Try the same on ubu01, where root is locked

Reset ubu01 and open its boot menu (hold Shift on a BIOS VM, press Esc on a UEFI one). Press e on the first Ubuntu entry, add the same text to the end of the linux line, and boot with Ctrl+X:

systemd.unit=emergency.target

You’ll get this on the console:

You are in emergency mode. After logging in, type "journalctl -xb" to view
system logs, "systemctl reboot" to reboot, or "exit"
to continue bootup.

Cannot open access to console, the root account is locked.
See sulogin(8) man page for more details.

Press Enter to continue.

Ubuntu ships with no root password, so sulogin has nothing to check and refuses. Pressing Enter only shows the message again. Reset ubu01, open the menu again, and this time add both of these to the end of the linux line:

systemd.unit=emergency.target systemd.setenv=SYSTEMD_SULOGIN_FORCE=1

You should see something like this:

You are in emergency mode. After logging in, type "journalctl -xb" to view
system logs, "systemctl reboot" to reboot, or "exit"
to continue bootup.

Press Enter for maintenance
(or press Control-D to continue):
root@ubu01:~#

systemd.setenv= puts a variable into the environment systemd gives its services, and SYSTEMD_SULOGIN_FORCE=1 tells sulogin to open a root shell when root has no usable password. It needs physical or console access to the boot menu, which is the same trust you need for any recovery. Type systemctl default to carry on, and check that SSH works again.

If your boot menu is protected with a GRUB password (Lesson 71 shows how), you’ll need that password before you can press e. That’s the point of it, so keep it in the same vault as the root passwords.

✅ Checkpoint: ubu01 refused the plain emergency shell, gave you a root prompt with SYSTEMD_SULOGIN_FORCE=1, and then booted normally.