GRUB and dracut: boot parameters and initramfs rebuilds
GRUB and dracut are the two tools you touch when a server needs a different boot parameter or a rebuilt initramfs. GRUB2 decides which kernel starts and which command line it gets, and dracut builds the small initramfs image that the kernel uses to find your root disk. Both run before any service starts, so a mistake here means a server that doesn’t come back, and the fix is done at the console.
In Lesson 9 you watched the boot chain from the outside, and in Lesson 49 you patched kernels and picked the default one. Now you’ll change kernel arguments the supported way on Rocky 10 and Ubuntu 26.04, add a storage driver to the initramfs the way you would before a migration, and bring back a VM that won’t boot because of one wrong word on the kernel line.
Where this lesson fits
Lesson 50 of 82 in Linux Administration · Phase 3: Enterprise infrastructure and storage · Before this: Lesson 49, Linux kernel patching · Next: Lesson 51, kernel tuning with sysctl and limits
| Time | You need | By the end you can |
|---|---|---|
| About 90 minutes | rocky01 and ubu01, the VirtualBox console window for both (SSH alone is not enough), and a snapshot before the Break-It part | Add and remove kernel arguments on Rocky and Ubuntu, rebuild and inspect an initramfs, and boot a broken VM by editing its GRUB entry |
What it is, in one line
GRUB passes the kernel its command line and dracut packs the drivers the kernel needs to reach the root disk, so you change boot options with grubby or update-grub and rebuild the initramfs with dracut.
In Simple Words
The trip sheet and the driver’s bag
Think of an overnight bus from Bhubaneswar to Kolkata. Before it leaves, the depot clerk writes a trip sheet: which bus, which route, which stops, any special note such as “no AC tonight”. The driver doesn’t argue with the trip sheet, he follows it.
The driver also carries a small bag with only what he needs to reach the first big stop: the keys, a route card and the phone numbers for that road. If the bus company swaps his bus for a different make at the last minute, the old key in that bag won’t start it, and the bus stays in the depot even though the road and the passengers are fine.
The trip sheet is the kernel command line that GRUB hands over, and the small bag is the initramfs that dracut packs.
Engineer's View
Where each piece lives on Rocky 10 and Ubuntu 26.04
| Job | Rocky Linux 10 / RHEL 10 | Ubuntu 26.04 |
|---|---|---|
| Config GRUB reads at boot | /boot/grub2/grub.cfg plus one BLS file per kernel in /boot/loader/entries/. On UEFI, /boot/efi/EFI/rocky/grub.cfg is a four-line stub that points at the real file | /boot/grub/grub.cfg. On UEFI, /boot/efi/EFI/ubuntu/grub.cfg is a small stub too |
| Where you set defaults | /etc/default/grub (GRUB_CMDLINE_LINUX, GRUB_TIMEOUT) | /etc/default/grub and drop-ins in /etc/default/grub.d/*.cfg |
| Change kernel arguments | grubby --update-kernel=ALL --args="..." or --remove-args. Used at the next boot, no rebuild needed | Edit the file, then update-grub |
| Rebuild grub.cfg (rarely needed) | grub2-mkconfig -o /boot/grub2/grub.cfg, on BIOS and UEFI alike | update-grub (a wrapper for grub-mkconfig -o /boot/grub/grub.cfg) |
| Build the initramfs | dracut -f for the running kernel, dracut -f --regenerate-all for all | update-initramfs -u, which is a small wrapper that calls dracut |
| Image file and viewer | /boot/initramfs-KVER.img, lsinitrd | /boot/initrd.img-KVER, lsinitrd |
| Your dracut settings | /etc/dracut.conf.d/*.conf | /etc/dracut.conf.d/*.conf |
| What the image holds | Host-only: only the drivers this machine uses today, plus a separate generic rescue image | Generic: most storage drivers, so it’s bigger but more forgiving |
Rocky and RHEL since version 8 follow the Boot Loader Specification. Each kernel has its own small file in /boot/loader/entries/ with a title, the kernel and initramfs paths and an options line, and GRUB’s blscfg command reads those files at boot. That’s why grubby edits the entry files directly, and why installing a new kernel doesn’t need grub2-mkconfig at all.
With --update-kernel=ALL, grubby also writes the same change into GRUB_CMDLINE_LINUX in /etc/default/grub, so a later grub2-mkconfig doesn’t quietly undo it, and kernels you install later take the same arguments. With a single kernel path, only that one entry changes, which is what you want for a test.
Ubuntu builds one grub.cfg from templates in /etc/grub.d, and every linux line gets GRUB_CMDLINE_LINUX followed by GRUB_CMDLINE_LINUX_DEFAULT. The recovery entries only get the first one, so anything you need in recovery mode too belongs in GRUB_CMDLINE_LINUX.
The host-only difference matters at work. Rocky’s dracut is set to hostonly="yes", so the image on a VMware VM carries the VMware disk driver and nothing else, while the “0-rescue” entry has a generic image with almost every driver. Ubuntu’s dracut configuration doesn’t set hostonly, so its images stay generic.
How it works, step by step
Here is the path a kernel argument takes, from the moment you type a command to the moment the kernel uses it. The diagram after these steps shows the same path.
1You change the source, not the output
On Rocky that’s grubby, which rewrites the options line in the BLS entry files and keeps /etc/default/grub in step. On Ubuntu it’s /etc/default/grub or, better, a drop-in in /etc/default/grub.d, because a package update may want to replace the main file and will ask you about your edits.
2The config GRUB reads is written
On Rocky the entry files are already what GRUB reads, so you’re done. On Ubuntu, update-grub sources /etc/default/grub and every .cfg in grub.d, runs the scripts in /etc/grub.d and writes a fresh /boot/grub/grub.cfg. Until you run it, the change exists only in a text file that nothing at boot looks at.
3GRUB passes the line at boot
GRUB loads the kernel and initramfs from /boot and hands over the command line. You can still change it for one boot from the menu by pressing e, which is your way back in when a bad argument stops the boot. After boot, /proc/cmdline shows what the kernel received.
4Each part of the system takes its own arguments
The kernel itself uses arguments such as root=, ro, quiet and audit=1, and a driver takes module.parameter=value. dracut’s scripts in the initramfs read the rd. ones (rd.lvm.lv=, rd.break), and systemd reads the systemd. ones you met in Lesson 9. Anything nobody recognises is passed on to user space, and the kernel logs a line saying so.
5dracut builds the initramfs
dracut reads /usr/lib/dracut/dracut.conf.d, then /etc/dracut.conf and /etc/dracut.conf.d, picks its own modules (systemd, lvm, dm and the rest) and the kernel drivers they need, copies config files such as lvm.conf and crypttab, and packs everything into one compressed cpio archive. In host-only mode it asks the running system which drivers it uses right now and leaves out the rest.
6When you need to rebuild it
Kernel updates build a new image on their own (kernel-install on Rocky, /etc/kernel/postinst.d/dracut on Ubuntu). You rebuild by hand after you add a driver or dracut setting, after you change a file the image carries a copy of (an LVM filter, /etc/multipath.conf, crypttab, a module blacklist), and before a VM moves to a different hypervisor or disk controller.
The picture below shows what goes into the image and where each part comes from. When a server boots fine on old hardware and drops to a dracut shell on new hardware, one of these four boxes is usually missing something.
Editing at the GRUB menu is the skill that saves you at 3 a.m. Highlight the entry, press e, move to the line that starts with linux, change it, and press Ctrl+X (or F10) to boot. Nothing is saved, so the next normal boot uses the stored entry again, and that’s why you then make the real fix with grubby or update-grub.
Rocky shows the menu for 5 seconds. Ubuntu hides it (GRUB_TIMEOUT_STYLE=hidden), so you press Esc on a UEFI VM or hold Shift on a BIOS VM as it starts, the same trick as Lesson 9. Lesson 71 uses this menu again with rd.break to reset a lost root password.
Hands-on lab: change kernel arguments and rebuild the initramfs on rocky01 and ubu01
Before you run any command, double-check it
Read every command twice before you press Enter. Check the target: the server name, the user, the disk or device name, the path and the sizes. These commands are for learning and were tested in our lab, not on your system. At work, run the read-only commands first, follow your change process, make sure a backup exists, and ask a second person to check anything that changes or deletes data. You run them at your own risk.
Steps 1 to 4 and 6 to 8 are on rocky01, step 5 is on ubu01, and step 9 puts both back the way they were. Two reboots are needed, so keep both console windows open. Every change here is undone at the end, and you’ll make the same kind of changes for real in later lessons (kdump in Lesson 79, hardening in Lesson 55).
1Look at the GRUB files on rocky01
Run this:
grep -v '^#' /etc/default/grub
sudo ls /boot/loader/entries/
sudo cat /boot/loader/entries/*$(uname -r).confYou should see something like this:
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=2G-64G:256M,64G-:512M resume=/dev/mapper/rl-swap rd.lvm.lv=rl/root rd.lvm.lv=rl/swap rhgb quiet"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true
8c1f2e7a4b9d4f6e9a2c3b1d5e7f9a0c-0-rescue.conf
8c1f2e7a4b9d4f6e9a2c3b1d5e7f9a0c-6.12.0-211.16.1.el10_2.0.1.x86_64.conf
title Rocky Linux (6.12.0-211.16.1.el10_2.0.1.x86_64) 10.2 (Red Quartz)
version 6.12.0-211.16.1.el10_2.0.1.x86_64
linux /vmlinuz-6.12.0-211.16.1.el10_2.0.1.x86_64
initrd /initramfs-6.12.0-211.16.1.el10_2.0.1.x86_64.img
options root=/dev/mapper/rl-root ro crashkernel=2G-64G:256M,64G-:512M resume=/dev/mapper/rl-swap rd.lvm.lv=rl/root rd.lvm.lv=rl/swap rhgb quiet
grub_users $grub_users
grub_arg --unrestricted
grub_class rocky/etc/default/grub holds the defaults, with GRUB_ENABLE_BLSCFG=true switching on the entry files. There are two entries: your kernel and the rescue kernel (one more if you kept an older kernel in Lesson 49). Each file is named after the machine ID and the kernel version, and the options line is the kernel command line for that entry.
If you also see $tuned_params at the end of the options and $tuned_initrd after the initrd, those are GRUB variables that tuned uses for its own boot options. Leave them alone.
✅ Checkpoint: You can point at the options line and see that it matches GRUB_CMDLINE_LINUX plus root= and ro.
2Ask grubby what it knows
Run this:
sudo grubby --default-kernel
sudo grubby --info=DEFAULTYou should see something like this:
/boot/vmlinuz-6.12.0-211.16.1.el10_2.0.1.x86_64
index=0
kernel="/boot/vmlinuz-6.12.0-211.16.1.el10_2.0.1.x86_64"
args="ro crashkernel=2G-64G:256M,64G-:512M resume=/dev/mapper/rl-swap rd.lvm.lv=rl/root rd.lvm.lv=rl/swap rhgb quiet"
root="/dev/mapper/rl-root"
initrd="/boot/initramfs-6.12.0-211.16.1.el10_2.0.1.x86_64.img"
title="Rocky Linux (6.12.0-211.16.1.el10_2.0.1.x86_64) 10.2 (Red Quartz)"
id="8c1f2e7a4b9d4f6e9a2c3b1d5e7f9a0c-6.12.0-211.16.1.el10_2.0.1.x86_64"grubby reads the same entry files and shows root= on its own line, apart from the other args. --info=ALL lists every entry with its index, which is the number GRUB uses for saved_entry (Lesson 49).
✅ Checkpoint: grubby’s default kernel matches the output of uname -r.
3Add an argument to every kernel on Rocky
Run this:
sudo grubby --update-kernel=ALL --args="audit=1 audit_backlog_limit=8192"
sudo grubby --info=ALL | grep -E '^(kernel|args)'
grep GRUB_CMDLINE_LINUX /etc/default/grubYou should see something like this:
kernel="/boot/vmlinuz-6.12.0-211.16.1.el10_2.0.1.x86_64"
args="ro crashkernel=2G-64G:256M,64G-:512M resume=/dev/mapper/rl-swap rd.lvm.lv=rl/root rd.lvm.lv=rl/swap rhgb quiet audit=1 audit_backlog_limit=8192"
kernel="/boot/vmlinuz-0-rescue-8c1f2e7a4b9d4f6e9a2c3b1d5e7f9a0c"
args="ro crashkernel=2G-64G:256M,64G-:512M resume=/dev/mapper/rl-swap rd.lvm.lv=rl/root rd.lvm.lv=rl/swap rhgb quiet audit=1 audit_backlog_limit=8192"
GRUB_CMDLINE_LINUX="crashkernel=2G-64G:256M,64G-:512M resume=/dev/mapper/rl-swap rd.lvm.lv=rl/root rd.lvm.lv=rl/swap rhgb quiet audit=1 audit_backlog_limit=8192"grubby prints nothing when it succeeds. Both entries, including rescue, now end with the two audit arguments, and GRUB_CMDLINE_LINUX got them too, so a future grub2-mkconfig keeps them. audit=1 starts kernel auditing before auditd is up and audit_backlog_limit=8192 gives early audit messages room to queue, a pair CIS benchmarks ask for and Lesson 55 comes back to.
Running grubby with --args for a key that’s already there replaces its value, it doesn’t add a second copy. That’s handy for crashkernel= and dangerous for root=, as the Break-It part shows.
✅ Checkpoint: Both entries and GRUB_CMDLINE_LINUX end with audit=1 audit_backlog_limit=8192.
4Reboot and prove the kernel got it
Reboot, log in again, then run:
sudo systemctl reboot
cat /proc/cmdline
sudo journalctl -k -b | grep -m1 'Command line'You should see something like this:
BOOT_IMAGE=(hd0,gpt2)/vmlinuz-6.12.0-211.16.1.el10_2.0.1.x86_64 root=/dev/mapper/rl-root ro crashkernel=2G-64G:256M,64G-:512M resume=/dev/mapper/rl-swap rd.lvm.lv=rl/root rd.lvm.lv=rl/swap rhgb quiet audit=1 audit_backlog_limit=8192
Oct 27 11:42:06 rocky01 kernel: Command line: BOOT_IMAGE=(hd0,gpt2)/vmlinuz-6.12.0-211.16.1.el10_2.0.1.x86_64 root=/dev/mapper/rl-root ro crashkernel=2G-64G:256M,64G-:512M resume=/dev/mapper/rl-swap rd.lvm.lv=rl/root rd.lvm.lv=rl/swap rhgb quiet audit=1 audit_backlog_limit=8192/proc/cmdline is the truth for this boot, and the kernel log keeps the same line from the very start of the boot. Make checking it a habit, because a change in a file that never reached the kernel is the most common “I set it, but it didn’t work” ticket.
✅ Checkpoint: /proc/cmdline on rocky01 ends with the two audit arguments.
5The same change on Ubuntu, with a drop-in
On ubu01, look at the defaults first:
grep -E '^GRUB_(DEFAULT|TIMEOUT|CMDLINE)' /etc/default/grubYou should see something like this:
GRUB_DEFAULT=0
GRUB_TIMEOUT_STYLE=hidden
GRUB_TIMEOUT=0
GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_CMDLINE_LINUX=""Add a drop-in, rebuild grub.cfg and check the result before you reboot:
sudo tee /etc/default/grub.d/90-saat-audit.cfg >/dev/null <<'EOF'
# Lesson 50: early kernel auditing
GRUB_CMDLINE_LINUX="$GRUB_CMDLINE_LINUX audit=1 audit_backlog_limit=8192"
EOF
sudo update-grub
grep -m1 -P '^\tlinux\t' /boot/grub/grub.cfgYou should see something like this:
Sourcing file `/etc/default/grub'
Sourcing file `/etc/default/grub.d/90-saat-audit.cfg'
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-7.0.0-15-generic
Found initrd image: /boot/initrd.img-7.0.0-15-generic
Warning: os-prober will not be executed to detect other bootable partitions.
Systems on them will not be added to the GRUB boot configuration.
Check GRUB_DISABLE_OS_PROBER documentation entry.
done
linux /vmlinuz-7.0.0-15-generic root=/dev/mapper/ubuntu--vg-ubuntu--lv ro audit=1 audit_backlog_limit=8192update-grub sourced the main file and then your drop-in, which appends to the variable instead of replacing it. On a UEFI VM you’ll also see a line about a UEFI Firmware Settings entry. The grep shows the first linux line in the new grub.cfg, so you know the change is there before you risk a reboot.
Reboot ubu01 and check:
sudo systemctl reboot
cat /proc/cmdlineYou should see something like this:
BOOT_IMAGE=/vmlinuz-7.0.0-15-generic root=/dev/mapper/ubuntu--vg-ubuntu--lv ro audit=1 audit_backlog_limit=8192✅ Checkpoint: /proc/cmdline on ubu01 shows audit=1 audit_backlog_limit=8192.
6See which disk drivers the Rocky initramfs carries
Back on rocky01, run this:
sudo lsinitrd | grep -E '/(ahci|virtio_scsi|virtio_blk)\.ko'
modinfo -F filename virtio_scsi virtio_blkYou should see something like this:
-rw-r--r-- 1 root root 43512 Oct 20 18:20 usr/lib/modules/6.12.0-211.16.1.el10_2.0.1.x86_64/kernel/drivers/ata/ahci.ko.xz
/lib/modules/6.12.0-211.16.1.el10_2.0.1.x86_64/kernel/drivers/scsi/virtio_scsi.ko.xz
/lib/modules/6.12.0-211.16.1.el10_2.0.1.x86_64/kernel/drivers/block/virtio_blk.ko.xzThe image has ahci, the SATA driver for VirtualBox’s controller, and nothing for virtio, even though both virtio drivers are installed on disk under /lib/modules. That’s host-only mode at work: rocky01 boots from SATA today, so dracut packed only the SATA driver. If this VM moved to a KVM host with virtio disks, the kernel would never see its root disk.
✅ Checkpoint: ahci is in the image and virtio_scsi is not.
7Add a driver with a dracut drop-in and rebuild
Keep a copy of the working image, add the setting, rebuild and check:
sudo cp -p /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.bak
echo 'add_drivers+=" virtio_scsi virtio_blk "' | sudo tee /etc/dracut.conf.d/90-virtio.conf
sudo dracut -f -v 2>&1 | tail -n 4
sudo lsinitrd | grep -E '/(ahci|virtio_scsi|virtio_blk)\.ko'You should see something like this:
add_drivers+=" virtio_scsi virtio_blk "
dracut[I]: *** Hardlinking files done ***
dracut[I]: *** Generating early-microcode cpio image ***
dracut[I]: *** Creating image file '/boot/initramfs-6.12.0-211.16.1.el10_2.0.1.x86_64.img' ***
dracut[I]: *** Creating initramfs image file '/boot/initramfs-6.12.0-211.16.1.el10_2.0.1.x86_64.img' done ***
-rw-r--r-- 1 root root 43512 Oct 20 18:20 usr/lib/modules/6.12.0-211.16.1.el10_2.0.1.x86_64/kernel/drivers/ata/ahci.ko.xz
-rw-r--r-- 1 root root 19840 Oct 20 18:20 usr/lib/modules/6.12.0-211.16.1.el10_2.0.1.x86_64/kernel/drivers/block/virtio_blk.ko.xz
-rw-r--r-- 1 root root 13968 Oct 20 18:20 usr/lib/modules/6.12.0-211.16.1.el10_2.0.1.x86_64/kernel/drivers/scsi/virtio_scsi.ko.xzThe spaces inside the quotes matter: dracut joins every add_drivers+= it finds, and without them two files would glue their names into one word that matches no driver. -f overwrites the existing image for the running kernel and -v shows what it does. The new image now has both virtio drivers next to ahci, and it’s a little bigger.
✅ Checkpoint: lsinitrd lists ahci, virtio_blk and virtio_scsi.
8Rebuild all images, and the Ubuntu way
On rocky01:
sudo dracut -f --regenerate-all
ls -lh /boot/initramfs-*--regenerate-all rebuilds the image of every installed kernel, which is what you want before a migration, because the old kernel is your fallback and needs the driver too. The rescue image isn’t touched; it’s generic already.
On ubu01:
sudo update-initramfs -u -k all
modinfo -F filename virtio_scsiYou should see something like this for the first command:
update-initramfs: Generating /boot/initrd.img-7.0.0-15-genericOn Ubuntu 26.04 update-initramfs is a wrapper that calls dracut with Ubuntu’s file name. Without -k, -u rebuilds only the newest installed kernel, which may not be the one you’re running, so say -k all or -k $(uname -r). If modinfo prints (builtin), the driver is compiled into Ubuntu’s kernel and needs no initramfs entry at all.
✅ Checkpoint: Each kernel on rocky01 has a fresh initramfs file and ubu01 printed one “Generating” line per kernel.
9Put both VMs back
On rocky01:
sudo grubby --update-kernel=ALL --remove-args="audit=1 audit_backlog_limit=8192"
sudo rm /etc/dracut.conf.d/90-virtio.conf /boot/initramfs-$(uname -r).img.bak
sudo dracut -f --regenerate-all
sudo grubby --info=DEFAULT | grep ^argsOn ubu01:
sudo rm /etc/default/grub.d/90-saat-audit.cfg
sudo update-grubReboot both when it suits you. The order matters on Rocky: remove the drop-in first, then rebuild, or the rebuild packs the virtio drivers again.
✅ Checkpoint: After a reboot, /proc/cmdline on both VMs no longer shows audit=1, and lsinitrd on rocky01 shows no virtio drivers.
Break It, Then Fix It
Take a snapshot of rocky01 and ubu01 first (Machine, Take Snapshot, “before-l50-break”). Failure 1 stops rocky01 from booting, so you need its console window and about five minutes.
1One typo in root= and rocky01 won’t boot
Change only the default entry, so the rescue entry stays as a second way in:
sudo grubby --update-kernel=/boot/vmlinuz-$(uname -r) --args="root=/dev/mapper/rl-rooot"
sudo grubby --info=DEFAULT | grep ^root
sudo systemctl rebootAfter about three minutes of waiting, the console shows something like this:
[ TIME ] Timed out waiting for device dev-mapper-rl\x2drooot.device - /dev/mapper/rl-rooot.
[DEPEND] Dependency failed for initrd-root-device.target - Initrd Root Device.
dracut-initqueue[412]: Warning: dracut-initqueue: timeout, still waiting for following initqueue hooks:
dracut-initqueue[412]: Warning: Could not boot.
Starting dracut-emergency.service - Dracut Emergency Shell...
Warning: /dev/mapper/rl-rooot does not exist
Generating "/run/initramfs/rdsosreport.txt"
Entering emergency mode. Exit the shell to continue.
Type "journalctl" to view system logs.
You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot
after mounting them and attach it to a bug report.
dracut:/#The kernel and initramfs loaded fine, and LVM even activated the volumes, but the initramfs was told to wait for a device name that will never appear. You’re in the dracut shell inside the initramfs, and ls /dev/mapper there shows rl-root sitting right next to the name it was looking for. That one look tells you the disk is fine and the kernel line is wrong.
Fix, part 1: in VirtualBox choose Machine, Reset. When the GRUB menu appears, keep the first entry highlighted, press e, find the line that starts with linux and correct rl-rooot to rl-root. Press Ctrl+X to boot this once.
Fix, part 2: once you’re logged in, make the fix permanent and check it:
sudo grubby --update-kernel=/boot/vmlinuz-$(uname -r) --args="root=/dev/mapper/rl-root"
sudo grubby --info=DEFAULT | grep ^root
sudo systemctl rebootThe “0-rescue” entry would also have booted, because you changed only the default entry. Keep that in mind at work: when one entry is broken, try another before you reach for rescue media.
✅ Checkpoint: rocky01 boots on its own again and grubby shows root=”/dev/mapper/rl-root”.
2An unclosed quote breaks update-grub on ubu01
Make the classic editing slip, a missing closing quote:
echo 'GRUB_CMDLINE_LINUX="$GRUB_CMDLINE_LINUX consoleblank=0' | sudo tee /etc/default/grub.d/91-typo.cfg
stat -c %y /boot/grub/grub.cfg
sudo update-grubYou’ll get this error:
Sourcing file `/etc/default/grub'
Sourcing file `/etc/default/grub.d/91-typo.cfg'
/usr/sbin/grub-mkconfig: 2: /etc/default/grub.d/91-typo.cfg: Syntax error: Unterminated quoted stringgrub-mkconfig is a /bin/sh script, which on Ubuntu is dash, and it reads your files with the shell’s dot command, so a shell syntax error in any of them stops the whole run. The number is the line where dash gave up (the end of the file), not always the line with the mistake. The good news is that grub-mkconfig fails before it starts writing, so your working grub.cfg is untouched, and stat shows the same time as before.
Fix: correct or remove the file, run update-grub again and confirm grub.cfg was rewritten:
sudo rm /etc/default/grub.d/91-typo.cfg
sudo update-grub
stat -c %y /boot/grub/grub.cfg✅ Checkpoint: update-grub ends with “done” and grub.cfg has a new time.
3Follow an old blog post on a UEFI Rocky server
Many older guides tell you to write grub.cfg into the EFI partition. Try it on rocky01:
sudo grub2-mkconfig -o /boot/efi/EFI/rocky/grub.cfgYou’ll get this error:
Running `grub2-mkconfig -o /boot/efi/EFI/rocky/grub.cfg' will overwrite the GRUB wrapper.
Please run `grub2-mkconfig -o /boot/grub2/grub.cfg' instead to update grub.cfg.
GRUB configuration file was not updated.On RHEL 7 and 8 with UEFI the real config lived in the EFI partition. Since RHEL 9 that file is a small stub that finds /boot and loads /boot/grub2/grub.cfg, so the same path works for BIOS and UEFI, and grub2-mkconfig now refuses to overwrite the stub. On an older system without this check, the command would replace the stub, and the next boot would show a menu built from the wrong place.
Fix: use the one path for both firmware types, and only when you need a full rebuild:
sudo cat /boot/efi/EFI/rocky/grub.cfg
sudo grub2-mkconfig -o /boot/grub2/grub.cfg✅ Checkpoint: The stub shows a configfile line pointing at grub.cfg under /grub2, and grub2-mkconfig ends with “done”. Restore the “before-l50-break” snapshots if anything else looks wrong.
This happened at work
The migration that stopped at a dracut shell
My team lead tells this one from her time at a bus booking app in Cuttack. It was June, two weeks before Rath Yatra, when bookings to Puri go through the roof, and the company was moving its 18 Rocky Linux 9 VMs from an ageing VMware cluster to a new KVM-based private cloud. The plan was one test VM on Friday, the rest over the weekend.
The test VM, a copy of the seat-inventory API, converted cleanly and then sat at a dracut emergency shell with “Warning: /dev/mapper/rl-root does not exist”. The first theory was a broken conversion, so they ran it again, which took 40 minutes and gave the same shell. Someone suggested switching the new disk to the IDE type to “make it look like VMware”, and that’s when she stopped the room and asked for the console.
From the GRUB menu she booted the rescue entry, and it came up fine. lsinitrd on the normal image showed vmw_pvscsi and no virtio driver at all, because Rocky builds host-only images and this one had been built on VMware. The rescue image is generic, so it had the virtio drivers and found the disk at once.
The fix took ten minutes per VM, done before the move: a file /etc/dracut.conf.d/90-virtio.conf with add_drivers+=" virtio_blk virtio_scsi ", then dracut -f --regenerate-all, then lsinitrd | grep virtio as proof in the change ticket. All 18 VMs moved that weekend, and the migration checklist now has a line nobody skips: rebuild and check the initramfs on the source, and never delete the rescue entry to save space in /boot.
Common mistakes and troubleshooting
| Mistake | What happens | What to do instead |
|---|---|---|
| Editing grub.cfg by hand | Your change disappears at the next kernel update or update-grub run | Use grubby on Rocky, or /etc/default/grub.d plus update-grub on Ubuntu |
| Editing /etc/default/grub on Ubuntu and rebooting without update-grub | /proc/cmdline shows no change, because grub.cfg was never rebuilt | Run sudo update-grub and grep the linux line in /boot/grub/grub.cfg before you reboot |
| Putting a must-have argument in GRUB_CMDLINE_LINUX_DEFAULT | Recovery mode boots without it | Use GRUB_CMDLINE_LINUX for anything recovery needs too |
Running grub2-mkconfig -o /boot/efi/EFI/rocky/grub.cfg | Refused on Rocky 10; on older systems it overwrites the EFI stub | Use /boot/grub2/grub.cfg on BIOS and UEFI alike |
Writing add_drivers+="virtio_blk" without spaces | Values from two files join into one name that matches no driver | Always add_drivers+=" name " with spaces inside the quotes |
Expecting update-initramfs -u to rebuild the running kernel | It rebuilds only the newest installed kernel | Use -k all or -k $(uname -r) |
| Deleting the rescue kernel to free space in /boot | You lose the one generic image that boots on new hardware | Keep it; clean old kernels with the package manager instead (Lesson 49) |
Interview questions
Click a question to open it. Each one is written the way an interviewer would ask it. Under it you will see why they ask it and one way you could answer, in your own words.
Fresher level
1. How do you add a kernel parameter permanently on a RHEL or Rocky server?
Why they ask this: They want to hear grubby rather than hand-editing grub.cfg, and a check after the reboot.
How you might answer: I use grubby --update-kernel=ALL --args="name=value", which updates every boot entry in /boot/loader/entries and also GRUB_CMDLINE_LINUX in /etc/default/grub, so new kernels and a later grub2-mkconfig keep it.
After the reboot I check cat /proc/cmdline. To remove it I use --remove-args, and for a one-off test I change only one entry by giving its kernel path.
2. What is the initramfs, and why does a server need one?
Why they ask this: A fundamentals question. They’re checking that you understand the gap between the kernel loading and the root file system being available.
How you might answer: The initramfs is a small compressed file system that GRUB loads into memory with the kernel. It holds the drivers and tools needed to reach the real root disk, such as the SATA or virtio driver, LVM and sometimes multipath or decryption.
The kernel can’t mount a root that sits on LVM over a driver that lives as a module on that same disk, so the initramfs finds and mounts it on /sysroot and then hands over. On RHEL and Ubuntu 26.04 it’s built by dracut and you can look inside with lsinitrd.
2–5 years experience
3. You changed /etc/default/grub on an Ubuntu server and rebooted, but /proc/cmdline doesn’t show your change. What happened?
Why they ask this: They want to know whether you understand that grub.cfg is generated, and how you verify before rebooting.
How you might answer: Most likely nobody ran update-grub, so grub.cfg still had the old line. I’d run it, read its output for errors, and grep the linux line in /boot/grub/grub.cfg before rebooting again.
If update-grub did run, I’d check whether the value went into GRUB_CMDLINE_LINUX_DEFAULT and the server booted a recovery entry, or whether a file in /etc/default/grub.d, which is read after the main file, set the variable again.
4. We’re moving 50 Rocky VMs from VMware to KVM. What would you check in the initramfs before the move?
Why they ask this: This is a real migration problem. They’re testing whether you know about host-only images and plan for the target hardware.
How you might answer: Rocky builds host-only images, so the current image probably has the VMware disk driver and no virtio driver. I’d check with lsinitrd | grep virtio on a few VMs first.
Then I’d add /etc/dracut.conf.d/90-virtio.conf with add_drivers+=" virtio_blk virtio_scsi ", run dracut -f --regenerate-all so the fallback kernel has it too, and record the lsinitrd output in the ticket. I’d also keep the rescue entry, because its generic image is the backup plan.
Senior level
5. A server stops at the dracut emergency shell after a change last night. Walk me through getting it back without reinstalling.
Why they ask this: Seniors should stay calm, read the evidence the shell gives them and choose the least risky way back.
How you might answer: First I read what the shell says and look at /run/initramfs/rdsosreport.txt, then I compare cat /proc/cmdline with ls /dev/mapper and blkid. That tells me whether the disk is missing (a driver or storage problem) or the kernel line points at the wrong name.
If it’s the kernel line, I reset, edit the entry at the GRUB menu, boot once and fix it with grubby. If a driver is missing, I boot the previous kernel or the rescue entry, rebuild the image with dracut and the right drivers, and check it with lsinitrd before the next reboot. Then I find out what changed last night, because that’s the real cause.
6. How would you roll out a kernel argument change to 300 servers safely?
Why they ask this: They want a staged, reversible plan with checks, not a loop over SSH.
How you might answer: I’d use Ansible with grubby on RHEL and a grub.d drop-in plus update-grub on Ubuntu, and run it on a few canary servers first. Those reboot in a planned slot with console access ready, and the playbook checks /proc/cmdline afterwards and fails loudly if the argument is missing.
Then I’d go in batches small enough that the service stays up. The rollback is the same playbook with --remove-args or the drop-in deleted, and I’d tell the monitoring team which reboots to expect.
Quick quiz
Q1. On Rocky Linux 10, which command adds a kernel argument to every boot entry?
grub2-mkconfig --argsgrubby --update-kernel=ALL --args="..."dracut --add-argsupdate-grub
Show answer
B — grubby edits the BLS entry files in /boot/loader/entries and keeps GRUB_CMDLINE_LINUX in step.
Q2. You edited /etc/default/grub on Ubuntu. What must you run before rebooting?
grubby --update-kernel=ALLdracut -fsudo update-grub- Nothing, GRUB reads the file at boot
Show answer
C — update-grub rebuilds /boot/grub/grub.cfg, which is the only file GRUB reads.
Q3. Why might a Rocky VM moved from VMware to KVM drop to a dracut shell?
- The kernel is too new
- Its host-only initramfs has no virtio disk driver
- GRUB can’t read XFS
- SELinux blocks the disk
Show answer
B — Rocky builds host-only images with only the drivers in use when they were built.
Q4. Which line in /etc/dracut.conf.d/90-virtio.conf is written correctly?
add_drivers="virtio_blk"add_drivers+="virtio_blk"add_drivers+=" virtio_blk "drivers=virtio_blk
Show answer
C — Use += so other files still count, and keep spaces inside the quotes so names don’t run together.
Q5. A bad kernel argument stops the boot. What is the quickest way in?
- Reinstall the OS
- Press e at the GRUB menu, fix the linux line and press Ctrl+X
- Run update-grub from the dracut shell
- Delete the initramfs
Show answer
B — Editing the entry at the menu changes the command line for one boot; then you fix it for good with grubby.
Quick recap
- GRUB and dracut work before any service starts: GRUB passes the kernel command line, dracut builds the initramfs that finds the root disk.
- On Rocky, change arguments with
grubby --update-kernel=ALL --args/--remove-args; it edits the BLS entries and GRUB_CMDLINE_LINUX. - On Ubuntu, use a drop-in in /etc/default/grub.d, run
update-gruband grep grub.cfg before you reboot. - Check every change in
/proc/cmdlineafter the reboot. - Rocky’s images are host-only, so add drivers with
add_drivers+=" ... "in /etc/dracut.conf.d, rebuild withdracut -f --regenerate-alland prove it with lsinitrd. - A broken kernel line is fixed for one boot with e at the GRUB menu, then for good with grubby; the rescue entry is your spare key.
Next lesson
Lesson 51: Kernel tuning: sysctl, tuned, ulimit and systemd LimitNOFILE → You’ll change kernel parameters at runtime and make them stick with /etc/sysctl.d, see what tuned does on Rocky, and fix the open-files limit for a systemd service the way Kafka needs it.
Further reading: GRUB and dracut
Want to go deeper into GRUB and dracut? Read Red Hat’s guide to managing the kernel on RHEL 10 (kernel command-line parameters and grubby), the GNU GRUB manual’s simple configuration page for /etc/default/grub, the dracut project wiki, and the kernel’s list of command-line parameters. You can also see the full plan for this course on the Linux Administration course page.