The Linux troubleshooting method: what changed, layer by layer
Linux troubleshooting goes faster when you follow the same routine every time: write the symptom down word for word, find out what changed, check the layers from the hardware up with read-only commands, change one thing, and prove the fix. Engineers who skip the routine and jump to restarts often make the problem go away for an hour and lose the evidence that would have told them why it happened.
This lesson opens the troubleshooting phase of the course. The fourteen lessons after it each take one classic failure (full disks, failed services, lost routes, emergency mode, SSH, memory, DNS and more), and every one of them uses the method you practise here, on a real fault in the log forwarding you built between rocky01 and ubu01 in Lesson 29.
Where this lesson fits
Lesson 65 of 82 in Linux Administration · Phase 4: Troubleshooting step by step · Before this: Lesson 64, Ansible for Linux admins · Next: Lesson 66, du and df don’t match
| Time | You need | By the end you can |
|---|---|---|
| About 75 minutes | rocky01 and ubu01 with the rsyslog forwarding from Lesson 29, saat-health.sh from Lesson 39, and a snapshot of rocky01 | Work through any server problem with a fixed method: describe it, list what changed, walk the layers, fix one thing, prove it and write a short incident note |
What it is, in one line
Linux troubleshooting is a routine: describe the symptom precisely, ask what changed, check each layer from the bottom up without changing anything, then change one thing and prove it worked.
In Simple Words
How a good family doctor works
Think of the family doctor in your colony. Before writing anything on the prescription pad, she asks: since when, what did you eat, did you start any new medicine, did you travel? Then she checks your temperature, pulse and blood pressure, listens to your chest and only then decides.
If she gives you a medicine, she gives one, and asks you to come back in three days so she knows whether it worked.
A bad doctor gives five medicines on the first visit. You feel better, but nobody knows which one helped, and next time the same illness comes back the guessing starts again.
On a server, “what did you eat” is “what changed”, the temperature and pulse are your read-only checks, and the one medicine is the one change you make.
Engineer's View
Where Linux keeps the record of what changed
Most incidents follow a change: a package update, a config edit, a reboot, a new disk, someone’s login. Linux records most of these, but in different places on each family, and a few of the old tools have moved.
| Question | Rocky Linux 10 / RHEL 10 | Ubuntu 26.04 |
|---|---|---|
| Which packages changed, and when? | rpm -qa --last | head, dnf history list and dnf history info N | /var/log/dpkg.log, /var/log/apt/history.log, and the new apt history-list / apt history-info N (APT 3.x) |
| When did it boot or shut down? | last -x reboot shutdown (reads /var/log/wtmp), journalctl --list-boots | journalctl --list-boots. last is not installed by default; it comes from the wtmpdb package in universe |
| Who logged in? | last -n 10, journalctl -u sshd | journalctl -u ssh (the unit is ssh, not sshd) |
| Which config files changed? | find /etc -xdev -type f -mmin -60, rpm -V rsyslog for files a package owns | find /etc -xdev -type f -mmin -60, dpkg --verify rsyslog |
| What did the kernel say? | dmesg -T (this boot only), journalctl -k -b -1 (previous boot) | Same commands |
| What happened before the reboot? | journalctl -b -1 -p warning. Needs a persistent journal, which you set up on rocky01 in Lesson 29 | journalctl -b -1 -p warning. Ubuntu keeps the journal on disk by default |
If the team keeps /etc in git (with etckeeper or a plain repository), git -C /etc log --stat answers the config question better than any timestamp, because it also shows who changed what. Ansible and other tools from Lesson 64 leave their own trail too: the playbook run history in your CI system is a change record.
A few details catch people. dmesg shows only the kernel ring buffer since the last boot, and its -T times can drift on a VM that was paused or a laptop that slept, so trust the journal’s timestamps over dmesg -T. On Ubuntu 26.04 /var/log/wtmp is gone; wtmpdb keeps login records in an SQLite file instead, and only once the package is installed.
How it works, step by step
The method has five moves, always in the same order. The diagram below shows them; the rest of this section explains each one.
1Write the symptom down precisely
“The server is down” is not a symptom. “Since 14:05, test messages from rocky01 no longer appear in /var/log/remote/rocky01/ on ubu01; local logging on rocky01 still works” is one. Note who reported it, what fails, since when, for whom, and copy any error text word for word.
Half of all troubleshooting time goes into chasing a problem that was described wrongly.
2Ask what changed, and when
Put the start time of the symptom next to the change records: package history, reboots, logins, files modified in /etc, deployments, and anything the network or storage team did. A change five minutes before the symptom isn’t proof, but it’s the first place to look. If nothing changed on the server, the change is often outside it: a certificate expired, a disk filled up, a partner system moved.
3Walk the layers from the bottom up
Check the layers in order: the VM or hardware, the kernel, storage, network, the service, and finally the application and its config. A full disk or a dead network makes every layer above it look broken, so starting at the bottom saves you from debugging an app that is fine. Each check is read-only and quick, and each one either clears a layer or points into it.
4Make one theory and test it without changing anything
Once a layer looks wrong, say your theory in one sentence (“rsyslog is sending to the wrong IP”) and look for a read-only command that would prove or kill it. The journal line, the config line, a nc -zv to the port: these cost nothing and change nothing.
5Change one thing, then prove it
Make the smallest change that fixes the cause, keep a backup of what you touched, and re-run the exact test that showed the symptom. If the symptom stays, undo your change before trying the next idea, so you never stack guesses on top of each other. The picture below shows the layers in the order you check them, with the commands for each and what the lab found.
6Write the incident note while it’s fresh
Five to ten lines are enough: symptom, impact, timeline, what changed, the checks you ran, the cause, the fix, and one follow-up that stops it happening again. Tomorrow you won’t remember whether you restarted rsyslog before or after the edit, and the next on-call engineer will search old tickets for the same error text.
After a crash or an unplanned reboot the evidence is in the previous boot, not this one. journalctl --list-boots lists the boots the journal knows, journalctl -b -1 -p warning shows warnings and worse from the boot before this one, and journalctl -k -b -1 shows that boot’s kernel messages. A log that stops in the middle of normal messages, with no shutdown lines, points to a power cut, a hard reset or a kernel panic (Lesson 79 covers kdump for that case).
Two habits make all of this easier. Run the read-only checks from Lesson 1 (now saved as saat-health.sh from Lesson 39) before you touch anything, and know what “normal” looks like on your servers, because a check result means little if you have nothing to compare it with.
Hands-on lab: find and fix a broken log forward with the method
Before you run any command, double-check it
Read every command twice before you press Enter. Check the target: the server name, the user, the disk or device name, the path and the sizes. These commands are for learning and were tested in our lab, not on your system. At work, run the read-only commands first, follow your change process, make sure a backup exists, and ask a second person to check anything that changes or deletes data. You run them at your own risk.
You’ll break the log forwarding from Lesson 29 with a small script, then find and fix the fault using only the method: symptom, what changed, layers, one change, proof, note. Work on rocky01 as student, with a second SSH session to ubu01 for checking the far end. Take a VirtualBox snapshot of rocky01 first (VBoxManage snapshot rocky01 take before-l65 on the laptop), and create a work folder with mkdir -p ~/lab65.
Outputs from the VMs are examples; PIDs, times and boot IDs on yours will differ.
1Prove the log path works before anything breaks
Run this on rocky01:
logger -t lab65 "baseline $(date +%T)"Then on ubu01:
sudo tail -n 1 /var/log/remote/rocky01/lab65.logYou should see something like this:
2026-10-27T14:02:11+05:30 rocky01 lab65[2611]: baseline 14:02:11The ubu01 template from Lesson 29 writes one file per program, so the tag lab65 gets its own file, lab65.log. This is your known-good result. Without it you could spend an hour fixing something that never worked in the first place.
✅ Checkpoint: ubu01 shows your baseline line in /var/log/remote/rocky01/lab65.log.
2Record what normal looks like
Run this on rocky01:
sudo /usr/local/bin/saat-health.sh -q
systemctl is-active rsyslog
sudo ss -tnp | grep ':514'You should see something like this:
RESULT: OK (0 critical, 0 warning, 0 unknown)
active
ESTAB 0 0 10.10.10.11:41862 10.10.10.21:514 users:(("rsyslogd",pid=1108,fd=12))The health script from Lesson 39 says OK, rsyslog is running, and there is one established TCP connection from rocky01 to ubu01 port 514, owned by rsyslogd. Save these three lines in your notes. When something breaks later, “the connection to 514 is gone” is only useful if you know it was there before.
✅ Checkpoint: You have the RESULT line, “active”, and one ESTAB line to 10.10.10.21:514.
3Let a colleague make a change
Create the script and run it on rocky01 (paste this whole block):
cat > ~/lab65/mystery65.sh <<'EOF'
#!/bin/bash
# mystery65.sh: one quick change, the kind a colleague makes on a busy afternoon
set -euo pipefail
cp -a /etc/rsyslog.d/05-forward-ubu01.conf /root/05-forward-ubu01.conf.l65
sed -i 's/target="10.10.10.21"/target="10.10.10.12"/' /etc/rsyslog.d/05-forward-ubu01.conf
systemctl restart rsyslog
echo "mystery65: done"
EOF
bash -n ~/lab65/mystery65.sh && sudo bash ~/lab65/mystery65.shYou should see this:
mystery65: doneYou can read the script, and in a lab that’s fine, but play along: at work you would get a ticket and nothing else. Now reproduce the symptom the way the ticket would describe it. On rocky01 run logger -t lab65 "test 1 $(date +%T)", and on ubu01 run the same tail as in step 1.
You should see something like this on ubu01:
2026-10-27T14:02:11+05:30 rocky01 lab65[2611]: baseline 14:02:11Only the baseline line. Write the symptom down now, in one line: “From about 14:05, messages from rocky01 do not arrive on ubu01; the baseline at 14:02 did.”
✅ Checkpoint: test 1 is missing on ubu01 and you have written the symptom down with a time.
4Run the first five minutes of read-only checks
Run this on rocky01:
uptime
sudo /usr/local/bin/saat-health.sh -q
systemctl --failed
sudo journalctl -p err --since "30 min ago" --no-pager | tail -n 2You should see something like this:
14:07:40 up 5:06, 1 user, load average: 0.00, 0.02, 0.00
RESULT: OK (0 critical, 0 warning, 0 unknown)
UNIT LOAD ACTIVE SUB DESCRIPTION
0 loaded units listed.
Oct 27 14:05:31 rocky01 rsyslogd[2690]: cannot connect to 10.10.10.12:514: No route to host [v8.2412.0 try https://www.rsyslog.com/e/2027 ]
Oct 27 14:06:02 rocky01 rsyslogd[2690]: cannot connect to 10.10.10.12:514: No route to host [v8.2412.0 try https://www.rsyslog.com/e/2027 ]The box is up, not loaded and has no failed units, and the health script is still green. That’s worth remembering: a monitoring check that says OK only means the things it checks are fine. The error-level journal has the real clue, rsyslog failing to connect to 10.10.10.12, which is not ubu01’s address.
Your health script may already say WARNING if rsyslog has logged more than ten errors in the last hour, which is the same clue arriving another way.
✅ Checkpoint: You found a “cannot connect” line in the journal and noticed the IP in it.
5Ask what changed
Run this on rocky01:
last -x reboot shutdown | head -n 3
journalctl --list-boots --no-pager | tail -n 2
sudo dnf history list | head -n 4
sudo find /etc -xdev -type f -mmin -30
ls -l /etc/rsyslog.d/05-forward-ubu01.confYou should see something like this:
reboot system boot 6.12.0-211.16.1. Tue Oct 27 09:01 still running
shutdown system down 6.12.0-211.16.1. Mon Oct 26 19:48 - 09:01 (13:12)
reboot system boot 6.12.0-211.16.1. Mon Oct 26 13:21 - 19:48 (06:27)
-1 b2d9f04a6c3e4d7a8f1b0c2e9d5a7f36 Mon 2026-10-26 13:21:12 IST Mon 2026-10-26 19:48:31 IST
0 e41a7c9d0b2f4c86a3d5e7f9b1c0d2a4 Tue 2026-10-27 09:01:12 IST Tue 2026-10-27 14:08:02 IST
ID | Command line | Date and time | Action(s) | Altered
-------------------------------------------------------------------------------
27 | install -y rsync | 2026-10-26 18:12 | Install | 1
26 | install -y python3-pip | 2026-10-26 17:40 | Install | 4
/etc/rsyslog.d/05-forward-ubu01.conf
-rw-r--r--. 1 root root 291 Oct 27 14:05 /etc/rsyslog.d/05-forward-ubu01.confRead it as four answers. No reboot since 09:01 this morning (last cuts the kernel version at 16 characters, which is why it ends in a dot). The journal agrees on the boots.
The last package transaction was yesterday evening, so no update is involved. One file in /etc was modified in the last 30 minutes, the forwarding rule, at 14:05, the same minute the symptom started. -mmin -30 means “modified less than 30 minutes ago” and -xdev keeps find on the root file system.
✅ Checkpoint: You can name the one thing that changed and its time: 05-forward-ubu01.conf at 14:05.
6Walk the layers from the bottom up
A changed file is a strong lead but not yet proof, so check the layers under it first. The VM is up and the kernel is quiet (step 4). Now the network, the far end’s port and the service:
Run this on rocky01:
ping -c 2 ubu01
nc -zv 10.10.10.21 514
systemctl is-active rsyslog
sudo ss -tnp | grep ':514' || echo "no connection on 514"You should see something like this:
PING ubu01.saatlab.test (10.10.10.21) 56(84) bytes of data.
64 bytes from ubu01.saatlab.test (10.10.10.21): icmp_seq=1 ttl=64 time=0.412 ms
64 bytes from ubu01.saatlab.test (10.10.10.21): icmp_seq=2 ttl=64 time=0.388 ms
--- ubu01.saatlab.test ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 0.388/0.400/0.412/0.012 ms
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to 10.10.10.21:514.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.
active
no connection on 514The network to ubu01 works, ubu01 accepts TCP on 514 from rocky01 (so its ufw rule from Lesson 34 and its rsyslog listener are fine), rsyslog on rocky01 runs, but it holds no connection to 514. Everything below the service is clean. Now read the service’s own log and the line it was configured with:
sudo journalctl -u rsyslog --since "30 min ago" --no-pager | tail -n 3
grep -n target /etc/rsyslog.d/05-forward-ubu01.confYou should see something like this:
Oct 27 14:05:31 rocky01 systemd[1]: Started rsyslog.service - System Logging Service.
Oct 27 14:05:31 rocky01 rsyslogd[2690]: cannot connect to 10.10.10.12:514: No route to host [v8.2412.0 try https://www.rsyslog.com/e/2027 ]
Oct 27 14:05:31 rocky01 rsyslogd[2690]: action 'action-1-builtin:omfwd' suspended (module 'builtin:omfwd'), retry 0. There should be messages before this one giving the reason for suspension. [v8.2412.0 try https://www.rsyslog.com/e/2007 ]
2:*.* action(type="omfwd" target="10.10.10.12" port="514" protocol="tcp"Theory in one sentence: rsyslog forwards to 10.10.10.12 because the target in the config was mistyped. Every check agrees, and you haven’t changed anything yet. “No route to host” is what Linux reports when nothing on the local network answers for that IP; the action number in the second line can differ on your VM.
✅ Checkpoint: Network, port and service are all clean, and the config shows target=”10.10.10.12″.
7Change one thing and prove it
Fix the target, check the syntax, restart, and test:
sudo cp -a /etc/rsyslog.d/05-forward-ubu01.conf ~/lab65/05-forward-ubu01.conf.broken
sudo sed -i 's/target="10.10.10.12"/target="10.10.10.21"/' /etc/rsyslog.d/05-forward-ubu01.conf
sudo rsyslogd -N1 && sudo systemctl restart rsyslog
grep -n target /etc/rsyslog.d/05-forward-ubu01.conf
logger -t lab65 "test 2 $(date +%T)"You should see something like this:
rsyslogd: version 8.2412.0, config validation run (level 1), master config /etc/rsyslog.conf
rsyslogd: End of config validation run. Bye.
2:*.* action(type="omfwd" target="10.10.10.21" port="514" protocol="tcp"Then on ubu01:
sudo tail -n 3 /var/log/remote/rocky01/lab65.logYou should see something like this:
2026-10-27T14:02:11+05:30 rocky01 lab65[2611]: baseline 14:02:11
2026-10-27T14:06:18+05:30 rocky01 lab65[2702]: test 1 14:06:18
2026-10-27T14:19:44+05:30 rocky01 lab65[2771]: test 2 14:19:44test 2 arrived, and test 1 probably arrived too, a few seconds after the restart. The disk-assisted queue you gave the forward action in Lesson 29 held the messages while the target was wrong and delivered them once it was right. You kept a copy of the broken file before editing, so the change can be undone or shown to whoever made it.
✅ Checkpoint: test 2 is on ubu01, and sudo ss -tnp | grep :514 shows an ESTAB line again.
8Look back at the previous boot
Today’s fault happened in the current boot. Practise the other case now, because after a crash it’s the first thing you need.
Run this on rocky01:
journalctl --list-boots --no-pager | tail -n 2
sudo journalctl -b -1 -n 5 --no-pagerYou should see something like this:
IDX BOOT ID FIRST ENTRY LAST ENTRY
-1 b2d9f04a6c3e4d7a8f1b0c2e9d5a7f36 Mon 2026-10-26 13:21:12 IST Mon 2026-10-26 19:48:31 IST
0 e41a7c9d0b2f4c86a3d5e7f9b1c0d2a4 Tue 2026-10-27 09:01:12 IST Tue 2026-10-27 14:21:30 IST
Oct 26 19:48:22 rocky01 systemd-logind[811]: The system will power off now!
Oct 26 19:48:22 rocky01 systemd-logind[811]: System is powering down.
Oct 26 19:48:29 rocky01 systemd-shutdown[1]: Syncing filesystems and block devices.
Oct 26 19:48:30 rocky01 systemd-shutdown[1]: Sending SIGTERM to remaining processes...
Oct 26 19:48:31 rocky01 systemd-journald[612]: Journal stopped-b -1 is the boot before this one, and -n 5 shows its last five lines. These end with a clean shutdown (“Journal stopped”), so the last reboot was planned. If the previous boot ended in the middle of ordinary messages with no shutdown lines, you would look at sudo journalctl -k -b -1 | tail -n 30 for kernel errors and at the VirtualBox or hypervisor logs for a reset.
✅ Checkpoint: You can say whether rocky01’s last shutdown was clean, and what time it happened.
9Write the incident note
Paste this whole block on rocky01, with your own times:
cat > ~/lab65/incident-2026-10-27.txt <<'EOF'
Incident: rocky01 logs not reaching ubu01 (27 Oct 2026)
Symptom: lab65 test messages from rocky01 missing in /var/log/remote/rocky01/ on ubu01 from 14:05
Impact: no central copy of rocky01 logs for 14 minutes (queued on rocky01, delivered after the fix)
What changed: 14:05 /etc/rsyslog.d/05-forward-ubu01.conf edited and rsyslog restarted; no reboot, no packages
Checks: ping and nc to 10.10.10.21:514 OK; rsyslog active; journal: cannot connect to 10.10.10.12:514
Cause: forward target typo, 10.10.10.21 changed to 10.10.10.12
Fix: 14:19 target corrected, rsyslogd -N1 clean, rsyslog restarted, test 2 arrived on ubu01
Follow-up: keep /etc in git, second pair of eyes on rsyslog.d edits, alert when a host sends nothing for 15 min
EOF
wc -l ~/lab65/incident-2026-10-27.txtYou should see this:
8 /home/student/lab65/incident-2026-10-27.txtEight lines, and anyone reading it next month knows what broke, how it was found and what to change so it doesn’t happen again. Notice the note has times on everything and quotes the error text, which is what someone will search for. Keep /root/05-forward-ubu01.conf.l65 for now; the Break-It section uses it.
✅ Checkpoint: ~/lab65/incident-2026-10-27.txt exists with symptom, change, checks, cause, fix and follow-up.
Break It, Then Fix It
Take a fresh snapshot of rocky01 first. The first exercise is the real trap this lesson is about; the other two are smaller ones that waste time in almost every incident.
1Two changes at once hide the cause
Run this on rocky01 (paste this whole block):
cat > ~/lab65/break65.sh <<'SCRIPT'
#!/bin/bash
# break65.sh: two changes in one go, like a busy Friday evening
set -euo pipefail
sed -i 's/target="10.10.10.21"/target="10.10.10.12"/' /etc/rsyslog.d/05-forward-ubu01.conf
cat > /etc/rsyslog.d/04-quiet.conf <<'EOF'
# Silence the chatty lab65 app (ticket 4471)
if $programname == 'lab65' then stop
EOF
systemctl restart rsyslog
echo "break65: done"
SCRIPT
sudo bash ~/lab65/break65.shNow act like an engineer who found the first cause quickly. The journal shows “cannot connect to 10.10.10.12” again, so fix the target the same way as in lab step 7 and test:
sudo sed -i 's/target="10.10.10.12"/target="10.10.10.21"/' /etc/rsyslog.d/05-forward-ubu01.conf
sudo rsyslogd -N1 && sudo systemctl restart rsyslog
logger -t lab65 "test 3 $(date +%T)"
sudo grep -c 'test 3' /var/log/messages
journalctl -t lab65 -n 1 --no-pagerYou’ll get this:
rsyslogd: version 8.2412.0, config validation run (level 1), master config /etc/rsyslog.conf
rsyslogd: End of config validation run. Bye.
0
Oct 27 14:41:09 rocky01 lab65[2903]: test 3 14:41:09test 3 is not on ubu01 either (check with the tail from step 1). This is the moment people say “so the IP wasn’t the problem”, put the old value back and start guessing elsewhere. The IP was a real fault; it wasn’t the only one.
The new clue is that test 3 is in the journal but not in /var/log/messages, so rsyslog received it and threw it away before any rule wrote it out.
Fix: go back to the change list instead of guessing, and read every file it shows.
sudo find /etc -xdev -type f -mmin -30
cat /etc/rsyslog.d/04-quiet.conf
sudo rm /etc/rsyslog.d/04-quiet.conf
sudo rsyslogd -N1 && sudo systemctl restart rsyslog
logger -t lab65 "test 4 $(date +%T)"You should see something like this:
/etc/rsyslog.d/05-forward-ubu01.conf
/etc/rsyslog.d/04-quiet.conf
# Silence the chatty lab65 app (ticket 4471)
if $programname == 'lab65' then stop
rsyslogd: version 8.2412.0, config validation run (level 1), master config /etc/rsyslog.conf
rsyslogd: End of config validation run. Bye.rsyslog reads /etc/rsyslog.d in name order, so 04-quiet.conf runs before 05-forward-ubu01.conf and its stop discards lab65 messages before the forward rule or /var/log/messages ever see them. With two changes in the same slot, fixing one and seeing no improvement tells you nothing about that one. Undo or check every change in the window, then test.
✅ Checkpoint: test 4 appears on ubu01, and the IP fix from before is still in place.
2Search the wrong boot for the evidence
Leave a marker in the log, then reboot rocky01:
logger -p user.err -t lab65boot "disk controller timeout (lab65 test)"
sudo systemctl rebootLog back in and look for it the way most people do:
journalctl -b -t lab65boot --no-pager
sudo dmesg -T | grep -c lab65You’ll get this:
-- No entries --
0-b on its own means the current boot, which started after your marker, and dmesg holds only kernel messages from this boot, so neither can see it. At 3 a.m. after a surprise reboot, this is how people conclude “nothing in the logs”.
Fix: ask for the previous boot by number.
journalctl -b -1 -t lab65boot --no-pagerYou should see something like this:
Oct 27 14:47:52 rocky01 lab65boot[2988]: disk controller timeout (lab65 test)If the journal is kept only in memory (no /var/log/journal), the same command prints Specifying boot ID or boot offset has no effect, no persistent journal was found. and the evidence is gone for good. That’s why Lesson 29 made rocky01’s journal persistent, and why you check it on every server you look after.
✅ Checkpoint: journalctl -b -1 -t lab65boot shows your marker line.
3Reach for last on Ubuntu 26.04
Run this on ubu01:
last -x | head -n 3You’ll get something like this:
Command 'last' not found, but can be installed with:
sudo apt install wtmpdbThe classic /var/log/wtmp file has a 32-bit time field that runs out in 2038, so Debian and Ubuntu moved login records to wtmpdb, and the package isn’t part of a standard Ubuntu Server install. Installing it now won’t show anything from before the install.
Fix: use the journal, which has the same facts.
journalctl --list-boots --no-pager | tail -n 3
journalctl -u ssh --since today --no-pager | grep Accepted | tail -n 3The first command gives boot and shutdown times; the second gives SSH logins with user, source IP and key type. If your team wants last back, put wtmpdb in your build so it records from day one.
✅ Checkpoint: You can list ubu01’s boots and today’s SSH logins without last. Clean up on rocky01 with sudo rm /root/05-forward-ubu01.conf.l65 and keep ~/lab65 for your notes.
This happened at work
The patch weekend that got the blame
My team lead tells this one from her time at a microfinance institution in Toronto that gives small loans through field officers. On a Monday in February, at about 8:15 a.m. Eastern, the officers’ tablets stopped syncing new loan applications, and the help desk had forty calls before nine.
The only change anyone remembered was the Saturday patch run: about 140 package updates on the four RHEL application servers.
The on-call engineer did what most of us would. He saw the update in dnf history, ran dnf history undo on one server, restarted the app, and nothing changed. Two more hours went into comparing package lists before my lead asked for the symptom word for word.
The sync service’s log said the TLS handshake with the credit-check partner failed, and only for that partner.
She ran find /etc -xdev -mmin -4320 on one box and found /etc/crypto-policies/config changed on Saturday afternoon, three hours after the patching. A security engineer had set the system-wide crypto policy to FUTURE as part of a hardening task, on the same maintenance slot, in a separate ticket. FUTURE refuses RSA keys shorter than 3072 bits, and the partner still used a 2048-bit certificate.
Setting the policy back to DEFAULT with update-crypto-policies --set DEFAULT and restarting the service brought the sync back by 11:40.
The rollback of the packages had to be undone too, which cost another evening. Afterwards the team changed two things: every server change goes into one list per maintenance slot, whoever makes it, and /etc on every server is tracked with etckeeper so “what changed” is one git log away. The partner was asked for a stronger certificate, and the FUTURE policy went in a month later after a test against every partner endpoint.
Common mistakes and troubleshooting
| Mistake | What happens | What to do instead |
|---|---|---|
| Restarting the service before looking | The symptom goes away for a while and the evidence (state, connections, logs in memory) is lost | Run the read-only checks and save their output first, then restart if you must |
| Starting with the layer you suspect | You debug a healthy app while the disk or network underneath is the real problem | Walk the layers from the VM up; each check takes seconds |
| Undoing one change and declaring it innocent | A second change made in the same slot keeps the fault alive | List every change in the window and test them one at a time |
| Changing several things at once | It works, but nobody knows which change fixed it, and the next incident starts from zero | One change, one test, undo it if it didn’t help |
| Reading only the current boot after a crash | journalctl -b and dmesg show nothing from before the reboot | Use journalctl -b -1 and journalctl -k -b -1, with a persistent journal |
| Trusting a green health check | The check covers what it was written for, not your symptom | Test the symptom itself, the way the user sees it |
| Writing the incident note next week | Times, error texts and order of steps are forgotten or wrong | Write five to ten lines the same day, with times and exact error text |
Interview questions
Click a question to open it. Each one is written the way an interviewer would ask it. Under it you will see why they ask it and one way you could answer, in your own words.
Fresher level
1. A user says “the server is down”. What do you do in the first five minutes?
Why they ask this: They want to see a calm, ordered approach instead of a jump to restarts. Clear questions to the user count as much as commands.
How you might answer: I first ask what fails, for whom and since when, and get any error text. Then I log in and run read-only checks: uptime, df -h, free -h, systemctl --failed and journalctl -p err --since "30 min ago".
In parallel I ask what changed: dnf history or the dpkg log, reboots with journalctl --list-boots, and files in /etc modified recently. Only when I know which layer is broken do I change anything, and I write down what I ran.
2. How do you find out whether a server rebooted last night, and why?
Why they ask this: A basic check of the tools, plus whether you know the difference between the current and the previous boot.
How you might answer: uptime tells me when it last started, last -x reboot shutdown on RHEL or journalctl --list-boots anywhere shows the history. To see why, I read the end of the previous boot with journalctl -b -1 -n 50.
If the log ends with normal shutdown messages, someone or something rebooted it on purpose. If it stops in the middle of ordinary lines, I look for a crash or a hard reset in journalctl -k -b -1 and the hypervisor logs.
2–5 years experience
3. Walk me through how you work out “what changed” on a RHEL server and on an Ubuntu server.
Why they ask this: They want to know you can find change evidence on both families and know where each one keeps it.
How you might answer: On RHEL I use rpm -qa --last | head and dnf history list for packages, last -x and journalctl --list-boots for reboots, and find /etc -xdev -type f -mmin -120 for config. On Ubuntu the packages are in /var/log/dpkg.log and /var/log/apt/history.log, and 26.04 also has apt history-list.
On 26.04 last isn’t installed by default, so I use the journal for logins and boots. If /etc is in etckeeper or git, git log --stat beats all of these, and I also check the deployment and Ansible history, because many changes never touch the package manager.
4. You fixed one problem but the symptom is still there. What do you do next?
Why they ask this: This checks whether you understand stacked faults and avoid undoing a correct fix out of frustration.
How you might answer: I don’t assume my fix was wrong. I re-check that the fix is in place and that the evidence for the first fault is gone, for example the “cannot connect” errors stopped. Then I look for a second cause, starting again from the change list for the same window.
I compare where the data stops now: in a log forwarding case, if the message is in the journal but not in /var/log/messages, the drop happens inside rsyslog. Each layer check narrows it down until the second change shows up.
Senior level
5. How would you make troubleshooting faster across a team of twenty engineers and five hundred servers?
Why they ask this: Seniors are expected to think about systems and habits, not only commands. They want concrete practices.
How you might answer: I’d make “what changed” easy to answer: etckeeper or config management for /etc, one change record per maintenance slot, and packages and deployments logged centrally. Persistent journals and central log forwarding on every server, so evidence survives reboots.
Then a shared runbook with the same first-five-minutes checks, a known-good baseline per service, and incident notes in one searchable place with exact error texts. After each major incident we pick one follow-up that prevents a repeat, and someone owns it.
6. During a major outage, people in the call want you to try several fixes at once. How do you handle it?
Why they ask this: They want to see judgement under pressure and how you balance speed with knowing what fixed the problem.
How you might answer: I agree on one theory at a time and say out loud what we’re changing and how we’ll know if it worked, with a time limit. If restoring service matters more than finding the cause, I’ll do a known-safe restore such as failing over or rolling back, but I save the evidence first.
I keep a running timeline in the incident channel with every change and its time. Afterwards that timeline shows which step brought service back, and the cause is found in the review, not guessed during the outage.
Quick quiz
Q1. Which statement is the best way to write down a symptom?
- The logging server is broken
- Since 14:05, messages from rocky01 do not appear on ubu01; local logging works
- rsyslog needs a restart
- Something changed this afternoon
Show answer
B — A good symptom says what fails, where, since when and what still works, without guessing the cause.
Q2. Your server rebooted unexpectedly overnight. Which command shows warnings from before the reboot?
dmesg -Tjournalctl -b -p warningjournalctl -b -1 -p warninglast -n 5
Show answer
C — -b -1 is the previous boot. dmesg and plain -b only cover the current boot.
Q3. On Ubuntu 26.04, last says command not found. Why?
- Ubuntu removed login records completely
- last moved to the wtmpdb package, which is not installed by default
- You must run it with sudo
- It only works on RHEL
Show answer
B — wtmp was replaced by wtmpdb (Y2038), and the package is in universe. The journal still has boots and SSH logins.
Q4. You undo one recent change and the problem stays. What is the right conclusion?
- That change was not a fault, so put it back
- There may be a second change; check every change in the same window
- Reboot the server
- Reinstall the package
Show answer
B — Two faults can stack. Fixing one shows no improvement until the other is fixed too.
Q5. Which command lists config files under /etc modified in the last 30 minutes, staying on the root file system?
find /etc -mtime -30find /etc -xdev -type f -mmin -30ls -lt /etc | headrpm -qa --last
Show answer
B — -mmin -30 is minutes, while -mtime -30 would be 30 days. ls -lt only shows the top folder.
Quick recap
- Linux troubleshooting follows one order: exact symptom, what changed, layers from the bottom up, one change, proof, note.
- What changed:
rpm -qa --last,dnf history, /var/log/dpkg.log andapt history-list,journalctl --list-boots,last -xon RHEL,find /etc -xdev -type f -mmin -N. - Layers: VM and hardware, kernel, storage, network, service, app and config. Each check is read-only.
- After a crash, read the previous boot:
journalctl -b -1andjournalctl -k -b -1, which needs a persistent journal. - Two changes in one slot can hide each other; undo or test every change in the window.
- Write an eight-line incident note the same day, with times and exact error text.
Next lesson
Lesson 66: du and df don’t match: deleted files still held open → You’ll make a file system look full with no big files on it, find the deleted file a running process still holds with lsof +L1 and /proc, and get the space back with a restart, a HUP or a careful truncate.
Further reading: Linux troubleshooting
Want to go deeper into Linux troubleshooting? Read the man7.org pages for journalctl, last and dmesg, the dnf command reference (history), the Ubuntu 26.04 release notes (APT history commands) and Red Hat’s RHEL 10 documentation. You can also see the full plan for this course on the Linux Administration course page.