Bash scripting for admins: a real health-check script
-
Phase 0: Before you start
-
Phase 1: The origins and OS architecture
-
Phase 2: Day-to-day administration and the command builder
-
Phase 3: Enterprise infrastructure and storage
-
Phase 4: Troubleshooting step by step
-
Phase 5: Final project and career
Bash scripting for admins is the step from a script that works while you watch it to one you can leave running at 3 a.m.: it stops at the first unexpected error, cleans up after itself, writes to the system log, takes options like a real command and ends with an exit code a monitoring system can act on. Those habits are what separate a quick hack from a tool your team trusts.
In Lesson 38 you wrote small scripts with variables, tests and loops. Here you add set -Eeuo pipefail (and the traps hidden inside it), functions with local variables, logging with logger, trap, getopts and here-documents. Then you put all of it into /usr/local/bin/saat-health.sh, the nine checks from Lesson 1 with limits and exit codes.
Later lessons run this script, so keep its name and path as shown.
Where this lesson fits
Lesson 39 of 82 in Linux Administration · Phase 2: Day-to-day administration and the command builder · Before this: Lesson 38, bash scripting basics · Next: Lesson 40, time sync with chrony
| Time | You need | By the end you can |
|---|---|---|
| About 80 minutes | rocky01 with sudo, the ~/scripts folder from Lesson 38, and ubu01 if you want to run shellcheck there | Write admin scripts that fail safely, log and clean up, accept options, and install a health check whose exit code tells a monitor OK, WARNING, CRITICAL or UNKNOWN |
What it is, in one line
An admin script is a bash script built to run without you: it stops on surprises, cleans up, logs what happened and reports one honest exit code.
In Simple Words
The night watchman’s round
A housing society hires a night watchman to walk the building at 2 a.m. He checks the gate, the water tank, the lift and the generator, and writes each one in a register with the time. If something is badly wrong he stops the round and wakes the secretary, and before he leaves the tank room he always switches off the light, whatever he found.
In the morning the secretary doesn’t read every line. She looks at the last line in the register: all fine, one small problem, a big problem, or “couldn’t check the generator room, it was locked”. The exit code of an admin script is that last line, and the system log is the register.
Engineer's View
The safety settings and where they stop helping
| Setting | What it does | Where it doesn’t help |
|---|---|---|
set -e (errexit) | Stops the script when a command fails | Not inside if/while conditions, not in a && b except the last command, not after !. local x=$(cmd) hides cmd’s failure |
set -u (nounset) | Unset variables are an error instead of an empty string | Use ${1:-} or ${VAR:-default} where empty is allowed |
set -o pipefail | A pipeline fails if any part fails, not only the last | grep exits 1 for “no match”, which pipefail and -e treat as a failure |
set -E (errtrace) | Lets functions and subshells use the ERR trap | Only matters when you set an ERR trap |
trap '...' EXIT | Runs cleanup however the script ends: normal end, exit, error or Ctrl+C | Can’t run after SIGKILL (kill -9) or a power cut |
trap '...' ERR | Runs when a command fails, before -e stops the script | Same blind spots as set -e |
Admin scripts also send messages to the system log with logger, from util-linux. logger -t TAG -p user.warning "text" writes one line with a tag and a priority. On both distributions it lands in the journal (journalctl -t TAG); rsyslog also copies it to /var/log/messages on Rocky and /var/log/syslog on Ubuntu (Lesson 29).
Check scripts follow the Nagios plugin convention for exit codes: 0 OK, 1 WARNING, 2 CRITICAL, 3 UNKNOWN. Nagios, Icinga and Checkmk read the code directly, and a Zabbix agent (Lesson 61) can run the script with a user parameter. Note the trap in that table: a plain set -e failure exits with 1, which a monitor reads as WARNING, so saat-health.sh turns unexpected failures into 3.
How it works, step by step
Most admin scripts have the same shape from top to bottom, and saat-health.sh follows it. The diagram after these steps shows one run.
1Header and safety line
The shebang, a comment that says what the script does and what its exit codes mean, then set -Eeuo pipefail. Put it on line one of the code, before anything can go wrong.
2Defaults, functions and traps, then the main part
Limits sit near the top as plain variables so anyone can see and change them. Every check is a small function, the traps are set before any work starts, and the main part at the bottom reads the options, checks they make sense, calls the functions in order and prints the result.
3Functions keep their own variables with local
Variables in bash are global unless you say local inside a function, so a helper that uses n can quietly change the main script’s n. A function hands back text by printing it (you catch it with $(...)) and success or failure with return 0 or return 1. Declare first and assign on the next line, local out then out=$(cmd), because local out=$(cmd) takes local’s exit code (always 0) and hides cmd’s failure from set -e.
4getopts reads options, a here-document prints help
getopts ':w:c:qh' opt walks through the options one by one: a letter followed by a colon takes a value, which arrives in $OPTARG. The leading colon lets you print your own messages for an unknown option (?) or a missing value (:). The help text is a here-document, cat <<EOF ...
EOF, which keeps a block of text readable in the script and still expands variables such as the defaults.
5Every check reports a level, and the worst one wins
Each check calls result LEVEL NAME MESSAGE, which adds a line to the report and passes the level to raise. raise only ever makes the status worse: CRITICAL beats WARNING, and WARNING beats UNKNOWN, because a known problem is more useful to the on-call engineer than a missing reading. The final status becomes the exit code.
Monitoring tools treat the four codes very differently, as the picture below shows. Get them right and the same script feeds a dashboard, a cron job and a pager without changes.
6Logs and traps make it safe to run unattended
Human-readable output goes to standard output, problems go to standard error with >&2, and one summary line goes to the system log with logger, so there’s a record even when nobody reads cron mail. trap 'rm -f "$TMP"' EXIT removes the temporary report on every way out, and the ERR trap names the line number when something unexpected fails.
Test the script three ways before you trust it: bash -n for syntax, shellcheck for quoting and common mistakes, and bash -x script.sh to watch every line as it runs when the result surprises you. None of them replaces running it on a test VM with the limits pushed on purpose, which is what the lab does.
Hands-on lab: build and install saat-health.sh
Before you run any command, double-check it
Read every command twice before you press Enter. Check the target: the server name, the user, the disk or device name, the path and the sizes. These commands are for learning and were tested in our lab, not on your system. At work, run the read-only commands first, follow your change process, make sure a backup exists, and ask a second person to check anything that changes or deletes data. You run them at your own risk.
You’ll work as student in ~/lab39 on rocky01, and only two steps change the system: step 5 may install ShellCheck, and step 7 installs saat-health.sh in /usr/local/bin. Steps 1 to 4 show each building block on its own with a tiny script; step 5 puts them together. Start with mkdir -p ~/lab39 && cd ~/lab39.
1See what set -euo pipefail changes
Run these one at a time at the prompt:
bash -c 'false; echo "still running"'
bash -e -c 'false; echo "still running"'; echo "exit $?"
bash -c 'echo "folder: $NO_SUCH_VAR/tmp"'
bash -u -c 'echo "folder: $NO_SUCH_VAR/tmp"'
bash -c 'grep ERROR /var/log/nothing.log | wc -l; echo "pipeline status: $?"'
bash -o pipefail -c 'grep ERROR /var/log/nothing.log | wc -l; echo "pipeline status: $?"'You should see this:
still running
exit 1
folder: /tmp
bash: line 1: NO_SUCH_VAR: unbound variable
grep: /var/log/nothing.log: No such file or directory
0
pipeline status: 0
grep: /var/log/nothing.log: No such file or directory
0
pipeline status: 2Without -e, bash shrugs off the failed command and carries on. Without -u, a typo in a variable name turns $NO_SUCH_VAR/tmp into /tmp, which is how rm -rf "$DIR/" accidents happen. Without pipefail, the pipeline reports the status of wc (0) and hides grep’s error (2).
In a script you switch all three on with one line, set -euo pipefail.
✅ Checkpoint: You saw the script stop with -e, the unbound variable error with -u, and status 2 with pipefail.
2Write functions with local variables and return codes
Paste this whole block:
cat > functions.sh <<'EOF'
#!/bin/bash
# functions.sh: functions, local variables and return codes
set -euo pipefail
# pct_used MOUNT: print how full a mount point is, as a plain number
pct_used() {
local mnt=$1
df --output=pcent "$mnt" | tail -n 1 | tr -d ' %'
}
# is_up HOST: return 0 if HOST answers one ping, 1 if it doesn't
is_up() {
local host=$1
if ping -c 1 -W 1 "$host" >/dev/null 2>&1; then
return 0
fi
return 1
}
where="outside"
show_scope() {
local where="inside"
echo "in the function, where=$where"
}
show_scope
echo "after the function, where=$where"
echo "/ is $(pct_used /)% used"
for h in ubu01 kafka01; do
if is_up "$h"; then
echo "$h is up"
else
echo "$h is down (is_up returned $?)"
fi
done
EOF
chmod +x functions.sh
./functions.shYou should see something like this:
in the function, where=inside
after the function, where=outside
/ is 12% used
ubu01 is up
kafka01 is down (is_up returned 1)The local where inside show_scope is a separate variable, so the outer where keeps its value. pct_used hands its answer back by printing it, and is_up hands back only a status with return. A failing test inside if doesn’t trigger set -e, which is why you test commands with if instead of running them bare.
✅ Checkpoint: The outer variable still says outside, and kafka01 is reported down with return code 1.
3Log with timestamps and logger
Paste this whole block:
cat > log-demo.sh <<'EOF'
#!/bin/bash
# log-demo.sh: one log function, two places the message ends up
set -euo pipefail
# log LEVEL MESSAGE: timestamped line on stderr and a copy in the system log
log() {
local level=$1
shift
printf '%s [%s] %s\n' "$(date '+%F %T')" "$level" "$*" >&2
logger -t lab39 -p "user.${level,,}" -- "$*"
}
log INFO "starting checks on $(hostname -s)"
log WARNING "/boot is 31% full"
log ERR "port 3306 is not listening"
EOF
chmod +x log-demo.sh
./log-demo.shYou should see something like this:
2026-10-27 11:12:40 [INFO] starting checks on rocky01
2026-10-27 11:12:40 [WARNING] /boot is 31% full
2026-10-27 11:12:40 [ERR] port 3306 is not listeningNow read the same messages from the journal:
journalctl -t lab39 --no-pager
journalctl -t lab39 -p warning --no-pagerYou should see something like this:
Oct 27 11:12:40 rocky01 lab39[2611]: starting checks on rocky01
Oct 27 11:12:40 rocky01 lab39[2612]: /boot is 31% full
Oct 27 11:12:40 rocky01 lab39[2613]: port 3306 is not listening
Oct 27 11:12:40 rocky01 lab39[2612]: /boot is 31% full
Oct 27 11:12:40 rocky01 lab39[2613]: port 3306 is not listeningThe date '+%F %T' timestamp makes the screen output sortable, and >&2 keeps log lines out of anything a caller captures from standard output. ${level,,} lowercases the level so it becomes a syslog priority (info, warning, err). The second journalctl shows only warning and worse, which is how you filter a noisy script. The number in brackets is the PID of each logger process.
✅ Checkpoint: journalctl -t lab39 shows three lines, and -p warning shows two.
4Clean up with trap, even after Ctrl+C
Paste this whole block:
cat > trap-demo.sh <<'EOF'
#!/bin/bash
# trap-demo.sh: a temporary folder that is removed however the script ends
set -euo pipefail
work=$(mktemp -d)
trap 'echo "cleanup: removing $work"; rm -rf "$work"' EXIT
echo "working in $work"
date '+%F %T' > "$work/report.txt"
ls "$work"
sleep "${1:-0}"
echo "finished normally"
EOF
chmod +x trap-demo.sh
./trap-demo.shYou should see something like this:
working in /tmp/tmp.V7LOFdYQ17
report.txt
finished normally
cleanup: removing /tmp/tmp.V7LOFdYQ17Now give it 30 seconds of work and press Ctrl+C while it waits, then check the exit code and the folder (use the name your run printed):
./trap-demo.sh 30
echo "exit code: $?"
ls -d /tmp/tmp.bToLrVp67WYou should see something like this:
working in /tmp/tmp.bToLrVp67W
report.txt
^Ccleanup: removing /tmp/tmp.bToLrVp67W
exit code: 130
ls: cannot access '/tmp/tmp.bToLrVp67W': No such file or directorymktemp -d creates a folder with a random name that nobody else can guess, and the EXIT trap removes it on a normal end and after Ctrl+C alike. Exit code 130 is 128 plus 2, the number of SIGINT, which is how bash reports “stopped by a signal”. The trap is written in single quotes so $work is expanded when the trap runs, not when it’s set.
✅ Checkpoint: Both runs print the cleanup line, and the folder is gone after Ctrl+C.
5Write saat-health.sh and check it
Paste this whole block (it’s long, so copy all of it):
cat > saat-health.sh <<'EOF'
#!/bin/bash
# saat-health.sh: the nine read-only checks from Lesson 1, with limits and exit codes.
# SaatVedha Linux Administration, Lesson 39. Installed as /usr/local/bin/saat-health.sh
# Exit codes (the Nagios plugin convention, also read by Icinga and Checkmk):
# 0 OK, 1 WARNING, 2 CRITICAL, 3 UNKNOWN (the script could not check something)
set -Eeuo pipefail
readonly PROG=saat-health
readonly VERSION=1.0
# Default limits. Options below change the common ones.
DISK_WARN=80 # % used on any local file system
DISK_CRIT=90
MEM_WARN=15 # % of RAM still available
MEM_CRIT=5
LOAD_WARN=1.0 # 5-minute load average per CPU
LOAD_CRIT=2.0
TOP_WARN=50 # % of RAM held by the single biggest process
ERR_WARN=10 # error messages in the journal in the last hour
PORTS=22 # TCP ports that must be listening, comma separated
OUTFILE=""
QUIET=0
USE_SYSLOG=1
STATUS=0
N_CRIT=0
N_WARN=0
N_UNK=0
TMP=""
usage() {
cat <<EOF_USAGE
Usage: $PROG [-w PCT] [-c PCT] [-p PORTS] [-e N] [-o FILE] [-q] [-n] [-h]
Runs nine read-only health checks and exits with
0 OK, 1 WARNING, 2 CRITICAL or 3 UNKNOWN.
-w PCT disk use warning limit in % (default $DISK_WARN)
-c PCT disk use critical limit in % (default $DISK_CRIT)
-p PORTS TCP ports that must be listening, comma separated (default $PORTS)
-e N error messages in the last hour before a warning (default $ERR_WARN)
-o FILE also save the report to FILE
-q quiet: print only the RESULT line
-n do not send the result to syslog
-h show this help
EOF_USAGE
}
# fail MESSAGE: explain on stderr and exit UNKNOWN
fail() {
printf '%s %s: %s\n' "$(date '+%F %T')" "$PROG" "$*" >&2
exit 3
}
# Remove the temporary report however the script ends, and turn any
# unexpected failure into UNKNOWN (exit 3) instead of a silent exit 1.
trap 'rm -f "$TMP"' EXIT
trap 'fail "unexpected error on line $LINENO (exit code $?)"' ERR
have() {
command -v "$1" >/dev/null 2>&1
}
# raise CODE: keep the worst result. CRITICAL beats WARNING, WARNING beats UNKNOWN.
raise() {
case "$1" in
2) STATUS=2 ;;
1) if [ "$STATUS" -ne 2 ]; then STATUS=1; fi ;;
3) if [ "$STATUS" -eq 0 ]; then STATUS=3; fi ;;
esac
}
# result LEVEL CHECK MESSAGE: add one line to the report and remember the level
result() {
local level=$1 check=$2 msg=$3
case "$level" in
WARNING) N_WARN=$((N_WARN + 1)); raise 1 ;;
CRITICAL) N_CRIT=$((N_CRIT + 1)); raise 2 ;;
UNKNOWN) N_UNK=$((N_UNK + 1)); raise 3 ;;
esac
printf '%-8s %-8s %s\n' "$level" "$check" "$msg" >> "$TMP"
}
# above VALUE LIMIT: true if VALUE >= LIMIT, works with decimals like 0.35
above() {
awk -v v="$1" -v l="$2" 'BEGIN { exit !(v >= l) }'
}
meminfo() {
awk -v k="$1:" '$1 == k { print $2 }' /proc/meminfo
}
check_identity() {
local os
os=$(awk -F= '$1 == "PRETTY_NAME" { gsub(/"/, "", $2); print $2 }' /etc/os-release)
result INFO host "$(uname -n), kernel $(uname -r)"
result INFO os "${os:-unknown}"
}
check_load() {
local load5 cpus per msg
read -r _ load5 _ < /proc/loadavg
cpus=$(nproc)
per=$(awk -v l="$load5" -v c="$cpus" 'BEGIN { printf "%.2f", l / c }')
msg="5-min load $load5 on $cpus CPUs ($per per CPU), $(uptime -p)"
if above "$per" "$LOAD_CRIT"; then
result CRITICAL load "$msg"
elif above "$per" "$LOAD_WARN"; then
result WARNING load "$msg"
else
result OK load "$msg"
fi
}
check_disk() {
local out mnt pct n=0 worst=0 worst_mnt="" over=""
if ! out=$(df -l --output=target,pcent -x tmpfs -x devtmpfs -x efivarfs -x squashfs -x overlay 2>/dev/null); then
result UNKNOWN disk "df found no local file systems to check"
return 0
fi
while read -r mnt pct; do
pct=${pct%\%}
n=$((n + 1))
if [ "$pct" -gt "$worst" ]; then
worst=$pct
worst_mnt=$mnt
fi
if [ "$pct" -ge "$DISK_WARN" ]; then
over="$over $mnt ${pct}%"
fi
done < <(printf '%s\n' "$out" | tail -n +2)
if [ "$worst" -ge "$DISK_CRIT" ]; then
result CRITICAL disk "at or over ${DISK_CRIT}%:$over"
elif [ "$worst" -ge "$DISK_WARN" ]; then
result WARNING disk "at or over ${DISK_WARN}%:$over"
else
result OK disk "$n file systems below ${DISK_WARN}% (fullest: $worst_mnt at ${worst}%)"
fi
}
check_memory() {
local total avail stotal sfree pct spct=0 msg
total=$(meminfo MemTotal)
avail=$(meminfo MemAvailable)
stotal=$(meminfo SwapTotal)
sfree=$(meminfo SwapFree)
pct=$((avail * 100 / total))
if [ "$stotal" -gt 0 ]; then
spct=$(( (stotal - sfree) * 100 / stotal ))
fi
msg="${pct}% of RAM available ($((avail / 1024)) of $((total / 1024)) MiB), swap ${spct}% used"
if [ "$pct" -lt "$MEM_CRIT" ]; then
result CRITICAL memory "$msg"
elif [ "$pct" -lt "$MEM_WARN" ]; then
result WARNING memory "$msg"
else
result OK memory "$msg"
fi
}
check_top() {
local rss comm total pct msg
read -r rss comm < <(ps -eo rss=,comm= --sort=-rss | awk 'NR == 1')
total=$(meminfo MemTotal)
pct=$((rss * 100 / total))
msg="biggest process: $comm, $((rss / 1024)) MiB (${pct}% of RAM)"
if [ "$pct" -ge "$TOP_WARN" ]; then
result WARNING top "$msg"
else
result OK top "$msg"
fi
}
check_ports() {
local port ports missing="" found=""
if ! have ss; then
result UNKNOWN ports "ss not found (package iproute2)"
return 0
fi
IFS=, read -r -a ports <<< "$PORTS"
for port in "${ports[@]}"; do
if [ -n "$(ss -Htln "sport = :$port")" ]; then
found="$found $port"
else
missing="$missing $port"
fi
done
if [ -n "$missing" ]; then
result CRITICAL ports "not listening:$missing"
else
result OK ports "listening:$found"
fi
}
check_services() {
local out n names
if ! have systemctl || ! out=$(systemctl --failed --no-legend --plain 2>/dev/null); then
result UNKNOWN services "systemctl did not answer (is systemd running?)"
return 0
fi
if [ -z "$out" ]; then
result OK services "no failed units"
return 0
fi
n=$(printf '%s\n' "$out" | wc -l)
names=$(printf '%s\n' "$out" | awk '{ print $1 }' | paste -sd ' ' -)
result WARNING services "$n failed: $names"
}
check_errors() {
local n
if ! have journalctl || ! n=$(journalctl -q -p err --since "1 hour ago" --no-pager -o cat 2>/dev/null | wc -l); then
result UNKNOWN errors "could not read the journal"
return 0
fi
if [ "$n" -ge "$ERR_WARN" ]; then
result WARNING errors "$n error messages in the last hour (limit $ERR_WARN)"
else
result OK errors "$n error messages in the last hour"
fi
}
# ---- main ----
while getopts ':w:c:p:e:o:qnh' opt; do
case "$opt" in
w) DISK_WARN=$OPTARG ;;
c) DISK_CRIT=$OPTARG ;;
p) PORTS=$OPTARG ;;
e) ERR_WARN=$OPTARG ;;
o) OUTFILE=$OPTARG ;;
q) QUIET=1 ;;
n) USE_SYSLOG=0 ;;
h) usage; exit 0 ;;
:) echo "$PROG: option -$OPTARG needs a value" >&2; usage >&2; exit 3 ;;
*) echo "$PROG: unknown option -$OPTARG" >&2; usage >&2; exit 3 ;;
esac
done
shift $((OPTIND - 1))
for n in "$DISK_WARN" "$DISK_CRIT" "$ERR_WARN"; do
[[ $n =~ ^[0-9]+$ ]] || fail "limits must be whole numbers, got '$n'"
done
[ "$DISK_WARN" -lt "$DISK_CRIT" ] || fail "-w ($DISK_WARN) must be lower than -c ($DISK_CRIT)"
[[ $PORTS =~ ^[0-9]+(,[0-9]+)*$ ]] || fail "ports must look like 22 or 22,80,443, got '$PORTS'"
TMP=$(mktemp "/tmp/$PROG.XXXXXX")
HEADER="$PROG $VERSION on $(uname -n) at $(date '+%F %T %Z')"
check_identity
check_load
check_disk
check_memory
check_top
check_ports
check_services
check_errors
case "$STATUS" in
0) word=OK; prio=notice ;;
1) word=WARNING; prio=warning ;;
2) word=CRITICAL; prio=crit ;;
*) word=UNKNOWN; prio=err ;;
esac
summary="RESULT: $word ($N_CRIT critical, $N_WARN warning, $N_UNK unknown)"
report() {
echo "$HEADER"
cat "$TMP"
echo "$summary"
}
if [ "$QUIET" -eq 1 ]; then
echo "$summary"
else
report
fi
if [ -n "$OUTFILE" ]; then
report > "$OUTFILE"
fi
if [ "$USE_SYSLOG" -eq 1 ] && have logger; then
logger -t "$PROG" -p "user.$prio" -- "$summary" || echo "$PROG: could not write to syslog" >&2
fi
exit "$STATUS"
EOF
chmod +x saat-health.sh
bash -n saat-health.sh && echo "syntax ok"You should see this:
syntax okRead it top to bottom once. The limits are plain variables near the top, each check is a function that ends in result, the traps sit right after fail, and the main part at the bottom parses options, validates them, runs the eight check functions (identity covers the first two of Lesson 1’s nine) and prints the report. Notice what it avoids: no ((n++)), no grep -c that could fail on zero matches, and no | head that could end a pipeline early under pipefail.
Now run shellcheck. On rocky01 it comes from EPEL:
sudo dnf install -y epel-release
sudo dnf install -y ShellCheck
shellcheck saat-health.sh && echo "shellcheck: no warnings"You should see this at the end:
shellcheck: no warningsIf your company doesn’t allow EPEL, copy the file to ubu01 with scp (Lesson 33), install shellcheck there with sudo apt install -y shellcheck and check it on ubu01. The script was checked with shellcheck 0.9 and 0.11 and both are silent.
✅ Checkpoint: bash -n says syntax ok and shellcheck prints no warnings.
6Try the options and the help text
Run this:
./saat-health.sh -h
./saat-health.sh -x; echo "exit $?"
./saat-health.sh -w 95 -c 90; echo "exit $?"You should see something like this:
Usage: saat-health [-w PCT] [-c PCT] [-p PORTS] [-e N] [-o FILE] [-q] [-n] [-h]
Runs nine read-only health checks and exits with
0 OK, 1 WARNING, 2 CRITICAL or 3 UNKNOWN.
-w PCT disk use warning limit in % (default 80)
-c PCT disk use critical limit in % (default 90)
-p PORTS TCP ports that must be listening, comma separated (default 22)
-e N error messages in the last hour before a warning (default 10)
-o FILE also save the report to FILE
-q quiet: print only the RESULT line
-n do not send the result to syslog
-h show this help
saat-health: unknown option -x
Usage: saat-health [-w PCT] [-c PCT] [-p PORTS] [-e N] [-o FILE] [-q] [-n] [-h]
Runs nine read-only health checks and exits with
0 OK, 1 WARNING, 2 CRITICAL or 3 UNKNOWN.
-w PCT disk use warning limit in % (default 80)
-c PCT disk use critical limit in % (default 90)
-p PORTS TCP ports that must be listening, comma separated (default 22)
-e N error messages in the last hour before a warning (default 10)
-o FILE also save the report to FILE
-q quiet: print only the RESULT line
-n do not send the result to syslog
-h show this help
exit 3
2026-10-27 11:18:22 saat-health: -w (95) must be lower than -c (90)
exit 3-h prints the here-document and exits 0. An unknown option or a missing value prints a short reason plus the help on standard error and exits 3, because a monitor that calls the script wrongly hasn’t learned anything about the server. The same goes for limits that make no sense, caught by fail before any check runs.
✅ Checkpoint: The bad option and the bad limits both end with exit 3.

7Install it and run it
Install it with root as owner, then run it with sudo:
sudo install -o root -g root -m 0755 saat-health.sh /usr/local/bin/saat-health.sh
ls -lZ /usr/local/bin/saat-health.sh
sudo /usr/local/bin/saat-health.sh; echo "exit $?"You should see something like this:
-rwxr-xr-x. 1 root root unconfined_u:object_r:bin_t:s0 8528 Oct 27 11:19 /usr/local/bin/saat-health.sh
saat-health 1.0 on rocky01 at 2026-10-27 11:20:03 IST
INFO host rocky01, kernel 6.12.0-211.16.1.el10_2.0.1.x86_64
INFO os Rocky Linux 10.2 (Red Quartz)
OK load 5-min load 0.05 on 2 CPUs (0.03 per CPU), up 2 hours, 21 minutes
OK disk 3 file systems below 80% (fullest: /boot at 31%)
OK memory 73% of RAM available (1285 of 1742 MiB), swap 0% used
OK top biggest process: firewalld, 45 MiB (2% of RAM)
OK ports listening: 22
OK services no failed units
OK errors 0 error messages in the last hour
RESULT: OK (0 critical, 0 warning, 0 unknown)
exit 0install copies the file and sets owner and mode in one go. Because it creates a new file, the file gets the SELinux label of its new folder, bin_t. mv from your home folder would keep the home label, and a systemd service could then fail to run it (Lessons 41 and 68 cover that). The full path after sudo works whatever sudo’s secure_path contains.
Read the report like Lesson 1’s checks: load is low, every file system is under 80 percent, three quarters of the RAM is available, firewalld is the biggest process, sshd is listening, no unit has failed and the journal is quiet. Run as root so the error count sees every message.
✅ Checkpoint: The report ends with RESULT: OK and the exit code is 0.
8Push it past its limits
Lower the disk limit, then ask for a port nothing listens on:
sudo /usr/local/bin/saat-health.sh -w 10 -c 95; echo "exit $?"
sudo /usr/local/bin/saat-health.sh -q -p 22,3306; echo "exit $?"You should see something like this:
saat-health 1.0 on rocky01 at 2026-10-27 11:21:40 IST
INFO host rocky01, kernel 6.12.0-211.16.1.el10_2.0.1.x86_64
INFO os Rocky Linux 10.2 (Red Quartz)
OK load 5-min load 0.05 on 2 CPUs (0.03 per CPU), up 2 hours, 23 minutes
WARNING disk at or over 10%: / 12% /boot 31%
OK memory 73% of RAM available (1285 of 1742 MiB), swap 0% used
OK top biggest process: firewalld, 45 MiB (2% of RAM)
OK ports listening: 22
OK services no failed units
OK errors 0 error messages in the last hour
RESULT: WARNING (0 critical, 1 warning, 0 unknown)
exit 1
RESULT: CRITICAL (1 critical, 0 warning, 0 unknown)
exit 2Now make a unit fail on purpose, check, and clean up:
sudo systemd-run --unit=saat-test /usr/bin/false
sudo /usr/local/bin/saat-health.sh | grep -E 'services|RESULT'
sudo systemctl reset-failed saat-test.serviceYou should see something like this:
Running as unit: saat-test.service; invocation ID: 6c1f3a9e2d7b4e8f9a0c1b2d3e4f5a6b
WARNING services 1 failed: saat-test.service
RESULT: WARNING (0 critical, 1 warning, 0 unknown)systemd-run starts a throwaway service that runs false, which exits 1, so the unit lands in the failed state and the services check sees it. reset-failed clears it again. MariaDB arrives in Lesson 56, and then -p 22,3306 becomes a real check rather than a test.
✅ Checkpoint: You saw exit codes 1 and 2, and the failed saat-test unit turned the result into WARNING.
9Read the result the way a monitor does
Run this:
sudo /usr/local/bin/saat-health.sh -q -n; rc=$?
case $rc in
0) echo "green" ;;
1) echo "yellow" ;;
2) echo "red" ;;
*) echo "grey: the check itself failed" ;;
esac
journalctl -t saat-health -n 4 --no-pagerYou should see something like this:
RESULT: OK (0 critical, 0 warning, 0 unknown)
green
Oct 27 11:20:03 rocky01 saat-health[2690]: RESULT: OK (0 critical, 0 warning, 0 unknown)
Oct 27 11:21:40 rocky01 saat-health[2742]: RESULT: WARNING (0 critical, 1 warning, 0 unknown)
Oct 27 11:21:52 rocky01 saat-health[2779]: RESULT: CRITICAL (1 critical, 0 warning, 0 unknown)
Oct 27 11:22:31 rocky01 saat-health[2815]: RESULT: WARNING (0 critical, 1 warning, 0 unknown)That case statement is what every monitoring system does with a check: it only looks at the exit code, and keeps the text for the human. -n kept this run out of the log, so the journal shows the four earlier runs, each with its own priority. To keep a copy for a ticket, add -o ~/health-$(date +%F).txt; to run it every hour, wrap it in a service and timer the way Lesson 27 did for disk-report.sh.
✅ Checkpoint: You can turn any exit code from saat-health.sh into green, yellow, red or grey, and you found its runs in the journal.
Break It, Then Fix It
Each of these scripts passes bash -n and shellcheck, and each one still fails in a way that hurts at 3 a.m. They’re small, so work in ~/lab39.
1Count with ((count++)) under set -e
Paste this whole block:
cat > count-missing.sh <<'EOF'
#!/bin/bash
# count-missing.sh: count files that should exist but don't
set -euo pipefail
count=0
for f in /etc/hostname /etc/no-such-file /etc/os-release; do
if [ ! -e "$f" ]; then
((count++))
fi
done
echo "$count file(s) missing"
EOF
chmod +x count-missing.sh
./count-missing.sh; echo "exit $?"You’ll get this:
exit 1No message at all, and the script stopped halfway. ((count++)) returns the old value, 0, and in bash arithmetic a result of 0 counts as false, so the command “fails” and set -e ends the script the first time a file is missing. It works from the second missing file on, which is why this bug hides in testing.
Fix: add with $(( )), which is an assignment and always succeeds.
sed -i 's/((count++))/count=$((count + 1))/' count-missing.sh
./count-missing.sh; echo "exit $?"1 file(s) missing
exit 0✅ Checkpoint: The script prints 1 file(s) missing and exits 0.
2grep -c on a healthy log
Paste this whole block. The log has no errors, which is good news:
cat > app.log <<'EOF'
2026-10-27T11:40:01+05:30 rocky01 saatshop[2214]: INFO order 1051 placed
2026-10-27T11:40:09+05:30 rocky01 saatshop[2214]: INFO order 1052 placed
EOF
cat > count-errors.sh <<'EOF'
#!/bin/bash
# count-errors.sh: count ERROR lines in a log file
set -euo pipefail
log=${1:-app.log}
errors=$(grep -c ERROR "$log")
echo "$errors error line(s) in $log"
EOF
chmod +x count-errors.sh
./count-errors.sh; echo "exit $?"You’ll get this:
exit 1grep -c prints 0 and exits 1 when nothing matches, and set -e treats that 1 as a failure inside the assignment. So the script dies on the good days and works on the bad ones. Adding || true alone would also hide a real error, such as a missing log file (grep exit 2), so check the file first.
Fix: test that the file is readable, then accept “no match”. Paste this whole block:
cat > count-errors.sh <<'EOF'
#!/bin/bash
# count-errors.sh: count ERROR lines in a log file
set -euo pipefail
log=${1:-app.log}
if [ ! -r "$log" ]; then
echo "Cannot read $log" >&2
exit 3
fi
# grep -c prints 0 and exits 1 when nothing matches; that is not an error here
errors=$(grep -c ERROR "$log" || true)
echo "$errors error line(s) in $log"
EOF
./count-errors.sh; echo "exit $?"
./count-errors.sh /var/log/nothing.log; echo "exit $?"You should see this:
0 error line(s) in app.log
exit 0
Cannot read /var/log/nothing.log
exit 3✅ Checkpoint: A clean log gives 0 with exit 0, and a missing log gives a clear message with exit 3.
3Call a set -u script without its argument
Paste this whole block:
cat > reach.sh <<'EOF'
#!/bin/bash
# reach.sh: say whether a host answers ping
set -euo pipefail
host=$1
if ping -c 1 -W 1 "$host" >/dev/null 2>&1; then
echo "$host is up"
else
echo "$host is down"
fi
EOF
chmod +x reach.sh
./reach.sh; echo "exit $?"You’ll get this error:
./reach.sh: line 4: $1: unbound variable
exit 1set -u did its job: it refused to run ping with an empty host. The message is accurate but unfriendly, and a monitor reading exit 1 would call it a WARNING about a host nobody named.
Fix: read optional arguments with a default, and print usage with its own exit code. Paste this whole block:
cat > reach.sh <<'EOF'
#!/bin/bash
# reach.sh: say whether a host answers ping
set -euo pipefail
host=${1:-}
if [ -z "$host" ]; then
echo "Usage: $0 HOST" >&2
exit 2
fi
if ping -c 1 -W 1 "$host" >/dev/null 2>&1; then
echo "$host is up"
else
echo "$host is down"
fi
EOF
./reach.sh; echo "exit $?"
./reach.sh ubu01; echo "exit $?"You should see this:
Usage: ./reach.sh HOST
exit 2
ubu01 is up
exit 0✅ Checkpoint: Without an argument the script prints usage and exits 2; with ubu01 it says up.
This happened at work
The price sync that only failed on good nights
A colleague of mine was on call for an agritech marketplace in London that sold produce from farms to restaurants. Every night at 1 a.m. a bash script imported the suppliers’ new prices, counted how many rows were rejected, and published the price list for the morning orders. It had a proper set -euo pipefail at the top, and everyone was proud of that.
One Tuesday the restaurants’ buyers started calling at 7 a.m.: tomatoes were still at last week’s price, and two suppliers had already complained. The first theory was a supplier file in the wrong format, so the team spent an hour opening CSVs that turned out to be fine. Then my colleague ran the script by hand with bash -x and watched it stop, with no message and exit code 1, on the line rejected=$(grep -c REJECTED import.log).
For the first time in months, no row had been rejected. grep printed 0, exited 1, and set -e ended the script before it published anything. The cron output went to a local mailbox nobody read, and the job monitor only checked that the script had started.
The fix that morning was one || true after a check that the log existed. The lasting changes were bigger: every script got an ERR trap that sends the line number and exit code to the journal with logger, the monitor now waits for a “published” log line by 1:30 a.m. and pages if it doesn’t arrive, and the team added a test run with a clean import log to their checklist.
Common mistakes and troubleshooting
| Mistake | What happens | What to do instead |
|---|---|---|
| Trusting set -e to catch every failure | Failures inside if, && lists and local x=$(cmd) go unnoticed | Check important commands with if, and declare local on its own line |
Using ((n++)) or grep -c under set -e | The script dies the first time the count is 0 | Use n=$((n + 1)), and || true only after checking the file exists |
| Letting unexpected errors exit with 1 | A monitor shows WARNING for a broken check | Trap ERR and exit 3 (UNKNOWN), as saat-health.sh does |
| Temporary files with fixed names like /tmp/report.txt | Two runs overwrite each other, files are left behind | Use mktemp and remove it in an EXIT trap |
| Error messages on standard output | They end up inside captured results or reports | Send them to standard error with >&2 and to the journal with logger |
| Moving a finished script into /usr/local/bin with mv | On Rocky it keeps the home SELinux label and services can’t run it | Use install -m 0755 or cp, or run restorecon afterwards |
| Parsing options by hand with $1, $2 | Options in a different order break the script | Use getopts and print help with a here-document |
Interview questions
Click a question to open it. Each one is written the way an interviewer would ask it. Under it you will see why they ask it and one way you could answer, in your own words.
Fresher level
1. What does set -euo pipefail do, in your own words?
Why they ask this: A very common screening question for anyone who will write scripts. They want each part explained, not the line recited.
How you might answer: -e stops the script when a command fails, -u makes using an unset variable an error, and pipefail makes a pipeline fail when any command in it fails, not only the last. Together they turn silent mistakes into a stop.
I also know where -e doesn’t help: inside if conditions and && lists, and after local x=$(cmd). So I still check important commands with if, and I add an ERR trap that prints the line number.
2. How do you make sure a temporary file your script creates is always removed?
Why they ask this: They’re checking whether you know trap and mktemp, which shows you’ve written scripts that run unattended.
How you might answer: I create it with mktemp, so the name is unique, and set trap 'rm -f "$tmp"' EXIT straight after. The EXIT trap runs on a normal end, on exit, on errors with set -e and on Ctrl+C.
It can’t run after kill -9 or a crash, so for anything important I also put temporary files in a known folder that a tmpfiles rule or a cleanup job empties.
2–5 years experience
3. A script with set -e stops with no message on some nights and works on others. How do you find out why?
Why they ask this: They want a debugging method, and bonus points for knowing the classic causes rather than guessing.
How you might answer: I’d run it by hand with bash -x using the same input as a failing night and see the last line it reaches. Then I’d add trap 'echo "failed at line $LINENO" >&2' ERR so the next failure names its line in the log.
The usual causes are commands that legally return non-zero: grep -c or grep -q with no match, ((n++)) when n is 0, or diff finding differences. I’d handle those cases explicitly instead of turning set -e off.
4. How would you write a check script so that Nagios, Icinga or a cron job can all use it?
Why they ask this: This is about contracts: exit codes, output format and behaviour on bad input.
How you might answer: I follow the plugin convention: exit 0 OK, 1 WARNING, 2 CRITICAL and 3 UNKNOWN, with one summary line first or last so it fits in an alert. The worst result wins, and anything the script couldn’t check is UNKNOWN, never OK.
Limits come from options with sensible defaults, bad options exit 3 with a usage message, and the script never changes anything on the server. Like saat-health.sh, I also log the summary with logger so there’s a history on the host itself.
Senior level
5. What are the limits of set -e that you’d warn a junior about?
Why they ask this: Seniors are expected to know that set -e is a safety net with holes, and to teach good habits rather than rely on it.
How you might answer: set -e is ignored in conditions of if, while and until, in every command of an && or || list except the last, in a function called from a condition, and after !. local x=$(cmd) and export x=$(cmd) hide the failure, and inside a command substitution the setting isn’t inherited unless you use shopt -s inherit_errexit.
So I teach it as a net, not a plan: check every important command explicitly, keep functions small, add an ERR trap that logs the line, and run shellcheck in CI. When a script grows past a few hundred lines with complex error handling, I’d rather move it to Python or Ansible.
6. You’re asked to roll saat-health.sh out to 300 servers and wire it into monitoring. What changes before you do?
Why they ask this: They want to hear about packaging, configuration, safety on unusual hosts and testing at scale.
How you might answer: First, the script goes into Git with a version number, shellcheck in CI and a test run on both Rocky and Ubuntu VMs. I’d deploy it with Ansible (Lesson 64) or an RPM and deb package, with host-specific limits in a small config file instead of edited copies.
Then I’d make it safe on odd hosts: a timeout around each check so a hung command can’t block the agent, UNKNOWN when a tool is missing, and a soft rollout to 10 servers first. Monitoring gets the exit code and the RESULT line, and we review the first week of WARNINGs to tune the limits before anyone gets paged.
Quick quiz
Q1. Which line makes a script stop at a failed command, treat unset variables as errors and notice failures anywhere in a pipeline?
set -xset -euo pipefailset +eshopt -s nullglob
Show answer
B — -e stops on errors, -u catches unset variables, and pipefail checks every part of a pipeline.
Q2. Why does count=0; ((count++)) stop a script that uses set -e?
- count is read-only
- ((count++)) returns the old value 0, which bash treats as failure
- The syntax is wrong in bash 5
- set -e blocks arithmetic
Show answer
B — The expression evaluates to 0, so the command exits 1. Use count=$((count + 1)).
Q3. What does trap 'rm -f "$TMP"' EXIT guarantee?
- The file is removed only after a normal end
- The file is removed on a normal end, an error exit and Ctrl+C
- The file is removed even after kill -9
- The file is never created
Show answer
B — The EXIT trap runs on every way out bash controls. SIGKILL can’t be trapped.
Q4. saat-health.sh finds one file system at 85% and cannot find the ss command. What does it exit with?
- 0
- 1 (WARNING)
- 2 (CRITICAL)
- 3 (UNKNOWN)
Show answer
B — WARNING beats UNKNOWN, so the known problem wins. It would only be CRITICAL at 90% disk use or with a required port missing.
Q5. In getopts ':w:c:qh' opt, which options expect a value?
- q and h
- w and c
- All four
- None
Show answer
B — A letter followed by a colon takes a value in $OPTARG. The leading colon only turns on quiet error handling.
Quick recap
- Bash scripting for admins starts with
set -Eeuo pipefail, and with knowing where it doesn’t protect you. - Keep helpers in functions with
localvariables; return status with return and data with echo. - Log with
logger -t TAG -p user.LEVEL, send problems to standard error, and usemktempplustrap '...' EXITfor cleanup. - Parse options with getopts and print help from a here-document; bad usage exits with its own code.
- /usr/local/bin/saat-health.sh runs Lesson 1’s checks with limits and exits 0 OK, 1 WARNING, 2 CRITICAL or 3 UNKNOWN.
- Test with
bash -n, shellcheck andbash -x, then push the limits on a test VM before a monitor depends on it.
Next lesson
Lesson 40: Time sync with chrony → You’ll read the clock with timedatectl, check chrony’s sources and tracking on rocky01 and ubu01, set the time zone, turn rocky01 into the lab’s time server and fix a VM whose clock and firewall are both wrong.
Further reading: bash scripting for admins
Want to go deeper into bash scripting for admins? Read the set builtin and trap and getopts sections of the GNU Bash manual, the man7.org page for logger(1), the Monitoring Plugins development guidelines for the exit code convention, and the ShellCheck wiki. You can also see the full plan for this course on the Linux Administration course page.
