Final project: build it, break it, fix it

Steps 2 to 6 each check one layer. On the course VMs as they are, run the check, and build only what’s missing using the lesson named in the step. For a full rebuild, use the fresh path in step 1 and build each stage from its lessons before you run its check.

Commands run on rocky01 as student unless the step says ubu01.

1Choose your starting point and take snapshots

On the laptop, with both VMs shut down, snapshot the state you have now:

VBoxManage snapshot rocky01 take before-l81 --description "course state before the final project"
VBoxManage snapshot ubu01 take before-l81 --description "course state before the final project"
VBoxManage snapshot rocky01 list

You should see something like this (your list is longer):

   Name: base (UUID: 3c1d2e4f-5a6b-4c7d-8e9f-0a1b2c3d4e5f)
   ...
      Name: before-l81 (UUID: 9f8e7d6c-5b4a-4392-8170-6f5e4d3c2b1a) *

The star marks the current snapshot. Now pick one path:

The fresh path needs the Lesson 33 SSH key from ubu01 on rocky01 and the hardening drop-in again, because “base” is older than both. Expect it to take a full day; that is the point of it.

✅ Checkpoint: Both VMs have a before-l81 snapshot and you know which path you’re on.

2Stage 1: users, groups and sudo

Run this on rocky01:

id priya; id arjun; id appsvc
getent group webteam
sudo visudo -c

You should see something like this:

uid=1001(priya) gid=1001(priya) groups=1001(priya),2001(webteam)
uid=1002(arjun) gid=1002(arjun) groups=1002(arjun),2001(webteam)
uid=991(appsvc) gid=991(appsvc) groups=991(appsvc)
webteam:x:2001:priya,arjun
/etc/sudoers: parsed OK
/etc/sudoers.d/ad-linux-admins: parsed OK
/etc/sudoers.d/bkpsvc: parsed OK
/etc/sudoers.d/saatops: parsed OK
/etc/sudoers.d/webteam: parsed OK

On the fresh path you’ll see only webteam (and bkpsvc after stage 5); ad-linux-admins and saatops come from Lessons 54 and 64, which SaatShop doesn’t need. Missing something? These lines rebuild stage 1, and Lesson 16 step 4 has the webteam rule:

sudo groupadd -g 2001 webteam
sudo useradd -m -u 1001 -c "Priya Nair" -G webteam priya
sudo useradd -m -u 1002 -c "Arjun Rao" -G webteam arjun
sudo useradd -r -s /sbin/nologin appsvc
sudo visudo -f /etc/sudoers.d/webteam

✅ Checkpoint: priya and arjun are in webteam (GID 2001), appsvc exists and visudo -c parses every file.

3Stage 2: volumes and mounts

Run this on rocky01:

sudo vgs vg_data
sudo lvs vg_data
df -hT /srv/app /srv/logs /var/lib/mysql
sudo findmnt --verify | tail -n 1

You should see something like this:

  VG      #PV #LV #SN Attr   VSize VFree
  vg_data   2   3   0 wz--n- 9.99g 2.24g
  LV      VG      Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  lv_app  vg_data -wi-ao---- 4.25g
  lv_db   vg_data -wi-ao---- 2.00g
  lv_logs vg_data -wi-ao---- 1.50g
Filesystem                  Type  Size  Used Avail Use% Mounted on
/dev/mapper/vg_data-lv_app  xfs   4.2G  117M  4.1G   3% /srv/app
/dev/mapper/vg_data-lv_logs ext4  1.5G   28K  1.4G   1% /srv/logs
/dev/mapper/vg_data-lv_db   xfs   2.0G  178M  1.8G   9% /var/lib/mysql
0 parse errors, 0 errors, 0 warnings

Two PVs, three LVs and 2.24 GiB unallocated, which the Lesson 63 backup needs for its 512 MiB snapshot of lv_db. On the fresh path, build it like this, then add the three UUID lines to fstab as in Lesson 44 and run sudo mount -a and sudo restorecon -Rv /srv/app /srv/logs /var/lib/mysql:

sudo pvcreate /dev/sdb /dev/sdd
sudo vgcreate vg_data /dev/sdb /dev/sdd
sudo lvcreate -n lv_app -L 4.25G vg_data && sudo mkfs.xfs /dev/vg_data/lv_app
sudo lvcreate -n lv_logs -L 1.5G vg_data && sudo mkfs.ext4 /dev/vg_data/lv_logs
sudo lvcreate -n lv_db -L 2G vg_data && sudo mkfs.xfs -L dbdata /dev/vg_data/lv_db
sudo mkdir -p /srv/app /srv/logs /var/lib/mysql

Try it yourself: build the PV, VG, LV and fstab steps in the Command Builder.

✅ Checkpoint: vgs shows 3 LVs and 2.24g free, df shows all three mounts, and findmnt --verify ends with 0 errors.

4Stage 3: the web shop

Run this on rocky01:

systemctl is-active nginx php-fpm mariadb
curl -s http://rocky01.saatlab.test/health.php
ls -Z /srv/app/saatshop/health.php
getenforce; sudo firewall-cmd --list-services

You should see something like this:

active
active
active
OK db=up orders=5
unconfined_u:object_r:httpd_sys_content_t:s0 /srv/app/saatshop/health.php
Enforcing
dhcpv6-client http ntp ssh

Five orders, the web content label, SELinux enforcing and http allowed. On the fresh path, Lesson 56 steps 2 to 8 build all of it (skip its step 1, lv_db already exists), and ~/saatshop.sql recreates the database. ntp only appears if you did Lesson 40.

✅ Checkpoint: All three services are active and health.php says “OK db=up orders=5”.

5Stage 4: the share and the central logs

On rocky01, run this:

findmnt -n -o SOURCE,FSTYPE /mnt/share
ls /mnt/share
logger -t l81 "final project check"

You should see something like this:

ubu01:/srv/nfs/share nfs4
README.txt  from-rocky01.txt

On ubu01, run this:

sudo ufw status | grep -E '514|2049'
sudo tail -n 1 /var/log/remote/rocky01/l81.log

You should see something like this:

514/tcp                    ALLOW       10.10.10.11
2049/tcp                   ALLOW       10.10.10.0/24
2026-10-27T10:31:05+05:30 rocky01 l81: final project check

The share is mounted with NFSv4 and the test line crossed from rocky01 to ubu01 through TCP 514. Lessons 47 and 29 build these two paths; each needs its ufw rule on ubu01 (Lesson 34).

✅ Checkpoint: /mnt/share lists two files and ubu01 holds the l81 test line.

6Stage 5: the nightly backup

On ubu01, check the timer and run the job once:

systemctl list-timers backup-rocky01.timer --no-pager | head -n 2
sudo systemctl start backup-rocky01.service
journalctl -u backup-rocky01.service -n 2 -o cat --no-pager

You should see something like this:

NEXT                            LEFT LAST                            PASSED UNIT                 ACTIVATES
Wed 2026-10-28 01:41:09 IST      15h Tue 2026-10-27 01:37:12 IST     9h ago backup-rocky01.timer backup-rocky01.service
backup-post: backup result recorded: ok
backup-rocky01: OK: /srv/backup/rocky01/2026-10-27 (158M)

The job froze MariaDB, took the lv_db snapshot, pulled /etc, /srv/app, the metadata and the database copy, then removed the snapshot. On the fresh path, Lesson 62 steps 3 to 7 and Lesson 63 step 5 and 6 build it in that order.

✅ Checkpoint: The job ends with “backup-rocky01: OK” and the timer shows its next run at about 01:30.

7Stage 6: add the SaatShop check and its timer

On rocky01, write the check script to your home folder (paste the whole block):

cat > ~/saatshop-check.sh <<'EOF'
#!/bin/bash
# /usr/local/bin/saatshop-check.sh  (SaatVedha Linux course, Lesson 81)
# Checks every layer SaatShop needs on rocky01. Exit 0 = all OK, 2 = something failed.
set -uo pipefail
URL=http://rocky01.saatlab.test/health.php
META=/var/local/backup-meta
fails=0
ok()  { printf 'OK    %-7s %s\n' "$1" "$2"; }
bad() { printf 'FAIL  %-7s %s\n' "$1" "$2"; fails=$((fails + 1)); }

# 1. The page that users and monitoring ask for, by its real name
if out=$(curl -fsS -m 5 "$URL" 2>&1) && [[ $out == OK* ]]; then
    ok web "$out"
else
    bad web "$URL: ${out:-no answer}"
fi

# 2. The three services behind it
for s in nginx php-fpm mariadb; do
    state=$(systemctl is-active "$s" 2>/dev/null)
    if [[ $state == active ]]; then ok service "$s active"; else bad service "$s $state"; fi
done

# 3. Mounts and space (timeout, because a dead NFS server makes df hang)
for m in /srv/app /srv/logs /var/lib/mysql /mnt/share; do
    if ! findmnt -n "$m" >/dev/null; then bad mount "$m is not mounted"; continue; fi
    use=$(timeout 5 df -P "$m" 2>/dev/null | awk 'NR==2 {sub("%", "", $5); print $5}')
    if [[ -z $use ]]; then bad space "$m did not answer in 5 s"
    elif (( use >= 90 )); then bad space "$m ${use}% used"
    else ok space "$m ${use}% used"; fi
done

# 4. ubu01 holds the NFS share, the central logs and the backups
if timeout 3 bash -c '</dev/tcp/10.10.10.21/22' 2>/dev/null; then
    ok network "ubu01 10.10.10.21:22 reachable"
else
    bad network "ubu01 10.10.10.21:22 not reachable"
fi

# 5. Last good backup (backup-post.sh touches last-ok after every good run)
if [[ -f $META/last-ok ]]; then
    age=$(( ($(date +%s) - $(stat -c %Y "$META/last-ok")) / 3600 ))
    if (( age <= 26 )); then ok backup "last good run ${age} h ago"; else bad backup "last good run ${age} h ago"; fi
else
    bad backup "no good run recorded in $META"
fi

# 6. The general health script from Lesson 39, for information
if [[ -x /usr/local/bin/saat-health.sh ]]; then
    printf 'INFO  %-7s %s\n' health "$(/usr/local/bin/saat-health.sh -q -n -p 22,80)"
fi

if (( fails == 0 )); then
    echo "SAATSHOP: OK"
    logger -t saatshop-check "SAATSHOP: OK"
    exit 0
fi
echo "SAATSHOP: FAIL ($fails failed)"
logger -t saatshop-check -p user.warning "SAATSHOP: FAIL ($fails failed)"
exit 2
EOF
bash -n ~/saatshop-check.sh && sudo install -o root -g root -m 0755 ~/saatshop-check.sh /usr/local/bin/

It checks the page users ask for (by its real name), the three services, space on the four SaatShop mounts with a 5-second limit (a dead NFS server makes df hang), the path to ubu01, the age of the last good backup, and prints saat-health.sh’s summary from Lesson 39 for information. install creates a new file in /usr/local/bin, so it gets the right SELinux label (bin_t), not the label of your home folder. Now the unit and the timer:

sudo tee /etc/systemd/system/saatshop-check.service >/dev/null <<'EOF'
[Unit]
Description=SaatShop end-to-end check (Lesson 81)
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/saatshop-check.sh
EOF
sudo tee /etc/systemd/system/saatshop-check.timer >/dev/null <<'EOF'
[Unit]
Description=Run the SaatShop check every 5 minutes

[Timer]
OnCalendar=*:0/5
Persistent=true

[Install]
WantedBy=timers.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now saatshop-check.timer
sudo /usr/local/bin/saatshop-check.sh; echo "exit $?"

You should see something like this:

Created symlink '/etc/systemd/system/timers.target.wants/saatshop-check.timer' → '/etc/systemd/system/saatshop-check.timer'.
OK    web     OK db=up orders=5
OK    service nginx active
OK    service php-fpm active
OK    service mariadb active
OK    space   /srv/app 3% used
OK    space   /srv/logs 1% used
OK    space   /var/lib/mysql 9% used
OK    space   /mnt/share 38% used
OK    network ubu01 10.10.10.21:22 reachable
OK    backup  last good run 9 h ago
INFO  health  RESULT: OK (0 critical, 0 warning, 0 unknown)
SAATSHOP: OK
exit 0

Try it yourself: build a service and timer pair in the Command Builder.

✅ Checkpoint: The check prints SAATSHOP: OK with exit 0, and systemctl list-timers saatshop-check.timer shows a run every five minutes.

8Install break-saatshop.sh and take the “built” snapshot

On rocky01, paste the whole block:

cat > ~/break-saatshop.sh <<'SCRIPT_END'
#!/bin/bash
# /usr/local/sbin/break-saatshop.sh  (SaatVedha Linux course, Lesson 81)
# Puts one realistic fault into the SaatShop lab on rocky01, and takes it out again.
#   sudo break-saatshop.sh list        what each scenario looks like from outside
#   sudo break-saatshop.sh N           inject scenario N (1-10)
#   sudo break-saatshop.sh random      inject a scenario without saying which
#   sudo break-saatshop.sh status      which scenarios are active
#   sudo break-saatshop.sh reset [N]   undo scenario N, or every active one
set -euo pipefail

STATE=/var/lib/break-saatshop
UBU=10.10.10.21
BADGW=10.10.10.254
SYMPTOM=(""
  "Monitoring says a SaatShop file system is almost full"
  "Another file system is filling up, but nobody can find the files"
  "Nothing looks wrong yet. Reboot rocky01 and watch the console"
  "The 5-minute SaatShop check has stopped running"
  "The health page is broken, but the shop page still opens"
  "SaatShop works on rocky01 itself, but not from ubu01"
  "rocky01 can no longer talk to ubu01 (share, logs, backups)"
  "The check fails on the web test, yet Nginx looks fine"
  "The shop hangs, then times out, for everyone"
  "Tonight's backup of rocky01 failed (run the job on ubu01)")

fail() { echo "break-saatshop: $*" >&2; exit 1; }
say()  { echo "break-saatshop: $*"; }
need_mount() { mountpoint -q "$1" || fail "$1 is not a mount point ($2)"; }
need_file()  { [[ -e $1 ]] || fail "$1 not found ($2)"; }
active_list() {
    local f out=""
    for f in "$STATE"/active-*; do [[ -e $f ]] && out+="${f##*-} "; done
    echo "${out% }"
}
saved() { cat "$STATE/active-$1"; }

# 1. A forgotten dump file fills the database volume
break_1() {
    need_mount /var/lib/mysql "lv_db, Lesson 56"
    local f avail
    f=/var/lib/mysql/saatshop-full-$(date -d '6 days ago' +%F).sql
    avail=$(df -Pk /var/lib/mysql | awk 'NR==2 {print $4}')
    (( avail > 40960 )) || fail "/var/lib/mysql already has less than 40 MiB free"
    echo "$f" > "$STATE/active-1"
    fallocate -l "$(( avail - 20480 ))K" "$f"
    touch -d '6 days ago' "$f"
}
undo_1() { local f; f=$(saved 1); [[ -n $f ]] && rm -f -- "$f"; return 0; }

# 2. A writer keeps a deleted log open on /srv/logs
break_2() {
    need_mount /srv/logs "lv_logs, Lesson 44"
    local f=/srv/logs/saatshop-audit.log avail
    avail=$(df -Pk /srv/logs | awk 'NR==2 {print $4}')
    (( avail > 81920 )) || fail "/srv/logs already has less than 80 MiB free"
    touch "$STATE/active-2"
    fallocate -l "$(( avail - 40960 ))K" "$f"
    systemd-run --quiet --unit=saatshop-audit --description="SaatShop audit writer" \
        /bin/bash -c "exec 3>>$f; exec sleep infinity"
    sleep 1
    rm -f "$f"
}
undo_2() {
    systemctl stop saatshop-audit.service 2>/dev/null || true
    systemctl reset-failed saatshop-audit.service 2>/dev/null || true
    rm -f /srv/logs/saatshop-audit.log
}

# 3. An fstab line for a disk that is not there, without nofail
break_3() {
    local uuid
    uuid=$(cat /proc/sys/kernel/random/uuid)
    echo "$uuid" > "$STATE/active-3"
    cp -a /etc/fstab "$STATE/fstab.before-3"
    mkdir -p /srv/archive
    printf '%s\n' "# SaatShop invoice archive LUN (CHG-2026-1027)" \
        "UUID=$uuid  /srv/archive  xfs  defaults  0 0" >> /etc/fstab
    systemctl daemon-reload
}
undo_3() {
    local uuid; uuid=$(saved 3)
    [[ -n $uuid ]] && sed -i "/$uuid/d; /CHG-2026-1027/d" /etc/fstab
    systemctl daemon-reload
    rmdir /srv/archive 2>/dev/null || true
}

# 4. The check script loses its execute bit
break_4() {
    local f=/usr/local/bin/saatshop-check.sh
    need_file "$f" "Lesson 81, stage 6"
    stat -c %a "$f" > "$STATE/active-4"
    chmod 0644 "$f"
}
undo_4() { local m; m=$(saved 4); chmod "${m:-755}" /usr/local/bin/saatshop-check.sh; }

# 5. health.php gets a label from a home folder
break_5() {
    local f=/srv/app/saatshop/health.php
    need_file "$f" "Lesson 56"
    [[ $(getenforce) == Enforcing ]] || fail "SELinux is not enforcing, so this scenario would show nothing"
    touch "$STATE/active-5"
    chcon -t admin_home_t "$f"
}
undo_5() { restorecon /srv/app/saatshop/health.php; }

# 6. HTTP disappears from the permanent firewall rules
break_6() {
    local zone
    zone=$(firewall-cmd --get-zone-of-interface=enp0s3 2>/dev/null || firewall-cmd --get-default-zone)
    firewall-cmd --quiet --permanent --zone="$zone" --query-service=http \
        || fail "http is not open in zone $zone (Lesson 56)"
    echo "$zone" > "$STATE/active-6"
    firewall-cmd --quiet --permanent --zone="$zone" --remove-service=http
    firewall-cmd --quiet --reload
}
undo_6() {
    local zone; zone=$(saved 6)
    firewall-cmd --quiet --permanent --zone="${zone:-public}" --add-service=http || true
    firewall-cmd --quiet --reload
}

# 7. A host route to ubu01 through a gateway that does not exist
break_7() {
    local con
    con=$(nmcli -g GENERAL.CONNECTION device show enp0s3)
    [[ -n $con ]] || fail "no NetworkManager connection is active on enp0s3"
    echo "$con" > "$STATE/active-7"
    nmcli connection modify "$con" +ipv4.routes "$UBU/32 $BADGW"
    ip route replace "$UBU/32" via "$BADGW" dev enp0s3
}
undo_7() {
    local con; con=$(saved 7)
    nmcli connection modify "${con:-enp0s3}" -ipv4.routes "$UBU/32 $BADGW" 2>/dev/null || true
    ip route del "$UBU/32" via "$BADGW" dev enp0s3 2>/dev/null || true
}

# 8. A new /etc/hosts line sends the web name to the wrong address
break_8() {
    touch "$STATE/active-8"
    cp -a /etc/hosts "$STATE/hosts.before-8"
    sed -i '1i 10.10.10.12   rocky01.saatlab.test   # new web VIP (CHG-2026-1031)' /etc/hosts
}
undo_8() { sed -i '/^10\.10\.10\.12[[:space:]]\+rocky01\.saatlab\.test/d' /etc/hosts; }

# 9. Nginx gets a tiny open-files limit while clients keep connecting
break_9() {
    local d=/etc/systemd/system/nginx.service.d
    systemctl is-active --quiet nginx || fail "nginx is not running (Lesson 56)"
    [[ -e $d/50-tuning.conf ]] && fail "$d/50-tuning.conf already exists, move it away first"
    touch "$STATE/active-9"
    mkdir -p "$d"
    printf '%s\n' "# copied from the old web server build (CHG-2026-1029)" \
        "[Service]" "LimitNOFILE=64" > "$d/50-tuning.conf"
    systemctl daemon-reload
    systemctl restart nginx
    cat > "$STATE/clients.py" <<'EOF'
# Keeps about 400 idle client connections open to Nginx (Lesson 81, scenario 9)
import socket, time
conns = []
def alive(s):
    try:
        return s.recv(1, socket.MSG_PEEK | socket.MSG_DONTWAIT) != b""
    except BlockingIOError:
        return True
    except OSError:
        return False
while True:
    keep = []
    for s in conns:
        if alive(s):
            keep.append(s)
        else:
            s.close()
    conns = keep
    while len(conns) < 400:
        try:
            conns.append(socket.create_connection(("127.0.0.1", 80), timeout=2))
        except OSError:
            break
    time.sleep(5)
EOF
    systemd-run --quiet --unit=saatshop-clients --description="SaatShop busy-hour clients" \
        /usr/bin/python3 "$STATE/clients.py"
}
undo_9() {
    systemctl stop saatshop-clients.service 2>/dev/null || true
    systemctl reset-failed saatshop-clients.service 2>/dev/null || true
    rm -f /etc/systemd/system/nginx.service.d/50-tuning.conf "$STATE/clients.py"
    rmdir /etc/systemd/system/nginx.service.d 2>/dev/null || true
    systemctl daemon-reload
    systemctl restart nginx
}

# 10. The backup account expires
break_10() {
    id bkpsvc >/dev/null 2>&1 || fail "user bkpsvc not found (Lesson 62)"
    getent shadow bkpsvc | cut -d: -f8 > "$STATE/active-10"
    chage -E "$(date -d yesterday +%F)" bkpsvc
}
undo_10() { local e; e=$(saved 10); chage -E "${e:--1}" bkpsvc; }

inject() {
    local n=$1 now
    now=$(active_list)
    [[ -z $now ]] || fail "scenario $now is still active. Run: sudo break-saatshop.sh reset"
    "break_$n"
    logger -t break-saatshop "a scenario was injected"
    if [[ ${2:-} == quiet ]]; then
        say "a random scenario is now active."
    else
        say "scenario $n is now active."
    fi
    say "symptom: ${SYMPTOM[$n]}"
    say "start with: sudo /usr/local/bin/saatshop-check.sh"
    if [[ $n == 7 ]]; then say "work from the VirtualBox console or your laptop SSH (port 2211), not through ubu01"; fi
}

reset_one() {
    local n=$1
    [[ $n =~ ^([1-9]|10)$ ]] || fail "no scenario $n"
    if [[ ! -e $STATE/active-$n ]]; then say "scenario $n is not active"; return 0; fi
    "undo_$n"
    rm -f "$STATE/active-$n"
    logger -t break-saatshop "scenario $n reset"
    say "scenario $n undone"
}

[[ $EUID -eq 0 ]] || fail "run me with sudo"
[[ $(hostname -s) == rocky01 ]] || fail "this script is for rocky01 only (this is $(hostname -s))"
install -d -m 0700 "$STATE"

case ${1:-} in
    list)   for i in {1..10}; do printf '%2d  %s\n' "$i" "${SYMPTOM[$i]}"; done ;;
    status) now=$(active_list); say "active: ${now:-none}" ;;
    random) inject $(( RANDOM % 10 + 1 )) quiet ;;
    reset)
        if [[ -n ${2:-} ]]; then
            reset_one "$2"
        else
            now=$(active_list)
            [[ -n $now ]] || say "nothing to reset"
            for n in $now; do reset_one "$n"; done
        fi ;;
    [1-9]|10) inject "$1" ;;
    *) sed -n '3,8p' "$0"; exit 1 ;;
esac
SCRIPT_END
bash -n ~/break-saatshop.sh && sudo install -o root -g root -m 0750 ~/break-saatshop.sh /usr/local/sbin/
sudo /usr/local/sbin/break-saatshop.sh list

You should see this:

 1  Monitoring says a SaatShop file system is almost full
 2  Another file system is filling up, but nobody can find the files
 3  Nothing looks wrong yet. Reboot rocky01 and watch the console
 4  The 5-minute SaatShop check has stopped running
 5  The health page is broken, but the shop page still opens
 6  SaatShop works on rocky01 itself, but not from ubu01
 7  rocky01 can no longer talk to ubu01 (share, logs, backups)
 8  The check fails on the web test, yet Nginx looks fine
 9  The shop hangs, then times out, for everyone
10  Tonight's backup of rocky01 failed (run the job on ubu01)

The list shows symptoms, not causes, because that’s all you get from a ticket. Read the script once now (each scenario is a short break_N function with its undo_N under it), then try not to look at it again while you play. Shut both VMs down and take the snapshot you’ll come back to:

VBoxManage snapshot rocky01 take l81-built
VBoxManage snapshot ubu01 take l81-built

✅ Checkpoint: The script lists ten scenarios and both VMs have an l81-built snapshot.