Prepare Linux for Commvault and Rubrik backup agents

You’ll switch between rocky01 (the client) and ubu01 (the backup server), and step 2 uses kafka01 once. Take snapshots first on the laptop: VBoxManage snapshot rocky01 take before-l62 and the same for ubu01. On ubu01, load your Lesson 33 key into the agent before you start (eval "$(ssh-agent -s)"; ssh-add), because a few steps copy files with it.

1Run the agent pre-checks on rocky01

Run this on rocky01:

cat /etc/redhat-release; uname -r; getenforce
timedatectl | grep -E 'Time zone|synchronized'
getent hosts rocky01.saatlab.test ubu01.saatlab.test
getent hosts 10.10.10.11
df -h --output=target,avail /opt /var/log
free -h | head -n 2

You should see something like this:

Rocky Linux release 10.2 (Red Quartz)
6.12.0-211.16.1.el10_2.0.1.x86_64
Enforcing
                Time zone: Asia/Kolkata (IST, +0530)
System clock synchronized: yes
10.10.10.11     rocky01.saatlab.test rocky01
10.10.10.21     ubu01.saatlab.test ubu01
10.10.10.11     rocky01.saatlab.test rocky01
Mounted on Avail
/            14G
/            14G
               total        used        free      shared  buff/cache   available
Mem:           1.7Gi       1.1Gi       142Mi        18Mi       628Mi       612Mi

These are the lines a vendor’s pre-install checklist asks for, and you’ll paste them into the ticket. The OS and kernel go against the support list, SELinux stays Enforcing, chrony keeps the clock in sync (Lesson 40), and the name of each machine resolves to the right IP both ways. /opt and /var/log share the root file system here, with plenty of room for 1 GB of software and 1 GB of logs; also run findmnt /tmp to see whether /tmp is its own file system with less space. rocky01’s 2 GB of RAM would be the bare minimum for a real Commvault client, which is worth saying in the ticket rather than finding out during the first big backup.

At work, check names against DNS, not only /etc/hosts: dig +short rocky01.saatlab.test and dig +short -x 10.10.10.11. The backup server resolves your name through DNS, and a reverse record that still points to a decommissioned server is a classic reason for “client not reachable”.

✅ Checkpoint: You have the OS, kernel, SELinux mode, time sync, name lookups, space and memory for rocky01, and all of them look sane.

2Open the agent ports for the backup server only

On rocky01, define a firewalld service for Commvault’s two ports and allow it only from ubu01, which plays the backup server:

sudo firewall-cmd --permanent --new-service=commvault
sudo firewall-cmd --permanent --service=commvault --set-description="Commvault CVD 8400 and data/control 8403 (Lesson 62)"
sudo firewall-cmd --permanent --service=commvault --add-port=8400/tcp
sudo firewall-cmd --permanent --service=commvault --add-port=8403/tcp
sudo firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="10.10.10.21" service name="commvault" accept'
sudo firewall-cmd --reload
sudo firewall-cmd --info-service=commvault
sudo firewall-cmd --zone=public --list-rich-rules

You should see something like this:

success
success
success
success
success
success
commvault
  ports: 8400/tcp 8403/tcp
  protocols:
  source-ports:
  modules:
  destination:
  includes:
  helpers:
rule family="ipv4" source address="10.10.10.21" service name="commvault" accept

A named service keeps the rule readable: six months from now, “commvault” in --list-all tells the next admin why the ports are open, where “8400/tcp” alone doesn’t. The rich rule lets only 10.10.10.21 use it. Now prove it from both sides.

On ubu01, run nc -zv -w 3 10.10.10.11 8403, and on kafka01 run timeout 3 bash -c '</dev/tcp/10.10.10.11/8403':

nc: connect to 10.10.10.11 port 8403 (tcp) failed: Connection refused
bash: connect: No route to host
bash: line 1: /dev/tcp/10.10.10.11/8403: No route to host

The first line is from ubu01 and it’s the result you want: “Connection refused” means the firewall let the packet in and nothing is listening yet, which is right before the agent is installed. kafka01 got “No route to host”, firewalld’s reject from Lesson 34, so other machines can’t even reach the port. For Rubrik you’d do the same with ports 12800/tcp and 12801/tcp, allowed from the cluster’s addresses.

✅ Checkpoint: --info-service=commvault shows 8400/tcp and 8403/tcp, ubu01 gets “Connection refused” and kafka01 gets “No route to host”.

3Write the pre and post scripts on rocky01

Paste this whole block on rocky01. It creates the pre-script:

sudo tee /usr/local/sbin/backup-pre.sh >/dev/null <<'EOF'
#!/bin/bash
# /usr/local/sbin/backup-pre.sh  (SaatVedha Linux course, Lesson 62)
# Runs as root on rocky01 before every backup. Exit 0 = go ahead, anything else stops the job.
set -euo pipefail
META=/var/local/backup-meta
say() { logger -t backup-pre "$*"; echo "backup-pre: $*"; }

# 1. Only back up a healthy app, or you save a broken state
for svc in mariadb php-fpm nginx; do
    if ! systemctl is-active --quiet "$svc"; then
        say "FAIL: $svc is not running"
        exit 2
    fi
done

# 2. Save what you need to rebuild the server, not only its files
install -d -m 0700 "$META"
rpm -qa --qf '%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH}\n' | sort > "$META/packages.txt"
lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINTS > "$META/lsblk.txt"
{ vgs; lvs; } > "$META/lvm.txt" 2>&1
cp /etc/fstab "$META/fstab"

# 3. A logical dump of the SaatShop database (Lesson 63 adds an LVM snapshot)
mariadb-dump --single-transaction --routines --triggers --databases saatshop > "$META/saatshop.sql"

say "OK: metadata and saatshop.sql ready in $META"
EOF

Then the post-script:

sudo tee /usr/local/sbin/backup-post.sh >/dev/null <<'EOF'
#!/bin/bash
# /usr/local/sbin/backup-post.sh  (SaatVedha Linux course, Lesson 62)
# Runs as root on rocky01 after every backup attempt. $1 is "ok" or "failed".
set -euo pipefail
META=/var/local/backup-meta
result=${1:-unknown}
say() { logger -t backup-post "$*"; echo "backup-post: $*"; }

# The dump holds customer data: once it is on the backup server, don't keep a spare copy here
rm -f "$META/saatshop.sql"
date '+%F %T' > "$META/last-$result"
say "backup result recorded: $result"
EOF
sudo chmod 0750 /usr/local/sbin/backup-pre.sh /usr/local/sbin/backup-post.sh
sudo bash -n /usr/local/sbin/backup-pre.sh && sudo bash -n /usr/local/sbin/backup-post.sh && echo "syntax OK"

Run both by hand once, the way the backup job will:

sudo /usr/local/sbin/backup-pre.sh; echo "pre exit code: $?"
sudo ls -l /var/local/backup-meta
sudo /usr/local/sbin/backup-post.sh ok

You should see something like this:

backup-pre: OK: metadata and saatshop.sql ready in /var/local/backup-meta
pre exit code: 0
total 32
-rw-r--r--. 1 root root   612 Oct 27 15:20 fstab
-rw-r--r--. 1 root root   724 Oct 27 15:20 lsblk.txt
-rw-r--r--. 1 root root   689 Oct 27 15:20 lvm.txt
-rw-r--r--. 1 root root 18734 Oct 27 15:20 packages.txt
-rw-r--r--. 1 root root  2760 Oct 27 15:20 saatshop.sql
backup-post: backup result recorded: ok

The pre-script refuses to go on if MariaDB, PHP-FPM or Nginx is down, because a backup of a broken app is a broken backup. It saves what you need to rebuild the server (the package list, disks, LVM layout and fstab) next to a mariadb-dump of saatshop, taken with --single-transaction so InnoDB gives one consistent view without locking the shop. set -euo pipefail (Lesson 39) makes any failed command end the script with a non-zero code, which is what tells the job to stop.

The post-script deletes the dump once it’s been copied, so a file full of customer orders doesn’t sit on the server for anyone with root to read later, and it leaves a timestamp in last-ok or last-failed. Lesson 61’s Zabbix can alert when last-ok gets older than a day.

✅ Checkpoint: The pre-script exits 0, /var/local/backup-meta holds the five files, and the post-script records “ok”.

4Create the backup service account and its narrow rights

On rocky01, create the account, the gate script that will be its only command, and the sudo rule (paste the whole block):

sudo useradd --system --create-home --home-dir /var/lib/bkpsvc --shell /bin/bash --comment "Backup service account" bkpsvc
sudo tee /usr/local/sbin/backup-gate.sh >/dev/null <<'EOF'
#!/bin/bash
# /usr/local/sbin/backup-gate.sh  (SaatVedha Linux course, Lesson 62)
# Forced command for the backup key of bkpsvc. It allows these requests and nothing else:
#   pre | post ok | post failed | rsync in sender (read-only) mode
set -euo pipefail
set -f                                  # never expand wildcards in the request
req=${SSH_ORIGINAL_COMMAND:-}
from=${SSH_CLIENT:-local}
logger -t backup-gate "request from ${from%% *}: ${req:0:120}"

case $req in
    pre)
        exec sudo -n /usr/local/sbin/backup-pre.sh ;;
    "post ok"|"post failed")
        exec sudo -n /usr/local/sbin/backup-post.sh "${req#post }" ;;
    "rsync --server --sender "*)
        case $req in
            *--remove*|*--log-file*) ;;   # these would let the caller delete or write files
            *)  # shellcheck disable=SC2086  # split the request into rsync's own arguments
                exec sudo -n /usr/bin/rsync ${req#rsync } ;;
        esac ;;
esac
echo "backup-gate: request not allowed" >&2
exit 1
EOF
sudo chmod 0755 /usr/local/sbin/backup-gate.sh
sudo tee /etc/sudoers.d/bkpsvc >/dev/null <<'EOF'
# /etc/sudoers.d/bkpsvc  (Lesson 62): what the backup account may run as root, nothing more
bkpsvc ALL=(root) NOPASSWD: /usr/local/sbin/backup-pre.sh, \
                            /usr/local/sbin/backup-post.sh ok, \
                            /usr/local/sbin/backup-post.sh failed, \
                            /usr/bin/rsync --server --sender *
EOF
sudo chmod 0440 /etc/sudoers.d/bkpsvc
sudo visudo -cf /etc/sudoers.d/bkpsvc
id bkpsvc
sudo -l -U bkpsvc | tail -n 2

You should see something like this:

/etc/sudoers.d/bkpsvc: parsed OK
uid=976(bkpsvc) gid=976(bkpsvc) groups=976(bkpsvc)
User bkpsvc may run the following commands on rocky01:
    (root) NOPASSWD: /usr/local/sbin/backup-pre.sh, /usr/local/sbin/backup-post.sh ok, /usr/local/sbin/backup-post.sh failed, /usr/bin/rsync --server --sender *

bkpsvc is a system account with a locked password, so nobody can log in as it with a password. Its sudo rule names the exact commands and arguments, the way Lesson 16 taught: the two scripts (and the post-script only with “ok” or “failed”), and rsync only in --server --sender mode, which is the mode where rsync on this side reads and sends files. The gate script adds a second check, because sudo’s * would also match rsync options such as --remove-source-files, which deletes the files it has sent.

✅ Checkpoint: visudo -cf says “parsed OK”, and sudo -l -U bkpsvc lists four commands.

5Give the backup server its own key, locked to the gate

On ubu01, create a key without a passphrase for the job (a timer can’t type one) and copy the public half to rocky01 with your normal key:

ssh-keygen -t ed25519 -N '' -C 'backup-job@ubu01' -f ~/.ssh/backup_ed25519
scp ~/.ssh/backup_ed25519.pub rocky01:/tmp/

On rocky01, install it for bkpsvc with its restrictions, and give the folder the right SELinux label:

sudo install -d -m 0700 -o bkpsvc -g bkpsvc /var/lib/bkpsvc/.ssh
echo 'restrict,from="10.10.10.21",command="/usr/local/sbin/backup-gate.sh" '"$(cat /tmp/backup_ed25519.pub)" | sudo tee /var/lib/bkpsvc/.ssh/authorized_keys
sudo chown bkpsvc:bkpsvc /var/lib/bkpsvc/.ssh/authorized_keys
sudo chmod 0600 /var/lib/bkpsvc/.ssh/authorized_keys
sudo restorecon -Rv /var/lib/bkpsvc/.ssh
rm /tmp/backup_ed25519.pub

You should see something like this:

restrict,from="10.10.10.21",command="/usr/local/sbin/backup-gate.sh" ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIL0d3Wq8bV2nYk7rT4xPz1mHc9sJ5uFgE6aQyN2oR8tK backup-job@ubu01
Relabeled /var/lib/bkpsvc/.ssh from unconfined_u:object_r:var_lib_t:s0 to unconfined_u:object_r:ssh_home_t:s0
Relabeled /var/lib/bkpsvc/.ssh/authorized_keys from unconfined_u:object_r:var_lib_t:s0 to unconfined_u:object_r:ssh_home_t:s0

The options in front of the key are the real protection. restrict turns off terminals and every kind of forwarding, from= accepts the key only from ubu01’s address, and command= means sshd runs the gate whatever the client asks for, passing the request in SSH_ORIGINAL_COMMAND. The SELinux policy already has a rule for .ssh folders under /var/lib, so restorecon gives them the ssh_home_t label that sshd expects.

Back on ubu01, add a host alias for the job and test what the key can and can’t do:

cat >> ~/.ssh/config <<'EOF'

Host rocky01-backup
    HostName 10.10.10.11
    User bkpsvc
    IdentityFile ~/.ssh/backup_ed25519
    IdentitiesOnly yes
    BatchMode yes
    ConnectTimeout 10
EOF
ssh rocky01-backup pre
ssh rocky01-backup "post ok"
ssh rocky01-backup "cat /etc/shadow"

You should see something like this:

backup-pre: OK: metadata and saatshop.sql ready in /var/local/backup-meta
backup-post: backup result recorded: ok
backup-gate: request not allowed

The alias keeps the job script short and sets IdentitiesOnly yes, so SSH offers only the backup key. Without it, SSH could also offer your Lesson 33 key from the agent, and each rejected key counts as a failed login for fail2ban on rocky01 (Lesson 55 bans after three). BatchMode yes makes SSH fail at once instead of waiting for a prompt nobody will answer at 1:30 at night. rocky01’s journal shows every request: sudo journalctl -t backup-gate -n 3.

✅ Checkpoint: pre and “post ok” work through the alias, and “cat /etc/shadow” is refused with “request not allowed”.

6Run the backup job from ubu01

On ubu01, install the job script (paste the whole block):

sudo tee /usr/local/bin/backup-rocky01.sh >/dev/null <<'EOF'
#!/bin/bash
# /usr/local/bin/backup-rocky01.sh  (SaatVedha Linux course, Lesson 62)
# Runs on ubu01 as student: pre-script on rocky01, pull the files, post-script on rocky01.
set -euo pipefail
CLIENT=rocky01-backup                     # Host block in ~/.ssh/config
DEST=/srv/backup/rocky01
SOURCES=(/etc /srv/app /var/local/backup-meta)
MIN_FREE_MB=${MIN_FREE_MB:-1024}
TODAY=$(date +%F)
say() { echo "backup-rocky01: $*"; }

# 1. Is there room on the backup target?
free_mb=$(df -Pm "$DEST" | awk 'NR==2 {print $4}')
if (( free_mb < MIN_FREE_MB )); then
    say "FAIL: only ${free_mb} MB free in $DEST, need ${MIN_FREE_MB} MB"
    exit 3
fi

# 2. Pre-script on the client. If it fails, no copy is taken.
if ! ssh "$CLIENT" pre; then
    say "FAIL: pre-script failed on rocky01, nothing copied"
    ssh "$CLIENT" "post failed" || true
    exit 2
fi

# 3. Copy into a dated folder; unchanged files become hard links to the last good copy
prev=$(find "$DEST" -mindepth 1 -maxdepth 1 -type d -name '20??-??-??' ! -name "$TODAY" | sort | tail -n 1)
link=()
if [[ -n $prev ]]; then link=(--link-dest="$prev"); fi
remote=()
for s in "${SOURCES[@]}"; do remote+=("$CLIENT:$s"); done
status=ok
if ! rsync -aHAXR --delete --numeric-ids --fake-super "${link[@]}" "${remote[@]}" "$DEST/$TODAY/"; then
    status=failed
fi

# 4. Post-script always runs, and gets the result
# shellcheck disable=SC2029  # $status is meant to expand here, on ubu01
ssh "$CLIENT" "post $status" || say "WARN: post-script failed on rocky01"
if [[ $status != ok ]]; then
    say "FAIL: copy to $DEST/$TODAY did not complete"
    exit 1
fi
say "OK: $DEST/$TODAY ($(du -sh "$DEST/$TODAY" | cut -f1))"
EOF
sudo chmod 0755 /usr/local/bin/backup-rocky01.sh
bash -n /usr/local/bin/backup-rocky01.sh && echo "syntax OK"

Run it as student, then look at what arrived:

/usr/local/bin/backup-rocky01.sh; echo "job exit code: $?"
ls /srv/backup/rocky01
ls /srv/backup/rocky01/$(date +%F)
sudo apt install -y attr
cd /srv/backup/rocky01/$(date +%F) && getfattr -n user.rsync.%stat etc/shadow; cd

You should see something like this:

backup-pre: OK: metadata and saatshop.sql ready in /var/local/backup-meta
backup-post: backup result recorded: ok
backup-rocky01: OK: /srv/backup/rocky01/2026-10-27 (31M)
job exit code: 0
2026-10-27  site
etc  srv  var
# file: etc/shadow
user.rsync.%stat="100000 0,0 0:0"

The job checked for 1 GB of room, ran the pre-script, pulled /etc, /srv/app and the metadata folder, and ran the post-script with the result. -R kept the full paths, so the copy has etc, srv and var at the top, next to Lesson 37’s site folder. rsync on rocky01 ran as root through the gate and sudo, which is why even /etc/shadow came across.

student on ubu01 can’t create files owned by root, so --fake-super stores the real owner and mode in an extended attribute instead: “100000 0,0 0:0” means a regular file with mode 0000, owned by UID 0 and GID 0, which matches /etc/shadow on rocky01. --numeric-ids keeps the numbers rather than mapping names, because UID 27 is mysql on Rocky and nothing on Ubuntu. When you restore with the same option as root, rsync puts the owners and modes back.

✅ Checkpoint: The job prints “OK” with exit code 0, today’s folder holds etc, srv and var, and getfattr shows the saved owner and mode of etc/shadow.

7Schedule it with a systemd timer on ubu01

Create the service and the timer (paste the whole block):

sudo tee /etc/systemd/system/backup-rocky01.service >/dev/null <<'EOF'
[Unit]
Description=Pull a backup of rocky01 into /srv/backup (SaatVedha Lesson 62)
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
User=student
Group=student
ExecStart=/usr/local/bin/backup-rocky01.sh
Nice=10
IOSchedulingClass=idle
EOF
sudo tee /etc/systemd/system/backup-rocky01.timer >/dev/null <<'EOF'
[Unit]
Description=Nightly backup of rocky01 (SaatVedha Lesson 62)

[Timer]
OnCalendar=*-*-* 01:30:00
RandomizedDelaySec=15min
Persistent=true

[Install]
WantedBy=timers.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now backup-rocky01.timer
sudo systemctl start backup-rocky01.service
systemctl list-timers backup-rocky01.timer --no-legend
journalctl -u backup-rocky01.service -n 6 --no-pager

You should see something like this:

Created symlink '/etc/systemd/system/timers.target.wants/backup-rocky01.timer' → '/etc/systemd/system/backup-rocky01.timer'.
Wed 2026-10-28 01:41:17 IST 10h -         - backup-rocky01.timer backup-rocky01.service
Oct 27 15:36:02 ubu01 systemd[1]: Starting backup-rocky01.service - Pull a backup of rocky01 into /srv/backup (SaatVedha Lesson 62)...
Oct 27 15:36:03 ubu01 backup-rocky01.sh[3120]: backup-pre: OK: metadata and saatshop.sql ready in /var/local/backup-meta
Oct 27 15:36:05 ubu01 backup-rocky01.sh[3120]: backup-post: backup result recorded: ok
Oct 27 15:36:05 ubu01 backup-rocky01.sh[3120]: backup-rocky01: OK: /srv/backup/rocky01/2026-10-27 (31M)
Oct 27 15:36:05 ubu01 systemd[1]: backup-rocky01.service: Deactivated successfully.
Oct 27 15:36:05 ubu01 systemd[1]: Finished backup-rocky01.service - Pull a backup of rocky01 into /srv/backup (SaatVedha Lesson 62).

This is Lesson 27’s pattern with three additions. User=student runs the job without root on the backup server, because it only needs to write into /srv/backup. RandomizedDelaySec=15min spreads the start, so fifty clients on one backup server don’t all start at 01:30:00, and Persistent=true runs a missed backup after ubu01 was off at that time.

The second run of the day went into the same dated folder; --delete keeps that folder an exact copy of rocky01 at the last run.

At work you’d run this as a dedicated account on the backup server, not your own login. Here /srv/backup belongs to student since Lesson 37, so the lab keeps it simple.

✅ Checkpoint: list-timers shows the next run around 01:30 to 01:45, and the journal ends with “Deactivated successfully” and “Finished”.

8See hard links save space, and check a restore

Pretend today’s copy is yesterday’s, run the job again and compare:

cd /srv/backup/rocky01
mv "$(date +%F)" "$(date -d yesterday +%F)"
/usr/local/bin/backup-rocky01.sh | tail -n 1
du -sh "$(date -d yesterday +%F)" "$(date +%F)"
stat -c '%h %n' 20*/etc/hosts
ssh rocky01 cat /etc/hosts | diff - "$(date +%F)/etc/hosts" && echo "backup matches rocky01"

You should see something like this:

backup-rocky01: OK: /srv/backup/rocky01/2026-10-27 (31M)
31M	2026-10-26
1.2M	2026-10-27
2 2026-10-26/etc/hosts
2 2026-10-27/etc/hosts
backup matches rocky01

The job found the older folder and used it with --link-dest: every unchanged file became a hard link to yesterday’s copy (link count 2), so the new night costs about 1.2 MB, the files that did change, while each folder still looks like a full backup you can restore from. du counts a hard-linked file only once, the first time it meets it. The diff shows the copy of /etc/hosts is identical to the live one, and that is the smallest restore test there is.

✅ Checkpoint: du shows the second folder at a small fraction of the first, /etc/hosts has two links, and the diff prints “backup matches rocky01”.