Custom checks: UserParameter, system.run, zabbix_get and zabbix_sender

Agent work happens on web01, testing from zbx01, and items in the frontend at https://zbx01.lab. Agent 2 on Ubuntu reads /etc/zabbix/zabbix_agent2.d/*.conf, so all custom keys go into their own files there, which survive package upgrades.

1Write a first UserParameter

Run this on web01:

sudo tee /etc/zabbix/zabbix_agent2.d/userparameter_lab.conf >/dev/null <<'EOF'
# SaatVedha lab custom keys
UserParameter=nginx.workers,pgrep -c -f 'nginx: worker process'
EOF
sudo zabbix_agent2 -t nginx.workers

You should see:

nginx.workers                                 [s|1]

-t loads the config and runs the key once without needing the service. [s|1] means a successful string result of 1, one worker for a 1 vCPU VM. The running agent doesn’t know the key yet, which you’ll see in the Break It section.

✅ Checkpoint: zabbix_agent2 -t nginx.workers returns a number on web01.

2Reload and test from zbx01

Run this on web01, then the second command on zbx01:

sudo zabbix_agent2 -R userparameter_reload
zabbix_get -s 192.168.56.81 --tls-connect psk --tls-psk-identity "PSK web01" --tls-psk-file /tmp/web01.psk -k nginx.workers          # on zbx01

You should see:

User parameters reloaded
1

-R userparameter_reload tells the running agent to re-read UserParameter lines without a restart; other config changes still need systemctl restart zabbix-agent2. zabbix_get is the server’s view: if it works here, a passive item on web01 will work too. Since Lesson 6 web01 accepts only PSK connections, so every zabbix_get in this lesson passes web01’s key from /tmp/web01.psk.

✅ Checkpoint: zabbix_get from zbx01 returns the worker count.

3A flexible key: age of any file

Run this on web01:

echo 'UserParameter=file.age[*],echo $$(( $$(date +%s) - $$(stat -c %Y "$1") ))' | sudo tee -a /etc/zabbix/zabbix_agent2.d/userparameter_lab.conf
sudo zabbix_agent2 -R userparameter_reload
zabbix_get -s 192.168.56.81 --tls-connect psk --tls-psk-identity "PSK web01" --tls-psk-file /tmp/web01.psk -k 'file.age[/etc/hostname]'     # on zbx01

You should see this (the age in seconds will differ):

UserParameter=file.age[*],echo $$(( $$(date +%s) - $$(stat -c %Y "$1") ))
User parameters reloaded
8057

The answer is the file’s age in seconds. Every $ meant for the shell is doubled, because in a flexible key Zabbix replaces $1 to $9 itself and turns $$ into a single $ before the shell sees it. Create the item on web01: Name “Age of last backup”, Type Zabbix agent, key file.age[/var/backups/pos/latest.tar.gz], Numeric (unsigned), Units s, interval 5m.

✅ Checkpoint: file.age[/etc/hostname] returns a number of seconds.

4Call a script for the POS queue

Real checks are usually a script. Here a “billing queue” is a folder of .txn files that a POS sync job should empty:

Run this on web01:

sudo mkdir -p /var/spool/pos /etc/zabbix/scripts
sudo touch /var/spool/pos/{1001,1002,1003}.txn
sudo tee /etc/zabbix/scripts/pos_queue.sh >/dev/null <<'EOF'
#!/bin/bash
# number of unsent POS transactions
find /var/spool/pos -maxdepth 1 -name '*.txn' | wc -l
EOF
sudo chmod 755 /etc/zabbix/scripts/pos_queue.sh
echo 'UserParameter=pos.queue,/etc/zabbix/scripts/pos_queue.sh' | sudo tee -a /etc/zabbix/zabbix_agent2.d/userparameter_lab.conf
sudo zabbix_agent2 -R userparameter_reload
zabbix_get -s 192.168.56.81 --tls-connect psk --tls-psk-identity "PSK web01" --tls-psk-file /tmp/web01.psk -k pos.queue      # on zbx01

You should see:

UserParameter=pos.queue,/etc/zabbix/scripts/pos_queue.sh
User parameters reloaded
3

Add an item pos.queue (Zabbix agent, Numeric unsigned, 1m) and a trigger min(/web01/pos.queue,15m)>50 named “POS queue stuck on {HOST.NAME}”. Using min() over 15 minutes means the queue must stay high, so a normal burst at closing time doesn’t page anyone.

✅ Checkpoint: pos.queue returns 3 and the item and trigger exist on web01.

5Allow one system.run command, safely

system.run lets the server ask the agent to run any command written in the key, which is why it’s denied by default. If you need one specific command, allow only that pattern:

Run this on web01, then the tests on zbx01:

echo 'AllowKey=system.run[uptime -p]' | sudo tee /etc/zabbix/zabbix_agent2.d/allowkey.conf
sudo systemctl restart zabbix-agent2
zabbix_get -s 192.168.56.81 --tls-connect psk --tls-psk-identity "PSK web01" --tls-psk-file /tmp/web01.psk -k 'system.run[uptime -p]'       # on zbx01
zabbix_get -s 192.168.56.81 --tls-connect psk --tls-psk-identity "PSK web01" --tls-psk-file /tmp/web01.psk -k 'system.run[cat /etc/shadow]' # on zbx01

You should see this (your uptime will differ):

AllowKey=system.run[uptime -p]
up 2 hours, 14 minutes
ZBX_NOTSUPPORTED: Unsupported item key.

The exact allowed command works, and anything else is refused with the same message as an unknown key, so an attacker learns nothing. Rules are checked in order, and once you add any rule for system.run the automatic DenyKey=system.run[*] is applied after yours. Prefer a named UserParameter over system.run whenever you can, because the command then lives on the host, not in a key anyone with frontend rights can edit.

✅ Checkpoint: system.run[uptime -p] works from zbx01 and system.run[cat /etc/shadow] is refused.

6Create trapper items and widen Allowed hosts

Create three items on web01 with Type Zabbix trapper:

Look at Allowed hosts on each: in 8.0 it’s pre-filled with {$TRAPPER.ALLOWED_HOSTS}, a global macro (Administration > Macros) whose default is 127.0.0.1,::1. web01 sends from 192.168.56.81, so change Allowed hosts on these three items to 192.168.56.81. In 7.0 the field was empty, which meant any host could send.

In 8.0 an empty field rejects every sender, and an upgrade from 7.0 rewrites empty fields to 0.0.0.0/0,::/0 so old senders keep working.

✅ Checkpoint: Three trapper items exist on web01 with Allowed hosts 192.168.56.81.

7Send values from a backup script

Run this on web01:

sudo apt install -y zabbix-sender
sudo mkdir -p /var/backups/pos
sudo tee /usr/local/bin/pos-backup.sh >/dev/null <<'EOF'
#!/bin/bash
# nightly POS backup, then report the result to Zabbix
START=$(date +%s)
DEST=/var/backups/pos/pos-$(date +%F).tar.gz
if tar -czf "$DEST" -C /var/spool/pos . 2>/dev/null; then STATUS=0; else STATUS=1; fi
ln -sf "$DEST" /var/backups/pos/latest.tar.gz
SIZE=$(stat -c %s "$DEST" 2>/dev/null || echo 0)
DUR=$(( $(date +%s) - START ))
zabbix_sender -z 192.168.56.80 -s web01 -i - <<END
web01 backup.status $STATUS
web01 backup.size $SIZE
web01 backup.duration $DUR
END
EOF
sudo chmod 755 /usr/local/bin/pos-backup.sh
sudo /usr/local/bin/pos-backup.sh; echo "exit code: $?"

You should see this (seconds spent will differ):

Response from "192.168.56.80:10051": "processed: 3; failed: 0; total: 3; seconds spent: 0.000016"
sent: 3; skipped: 0; total: 3
exit code: 0

That’s the real sender output format. “processed: 3; failed: 0” means the server accepted all three values; “sent” only counts what left the client. Add the trigger last(/web01/backup.status)<>0 or nodata(/web01/backup.status,26h)=1 named “POS backup failed or did not run on {HOST.NAME}”, and schedule the script with echo '30 1 * * * root /usr/local/bin/pos-backup.sh' | sudo tee /etc/cron.d/pos-backup.

✅ Checkpoint: Latest data on web01 shows backup.status 0, a backup size and a duration.

8A calculated item from what you have

Create an item on web01: Name “Backup size as % of root disk”, Type Calculated, key backup.size.pct, Formula 100*last(//backup.size)/last(//vfs.fs.size[/,total]), Numeric (float), Units %, interval 1h.

The formula needs an item with the key vfs.fs.size[/,total] on the host. The Linux template discovers file systems with keys like vfs.fs.dependent.size[/,total], so if your host has no exact match, add a small Zabbix agent (active) item with that key first. Calculated items run on the server, so they also work for hosts without an agent.

✅ Checkpoint: The calculated item shows a small percentage in Latest data.