Low-level discovery: custom LLD rules and prototypes

Agent work happens on web01 and frontend work at https://zbx01.lab. web01 must be linked to Linux by Zabbix agent active (Lesson 4). You’ll add a second disk-like mount with a loop file so discovery has something new to find.

1Look at the raw discovery JSON

Run this on web01:

sudo zabbix_agent2 -t 'vfs.fs.discovery' | head -c 400; echo

You should see the start of the JSON (your mount list will differ):

vfs.fs.discovery                              [s|[{"{#FSNAME}":"/proc","{#FSTYPE}":"proc","{#FSOPTIONS}":"rw,relatime"},{"{#FSNAME}":"/sys","{#FSTYPE}":"sysfs","{#FSOPTIONS}":"rw,relatime"},{"{#FSNAME}":"/dev","{#FSTYPE}":"devtmpfs","{#FSOPTIONS}":"rw,relatime,size=4097460k,nr_inodes=1024365,mode=755"},

That’s real agent 2 output: one object per mount, with field names already in LLD macro form. The template uses the newer vfs.fs.get, whose objects have plain field names (fsname, fstype, bytes, inodes) and need LLD macro paths, but the idea is the same.

✅ Checkpoint: You can see the JSON array of mounts and recognise {#FSNAME} and {#FSTYPE}.

2Add a new mount and let discovery find it

Run this on web01:

sudo fallocate -l 200M /srv/data.img
sudo mkfs.ext4 -q /srv/data.img
sudo mkdir -p /data && sudo mount -o loop /srv/data.img /data
df -h /data

You should see:

Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      172M   24K  158M   1% /data

Discovery runs every hour, so speed it up: in Data collection > Hosts > web01 > Discovery, tick “Mounted filesystem discovery” and press Execute now. Within a minute or two, Latest data for web01 shows new items for /data.

Latest data (filter “FS [/data]”) then shows the new items:

✅ Checkpoint: Items for FS [/data] appear on web01 without you creating any of them.

3Set a per-mount threshold with a macro context

Say /data may run fuller than the rest. On web01 open the Macros tab and add two host macros: {$VFS.FS.PUSED.MAX.WARN:"/data"} with value 95 and {$VFS.FS.PUSED.MAX.CRIT:"/data"} with value 98.

You need both, because the template’s “Space is critically low” trigger fires above 90% by default and the “Space is low” warning depends on it. Then fill /data past 80% and watch which trigger fires:

Run this on web01:

sudo fallocate -l 140M /data/fill.bin
df -h /data

You should see:

Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      172M  141M   18M  90% /data

Zabbix calculates about 89% here (df rounds up).

The general rule would fire “Space is low” (warn at 80), but /data’s own macro says 95, so no problem opens for /data while / still uses 80. Write a little more (sudo fallocate -l 10M /data/more.bin) to reach about 95.4%, and after five minutes (the trigger uses min(...,5m)) the warning appears for /data only. Context macros are how one template serves hundreds of mounts with different limits.

✅ Checkpoint: At about 90% no “Space is low” problem exists for /data; above 95% it does.

4Write a custom discovery script

Now your own rule: discover the services listed in a file and check each one is running. The script prints LLD JSON.

Run this on web01:

printf 'nginx\ncron\nssh\nzabbix-agent2\n' | sudo tee /etc/zabbix/watch-services.txt
sudo tee /etc/zabbix/scripts/svc_discovery.sh >/dev/null <<'EOF'
#!/bin/bash
# LLD: one entry per service name in watch-services.txt
LIST=/etc/zabbix/watch-services.txt
first=1
printf '['
while read -r svc; do
    [ -z "$svc" ] && continue
    case "$svc" in \#*) continue ;; esac
    [ $first -eq 0 ] && printf ','
    printf '{"{#SERVICE}":"%s"}' "$svc"
    first=0
done < "$LIST"
printf ']\n'
EOF
sudo chmod 755 /etc/zabbix/scripts/svc_discovery.sh
sudo -u zabbix /etc/zabbix/scripts/svc_discovery.sh

You should see:

nginx
cron
ssh
zabbix-agent2
[{"{#SERVICE}":"nginx"},{"{#SERVICE}":"cron"},{"{#SERVICE}":"ssh"},{"{#SERVICE}":"zabbix-agent2"}]

The first four lines are tee echoing the list; the last line is the discovery JSON, which I checked with Zabbix 8.0’s own JSON parser. Now register two keys, one for discovery and one flexible key to check a service:

Run this on web01:

sudo tee /etc/zabbix/zabbix_agent2.d/userparameter_lld.conf >/dev/null <<'EOF'
UserParameter=svc.discovery,/etc/zabbix/scripts/svc_discovery.sh
UserParameter=svc.active[*],systemctl is-active --quiet "$1" && echo 1 || echo 0
EOF
sudo zabbix_agent2 -R userparameter_reload
sudo zabbix_agent2 -t 'svc.active[nginx]'

You should see:

User parameters reloaded
svc.active[nginx]                             [s|1]

✅ Checkpoint: svc_discovery.sh prints valid JSON as the zabbix user, and svc.active[nginx] returns 1.

5Create the discovery rule and prototypes

On web01 (or better, in the SaatVedha Nginx web template) open Discovery > Create discovery rule: Name Watched services, Type Zabbix agent (active), Key svc.discovery, Update interval 10m, Delete lost resources After 1d, Disable lost resources Immediately. Then add prototypes:

Press Execute now on the rule. The trigger uses max(...,3m)=0 so a service must be down for three minutes in a row, which ignores a quick restart.

Latest data, filtered by the tag service, shows:

✅ Checkpoint: Four “Service …: running” items exist on web01, all created by discovery.

6Add and remove entries, watch lost resources

Run this on web01:

echo 'chrony' | sudo tee -a /etc/zabbix/watch-services.txt
sudo sed -i '/^cron$/d' /etc/zabbix/watch-services.txt
/etc/zabbix/scripts/svc_discovery.sh

You should see:

chrony
[{"{#SERVICE}":"nginx"},{"{#SERVICE}":"ssh"},{"{#SERVICE}":"zabbix-agent2"},{"{#SERVICE}":"chrony"}]

Execute the rule again. A new item for chrony appears, and the cron item shows a yellow info icon in the item list saying it’s no longer discovered and will be deleted in about a day; with “Disable immediately” it also stops collecting at once. If chrony isn’t installed, its item reports 0 and its trigger fires after three minutes, which is correct.

✅ Checkpoint: chrony has an item, and the cron item is disabled with a “no longer discovered” note.

7Use an override to skip a trigger for one service

Suppose ssh is managed by another team and you only want its item, not its trigger. In the rule’s Overrides tab add an override named No trigger for ssh: Filter {#SERVICE} matches ^ssh$, Operation: Object Trigger prototype, Condition name contains is not running, action Create enabled: No (or Discover: No). Execute the rule.

The ssh item stays, but its trigger is removed or created disabled. Overrides keep one discovery rule for everything instead of copying rules per team.

✅ Checkpoint: After execution, web01 has no enabled “Service ssh is not running” trigger, but still has the ssh item.