Zabbix in Grafana, and Prometheus exporters in Zabbix

Grafana runs on graf01 (192.168.56.70) if you did the Grafana course; otherwise install it on zbx01 in step 1 and use http://zbx01.lab:3000 wherever the lesson says graf01. node_exporter runs on web01 (192.168.56.81). The Zabbix side was tested on a real 8.0.0 server with node_exporter 1.12.1; Grafana screens are described from plugin 6.10.2 and Grafana 13.3.

1Have Grafana ready

If you have graf01, run this on graf01:

grafana server -v
echo '192.168.56.80 zbx01 zbx01.lab' | sudo tee -a /etc/hosts
curl -sS -o /dev/null -w '%{http_code}\n' -k https://zbx01.lab/api_jsonrpc.php

You should see this on Grafana 13.3.0:

Version 13.3.0 (commit: 752e3462ff71a43a5c1fb62d00a482fa2dc0ee6a, branch: HEAD)
192.168.56.80 zbx01 zbx01.lab
200

The 200 means graf01 can reach the Zabbix frontend over HTTPS through zbx01’s firewall (Lesson 3 allowed 443 from the lab subnet). The -k is only for this reachability test; Grafana will trust the real certificate in step 5.

No graf01? Install Grafana OSS on zbx01 instead:

sudo apt install -y apt-transport-https wget
sudo mkdir -p /etc/apt/keyrings
wget -q -O - https://apt.grafana.com/gpg-full.key | sudo tee /etc/apt/keyrings/grafana.asc >/dev/null
echo "deb [signed-by=/etc/apt/keyrings/grafana.asc] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt update && sudo apt install -y grafana
sudo systemctl enable --now grafana-server
sudo ufw allow from 192.168.56.0/24 to any port 3000 proto tcp

✅ Checkpoint: Grafana 13 is running (graf01 or zbx01) and that machine can reach https://zbx01.lab/api_jsonrpc.php.

2Install and enable the Zabbix app

Run this on the Grafana machine:

sudo grafana cli plugins install alexanderzobnin-zabbix-app
sudo systemctl restart grafana-server
sudo ls /var/lib/grafana/plugins/

You should see this (the plugin version may be newer):

✔ Downloaded and extracted alexanderzobnin-zabbix-app v6.10.2 zip successfully to /var/lib/grafana/plugins/alexanderzobnin-zabbix-app

Please restart Grafana after installing or removing plugins. Refer to Grafana documentation for instructions if necessary.

alexanderzobnin-zabbix-app

On Grafana 13 use grafana cli (two words); the old standalone grafana-cli binary is deprecated. Now open Grafana as admin, go to Administration > Plugins and data > Plugins, search Zabbix, open it and click Enable. Installing alone isn’t enough for an app plugin.

✅ Checkpoint: The Zabbix plugin page in Grafana shows version 6.10.x and a Disable button (meaning it is enabled).

3Create a Grafana role and token in Zabbix

Grafana needs more methods than the api-bot role from Lesson 22, mainly history and trends. In Zabbix go to Users > User roles > Create user role: name Grafana read-only, User type User, no UI elements or actions, Access to API Enabled with an Allow list:

hostgroup.get  host.get  item.get  history.get  trend.get  problem.get  trigger.get
event.get  valuemap.get  proxy.get  usermacro.get  service.get  sla.get

Create user grafana-bot in group API bots (Lab/Web Read, frontend access disabled) with this role, then Users > API tokens > Create API token for grafana-bot with a 90-day expiry. Copy the token. If you plan to graph web01-snmp or web01-prom too, give API bots Read on Lab/Network and add web01-prom to Lab/Web in step 8.

✅ Checkpoint: Zabbix has role “Grafana read-only”, user grafana-bot and a token you have copied.

4Copy the Zabbix CA certificate to the Grafana machine

Run this on zbx01:

cat /etc/ssl/certs/zbx01.lab.crt

Copy the whole block, from -----BEGIN CERTIFICATE----- to the END line. Grafana will use it to trust zbx01’s self-signed certificate instead of skipping TLS checks.

✅ Checkpoint: You have the PEM text of zbx01.lab.crt ready to paste.

5Add the Zabbix data source

In Grafana open Connections > Add new connection, search Zabbix, click it and then Add new data source. Fill in:

Click Save & test.

You should see:

Zabbix API version 8.0.0

The test calls hostgroup.get and apiinfo.version, so a green result proves the URL, TLS and token work. It does not prove the token may read history, which is why step 6 matters.

✅ Checkpoint: Save & test shows “Zabbix API version 8.0.0”.

6Build a dashboard: Metrics and Problems

Create a new dashboard, add a Time series visualization with data source Zabbix lab. Set Query type Metrics, Group Lab/Web, Host web01, Item /CPU utilization|Load average \(1m avg\)/. Add a second panel of type Zabbix Problems with Query type Problems, Group Lab/Web, Show Problems.

Save the dashboard as Zabbix lab overview.

Regex filters are wrapped in slashes, and brackets must be escaped. The Problems panel can acknowledge problems from Grafana if the user’s role allows it; our grafana-bot can’t, which is what you want for a shared TV screen.

The data source also has a Dashboards tab with three bundled dashboards: Zabbix System Status, Zabbix Template Linux Server and Zabbix Server Dashboard. Import them for ideas, but on 8.0 some panels stay empty because they use old item names, for example Zabbix server: Queue (now Queue) and Processor load (15 min average per core) (now Load average (15m avg)); edit those queries to the current names.

✅ Checkpoint: The dashboard shows web01 CPU and load lines and the current Lab/Web problems.

7Run node_exporter on web01

Paste this whole block on web01:

VER=1.12.1
cd /tmp && wget -q https://github.com/prometheus/node_exporter/releases/download/v$VER/node_exporter-$VER.linux-amd64.tar.gz
tar xzf node_exporter-$VER.linux-amd64.tar.gz
sudo install -m 755 node_exporter-$VER.linux-amd64/node_exporter /usr/local/bin/
sudo useradd --system --no-create-home --shell /usr/sbin/nologin node_exporter 2>/dev/null
sudo tee /etc/systemd/system/node_exporter.service >/dev/null <<'EOF'
[Unit]
Description=Prometheus node_exporter
After=network-online.target
[Service]
User=node_exporter
ExecStart=/usr/local/bin/node_exporter --web.listen-address=:9100
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload && sudo systemctl enable --now node_exporter
sudo ufw allow from 192.168.56.80 to any port 9100 proto tcp
curl -s localhost:9100/metrics | grep -E '^node_load1 |^node_exporter_build_info'

You should see this (your load value will differ):

node_exporter_build_info{branch="HEAD",goarch="amd64",goos="linux",goversion="go1.26.5",revision="6044da783597cc3b57aef7580ddcdcff58a4ee99",tags="unknown",version="1.12.1"} 1
node_load1 0.1

This is the same node_exporter setup as the Prometheus course; if you already have it on web01, skip to the ufw line. Only zbx01 may reach port 9100.

✅ Checkpoint: curl on web01 shows node_load1 and node_exporter_build_info with version 1.12.1.

8Scrape it from Zabbix with one HTTP agent item

In Zabbix create host web01-prom in group Lab/Web, monitored by Server, no interface needed, with host macros {$NODE_EXPORTER_HOST} = 192.168.56.81 and {$NODE_EXPORTER_PORT} = 9100. Add an item:

Click Test, then Get value: you get the whole metrics page (about a thousand lines). Now add three Dependent items with master item “node_exporter metrics (raw)” and one preprocessing step each:

After two minutes, Latest data for web01-prom shows (your values will differ):

One HTTP request per minute feeds all three items, the same master-and-dependent idea as Lesson 11 and SNMP walk[] in Lesson 20. Keep the raw item’s history short (1h), because the page is big text.

✅ Checkpoint: Latest data shows Free space on / and Load average 1m for web01-prom.

9Discover filesystems with Prometheus to JSON

On web01-prom create a Discovery rule: Name Filesystems (prom), Type Dependent item, Key lab.fs.discovery, Master item the raw item. Preprocessing: Prometheus to JSON with node_filesystem_avail_bytes{fstype=~"ext4|xfs"}. On the LLD macros tab add {#MOUNT} = $.labels.mountpoint and {#FSTYPE} = $.labels.fstype.

Test the preprocessing:

[{"name":"node_filesystem_avail_bytes","value":"1.7041027072e+10","line_raw":"node_filesystem_avail_bytes{device=\"/dev/sda2\",device_error=\"\",fstype=\"ext4\",mountpoint=\"/\"} 1.7041027072e+10","labels":{"device":"/dev/sda2","device_error":"","fstype":"ext4","mountpoint":"/"},"type":"gauge","help":"Filesystem space available to non-root users in bytes."}, ...]

Each matching line becomes an object with its name, value, labels, type and help text, and the LLD macro paths pick labels out of it. Add an item prototype: Name Free space on {#MOUNT}, Key lab.fs.avail[{#MOUNT}], Dependent item on the raw item, Numeric (unsigned), units B, preprocessing Prometheus pattern node_filesystem_avail_bytes{mountpoint="{#MOUNT}"}, value.

After the next discovery, Latest data shows one item per ext4 or xfs mount (your sizes will differ):

Free space on /        15.87 GB
Free space on /boot    1.62 GB

The shipped template “Linux by Prom” does all of this (and much more) with the same two macros, so at work you’d link it instead of building items by hand. Building one yourself is how you learn to adapt it to an application exporter that has no template.

✅ Checkpoint: web01-prom has discovered “Free space on” items for each ext4 or xfs mount.