Monitoring with Zabbix: agent 2, server and HA proxies

Step 1 runs on the laptop, steps 2 to 5 on ubu01, step 6 on rocky01, step 7 on kafka01, and steps 8 and 9 move between them; each step says where. The Zabbix web pages are reached from the laptop’s browser through a port forward.

1Plan the RAM and take snapshots

On the laptop, start the three VMs and snapshot them:

VBoxManage startvm ubu01 rocky01 kafka01 --type=headless
VBoxManage list runningvms
for vm in ubu01 rocky01 kafka01; do VBoxManage snapshot $vm take before-l61; done

You should see something like this (snapshot progress cut):

"rocky01" {6b0f4c1e-2d3a-4b8e-9f10-7c2d5e8a1b34}
"ubu01" {a3e9d7c2-5f1b-4e6a-8d20-1b9c4f7e2a56}
"kafka01" {e1c84b6a-9d2f-4a37-b5e1-3f6a2c8d9e70}

Together they use 8 GB (2 + 2 + 4), which is comfortable on a 16 GB laptop. On an 8 GB laptop, first stop the Kafka cluster on kafka01 (sudo systemctl stop kafka@1 kafka@2 kafka@3), shut kafka01 down and lower it to 2048 MB with VBoxManage modifyvm kafka01 --memory=2048; the Kafka items will then show “down”, which is still a useful test. Set it back to 4096 MB before Lesson 62. ubu01 uses ad01 as its first DNS server since Lesson 54, so if ad01 is off, apt waits a few seconds on every lookup before falling back to 10.10.10.1; that is slow but works.

✅ Checkpoint: VBoxManage list runningvms shows ubu01, rocky01 and kafka01, and each has a before-l61 snapshot.

2Install PostgreSQL and Zabbix 7.0 on ubu01

On ubu01, add the Zabbix 7.0 repository for Ubuntu 26.04 and install the server, frontend and agent:

sudo apt update
sudo apt install -y postgresql
cd /tmp
curl -fLO https://repo.zabbix.com/zabbix/7.0/ubuntu/pool/main/z/zabbix-release/zabbix-release_latest_7.0+ubuntu26.04_all.deb
sudo dpkg -i zabbix-release_latest_7.0+ubuntu26.04_all.deb
sudo apt update
sudo apt install -y zabbix-server-pgsql zabbix-frontend-php php-pgsql zabbix-nginx-conf zabbix-sql-scripts zabbix-agent2 zabbix-get
apt-cache policy zabbix-server-pgsql | head -n 3
psql --version
ls /etc/php

You should see something like this (apt output cut):

zabbix-server-pgsql:
  Installed: 1:7.0.32-1+ubuntu26.04
  Candidate: 1:7.0.32-1+ubuntu26.04
psql (PostgreSQL) 18.6 (Ubuntu 18.6-0ubuntu0.26.04.1)
8.5

The release package adds Zabbix’s apt source and signing key, so the server comes from repo.zabbix.com rather than Ubuntu’s archive. If the download returns 404, copy the current link from zabbix.com/download for Zabbix 7.0 LTS on Ubuntu 26.04; the file name changes now and then. “8.5” is the PHP version on Ubuntu 26.04, which names the PHP-FPM service in the next step.

✅ Checkpoint: apt-cache policy shows a 1:7.0.x version installed from the Zabbix repository and psql reports PostgreSQL 18.

3Create the database, start the server and open ufw

Paste this whole block on ubu01:

sudo -u postgres psql -c "CREATE USER zabbix WITH PASSWORD 'Lab-Only-Pass-61';"
sudo -u postgres createdb -O zabbix zabbix
SQL=$(dpkg -L zabbix-sql-scripts | grep 'postgresql/server.sql.gz$')
echo "$SQL"
zcat "$SQL" | sudo -u zabbix psql -q zabbix >/dev/null
sudo -u zabbix psql -d zabbix -tAc 'SELECT count(*) FROM users;'
sudo sed -i 's/^# DBPassword=$/DBPassword=Lab-Only-Pass-61/' /etc/zabbix/zabbix_server.conf
sudo sed -i -E -e 's/^#\s*listen\s+8080;/        listen          8080;/' \
  -e 's/^#\s*server_name\s+example\.com;/        server_name     ubu01.saatlab.test;/' /etc/zabbix/nginx.conf
sudo grep -E '^DB(Name|User|Password)=' /etc/zabbix/zabbix_server.conf
grep -nE '^\s*(listen|server_name)' /etc/zabbix/nginx.conf
sudo nginx -t
sudo systemctl restart zabbix-server nginx php8.5-fpm
sudo systemctl enable zabbix-server nginx php8.5-fpm
sudo ufw allow from 10.10.10.0/24 to any port 8080 proto tcp
sudo ufw allow from 10.10.10.11 to any port 10051 proto tcp
sudo ufw allow from 10.10.10.31 to any port 10051 proto tcp
sudo grep 'server #0 started' /var/log/zabbix/zabbix_server.log

You should see something like this:

CREATE ROLE
/usr/share/zabbix-sql-scripts/postgresql/server.sql.gz
2
DBName=zabbix
DBUser=zabbix
DBPassword=Lab-Only-Pass-61
2:        listen          8080;
3:        server_name     ubu01.saatlab.test;
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
Created symlink '/etc/systemd/system/multi-user.target.wants/zabbix-server.service' → '/usr/lib/systemd/system/zabbix-server.service'.
Rule added
Rule added
Rule added
  3418:20261027:143011.906 server #0 started [main process]

The schema import takes a minute or so; the 2 is the two built-in users (Admin and guest), proof the data loaded. The schema path differs between package builds, which is why dpkg finds it for you. The server reaches PostgreSQL over TCP on localhost with the password; the import used the zabbix OS user through peer authentication.

Port 8080 is used because Ubuntu’s default Nginx site already owns 80. The three ufw rules are the only openings ubu01 needs for this lesson.

✅ Checkpoint: The log has “server #0 started” and sudo ufw status lists 8080 from 10.10.10.0/24 and 10051 from 10.10.10.11 and 10.10.10.31.

4Finish the web installer and set up ubu01’s own agent

On the laptop, forward a port to the frontend (adjust for macOS/Linux shells as in Lesson 2):

VBoxManage natnetwork modify --netname=LinuxLab --port-forward-4="zabbix-web:tcp:[127.0.0.1]:8021:[10.10.10.21]:8080"

Open http://127.0.0.1:8021/ in your browser and follow the installer: at “Configure DB connection” pick PostgreSQL, host localhost, database zabbix, user zabbix, password Lab-Only-Pass-61; at “Settings” name the server SaatLab Zabbix and pick Asia/Kolkata. Log in as Admin with the password zabbix, and change it at once under User settings. Then open Data collection, Hosts, click “Zabbix server” and change its Host name to ubu01.

[Add your screenshot here]: the Zabbix 7.0 installer at the “Configure DB connection” page with PostgreSQL, localhost and database zabbix filled in.

Back on ubu01, give the agent the same name and test it:

sudo sed -i 's/^Hostname=Zabbix server$/Hostname=ubu01/' /etc/zabbix/zabbix_agent2.conf
sudo systemctl restart zabbix-agent2
sudo systemctl enable zabbix-agent2
zabbix_get -s 127.0.0.1 -k agent.hostname
zabbix_get -s 127.0.0.1 -k 'vfs.file.contents[/proc/sys/kernel/osrelease]'

You should see something like this:

ubu01
7.0.0-15-generic

The default host for the server itself watches 127.0.0.1:10050 with the “Linux by Zabbix agent” template, so ubu01 is monitored directly by the server, not through a proxy. Server= and ServerActive= stay at 127.0.0.1 here. Within a minute or two, the ZBX icon for ubu01 under Monitoring, Hosts turns green.

✅ Checkpoint: zabbix_get answers ubu01 and the kernel string, and the frontend shows host ubu01 with a green ZBX icon.

5Create the proxy group and the two proxies

In the frontend, open Administration, Proxy groups, Create proxy group, then Administration, Proxies, Create proxy (twice):

Then on ubu01, check that the server is listening for them and ufw lets them in:

ss -ltn '( sport = :10051 )'
sudo ufw status | grep 10051

You should see something like this:

State  Recv-Q Send-Q Local Address:Port  Peer Address:PortProcess
LISTEN 0      4096         0.0.0.0:10051      0.0.0.0:*
LISTEN 0      4096            [::]:10051         [::]:*
10051/tcp                  ALLOW       10.10.10.11
10051/tcp                  ALLOW       10.10.10.31

The proxy names are deliberately different from the VM names, so you can tell a proxy (proxy-rocky01) from the host it runs on (rocky01). “Address for active agents” is where the group sends redirected agents. “Proxy address” limits which IP may claim to be that proxy.

Both proxies show “Unknown” state and “Never” seen until they connect.

✅ Checkpoint: Administration, Proxies lists proxy-rocky01 and proxy-kafka01 in group lab-proxies, and the server listens on 10051.

6Install the proxy and agent 2 on rocky01

On rocky01, paste this whole block (the first line keeps EPEL’s Zabbix packages out of the way):

sudo dnf config-manager --save --setopt=epel.excludepkgs='zabbix*'
sudo rpm -Uvh https://repo.zabbix.com/zabbix/7.0/rocky/10/x86_64/zabbix-release-latest-7.0.el10.noarch.rpm
sudo dnf clean all
sudo dnf install -y zabbix-proxy-sqlite3 zabbix-selinux-policy zabbix-agent2 zabbix-get
sudo install -d -o zabbix -g zabbix -m 750 /var/lib/zabbix
sudo restorecon -R /var/lib/zabbix
sudo sed -i -e 's/^Server=127.0.0.1$/Server=10.10.10.21/' -e 's/^Hostname=Zabbix proxy$/Hostname=proxy-rocky01/' \
  -e 's#^DBName=zabbix_proxy$#DBName=/var/lib/zabbix/zabbix_proxy.db#' /etc/zabbix/zabbix_proxy.conf
sudo sed -i -e 's/^Server=127.0.0.1$/Server=10.10.10.11,10.10.10.31/' \
  -e 's/^ServerActive=127.0.0.1$/ServerActive=10.10.10.11;10.10.10.31/' \
  -e 's/^Hostname=Zabbix server$/Hostname=rocky01/' /etc/zabbix/zabbix_agent2.conf
sudo grep -E '^(Server|Hostname|DBName)=' /etc/zabbix/zabbix_proxy.conf
sudo grep -E '^(Server|ServerActive|Hostname)=' /etc/zabbix/zabbix_agent2.conf
sudo setsebool -P zabbix_can_network on
getsebool zabbix_can_network
sudo firewall-cmd --permanent --add-port={10050,10051}/tcp
sudo firewall-cmd --reload
sudo systemctl enable --now zabbix-proxy zabbix-agent2
sleep 20
sudo grep -E 'creating database|proxy #0 started|received configuration data' /var/log/zabbix/zabbix_proxy.log

You should see something like this (dnf output cut):

Server=10.10.10.21
Hostname=proxy-rocky01
DBName=/var/lib/zabbix/zabbix_proxy.db
Server=10.10.10.11,10.10.10.31
ServerActive=10.10.10.11;10.10.10.31
Hostname=rocky01
zabbix_can_network --> on
success
success
Created symlink '/etc/systemd/system/multi-user.target.wants/zabbix-proxy.service' → '/usr/lib/systemd/system/zabbix-proxy.service'.
Created symlink '/etc/systemd/system/multi-user.target.wants/zabbix-agent2.service' → '/usr/lib/systemd/system/zabbix-agent2.service'.
  2871:20261027:143522.104 creating database ...
  2871:20261027:143522.688 proxy #0 started [main process]
  2896:20261027:143523.311 received configuration data from server at "10.10.10.21", datalen 6871

With SQLite the proxy creates its own database file on first start, so there is no schema to import. ProxyMode isn’t set, so the proxy is active. “received configuration data” is the line to look for: it means the server recognised the name proxy-rocky01 and sent its configuration.

The proxy’s connection to 10.10.10.21:10051 is what zabbix_can_network allows; setsebool -P takes a little while because it rebuilds the policy. If the rpm URL returns 404, take the current one from zabbix.com/download for Zabbix 7.0 on Rocky Linux 10.

✅ Checkpoint: The proxy log shows “received configuration data from server”, and proxy-rocky01 is Online in Administration, Proxies.

7Install the second proxy and agent 2 on kafka01

On kafka01 (no EPEL here, so no exclude line), paste this whole block:

sudo rpm -Uvh https://repo.zabbix.com/zabbix/7.0/rocky/10/x86_64/zabbix-release-latest-7.0.el10.noarch.rpm
sudo dnf clean all
sudo dnf install -y zabbix-proxy-sqlite3 zabbix-selinux-policy zabbix-agent2 zabbix-get
sudo install -d -o zabbix -g zabbix -m 750 /var/lib/zabbix
sudo restorecon -R /var/lib/zabbix
sudo sed -i -e 's/^Server=127.0.0.1$/Server=10.10.10.21/' -e 's/^Hostname=Zabbix proxy$/Hostname=proxy-kafka01/' \
  -e 's#^DBName=zabbix_proxy$#DBName=/var/lib/zabbix/zabbix_proxy.db#' /etc/zabbix/zabbix_proxy.conf
sudo sed -i -e 's/^Server=127.0.0.1$/Server=10.10.10.11,10.10.10.31/' \
  -e 's/^ServerActive=127.0.0.1$/ServerActive=10.10.10.11;10.10.10.31/' \
  -e 's/^Hostname=Zabbix server$/Hostname=kafka01/' /etc/zabbix/zabbix_agent2.conf
sudo grep -E '^(Server|Hostname|DBName)=' /etc/zabbix/zabbix_proxy.conf
sudo grep -E '^(Server|ServerActive|Hostname)=' /etc/zabbix/zabbix_agent2.conf
sudo setsebool -P zabbix_can_network on
getsebool zabbix_can_network
sudo firewall-cmd --permanent --add-port={10050,10051}/tcp
sudo firewall-cmd --reload
sudo systemctl enable --now zabbix-proxy zabbix-agent2
sleep 20
sudo grep -E 'creating database|proxy #0 started|received configuration data' /var/log/zabbix/zabbix_proxy.log
sudo firewall-cmd --list-ports

You should see something like this:

Server=10.10.10.21
Hostname=proxy-kafka01
DBName=/var/lib/zabbix/zabbix_proxy.db
Server=10.10.10.11,10.10.10.31
ServerActive=10.10.10.11;10.10.10.31
Hostname=kafka01
zabbix_can_network --> on
success
success
  5530:20261027:144107.420 creating database ...
  5530:20261027:144107.977 proxy #0 started [main process]
  5554:20261027:144108.602 received configuration data from server at "10.10.10.21", datalen 6871
9094/tcp 9194/tcp 9294/tcp 10050/tcp 10051/tcp

Running a proxy on the Kafka VM is a lab shortcut to get two proxies without a fourth VM; at work, proxies get their own small servers. kafka01’s firewall now has the three SASL_SSL ports from Lesson 60 plus the two Zabbix ports. Until the next step the agents log “host [rocky01] not found” and “host [kafka01] not found”, because those hosts don’t exist in the frontend yet.

✅ Checkpoint: Both proxies show Online in Administration, Proxies, each with Version 7.0.32 and a “Last seen” of a few seconds.

8Add rocky01 and kafka01 to the proxy group

In Data collection, Hosts, create host rocky01: template “Linux by Zabbix agent active”, host group “Linux servers”, an Agent interface 10.10.10.11 port 10050, and Monitored by “Proxy group” lab-proxies. Create kafka01 the same way with 10.10.10.31, then add two items to kafka01, both of type “Zabbix agent (active)” with a 1m interval: “Kafka node 1 service” with key systemd.unit.info[kafka@1.service] and “Kafka SASL_SSL port 9094” with key net.tcp.listen[9094].

[Add your screenshot here]: Administration, Proxies with proxy-rocky01 and proxy-kafka01 Online in group lab-proxies, one host each.

On rocky01, ask kafka01’s agent directly, the same way a proxy runs a passive check:

zabbix_get -s 10.10.10.31 -k agent.hostname
zabbix_get -s 10.10.10.31 -k 'vfs.file.contents[/proc/sys/kernel/osrelease]'
zabbix_get -s 10.10.10.31 -k 'systemd.unit.info[kafka@1.service]'
zabbix_get -s 10.10.10.31 -k 'net.tcp.listen[9094]'

You should see this:

kafka01
6.12.0-211.16.1.el10_2.0.1.x86_64
active
1

systemd.unit.info comes from agent 2’s systemd plugin and returns the ActiveState of the unit, and net.tcp.listen[9094] returns 1 while the SASL_SSL listener is up. The server shares the two hosts between the proxies, so Administration, Proxies shows one host each, and Monitoring, Latest data for kafka01 fills in within a couple of minutes. To watch Kafka’s open files against its limit from Lesson 58, teams add a UserParameter that reads /proc through a narrow sudo rule, or use Kafka’s JMX metrics; the agent itself can’t read another user’s /proc/PID/fd.

✅ Checkpoint: zabbix_get returns kafka01, the kernel string, active and 1, and Latest data shows values for both new hosts.

9Fail a proxy and watch the group take over

On rocky01, stop its proxy. On ubu01, wait past the 1-minute failover period and read the server log:

sudo systemctl stop zabbix-proxy                     # on rocky01
sleep 90; sudo grep 'changed state' /var/log/zabbix/zabbix_server.log | tail -n 1   # on ubu01

You should see something like this:

  3461:20261027:150412.006 Proxy "proxy-rocky01" changed state from online to offline

In Administration, Proxies, proxy-rocky01 is Offline and proxy-kafka01 now has both hosts, and rocky01’s graphs carry on because its agent also knows 10.10.10.31. The group itself stayed online: one proxy is still at least the minimum of 1. Start the proxy again with sudo systemctl start zabbix-proxy on rocky01; the server logs the change back to online, and it may move a host back to balance the load.

✅ Checkpoint: The server logged proxy-rocky01 going offline, Latest data for rocky01 kept updating, and after the restart proxy-rocky01 is Online again.