Zabbix health, housekeeping, upgrades and troubleshooting

Everything runs on zbx01 (192.168.56.80) unless a step says otherwise. web01 is monitored through zbxprx01 (192.168.56.82) since Lesson 19, which matters in step 9. The zabbix_server -R outputs, log lines and error messages were captured on a real Zabbix 8.0.0 server.

1Read the server’s health numbers

Open Monitoring > Latest data, set Hosts to Zabbix server and type Utilization in Name. Then repeat with cache and Queue.

You should see these items (your values will differ):

On a lab this small everything is close to zero, which is your baseline. If “Zabbix server health” isn’t linked, add it on Data collection > Hosts > Zabbix server > Templates.

✅ Checkpoint: Latest data shows the Utilization, cache and Queue items for Zabbix server with fresh values.

2Ask the running server for its status

Run this on zbx01:

sudo zabbix_server -R status
ps -eo args | grep 'zabbix_server: poller #' | grep -v grep

You should see this (your timings will differ, and ps lists all five pollers):

configuration syncer #1 [synced configuration in 0.013639 sec, idle 10 sec]
event manager #1 [assessed:9 processed:0 discarded:0 events, idle 5.0s, during 5.0s]
preprocessing manager #1 [queued 1, processed 1 values, idle 5.009264 sec during 5.009377 sec]
service manager #1 [processed 0 events, updated 0 event tags, deleted 0 problems, synced 0 service updates, idle 5.004933 sec during 5.004973 sec]
/usr/sbin/zabbix_server: poller #1 [got 0 values in 0.000010 sec, idle 5 sec]
/usr/sbin/zabbix_server: poller #2 [got 0 values in 0.000040 sec, idle 5 sec]

-R status is new in 8.0 and asks the manager processes what they did in their last cycle. The ps line shows the title every Zabbix process sets for itself, which is the fastest way to see a poller stuck on “getting values”.

✅ Checkpoint: Both commands print process lines, and you can see five poller processes in ps.

3Look inside the history cache

Run this on zbx01:

sudo zabbix_server -R diaginfo=historycache

You should see:

Runtime control command was forwarded successfully

Then read the log:

sudo grep -A6 'history cache diagnostic' /var/log/zabbix/zabbix_server.log | tail -7

== history cache diagnostic information ==
Items:10 values:1 time:0.000007
Memory.data:
  size: free:16776760 used:48
  chunks: free:1 used:1 min:16776760 max:16776760
    buckets:
      256+:1

“values:1” means one value is waiting to be written, and “free:16776760” of the default 16M is unused. On a healthy server this number stays near the full size; in break 2 you’ll see it drop to zero.

✅ Checkpoint: The log shows “history cache diagnostic information” with a large free value.

4Change logging and reload the cache without a restart

Run this on zbx01:

sudo zabbix_server -R log_level_increase=poller
sudo zabbix_server -R log_level_decrease=poller
sudo zabbix_server -R config_cache_reload
sudo grep -E 'log level has been|forced reloading' /var/log/zabbix/zabbix_server.log | tail -4

You should see this (your PIDs and times will differ):

Runtime control command was forwarded successfully
Runtime control command was forwarded successfully
Runtime control command was forwarded successfully
 14926:20261108:101404.834 log level has been increased to 4 (debug)
 14926:20261108:101406.841 log level has been decreased to 3 (warning)
 14879:20261108:101410.112 forced reloading of the configuration cache
 14879:20261108:101410.130 finished forced reloading of the configuration cache

A target such as poller, "agent poller" or "history syncer,2" raises the level for those processes only, so the log doesn’t explode. A wrong name is refused with invalid process type: "nosuch", and you should always decrease again after collecting what you need.

✅ Checkpoint: The log shows the level going to 4 and back to 3, and a forced configuration cache reload.

5Set housekeeping and run it once

Open Administration > Housekeeping. In the History section keep “Enable internal housekeeping” on, switch on “Override item history period” and set Data storage period to 14d; in Trends keep 365d. Click Update.

Run this on zbx01:

sudo zabbix_server -R housekeeper_execute
sleep 5; sudo grep 'housekeeper \[' /var/log/zabbix/zabbix_server.log | tail -1

You should see this (your counts and times will differ):

Runtime control command was forwarded successfully
 14890:20261108:101408.048 housekeeper [deleted 203 hist/trends, 0 events, 0 problems, 0 sessions, 0 alarms, 0 audit, 0 autoreg_host, 0 records, 0 sets, 0 expired device authentication records in 0.054674 sec, idle for 1 hour(s)]

The line tells you how many rows each run removed and how long it took. If one run starts taking minutes on a big server, that’s the sign to look at partitioning the history tables (MySQL partitions, or TimescaleDB on PostgreSQL) and switching internal housekeeping off for history and trends.

Administration > Housekeeping: history override switched on with a 14d storage period

✅ Checkpoint: Housekeeping shows the 14d override, and the log has a “housekeeper [deleted …” line.

6Measure the database

Run this on zbx01:

sudo mysql -e "SELECT table_name, ROUND((data_length+index_length)/1024/1024,1) AS size_mb
  FROM information_schema.tables WHERE table_schema='zabbix'
  ORDER BY (data_length+index_length) DESC LIMIT 5;"
sudo du -sh /var/lib/mysql/zabbix

You should see this (your sizes will differ):

TABLE_NAME      size_mb
history_uint    48.5
history         31.2
trends_uint     12.8
items           10.4
history_text    6.1
182M    /var/lib/mysql/zabbix

The history tables are almost always at the top. Write this number down every month; a rough formula for planning is new values per second × bytes per value (about 90 on MySQL) × seconds of retention, so 1000 NVPS kept for 14 days is around 100 GB of history.

✅ Checkpoint: You know the size of your zabbix schema and which tables are the largest.

7Back up the database and configuration

Paste this whole block on zbx01:

mkdir -p ~/backup && cd ~/backup
sudo mysqldump --single-transaction --quick zabbix | gzip > zabbix-db-$(date +%F).sql.gz
sudo tar czf zabbix-etc-$(date +%F).tgz /etc/zabbix /etc/nginx/conf.d/zabbix.conf \
  /etc/ssl/certs/zbx01.lab.crt /etc/ssl/private/zbx01.lab.key
ls -lh ~/backup
gunzip -c zabbix-db-$(date +%F).sql.gz | grep -c 'CREATE TABLE'

You should see this (your dump size and dates will differ):

tar: Removing leading `/' from member names
tar: Removing leading `/' from hard link targets
total 41M
-rw-rw-r-- 1 student student 41M Nov  8 10:32 zabbix-db-2026-11-08.sql.gz
-rw-r--r-- 1 root    root    21K Nov  8 10:32 zabbix-etc-2026-11-08.tgz
230

The two tar lines are normal. tar stores the paths without the leading slash, so a later tar xzf unpacks into the folder you’re in instead of writing straight over /etc. The configuration archive comes to about 21K because zabbix_server.conf alone is about 35 KB of comments and defaults, and gzip shrinks that well. /etc/nginx/conf.d/zabbix.conf is only a symlink to /etc/zabbix/nginx.conf, created by the zabbix-nginx-conf package, so the real Nginx file is already inside /etc/zabbix.

--single-transaction takes a consistent copy of InnoDB tables without stopping the server. The second file holds the server, agent, frontend (/etc/zabbix/web/zabbix.conf.php) and Nginx configuration, plus the certificate. Copy both files off the VM, because a backup on the same disk isn’t a backup.

✅ Checkpoint: ~/backup has a .sql.gz with about 230 CREATE TABLE lines and a .tgz of the configuration.

8Rehearse the upgrade checks

Run this on zbx01:

zabbix_server -V | head -1
mysql -V
php -r 'echo PHP_VERSION, PHP_EOL;'
sudo mysql zabbix -e "SELECT mandatory, optional FROM dbversion;"

You should see:

zabbix_server (Zabbix) 8.0.0
mysql  Ver 8.4.11-0ubuntu0.26.04.1 for Linux on x86_64 ((Ubuntu))
8.5.4
mandatory  optional
8000000    8000000

Your lab is already on 8.0, so you can’t upgrade it, but these four checks are what you run first on a 7.0 server at work. Zabbix 8.0 needs MySQL 8.4 or newer (MariaDB 10.11, PostgreSQL 15) and PHP 8.2 or newer, and a 7.0 database shows 7000000 here.

The upgrade on Ubuntu, after a backup, is short:

sudo systemctl stop zabbix-server
wget https://repo.zabbix.com/zabbix/8.0/release/ubuntu/pool/main/z/zabbix-release/zabbix-release_latest_8.0+ubuntu26.04_all.deb
sudo dpkg -i zabbix-release_latest_8.0+ubuntu26.04_all.deb
sudo apt update
sudo apt install --only-upgrade zabbix-server-mysql zabbix-frontend-php zabbix-nginx-conf zabbix-sql-scripts zabbix-agent2
grep -n 'root' /etc/zabbix/nginx.conf        # must be /usr/share/zabbix/ui
sudo systemctl start zabbix-server
sudo tail -f /var/log/zabbix/zabbix_server.log

The server log during the database upgrade looks like this:

current database version (mandatory/optional): 07000000/07000030
required mandatory version: 07050196
starting automatic database upgrade
completed 1% of database upgrade
...
database upgrade fully completed

Note the new /release/ part of the repository URL, and that the frontend moved to /usr/share/zabbix/ui in 7.2, so an old Nginx root line gives a 404. Templates and media types are not upgraded for you: import the new versions afterwards, and upgrade proxies to the same major version soon after the server.

✅ Checkpoint: You have the four version numbers of your lab written next to the 8.0 minimums.

9Drill the troubleshooting order

Run the three checks on your own lab. First the host: open Data collection > Hosts, note that web01 shows “remote-proxy” under Monitored by, so any test must come from zbxprx01.

Run this on zbxprx01 (copy web01’s key from /etc/zabbix/zabbix_agent2.d/web01.psk first):

sudo apt install -y zabbix-get
echo 'PASTE_THE_64_HEX_KEY' > /tmp/web01.psk && chmod 600 /tmp/web01.psk
zabbix_get -s 192.168.56.81 -k agent.ping \
  --tls-connect psk --tls-psk-identity "PSK web01" --tls-psk-file /tmp/web01.psk

You should see:

1

Then the item: on web01 open Items, filter State Not supported and hover the red info icon; the text is the agent’s own reason. Then the alert: open Reports > Action log for the last day and check each sent, failed or missing message against Lesson 16’s action and the user’s media severity.

1. Collection : availability icon, zabbix_get from the server OR proxy that polls the host
2. Item       : item error text, Test > Get value, zabbix_agent2 -t KEY on the host
3. Trigger    : Monitoring > Problems (show suppressed), trigger expression on latest values
4. Action     : Reports > Action log, action conditions, maintenance, user media and severity

✅ Checkpoint: zabbix_get from zbxprx01 returns 1 for web01, and you know where each of the four checks lives.

10Final project: monitor a new service end to end

Pick a service on web01 you haven’t monitored yet, for example Mailpit on zbx01 or a second Nginx site, and use everything from the course in one go. Treat it like a real request from a team lead.

✅ Checkpoint: Your new service has data, triggers, an alert that reaches you, a dashboard and a backup taken after the change.