Troubleshooting Grafana and where to go next

Everything runs on graf01 (192.168.56.70) unless a step says app01. Save the health-check script from step 1; it’s worth keeping on every Grafana server you look after.

1Write a four-layer health check

Paste this whole block on graf01:

cat > ~/grafana-check.sh <<'EOF'
#!/bin/bash
# Layer 1: service and database
systemctl is-active grafana-server
curl -s localhost:3000/api/health | jq -c '{database, version}'
# Layer 2: every data source
for uid in $(curl -s -u admin:Lab@Grafana2026 localhost:3000/api/datasources | jq -r '.[].uid'); do
  printf '%-22s ' "$uid"
  curl -s -u admin:Lab@Grafana2026 "localhost:3000/api/datasources/uid/$uid/health" | jq -r .status
done
# Recent errors
sudo grep 'level=error' /var/log/grafana/grafana.log | tail -n 3
EOF
chmod +x ~/grafana-check.sh && ~/grafana-check.sh

You should see this (your UIDs will differ):

active
{"database":"ok","version":"13.3.0"}
es-app01-logs          OK
ae7h2kq0l9mpsd         OK
be2h4c8xn1qpse         OK
ce5d1m3bw7tfya         OK
de9k0p2rz4hxlb         OK
fa3m8t1qx6vzke         OK

The six lines are your six data sources in name order (Elasticsearch app01 logs, InfluxDB, Loki, MySQL app01, Prometheus, TestData), and every healthy one says OK; the health message behind each status differs by data source type. If the last part prints old error lines, such as the provisioning error from Lesson 13’s break lab, check their times before you worry. When someone reports a problem, run this first: if layer 1 or 2 fails, you already know where to look and nobody has to open a dashboard.

✅ Checkpoint: The script prints active, database ok, and OK for every data source.

2Read the logs and raise one logger only

Paste this whole block on graf01:

sudo tail -n 3 /var/log/grafana/grafana.log
sudo sed -i '/^\[log\]/,/^\[/ s/^;filters =.*/filters = plugins.signature:debug/' /etc/grafana/grafana.ini
sudo grep -A9 '^\[log\]' /etc/grafana/grafana.ini | grep -v -e '^[#;]' -e '^$'
sudo systemctl restart grafana-server && sleep 8
sudo grep 'logger=plugins.signature' /var/log/grafana/grafana.log | tail -n 2

The first three lines are whatever Grafana logged last, so yours will differ. After them you should see this (your times and plugin IDs can differ):

[log]
filters = plugins.signature:debug
logger=plugins.signature t=2026-11-08T14:03:02.441527806Z level=debug msg="Plugin signature valid" id=grafana-lokiexplore-app
logger=plugins.signature t=2026-11-08T14:03:02.447190214Z level=debug msg="Plugin signature valid" id=grafana-pyroscope-app

Grafana 13 installs a few app plugins on first start, such as Logs Drilldown (grafana-lokiexplore-app), and these are the signed plugins the debug logger reports on.

Each line starts with the logger name, which is what filters matches, so you can raise provisioning, ngalert or tsdb.prometheus to debug while everything else stays at info. Put the line back to ;filters = when you’re done; debug logs fill disks quickly.

✅ Checkpoint: grafana.ini has filters = plugins.signature:debug under [log] and the log shows debug lines from that logger only.

3Use the query inspector on a real panel

Open Server Overview, set the time range to the last 24 hours, and on the CPU panel open the menu and choose Inspect > Query, then click Refresh. Look at the request, then the Stats tab.

The Stats tab shows these rows (your numbers will differ):

Total request time      182 ms
Data processing time    1.2 ms
Number of queries       2
Total number rows       578

The Query tab shows the POST to /api/ds/query with the PromQL after $instance and $__rate_interval were filled in, and the intervalMs Grafana chose for this range. If a panel is slow, these numbers tell you whether it’s the data source (request time) or the browser (data processing time).

✅ Checkpoint: You can read the total request time, number of queries and rows for the CPU panel in the inspector.

4Back up grafana.db safely and test the copy

Paste this whole block on graf01:

sudo apt-get install -y sqlite3
sudo mkdir -p /var/backups/grafana
sudo sqlite3 /var/lib/grafana/grafana.db ".backup '/var/backups/grafana/grafana-$(date +%F).db'"
sudo sqlite3 /var/backups/grafana/grafana-$(date +%F).db 'PRAGMA integrity_check;'
sudo sqlite3 /var/backups/grafana/grafana-$(date +%F).db 'SELECT count(*) FROM dashboard;'
echo '30 1 * * * root sqlite3 /var/lib/grafana/grafana.db ".backup /var/backups/grafana/grafana-$(date +\%F).db"' \
  | sudo tee /etc/cron.d/grafana-backup

You should see this (your dashboard count will differ):

ok
17
30 1 * * * root sqlite3 /var/lib/grafana/grafana.db ".backup /var/backups/grafana/grafana-$(date +\%F).db"

.backup uses SQLite’s online backup, so it takes a consistent copy while Grafana keeps running; a plain cp of a database being written can give you a corrupt file. The integrity check proves the copy opens and is whole. To restore: stop grafana-server, copy the backup over /var/lib/grafana/grafana.db, chown grafana:grafana it, and start Grafana.

Add a line to delete backups older than 14 days, and copy them off the VM.

✅ Checkpoint: The backup file passes integrity_check, its dashboard count matches your Grafana, and /etc/cron.d/grafana-backup exists.

5Know what moving to MySQL or PostgreSQL involves

You won’t do this in the lab, but you should know the shape of it. Grafana’s database is chosen in [database]:

A grafana.ini section for PostgreSQL looks like this:

[database]
type = postgres
host = db01.lab:5432
name = grafana
user = grafana
password = """$__file{/etc/grafana/db.pass}"""
ssl_mode = require

On start-up Grafana creates its tables in the empty database. It does not copy anything from grafana.db, so teams usually move by recreating the content: data sources, dashboards and alerting from provisioning files (Lesson 13), users from LDAP or SSO, and anything else through the HTTP API. That’s one more reason to keep Grafana as code.

With an external database you can also run two Grafana servers behind a load balancer for high availability.

✅ Checkpoint: You can name the [database] keys for an external database and explain why grafana.db content isn’t copied automatically.

6Install the Zabbix plugin and plan upgrades

Paste this whole block on graf01:

sudo grafana cli plugins install alexanderzobnin-zabbix-app
sudo systemctl restart grafana-server
sudo grafana cli plugins ls | grep zabbix
apt-cache policy grafana | head -n 3

You should see this (the plugin version may be newer than 6.10.2):

✔ Downloaded and extracted alexanderzobnin-zabbix-app v6.10.2 zip successfully to /var/lib/grafana/plugins/alexanderzobnin-zabbix-app

Please restart Grafana after installing or removing plugins. Refer to Grafana documentation for instructions if necessary.

alexanderzobnin-zabbix-app @ 6.10.2
grafana:
  Installed: 13.3.0
  Candidate: 13.3.0

The Zabbix plugin is an app: enable it under Administration > Plugins and data > Plugins, then add a Zabbix data source pointing at a Zabbix API. The Zabbix monitoring course builds that Zabbix server. Plugins update separately from Grafana with grafana cli plugins update-all, and both should be tested before production.

For Grafana itself, keep the routine from Lesson 2: the package is held with apt-mark hold grafana, and an upgrade means a VM snapshot, a .backup of grafana.db, reading the release notes for breaking changes, apt-mark unhold grafana, installing the chosen version, checking the logs and plugin list, and holding the package again.

✅ Checkpoint: grafana cli plugins ls lists alexanderzobnin-zabbix-app and apt-cache policy shows the installed Grafana version.

7Final project: the Service Health dashboard

Build one dashboard, Service Health, in the NOC Wall folder, that answers “is the exam portal on app01 healthy?” in ten seconds. Use what you built in this course:

Then add one more Grafana-managed rule, Nginx 5xx high, on the Loki query sum(rate({job="nginx", status=~"5.."}[5m])) above 0.5 for 5 minutes with labels team=linux and severity=critical, so it follows the policy from Lesson 11. Finally run ~/grafana-as-code/export.sh and commit, so the dashboard is in Git.

Run this on graf01 to prove the whole chain:

~/exam-traffic.sh 600 &
sleep 420
curl -s localhost:8025/api/v1/messages | jq -r '.messages[0] | "\(.To[0].Address)  |  \(.Subject)"'
cd ~/grafana-as-code && ./export.sh >/dev/null; grep -l '"title": "Service Health"' dashboards/*.json; git add . && git commit -qm "Service Health" && echo committed

You should see this (the file name is your dashboard’s UID):

oncall@lab.local  |  [FIRING] Nginx 5xx high on  (critical)
dashboards/eg4pdt0c2v6mfb.json
committed

The exam traffic always includes failing /api/submit calls, so the rule fires; that’s intended here. The subject has no instance because the query sums across all lines; add host to the sum if you want it. Pause the rule or raise its threshold when you finish.

✅ Checkpoint: Service Health shows all seven parts, the Nginx 5xx high alert reached oncall@lab.local, and the dashboard is committed in ~/grafana-as-code.

8Snapshot the finished lab

Run this on your laptop:

VBoxManage snapshot graf01 take course-complete
VBoxManage snapshot app01 take course-complete

You should see this (your UUIDs will differ):

0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Snapshot taken. UUID: 0b7d5e29-3c81-4f46-a9d0-8e2f6c1a5b74
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Snapshot taken. UUID: 6e1c9a83-5f27-4d0b-b3e6-4a8d7f2c0e15

✅ Checkpoint: Both VMs have a snapshot called course-complete.