Grafana dashboards for Prometheus

Everything runs on prom01. Grafana will listen on 3000, and you’ll reach it from your laptop at http://prom01.lab:3000 (the laptop’s hosts file got prom01.lab in Lesson 3). The Grafana screens change a little between 13.x releases, so if a button label differs slightly, look for the nearest match.

1Add the Grafana APT repository

Paste this whole block:

sudo apt-get install -y apt-transport-https wget gnupg
sudo mkdir -p /etc/apt/keyrings
sudo wget -O /etc/apt/keyrings/grafana.asc https://apt.grafana.com/gpg-full.key
sudo chmod 644 /etc/apt/keyrings/grafana.asc
echo "deb [signed-by=/etc/apt/keyrings/grafana.asc] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt-get update

You should see the repository line echoed back by tee:

deb [signed-by=/etc/apt/keyrings/grafana.asc] https://apt.grafana.com stable main

Then apt-get update prints a Get or Hit line for https://apt.grafana.com stable InRelease and one for stable/main amd64 Packages, and finishes with Reading package lists... Done. If you get a GPG or NO_PUBKEY error instead, the key download failed, so run the wget line again.

The signed-by part means apt trusts this key only for the Grafana repository, not for every package on the system. This is the method Grafana documents for Debian and Ubuntu; the old apt-key method is deprecated and you shouldn’t use it on new systems.

✅ Checkpoint: apt-get update fetches the apt.grafana.com stable InRelease without a GPG error.

2Install Grafana OSS and start it

Run this:

sudo apt-get install -y grafana
sudo systemctl daemon-reload
sudo systemctl enable --now grafana-server
apt-cache policy grafana | head -7
systemctl is-active grafana-server

You should see this:

grafana:
  Installed: 13.3.0
  Candidate: 13.3.0
  Version table:
 *** 13.3.0 500
        500 https://apt.grafana.com stable/main amd64 Packages
        100 /var/lib/dpkg/status
active

*** marks the version that’s installed. 500 is the priority apt gives a normal enabled repository, and the 100 line is dpkg’s own record of the installed package. Without head the table goes on with every older release in the repository, starting with 13.2.5 and 13.2.4.

The package is called grafana (the open-source edition) and the service is grafana-server. Your minor version may be newer; anything starting with 13.3 behaves the same for this lesson. Logs go to the journal and to /var/log/grafana/grafana.log.

✅ Checkpoint: grafana-server is active and apt-cache shows a 13.3 version installed.

3Open port 3000 to your laptop and log in

Run this:

sudo ufw allow from 192.168.56.1 to any port 3000 proto tcp
curl -s http://localhost:3000/api/health

You should see this (the commit hash is cut here as “…”):

Rule added
{
  "database": "ok",
  "version": "13.3.0",
  "commit": "..."
}

Browse from your laptop to http://prom01.lab:3000, log in as admin with password admin, and set a new password when asked. Don’t skip that screen; a Grafana with admin/admin on a network is one of the most common findings in security audits.

✅ Checkpoint: You are logged in to Grafana as admin with your own password.

4Add Prometheus as a data source

Go to Connections > Add new connection, search for Prometheus, open it and click Add new data source. Fill in only these fields and leave the rest as they are:

Click Save & test at the bottom. You should see a green box saying Successfully queried the Prometheus API. Setting the scrape interval to the same 15 seconds as prometheus.yml matters, because Grafana uses it to work out $__rate_interval; if it’s wrong, rate() panels can come out empty when you zoom in.

At work you’d usually create the data source from a file instead, so it’s the same on every Grafana: put a YAML file with apiVersion: 1 and a datasources list (name Prometheus, type prometheus, access proxy, url http://localhost:9090) in /etc/grafana/provisioning/datasources/ and restart Grafana.

✅ Checkpoint: Save & test shows “Successfully queried the Prometheus API.”

5Import Node Exporter Full (ID 1860)

Go to Dashboards > New > Import dashboard. Type 1860 in the box and click Load. On the next screen choose your Prometheus data source in the drop-down at the bottom and click Import.

The dashboard opens with four drop-downs at the top: Datasource, Job (filled from label_values(node_uname_info, job)), Nodename and Instance. Pick Job node and Nodename node01, and you’ll see node01’s CPU, memory, disk and network panels. Switch Nodename to prom01 to see the monitoring server itself.

✅ Checkpoint: Node Exporter Full shows live CPU, memory and disk numbers for node01 and prom01.

6Create your own dashboard with an instance variable

Go to Dashboards > New > New dashboard. Before adding panels, add the variable: click Edit if needed, then the Add new element (plus) icon and choose Variable. Choose type Query and fill it in like this:

Click Preview, and the preview should list 192.168.56.60:9100 and 192.168.56.61:9100. Apply, then Save the dashboard with the title prom01 lab overview. The same thing written as a classic query is label_values(node_uname_info{job="node"}, instance).

✅ Checkpoint: The new dashboard has an Instance drop-down listing both lab servers plus All.

7Add the four panels

For each panel, use Add new element > Panel (or + Add visualization on an empty dashboard), pick the Prometheus data source, switch the query editor from Builder to Code, paste the query, click Refresh and choose the visualisation on the right.

Panel queries (paste one per panel):

# 1. Time series, title "CPU busy %", unit Percent (0-100), legend {{instance}}
100 * (1 - avg by (instance) (rate(node_cpu_seconds_total{job="node", mode="idle", instance=~"$instance"}[$__rate_interval])))

# 2. Stat, title "Memory available", unit Percent (0.0-1.0)
instance:node_memory_available:ratio{instance=~"$instance"}

# 3. Gauge, title "Root disk used", unit Percent (0-100), thresholds 70 orange, 85 red
100 * (1 - instance:node_filesystem_avail:ratio{instance=~"$instance"})

# 4. Table, title "Scrape targets", Format: Table, Type: Instant
up

Panels 2 and 3 use the recording rules you made in Lesson 7, so Grafana asks Prometheus for one cheap pre-computed series instead of doing the division on every refresh. For the table, add the Organize fields by name transformation to hide the Time column and rename Value to “Up”. Save the dashboard when all four look right.

To check what Grafana will ask for, run the CPU query against Prometheus yourself:

curl -s http://localhost:9090/api/v1/query --data-urlencode 'query=100 * (1 - avg by (instance) (rate(node_cpu_seconds_total{job="node",mode="idle"}[1m])))' | jq -c '.data.result[] | [.metric.instance, .value[1]]'

You should see this (your numbers will differ):

["192.168.56.60:9100","6.533333333332442"]
["192.168.56.61:9100","1.8666666666666742"]

If curl gives numbers but the panel says No data, the problem is in Grafana (variable, data source or time range), not in Prometheus. That one check saves a lot of guessing.

✅ Checkpoint: All four panels show data for All, and choosing one instance changes the CPU, memory and disk panels.

8Let Prometheus scrape Grafana

Grafana exposes its own metrics at /metrics on port 3000, switched on by default ([metrics] enabled = true in the defaults). Add a job at the end of scrape_configs in /etc/prometheus/prometheus.yml:

Add this job:

  - job_name: grafana
    static_configs:
      - targets: ['localhost:3000']

Then run this:

curl -s http://localhost:3000/metrics | grep '^grafana_build_info'
promtool check config /etc/prometheus/prometheus.yml && curl -s -X POST http://localhost:9090/-/reload
sleep 20; curl -s http://localhost:9090/api/v1/query --data-urlencode 'query=up{job="grafana"}' | jq -c '.data.result[] | [.metric.instance, .value[1]]'

You should see this (build details and the promtool lines are cut here as “…”):

grafana_build_info{branch="...",edition="oss",goversion="...",revision="...",version="13.3.0"} 1
...
 SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax
...
["localhost:3000","1"]

Now your monitoring watches its own dashboards server too, and you could add an InstanceDown-style alert for up{job="grafana"} == 0. With this, prom01 runs every job from the course: prometheus, node, alertmanager and grafana.

✅ Checkpoint: up{job=”grafana”} returns 1 for localhost:3000.