Your first Grafana dashboard: data sources and panels

You install Prometheus and node_exporter the short way here, because the SaatVedha Prometheus course explains every flag in detail. Then everything else happens in Grafana at https://graf01.lab/. Commands run on graf01 unless a step says app01.

1Install node_exporter on graf01 and app01

Paste this whole block on graf01, then again on app01:

VER=1.12.1
cd /tmp && wget -q https://github.com/prometheus/node_exporter/releases/download/v$VER/node_exporter-$VER.linux-amd64.tar.gz
tar xzf node_exporter-$VER.linux-amd64.tar.gz
sudo install -o root -g root -m 0755 node_exporter-$VER.linux-amd64/node_exporter /usr/local/bin/
sudo useradd --system --no-create-home --shell /usr/sbin/nologin node_exporter
sudo tee /etc/systemd/system/node_exporter.service >/dev/null <<'EOF'
[Unit]
Description=Prometheus Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
ExecStart=/usr/local/bin/node_exporter
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload && sudo systemctl enable --now node_exporter
curl -s localhost:9100/metrics | grep -E '^node_(load1|memory_MemAvailable_bytes) '

You should see (your values will differ):

Created symlink '/etc/systemd/system/multi-user.target.wants/node_exporter.service' → '/etc/systemd/system/node_exporter.service'.
node_load1 0.2
node_memory_MemAvailable_bytes 2.348716544e+09

Ubuntu 24.04 prints the symlink paths without the quotes. The values differ on every machine. On app01 the memory number is smaller, because app01 has 1.5 GB of RAM against graf01’s 3 GB.

✅ Checkpoint: curl localhost:9100/metrics returns node_load1 on both graf01 and app01.

2Let graf01 reach app01 on port 9100

Run this on app01:

sudo ufw allow OpenSSH
sudo ufw allow from 192.168.56.70 to any port 9100 proto tcp
sudo ufw --force enable
sudo ufw status | grep 9100

You should see:

9100/tcp                   ALLOW       192.168.56.70

Then run this on graf01:

curl -s -m 5 http://app01:9100/metrics | grep '^node_load1 '

node_load1 0.05

Only graf01 may read app01’s metrics now. graf01 needs no rule for its own 9100, because Prometheus reaches graf01:9100 through the loopback interface, which ufw always allows.

✅ Checkpoint: graf01 can curl app01:9100/metrics and ufw on app01 allows 9100 only from 192.168.56.70.

3Install Prometheus on graf01

Paste this whole block on graf01:

VER=3.16.0
cd /tmp && wget -q https://github.com/prometheus/prometheus/releases/download/v$VER/prometheus-$VER.linux-amd64.tar.gz
tar xzf prometheus-$VER.linux-amd64.tar.gz
sudo install -o root -g root -m 0755 prometheus-$VER.linux-amd64/{prometheus,promtool} /usr/local/bin/
sudo useradd --system --no-create-home --shell /usr/sbin/nologin prometheus
sudo mkdir -p /etc/prometheus /var/lib/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus
sudo tee /etc/prometheus/prometheus.yml >/dev/null <<'EOF'
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "node"
    static_configs:
      - targets: ["graf01:9100", "app01:9100"]
EOF
promtool check config /etc/prometheus/prometheus.yml

You should see:

Checking /etc/prometheus/prometheus.yml
 SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax

✅ Checkpoint: promtool reports SUCCESS for /etc/prometheus/prometheus.yml.

4Run Prometheus as a service

Paste this whole block on graf01:

sudo tee /etc/systemd/system/prometheus.service >/dev/null <<'EOF'
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --storage.tsdb.retention.time=15d \
  --web.enable-lifecycle
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload && sudo systemctl enable --now prometheus
sudo ufw allow from 192.168.56.0/24 to any port 9090 proto tcp
sleep 20
curl -s localhost:9090/api/v1/query --data-urlencode 'query=up' | python3 -m json.tool | grep -E '"(instance|job)"|"1"'

You should see (the three targets can come in a different order):

                    "instance": "localhost:9090",
                    "job": "prometheus"
                    "1"
                    "instance": "graf01:9100",
                    "job": "node"
                    "1"
                    "instance": "app01:9100",
                    "job": "node"
                    "1"

All three targets report up = 1. Prometheus listens on port 9090 without Nginx in front, and ufw lets only the lab network reach it, so you can open http://192.168.56.70:9090 from your laptop to check targets under Status > Target health.

✅ Checkpoint: up returns 1 for localhost:9090, graf01:9100 and app01:9100.

5Add Prometheus as a data source in Grafana

In Grafana open Connections > Data sources > Add new data source and pick Prometheus. Set these fields and leave the rest as they are:

Click Save & test. A green box saying “Successfully queried the Prometheus API.” means Grafana reached Prometheus and ran a test query.

Check from graf01 that Grafana stored it (use your admin password):

curl -s -u admin:'Lab@Grafana2026' --cacert /etc/ssl/graf01/graf01.crt \
  https://graf01.lab/api/datasources/name/Prometheus | python3 -m json.tool | grep -E '"(name|type|url|isDefault)"'

You should see:

    "name": "Prometheus",
    "type": "prometheus",
    "url": "http://localhost:9090",
    "isDefault": true,

Setting the scrape interval to match Prometheus (15s) matters later: Grafana uses it to choose sensible steps and the smallest $__rate_interval, which Lesson 5 explains.

✅ Checkpoint: Save & test shows “Successfully queried the Prometheus API.” and the API lists the data source with isDefault true.

6Create the folder and the dashboard with a time series panel

Go to Dashboards > New > New folder and call it SaatVedha Lab. Inside it choose New > New dashboard, click the blue Add new element icon, drag a panel onto the page and choose Configure visualization. Select the Prometheus data source, switch the query editor from Builder to Code and paste:

100 * (1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[$__rate_interval])))

Click Refresh (or press Shift+Enter in the query box). Pick Time series as the visualisation, set the panel title to CPU busy %, and in the options pane set:

You get two lines, graf01:9100 and app01:9100, with a legend table under the graph. Click Save, title the dashboard Server Overview, check the folder is SaatVedha Lab, add the note “first panel” and save. Then click Back to dashboard.

To see the numbers Grafana is drawing, run the same query at Prometheus directly:

Run this on graf01:

curl -s localhost:9090/api/v1/query --data-urlencode \
  'query=100 * (1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])))'; echo

You should see (your numbers and timestamps will differ):

{"status":"success","data":{"resultType":"vector","result":[{"metric":{"instance":"graf01:9100"},"value":[1794119005.116,"3.71"]},{"metric":{"instance":"app01:9100"},"value":[1794119005.116,"1.28"]}]}}

✅ Checkpoint: Server Overview is saved in folder SaatVedha Lab with a CPU busy % panel showing two series in percent.

7Add stat, gauge, bar gauge and table panels

In edit mode add four more panels the same way. For each, paste the query, pick the visualisation and set the options in the table. Leave the query Type on Range unless the table says Instant.

Drag and resize so the four new panels sit in one line under the CPU graph. The stat panel shows one tile per server, red if less than 15% memory is free, and the gauge turns orange at 70% disk used. In the Targets table you see three rows with columns instance, job and Up.

Check the memory query from graf01 if a panel looks odd:

curl -s localhost:9090/api/v1/query --data-urlencode \
  'query=100 * node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes'; echo

{"status":"success","data":{"resultType":"vector","result":[{"metric":{"instance":"graf01:9100","job":"node"},"value":[1794119023.061,"63.77"]},{"metric":{"instance":"app01:9100","job":"node"},"value":[1794119023.061,"58.92"]}]}}

✅ Checkpoint: The dashboard shows five panels: CPU time series, memory stat, disk gauge, load bar gauge and targets table, all with sensible units.

8Add an override and a row

Edit the CPU busy % panel and open Overrides > Add field override > Fields with name, pick graf01:9100, then Add override property > Graph styles > Line style and choose Dashes. Now the monitoring server’s own line is dashed, so nobody mistakes it for the application server.

Back on the dashboard, click Add new element > Add row/Group into rows. Your panels move into the new row; name it Linux hosts. Rows can be collapsed, which also stops their panels from querying until someone opens them, a simple way to keep big dashboards fast.

Open Dashboard options > View all settings and on the General page set Time zone to Browser Time, add the tag linux and set the default refresh to 30s. Save with the note “override and row”.

✅ Checkpoint: The graf01 CPU line is dashed, the panels sit in a collapsible row called Linux hosts, and the dashboard has the linux tag.

9Look at the version history

Click Edit > Dashboard options > View all settings > Versions. You should see at least two versions. The newest carries your note “override and row”, and the oldest is labelled “Initial save”, because Grafana always names a dashboard’s first version that way, whatever note you typed.

Tick both and click Compare versions: Grafana lists the changes in plain words, and View JSON diff shows the raw JSON lines added and removed.

You can also read them through the API. The dashboard’s UID is the short code in its URL after /d/, for example https://graf01.lab/d/fe3k9q2xw1ts0d/server-overview; copy yours into the first line.

Run this on graf01:

DUID=fe3k9q2xw1ts0d
curl -s -u admin:'Lab@Grafana2026' --cacert /etc/ssl/graf01/graf01.crt \
  "https://graf01.lab/api/dashboards/uid/$DUID/versions" | grep -oE '"version":[0-9]+,"created":"[^"]*","message":"[^"]*"'

You should see (your times will differ):

"version":2,"created":"2026-11-08T07:40:12Z","message":"override and row"
"version":1,"created":"2026-11-08T07:31:05Z","message":"Initial save"

The answer is one long line of JSON with a versions list, and each entry also carries the full dashboard JSON of that version, so grep -o cuts out the three fields you care about. The UID is the dashboard’s permanent ID, and yours is a different random string.

✅ Checkpoint: Versions lists at least two saves with your notes, and you can compare them.

10Export the dashboard as JSON

Exit edit mode, click Export > Export as code in the toolbar, leave the model on Classic and click Download file. Turn on Share dashboard with another instance first if the file is going to a different Grafana: it replaces your data source with an input you choose at import time.

Copy the file to graf01 and peek inside (on your laptop):

scp ~/Downloads/Server\ Overview-*.json student@192.168.56.70:server-overview.json
ssh student@192.168.56.70 "python3 -c 'import json; d=json.load(open(\"server-overview.json\")); print(d[\"title\"], d[\"uid\"])'"

You should see (with your own UID):

Server Overview fe3k9q2xw1ts0d

The file name your browser gives the download may differ; adjust the scp line to match. Inside, the classic model stores your row as a panel of type row next to the five panels, along with the time range, refresh and tags. Keep this file: Lesson 13 puts dashboards like it under version control and provisions them automatically.

✅ Checkpoint: You have server-overview.json on graf01 with the title Server Overview and the same UID as the dashboard.