Prometheus and Node Exporter dashboards in Grafana

Most steps are in Grafana at https://graf01.lab/ (log in as admin with Lab@Grafana2026). The Prometheus changes run on graf01 (192.168.56.70). app01 (192.168.56.71) only needs node_exporter running, which it has had since Lesson 4.

1Check what Prometheus already scrapes

Run this on graf01:

curl -s 'localhost:9090/api/v1/query?query=up' \
  | jq -r '.data.result[] | "\(.metric.job)  \(.metric.instance)  \(.value[1])"'
curl -s 'localhost:9090/api/v1/query?query=node_uname_info' \
  | jq -r '.data.result[] | "\(.metric.job)  \(.metric.instance)  \(.metric.nodename)"'

You should see this (the lines can come in a different order):

prometheus  localhost:9090  1
node  graf01:9100  1
node  app01:9100  1
node  graf01:9100  graf01
node  app01:9100  app01

Both node_exporters are up under job="node", and node_uname_info carries the hostname as nodename. Node Exporter Full builds its drop-downs from this metric. If app01 shows 0, check systemctl status node_exporter on app01 and its ufw rule for 9100 from graf01.

✅ Checkpoint: up returns 1 for prometheus, graf01:9100 and app01:9100.

2Import Node Exporter Full (1860)

In Grafana go to Dashboards > New > Import. Type 1860 in the grafana.com box and click Load. Grafana (on graf01, which has internet through its NAT adapter) downloads the latest revision.

Leave the name as Node Exporter Full, pick the folder SaatVedha Lab, leave the UID as it is and click Import.

At the top pick Job node, Nodename app01 and the panels fill in. Switch to graf01 and they change. If your laptop shows an error about reaching grafana.com, download the JSON on your laptop from the dashboard’s page and use Upload dashboard JSON file instead.

✅ Checkpoint: Node Exporter Full is in SaatVedha Lab and shows data for both app01 and graf01.

3Read the rows like an engineer

Spend five minutes on app01’s view with the time range set to the last 1 hour. The first two rows are open, the rest are collapsed; click a row title to open it.

On a 1.5 GB app01 with MySQL, the RAM Used gauge may look high, and the Memory Details row shows most of it is page cache that Linux gives back when needed. Reading that before raising a ticket saves a lot of embarrassment.

✅ Checkpoint: You can point to CPU Busy, RAM Used and Root FS Used for app01 and explain what iowait means.

4Save your own copy and leave the original alone

Open Node Exporter Full, click Edit, then the arrow next to Save dashboard and choose Save as copy. Name it Node Exporter (SaatVedha), folder SaatVedha Lab, and save. Grafana gives the copy a new UID.

Now make one change in the copy only: on Root FS Used set thresholds to green base, orange 75 and red 90, then save with the note “lab thresholds”.

Run this on graf01 to list both:

curl -s -u admin:Lab@Grafana2026 'localhost:3000/api/search?query=Node%20Exporter' \
  | jq -r '.[] | "\(.uid)  \(.title)  [\(.folderTitle)]"'

You should see this (your copy gets its own UID):

rYdddlPWk  Node Exporter Full  [SaatVedha Lab]
dg3oenca811aca  Node Exporter (SaatVedha)  [SaatVedha Lab]

The original keeps rYdddlPWk, which is the UID the community JSON ships with. When a new revision comes out you can import it over the original (same UID) without touching your copy, compare the two, and move over the panels you like.

✅ Checkpoint: Two dashboards exist: the untouched original with UID rYdddlPWk and your copy with a new UID and changed thresholds.

5Let Prometheus scrape Grafana itself

Grafana is bound to 127.0.0.1:3000 since Lesson 3, which is fine because Prometheus runs on the same VM. First look at the page, then add a grafana job. The new file is the Lesson 4 file plus the grafana job and a rule_files line you’ll use in step 7.

Paste this whole block on graf01:

curl -s localhost:3000/metrics | grep -E '^grafana_build_info|^grafana_stat_totals_dashboard '

sudo tee /etc/prometheus/prometheus.yml >/dev/null <<'EOF'
global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - /etc/prometheus/rules/*.yml

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "node"
    static_configs:
      - targets: ["graf01:9100", "app01:9100"]

  - job_name: "grafana"
    static_configs:
      - targets: ["localhost:3000"]
EOF
sudo mkdir -p /etc/prometheus/rules
promtool check config /etc/prometheus/prometheus.yml
curl -s -X POST localhost:9090/-/reload && echo reloaded

The first line of output is grafana_build_info, a constant 1 whose labels carry version="13.3.0", edition="oss", the Go version and the commit Grafana was built from. After it you should see this (your dashboard count will differ):

grafana_stat_totals_dashboard 6
Checking /etc/prometheus/prometheus.yml
 SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax

reloaded

The reload endpoint works because Lesson 4 started Prometheus with --web.enable-lifecycle. Open Status > Target health in the Prometheus UI (http://192.168.56.70:9090) and the grafana job shows one target, UP.

✅ Checkpoint: Target health shows job grafana with localhost:3000 UP.

6Stop /metrics leaking through Nginx

Nginx forwards everything under https://graf01.lab/ to Grafana, so right now anyone on the network can read /metrics through it. Prometheus reads it locally, so block it at the proxy:

Paste this whole block on graf01:

curl -sk -o /dev/null -w "%{http_code}\n" https://graf01.lab/metrics
sudo sed -i '0,/location \/ {/s//location = \/metrics { deny all; }\n\n    location \/ {/' /etc/nginx/sites-available/grafana
sudo nginx -t && sudo systemctl reload nginx
curl -sk -o /dev/null -w "%{http_code}\n" https://graf01.lab/metrics

You should see:

200
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
403

The sed adds a location = /metrics block just before the first location / in the site. If you’d rather keep it reachable for a remote Prometheus, set basic_auth_username and basic_auth_password under [metrics] in grafana.ini instead.

✅ Checkpoint: https://graf01.lab/metrics returns 403 while curl localhost:3000/metrics on graf01 still works.

7Add a recording rule for CPU busy

Paste this whole block on graf01:

sudo tee /etc/prometheus/rules/node.yml >/dev/null <<'EOF'
groups:
  - name: node-recording
    interval: 30s
    rules:
      - record: instance:node_cpu_utilisation:rate5m
        expr: 100 * (1 - avg by (instance, job) (rate(node_cpu_seconds_total{mode="idle"}[5m])))
EOF
promtool check rules /etc/prometheus/rules/node.yml
curl -s -X POST localhost:9090/-/reload && sleep 35
curl -s 'localhost:9090/api/v1/query?query=instance:node_cpu_utilisation:rate5m' \
  | jq -r '.data.result[] | "\(.metric.instance)  \(.value[1])"'

You should see this (your CPU numbers will differ):

Checking /etc/prometheus/rules/node.yml
  SUCCESS: 1 rules found

graf01:9100  6.833333333333329
app01:9100  2.1666666666666714

The avg by (instance, job) keeps the labels a dashboard filters on. In your copy of the dashboard you can now replace the CPU Busy query with instance:node_cpu_utilisation:rate5m{instance="$node", job="$job"}. On two servers you won’t feel a difference; on two hundred, with a TV screen refreshing all day, you will.

✅ Checkpoint: promtool reports 1 rule found and the recorded series returns one value per server.

8Build the Grafana health dashboard

Create a dashboard called Grafana health in SaatVedha Lab with the Prometheus data source and these panels:

Open a few dashboards in another tab and watch the request rate move. A climbing p95 with few requests usually means a slow data source, and a jump in 5xx status codes points at a plugin, the database or the proxy.

✅ Checkpoint: Grafana health shows the version 13.3.x, a request rate, p95 latency, dashboard count, memory and Scrape OK = UP.

9Look at the JSON model and UID

In Grafana health click Edit > Settings > JSON Model and scroll to the bottom: you’ll see "title", "uid" and "version". The same JSON is available from the API:

Run this on graf01:

curl -s -u admin:Lab@Grafana2026 localhost:3000/api/dashboards/uid/rYdddlPWk \
  | jq '{title: .dashboard.title, uid: .dashboard.uid, version: .dashboard.version, folder: .meta.folderTitle, panels: (.dashboard.panels | length)}'

You should see:

{
  "title": "Node Exporter Full",
  "uid": "rYdddlPWk",
  "version": 1,
  "folder": "SaatVedha Lab",
  "panels": 28
}

The UID appears in the URL (/d/rYdddlPWk/...), in links between dashboards and in API calls, so pick meaningful UIDs for your own dashboards (for example grafana-health) and never change them once people bookmark them. The panel count is top-level panels and rows; collapsed rows hold more inside.

✅ Checkpoint: The API returns the title, UID rYdddlPWk, folder SaatVedha Lab and a panel count for Node Exporter Full.

10Snapshot graf01

Run this on your laptop:

VBoxManage snapshot graf01 take node-dashboards

You should see this (your UUID will differ):

0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Snapshot taken. UUID: 7f2c4e81-0b9d-4a63-b5e7-29c8d1f4a036

✅ Checkpoint: graf01 has a snapshot called node-dashboards.