Grafana dashboard variables and repeating panels

Everything happens on the Server Overview dashboard at https://graf01.lab/ unless a step says graf01. Click Edit before each step and Save with a short note after it, so the version history tells the story of this lesson.

1Add a job variable

In edit mode click Add new element > Variable (or the Add variable plus at the top). Choose type Query and fill in:

The preview at the bottom should list node and prometheus. The same lookup in the old classic syntax is label_values(up, job), which you’ll see in many older dashboards.

Check the same values from graf01:

curl -s -G localhost:9090/api/v1/label/job/values --data-urlencode 'match[]=up'; echo

You should see:

{"status":"success","data":["node","prometheus"]}

✅ Checkpoint: A Job drop-down appears at the top of the dashboard with node and prometheus.

2Add a chained instance variable

Add another Query variable. This one reads job from the first, allows several servers and hides the port in the drop-down.

Set Job to node. The Server drop-down now shows graf01 and app01, while the values behind them stay graf01:9100 and app01:9100, because the regex only changes the display text. Values that don’t match a variable regex at all are dropped from the list, so the pattern must fit every value you want to keep.

Pick All.

Check the chained lookup from graf01:

curl -s -G localhost:9090/api/v1/label/instance/values --data-urlencode 'match[]=up{job=~"node"}'; echo

You should see:

{"status":"success","data":["app01:9100","graf01:9100"]}

✅ Checkpoint: Changing Job to prometheus shows localhost in Server, and changing it back to node shows graf01 and app01.

3Use the variables in the panels

Edit each panel and add the two matchers. For example the CPU and memory queries become:

100 * (1 - avg by (instance) (rate(node_cpu_seconds_total{job=~"$job", instance=~"$instance", mode="idle"}[$__rate_interval])))

100 * node_memory_MemAvailable_bytes{job=~"$job", instance=~"$instance"}
  / node_memory_MemTotal_bytes{job=~"$job", instance=~"$instance"}

Do the same for the disk gauge, load bar gauge and Targets table. Then open the CPU panel menu and choose Inspect > Query and click Refresh. The request still shows $__rate_interval, because the browser fills in your own variables and leaves the time variables to the Grafana server.

The response carries a field called executedQueryString with the query Prometheus ran.

With Job node, Server All and Last 1 hour, executedQueryString reads:

Expr: 100 * (1 - avg by (instance) (rate(node_cpu_seconds_total{job=~"node", instance=~"(app01:9100|graf01:9100)", mode="idle"}[1m0s])))
Step: 15s

All became a regex with both servers, and $__rate_interval became 1m0s, which is how the Go code inside Grafana writes 60 seconds. With a 1-hour range the step is 15s (the scrape interval you set on the data source), and max(15s + 15s, 4 x 15s) is 60s. Pick only app01 in the drop-down and every panel narrows to one server.

✅ Checkpoint: Selecting one server changes all five panels, and the query inspector shows the substituted query with =~.

4Add a custom variable for CPU modes

Add a Custom variable named cpu_mode, label “CPU mode”, values user,system,iowait,steal, Multi-value and Include All on. Add a new time series panel CPU time by mode with unit Percent (0.0-1.0), stacked (Graph styles > Stack series > Normal):

sum by (instance, mode) (
  rate(node_cpu_seconds_total{instance=~"$instance", mode=~"$cpu_mode"}[$__rate_interval])
)

Check the shape of the answer from graf01:

curl -s localhost:9090/api/v1/query --data-urlencode \
  'query=count(sum by (instance, mode) (rate(node_cpu_seconds_total{mode=~"user|system|iowait|steal"}[1m])))'; echo

You should see (your timestamp will differ):

{"status":"success","data":{"resultType":"vector","result":[{"metric":{},"value":[1794119650.021,"8"]}]}}

Eight series: four modes for each of two servers. Untick everything except iowait and the panel shows only disk wait time, a handy view when someone says “the server feels slow”.

✅ Checkpoint: The CPU time by mode panel follows both the Server and the CPU mode drop-downs.

5Add interval, text box and constant variables

Add three more variables:

Change the bar gauge query to avg_over_time(node_load1{instance=~"$instance"}[$smooth]) and its title to Load (avg over $smooth) in $env. In the disk gauge use mountpoint="$mount" instead of the fixed slash. Type /boot in the Mount box (if your VM has a separate /boot) and the gauge follows; an empty result means no such mount point on that server.

List the mount points node_exporter reports, from graf01:

curl -s -G localhost:9090/api/v1/label/mountpoint/values --data-urlencode 'match[]=node_filesystem_size_bytes{fstype!~"tmpfs|overlay"}'; echo

With Ubuntu’s default LVM layout, which puts /boot on its own partition, you should see:

{"status":"success","data":["/","/boot"]}

If you installed without LVM and without a separate /boot, the list holds only "/".

✅ Checkpoint: The bar gauge title reads “Load (avg over 5m) in lab” when Smooth is 5m, and the disk gauge follows the Mount text box.

6Make the data source a variable

Add a Data source variable named ds, type Prometheus. Then in each panel pick ${ds} at the top of the data source list instead of “Prometheus”, and set the same ${ds} as the data source of the job and instance variables. You have one Prometheus today, so nothing changes on screen, but when a second Prometheus (test or another site) is added later, one drop-down switches the whole dashboard.

✅ Checkpoint: Every panel and both query variables use ${ds}, and the dashboard looks the same as before.

7See $__interval and $__rate_interval change with the time range

Open the CPU panel’s query inspector again and note the Step and the range in brackets in executedQueryString. For Last 1 hour you saw Step: 15s and [1m0s]. Now change the time picker to Last 24 hours and then Last 7 days, refresh each time and write the two values down.

The step grows with the range, and how far depends on how wide your panel is, because Grafana aims for about one point per pixel. The rate window follows one fixed rule from Grafana’s code: it’s the step plus one scrape interval, but never less than four scrapes (1m0s here). Set the time picker back to Last 1 hour.

✅ Checkpoint: The query inspector shows a longer Step for 7 days than for 1 hour, the rate window is always 15s longer than the step or at least 1m0s, and the CPU graph has no gaps at either range.

8Repeat the CPU panel for each server

Edit the CPU busy % panel, open Repeat options in the panel options pane, set Repeat by variable to instance, direction Horizontal and Max per row 2. Change the panel title to CPU busy % on $instance and save.

With Server set to All you now get two CPU panels side by side, one per server, each with one line. Pick only graf01 and the dashboard shows a single panel. Only the first panel is editable; the clones follow it.

✅ Checkpoint: Two CPU panels appear when Server is All, titled with each server, and one when a single server is picked.

9Repeat a whole row per server

Add a new row with Add new element > Add row/Group into rows, title it Host: $instance, and move the memory stat and disk gauge into it. Select the row and in the edit pane set Repeat by variable to instance. With All selected you get one “Host: graf01:9100” row and one “Host: app01:9100” row, each with its own memory and disk panels.

Repeated rows are the clean way to build “one block per server” pages, and collapsing them keeps the dashboard light, because collapsed rows don’t query.

✅ Checkpoint: Two rows titled Host: graf01:9100 and Host: app01:9100 appear, each with a memory stat and a disk gauge.

10Add a dashboard link and a data link

Open Dashboard options > View all settings > Links > Add dashboard link. Type Dashboards, With tags linux, and tick Include current time range and Include current template variable values. Every dashboard with the linux tag now appears as a link at the top, and clicking it keeps your time range and server choice.

Then edit the Targets table, open Data links and actions > Add link, title it “Show only this server” and set the URL to:

/d/${__dashboard.uid}/server-overview?var-instance=${__data.fields.instance}&${__url_time_range}

Clicking any row in the table now reloads the dashboard filtered to that row’s instance with the same time range. Save with the note “variables, repeats and links”.

✅ Checkpoint: Clicking app01:9100 in the Targets table reloads Server Overview with Server set to app01 only.