PromQL examples you will use every day

Everything here runs on prom01, against the Prometheus you set up in Lessons 2 to 4 (listening on 127.0.0.1:9090, so curl localhost:9090 needs no password). You can type the same queries into the Query page at https://prom01.lab/ from your laptop. The numbers on your screen will differ from mine; the shape of the answer is what you’re checking.

1Install jq and make your first API query

Run this on prom01:

sudo apt install -y jq
curl -s 'http://localhost:9090/api/v1/query?query=up' | jq '.status, .data.resultType, (.data.result | length)'

You should see:

"success"
"vector"
3

Three series: the prometheus job plus the two node targets. jq lets you pull out only the parts you want, so the next command prints each target on one line.

curl -s 'http://localhost:9090/api/v1/query?query=up' | jq -r '.data.result[] | [.metric.job, .metric.instance, .value[1]] | @tsv'

node        192.168.56.61:9100  1
prometheus  localhost:9090      1
node        192.168.56.60:9100  1

✅ Checkpoint: The API returns status “success” and all three targets show 1.

2Instant vector versus range vector

Queries with braces, quotes and brackets are easier through --data-urlencode, which does the URL encoding for you. Ask for the idle CPU counter of one core on node01, first as an instant vector and then with a 1 minute window.

Run this on prom01:

Q='node_cpu_seconds_total{instance="192.168.56.61:9100",cpu="0",mode="idle"}'
curl -s localhost:9090/api/v1/query --data-urlencode "query=$Q" | jq -c '.data.resultType, .data.result[0].value'
curl -s localhost:9090/api/v1/query --data-urlencode "query=$Q[1m]" | jq -c '.data.resultType, .data.result[0].values'

You should see this (your timestamps and numbers will differ):

"vector"
[1794112725.829,"758.61"]
"matrix"
[[1794112725.829,"758.61"],[1794112740.829,"773.09"],[1794112755.829,"787.59"],[1794112770.829,"802.19"]]

The instant vector has one pair: a Unix timestamp and the value as a string. The range vector has every scrape in the window, 15 seconds apart, and the counter grows by about 14.5 seconds each time because core 0 sat idle for nearly all of those 15 seconds. Note that values come back as strings in JSON, so convert them if a script does maths.

✅ Checkpoint: You saw resultType “vector” with one value and “matrix” with four or so values 15 s apart.

3Filter with the four label matchers

Run this on prom01:

q(){ curl -s localhost:9090/api/v1/query --data-urlencode "query=$1" | jq -r '.data.result[] | [.metric.instance, (.metric.role // "-"), .value[1]] | @tsv'; }
q 'up{job="node"}'
q 'up{job="node", role!="monitoring"}'
q 'up{instance=~"192.168.56.6[01]:9100"}'
q 'up{instance!~".*:9100"}'

You should see:

192.168.56.61:9100  app         1
192.168.56.60:9100  monitoring  1
192.168.56.61:9100  app         1
192.168.56.61:9100  app         1
192.168.56.60:9100  monitoring  1
localhost:9090      -           1

The small q shell function saves typing for the rest of the lab (it lasts until you close the SSH session). The last query keeps only targets whose instance does not end in :9100, which leaves Prometheus itself. Keep the role labels from Lesson 4 in mind, because they let you write role="app" instead of listing IP addresses.

✅ Checkpoint: Each matcher returned the targets you expected, and role!=”monitoring” left only node01.

4CPU busy % per instance

node_cpu_seconds_total counts seconds spent in each mode (idle, user, system, iowait and so on) for each core. The rate of the idle counter is the fraction of time the core was idle, so one minus the average idle fraction is the busy fraction.

Run this on prom01:

q '100 * (1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])))'
q 'count by (instance) (node_cpu_seconds_total{mode="idle"})'

You should see this (your numbers will differ):

192.168.56.61:9100  -  1.84
192.168.56.60:9100  -  4.61
192.168.56.61:9100  -  1
192.168.56.60:9100  -  2

The role column shows “-” now because by (instance) throws every other label away. The second query counts idle series per instance, which is a neat way to get the number of CPUs: 1 on node01 and 2 on prom01, matching the VM settings from Lesson 1. If you want to keep job and role, write avg without (cpu, mode) (...) instead.

✅ Checkpoint: You get a CPU busy % per instance, and the CPU count matches your VM settings.

5Memory available % and disk used %

Run this on prom01:

q '100 * node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes'
FS='fstype!~"tmpfs|overlay|squashfs"'
curl -s localhost:9090/api/v1/query --data-urlencode "query=100 * (1 - node_filesystem_avail_bytes{$FS} / node_filesystem_size_bytes{$FS})" \
  | jq -r '.data.result[] | [.metric.instance, .metric.mountpoint, (.value[1] | tonumber | floor)] | @tsv'

You should see this (your numbers will differ):

192.168.56.61:9100  app         71.92
192.168.56.60:9100  monitoring  48.37
192.168.56.61:9100  /      38
192.168.56.61:9100  /boot  10
192.168.56.60:9100  /      44
192.168.56.60:9100  /boot  10

Use MemAvailable, not MemFree: Linux keeps spare memory busy as page cache, so MemFree looks frighteningly low on a healthy server, and MemAvailable is the kernel’s own estimate of what programs can still get. For disk we use avail_bytes (space a normal user can write) rather than free_bytes, which also counts the 5% ext4 keeps for root. Both sides of the disk division have the same labels, so no on() is needed.

✅ Checkpoint: You see memory available % for both VMs and disk used % for / and /boot only, with no tmpfs rows.

6Network bytes per second: rate, irate and increase

Run this on prom01:

q 'rate(node_network_receive_bytes_total{device="enp0s8"}[5m])'
q 'irate(node_network_receive_bytes_total{device="enp0s8"}[5m])'
q 'increase(node_network_receive_bytes_total{device="enp0s8"}[1h])'

You should see this (your numbers will differ):

192.168.56.61:9100  app         412.36
192.168.56.60:9100  monitoring  1873.05
192.168.56.61:9100  app         398.4
192.168.56.60:9100  monitoring  2240.67
192.168.56.61:9100  app         1481203.11
192.168.56.60:9100  monitoring  6735418.52

node01 receives about 400 bytes a second on enp0s8: that’s Prometheus scraping it every 15 seconds. irate() uses only the last two samples, so it jumps around more and is good for spotting short spikes on a zoomed-in graph. rate() averages the whole window and is what you want in alerts and recording rules. The increase() result is bytes in the last hour, which is about 1.4 MB for node01; in the Graph tab multiply rate() by 8 if your team thinks in bits per second.

✅ Checkpoint: rate, irate and increase all return two series for enp0s8, and increase over 1h is roughly rate x 3600.

7Look back in time with offset and forward with predict_linear

Run this on prom01:

q 'node_load1'
q 'node_load1 offset 10m'
q 'predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[1h], 24*3600)'

You should see this (your numbers will differ):

192.168.56.61:9100  app         0.02
192.168.56.60:9100  monitoring  0.11
192.168.56.61:9100  app         0
192.168.56.60:9100  monitoring  0.07
192.168.56.61:9100  app         8724713472
192.168.56.60:9100  monitoring  12941312000

offset 10m gives you the value as it was ten minutes ago, which is handy for “compared with yesterday” queries like rate(x[5m]) / rate(x[5m] offset 1d). predict_linear() fits a straight line through the last hour of free space and projects it 24 hours ahead, in bytes. A positive answer means the disk won’t fill in that time; Lesson 8 turns < 0 on this into the HostDiskWillFillIn24h alert.

predict_linear on a 1 hour window is easily fooled by one big file copy. For real alerts use a longer window such as [6h] and combine it with a "less than 20% space left" condition, as Lesson 8 does.

✅ Checkpoint: offset returns older values and predict_linear returns a positive number of bytes for / on both VMs.

8Comparisons, bool, topk and absent

Run this on prom01:

q 'node_load1 > 0.05'
q 'node_load1 > bool 0.05'
q 'topk(1, node_load1)'
q 'absent(up{job="node"})'
q 'absent(up{job="nodes"})'

You should see this (your numbers will differ):

192.168.56.60:9100  monitoring  0.11
192.168.56.61:9100  app         0
192.168.56.60:9100  monitoring  1
192.168.56.60:9100  monitoring  0.11
                    -           1

Without bool the comparison drops node01 because its load is below 0.05; with bool both come back as 0 or 1. absent(up{job="node"}) prints nothing because the series exist. absent(up{job="nodes"}) returns one series with value 1 and the label job="nodes" (its instance column is empty, because it has no instance label), which is how you alert when a whole job vanishes from the config or service discovery, a case up == 0 can't catch because there are no up series left to be 0.

✅ Checkpoint: bool returned 0/1 for both VMs, and absent() returned 1 only for the job name that doesn't exist.

9A first histogram_quantile

A histogram metric comes as a set of _bucket series with an le ("less than or equal") label, plus _sum and _count. Prometheus exposes one for its own HTTP API, so you can ask how long its query endpoint takes for 90% of requests.

Run this on prom01:

curl -s localhost:9090/api/v1/query --data-urlencode \
  'query=histogram_quantile(0.9, sum by (le) (rate(prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query"}[5m])))' \
  | jq -c '.data.result'

You should see this (your timestamp and number will differ):

[{"metric":{},"value":[1794113021.402,"0.09000000000000001"]}]

About 90 ms for the 90th percentile. Rate the buckets first, then sum by le (keep le or the function has nothing to work with), then take the quantile. The answer is an estimate inside a bucket boundary (this histogram's buckets are 0.1, 0.2, 0.4 and so on), so 0.09 means "somewhere under 0.1 s".

✅ Checkpoint: histogram_quantile returned one value below 1 second.

10Graph tab habits that save time

From your laptop open https://prom01.lab/, go to Query, paste the CPU query from step 4 and switch to the Graph tab. Set the range to 1h, then change [5m] to [1m] and back: a short window looks noisy and a long one looks smooth but late. Open the Explain tab (next to Table and Graph in the Prometheus 3 UI) to see how the expression breaks down, and open the three-dot menu at the right end of the query box and pick Explore metrics to search metric names instead of guessing.

One habit worth building: always start a new query with only the selector, check the table shows the series and labels you expect, and add functions one layer at a time. The table view also tells you how many series you're about to graph, so you don't freeze your browser with 3,000 lines.

✅ Checkpoint: You see two CPU lines in the Graph tab, and you know where Explain and Explore metrics are.