Prometheus recording rules

Everything runs on prom01. You’ll create /etc/prometheus/rules/recording.yml with the three rules from the shared lab plan, confirm prometheus.yml already loads that folder, then write a unit test next to it. Keep the rule names as written, because Lessons 8 and 10 use them.

1Check the rules folder and the rule_files glob

Run this on prom01:

ls -ld /etc/prometheus/rules
grep -n -A1 'rule_files' /etc/prometheus/prometheus.yml
promtool check config /etc/prometheus/prometheus.yml

You should see this (your dates and line numbers may differ):

drwxr-xr-x 2 prometheus prometheus 4096 Nov  7 18:02 /etc/prometheus/rules
5:rule_files:
6-  - /etc/prometheus/rules/*.yml
Checking /etc/prometheus/prometheus.yml
 SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax

Lesson 2 created the folder and wrote the glob /etc/prometheus/rules/*.yml into prometheus.yml, so every .yml file you drop in the folder is loaded on the next reload (Lesson 8 adds alerts.yml next to today’s file). An empty folder is fine, which is why promtool doesn’t mention any rule files yet. If your line numbers differ because of the storage block from Lesson 6, that’s expected.

✅ Checkpoint: /etc/prometheus/rules exists, owned by prometheus, and rule_files lists /etc/prometheus/rules/*.yml.

2Write the recording rules

Paste this whole block:

sudo tee /etc/prometheus/rules/recording.yml >/dev/null <<'EOF'
groups:
  - name: node-recording
    interval: 30s
    rules:
      # share of time the CPUs were busy, 0 to 1, one series per server
      - record: instance:node_cpu_utilisation:rate5m
        expr: 1 - avg without (cpu, mode) (rate(node_cpu_seconds_total{job="node", mode="idle"}[5m]))

      # share of memory still available to programs, 0 to 1
      - record: instance:node_memory_available:ratio
        expr: node_memory_MemAvailable_bytes{job="node"} / node_memory_MemTotal_bytes{job="node"}

      # share of each real filesystem still free for normal users, 0 to 1
      - record: instance:node_filesystem_avail:ratio
        expr: |
          node_filesystem_avail_bytes{job="node", fstype!~"tmpfs|overlay|squashfs"}
            / node_filesystem_size_bytes{job="node", fstype!~"tmpfs|overlay|squashfs"}
EOF
sudo chown prometheus:prometheus /etc/prometheus/rules/recording.yml

All three store ratios between 0 and 1 rather than percentages; that’s the Prometheus convention, and Grafana can show them as percent with a unit setting. avg without (cpu, mode) keeps job, instance and role, so you can still filter by role="app" later. The expr: | form lets a long expression run over several lines.

✅ Checkpoint: /etc/prometheus/rules/recording.yml exists, owned by prometheus, with one group and three rules.

3Check the rules and the config with promtool

Run this on prom01:

promtool check rules /etc/prometheus/rules/recording.yml
promtool check config /etc/prometheus/prometheus.yml

You should see this:

Checking /etc/prometheus/rules/recording.yml
  SUCCESS: 3 rules found

Checking /etc/prometheus/prometheus.yml
  SUCCESS: 1 rule files found
 SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax

Checking /etc/prometheus/rules/recording.yml
  SUCCESS: 3 rules found

check config also follows the rule_files glob and checks each file it finds, so in practice one command covers both. Both commands parse every PromQL expression too, so a missing [5m] is caught here and not at 2 am.

✅ Checkpoint: Both commands print SUCCESS and “3 rules found”.

4Reload and look at the Rules page

Run this on prom01:

curl -s -X POST localhost:9090/-/reload
sleep 35
curl -s 'localhost:9090/api/v1/rules?type=record' | jq '.data.groups[] | {name, file, interval, rules: [.rules[] | {name, health, lastError}]}'

You should see:

{
  "name": "node-recording",
  "file": "/etc/prometheus/rules/recording.yml",
  "interval": 30,
  "rules": [
    {
      "name": "instance:node_cpu_utilisation:rate5m",
      "health": "ok",
      "lastError": null
    },
    {
      "name": "instance:node_memory_available:ratio",
      "health": "ok",
      "lastError": null
    },
    {
      "name": "instance:node_filesystem_avail:ratio",
      "health": "ok",
      "lastError": null
    }
  ]
}

Health “ok” means the last evaluation ran without an error; it does not mean the rule returned data, as you’ll see in the break-it section. In the browser, https://prom01.lab/ under Status > Rule health shows the same list with each rule’s last evaluation time and duration.

✅ Checkpoint: The API lists group node-recording with three rules, all health “ok”.

5Query the recorded series

Run this on prom01:

q(){ curl -s localhost:9090/api/v1/query --data-urlencode "query=$1" | jq -r '.data.result[] | [.metric.__name__, .metric.instance, (.metric.mountpoint // "-"), (.value[1] | tonumber * 1000 | round / 1000)] | @tsv'; }
q 'instance:node_cpu_utilisation:rate5m'
q 'instance:node_memory_available:ratio'
q 'instance:node_filesystem_avail:ratio{mountpoint="/"}'

You should see this (your values will differ):

instance:node_cpu_utilisation:rate5m   192.168.56.61:9100  -  0.018
instance:node_cpu_utilisation:rate5m   192.168.56.60:9100  -  0.046
instance:node_memory_available:ratio   192.168.56.61:9100  -  0.719
instance:node_memory_available:ratio   192.168.56.60:9100  -  0.484
instance:node_filesystem_avail:ratio   192.168.56.61:9100  /  0.62
instance:node_filesystem_avail:ratio   192.168.56.60:9100  /  0.56

These match what the long queries from Lesson 5 gave you, as ratios instead of percentages. In the Graph tab, compare instance:node_cpu_utilisation:rate5m with the raw expression for the last 15 minutes: the lines overlap, and the recorded one has points every 30 seconds starting from the moment you reloaded.

✅ Checkpoint: All three recorded names return one series per instance (one per mount for the filesystem rule).

6Use recorded series in new queries

Recorded series are ordinary series, so you can build on them. These are queries Lesson 8 turns into alerts and Lesson 10 into dashboard panels:

Run this on prom01:

q '100 * instance:node_cpu_utilisation:rate5m{role="app"}'
q 'instance:node_filesystem_avail:ratio < 0.20'
q 'topk(1, 1 - instance:node_memory_available:ratio)'
q 'max_over_time(instance:node_cpu_utilisation:rate5m[1h])'

You should see this (your values will differ):

192.168.56.61:9100  -  1.8
192.168.56.60:9100  -  0.516
192.168.56.61:9100  -  0.031
192.168.56.60:9100  -  0.072

The second query prints nothing because no filesystem is below 20% free, which is the healthy answer. Notice the fourth query: max_over_time() over a recorded series is cheap and needs no subquery, which was the expensive fix from Lesson 5’s break-it section.

✅ Checkpoint: You get CPU % for node01 only, nothing for the 20% disk filter, and a peak CPU value for each server over the last hour.

7Write a unit test for the rules

promtool test rules starts an in-memory Prometheus, feeds it series you describe, runs your rules and compares the results. The notation '0+7.5x40' means “start at 0 and add 7.5 each step, 40 times”, and '256000000x40' means the same value 41 times.

Paste this whole block:

sudo tee /etc/prometheus/rules/recording_test.yml.test >/dev/null <<'EOF'
rule_files:
  - recording.yml

evaluation_interval: 30s

tests:
  - interval: 15s
    input_series:
      # node01 has 1 CPU, idle 7.5 of every 15 seconds = 50% busy
      - series: 'node_cpu_seconds_total{job="node", instance="192.168.56.61:9100", cpu="0", mode="idle"}'
        values: '0+7.5x40'
      - series: 'node_memory_MemAvailable_bytes{job="node", instance="192.168.56.61:9100"}'
        values: '256000000x40'
      - series: 'node_memory_MemTotal_bytes{job="node", instance="192.168.56.61:9100"}'
        values: '1024000000x40'
    promql_expr_test:
      - expr: instance:node_cpu_utilisation:rate5m
        eval_time: 10m
        exp_samples:
          - labels: 'instance:node_cpu_utilisation:rate5m{job="node", instance="192.168.56.61:9100"}'
            value: 0.5
      - expr: instance:node_memory_available:ratio
        eval_time: 10m
        exp_samples:
          - labels: 'instance:node_memory_available:ratio{job="node", instance="192.168.56.61:9100"}'
            value: 0.25
EOF
cd /etc/prometheus/rules && promtool test rules recording_test.yml.test

You should see this:

Processing test file: recording_test.yml.test
  SUCCESS

The test file ends in .test on purpose: the *.yml glob in prometheus.yml would otherwise try to load it as a rule file and the reload would fail. Paths in rule_files inside the test are relative to the test file, which is why you cd into the folder. Pick input values that give exact answers, because the test compares floats digit for digit and 0.2 can come out as 0.20000000000000007.

✅ Checkpoint: promtool test rules prints SUCCESS for recording_test.yml.test.

8Make check and test part of every change

From now on, every rule change goes through the same three commands before the reload. Put them in a small script so nobody skips a step:

Run this on prom01:

sudo tee /usr/local/bin/prom-rules-reload >/dev/null <<'EOF'
#!/bin/bash
set -e
cd /etc/prometheus/rules
promtool check config /etc/prometheus/prometheus.yml
for t in *.test; do [ -e "$t" ] && promtool test rules "$t"; done
curl -sf -X POST localhost:9090/-/reload && echo "reloaded OK"
EOF
sudo chmod 755 /usr/local/bin/prom-rules-reload
prom-rules-reload

You should see:

Checking /etc/prometheus/prometheus.yml
  SUCCESS: 1 rule files found
 SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax

Checking /etc/prometheus/rules/recording.yml
  SUCCESS: 3 rules found

Processing test file: recording_test.yml.test
  SUCCESS

reloaded OK

set -e stops the script at the first failure, so a broken file never reaches the reload, and curl -f makes a failed reload return an error too. At work the same commands usually run in a CI pipeline on every merge request to the monitoring Git repository.

✅ Checkpoint: prom-rules-reload prints SUCCESS for config, rules and tests, then “reloaded OK”.