Prometheus alerting rules

All commands run on prom01 unless the step says node01. Your Prometheus already loads /etc/prometheus/rules/*.yml from Lesson 7, so you’ll add a second file next to recording.yml and reload. If jq isn’t installed yet, run sudo apt install -y jq first.

1Check the rules folder and the rule_files line

Run this:

ls -l /etc/prometheus/rules/
grep -A2 rule_files /etc/prometheus/prometheus.yml
grep evaluation_interval /etc/prometheus/prometheus.yml

You should see this (your dates will differ):

total 8
-rw-r--r-- 1 prometheus prometheus  816 Nov  8 09:41 recording.yml
-rw-r--r-- 1 root       root       1005 Nov  8 09:52 recording_test.yml.test
rule_files:
  - /etc/prometheus/rules/*.yml

  evaluation_interval: 15s

The glob means any new .yml file in that folder is picked up on the next reload, with no change to prometheus.yml. The 15 second evaluation interval is the clock your for times are measured against.

✅ Checkpoint: recording.yml (and its .test file from Lesson 7) is in /etc/prometheus/rules and rule_files points at /etc/prometheus/rules/*.yml.

2Write the four alerting rules

Paste this whole block:

sudo tee /etc/prometheus/rules/alerts.yml >/dev/null <<'EOF'
groups:
  - name: node-alerts
    rules:
      - alert: InstanceDown
        expr: up{job="node"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Instance {{ $labels.instance }} is down"
          description: "{{ $labels.instance }} (job {{ $labels.job }}) has not answered scrapes for more than 1 minute."

      - alert: HostHighCpu
        expr: 100 * (1 - avg by (instance, job) (rate(node_cpu_seconds_total{job="node",mode="idle"}[5m]))) > 80
        for: 5m
        keep_firing_for: 2m
        labels:
          severity: warning
        annotations:
          summary: "High CPU on {{ $labels.instance }}"
          description: "CPU use is {{ $value | humanize }}% on {{ $labels.instance }} for 5 minutes."

      - alert: HostLowMemory
        expr: node_memory_MemAvailable_bytes{job="node"} / node_memory_MemTotal_bytes{job="node"} < 0.10
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Low memory on {{ $labels.instance }}"
          description: "Only {{ $value | humanizePercentage }} of memory is available on {{ $labels.instance }}."

      - alert: HostDiskWillFillIn24h
        expr: |
          node_filesystem_avail_bytes{job="node",mountpoint="/"} / node_filesystem_size_bytes{job="node",mountpoint="/"} < 0.40
          and
          predict_linear(node_filesystem_avail_bytes{job="node",mountpoint="/"}[6h], 24 * 3600) < 0
        for: 30m
        labels:
          severity: warning
        annotations:
          summary: "Disk on {{ $labels.instance }} will fill within 24 hours"
          description: "{{ $labels.mountpoint }} on {{ $labels.instance }} has {{ $value | humanizePercentage }} left and is filling fast."
EOF
sudo chown prometheus:prometheus /etc/prometheus/rules/alerts.yml
promtool check rules /etc/prometheus/rules/alerts.yml

You should see this:

Checking /etc/prometheus/rules/alerts.yml
  SUCCESS: 4 rules found

A few details are worth reading slowly. The disk rule puts the “less than 40% free” part first, so {{ $value }} is the free ratio, which humanizePercentage turns into “26%”. The and keeps only filesystems that are both low and trending to zero, so a big empty disk with a temporary burst of writes stays quiet.

In the memory rule $value is a ratio too (0.05), and humanize in the CPU rule prints a plain number like 92.34.

If your Lesson 7 rule for instance:node_cpu_utilisation:rate5m keeps the job label, you can write HostHighCpu as instance:node_cpu_utilisation:rate5m{job="node"} > 0.8. Alerting on recorded series is common at work because the heavy maths runs once; we keep the raw query here so the lesson works on its own.

✅ Checkpoint: promtool reports SUCCESS: 4 rules found for alerts.yml.

3Reload Prometheus and confirm the rules loaded

Run this:

curl -s -X POST http://localhost:9090/-/reload
curl -s http://localhost:9090/api/v1/rules?type=alert | jq -c '.data.groups[].rules[] | {name, health, state, duration, keepFiringFor}'

You should see this:

{"name":"InstanceDown","health":"ok","state":"inactive","duration":60,"keepFiringFor":0}
{"name":"HostHighCpu","health":"ok","state":"inactive","duration":300,"keepFiringFor":120}
{"name":"HostLowMemory","health":"ok","state":"inactive","duration":300,"keepFiringFor":0}
{"name":"HostDiskWillFillIn24h","health":"ok","state":"inactive","duration":1800,"keepFiringFor":0}

The reload prints nothing when it works. duration is the for time in seconds, and health: ok means the last evaluation ran without a query error. Open https://prom01.lab/alerts in your browser (through the Nginx proxy from Lesson 3) and you’ll see the group node-alerts with four green inactive rules; Status > Rule health shows the same rules with their last evaluation time.

✅ Checkpoint: All four alerting rules show health ok and state inactive.

4Fire InstanceDown by stopping node_exporter on node01

Run this on node01:

sudo systemctl stop node_exporter

Then run this on prom01 a few times over the next two minutes:

curl -s http://localhost:9090/api/v1/alerts | jq -c '.data.alerts[] | {alert: .labels.alertname, instance: .labels.instance, state, activeAt}'

In the first 15 to 30 seconds you should see this (your times will differ):

{"alert":"InstanceDown","instance":"192.168.56.61:9100","state":"pending","activeAt":"2026-11-08T04:41:21.583359519Z"}

And about a minute later:

{"alert":"InstanceDown","instance":"192.168.56.61:9100","state":"firing","activeAt":"2026-11-08T04:41:21.583359519Z"}

In my run the exporter stopped at 04:41:17, the scrape failed and the rule went pending at 04:41:21, and it turned to firing at about 04:42:21, one minute later as for: 1m asks. activeAt doesn’t change when the state moves to firing, because it records when the condition first became true. On the Alerts page the rule turns yellow for pending, then red for firing.

✅ Checkpoint: InstanceDown for 192.168.56.61:9100 went pending, then firing about a minute later.

5Read the full alert and the ALERTS series

Run this:

curl -s http://localhost:9090/api/v1/alerts | jq '.data.alerts[] | select(.labels.alertname=="InstanceDown")'
curl -s http://localhost:9090/api/v1/query --data-urlencode 'query=ALERTS' | jq -c '.data.result[].metric'

You should see this (your activeAt time will differ):

{
  "labels": {
    "alertname": "InstanceDown",
    "instance": "192.168.56.61:9100",
    "job": "node",
    "role": "app",
    "severity": "critical"
  },
  "annotations": {
    "description": "192.168.56.61:9100 (job node) has not answered scrapes for more than 1 minute.",
    "summary": "Instance 192.168.56.61:9100 is down"
  },
  "state": "firing",
  "activeAt": "2026-11-08T04:41:21.583359519Z",
  "value": "0e+00"
}
{"__name__":"ALERTS","alertname":"InstanceDown","alertstate":"firing","instance":"192.168.56.61:9100","job":"node","role":"app","severity":"critical"}

The alert kept role="app" from the scrape config and gained severity="critical" from the rule; Lesson 9 routes on those labels. value is 0 because that’s what up returned. ALERTS is a normal series, so you can graph ALERTS{alertstate="firing"} to see when alerts fired over the last week.

✅ Checkpoint: The annotations show the real instance address and ALERTS has a firing series for InstanceDown.

6Bring node01 back and watch the alert resolve

Run this on node01, then the query on prom01 after 30 seconds:

sudo systemctl start node_exporter
# on prom01:
curl -s http://localhost:9090/api/v1/alerts | jq '.data.alerts | length'

You should see this:

0

As soon as the next scrape succeeds, up is 1, the query returns nothing and the alert disappears from the active list. Prometheus sends Alertmanager a final copy with an end time so it can mail a “resolved” message, which you’ll see in Lesson 9.

✅ Checkpoint: No active alerts remain and the node target is UP again.

7Push CPU on node01 and watch HostHighCpu

Run this on node01:

sudo apt install -y stress-ng
stress-ng --cpu 1 --timeout 12m

And this on prom01 every minute or so:

curl -s http://localhost:9090/api/v1/alerts | jq -c '.data.alerts[] | {alert: .labels.alertname, state, value, d: .annotations.description}'

You should see this (your values and times will differ):

{"alert":"HostHighCpu","state":"pending","value":"8.005614035087719e+01","d":"CPU use is 80.06% on 192.168.56.61:9100 for 5 minutes."}
...
{"alert":"HostHighCpu","state":"firing","value":"9.999122807017544e+01","d":"CPU use is 99.99% on 192.168.56.61:9100 for 5 minutes."}

node01 has one vCPU, so one stress worker takes it to about 100%. Because the rule uses rate(...[5m]), the average climbs slowly and crosses 80% after a few minutes; then for: 5m adds five more, so expect firing roughly 8 to 10 minutes after you start. When stress-ng stops, the alert stays firing for another 2 minutes because of keep_firing_for.

If you’re short of time, change for: 5m to for: 1m for the test and put it back afterwards.

✅ Checkpoint: HostHighCpu fired for 192.168.56.61:9100 with a description showing a CPU number above 80.

8Unit-test the alerts with promtool test rules

Testing by stopping services is fine in a lab, but you can’t wait ten minutes for every change at work. promtool test rules feeds made-up series into your rules on a fake clock and checks which alerts fire, with which labels and text.

Paste this whole block:

sudo mkdir -p /etc/prometheus/tests
sudo tee /etc/prometheus/tests/alerts_test.yml >/dev/null <<'EOF'
rule_files:
  - ../rules/alerts.yml

evaluation_interval: 1m

tests:
  # 1. node01 stops answering at minute 2
  - interval: 1m
    input_series:
      - series: 'up{job="node", instance="192.168.56.61:9100"}'
        values: '1 1 0 0 0 0'
    alert_rule_test:
      - eval_time: 2m
        alertname: InstanceDown
        exp_alerts: []
      - eval_time: 4m
        alertname: InstanceDown
        exp_alerts:
          - exp_labels:
              severity: critical
              job: node
              instance: 192.168.56.61:9100
            exp_annotations:
              summary: "Instance 192.168.56.61:9100 is down"
              description: "192.168.56.61:9100 (job node) has not answered scrapes for more than 1 minute."

  # 2. memory drops from 50% to 5% available at minute 11
  - interval: 1m
    input_series:
      - series: 'node_memory_MemAvailable_bytes{job="node", instance="192.168.56.61:9100"}'
        values: '500000000x10 50000000x10'
      - series: 'node_memory_MemTotal_bytes{job="node", instance="192.168.56.61:9100"}'
        values: '1000000000x20'
    alert_rule_test:
      - eval_time: 14m
        alertname: HostLowMemory
        exp_alerts: []
      - eval_time: 17m
        alertname: HostLowMemory
        exp_alerts:
          - exp_labels:
              severity: warning
              job: node
              instance: 192.168.56.61:9100
            exp_annotations:
              summary: "Low memory on 192.168.56.61:9100"
              description: "Only 5% of memory is available on 192.168.56.61:9100."

  # 3. root disk on node01 loses 5 MB every minute (about 7 GB a day)
  - interval: 1m
    input_series:
      - series: 'node_filesystem_avail_bytes{job="node", instance="192.168.56.61:9100", mountpoint="/", fstype="ext4", device="/dev/sda2"}'
        values: '6000000000-5000000x480'
      - series: 'node_filesystem_size_bytes{job="node", instance="192.168.56.61:9100", mountpoint="/", fstype="ext4", device="/dev/sda2"}'
        values: '15000000000x480'
    alert_rule_test:
      - eval_time: 7h
        alertname: HostDiskWillFillIn24h
        exp_alerts:
          - exp_labels:
              severity: warning
              job: node
              instance: 192.168.56.61:9100
              mountpoint: /
              fstype: ext4
              device: /dev/sda2
            exp_annotations:
              summary: "Disk on 192.168.56.61:9100 will fill within 24 hours"
              description: "/ on 192.168.56.61:9100 has 26% left and is filling fast."
EOF
cd /etc/prometheus/tests && promtool test rules alerts_test.yml

You should see this:

Processing test file: alerts_test.yml
  SUCCESS

The values strings are shorthand: 500000000x10 means that value 11 times (the first plus 10 repeats), and 6000000000-5000000x480 starts at 6 GB and drops 5 MB a minute. In test 1, up goes to 0 at minute 2, so at minute 2 the alert is only pending (and pending alerts don’t count in exp_alerts), and at minute 4 it’s firing. In test 3 the disk is below 40% free after a few hours, and after 30 more minutes of the 6 hour trend pointing below zero the alert fires with 26% left.

✅ Checkpoint: promtool test rules prints SUCCESS for all three test groups.