Prometheus Alertmanager: routing alerts to email

Everything runs on prom01 unless the step says otherwise. Alertmanager will listen on 9093 and Mailpit catches mail on 127.0.0.1:1025 and shows it on a web page at 8025. Take a snapshot of prom01 first (call it “before-alertmanager”) so you can roll back if something goes wrong.

1Download and verify Alertmanager

Run this on prom01:

VER=0.34.2
cd /tmp
wget -q https://github.com/prometheus/alertmanager/releases/download/v$VER/alertmanager-$VER.linux-amd64.tar.gz
wget -q https://github.com/prometheus/alertmanager/releases/download/v$VER/sha256sums.txt
sha256sum -c --ignore-missing sha256sums.txt
tar xzf alertmanager-$VER.linux-amd64.tar.gz
ls alertmanager-$VER.linux-amd64

You should see this:

alertmanager-0.34.2.linux-amd64.tar.gz: OK
LICENSE  NOTICE  alertmanager  alertmanager.yml  amtool

The checksum line must say OK before you go further. The tarball holds two programs: alertmanager (the server) and amtool (the command-line client that checks config, lists alerts and manages silences).

✅ Checkpoint: sha256sum says OK and the folder holds alertmanager and amtool.

2Create the user, folders and binaries

Paste this whole block:

VER=0.34.2
sudo useradd --system --no-create-home --shell /usr/sbin/nologin alertmanager
sudo install -m 0755 /tmp/alertmanager-$VER.linux-amd64/alertmanager /tmp/alertmanager-$VER.linux-amd64/amtool /usr/local/bin/
sudo mkdir -p /etc/alertmanager /var/lib/alertmanager
sudo chown alertmanager:alertmanager /var/lib/alertmanager
alertmanager --version | head -1
amtool --version | head -1

You should see this:

alertmanager, version 0.34.2 (branch: HEAD, revision: e8349048265ccc0871c3d0d668b05eacc68baf50)
amtool, version 0.34.2 (branch: HEAD, revision: e8349048265ccc0871c3d0d668b05eacc68baf50)

Same pattern as Prometheus in Lesson 2: a system user with no shell, binaries owned by root, and only the data folder writable by the service.

✅ Checkpoint: Both alertmanager and amtool report version 0.34.2.

3Install Mailpit as a local mail catcher

Mailpit is a small open-source SMTP server that accepts every mail and shows it on a web page instead of delivering it. It’s perfect for testing alert email without a real mail account.

Paste this whole block:

MPVER=1.33.1
cd /tmp
wget -q https://github.com/axllent/mailpit/releases/download/v$MPVER/mailpit-linux-amd64.tar.gz
tar xzf mailpit-linux-amd64.tar.gz mailpit
sudo install -m 0755 mailpit /usr/local/bin/mailpit
sudo useradd --system --no-create-home --shell /usr/sbin/nologin mailpit
sudo mkdir -p /var/lib/mailpit && sudo chown mailpit:mailpit /var/lib/mailpit
sudo tee /etc/systemd/system/mailpit.service >/dev/null <<'EOF'
[Unit]
Description=Mailpit test mail server
After=network-online.target

[Service]
User=mailpit
Group=mailpit
ExecStart=/usr/local/bin/mailpit --smtp 127.0.0.1:1025 --listen 0.0.0.0:8025 --database /var/lib/mailpit/mailpit.db
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now mailpit
sudo journalctl -u mailpit -n 4 --no-pager

You should see this (each line starts with the date and time, cut here as “…”):

... level=info msg="[smtpd] starting on 127.0.0.1:1025 (no encryption)"
... level=warning msg="[http] listening on 0.0.0.0:8025 without --ui-auth-file or --allowed-hosts; the API is reachable from any host that can route to this interface"
... level=info msg="[http] starting on 0.0.0.0:8025"
... level=info msg="[http] accessible via http://0.0.0.0:8025/"

SMTP listens only on localhost, so only programs on prom01 can send mail through it. The web page listens on all interfaces, and Mailpit warns about that; in step 6 ufw lets only your laptop reach it. Note “(no encryption)”: Mailpit doesn’t offer STARTTLS unless you give it a certificate, which matters in the Break It section.

✅ Checkpoint: mailpit.service is running and the journal shows SMTP on 127.0.0.1:1025.

4Write alertmanager.yml and check it

Paste this whole block:

sudo tee /etc/alertmanager/alertmanager.yml >/dev/null <<'EOF'
global:
  smtp_smarthost: '127.0.0.1:1025'
  smtp_from: 'alertmanager@prom01.lab'
  smtp_require_tls: false

route:
  receiver: team-email
  group_by: ['alertname', 'job']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

receivers:
  - name: team-email
    email_configs:
      - to: 'oncall@lab.local'
        send_resolved: true

inhibit_rules:
  - source_matchers: [alertname="InstanceDown"]
    target_matchers: [severity="warning"]
    equal: [instance]
EOF
sudo chown root:alertmanager /etc/alertmanager/alertmanager.yml
sudo chmod 640 /etc/alertmanager/alertmanager.yml
amtool check-config /etc/alertmanager/alertmanager.yml

You should see this:

Checking '/etc/alertmanager/alertmanager.yml'  SUCCESS
Found:
 - global config
 - route
 - 1 inhibit rules
 - 1 receivers
 - 0 templates

smtp_require_tls: false is needed because Alertmanager insists on STARTTLS by default, and Mailpit on localhost doesn’t offer it. With a real mail server you’d leave TLS on and add smtp_auth_username and smtp_auth_password_file. Email receivers don’t send “resolved” mails unless you ask, hence send_resolved: true.

The file is readable by the alertmanager group only (640), because at work it holds SMTP passwords and webhook URLs. If amtool says permission denied as student, run it with sudo.

✅ Checkpoint: amtool check-config prints SUCCESS with 1 receiver and 1 inhibit rule.

5Run Alertmanager as a systemd service

Paste this whole block:

sudo tee /etc/systemd/system/alertmanager.service >/dev/null <<'EOF'
[Unit]
Description=Prometheus Alertmanager
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=alertmanager
Group=alertmanager
ExecStart=/usr/local/bin/alertmanager \
  --config.file=/etc/alertmanager/alertmanager.yml \
  --storage.path=/var/lib/alertmanager \
  --web.listen-address=:9093 \
  --web.external-url=http://prom01.lab:9093/ \
  --cluster.listen-address=
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now alertmanager
sudo journalctl -u alertmanager -n 4 --no-pager -o cat
curl -s http://localhost:9093/-/ready; echo

You should see this (your times will differ):

time=2026-11-08T17:24:42.131Z level=INFO source=app.go:194 msg="Starting Alertmanager" version="(version=0.34.2, branch=HEAD, revision=e8349048265ccc0871c3d0d668b05eacc68baf50)"
time=2026-11-08T17:24:42.138Z level=INFO source=app.go:196 msg="Build context" build_context="(go=go1.26.10, platform=linux/amd64, user=root@bb54f2fc4768, date=20261029-09:50:10, tags=netgo)"
time=2026-11-08T17:24:42.678Z level=INFO source=tls_config.go:372 msg="Listening on" address=[::]:9093
time=2026-11-08T17:24:42.679Z level=INFO source=tls_config.go:375 msg="TLS is disabled." http2=false address=[::]:9093
OK

-o cat prints only the messages, without the date, host name and PID that journalctl normally puts in front. address=[::]:9093 is one socket that takes both IPv6 and IPv4 connections on port 9093. IPv6 is on by default in Ubuntu 26.04, so this is the line you’ll get. On a VM with IPv6 switched off, the same two lines say address=0.0.0.0:9093.

An empty --cluster.listen-address= switches off the gossip port 9094, which a single Alertmanager doesn’t need. --web.external-url is the address put into the “View in Alertmanager” link of every email, so it must be a name your laptop can open.

✅ Checkpoint: alertmanager.service is active and /-/ready returns OK.

6Open 9093 and 8025 to your laptop only

Run this on prom01:

sudo ufw allow from 192.168.56.1 to any port 9093 proto tcp
sudo ufw allow from 192.168.56.1 to any port 8025 proto tcp
sudo ufw status | grep -E '9093|8025'

You should see this:

9093/tcp                   ALLOW       192.168.56.1
8025/tcp                   ALLOW       192.168.56.1

Now browse from the laptop to http://prom01.lab:9093 for the Alertmanager UI and http://prom01.lab:8025 for the Mailpit inbox. Alertmanager has no login of its own, so at work you’d put it behind the same Nginx and basic auth you built for Prometheus in Lesson 3.

✅ Checkpoint: Both pages open from your laptop and nothing else can reach those ports.

7Point Prometheus at Alertmanager

Open /etc/prometheus/prometheus.yml with sudo nano and add the alerting block above rule_files, plus a scrape job for Alertmanager’s own metrics at the end of scrape_configs.

Add these parts:

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['localhost:9093']

# ...under scrape_configs:
  - job_name: alertmanager
    static_configs:
      - targets: ['localhost:9093']

Then run this:

promtool check config /etc/prometheus/prometheus.yml
curl -s -X POST http://localhost:9090/-/reload
curl -s http://localhost:9090/api/v1/alertmanagers | jq -c .data

You should see:

Checking /etc/prometheus/prometheus.yml
  SUCCESS: 2 rule files found
 SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax

Checking /etc/prometheus/rules/alerts.yml
  SUCCESS: 4 rules found

Checking /etc/prometheus/rules/recording.yml
  SUCCESS: 3 rules found

{"activeAlertmanagers":[{"url":"http://localhost:9093/api/v2/alerts"}],"droppedAlertmanagers":[]}

activeAlertmanagers lists the address Prometheus will post alerts to. If it’s empty, the alerting block is in the wrong place or indented wrongly. On Status > Target health you’ll also see the new alertmanager job as UP.

✅ Checkpoint: /api/v1/alertmanagers shows http://localhost:9093/api/v2/alerts as active.

8Fire InstanceDown and read the email

Run this on node01:

sudo systemctl stop node_exporter

Wait about two minutes, then run this on prom01:

curl -s http://localhost:8025/api/v1/messages | jq -c '.messages[] | {Subject, From: .From.Address, To: [.To[].Address]}'

You should see:

{"Subject":"[FIRING:1] InstanceDown node (192.168.56.61:9100 app critical)","From":"alertmanager@prom01.lab","To":["oncall@lab.local"]}

The timing adds up: about 15 seconds for the failed scrape and first evaluation, one minute of for, then 30 seconds of group_wait. The subject shows the group labels (InstanceDown node) and, in brackets, the labels that differ. Open the mail in the Mailpit page to see the full body:

1 alert for alertname=InstanceDown job=node

View in Alertmanager ( http://prom01.lab:9093/#/alerts?receiver=team-email ) *[1] Firing* *Labels*
alertname = InstanceDown
instance = 192.168.56.61:9100
job = node
role = app
severity = critical
*Annotations*
description = 192.168.56.61:9100 (job node) has not answered scrapes for more than 1 minute.
summary = Instance 192.168.56.61:9100 is down
Source ( https://prom01.lab/graph?g0.expr=up%7Bjob%3D%22node%22%7D+%3D%3D+0&g0.tab=1 )

✅ Checkpoint: Mailpit has a [FIRING:1] InstanceDown email for 192.168.56.61:9100.

9Watch inhibition mute a warning

While node01 is still down, pretend its memory warning also fired. amtool alert add posts a test alert straight to Alertmanager. First save the URL so you don’t type it every time:

Run this on prom01:

mkdir -p ~/.config/amtool
echo 'alertmanager.url: http://localhost:9093' > ~/.config/amtool/config.yml
amtool alert add HostLowMemory severity=warning job=node instance=192.168.56.61:9100 --annotation='summary="Low memory test"'
amtool alert query
amtool alert query --inhibited

You should see this (your times will differ):

Alertname     Starts At                Summary                             State
InstanceDown  2026-11-08 05:41:21 UTC  Instance 192.168.56.61:9100 is down  active
Alertname      Starts At                Summary          State
HostLowMemory  2026-11-08 05:44:18 UTC  Low memory test  suppressed  

Plain alert query hides muted alerts, so only InstanceDown appears. With --inhibited you see HostLowMemory as suppressed, because a firing InstanceDown has the same instance label. No email goes out for it.

Put the value in quotes inside --annotation; newer amtool versions warn about unquoted values with spaces.

✅ Checkpoint: HostLowMemory for 192.168.56.61:9100 shows as suppressed and no email arrives for it.

10Silence node01 during planned work, then bring it back

Run this on prom01:

amtool silence add alertname=InstanceDown instance=192.168.56.61:9100 --duration=2h --comment="node01 maintenance, patching"
amtool silence query
amtool alert query --silenced

You should see this (your silence ID and times will differ):

13d87691-aa9b-4656-a893-e00715068e29
ID                                    Matchers                                                Ends At                  Created By  Comment
13d87691-aa9b-4656-a893-e00715068e29  alertname="InstanceDown" instance="192.168.56.61:9100"  2026-11-08 07:45:07 UTC  student     node01 maintenance, patching
Alertname     Starts At                Summary                             State
InstanceDown  2026-11-08 05:41:21 UTC  Instance 192.168.56.61:9100 is down  suppressed  

The first line is the silence ID. Created By defaults to your Linux user name; use --author to set a real name. While the silence lasts, no reminders go out after repeat_interval.

Now finish the “maintenance”, expire the silence and start node01’s exporter:

Run this (the silence expire on prom01, the start on node01):

amtool silence expire $(amtool silence query -q)
# on node01:
sudo systemctl start node_exporter

silence query -q prints only IDs, which is handy for scripts. Within about five minutes (the group_interval) Mailpit gets a second mail whose subject starts with [RESOLVED].

✅ Checkpoint: The silence is gone, node01 is UP and Mailpit holds both a FIRING and a RESOLVED InstanceDown email.