Install Prometheus on Ubuntu as a systemd service

Restore prom01 to the base snapshot from Lesson 1 (or delete the test folder in your home), start it and SSH in as student. Every command in this lab runs on prom01.

1Download Prometheus and check the checksum

Run this on prom01:

VER=3.16.0
mkdir -p ~/dl && cd ~/dl
wget -q https://github.com/prometheus/prometheus/releases/download/v$VER/prometheus-$VER.linux-amd64.tar.gz
wget -q https://github.com/prometheus/prometheus/releases/download/v$VER/sha256sums.txt
sha256sum -c sha256sums.txt --ignore-missing

You should see:

prometheus-3.16.0.linux-amd64.tar.gz: OK

sha256sums.txt lists a checksum for every file in the release, and --ignore-missing tells sha256sum to check only the ones you downloaded. “OK” means the file you got is byte for byte the one the project published; “FAILED” means a broken or tampered download, so delete it and fetch it again.

✅ Checkpoint: sha256sum prints OK for prometheus-3.16.0.linux-amd64.tar.gz.

2Create the prometheus user and folders

Run this on prom01:

sudo useradd --system --no-create-home --shell /usr/sbin/nologin prometheus
sudo mkdir -p /etc/prometheus/rules /var/lib/prometheus
sudo chown -R prometheus:prometheus /etc/prometheus /var/lib/prometheus
id prometheus
ls -ld /etc/prometheus /var/lib/prometheus

You should see this (your UID and GID numbers and times will differ):

uid=996(prometheus) gid=995(prometheus) groups=995(prometheus)
drwxr-xr-x 3 prometheus prometheus 4096 Nov  8 05:20 /etc/prometheus
drwxr-xr-x 2 prometheus prometheus 4096 Nov  8 05:20 /var/lib/prometheus

--system gives a low UID (below 1000) and no password ageing, which is how service accounts are set up on Ubuntu. Your UID may be a different number, and that’s fine.

✅ Checkpoint: id prometheus works and both folders are owned by prometheus.

3Install the binaries and the first config

Paste this whole block:

cd ~/dl && tar xzf prometheus-$VER.linux-amd64.tar.gz
sudo install -m 755 -o root -g root prometheus-$VER.linux-amd64/prometheus prometheus-$VER.linux-amd64/promtool /usr/local/bin/
sudo tee /etc/prometheus/prometheus.yml >/dev/null <<'EOF'
global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - /etc/prometheus/rules/*.yml

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]
EOF
sudo chown prometheus:prometheus /etc/prometheus/prometheus.yml
prometheus --version

You should see:

prometheus, version 3.16.0 (branch: HEAD, revision: e2e28adeee2c5883d000fc82aca6008cfbf7e96d)
  build user:       root@880187fc0808
  build date:       20261105-15:13:16
  go version:       go1.27.3
  platform:         linux/amd64
  tags:             netgo,builtinassets

The config is shorter than the sample in the tarball on purpose, so you can read every line. global sets defaults for all jobs: scrape every 15 seconds (the built-in default is 1 minute) and run rules every 15 seconds. scrape_timeout is not set, so it stays at 10 seconds, and it must never be longer than the interval.

rule_files takes glob patterns, and an empty folder is fine, so Lessons 7 and 8 only need to drop files into /etc/prometheus/rules. Under scrape_configs each list item starting with - job_name is one job, and static_configs lists its targets as host:port. Prometheus adds /metrics and http:// itself.

✅ Checkpoint: The version check reports 3.16.0 and /etc/prometheus/prometheus.yml exists.

4Check the config with promtool

Run this on prom01:

promtool check config /etc/prometheus/prometheus.yml; echo "exit code: $?"

You should see this:

Checking /etc/prometheus/prometheus.yml
 SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax

exit code: 0

promtool parses the file the same way the server does, including any rule files it points to. Exit code 0 means valid and 1 means failed, which is handy in scripts and CI pipelines: you can block a Git merge if the check fails.

✅ Checkpoint: promtool prints SUCCESS and exit code 0.

5Write the systemd unit

Paste this whole block:

sudo tee /etc/systemd/system/prometheus.service >/dev/null <<'EOF'
[Unit]
Description=Prometheus monitoring server
Documentation=https://prometheus.io/docs/introduction/overview/
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=prometheus
Group=prometheus
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --storage.tsdb.retention.time=15d \
  --web.enable-lifecycle
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5s
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF
sudo systemd-analyze verify /etc/systemd/system/prometheus.service && echo "unit OK"

You should see this:

unit OK

The backslashes let one long ExecStart line wrap over several lines, and there must be nothing after them, not even a space. ExecReload sends SIGHUP to the main process, which Prometheus treats as “read your config again”. LimitNOFILE=65536 raises the open file limit, because a busy server keeps many block files and connections open.

Writing 15d for retention looks the same as the default, but writing it down means nobody has to guess, and Lesson 6 adds a size limit next to it.

✅ Checkpoint: systemd-analyze verify prints nothing and you see “unit OK”.

6Start it, enable it at boot and read the logs

Run this on prom01:

sudo systemctl daemon-reload
sudo systemctl enable --now prometheus
systemctl status prometheus --no-pager -n 0
sudo journalctl -u prometheus -n 5 --no-pager

You should see this (your times and PIDs will differ, and the long log lines are shortened):

Created symlink '/etc/systemd/system/multi-user.target.wants/prometheus.service' → '/etc/systemd/system/prometheus.service'.
● prometheus.service - Prometheus monitoring server
     Loaded: loaded (/etc/systemd/system/prometheus.service; enabled; preset: enabled)
     Active: active (running) since Sun 2026-11-08 05:24:02 UTC; 4s ago
 Invocation: 4c1f9a7e2b6d4e0f8a3c5b7d9e1f2a4c
       Docs: https://prometheus.io/docs/introduction/overview/
   Main PID: 1532 (prometheus)
      Tasks: 8 (limit: 2266)
     Memory: 38.6M (peak: 39.1M)
        CPU: 312ms
     CGroup: /system.slice/prometheus.service
             └─1532 /usr/local/bin/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/var/lib/prometheus --storage.tsdb.retention.time=15d --web.enable-lifecycle
Nov 08 05:24:02 prom01 prometheus[1532]: time=2026-11-08T05:24:02.497Z level=INFO source=main.go:1744 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
Nov 08 05:24:02 prom01 prometheus[1532]: time=2026-11-08T05:24:02.497Z level=INFO source=main.go:1148 msg="TSDB retention updated" duration=15d size=0B percentage=0
Nov 08 05:24:02 prom01 prometheus[1532]: time=2026-11-08T05:24:02.498Z level=INFO source=main.go:1786 msg="Completed loading of configuration file" db_storage=29.066µs ... filename=/etc/prometheus/prometheus.yml totalDuration=796.506µs
Nov 08 05:24:02 prom01 prometheus[1532]: time=2026-11-08T05:24:02.498Z level=INFO source=main.go:1506 msg="Server is ready to receive web requests."
Nov 08 05:24:02 prom01 prometheus[1532]: time=2026-11-08T05:24:02.498Z level=INFO source=manager.go:235 msg="Starting rule manager..." component="rule manager"

enable --now does two jobs in one: it links the unit into multi-user.target so it starts at boot, and starts it right away. In the log, each line has time=, level= (INFO, WARN, ERROR), source= (the Go file and line) and msg=, so you can grep for level=ERROR when something goes wrong.

✅ Checkpoint: systemctl shows “active (running)” and “enabled”, and the journal shows “Server is ready to receive web requests”.

7Check the health and ready endpoints

Run this on prom01:

curl -s http://localhost:9090/-/healthy; echo
curl -s http://localhost:9090/-/ready; echo
curl -s 'http://localhost:9090/api/v1/query?query=up'; echo
sudo ss -ltnp | grep 9090

You should see this (your timestamp and PID will differ):

Prometheus Server is Healthy.
Prometheus Server is Ready.
{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"up","instance":"localhost:9090","job":"prometheus"},"value":[1794115460.399,"1"]}]}}
LISTEN 0      4096         0.0.0.0:9090       0.0.0.0:*    users:(("prometheus",pid=1532,fd=8))

/-/healthy answers 200 as soon as the process is alive, while /-/ready waits until the WAL replay is done and queries will work. Load balancers and Kubernetes probes use them that way: healthy for “restart it if this fails”, ready for “send traffic now”. The API answer is the same up query you ran in the browser in Lesson 1, as JSON.

✅ Checkpoint: Both endpoints answer, and the API returns up with value “1”.

8Tour the web UI

On your laptop open http://192.168.56.60:9090. Prometheus 3 has a new interface; these are the pages you’ll use most.

Open Status > Command-line flags and type retention in the filter: you should see storage.tsdb.retention.time set to 15d. Then open Status > Configuration and note that Prometheus has filled in scrape_timeout: 10s even though you never wrote it.

Status > Configuration: the running config, with scrape_timeout: 10s and other defaults filled in

✅ Checkpoint: You opened Target health, Configuration and Command-line flags, and retention shows 15d.

9Reload the config two ways

Change the scrape interval to 30s, check it, reload with the HTTP endpoint, then put it back and reload with systemctl.

Run this on prom01:

sudo sed -i 's/scrape_interval: 15s/scrape_interval: 30s/' /etc/prometheus/prometheus.yml
promtool check config /etc/prometheus/prometheus.yml
curl -s -X POST http://localhost:9090/-/reload && echo "reload sent"
curl -s http://localhost:9090/api/v1/status/config | grep -o 'scrape_interval: [0-9a-z]*' | head -1
sudo sed -i 's/scrape_interval: 30s/scrape_interval: 15s/' /etc/prometheus/prometheus.yml
sudo systemctl reload prometheus
curl -s http://localhost:9090/metrics | grep '^prometheus_config_last_reload_successful'

You should see:

Checking /etc/prometheus/prometheus.yml
 SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax

reload sent
scrape_interval: 30s
prometheus_config_last_reload_successful 1

The POST works only because the unit has --web.enable-lifecycle; without it Prometheus answers “Lifecycle API is not enabled.” The systemctl reload sends SIGHUP through ExecReload. sudo journalctl -u prometheus -n 4 shows a “Completed loading of configuration file” line for each reload, and the PID in systemctl status doesn’t change, so the process never restarted.

✅ Checkpoint: Status > Configuration showed 30s after the first reload and 15s after the second, and prometheus_config_last_reload_successful is 1.

10Reboot test, snapshot, and how you would upgrade

Run this on prom01:

sudo reboot
# log in again after a minute, then:
systemctl is-active prometheus; systemctl is-enabled prometheus

You should see this:

active
enabled

Shut prom01 down and take a VirtualBox snapshot called prom-installed. Lesson 3 starts from it. When a new release comes out later, the upgrade on this layout looks like this (don’t run it now, 3.16.0 is current):

NEW=3.16.1                     # the version you are moving to
cd ~/dl && wget -q https://github.com/prometheus/prometheus/releases/download/v$NEW/prometheus-$NEW.linux-amd64.tar.gz
wget -q -O sha256sums-$NEW.txt https://github.com/prometheus/prometheus/releases/download/v$NEW/sha256sums.txt
sha256sum -c sha256sums-$NEW.txt --ignore-missing
tar xzf prometheus-$NEW.linux-amd64.tar.gz
prometheus-$NEW.linux-amd64/promtool check config /etc/prometheus/prometheus.yml
sudo systemctl stop prometheus
sudo cp /usr/local/bin/prometheus /usr/local/bin/prometheus.old
sudo install -m 755 prometheus-$NEW.linux-amd64/prometheus prometheus-$NEW.linux-amd64/promtool /usr/local/bin/
sudo systemctl start prometheus
prometheus --version | head -1; curl -s http://localhost:9090/-/ready

Checking your current config with the new promtool before you stop anything catches settings that the new version no longer accepts. If the new version misbehaves, stop it, copy prometheus.old back and start again, or restore the snapshot.

✅ Checkpoint: After the reboot Prometheus is active and enabled, and prom01 has a snapshot called prom-installed.