Scrape targets and Node Exporter

Start prom01 (secured snapshot) and node01. Each step says which VM to use. Prometheus on prom01 now listens on 127.0.0.1 only, so you query it with curl localhost:9090 on prom01, or in the browser at https://prom01.lab/ with your password.

1Download and check Node Exporter on node01

Run this on node01:

VER=1.12.1
mkdir -p ~/dl && cd ~/dl
wget -q https://github.com/prometheus/node_exporter/releases/download/v$VER/node_exporter-$VER.linux-amd64.tar.gz
wget -q https://github.com/prometheus/node_exporter/releases/download/v$VER/sha256sums.txt
sha256sum -c sha256sums.txt --ignore-missing
tar xzf node_exporter-$VER.linux-amd64.tar.gz && ls node_exporter-$VER.linux-amd64

You should see this:

node_exporter-1.12.1.linux-amd64.tar.gz: OK
LICENSE  node_exporter  NOTICE

✅ Checkpoint: The checksum says OK and the folder has the node_exporter binary.

2Create the user, the textfile folder and install the binary

Run this on node01:

sudo useradd --system --no-create-home --shell /usr/sbin/nologin node_exporter
sudo install -m 755 -o root -g root node_exporter-$VER.linux-amd64/node_exporter /usr/local/bin/
sudo mkdir -p /var/lib/node_exporter/textfile_collector
sudo chown -R node_exporter:node_exporter /var/lib/node_exporter
node_exporter --version | head -1

You should see this:

node_exporter, version 1.12.1 (branch: HEAD, revision: 6044da783597cc3b57aef7580ddcdcff58a4ee99)

Node Exporter doesn’t need root, and it logs a warning if you run it as root. The textfile folder is where scripts will drop extra metrics in step 6.

✅ Checkpoint: id node_exporter works and node_exporter reports version 1.12.1.

3Write the systemd unit and start it

Paste this whole block on node01:

sudo tee /etc/systemd/system/node_exporter.service >/dev/null <<'EOF'
[Unit]
Description=Prometheus Node Exporter
Documentation=https://github.com/prometheus/node_exporter
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=node_exporter
Group=node_exporter
ExecStart=/usr/local/bin/node_exporter \
  --collector.systemd \
  --collector.textfile.directory=/var/lib/node_exporter/textfile_collector
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now node_exporter
sudo journalctl -u node_exporter --no-pager | grep -E 'Starting|Listening|TLS'

You should see this (your times and PID will differ):

Nov 08 07:02:11 node01 node_exporter[1288]: time=2026-11-08T07:02:11.468Z level=INFO source=node_exporter.go:217 msg="Starting node_exporter" version="(version=1.12.1, branch=HEAD, revision=6044da783597cc3b57aef7580ddcdcff58a4ee99)"
Nov 08 07:02:11 node01 node_exporter[1288]: time=2026-11-08T07:02:11.470Z level=INFO source=tls_config.go:372 msg="Listening on" address=[::]:9100
Nov 08 07:02:11 node01 node_exporter[1288]: time=2026-11-08T07:02:11.470Z level=INFO source=tls_config.go:375 msg="TLS is disabled." http2=false address=[::]:9100

--collector.systemd turns on a collector that is off by default and exports the state of every systemd unit, so you can alert when a service you care about is failed. --collector.textfile.directory points the textfile collector at your folder. The full list of enabled collectors is also in the journal, under “Enabled collectors”.

✅ Checkpoint: systemctl is-active node_exporter prints active and the journal shows “Listening on” with port 9100.

4Read your own /metrics page

Run this on node01:

curl -s localhost:9100/metrics | wc -l
curl -s localhost:9100/metrics | grep -E '^(# (HELP|TYPE) )?node_load1 '
curl -s localhost:9100/metrics | grep 'node_systemd_unit_state{name="ssh.service"'
curl -s localhost:9100/metrics | grep -E '^node_scrape_collector_success\{collector="(cpu|systemd|textfile)"'

After the line count from wc, you should see this (your load value will differ):

# HELP node_load1 1m load average.
# TYPE node_load1 gauge
node_load1 0.02
node_systemd_unit_state{name="ssh.service",state="activating",type="notify"} 0
node_systemd_unit_state{name="ssh.service",state="active",type="notify"} 1
node_systemd_unit_state{name="ssh.service",state="deactivating",type="notify"} 0
node_systemd_unit_state{name="ssh.service",state="failed",type="notify"} 0
node_systemd_unit_state{name="ssh.service",state="inactive",type="notify"} 0
node_scrape_collector_success{collector="cpu"} 1
node_scrape_collector_success{collector="systemd"} 1
node_scrape_collector_success{collector="textfile"} 1

The line count comes first, and on a small VM it’s in the hundreds or low thousands. The systemd collector writes one series per unit and state, with 1 on the current state. node_scrape_collector_success tells you whether each collector worked on this scrape; a 0 there means a collector is failing and its metrics are missing, even though up is still 1.

✅ Checkpoint: wc shows several hundred lines, node_load1 has HELP and TYPE lines, and all three collectors show success 1.

5Allow 9100 from prom01 only

Run this on node01:

sudo ufw allow OpenSSH
sudo ufw allow from 192.168.56.60 to any port 9100 proto tcp
sudo ufw enable
sudo ufw status numbered

You should see:

Rules updated
Rules updated (v6)
Rules updated
Command may disrupt existing ssh connections. Proceed with operation (y|n)? y
Firewall is active and enabled on system startup
Status: active

     To                         Action      From
     --                         ------      ----
[ 1] OpenSSH                    ALLOW IN    Anywhere
[ 2] 9100/tcp                   ALLOW IN    192.168.56.60
[ 3] OpenSSH (v6)               ALLOW IN    Anywhere (v6)

The /metrics page shows your kernel version, mount points, network interfaces and running services, so it shouldn’t be open to everyone. Now only prom01 can scrape it. Test from prom01, and from your laptop to see the difference:

Run this on prom01, then the same curl on your laptop:

curl -s -m 5 http://192.168.56.61:9100/metrics | grep '^node_load1 '

node_load1 0.04            (from prom01)
                           (from the laptop: nothing for 5 seconds, then curl gives up quietly with exit code 28)

✅ Checkpoint: prom01 gets node_load1 from node01, and the laptop times out.

6Publish your own metric with the textfile collector

Scripts like a nightly backup can’t run a web server, but they can write a file. Node Exporter reads every *.prom file in the folder on each scrape.

Paste this whole block on node01:

D=/var/lib/node_exporter/textfile_collector
cat <<EOF | sudo -u node_exporter tee $D/backup.prom.$$ >/dev/null
# HELP backup_last_success_timestamp_seconds Unix time of the last good backup.
# TYPE backup_last_success_timestamp_seconds gauge
backup_last_success_timestamp_seconds $(date +%s)
EOF
sudo -u node_exporter mv $D/backup.prom.$$ $D/backup.prom
curl -s localhost:9100/metrics | grep -E '^backup_|^node_textfile_scrape_error'

You should see this (your timestamp will differ):

backup_last_success_timestamp_seconds 1.794118411e+09
node_textfile_scrape_error 0

Writing to a temporary name and then using mv makes the update atomic, so Node Exporter never reads a half-written file. In a real backup script you’d run the same three lines at the end, only when the backup succeeded, and later alert when time() - backup_last_success_timestamp_seconds is more than a day.

✅ Checkpoint: backup_last_success_timestamp_seconds appears in /metrics and node_textfile_scrape_error is 0.

7Install Node Exporter on prom01 too

The monitoring server needs watching as well. Repeat steps 1 to 3 on prom01 with the same commands and unit file. prom01’s ufw (from Lesson 3) doesn’t open 9100, which is right: Prometheus scrapes 192.168.56.60:9100 from the same machine, and Linux sends traffic to its own address through the loopback interface, which ufw allows.

Then run this on prom01:

systemctl is-active node_exporter
curl -s http://192.168.56.60:9100/metrics | grep '^node_load1 '

You should see this (your load value will differ):

active
node_load1 0.11

✅ Checkpoint: node_exporter is active on prom01 and answers on 192.168.56.60:9100.

8Add the node job with labels

Paste this whole block on prom01:

sudo cp /etc/prometheus/prometheus.yml /etc/prometheus/prometheus.yml.bak
sudo tee -a /etc/prometheus/prometheus.yml >/dev/null <<'EOF'

  - job_name: "node"
    static_configs:
      - targets: ["192.168.56.60:9100"]
        labels:
          role: "monitoring"
      - targets: ["192.168.56.61:9100"]
        labels:
          role: "app"
EOF
promtool check config /etc/prometheus/prometheus.yml && curl -s -X POST http://localhost:9090/-/reload && echo reloaded

You should see this:

Checking /etc/prometheus/prometheus.yml
 SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax

reloaded

tee -a appends to the end of the file, which works because scrape_configs is the last section and the new job is indented two spaces like the prometheus job. Open the file with cat and check that both - job_name lines line up. The job uses the global 15s interval; you could set scrape_interval and scrape_timeout inside the job to override it.

✅ Checkpoint: promtool says SUCCESS and the reload prints “reloaded”.

9Check targets and query up

Run this on prom01 (wait 15 seconds after the reload):

curl -s 'http://localhost:9090/api/v1/query?query=up' | python3 -c 'import json,sys
for r in json.load(sys.stdin)["data"]["result"]: print(r["metric"]["instance"], r["metric"]["job"], r["metric"].get("role","-"), r["value"][1])'
curl -s 'http://localhost:9090/api/v1/query?query=scrape_samples_scraped' | python3 -c 'import json,sys
for r in json.load(sys.stdin)["data"]["result"]: print(r["metric"]["instance"], r["value"][1])'

The first command prints this:

localhost:9090 prometheus - 1
192.168.56.60:9100 node monitoring 1
192.168.56.61:9100 node app 1

The small Python part prints one line per series from the JSON answer: instance, job, role and value. The second command then prints each instance with the number of samples its last scrape returned, from a few hundred to a few thousand depending on the collectors. You could use jq instead if you install it, and Lesson 5 does.

Three targets, all up 1, and the role label you set shows up on each node series. In the browser at https://prom01.lab/ open Status > Target health: the node pool shows 2/2 up with last scrape times a few seconds old. Then try node_memory_MemAvailable_bytes{role="app"} in the Query page.

✅ Checkpoint: up is 1 for localhost:9090, 192.168.56.60:9100 and 192.168.56.61:9100.