Secure Prometheus with Nginx, HTTPS and basic auth

Start prom01 from the prom-installed snapshot. Commands run on prom01 unless a step says laptop. You’ll use the password Lab@Prom2026 for user promadmin; pick your own if you like, but use it the same way everywhere.

1See what an open port 9090 gives away

Run this on your laptop (PowerShell, Terminal or any shell with curl):

curl -s http://192.168.56.60:9090/api/v1/status/config
curl -s http://192.168.56.60:9090/api/v1/status/flags

The output is long. It starts like this (the … marks parts left out):

{"status":"success","data":{"yaml":"global:\n  scrape_interval: 15s\n  scrape_timeout: 10s\n  evaluation_interval: 15s\n ... scrape_configs:\n- job_name: prometheus\n ...
{"status":"success","data":{... "storage.tsdb.path":"/var/lib/prometheus", ... "web.enable-lifecycle":"true", ... "web.listen-address":"0.0.0.0:9090", ...}}

No password, and you get the full config and every startup flag, including the fact that the lifecycle API is on. On Windows use curl.exe if PowerShell maps curl to its own command.

✅ Checkpoint: From the laptop you can read the config of prom01 without any login. That’s the problem you’re fixing.

2Give prom01 a lab domain name

There’s no need to buy a domain. Add a line to the hosts file on your laptop so prom01.lab points to the VM; prom01 already has it from Lesson 1.

Then run this on the laptop:

ping prom01.lab

On Windows you should see this (macOS and Linux print ping in their own format):

Pinging prom01.lab [192.168.56.60] with 32 bytes of data:
Reply from 192.168.56.60: bytes=32 time<1ms TTL=64

✅ Checkpoint: prom01.lab resolves to 192.168.56.60 on your laptop and on prom01.

3Create a self-signed certificate with SAN

Run this on prom01:

sudo mkdir -p /etc/ssl/prom01
sudo openssl req -x509 -newkey rsa:2048 -nodes -days 365 \
  -keyout /etc/ssl/prom01/prom01.key -out /etc/ssl/prom01/prom01.crt \
  -subj "/CN=prom01.lab" \
  -addext "subjectAltName=DNS:prom01.lab,IP:192.168.56.60"
sudo chmod 600 /etc/ssl/prom01/prom01.key
sudo openssl x509 -in /etc/ssl/prom01/prom01.crt -noout -subject -ext subjectAltName -dates

The last command prints this (your dates will differ):

subject=CN=prom01.lab
X509v3 Subject Alternative Name:
    DNS:prom01.lab, IP Address:192.168.56.60
notBefore=Nov  8 06:02:11 2026 GMT
notAfter=Nov  8 06:02:11 2027 GMT

While the key is being made, openssl may print a line of dots and plus signs; that’s normal. -nodes leaves the key without a passphrase so Nginx can start unattended, which is why the key gets mode 600. The SAN line is the part that matters to browsers: both the name and the IP are listed, so either works without a name mismatch.

With a real domain you’d skip this step and run sudo apt install certbot python3-certbot-nginx, then sudo certbot --nginx -d prometheus.example.com, which gets a trusted Let’s Encrypt certificate and renews it automatically.

✅ Checkpoint: The certificate shows DNS:prom01.lab and IP Address:192.168.56.60 under Subject Alternative Name.

4Bind Prometheus to localhost and set the external URL

Run this on prom01:

sudo sed -i 's#  --web.enable-lifecycle#  --web.enable-lifecycle \\\n  --web.listen-address=127.0.0.1:9090 \\\n  --web.external-url=https://prom01.lab/#' /etc/systemd/system/prometheus.service
grep -A6 '^ExecStart' /etc/systemd/system/prometheus.service
sudo systemctl daemon-reload && sudo systemctl restart prometheus
sudo ss -ltnp | grep 9090
curl -s http://localhost:9090/-/ready; echo

You should see this (your PID will differ):

ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --storage.tsdb.retention.time=15d \
  --web.enable-lifecycle \
  --web.listen-address=127.0.0.1:9090 \
  --web.external-url=https://prom01.lab/
LISTEN 0      4096       127.0.0.1:9090       0.0.0.0:*    users:(("prometheus",pid=1641,fd=8))
Prometheus Server is Ready.

If the sed line looks scary, open the unit with sudo nano /etc/systemd/system/prometheus.service and add the two lines by hand; each line except the last ends with a space and a backslash. The ss line now shows 127.0.0.1:9090 instead of 0.0.0.0:9090, and the laptop curl from step 1 now fails at once (curl exits with code 7, connection refused).

✅ Checkpoint: ss shows Prometheus on 127.0.0.1:9090 only and curl localhost still says Ready.

5Install Nginx and create the password file

Run this on prom01:

sudo apt update && sudo apt install -y nginx apache2-utils
sudo htpasswd -cB -C 10 /etc/nginx/.htpasswd promadmin
sudo chown root:www-data /etc/nginx/.htpasswd && sudo chmod 640 /etc/nginx/.htpasswd
sudo cat /etc/nginx/.htpasswd

You should see this (your hash will differ, because bcrypt adds a random salt):

New password:
Re-type new password:
Adding password for user promadmin
promadmin:$2y$10$maqMtdyb4RsFl/D7wwaaTuu6vz7wbvp2XHSyvKqhEGljG7kSBBFEC

apache2-utils gives you the htpasswd tool. -c creates the file (leave it off when adding a second user or it wipes the first), -B picks bcrypt, and -C 10 sets the cost. Nginx worker processes run as www-data, so that group must be able to read the file.

✅ Checkpoint: /etc/nginx/.htpasswd has one promadmin line starting with $2y$10$ and is owned by root:www-data, mode 640.

6Write the Nginx site and enable it

Paste this whole block:

sudo tee /etc/nginx/sites-available/prometheus >/dev/null <<'EOF'
server {
    listen 443 ssl;
    server_name prom01.lab;

    ssl_certificate     /etc/ssl/prom01/prom01.crt;
    ssl_certificate_key /etc/ssl/prom01/prom01.key;
    ssl_protocols       TLSv1.2 TLSv1.3;

    auth_basic           "Prometheus";
    auth_basic_user_file /etc/nginx/.htpasswd;

    location / {
        proxy_pass http://127.0.0.1:9090;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}
EOF
sudo ln -s /etc/nginx/sites-available/prometheus /etc/nginx/sites-enabled/prometheus
sudo rm -f /etc/nginx/sites-enabled/default
sudo nginx -t && sudo systemctl reload nginx

You should see this:

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

nginx -t is Nginx’s version of promtool: always run it before a reload. Removing the default site stops Nginx from showing its welcome page on port 80. The string after auth_basic is the realm, the text some browsers show in the login box.

✅ Checkpoint: nginx -t says the test is successful and systemctl is-active nginx prints active.

7Switch on the firewall

Run this on prom01:

sudo ufw allow OpenSSH
sudo ufw allow 443/tcp
sudo ufw enable
sudo ufw status

You should see:

Rules updated
Rules updated (v6)
Rules updated
Rules updated (v6)
Command may disrupt existing ssh connections. Proceed with operation (y|n)? y
Firewall is active and enabled on system startup
Status: active

To                         Action      From
--                         ------      ----
OpenSSH                    ALLOW       Anywhere
443/tcp                    ALLOW       Anywhere
OpenSSH (v6)               ALLOW       Anywhere (v6)
443/tcp (v6)               ALLOW       Anywhere (v6)

Allow SSH before enabling ufw, or you lock yourself out of the VM and need the VirtualBox console. Port 9090 needs no rule because it’s bound to 127.0.0.1 anyway; ufw is the second lock on the same door.

✅ Checkpoint: ufw status shows active with OpenSSH and 443/tcp allowed, and your SSH session still works.

8Test every layer with curl

Run this on prom01:

curl -s -o /dev/null -w "%{http_code}\n" https://prom01.lab/-/healthy
curl -sk -o /dev/null -w "%{http_code}\n" https://prom01.lab/-/healthy
curl -sk -u promadmin:Lab@Prom2026 https://prom01.lab/-/healthy; echo
curl -s --cacert /etc/ssl/prom01/prom01.crt -u promadmin:Lab@Prom2026 'https://prom01.lab/api/v1/query?query=up'; echo
sudo tail -1 /var/log/nginx/access.log

You should see this (your timestamp and time will differ):

000
401
Prometheus Server is Healthy.
{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"up","instance":"localhost:9090","job":"prometheus"},"value":[1794117826.763,"1"]}]}}
127.0.0.1 - promadmin [08/Nov/2026:06:23:46 +0000] "GET /api/v1/query?query=up HTTP/1.1" 200 167 "-" "curl/8.18.0"

Read them in order. 000 means curl refused the self-signed certificate and never sent a request. With -k (skip verification) you reach Nginx, which answers 401 because there’s no password. With the password you get through, and with --cacert curl trusts your certificate properly, so you don’t need -k at all.

✅ Checkpoint: You got 000, 401, “Healthy” and a JSON result, and the access log shows the user promadmin.

9Open it from the laptop browser

Browse to https://prom01.lab/. The browser warns that the certificate isn’t trusted (Chrome and Edge show NET::ERR_CERT_AUTHORITY_INVALID), because you signed it yourself. Click Advanced and continue, log in as promadmin, and you land on the Query page.

To get rid of the warning on your own laptop, copy /etc/ssl/prom01/prom01.crt over with scp and import it as a trusted root (Windows: double-click, Install Certificate, Local Machine, Trusted Root Certification Authorities). Only do this for certificates you made yourself.

Status > Target health, now reached through Nginx at https://prom01.lab/targets

✅ Checkpoint: https://prom01.lab/ asks for a password and then shows the Prometheus UI, and http://192.168.56.60:9090 no longer loads.

10Try native basic auth with bcrypt, then remove it

Now the no-Nginx option. You’ll make a bcrypt hash, write /etc/prometheus/web.yml, check it, and start Prometheus with it for a few minutes.

Paste this whole block on prom01:

HASH=$(htpasswd -nbBC 10 "" 'Lab@Prom2026' | tr -d ':\n')
echo "$HASH"
sudo tee /etc/prometheus/web.yml >/dev/null <<EOF
basic_auth_users:
  promadmin: $HASH
EOF
sudo chown prometheus:prometheus /etc/prometheus/web.yml && sudo chmod 640 /etc/prometheus/web.yml
promtool check web-config /etc/prometheus/web.yml

You should see this (your hash will differ):

$2y$10$W3FQQlOUltbttOylRPpu1ugpv1kZ.EkGDUKCUpVuiq4Q5XK2VgS1K
/etc/prometheus/web.yml SUCCESS

htpasswd -nbBC 10 "" prints :hash for an empty user name; tr removes the colon and newline so only the hash is left. Typing the password on the command line leaves it in your shell history; leave out the b and htpasswd asks for it instead. Now add the flag with a systemd drop-in, restart and test:

Paste this whole block on prom01:

sudo mkdir -p /etc/systemd/system/prometheus.service.d
sudo tee /etc/systemd/system/prometheus.service.d/web-config.conf >/dev/null <<'EOF'
[Service]
ExecStart=
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --storage.tsdb.retention.time=15d \
  --web.enable-lifecycle \
  --web.listen-address=127.0.0.1:9090 \
  --web.external-url=https://prom01.lab/ \
  --web.config.file=/etc/prometheus/web.yml
EOF
sudo systemctl daemon-reload && sudo systemctl restart prometheus
curl -s -i http://localhost:9090/-/healthy | head -1
curl -s -u promadmin:Lab@Prom2026 http://localhost:9090/-/healthy; echo
sleep 20; curl -s -u promadmin:Lab@Prom2026 http://localhost:9090/api/v1/targets | grep -o '"lastError":"[^"]*"'

A drop-in file overrides part of a unit without editing it. The empty ExecStart= line clears the original command, and the next one replaces it with the same flags plus the web config file.

You should see:

HTTP/1.1 401 Unauthorized
Prometheus Server is Healthy.
"lastError":"server returned HTTP status 401 Unauthorized"

Local curl now needs the password, and look at the last line: Prometheus’s own job can’t scrape itself any more, so up{job="prometheus"} drops to 0. You’d fix that with a basic_auth block (username plus password_file) in the scrape job, and do the same in Grafana and every script. Adding tls_server_config with a cert and key to the same file turns on HTTPS too.

Remove it again (Nginx is our front door):

sudo rm /etc/systemd/system/prometheus.service.d/web-config.conf
sudo systemctl daemon-reload && sudo systemctl restart prometheus
systemctl cat prometheus | grep -c web.config.file
curl -s http://localhost:9090/-/healthy; echo

You should see this:

0
Prometheus Server is Healthy.

Keep /etc/prometheus/web.yml for reference; it does nothing without the flag. Shut prom01 down and take a snapshot called secured.

✅ Checkpoint: Local curl works without a password again, https://prom01.lab/ still asks for one, and prom01 has a snapshot called secured.