Nginx logs, LogQL metrics and annotations

You’ll build a tiny exam portal on app01 (192.168.56.71), ship its access log through a new Alloy pipeline to Loki on graf01 (192.168.56.70), then build the dashboard in Grafana at https://graf01.lab/. The Alloy config from Lesson 7 stays as it is; you add new blocks to it.

1Install Nginx on app01 as the exam portal

Paste this whole block on app01:

sudo apt update && sudo apt install -y nginx
sudo mkdir -p /var/www/exam/exam /var/www/exam/results
echo '<h1>SaatVedha exam portal</h1>' | sudo tee /var/www/exam/index.html >/dev/null
echo 'Exam started'  | sudo tee /var/www/exam/exam/start >/dev/null
echo 'Results page'  | sudo tee /var/www/exam/results/index.html >/dev/null

sudo tee /etc/nginx/sites-available/exam >/dev/null <<'EOF'
server {
    listen 80 default_server;
    server_name app01.lab;
    root /var/www/exam;

    location = /health     { return 200 "ok\n"; }
    location = /api/submit { return 500; }
    location /             { try_files $uri $uri/ =404; }
}
EOF
sudo rm -f /etc/nginx/sites-enabled/default
sudo ln -s /etc/nginx/sites-available/exam /etc/nginx/sites-enabled/exam
sudo nginx -t && sudo systemctl reload nginx
sudo ufw allow from 192.168.56.0/24 to any port 80 proto tcp

The last lines of the output are:

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
Rule added

/api/submit always returns 500, so you have a reliable “broken” page to graph. The ufw rule lets graf01 and your laptop reach port 80; app01 still allows 9100 and 3306 only from graf01, as before.

✅ Checkpoint: nginx -t is successful and ufw shows port 80 allowed from 192.168.56.0/24 on app01.

2Read a few access log lines

Run this on graf01, then the tail on app01:

for p in / /exam/start /results/ /nope /api/submit /health; do
  curl -s -o /dev/null -w "%{http_code} $p\n" http://app01$p
done
# on app01:
sudo tail -n 6 /var/log/nginx/access.log

You should see this (your times and curl version will differ):

200 /
200 /exam/start
200 /results/
404 /nope
500 /api/submit
200 /health

192.168.56.70 - - [08/Nov/2026:09:42:16 +0000] "GET / HTTP/1.1" 200 31 "-" "curl/8.5.0"
192.168.56.70 - - [08/Nov/2026:09:42:16 +0000] "GET /exam/start HTTP/1.1" 200 13 "-" "curl/8.5.0"
192.168.56.70 - - [08/Nov/2026:09:42:16 +0000] "GET /results/ HTTP/1.1" 200 13 "-" "curl/8.5.0"
192.168.56.70 - - [08/Nov/2026:09:42:16 +0000] "GET /nope HTTP/1.1" 404 162 "-" "curl/8.5.0"
192.168.56.70 - - [08/Nov/2026:09:42:16 +0000] "GET /api/submit HTTP/1.1" 500 186 "-" "curl/8.5.0"
192.168.56.70 - - [08/Nov/2026:09:42:16 +0000] "GET /health HTTP/1.1" 200 3 "-" "curl/8.5.0"

This is the combined format. The time in brackets is the server’s local time with its offset (+0000 because Ubuntu Server defaults to UTC), the request line sits in the first pair of quotes, and the status follows it. Your curl version and byte counts may differ.

✅ Checkpoint: access.log on app01 shows lines with 200, 404 and 500 status codes.

3Add the Nginx pipeline to Alloy on app01

Append these blocks to the end of /etc/alloy/config.alloy. They reuse the loki.write "graf01" component from Lesson 7, so nothing else changes. Inside Alloy strings a backslash must be doubled, which is why the regex has \\S instead of \S.

Paste this whole block on app01:

sudo tee -a /etc/alloy/config.alloy >/dev/null <<'EOF'

// ---- Nginx access log (Lesson 8) ----
local.file_match "nginx" {
  path_targets = [
    {"__path__" = "/var/log/nginx/access.log", "job" = "nginx", "host" = "app01"},
  ]
}

loki.source.file "nginx" {
  targets    = local.file_match.nginx.targets
  forward_to = [loki.process.nginx.receiver]
}

loki.process "nginx" {
  // 1. throw away load balancer health checks
  stage.drop {
    expression          = "\"GET /health "
    drop_counter_reason = "healthcheck"
  }

  // 2. parse the combined format into the extracted map
  stage.regex {
    expression = "^(?P<remote_addr>\\S+) \\S+ (?P<remote_user>\\S+) \\[(?P<time_local>[^\\]]+)\\] \"(?P<method>\\S+) (?P<path>[^ ?\"]+)\\S* (?P<protocol>[^\"]+)\" (?P<status>\\d{3}) (?P<bytes>\\d+) \"(?P<referer>[^\"]*)\" \"(?P<user_agent>[^\"]*)\"$"
  }

  // 3. use the request time, not the read time
  stage.timestamp {
    source = "time_local"
    format = "02/Jan/2006:15:04:05 -0700"
  }

  // 4. only low-cardinality fields become labels
  stage.labels {
    values = { method = "", status = "" }
  }

  stage.static_labels {
    values = { env = "lab" }
  }

  forward_to = [loki.write.graf01.receiver]
}
EOF
alloy fmt /etc/alloy/config.alloy >/dev/null && echo "syntax OK"
sudo systemctl restart alloy && systemctl is-active alloy

You should see:

syntax OK
active

The format string 02/Jan/2006:15:04:05 -0700 is Go’s way of describing a layout: you write the reference date (2 January 2006, 15:04:05, zone -0700) the way your log writes dates. Note that path is extracted but not made a label; it’s there for the drop stage to ignore and for you to see in the Alloy UI’s live debugging, and you’ll parse it again at query time.

✅ Checkpoint: alloy fmt reports syntax OK and alloy is active on app01.

4Generate exam-day traffic from graf01

A small script sends a realistic mix: mostly normal pages, a few missing pages, some failing submits and a health check every second, the way a load balancer would.

Paste this whole block on graf01:

cat > ~/exam-traffic.sh <<'EOF'
#!/bin/bash
# usage: ./exam-traffic.sh [seconds]   (default 600)
end=$((SECONDS + ${1:-600}))
pages=(/ / / /exam/start /exam/start /results/ /results/ /nope /api/submit)
while [ $SECONDS -lt $end ]; do
  for i in 1 2 3 4 5; do
    p=${pages[$RANDOM % ${#pages[@]}]}
    curl -s -o /dev/null "http://app01$p"
  done
  curl -s -o /dev/null http://app01/health
  sleep 1
done
EOF
chmod +x ~/exam-traffic.sh
nohup ~/exam-traffic.sh 1800 >/dev/null 2>&1 &
echo "traffic running as PID $!"

You should see this (your PID will differ):

traffic running as PID 4127

That’s about five page requests and one health check per second for 30 minutes. Leave it running while you do the next steps; run it again later whenever a graph looks empty.

✅ Checkpoint: The traffic script is running on graf01 and app01’s access.log is growing (sudo tail -f shows new lines).

5Check the labels and the dropped health checks in Loki

Run this on graf01:

curl -s localhost:3100/loki/api/v1/label/status/values; echo
curl -s localhost:3100/loki/api/v1/label/method/values; echo
sudo apt install -y jq
curl -sG localhost:3100/loki/api/v1/query \
  --data-urlencode 'query=sum(count_over_time({job="nginx"} |= "/health" [10m]))' | jq -c '.data.result'

You should see:

{"status":"success","data":["200","404","500"]}
{"status":"success","data":["GET"]}
[]

The first two calls show the status and method values Alloy attached (Loki also adds a service_name label by itself, taken from job). Query responses carry a large stats block, so jq keeps only the result. The third asks for health check lines in the last 10 minutes and gets an empty result, because the drop stage removed them on app01 before they were ever sent.

On app01, curl -s localhost:12345/metrics | grep dropped_lines shows the counter loki_process_dropped_lines_total{reason="healthcheck"} going up.

✅ Checkpoint: Loki lists status values 200, 404 and 500, and the /health query returns an empty result.

6Write LogQL metric queries in Explore

In Grafana open Explore, pick the Loki data source, switch the editor to Code and set the time range to the last 15 minutes. Run these one by one and look at the graph each time.

Paste each query into Explore:

# lines per minute, one series per stream
count_over_time({job="nginx", host="app01"}[1m])

# requests per second by status
sum by (status) (rate({job="nginx", host="app01"}[$__auto]))

# 5xx per second
sum(rate({job="nginx", host="app01", status=~"5.."}[5m]))

# percentage of requests that failed with 5xx
100 * sum(rate({job="nginx", status=~"5.."}[5m])) / sum(rate({job="nginx"}[5m]))

# top 5 paths, parsed at query time (path is NOT a label)
topk(5, sum by (path) (count_over_time({job="nginx"} | regexp `"\S+ (?P<path>[^ ?"]+)` [$__auto])))

In Table view the status query gives about these values:

status   Value
200      3.89
404      0.556
500      0.556

The script sends 5 requests a second from 9 pages, and 7 of those pages return 200, so about 3.9 per second should be 200s and about 0.56 each should be 404 and 500. The numbers move a little on every refresh because the pages are random.

$__auto lets Grafana pick the range to match the graph’s step, so the graph stays smooth when you zoom out. For the top paths query, set Type to Instant in the query options so you get one number per path instead of a graph.

✅ Checkpoint: The status query shows three series (200, 404, 500) and the topk query lists paths such as /, /exam/start and /api/submit.

7Build the “Nginx app01” dashboard

Create a new dashboard, save it as Nginx app01 in the SaatVedha Lab folder, and add these four panels, all on the Loki data source.

The Logs panel matters more than people think. When the 5xx stat goes red, the on-call engineer scrolls down and sees the actual failing lines on the same screen, with client address and user agent, without opening Explore.

✅ Checkpoint: Nginx app01 is saved in SaatVedha Lab with four panels that all show data.

8Ship a deploy log for annotations

Releases on app01 will write one logfmt line to /var/log/deploy/deploy.log. Add a second, small pipeline that tails it with job="deploy" and uses the ts field as the timestamp.

Paste this whole block on app01:

sudo mkdir -p /var/log/deploy && sudo touch /var/log/deploy/deploy.log
sudo tee -a /etc/alloy/config.alloy >/dev/null <<'EOF'

// ---- Deploy markers (Lesson 8) ----
local.file_match "deploy" {
  path_targets = [
    {"__path__" = "/var/log/deploy/deploy.log", "job" = "deploy", "host" = "app01"},
  ]
}

loki.source.file "deploy" {
  targets    = local.file_match.deploy.targets
  forward_to = [loki.process.deploy.receiver]
}

loki.process "deploy" {
  stage.logfmt {
    mapping = { ts = "", app = "", version = "" }
  }
  stage.timestamp {
    source = "ts"
    format = "RFC3339"
  }
  forward_to = [loki.write.graf01.receiver]
}
EOF
alloy fmt /etc/alloy/config.alloy >/dev/null && sudo systemctl restart alloy

# a release happens:
echo "ts=$(date -u +%Y-%m-%dT%H:%M:%SZ) level=info msg=deployed app=exam-portal version=v2.4.1" \
  | sudo tee -a /var/log/deploy/deploy.log

tee prints the line back (with your own time):

ts=2026-11-08T10:05:31Z level=info msg=deployed app=exam-portal version=v2.4.1

date -u with a trailing Z gives an unambiguous UTC time, which RFC3339 parsing understands. Remember this choice, because Break 3 shows what happens without it.

✅ Checkpoint: The deploy line is in /var/log/deploy/deploy.log and {job=”deploy”} returns it in Explore.

9Add the annotation query to the dashboard

Open Nginx app01, click Edit, then Settings and the Annotations tab, and click Add annotation query. Fill it in like this and save the dashboard.

Back on the dashboard, a dashed vertical line now crosses the time series panel at the deploy time, and hovering the small marker at the bottom shows “exam-portal v2.4.1”. Write a second deploy line with a different version a few minutes later and press refresh to see a second marker.

✅ Checkpoint: The time series panel shows a deploy marker with the title exam-portal v2.4.1 at the time of your deploy line.

10Snapshot both VMs

Run this on your laptop (VMs can stay running):

VBoxManage snapshot graf01 take nginx-logs
VBoxManage snapshot app01 take nginx-logs

You should see this (your UUIDs will differ):

0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Snapshot taken. UUID: 3b8e61f2-5c0d-4f7a-9e21-7d4c0a9b6e13
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Snapshot taken. UUID: c41a7d09-2e6b-4b38-8f55-1a9e3d7c2b60

✅ Checkpoint: Both VMs have a snapshot called nginx-logs.