Logs in Grafana with Loki and Alloy

Loki and an Alloy instance run on graf01; a second Alloy runs on app01. Commands say which VM. Take a snapshot of both VMs before you start, because this lesson changes services on both.

1Install Loki on graf01 from the Grafana APT repository

graf01 already has the Grafana APT repository from Lesson 2, so Loki is one package away. The package installs /usr/bin/loki, a loki system user and a loki.service unit that runs /usr/bin/loki -config.file /etc/loki/config.yml.

Run this on graf01:

sudo apt-get update
sudo apt-get install -y loki
loki --version | head -1
systemctl cat loki | grep ExecStart

The first line starts with loki, version 3.7.9 (or a later 3.x release), followed by the branch and Git revision it was built from. The second line should read:

ExecStart=/usr/bin/loki -config.file /etc/loki/config.yml

The package enables and starts Loki straight away with its own sample config, so you’ll replace that file next and restart. Loki 3.7.x is current as I write this; any 3.x release takes the same config. If you prefer the zip from GitHub releases, put loki-linux-amd64 in /usr/local/bin and write the same unit by hand, the rest of the lesson is identical.

✅ Checkpoint: The version check prints a 3.x version and the unit starts /usr/bin/loki with /etc/loki/config.yml.

2Write the Loki config with filesystem storage and 7-day retention

Paste this whole block on graf01:

sudo tee /etc/loki/config.yml >/dev/null <<'EOF'
auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096
  log_level: info

common:
  instance_addr: 127.0.0.1
  path_prefix: /var/lib/loki
  storage:
    filesystem:
      chunks_directory: /var/lib/loki/chunks
      rules_directory: /var/lib/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2026-01-01
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

limits_config:
  retention_period: 168h
  reject_old_samples: true
  reject_old_samples_max_age: 168h

compactor:
  working_directory: /var/lib/loki/compactor
  retention_enabled: true
  delete_request_store: filesystem

analytics:
  reporting_enabled: false
EOF
sudo mkdir -p /var/lib/loki && sudo chown -R loki /var/lib/loki
sudo -u loki loki -config.file=/etc/loki/config.yml -verify-config

You should see (your time will differ):

level=info ts=2026-11-08T06:53:31.719913919Z caller=main.go:109 msg="config is valid"

auth_enabled: false means a single tenant (Loki calls it “fake” in messages), fine for one team. Schema v13 with the TSDB index is the current recommended layout, and the from date only has to be in the past. -verify-config checks the file without starting the server, so make it a habit before every restart.

✅ Checkpoint: loki -verify-config prints msg=”config is valid”.

3Start Loki and check it is ready

Run this on graf01:

sudo systemctl enable loki && sudo systemctl restart loki
sleep 20
curl -s localhost:3100/ready; echo
sudo journalctl -u loki --no-pager | grep -E 'Starting Loki|Loki started' | tail -2

You should see (your times will differ):

ready
level=info ts=2026-11-08T05:56:35.354768065Z caller=main.go:146 msg="Starting Loki" version="(version=3.7.9, branch=release-3.7.x, revision=d3b4a2f7)"
level=info ts=2026-11-08T05:56:35.584067572Z caller=loki.go:615 msg="Loki started" startup_time=296.44178ms

For the first 15 seconds /ready answers “Ingester not ready: waiting for 15s after being ready”, which is normal. Loki listens on all addresses on port 3100, so open it to the lab network only, because app01’s Alloy must push to it:

Run this on graf01:

sudo ufw allow from 192.168.56.0/24 to any port 3100 proto tcp
sudo ufw status | grep 3100

3100/tcp                   ALLOW       192.168.56.0/24

✅ Checkpoint: curl localhost:3100/ready answers ready and ufw allows 3100 from 192.168.56.0/24 only.

4Install Alloy on graf01 and app01

app01 doesn’t have the Grafana repository yet, so add it there first; on graf01 it’s already present.

Run this on app01 only (adds the repository):

sudo mkdir -p /etc/apt/keyrings
sudo wget -q -O /etc/apt/keyrings/grafana.asc https://apt.grafana.com/gpg-full.key
sudo chmod 644 /etc/apt/keyrings/grafana.asc
echo "deb [signed-by=/etc/apt/keyrings/grafana.asc] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list

Then run this on both VMs:

sudo apt-get update && sudo apt-get install -y alloy
alloy --version | head -1
id -nG alloy

The version line starts with alloy, version v1.21.3 (or a later v1.21.x), followed by the branch and Git revision of the build. The group list should read:

alloy adm systemd-journal

The package creates an alloy user and adds it to the adm and systemd-journal groups, because on Ubuntu /var/log/syslog and /var/log/auth.log belong to syslog:adm with mode 640. The service runs alloy run on /etc/alloy/config.alloy, and its UI listens on 127.0.0.1:12345.

✅ Checkpoint: The version check prints v1.21.x on both VMs and id -nG alloy shows the adm group.

5Write config.alloy on app01

The app also needs a log to ship. Create a small JSON log for the shop application first, then the Alloy config.

Paste this whole block on app01:

sudo mkdir -p /var/log/shopapp
sudo tee /var/log/shopapp/app.log >/dev/null <<'EOF'
{"level":"info","msg":"order placed","city":"Jaipur","amount":899,"duration_ms":42}
{"level":"error","msg":"payment gateway timeout","city":"Pune","amount":1299,"duration_ms":5003}
{"level":"info","msg":"order placed","city":"Delhi","amount":349,"duration_ms":38}
EOF
echo 'level=warn msg="slow query" table=orders duration_ms=812' | sudo tee -a /var/log/shopapp/app.log >/dev/null

sudo tee /etc/alloy/config.alloy >/dev/null <<'EOF'
logging {
  level  = "info"
  format = "logfmt"
}

// 1. Find the files to read
local.file_match "system" {
  path_targets = [
    {"__path__" = "/var/log/syslog",   "job" = "varlogs", "host" = "app01"},
    {"__path__" = "/var/log/auth.log", "job" = "varlogs", "host" = "app01"},
  ]
}

local.file_match "shopapp" {
  path_targets = [
    {"__path__" = "/var/log/shopapp/*.log", "job" = "shopapp", "host" = "app01"},
  ]
}

// 2. Tail them
loki.source.file "system" {
  targets    = local.file_match.system.targets
  forward_to = [loki.write.graf01.receiver]
}

loki.source.file "shopapp" {
  targets    = local.file_match.shopapp.targets
  forward_to = [loki.write.graf01.receiver]
}

// 3. Send to Loki on graf01
loki.write "graf01" {
  endpoint {
    url = "http://192.168.56.70:3100/loki/api/v1/push"
  }
}
EOF
alloy fmt /etc/alloy/config.alloy >/dev/null && echo "syntax OK"
sudo systemctl restart alloy && systemctl is-active alloy

You should see:

syntax OK
active

Each block is a component with a type and a label you choose, like loki.write "graf01". Components refer to each other’s exports by those names, local.file_match.system.targets and loki.write.graf01.receiver, which is how the pipeline is wired. alloy fmt parses the file and fails on any syntax error, a quick check before every restart; Alloy itself adds the filename label for each file it tails.

✅ Checkpoint: alloy fmt reports syntax OK and the alloy service is active on app01.

6Ship graf01’s own logs and open the Alloy UI

Run this on graf01:

sudo tee /etc/alloy/config.alloy >/dev/null <<'EOF'
logging {
  level  = "info"
  format = "logfmt"
}

local.file_match "system" {
  path_targets = [
    {"__path__" = "/var/log/syslog",   "job" = "varlogs", "host" = "graf01"},
    {"__path__" = "/var/log/auth.log", "job" = "varlogs", "host" = "graf01"},
  ]
}

loki.source.file "system" {
  targets    = local.file_match.system.targets
  forward_to = [loki.write.local.receiver]
}

loki.write "local" {
  endpoint {
    url = "http://localhost:3100/loki/api/v1/push"
  }
}
EOF
sudo systemctl restart alloy
curl -s localhost:12345/-/ready; echo

You should see:

Alloy is ready.

The Alloy UI listens only on 127.0.0.1, so reach it from your laptop with an SSH tunnel: ssh -L 12345:127.0.0.1:12345 student@192.168.56.71 and browse to http://localhost:12345. The home page lists each component with its health, and the Graph page draws the pipeline you wrote; open loki.source.file.shopapp to see the tailed path and its read offset.

✅ Checkpoint: /-/ready on graf01 answers “Alloy is ready.” and the Alloy UI on app01 shows all components healthy.

7Check the labels arriving in Loki

Run this on graf01:

curl -s localhost:3100/loki/api/v1/labels; echo
curl -s localhost:3100/loki/api/v1/label/job/values; echo
curl -s localhost:3100/loki/api/v1/label/host/values; echo

You should see:

{"status":"success","data":["filename","host","job","service_name"]}
{"status":"success","data":["shopapp","varlogs"]}
{"status":"success","data":["app01","graf01"]}

You set job and host, Alloy added filename, and Loki 3 adds service_name by itself (it copies job when no service label exists), which Grafana’s Logs Drilldown app uses. Four label names and a handful of values: this is what healthy cardinality looks like.

✅ Checkpoint: The labels API lists filename, host, job and service_name, with jobs shopapp and varlogs and hosts app01 and graf01.

8Add the Loki data source and run your first LogQL in Explore

In Grafana add a data source of type Loki named Loki with URL http://localhost:3100, then Save & test. Open Explore, pick Loki, switch the query editor to Code and run:

{job="varlogs", host="app01"}

You get the newest syslog and auth.log lines from app01, with a bar chart of log volume above them. Click a line to see its labels. Now generate a failed login and find it: from your laptop run ssh hacker@192.168.56.71 (a user that doesn’t exist on app01) and press Enter at the password prompt twice, then query:

{job="varlogs", host="app01"} |= "Failed password"

The same query from graf01 with curl:

curl -s -G localhost:3100/loki/api/v1/query_range \
  --data-urlencode 'query={job="varlogs", host="app01"} |= "Failed password"' \
  --data-urlencode 'since=15m' --data-urlencode 'limit=5' | python3 -m json.tool | grep -A2 '"values"'

You should see (your timestamps, PID and port will differ):

                "values": [
                    [
                        "1794120700787705223",
                        "2026-11-08T07:10:02.118042+00:00 app01 sshd-session[1240]: Failed password for invalid user hacker from 192.168.56.1 port 50130 ssh2"

On Ubuntu 26.04 the OpenSSH 10 login process is called sshd-session; on 24.04 the same line says sshd[1240].

✅ Checkpoint: Explore shows app01 log lines, and the Failed password filter returns your failed SSH attempt.

9Line filters and parsers

Try each query in Explore against the last 15 minutes. These are the forms you’ll use every day:

Check the JSON parser from graf01:

curl -s -G localhost:3100/loki/api/v1/query_range \
  --data-urlencode 'query={job="shopapp"} | json | duration_ms > 1000 | line_format "{{.city}} {{.msg}} {{.duration_ms}}ms"' \
  --data-urlencode 'since=1h' | python3 -m json.tool | grep -A1 '"values"' | tail -1

You should see (your timestamp will differ):

                    "1794120702987705223", "Pune payment gateway timeout 5003ms"

Order matters for speed: put the stream selector first, then cheap line filters like |=, and only then the parser, so Loki parses only the lines that survived the filter. Parsed fields are labels for that query only; they don’t create streams in the database.

✅ Checkpoint: The json, logfmt and pattern queries all return results, and the duration filter returns only the Pune timeout.

10Check retention is active and take a snapshot

Run this on graf01:

curl -s localhost:3100/config | grep -E '^\s+(retention_period|retention_enabled|reject_old_samples_max_age):'
sudo du -sh /var/lib/loki

You should see (the size will differ):

  retention_enabled: true
  reject_old_samples_max_age: 1w
  retention_period: 1w
  retention_period: 0s
1.6M	/var/lib/loki

The /config endpoint shows the settings Loki loaded, with 168h printed as 1w; the extra retention_period: 0s line belongs to the separate table manager section and doesn’t matter here. Shut down both VMs and take a snapshot called loki-alloy; Lesson 8 builds on it with Nginx logs.

✅ Checkpoint: /config shows retention_enabled true and retention_period 1w, and both VMs have a loki-alloy snapshot.