Provisioning Grafana as code, and Elasticsearch logs

Part A (steps 1 to 4) is pure Grafana provisioning on graf01. Part B (steps 5 to 11) adds Elasticsearch on graf01 and Filebeat on app01, then provisions the Elasticsearch data source from your Git repository. Take a snapshot of both VMs before step 5.

1Look at the provisioning folders

Run this on graf01:

sudo ls -R /etc/grafana/provisioning
sudo grep -E '^;?provisioning' /etc/grafana/grafana.ini

You should see:

/etc/grafana/provisioning:
access-control  alerting  dashboards  datasources  plugins

/etc/grafana/provisioning/access-control:
sample.yaml

/etc/grafana/provisioning/alerting:
sample.yaml

/etc/grafana/provisioning/dashboards:
sample.yaml

/etc/grafana/provisioning/datasources:
sample.yaml

/etc/grafana/provisioning/plugins:
sample.yaml
;provisioning = conf/provisioning

The package creates these folders with commented sample files. The provisioning line is commented in grafana.ini, but the packaged service passes cfg:default.paths.provisioning=/etc/grafana/provisioning, so that’s the folder in use.

✅ Checkpoint: /etc/grafana/provisioning has access-control, alerting, dashboards, datasources and plugins folders.

2Export dashboards to a Git repository with the backup-bot token

Paste this whole block on graf01:

sudo apt-get install -y git
mkdir -p ~/grafana-as-code/{dashboards,provisioning/datasources,provisioning/dashboards}
cd ~/grafana-as-code && git init -q && git config user.email student@lab.local && git config user.name student

cat > export.sh <<'EOF'
#!/bin/bash
# Export every dashboard the token can see into dashboards/<uid>.json
TOKEN=$(cat ~/.grafana-backup-token)
API=http://localhost:3000/api
for uid in $(curl -s -H "Authorization: Bearer $TOKEN" "$API/search?type=dash-db" | jq -r '.[].uid'); do
  curl -s -H "Authorization: Bearer $TOKEN" "$API/dashboards/uid/$uid" \
    | jq '.dashboard | del(.id, .version)' > "dashboards/$uid.json"
  echo "exported $uid"
done
EOF
chmod +x export.sh && ./export.sh | tail -n 4
ls dashboards | wc -l

The script prints one exported line per dashboard with its UID, for example exported rYdddlPWk for Node Exporter Full, and you see the last four. The final number is how many JSON files you now have, which depends on how many dashboards you built in the earlier lessons (about a dozen if you did them all).

del(.id, .version) removes the numeric database ID and version, which belong to one Grafana server and would cause conflicts on another; the uid stays, because it’s the portable identity. The token is read from the 600 file, never typed into the script. Run this nightly from cron and commit the result, and you have dashboard backups with a history.

✅ Checkpoint: dashboards/ in ~/grafana-as-code holds one JSON file per dashboard backup-bot can see.

3Provision a folder of dashboards from the repository

Make one dashboard that lives only in Git: a copy of Grafana health with its own UID and title. Then point a dashboard provider at a folder that holds it.

Paste this whole block on graf01:

cd ~/grafana-as-code
mkdir -p dashboards/provisioned
jq '.uid = "grafana-health-code" | .title = "Grafana health (as code)"' \
  dashboards/grafana-health-copy.json > dashboards/provisioned/grafana-health-code.json

cat > provisioning/dashboards/saatvedha.yaml <<'EOF'
apiVersion: 1
providers:
  - name: saatvedha-git
    orgId: 1
    folder: Provisioned
    type: file
    disableDeletion: true
    allowUiUpdates: false
    updateIntervalSeconds: 30
    options:
      path: /var/lib/grafana/dashboards/saatvedha
EOF

sudo mkdir -p /var/lib/grafana/dashboards/saatvedha
sudo cp dashboards/provisioned/*.json /var/lib/grafana/dashboards/saatvedha/
sudo cp provisioning/dashboards/saatvedha.yaml /etc/grafana/provisioning/dashboards/
sudo chown -R root:grafana /var/lib/grafana/dashboards /etc/grafana/provisioning/dashboards/saatvedha.yaml
sudo systemctl restart grafana-server && sleep 10
curl -s -u admin:Lab@Grafana2026 localhost:3000/api/dashboards/uid/grafana-health-code \
  | jq '{title: .dashboard.title, folder: .meta.folderTitle, provisioned: .meta.provisioned}'

You should see:

{
  "title": "Grafana health (as code)",
  "folder": "Provisioned",
  "provisioned": true
}

Grafana created the Provisioned folder, loaded the file and marked the dashboard provisioned: true. Because allowUiUpdates is false, changes have to go through the file. disableDeletion: true protects you from an accidental empty folder on disk wiping the dashboard.

✅ Checkpoint: Grafana health (as code) is in the Provisioned folder and the API reports provisioned true.

4Commit the first version

Run this on graf01:

cd ~/grafana-as-code
printf '*.pass\n*.token\n' > .gitignore
git add . && git commit -qm "Export dashboards, add provisioned folder" && git log --oneline | head -n 1

You should see this (your commit hash will differ):

3f2a9c1 Export dashboards, add provisioned folder

At work this repository lives on your Git server, changes come as merge requests, and a pipeline copies files to each Grafana. Secrets never go in: the .gitignore keeps password and token files out, and the data source file in step 10 only points at secret files with $__file{}.

✅ Checkpoint: git log shows your first commit in ~/grafana-as-code.

5Give graf01 4 GB and install Elasticsearch 9

Run this on your laptop with graf01 shut down (sudo poweroff), then start it:

VBoxManage snapshot graf01 take before-elastic
VBoxManage modifyvm graf01 --memory 4096
VBoxManage startvm graf01 --type headless

Then paste this whole block on graf01:

curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch \
  | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/9.x/apt stable main" \
  | sudo tee /etc/apt/sources.list.d/elastic-9.x.list
sudo apt-get update && sudo apt-get install -y elasticsearch | tee ~/elastic-install.txt | grep -A2 'generated password'

You should see this (your password will differ):

The generated password for the elastic built-in superuser is : h7Qz+Kp2vXm1sT9wLe4R

If this node should join an existing cluster, you can reconfigure this with

Elasticsearch 9 turns on security during install: TLS on the HTTP port with its own CA in /etc/elasticsearch/certs/http_ca.crt, and a random password for the elastic superuser. Save that password now in ~/.elastic.pass with chmod 600; elasticsearch-reset-password -u elastic makes a new one if you lose it.

✅ Checkpoint: elasticsearch is installed, ~/elastic-install.txt contains the generated password, and you saved it in ~/.elastic.pass.

6Fix the heap at 1 GB and start Elasticsearch

Paste this whole block on graf01:

printf -- '-Xms1g\n-Xmx1g\n' | sudo tee /etc/elasticsearch/jvm.options.d/heap.options
sudo systemctl daemon-reload && sudo systemctl enable --now elasticsearch
sudo curl -s --cacert /etc/elasticsearch/certs/http_ca.crt -u "elastic:$(cat ~/.elastic.pass)" \
  https://localhost:9200 | jq '{name, version: .version.number, tagline}'
free -m | head -n 2

You should see this (your memory numbers will differ):

-Xms1g
-Xmx1g
Created symlink '/etc/systemd/system/multi-user.target.wants/elasticsearch.service' → '/usr/lib/systemd/system/elasticsearch.service'.
{
  "name": "graf01",
  "version": "9.5.6",
  "tagline": "You Know, for Search"
}
               total        used        free      shared  buff/cache   available
Mem:            3916        2930         211          21         969         986

Without the heap file, Elasticsearch sizes its heap from the VM’s total memory and competes with Grafana, Prometheus, Loki and InfluxDB on the same box. Equal -Xms and -Xmx reserve the heap once at start, so it can’t grow later and surprise you. The first start takes up to a minute while the JVM warms up.

✅ Checkpoint: Elasticsearch answers on https://localhost:9200 with version 9.x and name graf01, and free shows memory still available.

7Create a read-only user for Grafana and open 9200 to app01

Paste this whole block on graf01:

openssl rand -base64 18 | tr -d '\n' > ~/es-grafana-reader.pass && chmod 600 ~/es-grafana-reader.pass
sudo curl -s --cacert /etc/elasticsearch/certs/http_ca.crt -u "elastic:$(cat ~/.elastic.pass)" \
  -X POST https://localhost:9200/_security/user/grafana_reader -H 'Content-Type: application/json' \
  -d "{\"password\":\"$(cat ~/es-grafana-reader.pass)\",\"roles\":[\"viewer\"]}"; echo
sudo install -o root -g grafana -m 640 ~/es-grafana-reader.pass /etc/grafana/es-grafana-reader.pass
sudo install -o root -g grafana -m 640 /etc/elasticsearch/certs/http_ca.crt /etc/grafana/es-http-ca.crt
sudo ufw allow from 192.168.56.71 to any port 9200 proto tcp

You should see:

{"created":true}
Rule added

The built-in viewer role can read indices but not write or change settings, which is all Grafana needs. Grafana reads the password and the CA certificate from files owned by root and readable by the grafana group, so neither ever appears in a provisioning file or in Git.

✅ Checkpoint: grafana_reader exists, both files are in /etc/grafana with mode 640 root:grafana, and ufw allows 9200 from 192.168.56.71 only.

8Install Filebeat on app01 with the Nginx module

Copy the CA first, on graf01:

sudo cat /etc/grafana/es-http-ca.crt | ssh student@app01 'cat > /tmp/es-http-ca.crt'

Then paste this whole block on app01:

curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch \
  | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/9.x/apt stable main" \
  | sudo tee /etc/apt/sources.list.d/elastic-9.x.list
sudo apt-get update && sudo apt-get install -y filebeat
sudo install -m 644 /tmp/es-http-ca.crt /etc/filebeat/es-http-ca.crt

sudo filebeat keystore create
sudo filebeat keystore add ES_PWD        # paste the elastic password when asked

sudo tee /etc/filebeat/filebeat.yml >/dev/null <<'EOF'
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

output.elasticsearch:
  hosts: ["https://192.168.56.70:9200"]
  username: "elastic"
  password: "${ES_PWD}"
  ssl.certificate_authorities: ["/etc/filebeat/es-http-ca.crt"]
  ssl.verification_mode: certificate

setup.ilm.enabled: true
EOF
sudo filebeat modules enable nginx
sudo tee /etc/filebeat/modules.d/nginx.yml >/dev/null <<'EOF'
- module: nginx
  access:
    enabled: true
    var.paths: ["/var/log/nginx/access.log*"]
  error:
    enabled: false
EOF
sudo filebeat test config && sudo filebeat test output

You should see:

Enabled nginx
Config OK
elasticsearch: https://192.168.56.70:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 192.168.56.70
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.3
    dial up... OK
  talk to server... OK
  version: 9.5.6

The keystore keeps the password out of filebeat.yml, and ${ES_PWD} pulls it in at start. verification_mode: certificate checks that the server certificate comes from your CA without insisting that its name matches 192.168.56.70. Using the elastic superuser keeps the lab short; at work you’d create a writer role limited to the filebeat indices.

✅ Checkpoint: filebeat test config says Config OK and test output ends with talk to server OK and the Elasticsearch version.

9Load the ingest pipeline and start shipping

Run this on app01, then the check on graf01:

sudo filebeat setup --pipelines --modules nginx
sudo systemctl enable --now filebeat
# on graf01, after running ~/exam-traffic.sh 120 from Lesson 8:
sudo curl -s --cacert /etc/elasticsearch/certs/http_ca.crt -u "elastic:$(cat ~/.elastic.pass)" \
  'https://localhost:9200/_cat/indices/.ds-filebeat-*?h=index,docs.count'

You should see this (your document count will differ):

Loaded Ingest pipelines
.ds-filebeat-9.5.6-2026.11.08-000001 1873

The Nginx module’s pipeline runs inside Elasticsearch and turns each raw line into fields like source.address, http.request.method, url.original and http.response.status_code, and it moves the raw line itself from message to event.original, so the stored documents have no message field. Filebeat writes to the filebeat-9.5.6 data stream, and the .ds- index behind it rolls over by size and age under its lifecycle policy.

✅ Checkpoint: A .ds-filebeat-9.x backing index exists on graf01 and its docs.count grows while traffic runs.

10Provision the Elasticsearch data source from Git

Paste this whole block on graf01:

cd ~/grafana-as-code
cat > provisioning/datasources/elasticsearch.yaml <<'EOF'
apiVersion: 1
datasources:
  - name: Elasticsearch app01 logs
    uid: es-app01-logs
    type: elasticsearch
    access: proxy
    url: https://localhost:9200
    basicAuth: true
    basicAuthUser: grafana_reader
    editable: false
    jsonData:
      index: 'filebeat-*'
      timeField: '@timestamp'
      logMessageField: event.original
      tlsAuthWithCACert: true
    secureJsonData:
      basicAuthPassword: $__file{/etc/grafana/es-grafana-reader.pass}
      tlsCACert: $__file{/etc/grafana/es-http-ca.crt}
EOF
sudo cp provisioning/datasources/elasticsearch.yaml /etc/grafana/provisioning/datasources/
sudo chown root:grafana /etc/grafana/provisioning/datasources/elasticsearch.yaml
sudo systemctl restart grafana-server && sleep 10
curl -s -u admin:Lab@Grafana2026 localhost:3000/api/datasources/uid/es-app01-logs/health | jq .
git add . && git commit -qm "Provision Elasticsearch data source" && echo committed

You should see:

{
  "message": "Elasticsearch data source is healthy.",
  "status": "OK"
}
committed

logMessageField: event.original is the setting the UI calls Message field name, and it points at event.original because that’s where the Nginx module keeps the raw line; with message your log rows would come up empty. Access logs carry no log.level, so the Level field name stays empty and timeField keeps the default @timestamp, which the pipeline fills from the Nginx time.

The file holds only the location of the secrets, so it’s safe in Git. editable: false stops anyone changing the data source in the UI, which would be undone at the next restart anyway.

✅ Checkpoint: The health API returns Elasticsearch data source is healthy and the commit succeeded.

11Query logs and counts from Elasticsearch in Grafana

In Explore pick Elasticsearch app01 logs, set the time range to the last 15 minutes and try both query types:

Add the Metrics query as a panel on a new dashboard and save it in SaatVedha Lab. Compare it with the Loki status graph from Lesson 8: same traffic, but here every field was parsed at ingest and can be searched or aggregated without a parser in the query.

✅ Checkpoint: Explore shows Nginx log lines from Elasticsearch, and a Count by status code graph shows 200, 404 and 500.