Prometheus TSDB: storage, retention and deleting series

All commands run on prom01. Prometheus is reached locally on localhost:9090 as set up in Lesson 3, and the data folder is /var/lib/prometheus, owned by the prometheus user. Your numbers will be different from mine, depending on how long your lab has been running.

1Look inside the data folder

Run this on prom01:

sudo ls -la /var/lib/prometheus
sudo du -sh /var/lib/prometheus/* | sort -h

You should see this (your block IDs, sizes and times will differ):

total 32
drwxr-xr-x  6 prometheus prometheus  4096 Nov  8 06:00 .
drwxr-xr-x 45 root       root        4096 Nov  7 17:12 ..
drwxr-xr-x  3 prometheus prometheus  4096 Nov  8 03:00 01K9JZ7Q3M6H4RZB2W5ESXK0TA
drwxr-xr-x  3 prometheus prometheus  4096 Nov  8 05:00 01K9K6386VGMD0XKT1A3T9S2QY
drwxr-xr-x  2 prometheus prometheus  4096 Nov  8 05:00 chunks_head
-rw-r--r--  1 prometheus prometheus     0 Nov  7 17:40 lock
-rw-r--r--  1 prometheus prometheus 20001 Nov  8 05:59 queries.active
drwxr-xr-x  3 prometheus prometheus  4096 Nov  8 05:00 wal
0       /var/lib/prometheus/lock
20K     /var/lib/prometheus/queries.active
380K    /var/lib/prometheus/01K9K6386VGMD0XKT1A3T9S2QY
1.6M    /var/lib/prometheus/01K9JZ7Q3M6H4RZB2W5ESXK0TA
2.1M    /var/lib/prometheus/chunks_head
4.8M    /var/lib/prometheus/wal

The two ULID folders are persisted blocks; the bigger one has already been through compaction. Your lab only has three targets, so everything is tiny, and the WAL is the biggest piece because it holds the last few hours uncompressed.

✅ Checkpoint: You can see at least the wal, chunks_head and lock entries, all owned by prometheus.

2Read one block’s meta.json

Run this on prom01:

B=$(sudo ls /var/lib/prometheus | grep '^01' | tail -1)
sudo cat /var/lib/prometheus/$B/meta.json
sudo ls /var/lib/prometheus/$B
sudo stat -c '%s %n' /var/lib/prometheus/$B/chunks/*

You should see this (your block ID and counts will differ):

{
	"ulid": "01K9K6386VGMD0XKT1A3T9S2QY",
	"minTime": 1794110400000,
	"maxTime": 1794117600000,
	"stats": {
		"numSamples": 158664,
		"numFloatSamples": 158664,
		"numSeries": 2117,
		"numChunks": 2117
	},
	"compaction": {
		"level": 1,
		"sources": [
			"01K9K6386VGMD0XKT1A3T9S2QY"
		]
	},
	"version": 1
}
chunks  index  meta.json  tombstones
271804 /var/lib/prometheus/01K9K6386VGMD0XKT1A3T9S2QY/chunks/000001

minTime and maxTime are Unix milliseconds two hours apart, so this is a fresh level 1 block cut from the head. 158,664 samples in a 271,804 byte chunk file works out at about 1.7 bytes per sample, which matches what the docs promise.

✅ Checkpoint: meta.json shows a two-hour range (maxTime minus minTime is 7200000) and sample and series counts.

3Check the TSDB status page and API

In the browser open https://prom01.lab/ and go to Status > TSDB status. It shows head statistics and the top metrics and labels by series count. The same data comes from the API:

Run this on prom01:

curl -s localhost:9090/api/v1/status/tsdb | jq -c '.data.headStats'
curl -s localhost:9090/api/v1/status/tsdb | jq -r '.data.seriesCountByMetricName[0:5][] | "\(.value)\t\(.name)"'
curl -s localhost:9090/api/v1/status/tsdb | jq -r '.data.labelValueCountByLabelName[0:5][] | "\(.value)\t\(.name)"'

You should see this (your numbers will differ):

{"numSeries":2071,"numLabelPairs":914,"chunkCount":2071,"minTime":1794112650830,"maxTime":1794113190829}
98      node_scrape_collector_success
98      node_scrape_collector_duration_seconds
66      prometheus_http_requests_total
40      prometheus_http_request_duration_seconds_bucket
36      prometheus_http_response_size_bytes_bucket
564     __name__
88      le
64      handler
49      collector
17      device

About 2,000 series in total for three targets is normal. Notice the two node_scrape_collector_* metrics at the top: node_exporter reports a success flag and a duration for every collector on every host. You’ll drop the duration one in step 8, because nobody graphs it.

✅ Checkpoint: headStats.numSeries is in the low thousands and you can name the top metric by series count.

4Work out how much disk you need

Run this on prom01:

q(){ curl -s localhost:9090/api/v1/query --data-urlencode "query=$1" | jq -r '.data.result[0].value[1]'; }
SPS=$(q 'sum(rate(prometheus_tsdb_head_samples_appended_total[1h]))')
BPS=$(q 'prometheus_tsdb_compaction_chunk_size_bytes_sum / prometheus_tsdb_compaction_chunk_samples_sum')
echo "samples/s=$SPS bytes/sample=$BPS"
python3 -c "print(round($SPS * 15*86400 * $BPS / 1024**2), 'MiB for 15 days')"

You should see this (your numbers will differ):

samples/s=145.93 bytes/sample=1.716
311 MiB for 15 days

About 146 samples a second (roughly 2,200 series every 15 seconds) costs about 300 MiB for 15 days of blocks. Add the WAL and chunks_head, which hold a few hours uncompressed, and leave space for compaction, and the 2 GB you set next is plenty for this lab. At work, 1,000 servers with node_exporter at 15 seconds is about 45,000 samples a second, which comes to roughly 100 GB for 15 days.

✅ Checkpoint: You have a samples per second figure and a disk estimate in MiB for 15 days.

5Set retention by time and size in prometheus.yml

The Lesson 2 unit file still has --storage.tsdb.retention.time=15d, which keeps working. In 3.16 the recommended place is the config file, and a value there replaces the flag values, so always write both time and size together. Open the config with sudo nano /etc/prometheus/prometheus.yml and add this block at the top level, next to global::

Add this to /etc/prometheus/prometheus.yml:

storage:
  tsdb:
    retention:
      time: 15d
      size: 2GB

Then run this on prom01:

promtool check config /etc/prometheus/prometheus.yml
curl -s -X POST localhost:9090/-/reload
sudo journalctl -u prometheus -n 20 --no-pager | grep -i retention
curl -s localhost:9090/api/v1/status/runtimeinfo | jq -r .data.storageRetention

You should see this (your times and PID will differ):

Checking /etc/prometheus/prometheus.yml
 SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax

Nov 08 06:12:40 prom01 prometheus[1187]: time=2026-11-08T06:12:40.285Z level=INFO source=main.go:1148 msg="TSDB retention updated" duration=15d size=2GiB percentage=0
15d or 2GiB

No restart was needed. storageRetention reads “15d or 2GiB”, meaning whichever limit is reached first. The size uses powers of two, so 2GB is 2 GiB here. If you prefer to stay with flags, the same thing is --storage.tsdb.retention.size=2GB in the unit file plus a restart, but expect the flags to go away in a later major release.

✅ Checkpoint: runtimeinfo shows storageRetention “15d or 2GiB” and the journal has a “TSDB retention updated” line.

6Turn on the admin API for a short while and take a snapshot

The admin API can delete data, so it stays off by default and you switch it on only while you need it. Take a VirtualBox snapshot of prom01 before this step if you haven’t already.

Run this on prom01:

sudo sed -i 's/--web.enable-lifecycle/--web.enable-lifecycle --web.enable-admin-api/' /etc/systemd/system/prometheus.service
grep -n 'admin-api' /etc/systemd/system/prometheus.service
sudo systemctl daemon-reload && sudo systemctl restart prometheus
sleep 5
curl -s -X POST localhost:9090/api/v1/admin/tsdb/snapshot

You should see this (your snapshot name will differ):

15:  --web.enable-lifecycle --web.enable-admin-api \
{"status":"success","data":{"name":"20261108T061840Z-1584f8aa3a6f7adc"}}

The snapshot is a folder under /var/lib/prometheus/snapshots/ holding hard links to the existing blocks plus a fresh block made from the head, so it’s fast and takes almost no extra space until the originals are deleted. Copy it off the server and remove the local copy:

SNAP=20261108T061840Z-1584f8aa3a6f7adc      # use the name you got
sudo tar -C /var/lib/prometheus/snapshots -czf ~/prom-snap-$SNAP.tar.gz $SNAP
ls -lh ~/prom-snap-*.tar.gz
sudo promtool tsdb list -r /var/lib/prometheus/snapshots/$SNAP

-rw-r--r-- 1 student student 1.9M Nov  8 06:19 /home/student/prom-snap-20261108T061840Z-1584f8aa3a6f7adc.tar.gz
BLOCK ULID                  MIN TIME                       MAX TIME                       DURATION     NUM SAMPLES  NUM CHUNKS   NUM SERIES   SIZE
01K9JZ7Q3M6H4RZB2W5ESXK0TA  2026-11-07 17:40:12 +0000 UTC  2026-11-08 03:00:00 +0000 UTC  9h19m48s     812416       9370         2185         1MiB614KiB409B
01K9K6386VGMD0XKT1A3T9S2QY  2026-11-08 03:00:00 +0000 UTC  2026-11-08 05:00:00 +0000 UTC  2h0m0s       158664       2117         2117         365KiB147B
01K9KB4P1E0Z8N7R6D5HCQW3JX  2026-11-08 05:00:00 +0000 UTC  2026-11-08 06:18:40 +0000 UTC  1h18m40s     103107       2205         2205         290KiB153B

To restore, you’d stop Prometheus, move the snapshot’s block folders into an empty data folder, fix ownership and start it again. Copy the tar file to your laptop with scp if you want a real off-box backup.

✅ Checkpoint: The snapshot API returned “success”, the tar file exists in your home folder, and tsdb list shows its blocks.

7Analyse the snapshot with promtool

promtool tsdb analyze reads a block and prints the labels and metrics that cost the most. Point it at the snapshot, not at the live folder: on a running server it opens the folder read-only and leaves a tmp_dbro_sandbox... directory behind in it.

Run this on prom01:

sudo promtool tsdb analyze --limit=5 /var/lib/prometheus/snapshots/$SNAP

You should see this (shortened; your block ID and counts will differ):

Block ID: 01K9KB4P1E0Z8N7R6D5HCQW3JX
Duration: 1h18m40s
Total Series: 2205
Label names: 57
Postings (unique label pairs): 914
Postings entries (total label pairs): 9805

Label pairs most involved in churning:
67 job=prometheus
67 instance=localhost:9090
...
Highest cardinality labels:
564 __name__
88 le
64 handler
49 collector
17 device

Highest cardinality metric names:
98 node_scrape_collector_duration_seconds
98 node_scrape_collector_success
66 prometheus_http_requests_total
60 prometheus_http_request_duration_seconds_bucket
54 prometheus_http_response_size_bytes_bucket

“Highest cardinality labels” tells you how many different values each label name has. In a healthy setup you see small numbers like these; a label with tens of thousands of values (user IDs, order IDs, full URLs) is the thing to hunt. “Churning” means series that started or stopped inside the block, which is high when pods or targets come and go a lot.

✅ Checkpoint: promtool printed the block summary and the highest cardinality labels and metric names.

8Stop collecting a metric and delete its history

First stop new samples for node_scrape_collector_duration_seconds with metric_relabel_configs, which runs after the scrape and before storage. Add these lines to the node job in prometheus.yml, at the same indentation as static_configs::

Add this to the node job:

    metric_relabel_configs:
      - source_labels: [__name__]
        regex: 'node_scrape_collector_duration_seconds'
        action: drop

Then run this on prom01:

promtool check config /etc/prometheus/prometheus.yml && curl -s -X POST localhost:9090/-/reload
sleep 30
q 'scrape_samples_scraped{job="node"}'; q 'scrape_samples_post_metric_relabeling{job="node"}'
curl -s -o /dev/null -w '%{http_code}\n' -X POST -g \
  'localhost:9090/api/v1/admin/tsdb/delete_series?match[]=node_scrape_collector_duration_seconds'
curl -s -o /dev/null -w '%{http_code}\n' -X POST localhost:9090/api/v1/admin/tsdb/clean_tombstones
q 'count(node_scrape_collector_duration_seconds)'

You should see this (your sample counts will differ):

643
594
204
204
null

Each node scrape still fetches 643 samples, but only 594 are stored now. The two 204 responses mean “done, nothing to return”: delete_series wrote tombstones and clean_tombstones rewrote the affected blocks without that data. The count query returns nothing (null from jq), so the history is gone from the blocks and the head.

delete_series without start and end deletes the whole history of every matching series. Add -d “start=2026-11-08T04:00:00Z” -d “end=2026-11-08T05:00:00Z” to delete only a time range, and always test your match[] selector in the Query page first, because there is no undo except the snapshot.

✅ Checkpoint: scrape_samples_post_metric_relabeling is lower than scrape_samples_scraped, both admin calls returned 204, and the deleted metric returns no data.

9Turn the admin API off again and clean up

Run this on prom01:

sudo sed -i 's/ --web.enable-admin-api//' /etc/systemd/system/prometheus.service
sudo systemctl daemon-reload && sudo systemctl restart prometheus
sleep 5
curl -s -X POST localhost:9090/api/v1/admin/tsdb/snapshot
sudo rm -rf /var/lib/prometheus/snapshots/$SNAP

You should see this:

{"status":"error","errorType":"unavailable","error":"admin APIs disabled"}

That error is what you want to see. Keep the tar file; the snapshot folder on the server was only a staging copy, and once the blocks it links to have been deleted by retention it would start using real disk space.

✅ Checkpoint: The snapshot call now answers “admin APIs disabled” and the unit file no longer contains enable-admin-api.