Out of memory and the OOM killer

This lab kills processes on purpose, so take a snapshot of rocky01 first: VBoxManage snapshot rocky01 take before-l74 on the laptop. Keep rocky01 at 2 GB RAM; with more, the numbers below change and the whole-machine test needs bigger values. Work over SSH as student, make a folder with mkdir -p ~/lab74, and check that stress-ng from Lesson 53 is there with stress-ng --version (if not: sudo dnf install -y stress-ng, or sudo apt install stress-ng from universe on Ubuntu).

1Check memory, swap and the overcommit rules

Run this:

free -h
swapon --show
sysctl vm.overcommit_memory vm.overcommit_ratio vm.panic_on_oom
grep -E '^(CommitLimit|Committed_AS)' /proc/meminfo

You should see something like this:

               total        used        free      shared  buff/cache   available
Mem:           1.7Gi       462Mi       988Mi       8.8Mi       421Mi       1.2Gi
Swap:          2.0Gi          0B       2.0Gi
NAME      TYPE      SIZE USED PRIO
/dev/dm-1 partition   2G   0B   -2
vm.overcommit_memory = 0
vm.overcommit_ratio = 50
vm.panic_on_oom = 0
CommitLimit:     2989154 kB
Committed_AS:    1187340 kB

About 1.2Gi is available and 2 GB of swap sits unused on the rl-swap logical volume (dm-1). overcommit_memory 0 is the heuristic mode: the kernel refuses only a single request that is obviously bigger than RAM plus swap. Committed_AS is what all processes have reserved so far, and CommitLimit (swap plus half of RAM) only matters in strict mode 2.

✅ Checkpoint: You know rocky01’s available memory, its swap size, and that overcommit is in mode 0 and panic_on_oom is 0.

2See who the OOM killer would pick right now

Create a small script that ranks every process by oom_score (paste this whole block):

cat > ~/lab74/oomtop.sh <<'EOF'
#!/bin/bash
# oomtop.sh: the processes the OOM killer would pick first (Lesson 74)
n=$1
[ -n "$n" ] || n=8
printf '%7s %5s %6s %s\n' PID SCORE ADJ COMMAND
for d in /proc/[0-9]*; do
    score=$(cat "$d/oom_score" 2>/dev/null) || continue
    adj=$(cat "$d/oom_score_adj" 2>/dev/null)
    printf '%7s %5s %6s %s\n' "$(basename "$d")" "$score" "$adj" "$(cat "$d/comm" 2>/dev/null)"
done | sort -k2,2nr | head -n "$n"
EOF
chmod +x ~/lab74/oomtop.sh
~/lab74/oomtop.sh 6
for p in sshd systemd-journal systemd-udevd; do printf '%-16s adj %5s  score %s\n' $p "$(cat /proc/$(pgrep -o -x $p)/oom_score_adj)" "$(cat /proc/$(pgrep -o -x $p)/oom_score)"; done

You should see something like this:

    PID SCORE    ADJ COMMAND
   1838   735    100 systemd
    812   674      0 firewalld
   1046   672      0 tuned
    868   670      0 NetworkManager
    790   670      0 polkitd
   2077   667      0 bash
sshd             adj -1000  score 0
systemd-journal  adj  -250  score 503
systemd-udevd    adj -1000  score 0

Anyone can read these files, so no sudo is needed. On kernel 6.12 a small process with adj 0 scores about 667, and the score rises with its share of RAM plus swap. Your own user manager (systemd --user, PID 1838 here) sits on top because user@.service gives it an adj of 100, and its children inherit that.

The sshd listener is at -1000 because OpenSSH sets that for itself on Linux (your login session gets 0 back), journald has -250 from its unit, and udevd -1000. PID 1 isn’t in the list at all: it can never be chosen.

Try it yourself: build the top-memory command in the Command Builder, to compare RSS with these scores.

✅ Checkpoint: oomtop.sh prints six processes and you can explain why sshd and systemd-udevd score 0.

3Run a service with a memory limit

Create the hog script, its settings file and a unit with MemoryMax=300M (paste this whole block):

sudo tee /usr/local/bin/saat-memhog.py >/dev/null <<'EOF'
#!/usr/bin/python3
# saat-memhog.py: grows by 50 MiB a second up to a target, then holds it (Lesson 74)
import sys
import time

target = int(sys.argv[1]) if len(sys.argv) > 1 else 200
held = []
while len(held) * 50 < target:
    held.append(b"x" * (50 * 1024 * 1024))
    print(f"saat-memhog: holding {len(held) * 50} MiB", flush=True)
    time.sleep(1)
print("saat-memhog: target reached, holding", flush=True)
time.sleep(86400)
EOF
sudo chmod 755 /usr/local/bin/saat-memhog.py
echo 'MB=200' | sudo tee /etc/saat-memhog.env
sudo tee /etc/systemd/system/saat-memhog.service >/dev/null <<'EOF'
[Unit]
Description=Memory hog for the OOM lab (Lesson 74)

[Service]
User=appsvc
EnvironmentFile=/etc/saat-memhog.env
ExecStart=/usr/local/bin/saat-memhog.py $MB
MemoryMax=300M
MemorySwapMax=0
EOF
sudo systemctl daemon-reload
sudo systemctl start saat-memhog
sleep 6; sudo journalctl -u saat-memhog -o cat --no-pager | grep saat-memhog:

You should see this:

MB=200
saat-memhog: holding 50 MiB
saat-memhog: holding 100 MiB
saat-memhog: holding 150 MiB
saat-memhog: holding 200 MiB
saat-memhog: target reached, holding

Now read the limit from the cgroup side:

cd /sys/fs/cgroup/system.slice/saat-memhog.service
cat memory.max memory.current memory.swap.max memory.events
cd

You should see something like this:

314572800
223870976
0
low 0
high 0
max 0
oom 0
oom_kill 0
oom_group_kill 0

systemd turned MemoryMax=300M into 314572800 bytes in memory.max, and MemorySwapMax=0 into a swap limit of 0. memory.current (about 213 MiB) counts the 200 MiB of data, Python itself and any page cache the service caused. In memory.events, max counts how often the limit was touched and oom_kill how many processes the kernel killed here: none yet.

✅ Checkpoint: saat-memhog holds 200 MiB, memory.max is 314572800 and oom_kill is 0.

4Push the service past its limit

Ask for 600 MiB and restart:

echo 'MB=600' | sudo tee /etc/saat-memhog.env
sudo systemctl restart saat-memhog
sleep 10; sudo journalctl -u saat-memhog -o cat --no-pager --since "20 sec ago"
systemctl status saat-memhog --no-pager | grep Active:

You should see something like this:

MB=600
Started saat-memhog.service - Memory hog for the OOM lab (Lesson 74).
saat-memhog: holding 50 MiB
saat-memhog: holding 100 MiB
saat-memhog: holding 150 MiB
saat-memhog: holding 200 MiB
saat-memhog: holding 250 MiB
saat-memhog.service: A process of this unit has been killed by the OOM killer.
saat-memhog.service: Main process exited, code=killed, status=9/KILL
saat-memhog.service: Failed with result 'oom-kill'.
saat-memhog.service: Consumed 1.204s CPU time, 300M memory peak.
     Active: failed (Result: oom-kill) since Tue 2026-10-27 11:32:10 IST; 15s ago

The script reached 250 MiB of data, the next 50 MiB would have taken the cgroup past 300M, and the kernel killed it. Read the systemd lines in order: the OOM killer struck a process of this unit, the main process died from signal 9 (SIGKILL), and with the default OOMPolicy=stop the unit ended as failed with result oom-kill. Nothing else on rocky01 noticed, because the limit belonged to this service alone.

On Ubuntu 26.04 (systemd 259) the first systemd line reads “The kernel OOM killer killed some processes in this unit.” instead; Rocky 10 runs systemd 257, which uses the older wording. Search your logs for oom-kill, which is the same on both.

✅ Checkpoint: systemctl status shows Active: failed (Result: oom-kill).

5Read the kernel’s side of the story

Run this:

sudo journalctl -k --since "5 min ago" -g 'invoked oom-killer|memory: usage|swap: usage|oom-kill:|killed process' --no-pager

You should see something like this:

Oct 27 11:32:10 rocky01 kernel: saat-memhog.py invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
Oct 27 11:32:10 rocky01 kernel: memory: usage 307200kB, limit 307200kB, failcnt 0
Oct 27 11:32:10 rocky01 kernel: swap: usage 0kB, limit 0kB, failcnt 0
Oct 27 11:32:10 rocky01 kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/system.slice/saat-memhog.service,task_memcg=/system.slice/saat-memhog.service,task=saat-memhog.py,pid=3128,uid=991
Oct 27 11:32:10 rocky01 kernel: Memory cgroup out of memory: Killed process 3128 (saat-memhog.py) total-vm:330112kB, anon-rss:297216kB, file-rss:6400kB, shmem-rss:0kB, UID:991 pgtables:672kB oom_score_adj:0

Five lines tell the whole story. Line one names the process that needed memory and couldn’t get it, the “invoked” process, here the same script. The memory and swap lines show usage at the limit and a swap limit of 0, which is what makes this a cgroup OOM (CONSTRAINT_MEMCG, oom_memcg=/system.slice/saat-memhog.service) and not a whole-machine one.

The last line is the one to put in the ticket: PID, name, user (UID 991 is appsvc) and anon-rss, the private memory it had built up, 297216 kB or about 290 MiB. Between those lines the full log also has a memory report and a candidate table; sudo dmesg -T | grep -A3 "Tasks state" shows the table, with sizes in 4 KiB pages:

Oct 27 11:32:10 rocky01 kernel: Tasks state (memory values in pages):
Oct 27 11:32:10 rocky01 kernel: [  pid  ]   uid  tgid total_vm      rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
Oct 27 11:32:10 rocky01 kernel: [   3128]   991  3128    82528    75904    74304     1600         0   688128        0             0 saat-memhog.py

✅ Checkpoint: You found the “Memory cgroup out of memory: Killed process” line and can say which process, which user and how much anon-rss.

6Change the limit without editing the unit

Run this:

sudo systemctl set-property saat-memhog MemoryMax=700M
systemctl show saat-memhog -p MemoryMax --value
cat /etc/systemd/system.control/saat-memhog.service.d/50-MemoryMax.conf
sudo systemctl restart saat-memhog
sleep 14; sudo journalctl -u saat-memhog -n 1 -o cat --no-pager
cat /sys/fs/cgroup/system.slice/saat-memhog.service/memory.max

You should see something like this:

734003200
# /etc/systemd/system.control/saat-memhog.service.d/50-MemoryMax.conf
# This is a drop-in unit file extension, created via "systemctl set-property"
# or an equivalent operation. Do not edit.
[Service]
MemoryMax=734003200
saat-memhog: target reached, holding
734003200

set-property changes the running cgroup at once and saves the value as a drop-in under /etc/systemd/system.control, so it survives a reboot (add --runtime for a temporary change). That makes it the quickest safe lever during an incident: a runaway service gets a ceiling without a restart. Here the restart was only needed to let the killed service start again.

✅ Checkpoint: saat-memhog reached its 600 MiB target and memory.max shows 734003200.

7Run the whole VM out of memory with stress-ng

Stop the hog, switch swap off for the test, and ask for more memory than rocky01 has:

sudo systemctl stop saat-memhog
sudo swapoff -a
free -h
stress-ng --vm 1 --vm-bytes 1500M --vm-keep --oomable -t 60s
sudo journalctl -k --since "2 min ago" -g 'invoked oom-killer|oom-kill:|killed process|oom_reaper' --no-pager
sudo swapon -a

You should see something like this:

               total        used        free      shared  buff/cache   available
Mem:           1.7Gi       455Mi       1.0Gi       8.8Mi       410Mi       1.2Gi
Swap:             0B          0B          0B
stress-ng: info:  [2911] setting to a 1 min run per stressor
stress-ng: info:  [2911] dispatching hogs: 1 vm
stress-ng: info:  [2911] skipped: 0
stress-ng: info:  [2911] passed: 1: vm (1)
stress-ng: info:  [2911] failed: 0
stress-ng: info:  [2911] metrics untrustworthy: 0
stress-ng: info:  [2911] successful run completed in 6.84 secs
Oct 27 11:41:52 rocky01 kernel: stress-ng-vm invoked oom-killer: gfp_mask=0x140dca(GFP_HIGHUSER_MOVABLE|__GFP_COMP|__GFP_ZERO), order=0, oom_score_adj=1000
Oct 27 11:41:52 rocky01 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/session-4.scope,task=stress-ng-vm,pid=2913,uid=1000
Oct 27 11:41:52 rocky01 kernel: Out of memory: Killed process 2913 (stress-ng-vm) total-vm:1573756kB, anon-rss:1203840kB, file-rss:2304kB, shmem-rss:0kB, UID:1000 pgtables:2412kB oom_score_adj:1000
Oct 27 11:41:52 rocky01 kernel: oom_reaper: reaped process 2913 (stress-ng-vm), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

With swap off, many cloud servers look like this. 1500M is less than the 1.7Gi total, so the heuristic overcommit let stress-ng reserve it, and the kernel only ran out when the worker wrote to those pages. This time the line says “Out of memory” and global_oom: the whole machine was short, not one cgroup.

The victim was stress-ng-vm with oom_score_adj 1000. stress-ng sets that on its own workers on purpose, and --oomable tells it to count an OOM kill as a pass instead of restarting the worker. A real memory hog doesn’t volunteer like that, which is what the Break-It section shows. swapon -a turns rl-swap back on from /etc/fstab.

✅ Checkpoint: The kernel log shows “Out of memory: Killed process … (stress-ng-vm)” with global_oom, and swapon --show lists /dev/dm-1 again.

8Set oom_score_adj by hand with choom

Run this:

choom -n 500 -- sleep 600 &
choom -p $!
choom -p $! -n -100
sudo choom -p $! -n -100
choom -p $!
kill %1

You should see something like this:

[1] 3301
pid 3301's current OOM score: 1000
pid 3301's current OOM score adjust value: 500
choom: failed to set score adjust value: Permission denied
pid 3301's OOM score adjust value changed from 500 to -100
pid 3301's current OOM score: 600
pid 3301's current OOM score adjust value: -100

choom (util-linux) reads and writes /proc/PID/oom_score_adj. Any user may make their own process a likelier victim, but going below 0 needs root (CAP_SYS_RESOURCE), otherwise every user would mark their own jobs as untouchable. For services, put the value in the unit with OOMScoreAdjust= instead, so it is there after every restart.

✅ Checkpoint: You raised the score as student, were refused going negative, and root set -100.

9Check systemd-oomd and memory pressure on ubu01

On ubu01, run this:

systemctl is-active systemd-oomd
apt-cache policy systemd-oomd | head -n 3
cat /proc/pressure/memory

You should see something like this:

inactive
systemd-oomd:
  Installed: (none)
  Candidate: 259.9-0ubuntu0.1
some avg10=0.00 avg60=0.00 avg300=0.00 total=1834
full avg10=0.00 avg60=0.00 avg300=0.00 total=1102

Ubuntu Server 26.04 doesn’t install systemd-oomd; the desktop flavours do. /proc/pressure/memory is the PSI file oomd would watch: “some” is the share of time at least one task waited for memory, “full” the time all of them did, averaged over 10, 60 and 300 seconds. On a healthy server they sit at 0.00. If you ever see oomd kill something on a desktop or on a server where someone installed it, its own journal (journalctl -u systemd-oomd) says which cgroup and why.

✅ Checkpoint: systemd-oomd is inactive on ubu01 and you can read the some and full lines in /proc/pressure/memory.