“Too many open files”: find the cause and fix it

Take a snapshot of rocky01 first (VBoxManage snapshot rocky01 take before-l75 on the laptop). Steps 1 to 8 run on rocky01 as student, with sudo where shown; ubu01 joins in the Break-It part. Make a folder with mkdir -p ~/lab75 and check that lsof is there (sudo dnf install -y lsof strace adds it and strace if not).

1Make the error happen in your own shell

Create a script that opens files until it can’t, and run it in a subshell with a soft limit of 256:

cat > ~/lab75/fd-eater.py <<'EOF'
#!/usr/bin/python3
# fd-eater.py: opens files until the kernel says no (Lesson 75)
import resource
soft, hard = resource.getrlimit(resource.RLIMIT_NOFILE)
print(f"soft limit {soft}, hard limit {hard}", flush=True)
held = []
while True:
    held.append(open("/etc/hostname"))
EOF
(ulimit -Sn 256; python3 ~/lab75/fd-eater.py)

You should see something like this:

soft limit 256, hard limit 524288
Traceback (most recent call last):
  File "/home/student/lab75/fd-eater.py", line 8, in <module>
OSError: [Errno 24] Too many open files: '/etc/hostname'

The brackets run the command in a subshell, so your login shell keeps its own limit. Notice the traceback has no source line under “line 8”: Python needs a free descriptor to open its own script and print that line, and there wasn’t one left. Now look one layer lower, at the system call that failed:

(ulimit -Sn 64; strace -e trace=openat python3 ~/lab75/fd-eater.py 2>&1 | grep -m1 -B1 EMFILE)

You should see this:

openat(AT_FDCWD, "/etc/hostname", O_RDONLY|O_CLOEXEC) = 63
openat(AT_FDCWD, "/etc/hostname", O_RDONLY|O_CLOEXEC) = -1 EMFILE (Too many open files)

Descriptor 63 was the last one allowed under a limit of 64 (0 to 63), and the next openat() returned EMFILE. Every language’s “Too many open files” message is this one kernel error with different words around it.

✅ Checkpoint: You saw Errno 24 from Python and EMFILE from strace, and ulimit -Sn in your own shell still prints 1024.

2Look at the whole system first

Create a script that compares each process’s open descriptors with its soft limit, then run it with sudo so it can read every process:

cat > ~/lab75/fdtop.sh <<'EOF'
#!/bin/bash
# fdtop.sh: open files per process against its soft limit (Lesson 75)
n=$1
[ -n "$n" ] || n=8
printf '%6s %7s %5s %7s %s\n' FDS LIMIT USE% PID COMMAND
for d in /proc/[0-9]*; do
    lim=$(awk '/^Max open files/ {print $4}' "$d/limits" 2>/dev/null)
    [ -n "$lim" ] || continue
    fds=$(find "$d/fd" -mindepth 1 2>/dev/null | wc -l)
    pct=0
    [ "$lim" = unlimited ] || pct=$((fds * 100 / lim))
    printf '%6s %7s %4s%% %7s %s\n' "$fds" "$lim" "$pct" "$(basename "$d")" "$(cat "$d/comm" 2>/dev/null)"
done | sort -k3,3nr -k1,1nr | head -n "$n"
EOF
chmod +x ~/lab75/fdtop.sh
cat /proc/sys/fs/file-nr
sysctl fs.file-max fs.nr_open
sudo ~/lab75/fdtop.sh 6

You should see something like this:

1408	0	9223372036854775807
fs.file-max = 9223372036854775807
fs.nr_open = 1073741816
   FDS   LIMIT  USE%     PID COMMAND
    31    1024    3%     868 NetworkManager
    14    1024    1%     812 firewalld
    13    1024    1%     790 polkitd
    12    1024    1%    1046 tuned
   118  524288    0%       1 systemd
    74  524288    0%     612 systemd-journal

file-nr has three numbers: open file handles on the whole machine (1408), a field that is always 0 on current kernels, and file-max. 1408 against nine quintillion means ENFILE is not your problem. fdtop.sh sorts by how full each process is; on a quiet rocky01 nothing is above a few percent, and services such as NetworkManager run with the default soft limit of 1024.

✅ Checkpoint: file-nr shows a small first number and no process in fdtop.sh is above 10 percent.

3Start a service that leaks file handles

Create a pretend billing API that opens a rates file on every request and never closes it (paste this whole block):

sudo mkdir -p /var/lib/saat-leaky
echo 'code,rate' | sudo tee /var/lib/saat-leaky/rates.csv
sudo tee /usr/local/bin/saat-leaky.py >/dev/null <<'EOF'
#!/usr/bin/python3
# saat-leaky.py: a pretend billing API that forgets to close a file on every request (Lesson 75)
import time

DATA = "/var/lib/saat-leaky/rates.csv"
forgotten = []  # the bug: every request keeps its file object, so nothing is closed
n = 0
while True:
    n += 1
    try:
        f = open(DATA)
        f.readline()
        forgotten.append(f)
        if n % 200 == 0:
            print(f"request {n}: ok", flush=True)
    except OSError as e:
        if n % 200 == 0:
            print(f"request {n}: ERROR {e}", flush=True)
    time.sleep(0.05)
EOF
sudo chmod 755 /usr/local/bin/saat-leaky.py
printf '[Unit]\nDescription=Leaky billing API (Lesson 75)\n[Service]\nUser=appsvc\nExecStart=/usr/local/bin/saat-leaky.py\n' | sudo tee /etc/systemd/system/saat-leaky.service >/dev/null
sudo systemctl daemon-reload
sudo systemctl start saat-leaky
PID=$(systemctl show -p MainPID --value saat-leaky)
for i in 1 2 3; do echo "$(date +%T) $(sudo ls /proc/$PID/fd | wc -l)"; sleep 10; done

You should see something like this:

code,rate
11:20:05 63
11:20:15 258
11:20:25 453

About 20 requests a second, and the count grows by nearly 200 every 10 seconds without ever falling. That straight climbing line is the signature of a leak. A busy but healthy service goes up and down with traffic; this one only goes up.

Keep the PID variable in this shell for the next steps.

✅ Checkpoint: The descriptor count rises by roughly 200 per 10 seconds.

4See what the handles are with lsof

Run this:

sudo lsof -a -p $PID -d 0-999999 | awk 'NR>1 {print $5, $9}' | sort | uniq -c | sort -rn | head -n 5
sudo lsof -p $PID | wc -l
sudo ls /proc/$PID/fd | wc -l

You should see something like this:

    612 REG /var/lib/saat-leaky/rates.csv
      2 unix type=STREAM
      1 CHR /dev/null
655
615

-a -p $PID -d 0-999999 means “this PID AND only numbered descriptors”. Without the -d filter lsof also lists the working directory, the program file and every memory-mapped library, which is why the plain count (655, including the header) is higher than /proc/PID/fd (615). TYPE and NAME grouped together give the answer in one line: hundreds of copies of the same rates.csv, plus stdin (/dev/null) and the two journal sockets for stdout and stderr.

✅ Checkpoint: lsof shows one file name with a count in the hundreds.

5Watch it hit the wall

Wait until about a minute has passed since the start, then run:

sudo journalctl -u saat-leaky -o cat --no-pager | tail -n 4
systemctl status saat-leaky --no-pager | grep Active:
sudo ~/lab75/fdtop.sh 2

You should see something like this:

request 800: ok
request 1000: ok
request 1200: ERROR [Errno 24] Too many open files: '/var/lib/saat-leaky/rates.csv'
request 1400: ERROR [Errno 24] Too many open files: '/var/lib/saat-leaky/rates.csv'
     Active: active (running) since Tue 2026-10-27 11:20:02 IST; 1min 12s ago
   FDS   LIMIT  USE%     PID COMMAND
  1024    1024  100%    2534 saat-leaky.py
    31    1024    3%     868 NetworkManager

The service is still “active (running)”, every request now fails, and fdtop.sh shows it at 1024 of 1024. This half-dead state is common at work: systemd sees a running process, a simple port check passes, and only the users and the application log know something is wrong.

✅ Checkpoint: The journal shows Errno 24 and fdtop.sh shows saat-leaky.py at 100 percent.

6Buy time with prlimit, without a restart

Raise the soft limit of the running process to 4096:

sudo prlimit --pid $PID --nofile=4096:524288
sudo prlimit --pid $PID --nofile
grep 'open files' /proc/$PID/limits
sleep 20; sudo journalctl -u saat-leaky -o cat --no-pager | tail -n 2

You should see something like this:

RESOURCE DESCRIPTION              SOFT   HARD UNITS
NOFILE   max number of open files 4096 524288 files
Max open files            4096                 524288               files
request 1600: ok
request 1800: ok

prlimit (util-linux) changes the limits of a running process, the same way ulimit does for your shell. Requests succeed again at once, which makes it a useful emergency lever when a restart would hurt. It’s also gone at the next restart, it doesn’t help programs that read their limit only at startup, and here it buys about two and a half minutes, because the leak keeps going at 20 handles a second.

✅ Checkpoint: /proc/$PID/limits shows 4096 and the journal says ok again.

7Fix the leak, and set a limit you chose

Install the fixed code, give the service its own limit with a drop-in, restart and measure again:

sudo tee /usr/local/bin/saat-leaky.py >/dev/null <<'EOF'
#!/usr/bin/python3
# saat-leaky.py, fixed: the with block closes the file after every request (Lesson 75)
import time

DATA = "/var/lib/saat-leaky/rates.csv"
n = 0
while True:
    n += 1
    try:
        with open(DATA) as f:
            f.readline()
        if n % 200 == 0:
            print(f"request {n}: ok", flush=True)
    except OSError as e:
        if n % 200 == 0:
            print(f"request {n}: ERROR {e}", flush=True)
    time.sleep(0.05)
EOF
printf '[Service]\nLimitNOFILE=8192\n' | sudo systemctl edit --stdin saat-leaky
sudo systemctl restart saat-leaky
systemctl show saat-leaky -p LimitNOFILE -p LimitNOFILESoft
PID=$(systemctl show -p MainPID --value saat-leaky)
grep 'open files' /proc/$PID/limits
for i in 1 2 3; do echo "$(date +%T) $(sudo ls /proc/$PID/fd | wc -l)"; sleep 10; done

You should see something like this:

LimitNOFILE=8192
LimitNOFILESoft=8192
Max open files            8192                 8192                 files
11:31:40 3
11:31:50 3
11:32:00 3

The with block closes the file at the end of every request, so the count stays at 3 (stdin, stdout, stderr) however long it runs. The drop-in sets 8192 for soft and hard, sized for a real API that might hold a few thousand connections at peak, and it survives restarts and reboots, unlike prlimit. For the full story of where LimitNOFILE comes from, see Lesson 51.

Try it yourself: build the LimitNOFILE drop-in for any service in the Command Builder.

✅ Checkpoint: The count stays flat at 3 and /proc/PID/limits shows 8192 in both columns.

8Check how Nginx gets its limit

Read the Nginx settings that matter and the limits its worker processes have:

grep -nE '^(user|worker_processes|worker_rlimit_nofile|error_log)|worker_connections' /etc/nginx/nginx.conf
for p in $(pgrep -f 'nginx: worker'); do grep 'open files' /proc/$p/limits; done
systemctl show nginx -p LimitNOFILE -p LimitNOFILESoft

You should see something like this:

5:user nginx;
6:worker_processes auto;
7:error_log /var/log/nginx/error.log;
14:    worker_connections 1024;
Max open files            1024                 524288               files
Max open files            1024                 524288               files
LimitNOFILE=524288
LimitNOFILESoft=1024

Two workers (one per vCPU), each allowed 1024 connections and each with a soft limit of 1024 descriptors, inherited from systemd’s default. A worker also needs descriptors for log files, the listening sockets and, when it proxies, a second socket per request to the backend. So worker_connections 1024 with a limit of 1024 is already tight, and raising worker_connections alone makes it worse.

The Break-It section shows what happens and the Nginx way to fix it, worker_rlimit_nofile.

✅ Checkpoint: You can say how many workers Nginx runs on rocky01 and that each has a soft limit of 1024.