What is a file descriptor — and why Too many open files happens with no files open

Your server dies with Too many open files. You check the code, and it doesn't touch files at all. It only accepts requests.

Here's the catch. To the system, almost everything is a "file." A network connection is too. And they all share one limit.

What is a file descriptor?

A file descriptor is a number the system hands a program for everything it opens.

Picture a coat check. You hand over your jacket and get a token. From then on you only show the token, and the attendant finds the jacket. A program does the same. It says "open this file," the system replies "here's number 3." Then the program asks "read from number 3."

These all get a token:

• regular files;
• network connections, meaning every visitor to your site;
• pipes between commands, the | kind;
• the terminal itself.

Now about the hooks. A program has a limited number of them. Usually 1024 on Linux and 256 in a macOS terminal. Check yours:

ulimit -n

Out of hooks, and the next visitor gets EMFILE. That's Too many open files.

How it works

The first three numbers are taken before the program even starts:

• 0 is stdin, where the program reads input;
• 1 is stdout, where it writes results;
• 2 is stderr, where it writes errors.

So the first file you open gets number 3. Try it:

python3 -c "import os; print(os.open('/etc/hostname', os.O_RDONLY))"
# 3

That also explains the odd-looking 2>&1. No magic: "send stream number 2 to wherever number 1 goes." More in what stdout and stderr are.

One more rule: the system always hands out the lowest free number. Close 0, and the next file gets 0.

Why the limit runs out with no files at all

I tested it in Python. I set the limit to 64 and opened network sockets, not a single file:

ulimit -n 64
python3 -c "
import socket
s = []
while True:
    s.append(socket.socket())
"
# OSError: [Errno 24] Too many open files

It failed at socket 61. Three numbers were already taken: 0, 1 and 2.

In practice one visitor rarely costs a single token. A server holds a connection to the browser, one to the database, and sometimes an open file. That's already three hooks. So a limit of 1024 runs out well before a thousand visitors.

The second common trap is code an AI will happily write for you. Reading 3000 files "in parallel":

await Promise.all(names.map((n) => readFile(n)));

With a limit of 256 this fails with EMFILE: too many open files. Every file opens at once and the hooks run out. The AI isn't wrong exactly, it just wrote the shortest version.

The fix is a cap: let, say, 50 readers run at a time.

const out = [];
let next = 0;
async function worker() {
  while (next < names.length) {
    const i = next++;
    out[i] = await readFile(names[i]);
  }
}
await Promise.all(Array.from({ length: 50 }, worker));

The same 3000 files with the same limit went through with no error. A bit more code, and no crash.

How to find a leak

If a program dies after a day of running rather than right away, it's almost always a leak. Something gets opened and never closed.

1. Get the process number: pgrep -f node or pgrep -f python.
2. Count its descriptors: ls /proc/PID/fd | wc -l. On macOS: lsof -p PID | wc -l.
3. Wait a minute and count again. If the number keeps growing, that's a leak.
4. See what's piling up: lsof -p PID | tail -20.

Hundreds of open sockets? Look for a connection that never closes. Hundreds of open files? Look for open without close.

The most reliable trick is to not close by hand and let the language do it:

with open("data.txt") as f:
    text = f.read()
# the file is closed here, even if an error happened inside

Does raising ulimit fix it?

Only if your program honestly needs many connections. It won't cure a leak, it just delays the crash. Check first whether the descriptor count is growing.

What about Windows?

Same idea, different name: a handle. EMFILE in Node shows up there too.

Why does a deleted file keep taking up space?

Because a program still holds the token. The system erases the data only after it's closed. Here's what that looks like in practice: why you get No space left on device.

If the terminal still feels like a black box, start there. The tokens will make more sense on their own.