Memory, Swap and Why Servers Die Under Load

CPU exhaustion is honest: things get slower in proportion. Memory exhaustion is not. A server low on memory behaves normally, then becomes strangely slow, then abruptly loses a process, and the logs frequently say nothing useful about why.

Reading free memory correctly

The most common mistake is looking at free memory and panicking. Linux deliberately uses spare memory as filesystem cache, so on a healthy busy server free memory is close to zero by design. That cache is returned to applications the moment they need it.

The figure to watch is available memory, which accounts for reclaimable cache. Available near zero is a genuine problem; free near zero is normal.

What happens as memory runs out

1. Cache is evicted. Reads that were served from memory now hit disk, and everything slows down.
2. If swap exists, pages are written to disk. Swap is orders of magnitude slower than memory, so the slowdown becomes severe.
3. If there is no swap or swap is full, the kernel’s out-of-memory killer terminates a process, usually the largest consumer, which is often your database or application.

The important detail is that stage three arrives without warning from the application’s point of view. It simply disappears.

Whether to use swap

Opinions differ; the useful distinction is what you want to happen at the limit.

A small amount of swap gives the system somewhere to put genuinely idle pages and provides a buffer that turns a sudden kill into a period of slowness you can detect and react to.

Large swap on a latency-sensitive server is usually worse than none. A system thrashing through swap is effectively unavailable while appearing to be up, which is harder to detect than a clean failure.

A common approach is a modest swap file plus monitoring that alerts on swap activity, treating any sustained swapping as a signal to add memory.

Memory-hungry patterns

• Runtimes with garbage-collected heaps that grow to fill what is available.
• Databases sized to keep a working set resident.
• In-memory caches, which are memory by definition.
• Per-connection worker models, where memory scales with concurrency.

Getting more memory

Most providers resize with a reboot. Providers that let you choose memory independently of processor count, such as the configurable model described in Kamatera and the configurable server model, are useful here because you can add memory without paying for cores you do not need.

Alert on available memory well before it reaches zero. By the time the out-of-memory killer acts, the incident has already happened.