Caching

Some answers are expensive to produce: a complex database query, a call to a slow third-party API, a rendered page. If the same answer is requested over and over, recomputing it every time is wasteful — and it puts the slow component under load it didn't need.

A cache keeps a copy of the answer somewhere fast (usually memory) so that the next time someone asks the same question, you can hand back the saved copy instead of doing the work again.

The problem: doing the same expensive work twice

Imagine a product page that runs a heavy database query on every view. The data barely changes minute to minute, yet each visitor triggers the full query. As traffic grows, the database becomes the bottleneck — even though almost every request is asking for the same thing. A database that comfortably handles a few hundred queries a second will drown at a thousand page views a second, and no amount of tuning changes the fact that it's answering the same question over and over.

The insight: reads usually outnumber writes, and the same few items tend to be requested far more than the rest (hot keys). That repetition is exactly what a cache exploits.

How it works

A cache sits between the application and the slow source (a database, an API). On each read, the app checks the cache first:

Cache hit — the answer is already there. Return it immediately. The database is never touched.
Cache miss — the answer isn't there. Fetch it from the source, store it in the cache, then return it. The next identical request will be a hit.

When the app does this itself, as in the code below, the pattern is called cache-aside. (Read-through is the same flow handled by the cache layer, so the app only ever talks to the cache.) The first request for an item pays the full cost; everyone after rides for nearly free, until the entry expires.

Step through the first two requests for one product. Before the second one runs, predict whether it hits and what it costs. Then, at the busy hour, flip to No cache to replay the same traffic without one.

Cache-aside: the first read pays, the rest ride free

Request #1 wants product 42. The checks the first, but it's empty: nobody has asked for p:42 yet. That's a miss.

1 / 5

Read all steps as text

Cold cache. Request #1 wants product 42. The app checks the cache first, but it's empty: nobody has asked for p:42 yet. That's a miss.
Read the database. On a miss the app runs the real query against the database, which takes about 50 ms. This is the slow path a cache exists to skip.
Keep a copy. Before replying, the app saves the answer in the cache as p:42 → $89. Request #1 took 52 ms in all: the miss, the query and the save.Question: Request #2 asks for product 42 a moment later. Hit or miss, and how long does it take? Answer: Hit, about 1 ms: the copy from request #1 is still there. The cache already holds p:42, so the app answers straight from memory and never touches the database.
Hit. Request #2 finds p:42 in the cache: a hit, answered in about 1 ms. The database never hears about it.Question: Now a busy hour: 1,000 reads a second, mostly for a few popular products, and 95% of them are hits. How many queries a second reach the database? Answer: 50. Only misses reach the database: 5% of 1,000 is 50 queries a second. Then flip to No cache to see the same hour without one.
Busy hour. At 1,000 reads a second with 95% hits, only 50 queries a second reach the database, far below its capacity. The average reply takes about 3.6 ms.

Note

Why it's so much faster: an in-memory cache like Redis or Memcached usually answers in well under a millisecond, while a query that joins tables or touches disk can take tens of milliseconds. (A cache inside the app's own process is faster still: microseconds.) If 95% of reads are hits, you've removed 95% of the read load from your database and made most requests dramatically faster at the same time.

Check yourself

Your database tops out at about 400 queries a second. You put a cache in front of it, traffic is 2,000 reads a second, and 90% of reads are hits. What load does the database see?

Keeping the cache fresh

A cached copy is a snapshot. The moment the underlying data changes, the cache can be stale — serving an old answer. Two mechanisms keep this under control:

TTL (time to live) — each entry is stamped with an expiry. After the TTL elapses, the entry is dropped and the next read is a miss, refreshing it from the source. A short TTL means fresher data but more misses; a long TTL means fewer misses but more staleness.
Eviction — memory is finite, so when the cache fills up it must drop something. LRU (least-recently-used) is the common choice: evict whatever hasn't been touched in the longest time, keeping the hot keys resident.

Step through a price change on a cached product. You'll predict what a shopper sees ten seconds after the price drops, then flip to Delete on write to see invalidation close the stale window. The last step shows a problem neither approach solves on its own: a hot key that vanishes from the cache.

The price changed, but the cache didn't hear

At t = 0 a miss loaded product 42's price, $89, into the with a 60 s . A shopper at t = 5 s straight from the cache.

1 / 5

Read all steps as text

Cached. At t = 0 a miss loaded product 42's price, $89, into the cache with a 60 s TTL. A shopper at t = 5 s gets $89 straight from the cache.
Price drops. At t = 20 s the price drops to $79 in the database. Nothing tells the cache, so it still holds $89.Question: A shopper opens the page at t = 30 s. Which price do they see? Answer: $89: the cached copy lives until t = 60 s. The cache has no idea the database changed. Until its TTL runs out, every read is a hit on the old copy.
Stale read. The shopper at t = 30 s sees $89: a stale read. Every hit serves the old price until the entry expires, and the stale window keeps growing.
Expires. At t = 60 s the entry expires. The read at 62 s misses, loads $79 and caches it. Shoppers saw the old price for 40 s: with a TTL alone, data can be as stale as the TTL is long.Question: Suppose product 42 is hot: 2,000 reads a second, and reloading it takes 50 ms. When the entry expires, about how many reads miss and query the database before the new copy is in? Answer: About 100. Every read in that 50 ms gap misses: 2,000 a second × 0.05 s = 100 identical queries at once.
Stampede. That burst is a cache stampede: about 100 identical queries hit the database in one go. The fix is to let one request reload the entry while the others wait for it, or keep serving the old copy for a moment.

Watch out

Cache invalidation is famously hard. Deleting on every write sounds airtight, but there are gaps: a write path that forgets to delete, a second service that writes to the same table, or a slow read that loads the old value just before the write and saves it to the cache just after the delete. Keep a sensible TTL as a safety net even when you invalidate explicitly, so any stale entry eventually ages out. As the saying goes, there are only two hard things in computer science: cache invalidation and naming things.

The trade-offs

Caching is not free:

Staleness — you accept that reads may be slightly out of date. Fine for a product description; dangerous for an account balance.
Cold starts — an empty cache (after a deploy or restart) sends a burst of misses straight to the database. If your database only survives because of the cache, a cold start can take it down.
Stampedes — when a hot key expires or is deleted, every request in the reload gap misses at once. Common defences: let one request reload while the others wait for it (often called request coalescing or single flight), keep serving the old value while one request refreshes it in the background, and add a little random jitter to TTLs so many keys don't expire in the same second.
Complexity — another moving part to run, monitor, and reason about, plus the invalidation logic.
Memory cost — fast storage isn't free; you cache the valuable subset, not everything.

Check yourself

Every 60 seconds, right as a very popular product's cache entry expires, your database CPU spikes. What's the best fix?

When to reach for it

Caching pays off when reads vastly outnumber writes, when the same items are requested repeatedly, and when slightly stale data is acceptable. It's one of the highest-leverage performance tools available — often a few lines in front of a hot query.

It pairs naturally with load balancing: the load balancer spreads requests across servers, and a shared cache keeps each of those servers from re-doing the same expensive work.