Caching

Cache Levels

From fastest to slowest:

L1/L2/L3 CPU cache     ~1-10ns   — managed by CPU
Process memory          ~100ns    — your in-process Map/LRU
Redis/Memcached         ~0.5ms    — network hop to cache server
Database                ~5-50ms   — disk or buffer cache
CDN edge                ~10-50ms  — geographically close edge node

Pick the right level for the data’s characteristics:

• In-process: fastest, no network, but lost on restart, not shared across instances
• Redis: shared across all instances, survives restart, slightly slower
• CDN: for public static content — offloads origin entirely

In-Process LRU Cache

import LRU from 'lru-cache';

const userCache = new LRU<string, User>({
  max: 1000,                // max 1000 entries
  ttl: 5 * 60 * 1000,      // 5 minute TTL
  updateAgeOnGet: false,    // TTL doesn't reset on read
});

async function getUser(userId: string): Promise<User> {
  const cached = userCache.get(userId);
  if (cached) return cached;

  const user = await db.query('SELECT * FROM users WHERE id = $1', [userId]);
  if (user) userCache.set(userId, user.rows[0]);
  return user.rows[0];
}

When to use: reference data that changes rarely (user roles, feature flags, config). Data that’s expensive to fetch but small enough to fit in memory.

When NOT to use: data that changes frequently, data where stale reads are unacceptable, data shared between requests in a stateless service (will be inconsistent across instances).

Redis Caching

import { createClient } from 'redis';

const redis = createClient({ url: process.env.REDIS_URL });
await redis.connect();

async function getCachedOrder(orderId: string): Promise<Order | null> {
  const cached = await redis.get(`order:${orderId}`);
  if (cached) return JSON.parse(cached);

  const order = await db.findOrder(orderId);
  if (order) {
    await redis.setEx(
      `order:${orderId}`,
      300,           // 5 minute TTL
      JSON.stringify(order)
    );
  }
  return order;
}

// Invalidate on update
async function updateOrder(orderId: string, data: Partial<Order>): Promise<Order> {
  const order = await db.updateOrder(orderId, data);
  await redis.del(`order:${orderId}`);   // invalidate cache
  return order;
}

Cache warming: pre-populate cache before traffic hits:

async function warmCache() {
  const topProducts = await db.query(
    'SELECT * FROM products ORDER BY view_count DESC LIMIT 1000'
  );
  
  await Promise.all(topProducts.rows.map(product =>
    redis.setEx(`product:${product.id}`, 3600, JSON.stringify(product))
  ));
}

Cache Stampede (Thundering Herd)

Cache expires. 1000 concurrent requests all see a cache miss and all hit the database simultaneously.

Time T:   cache expires
T+0ms:    request 1 sees miss, starts DB query
T+0ms:    request 2 sees miss, starts DB query
T+0ms:    request 1000 sees miss, starts DB query
T+50ms:   1000 DB queries complete, all populate cache

1000 DB queries instead of 1.

Fix 1: Mutex (Single Flight)

Only one request fetches, others wait:

import pLimit from 'p-limit';

const fetchLocks = new Map<string, Promise<any>>();

async function getWithLock<T>(
  key: string,
  fetcher: () => Promise<T>,
  ttl: number
): Promise<T> {
  const cached = await redis.get(key);
  if (cached) return JSON.parse(cached);

  // If another request is already fetching, wait for it
  if (fetchLocks.has(key)) {
    await fetchLocks.get(key);
    const result = await redis.get(key);
    return result ? JSON.parse(result) : fetcher();
  }

  // I'm the one fetching
  const fetchPromise = fetcher().then(async (data) => {
    await redis.setEx(key, ttl, JSON.stringify(data));
    fetchLocks.delete(key);
    return data;
  });

  fetchLocks.set(key, fetchPromise);
  return fetchPromise;
}

Fix 2: Probabilistic Early Expiration

Randomly refresh the cache before it expires, based on remaining TTL:

async function getWithEarlyExpiry<T>(
  key: string,
  fetcher: () => Promise<T>,
  ttl: number
): Promise<T> {
  const cached = await redis.get(key);
  const ttlRemaining = await redis.ttl(key);

  if (cached) {
    // Probabilistically refresh when less than 20% TTL remains
    const shouldEarlyRefresh = ttlRemaining < ttl * 0.2 && Math.random() < 0.1;

    if (!shouldEarlyRefresh) return JSON.parse(cached);
    // Fall through to refresh (current user sees cached value)
    fetcher().then(data => redis.setEx(key, ttl, JSON.stringify(data)));
    return JSON.parse(cached);
  }

  const data = await fetcher();
  await redis.setEx(key, ttl, JSON.stringify(data));
  return data;
}

Fix 3: Redis Lock

async function getWithRedisLock<T>(
  key: string,
  fetcher: () => Promise<T>,
  ttl: number
): Promise<T> {
  const cached = await redis.get(key);
  if (cached) return JSON.parse(cached);

  const lockKey = `lock:${key}`;
  const acquired = await redis.set(lockKey, '1', { NX: true, EX: 10 });

  if (!acquired) {
    // Someone else is fetching — wait and retry
    await sleep(100);
    const retried = await redis.get(key);
    return retried ? JSON.parse(retried) : getWithRedisLock(key, fetcher, ttl);
  }

  try {
    const data = await fetcher();
    await redis.setEx(key, ttl, JSON.stringify(data));
    return data;
  } finally {
    await redis.del(lockKey);
  }
}

Cache Invalidation

“There are only two hard things in computer science: cache invalidation and naming things.”

Time-based (TTL): simplest. Stale for up to TTL seconds. Fine for most cases.

Event-based: invalidate on write. No staleness, but requires coordinating cache with every write path.

// Pattern: write-through cache
async function updateProduct(productId: string, data: Partial<Product>): Promise<Product> {
  const product = await db.updateProduct(productId, data);
  
  // Update cache immediately (write-through)
  await redis.setEx(`product:${productId}`, 3600, JSON.stringify(product));
  
  // Also invalidate any list caches that include this product
  await redis.del('products:featured');
  await redis.del(`products:category:${product.categoryId}`);
  
  return product;
}

Tag-based invalidation:

// Associate cache keys with tags
async function setCached(key: string, value: any, ttl: number, tags: string[]) {
  await redis.setEx(key, ttl, JSON.stringify(value));
  for (const tag of tags) {
    await redis.sAdd(`tag:${tag}`, key);
    await redis.expire(`tag:${tag}`, ttl + 60);  // tag lives a bit longer
  }
}

// Invalidate all keys with a tag
async function invalidateTag(tag: string) {
  const keys = await redis.sMembers(`tag:${tag}`);
  if (keys.length > 0) {
    await redis.del(...keys, `tag:${tag}`);
  }
}

// Usage
await setCached('products:electronics', products, 3600, ['products', 'electronics']);
await setCached('products:featured', featured, 1800, ['products']);

// When any product changes:
await invalidateTag('products');   // invalidates both cache keys

CDN Caching

For public content (product pages, images, static assets):

// Express — set cache headers
app.get('/products/:id', async (req, res) => {
  const product = await getProduct(req.params.id);

  res.set({
    'Cache-Control': 'public, max-age=300, stale-while-revalidate=60',
    'Vary': 'Accept-Encoding',
    'ETag': `"${product.updatedAt.getTime()}"`,
  });

  // Check ETag
  if (req.headers['if-none-match'] === `"${product.updatedAt.getTime()}"`) {
    return res.sendStatus(304);   // not modified
  }

  res.json(product);
});

stale-while-revalidate=60: serve stale content for 60 seconds while refreshing in background. Zero latency on refresh.

Purge on update:

async function updateProduct(productId: string, data: Partial<Product>) {
  const product = await db.updateProduct(productId, data);
  
  // Purge CDN cache (Cloudflare example)
  await fetch(`https://api.cloudflare.com/client/v4/zones/${ZONE_ID}/purge_cache`, {
    method: 'POST',
    headers: { Authorization: `Bearer ${CF_TOKEN}` },
    body: JSON.stringify({ files: [`https://api.example.com/products/${productId}`] }),
  });
  
  return product;
}

What Not to Cache

✗ Data that changes faster than your TTL (real-time prices, live inventory)
✗ Data that must be consistent (account balances, order totals after payment)
✗ Data personalized per user at high cardinality (1M users × 100 products = 100M cache entries)
✗ Security-sensitive queries (permissions checks, rate limit state)
✓ Reference data (product catalog, config, localization strings)
✓ Expensive aggregations (daily stats, leaderboards)
✓ External API responses (weather, exchange rates)
✓ Session data (already in Redis anyway)

Cache hit rate below 80% often means you’re caching the wrong things, or TTLs are too short. Profile cache misses before adding more cache.