Why do systems fail on the days everyone knew were coming?
Systems fail on predictable peak days because they were never tested at that scale, not because they were broken. One component — usually the database or a third-party API — hits its limit first, and the rest of the stack queues up behind it. The fix is a rehearsal, not a rewrite: measure the target, load-test until something breaks, remove the first bottleneck, repeat. This is the list we work through with clients four to six weeks before the day. The stakes are commercial rather than technical: a Deloitte study of retail sites found that a 0.1-second improvement in mobile load time lifted conversion by about 8% (Deloitte, 2020) — and a stalled checkout on peak day is the same effect in reverse.
How do you measure before the day?
Set the target in numbers, run a load test that behaves like real customers, and have dashboards live before the peak, not after. Without a number you cannot know whether you passed; without a realistic test you find the bottleneck at 10:03 on the day; without dashboards you cannot tell which component is failing. Google's Core Web Vitals give a ready-made user-facing target — LCP under 2.5 seconds and INP under 200 milliseconds at the 75th percentile (web.dev, 2024) — and the Chrome UX Report lets you compare your field numbers with competitors before you start.
- 1. A target in numbers. Expected requests per second, concurrent users and orders per minute at peak — from last year's data times growth, not from a feeling.
- 2. A load test that resembles reality. k6 or Locust scripts that browse, search, add to cart and pay, run against a production-like environment until something breaks. The point is to find the first bottleneck, not to pass.
- 3. Dashboards and alerts before the day. Latency percentiles, error rate, queue depth, DB connections, third-party response times. If you cannot see it, you cannot fix it at 10:03.
How do you remove the obvious bottlenecks?
Take reads off the database with a cache, take slow work off the request with a queue, and give every third-party call a timeout and a fallback. Those three moves resolve most of the incidents we see, because the database and external APIs are where peak traffic lands hardest. Caching is the cheapest win: catalogue pages, search facets and prices do not change per user, so a CDN and Redis can serve them without touching the database. Google's SRE guidance on cascading failures makes the same point from the other side — the goal is to shed or defer load before the slowest dependency saturates (Google SRE, 2016).
- 4. Cache what does not change per user. Catalogue pages, search facets, prices: CDN and Redis take the bulk of reads off the database.
- 5. Queues for anything slow. Emails, invoices, ERP sync, analytics events — accept the order, queue the rest.
- 6. Database: indexes, pooling, replicas. Slow query log reviewed, connection pooling in place, read replicas for reporting so a dashboard cannot stall checkout.
- 7. Third parties with timeouts and fallbacks. Payment, shipping and tax APIs get strict timeouts, retries with backoff and a degraded path (e.g. "shipping cost calculated after order").
Cache or queue: which one fixes your bottleneck?
Use a cache when the problem is too many reads of the same data; use a queue when the problem is slow work inside the request. They solve different halves of a peak: the cache protects the database from the browsing crowd, the queue protects checkout from everything that happens after the order. Most high-load systems need both, and the table shows how to tell which one a given symptom calls for.
| Criterion | Cache (CDN, Redis) | Queue (Redis, RabbitMQ, SQS) |
|---|---|---|
| Symptom it fixes | DB CPU and read latency climb with traffic | Request time grows with emails, syncs, PDFs |
| What it changes | Serves repeated reads without the origin | Moves work out of the request into workers |
| Typical targets | Catalogue, search facets, prices, sessions | Emails, invoices, ERP/CRM sync, analytics |
| Main risk | Stale data; cache stampede on expiry | Backlog grows unseen; duplicate processing |
| Guardrail | TTLs, versioned keys, stampede locks | Queue-depth alerts, idempotent jobs, retries |
| Effort | Hours to days | Days; needs workers and monitoring |
How do you prepare to fail gracefully?
Decide in advance what you will switch off, prove that scaling works before you depend on it, and write down who does what when the numbers turn red. Graceful failure means the shop keeps selling with recommendations off, not that nothing ever breaks. Autoscaling is the item most often assumed and least often rehearsed: cloud quotas, warm-up time and database connection limits all cap how far a scale-out actually goes. The AWS Well-Architected reliability pillar puts the same idea as a rule: test recovery procedures, and design to scale horizontally before the peak rather than during it (AWS, 2024).
- 8. Feature flags for the expensive stuff. Recommendations, live stock badges, personalisation — switchable off in one click.
- 9. Autoscaling tested, not assumed. Scale-out rehearsed under load; limits and quotas checked with the cloud provider.
- 10. A runbook and a person on call. Who watches, what they do when queue depth climbs, how to roll back. Written down, rehearsed once.
Ten items, none exotic. Systems that pass this list survive the day; systems that skip it make the news. If your peak is coming, ask for a load-readiness audit.