The symptom and the actual bottleneck
The reported symptom was that the site got slow every morning and occasionally returned errors. The instinct in the room was to add servers. I asked for two things first: request-level latency percentiles by endpoint, and database connection counts over the same window. Those two graphs told the whole story in about twenty minutes.
Latency was fine at the median and catastrophic at the 95th percentile, and the database connection count was pinned at its ceiling every morning. This was not a shortage of application capacity. It was connection exhaustion — every application instance held its own pool, instances had been added over time, and the sum exceeded what the database could accept. Adding servers would have made it strictly worse.
Fix the cheap things before redesigning
Before any architecture work I did the unglamorous pass. The slowest endpoints were reviewed for missing indexes and found three, one of which was doing a sequential scan on a table of several million rows on every dashboard load. Query plans, not guesses. That alone moved the 95th percentile substantially.
I also right-sized the connection pools. Every instance had been configured with a generous pool copied from a tutorial, and the arithmetic across the fleet exceeded the database limit. Reducing per-instance pool size and putting a connection proxy in front stopped the exhaustion entirely. Neither change touched the architecture, and together they bought the room to redesign calmly rather than in a panic.
Reads: caching and replicas
The traffic was heavily read-dominated, and much of it was for data that changed infrequently — configuration, catalogue, reference records. Putting ElastiCache in front of those reads removed a large fraction of database queries immediately. The important decision was invalidation: cache on write rather than relying purely on expiry, so users never saw a stale value after their own edit.
Reporting queries were then moved to read replicas. Those queries were heavy, tolerant of a second or two of lag, and had been competing with transactional traffic for the same resources. Separating them meant a slow analytical query could no longer degrade checkout. Routing was explicit at the data-access layer rather than automatic, so it was obvious in code which queries accepted replica lag.
Writes: decouple everything that can wait
The write path was doing far too much synchronously. Placing an order wrote the order, generated a PDF, sent confirmation email, updated the analytics store and notified a partner API — all inside the request. The user waited for every one of those, and any failure failed the whole order.
Moving the non-essential steps behind SQS changed both latency and reliability. The request now writes the order and enqueues the rest, returning in a fraction of the time. If the partner API is down, the order still succeeds and the notification retries. The critical design detail was the outbox pattern: the message is written in the same transaction as the order, so it is impossible to have an order without its follow-up work or vice versa.
Scaling on the right signal
Auto-scaling existed but was configured on CPU, and this workload was not CPU-bound — it was IO-bound, so instances sat at moderate CPU while requests queued. Scaling on request count per target and on latency reflected actual load and made the group respond to real pressure rather than a proxy that happened to be flat.
Scale-out was set aggressive and scale-in conservative. Scaling up late is a user-visible outage; scaling down late is a small amount of money. Getting that asymmetry right matters more than tuning the exact thresholds, and it is the setting most often left at defaults that assume symmetry.
Proving it with load tests
The target was 50,000 requests per minute sustained. We built a load profile from real traffic logs rather than uniform synthetic requests, because real traffic is bursty and unevenly distributed across endpoints — a flat load test would have passed while production still failed.
The system held the target with headroom, and more importantly the failure mode changed. Under deliberate overload it degraded predictably: latency rose, the queue absorbed the write backlog, and the circuit breakers shed load on non-essential paths. It stopped falling over and started bending, which is the real goal. Predictable degradation under overload is worth more than a higher peak number.
Key takeaways
- Diagnose before scaling — the bottleneck here was database connection exhaustion, which more servers would have worsened
- Do the cheap pass first: query plans, missing indexes, and right-sized connection pools bought room to redesign calmly
- Cache infrequently-changing reads and invalidate on write so users always see their own edits
- Move heavy, lag-tolerant reporting queries to read replicas with explicit routing at the data layer
- Decouple non-essential write-path work behind a queue using the outbox pattern for transactional safety
- Auto-scale on request count and latency, not CPU, for IO-bound workloads; scale out fast, scale in slow
Conclusion
Nothing here was exotic. Indexes, pooling, caching, replicas, queues and correct scaling signals — applied in the order the evidence pointed to. The engineering discipline was refusing to redesign before diagnosing, and measuring the result against a load profile built from real traffic rather than a convenient synthetic one.
Enjoyed this article?
Vivek Kumar Singh
Technical Expert · Full Stack Cloud Engineer · Tokyo, Japan
