The Sequence of Events
The bug was a null check that worked correctly in staging because our staging data always had the relevant field populated. In production, a subset of older accounts had the field missing — an edge case that existed in our database but not in our test fixtures. The test passed. The deployment proceeded. The payment handler threw an unhandled exception that was caught by a generic error handler that returned a 200 with an error payload.
200 with an error payload: the payment UI showed success. The transaction didn't process. The user thought they'd paid. The charge didn't go through. We found out 47 minutes later from a support ticket.
What Changed After
Three immediate changes: every error response returns an HTTP error code — never 200 with an error body. Payment flows have integration tests against a realistic copy of production data (with PII stripped), not just clean test fixtures. Monitoring alerts trigger on payment error rate, not just server error rate.
The deeper change: we now do pre-production smoke tests against a staging environment seeded with a representative sample of production data shapes. The edge case that caused the bug — accounts without the optional field — would have been caught in staging if we'd been testing with realistic data.
Key takeaways
- Never return HTTP 200 with an error payload — use appropriate HTTP status codes so monitoring, client code, and logs can distinguish success from failure without parsing the body
- Seed your staging environment with representative production data shapes (PII stripped) — clean test data misses the edge cases that exist in real user accounts
- Monitor business-critical metrics (payment success rate, checkout completion rate) separately from infrastructure metrics — errors in these flows may not surface as server errors
Conclusion
The most preventable bugs are the ones that pass all your tests because your tests don't test the right things. The fix isn't just better code — it's better tests with better data, monitoring that catches business failures, and a culture where error responses tell the truth.
Enjoyed this article?
Vivek Kumar Singh
Technical Expert · Full Stack Cloud Engineer · Tokyo, Japan
