The First Thirty Minutes
Rule one of production incidents: don't fix, understand first. I spent twenty minutes reading logs before touching anything. The error was an out-of-memory exception in a Cloud Function that had been running fine for three months. Something had changed — either the data, the code, or the environment.
Git blame on the last deploy. The deploy three hours earlier had changed a batch processing function. The change looked benign — a new sort step before the processing loop. But the sort was loading the entire dataset into memory at once, and overnight a data import had doubled the dataset size. The sort that worked on 50,000 records failed on 100,000.
The Fix and What Changed After
Immediate fix: rolled back the deploy, confirmed recovery. Root cause fix: rewrote the sort to use a streaming approach with a cursor instead of loading the full dataset. Deployed with a canary — 10% of traffic, monitored for 20 minutes, then full rollout.
The changes after: all data processing functions now have explicit memory limit configuration and an alert if they approach 70% of the limit. The data import pipeline logs its row count before and after. Any import that exceeds expected size triggers a review before committing.
Key takeaways
- Read logs before touching anything in a production incident — understand the cause before applying a fix, or you may fix the wrong thing
- Always check the most recent deploy first in a regression — correlate the incident timeline with deployment history before investigating further
- After every incident: implement the alerting that would have caught it earlier — the goal is to catch the next incident of the same type before users do
Conclusion
Production incidents at 2am are part of the job. The engineers who handle them well aren't calmer by nature — they have a process. Read first, understand, fix minimally, verify, then fix properly. Having that process before you need it changes everything.
Enjoyed this article?
Vivek Kumar Singh
Technical Expert · Full Stack Cloud Engineer · Tokyo, Japan
