What Happened
The feature used a third-party API with region-specific endpoints. The endpoint URL was stored in an environment variable. We'd set it correctly in staging. We'd set it correctly in production for the primary region. We'd forgotten to set it for the secondary region deployment.
Because the variable had a fallback value (the primary region endpoint), it didn't fail — it just sent all secondary-region traffic to the wrong endpoint, which accepted the requests and returned responses that were subtly wrong for that region. No 500 errors. No alerts. Just silent misbehaviour.
The Fixes That Prevent This
Validate environment variables at startup, not at use. We added a startup check that throws if any required environment variable is missing or invalid. The application refuses to start with an incomplete configuration. A failed deployment with a clear error message is far better than a running application behaving incorrectly.
Use a validation library like zod for environment variables. Define the schema explicitly — which variables are required, what format they should be in, whether they have valid defaults. The schema is documentation and runtime validation at the same time.
Key takeaways
- Validate required environment variables at application startup and crash hard if any are missing — a failed deployment with a clear error is better than silent misbehaviour
- Use zod or a similar schema library to define your env var schema — you get type inference, format validation, and documentation in one place
- Never rely on fallback values for configuration that varies by environment — if staging and production need different values, both need explicit values
Conclusion
Environment variable incidents are almost always preventable. The cost of rigorous startup validation is low. The cost of a six-hour silent degradation in production is much higher. Validate early, validate explicitly, crash on invalid configuration.
Enjoyed this article?
Vivek Kumar Singh
Technical Expert · Full Stack Cloud Engineer · Tokyo, Japan
