Health checks that mean different things

Liveness and readiness are not the same probe and conflating them causes outages. Liveness answers whether the process is broken enough to need restarting. Readiness answers whether it should receive traffic right now. A service waiting on a slow dependency at startup is not ready, but it is very much alive, and a liveness probe that fails on that will restart it forever.

Readiness should check the dependencies the service genuinely cannot function without, and nothing else. I have seen readiness probes that checked six downstream services, which meant any one of them going down removed a perfectly capable instance from the load balancer — turning a partial dependency failure into a full outage of the service in front of it.

Graceful shutdown, or you drop requests every deploy

By default a container receiving a termination signal can kill in-flight requests. Enabling graceful shutdown with a timeout, and setting the pod's termination grace period comfortably longer than that, lets in-flight work finish. Without it, every single deployment produces a small burst of errors that teams learn to ignore and users experience as random flakiness.

The ordering detail people miss is that the instance must be removed from the load balancer before it stops accepting connections. A brief pre-stop delay before shutdown begins gives the routing layer time to notice. It looks like superstition and it is the difference between clean deploys and mysterious 502s.

Timeouts, retries and circuit breakers on every call

Every outbound call needs an explicit connect and read timeout. Client defaults are typically far too generous, and one slow dependency with no timeout will consume the thread pool and take down endpoints unrelated to it. This is the single most common cause of cascading failure I encounter.

Retries need a budget and jitter, and must only apply to idempotent operations — a retried POST that creates a payment is a serious bug, not a resilience feature. Add a circuit breaker so a persistently failing dependency is skipped rather than repeatedly hammered, and define the fallback deliberately: degraded data, cached data, or a clean error, decided in advance rather than improvised.

Configuration and secrets

Configuration comes from the environment, differs per environment, and never ships in the artefact. The same container image should run in staging and production with only its configuration changing — if you are building environment-specific images, you are testing something other than what you deploy.

Secrets come from a secret manager, and the application should tolerate rotation without a redeploy. Validate configuration at startup and fail fast on anything missing or malformed. A service that starts happily with an absent database password and then fails on the first user request has converted a deployment error into a production incident.

Observability: logs, metrics, traces

Structured JSON logs with a correlation ID propagated across service boundaries. Without that ID, tracing one user's journey through several services is guesswork, and adding it later means touching every service. Include the ID in error responses so support can connect a user's complaint to the exact request.

Expose metrics for request rate, error rate and latency percentiles at minimum, plus the health of every dependency. Distributed tracing is the piece most often deferred and most regretted — it is what turns 'the checkout is slow sometimes' into 'this specific call to this specific service has a p99 of four seconds'.

Resource limits, database hygiene and rollback

Set JVM heap explicitly relative to the container memory limit. A JVM that assumes it owns the host will size its heap for the node and be killed by the container runtime, producing restarts that look mysterious until you find the exit code. Set CPU and memory requests and limits so the scheduler can place the workload sensibly.

Size connection pools with the total fleet in mind, not per instance — the sum across all instances must sit inside the database's limit. And make schema migrations backward compatible so the previous version still runs against the new schema, because a deployment you cannot roll back is not a deployment, it is a commitment.

Key takeaways

  • Separate liveness from readiness, and keep non-essential dependencies out of readiness checks
  • Enable graceful shutdown with a pre-stop delay, or every deploy drops in-flight requests
  • Set explicit timeouts on every outbound call; retry only idempotent operations, with jitter and a budget
  • Ship one image per artefact, configure from the environment, and validate config at startup
  • Propagate a correlation ID everywhere and return it in error responses; add tracing before you need it
  • Size the JVM heap against the container limit and connection pools against the whole fleet, not one instance

Conclusion

None of this is advanced. It is the difference between a service that runs and a service you can operate at 3am. Run the list before the first production deploy, because every item on it is far cheaper to add before there are users than after.

Enjoyed this article?

Vivek Kumar Singh

Vivek Kumar Singh

Technical Expert · Full Stack Cloud Engineer · Tokyo, Japan