API Timeout Playbook (502/504, Deadlines, Safe Retries)

Quick Triage

• -Classify protocol semantics first: 502 means a gateway/proxy received an invalid upstream response; 504 means it did not receive a timely upstream response.
• -Capture one failing request with correlation ID, elapsed latency, and timeout/deadline values at caller, intermediary, and upstream service.
• -For gRPC paths, record request deadline and verify whether failures map to `DEADLINE_EXCEEDED` status.

Execution Steps

1. 1Reproduce one failure path and preserve raw evidence (status code, elapsed time, request ID, and timeout/deadline config).
2. 2Identify timeout ownership by hop: client, gateway/proxy, service, or downstream dependency.
3. 3Separate invalid-upstream-response failures (502) from upstream-wait-expired failures (504) before changing timeout values.
4. 4For gRPC services, set explicit deadlines on calls and confirm deadline propagation behavior across service-to-service hops.
5. 5Apply bounded retries with exponential backoff and jitter only for retry-safe operations; avoid blind retries for non-idempotent writes.
6. 6Roll out minimal fixes, then validate 502/504/DEADLINE_EXCEEDED rates and tail latency under representative traffic.

Diagnosis Dimensions

• -HTTP semantics boundary: distinguish 502 invalid-upstream-response from 504 upstream-timeout and 408 request-timeout contexts.
• -Deadline continuity: whether explicit deadlines/timeouts are consistently configured across caller, intermediary, and upstream hops.
• -Latency shape: whether p95/p99 tails exceed configured time budgets even when average latency looks healthy.
• -Retry safety: whether retry policies are bounded, jittered, and limited to retry-safe operations.

Verify Recovery

• -Replay the original failing flow and confirm timeout-class errors are cleared or reduced to normal baseline.
• -Observe a sustained traffic window and verify p95/p99 latency remains within configured deadlines/timeouts.
• -Verify retry behavior stays bounded and non-idempotent operations did not produce duplicate side effects.

What To Avoid

• -Increasing all timeout values globally without first identifying the failing hop and failure class.
• -Treating every timeout-like failure as 504 while ignoring the 502 invalid-upstream-response path.
• -Running synchronized fixed-interval retries across many clients during an ongoing latency incident.
• -Leaving gRPC calls without explicit deadlines, which can cause unbounded waits.

Pro Tip

Frequently Asked Questions

Official References

• -RFC 9110: 502 Bad Gateway
• -RFC 9110: 504 Gateway Timeout
• -RFC 9110: 408 Request Timeout
• -gRPC Deadlines Guide
• -gRPC Status Codes
• -AWS SDK Retry Behavior