INC-902

P1

Elevated 5xx on API gateway south

Timeline

  1. Failover traffic shifted to gw-south-08. Error rate dropped to 2.1%.
  2. Root cause hypothesis: upstream timeout on auth validator pod 3/3 unhealthy.
  3. Pager opened for on-call platform. N. Pillay assigned commander.
  4. Alert storm: HTTP 502 rate > 8% on gw-south-07 for 5m window.

Impacted nodes

Fleet view
Node Signal Value Since
gw-south-07 http_5xx_rate 8.4% 11:04
auth-val-03 pod_ready 0/3 11:02