INC-902
Elevated 5xx on API gateway south
Timeline
- Failover traffic shifted to gw-south-08. Error rate dropped to 2.1%.
- Root cause hypothesis: upstream timeout on auth validator pod 3/3 unhealthy.
- Pager opened for on-call platform. N. Pillay assigned commander.
- Alert storm: HTTP 502 rate > 8% on gw-south-07 for 5m window.
Impacted nodes
Fleet view| Node | Signal | Value | Since |
|---|---|---|---|
| gw-south-07 | http_5xx_rate | 8.4% | 11:04 |
| auth-val-03 | pod_ready | 0/3 | 11:02 |