Sahan Serasinghe • 7/22/2026

When "no healthy upstream" isn't about the upstream you think

This article details the investigation of an intermittent 'no healthy upstream' error in a search backend service. The author describes how initial root-cause analysis blamed CPU throttling, but systematic testing disproved this hypothesis. Through careful observation of latency patterns and system metrics, the author uncovers a different failure mode where healthy instances gradually remove themselves from service. The post emphasizes treating RCAs as hypotheses and validating them with evidence, offering insights into debugging elusive production issues in distributed systems.

0 comments

#Backend #Load Balancer #Root Cause Analysis