When Your AI Agent Lies: Silent LLM Fallbacks

This article details a debugging case where an AI-powered system appeared to work perfectly—returning fast 200 responses with no errors—but was actually running entirely on deterministic template fallbacks due to three combined bugs. The bugs included: an API version mismatch with Azure OpenAI's new request surface, unsupported sampling parameters for reasoning models, and a config loader prefix-trimming issue that caused all keys to miss. Each bug was individually harmless and logged at an unmonitored verbosity level, making the system look fine while doing none of its intended work. The article highlights how silent fallback logic can mask critical failures in production AI systems and emphasizes the need for monitoring beyond just error spikes.