Is your eval lying to you?

When building evals for AI coding agents, string contains is incredibly tempting. It’s fast and deterministic. And it costs practically nothing, so you can run it on every change in CI without worrying about inference cost or latency. But is using it making your eval lie to you? Want to know whether the agent used Azure? Look for Azure, right? Want to know whether it used a particular API? Look fo