How we built a real-world evaluation platform for autonomous SRE agents at scale
Bits AI SRE improved in isolated scenarios but lacked a way to detect regressions across the broader range of production incidents. The team found that tool-level tests and live replays could not capture failures caused by multi-step reasoning or changing telemetry. They built a replayable evaluation platform combining realistic investigation labels, scalable orchestration, and longitudinal performance tracking. ## Subtle Regressions from Well-Intentioned Features - Adding a monitor’s service name to the agent’s initial context improved some internal investigations. - Across broader scenarios, it introduced irrelevant signals that confused the agent and degraded unrelated investigations. - Because there was no representative evaluation set, the team could not measure the change’s wider impact before internal misses exposed it. - The incident demonstrated the need to evaluate every change across diverse investigation types. ## Limits of Tool Tests and Live Replay - Testing tools individually failed to capture errors caused by incorrect interactions between valid tool outputs. - Live investigation replay was difficult to scale because: - Results were not consistently aggregated. - Production environments changed. - Telemetry expired, making investigations unreplayable. - Standard evaluation frameworks assumed clean inputs and static datasets, unlike agents operating over production telemetry. - The team needed controlled, offline replay of realistic end-to-end investigations. ## Evaluation Labels and World Snapshots - Each label represents one production-style investigation. - It contains: - **Ground truth:** the issue’s actual root cause. - **World snapshot:** the queries and signals available when the issue occurred. - The agent is never shown the root cause directly; it must reason from the preserved signals. - Labels must cover varied technologies and failure modes, including: - Kubernetes pod failures - Kafka lag - Bad-code deployments - Complex multi-service business failures - A narrow or overly clean dataset would make performance appear better than it really is. ## Orchestrating Evaluations at Scale - The platform runs Bits against labels, scores the outcomes, and tracks quality over time. - It supports comparisons across: - Investigation categories - Model variants - Configuration versions - Evaluation runs - The architecture consists of a shared label set, an orchestration layer, and reporting infrastructure. - This allows teams to determine whether improvements in one domain, such as Kafka, regress another, such as Kubernetes. ## Scaling Label Creation - The team initially created labels manually from Datadog alerts. - Manual labeling provided early coverage but consumed engineering time and remained far from representative. - They embedded label generation into Bits itself: - Customer feedback and investigation data are used to derive root causes. - Relevant queries are preserved as the world snapshot. - Each user interaction becomes a potential evaluation case. - This increased label creation rates by an order of magnitude and allowed coverage to grow with product usage. ## Agent-Assisted Validation - Early labels required extensive human review, especially when feedback was ambiguous or reconstructed signals were uncertain. - As ingestion grew, manual review became a bottleneck. - Bits was then used to assist with validation by aggregating related signals, identifying relationships, and resolving ambiguous feedback before human review. ## Practical Conclusion Reliable agent improvement requires more than testing individual tools or replaying live incidents. A representative, production-derived label set combined with reproducible end-to-end evaluations makes regressions visible and enables safer iteration.
Read original(opens in new tab)
