Build your own vulnerability harness (opens in new tab)
A scalable AI vulnerability program should be built around a model-agnostic harness rather than a single model, prompt, or agent session. The harness must preserve state, support resumable investigations, cross-check findings with different models, and trace issues across repositories. The authors recommend starting small with database-backed Recon, Hunt, and Validate stages, expanding only when operational bottlenecks justify it. ## Why a Harness Is Needed - Generic coding agents are poorly suited to large-scale security analysis because they: - Hold only one hypothesis at a time. - Exhaust their context windows while exploring real repositories. - Lose important information during context compaction. - Subagents help, but they do not provide the persistence, deduplication, resumability, and cross-run coordination required for security investigations. - The system should treat models as interchangeable components: - One model can discover vulnerabilities. - Another can independently validate them. - Different models expose different classes of bugs and reduce shared blind spots. - The harness, rather than any particular frontier model, is intended to be the durable investment. ## The Original Security-Audit Skill The authors began with an approximately 450-line skill designed to audit one repository in a single session. Its seven-phase workflow included: - Three parallel reconnaissance agents producing `architecture.md`. - Hunter agents attacking the code by vulnerability class. - Adversarial validators attempting to disprove findings. - A human-readable vulnerability report for surviving findings. - A schema-checked `findings.json` file. - Mechanical validation of referenced functions and line numbers. - A fresh agent independently re-verifying every finding before submission to an ingest API. This skill became the blueprint for the later pipeline: - Recon agents became the Recon stage. - Attack-class hunters became Hunt. - Adversarial reviewers became Validate. - Reports became structured findings. - Independent re-verification remained a separate validation step. ## Limitations of Single-Session Audits A single run found only about half of the bugs discovered across multiple runs, and it tended to find simpler vulnerabilities rather than subtle ones. Repeating the skill many times and manually diffing results quickly became impractical. The authors identified three major bottlenecks: - **Context exhaustion:** Long sessions cause the model to forget previously investigated bugs. The solution is to externalize state and use the model as a stateless computation engine. - **Poor persistence:** Crashes, rate limits, and connection failures can erase hours of progress if work is not stored incrementally. - **Lack of cross-repository reasoning:** Auditing one repository in isolation misses vulnerabilities at the interfaces between applications and shared components. ## Recommended Minimal Architecture The authors advise building only the infrastructure needed to address current problems: - Store Recon, Hunt, and Validate stages in a database. - Use a separate validator that cannot submit its own findings, reducing confirmation bias. - Defer cross-repository tracing until multiple important repositories need to be analyzed together. - Defer a dedicated deduplication agent until the system produces too much duplicate or low-quality output. - Begin with a well-tuned development skill, then add pipeline stages only when a specific limitation is slowing the work. ## Enterprise-Scale Direction A mature vulnerability harness should continuously scan a fleet of repositories, trace dependencies across them, and reduce thousands of raw candidates to a smaller queue of verified, actionable fixes. Frequent model interchange and independent validation are central to maintaining coverage as models change or become unavailable. The practical recommendation is to invest first in durable orchestration and state management, not allegiance to a particular model. A simple, resumable Recon–Hunt–Validate pipeline is the appropriate starting point, with cross-repository analysis and advanced deduplication added only as scale demands.
