Advancing Guardrail Models through Automated Vulnerability Collection and Generation Using Coding Agents

LLM guardrails must detect prompt injection and jailbreak attempts without blocking legitimate requests that merely contain security-related keywords. The post argues that benchmark scores alone do not reflect production performance, especially false positives caused by missing input diversity. It presents a Codex-based, automated testing pipeline that generates categorized test data, evaluates the guardrail model, and analyzes failures reproducibly.

The Gap Between Benchmark and Production Performance

• The initial guardrail model performed well on external benchmarks but produced unexpected false positives in production-like tests.
• Legitimate requests containing terms such as “ignore,” “bypass,” “override,” “system prompt,” or “jailbreak” were sometimes classified as attacks.
• Examples included:
  • Development questions about temporarily bypassing authentication in a local test environment.
  • Educational requests about jailbreak techniques and defensive guidelines.
• The core issue was insufficient representation of real-world input diversity, not simply poor model quality.
• This motivated an automated environment for repeatedly discovering and analyzing guardrail weaknesses.

Using Codex as a Test Automation Tool

• The team adapted coding agents from software development tasks to complex, repeatable security testing.
• Codex was used through its CLI capabilities to:
  • Read and create project files.
  • Edit code.
  • Execute evaluation scripts.
• The pipeline relies on three Codex concepts:
  • AGENTS.md: Defines global rules, project conventions, commands, and security constraints.
  • Sub-agents: Allow a main orchestrator to delegate independent category tests to parallel worker agents.
  • Skills: Package repeatable procedures, input/output specifications, prompts, and scripts into reusable modules.

Category-Based Experiments

• Instead of sending thousands of random samples, experiments are divided into vulnerability and false-positive categories.
• Example categories include:
  • Normal development or IT requests containing security-related keywords.
  • Educational or preventive requests involving sensitive topics such as jailbreaks or drug abuse prevention.
• Categorization improves:
  • Root-cause analysis.
  • Parallel execution through independent workers.
  • Context clarity.
  • Regression testing after model changes.

Separate Generation and Evaluation Skills

synthetic-generator

• Creates test queries according to each category’s specification.
• Enforces constraints such as:
  • Attack type.
  • Sentence length.
  • Safe or dangerous target labels.
• Produces varied, realistic phrasing and stores the dataset as JSONL.

injection-classifier

• Sends generated inputs to the guardrail model API through Python scripts.
• Compares predictions with ground-truth labels.
• Calculates false-positive and false-negative statistics.
• Stores the original text, labels, predictions, and metrics in a consolidated JSONL file.

Separating these procedures into skills provides intermediate artifacts for debugging, fixed input/output contracts for reproducibility, and independent maintenance of generation and evaluation logic.

Pipeline Architecture

• A main agent:

  • Reads AGENTS.md and TEST_CATEGORY.md.
  • Determines categories, sample counts, and constraints.
  • Creates and assigns work to category-specific workers.
  • Collects completion reports and verifies the run.

• Each category worker:

  • Generates input.jsonl using synthetic-generator.
  • Evaluates the guardrail model using injection-classifier.
  • Produces result.jsonl with predictions and metrics.
  • Analyzes false positives and false negatives.
  • Writes a Markdown analysis report.
  • Stores outputs under outputs/<run_id>/, organized by category.

Results and Practical Recommendation

The pipeline enables systematic, repeatable testing rather than isolated discovery of misclassifications. For production guardrails, teams should combine benchmark evaluation with categorized real-world simulations, modular generation and evaluation steps, parallel test agents, and preserved JSONL artifacts for debugging and regression analysis.