line·Mar 25, 2026·3 min readCurated summary

Advancing Guardrail Models through Automated Vulnerability Collection and Generation Using Coding Agents

LLM guardrails must detect prompt injection and jailbreak attempts without blocking legitimate requests that merely contain security-related keywords. The post argues that benchmark scores alone do not reflect production performance, especially false positives caused by missing input diversity. It presents a Codex-based, automated testing pipeline that generates categorized test data, evaluates the guardrail model, and analyzes failures reproducibly. ## The Gap Between Benchmark and Production Performance - The initial guardrail model performed well on external benchmarks but produced unexpected false positives in production-like tests. - Legitimate requests containing terms such as “ignore,” “bypass,” “override,” “system prompt,” or “jailbreak” were sometimes classified as attacks. - Examples included: - Development questions about temporarily bypassing authentication in a local test environment. - Educational requests about jailbreak techniques and defensive guidelines. - The core issue was insufficient representation of real-world input diversity, not simply poor model quality. - This motivated an automated environment for repeatedly discovering and analyzing guardrail weaknesses. ## Using Codex as a Test Automation Tool - The team adapted coding agents from software development tasks to complex, repeatable security testing. - Codex was used through its CLI capabilities to: - Read and create project files. - Edit code. - Execute evaluation scripts. - The pipeline relies on three Codex concepts: - **AGENTS.md:** Defines global rules, project conventions, commands, and security constraints. - **Sub-agents:** Allow a main orchestrator to delegate independent category tests to parallel worker agents. - **Skills:** Package repeatable procedures, input/output specifications, prompts, and scripts into reusable modules. ## Category-Based Experiments - Instead of sending thousands of random samples, experiments are divided into vulnerability and false-positive categories. - Example categories include: - Normal development or IT requests containing security-related keywords. - Educational or preventive requests involving sensitive topics such as jailbreaks or drug abuse prevention. - Categorization improves: - Root-cause analysis. - Parallel execution through independent workers. - Context clarity. - Regression testing after model changes. ## Separate Generation and Evaluation Skills ### `synthetic-generator` - Creates test queries according to each category’s specification. - Enforces constraints such as: - Attack type. - Sentence length. - Safe or dangerous target labels. - Produces varied, realistic phrasing and stores the dataset as JSONL. ### `injection-classifier` - Sends generated inputs to the guardrail model API through Python scripts. - Compares predictions with ground-truth labels. - Calculates false-positive and false-negative statistics. - Stores the original text, labels, predictions, and metrics in a consolidated JSONL file. Separating these procedures into skills provides intermediate artifacts for debugging, fixed input/output contracts for reproducibility, and independent maintenance of generation and evaluation logic. ## Pipeline Architecture - A **main agent**: - Reads `AGENTS.md` and `TEST_CATEGORY.md`. - Determines categories, sample counts, and constraints. - Creates and assigns work to category-specific workers. - Collects completion reports and verifies the run. - Each **category worker**: - Generates `input.jsonl` using `synthetic-generator`. - Evaluates the guardrail model using `injection-classifier`. - Produces `result.jsonl` with predictions and metrics. - Analyzes false positives and false negatives. - Writes a Markdown analysis report. - Stores outputs under `outputs/<run_id>/`, organized by category. ## Results and Practical Recommendation The pipeline enables systematic, repeatable testing rather than isolated discovery of misclassifications. For production guardrails, teams should combine benchmark evaluation with categorized real-world simulations, modular generation and evaluation steps, parallel test agents, and preserved JSONL artifacts for debugging and regression analysis.

Read original(opens in new tab)

JailbreakingPythonAPIPrompt Injection+4

line·Dec 17, 2025Original article

Safety is a Given, Cost (opens in new tab)

AI developers often rely on system prompts to enforce safety rules, but this integrated approach frequently leads to "over-refusal" and unpredictable shifts in model performance. To ensure both security and operational efficiency, it is increasingly necessary to decouple safety mechanisms into separate guardrail systems that operate independently of the primary model's logic. ## Negative Impact on Model Utility * Integrating safety instructions directly into system prompts often leads to a high False Positive Rate (FPR), where the model rejects harmless requests alongside harmful ones. * Technical analysis using Principal Component Analysis (PCA) reveals that guardrail prompts shift the model's embedding results in a consistent direction toward refusal, regardless of the input's actual intent. * Studies show that aggressive safety prompting can cause models to refuse benign technical queries—such as "how to kill a Python process"—because the model adopts an overly conservative decision boundary. ## Positional Bias and Context Neglect * Research on the "Lost in the Middle" phenomenon indicates that LLMs are most sensitive to information at the beginning and end of a prompt, while accuracy drops significantly for information placed in the center. * The "Constraint Difficulty Distribution Index" (CDDI) demonstrates that the order of instructions matters; models generally follow instructions better when difficult constraints are placed at the beginning of the prompt. * In complex system prompts where safety rules are buried in the middle, the model may fail to prioritize these guardrails, leading to inconsistent safety enforcement depending on the prompt's structure. ## The Butterfly Effect of Prompt Alterations * Small, seemingly insignificant changes to a system prompt—such as adding a single whitespace, a "Thank you" note, or changing the output format to JSON—can alter more than 10% of a model's predictions. * Modifying safety-related lines within a unified system prompt can cause "catastrophic performance collapse," where the model's internal reasoning path is diverted, affecting unrelated tasks. * Because LLMs treat every part of the prompt as a signal that moves their decision boundaries, managing safety and task logic in a single string makes the system brittle and difficult to iterate upon. To build robust and high-performing AI applications, developers should move away from bloated system prompts and instead implement external guardrails. This modular approach allows for precise security filtering without compromising the model's creative or logical capabilities.

JailbreakingAILarge Language ModelsPrompt Engineering+4