cloudflare·May 18, 2026·3 min readCurated summary

Project Glasswing: what Mythos showed us

Project Glasswing found that Anthropic’s Mythos Preview represents a major advance in AI-assisted vulnerability research. Unlike conventional scanners, it can combine multiple low-level bugs into a credible exploit chain and generate working proofs by writing, compiling, and testing code iteratively. However, inconsistent refusals and a high rate of speculative findings mean capable models still require strong safeguards and human-led validation before large-scale deployment. ## Exploit Chain Construction - Mythos Preview can combine several seemingly minor vulnerabilities into a complete attack. - It can reason from primitives such as use-after-free bugs to arbitrary read/write access, control-flow hijacking, and ROP-based system takeover. - Earlier frontier models often identified individual bugs but failed to connect them into a working exploit. - This ability can elevate low-severity findings that might otherwise remain ignored in vulnerability backlogs. ## Automated Proof Generation - The model does more than describe suspected vulnerabilities: - Writes proof-of-concept code. - Compiles it in a scratch environment. - Executes it and checks whether the expected behavior occurs. - Revises its hypothesis when testing fails. - This feedback loop distinguishes plausible speculation from demonstrated exploitability. ## Inconsistent Model Refusals - Mythos Preview lacked the additional safeguards used in generally available models, but still developed emergent refusals around some offensive security tasks. - These refusals were inconsistent: - The same research task could succeed after an unrelated environmental change. - The model might confirm serious memory bugs but refuse to create an exploit. - Rephrasing the request or repeating it could produce a different result. - Organic model guardrails are therefore not reliable enough to act as a complete safety boundary. - Future publicly available cyber-capable models will need additional, deliberate safeguards beyond their learned behavior. ## The Signal-to-Noise Problem - Vulnerability research still requires determining which findings are real, exploitable, and urgent. - AI tools increase the volume of speculative findings, making triage more difficult. - Two major factors affect noise levels: - **Programming language:** C and C++ expose developers to memory bugs such as buffer overflows and out-of-bounds access, while memory-safe languages such as Rust eliminate many of these classes at compile time. Memory-unsafe projects produced more false positives. - **Model bias:** Models tend to report possible vulnerabilities even when evidence is weak, using qualifications such as “possibly” or “could in theory.” - Exploratory over-reporting may help discover novel issues, but it is costly in a production triage queue because each speculative finding consumes analyst time and model resources. ## Scaling AI-Assisted Security Research - Mythos Preview’s capabilities justify treating it as a different class of security tool rather than simply a better conventional scanner. - Scaling these systems will require: - Post-validation stages to filter speculative findings. - Sandboxed environments for compiling and testing proofs. - Human review of exploit chains and severity. - Explicit safety controls that do not depend solely on model refusals. - The main challenge is no longer only whether models can find vulnerabilities, but whether organizations can reliably validate, prioritize, and safely manage their output. Organizations should use advanced security models in controlled environments with layered safeguards and rigorous validation. Their ability to construct exploits is powerful, but their inconsistent safety behavior and noisy findings make unsupervised use inappropriate.

Read original(opens in new tab)

Llm SecurityRustC++Memory Safety+4

line·Dec 17, 2025Original article

Safety is a Given, Cost (opens in new tab)

AI developers often rely on system prompts to enforce safety rules, but this integrated approach frequently leads to "over-refusal" and unpredictable shifts in model performance. To ensure both security and operational efficiency, it is increasingly necessary to decouple safety mechanisms into separate guardrail systems that operate independently of the primary model's logic. ## Negative Impact on Model Utility * Integrating safety instructions directly into system prompts often leads to a high False Positive Rate (FPR), where the model rejects harmless requests alongside harmful ones. * Technical analysis using Principal Component Analysis (PCA) reveals that guardrail prompts shift the model's embedding results in a consistent direction toward refusal, regardless of the input's actual intent. * Studies show that aggressive safety prompting can cause models to refuse benign technical queries—such as "how to kill a Python process"—because the model adopts an overly conservative decision boundary. ## Positional Bias and Context Neglect * Research on the "Lost in the Middle" phenomenon indicates that LLMs are most sensitive to information at the beginning and end of a prompt, while accuracy drops significantly for information placed in the center. * The "Constraint Difficulty Distribution Index" (CDDI) demonstrates that the order of instructions matters; models generally follow instructions better when difficult constraints are placed at the beginning of the prompt. * In complex system prompts where safety rules are buried in the middle, the model may fail to prioritize these guardrails, leading to inconsistent safety enforcement depending on the prompt's structure. ## The Butterfly Effect of Prompt Alterations * Small, seemingly insignificant changes to a system prompt—such as adding a single whitespace, a "Thank you" note, or changing the output format to JSON—can alter more than 10% of a model's predictions. * Modifying safety-related lines within a unified system prompt can cause "catastrophic performance collapse," where the model's internal reasoning path is diverted, affecting unrelated tasks. * Because LLMs treat every part of the prompt as a signal that moves their decision boundaries, managing safety and task logic in a single string makes the system brittle and difficult to iterate upon. To build robust and high-performing AI applications, developers should move away from bloated system prompts and instead implement external guardrails. This modular approach allows for precise security filtering without compromising the model's creative or logical capabilities.

Llm SecurityAILarge Language ModelsPrompt Engineering+4