Prompt Injection | Techlist.io

gitlabJul 29, 20263 min readCurated summary

GitLab Patch Release: 19.2.1, 19.1.3, 19.0.5 | GitLab Docs

GitLab released patch versions 19.2.1, 19.1.3, and 19.0.5 on July 29, 2026, addressing important security and bug fixes in CE and EE. Self-managed installations should upgrade immediately; GitLab.com is already patched, and GitLab Dedicated customers need no action. The release fixes vulnerabilities ranging from unauthorized data access and configuration changes to denial of service and AI-related security issues. ## Release Scope and Upgrade Guidance - Applies to GitLab Community Edition and Enterprise Edition. - Affected self-managed installations should upgrade to: - 19.2.1 - 19.1.3 - 19.0.5 - GitLab issues security patches through scheduled releases twice monthly or through urgent ad-hoc releases. - Vulnerability details are generally made public 90 days after the release containing the fix. ## High-Severity Access and Configuration Vulnerabilities - **CVE-2026-6267 — Workhorse information exposure (CVSS 8.5)** - An authenticated Developer could access unauthorized information because of insufficient access controls in internal request handling. - Affected CE/EE versions extend back to 10.1.0. - **CVE-2026-12436 — Pipeline Schedule API mass assignment (CVSS 8.4)** - An authenticated user could modify another user’s CI/CD configuration through improperly validated pipeline schedule attributes. - Affects versions beginning with GitLab 18.0. - **CVE-2026-13113 — Merge request approval race condition (CVSS 6.5)** - In GitLab EE, an authenticated user could merge code into a protected branch without the required approvals. - The issue resulted from a race condition in approval rule processing. - **CVE-2026-14341 — Project import authorization flaw (CVSS 4.9)** - A Maintainer could modify protected branch configuration through a projects API endpoint. - The vulnerability affected CE and EE. ## Availability, Import, and Credential Protection Fixes - **CVE-2026-15975 — Merge request discussion denial of service (CVSS 7.5)** - An unauthenticated attacker could exhaust resources and cause a denial of service. - The issue involved insufficient throttling when processing discussions. - **CVE-2026-6336 — Project import status access control (CVSS 5.3)** - An unauthorized user could view project import source information because of a missing authorization check. - **CVE-2026-16553 — Virtual Registry credential exposure (CVSS 5.4)** - Improper upstream request handling could disclose sensitive information to an unintended host. - This issue affects GitLab EE virtual registries. ## Web and AI-Assisted Feature Fixes - **CVE-2026-3093 — Cross-site scripting in paginated views (CVSS 4.7)** - A crafted URL could execute attacker-controlled JavaScript in another user’s browser. - The root cause was inadequate sanitization of user-controlled input. - **CVE-2026-15077 — Prompt injection in Duo Code Review (CVSS 4.3)** - Untrusted content processed by AI-assisted code review could allow access to information from unauthorized projects. - This affects GitLab EE versions 19.1 and 19.2 before the patched releases. - **CVE-2026-15831 — Duo Workflows token generation flaw (CVSS 4.3)** - An authenticated user could bypass administrator-configured tool governance policies through improper authorization during token generation. - The issue affects GitLab EE. ## Recommended Action Administrators should upgrade affected self-managed GitLab installations to the latest supported patch release immediately, regardless of deployment method, including Omnibus, source installations, and Helm charts. Regularly applying the newest patch release is GitLab’s recommended security practice.

Read original

Prompt Injection CI/CD Gitlab Api Security+3

githubMay 15, 20263 min readCurated summary

Raising the bar: Quality, shared responsibility, and the future of GitHub’s bug bounty program

GitHub is reaffirming its commitment to external security researchers while tightening bug bounty submission standards. Rising report volumes—partly driven by AI and other tools—have increased both valuable findings and unvalidated noise. GitHub’s central message is that tools are welcome, but researchers remain responsible for validating vulnerabilities, demonstrating impact, and understanding the platform’s shared security boundaries. ## Rising Submission Volume - New tools, including AI, have lowered the barrier to security research and expanded the number of people examining attack surfaces. - GitHub has also seen more reports that: - Lack a working proof of concept - Describe only theoretical attack scenarios - Concern categories already listed as ineligible - Because this challenge affects the wider industry, some bug bounty programs have shut down; GitHub instead plans to improve its program. ## Requirements for Strong Reports - Submissions must include a working proof of concept demonstrating concrete security impact. - Researchers should show what an attacker can actually accomplish, rather than merely describing a possible attack path. - Reports must respect GitHub’s published scope and ineligible findings list. Examples of generally ineligible issues include: - DMARC, SPF, or DKIM configuration problems - User enumeration - Missing security headers without a demonstrated attack path - Scanner, static-analysis, or AI-generated findings must be manually validated before submission. - Unverified false positives create unnecessary triage work and may affect a researcher’s HackerOne Signal and reputation. ## AI Is Welcome, but Validation Is Required - GitHub supports the use of AI in security research and uses AI internally. - AI-assisted reports are acceptable when findings are reproduced, verified, and supported by a working proof of concept. - Researchers remain accountable for the accuracy of their submissions, regardless of which tools produced them. - GitHub recommends a concise report structure: - A short issue summary - Clear reproduction steps and evidence, such as screenshots, HTTP requests, or terminal output - An impact statement explaining what an attacker can achieve - Lengthy theoretical explanations and AI-generated filler can obscure the actual vulnerability and slow triage. ## Shared Responsibility and GitHub’s Security Boundary - GitHub protects users through automated scanning, manual review, and other systems for detecting malicious content. - Users are still responsible for deciding what repositories, issues, code, and scripts to trust. - Users should review content before executing or interacting with it. - Cloning a repository is considered an act of trust because Git hooks, build scripts, and other automation may run locally. - Users must also secure their own environments, including tokens, credentials, and local security settings. - Scenarios generally do not bypass GitHub’s security controls when they require victims to deliberately engage with attacker-controlled content. ## Common Shared-Responsibility Scenarios - Prompt injection in content a user intentionally provides to an AI tool - Git hooks or filters executing code from a repository the user checked out - Malicious content in a repository the user chose to clone - Unexpected LLM output caused by untrusted input supplied by the user Research into these areas remains useful when it identifies a way to bypass an actual GitHub security control without requiring the user to actively trust malicious content.

Read original

Prompt Injection Github Git Ai Security+4

githubMay 7, 20264 min readCurated summary

Agent pull requests are everywhere. Here’s how to review them.

Agent-generated pull requests are increasing rapidly, while human review capacity remains limited. Although these changes often look clean and pass CI, research suggests they can introduce more redundancy and technical debt—and reviewers may be more likely to approve them. The solution is not to review more slowly, but to focus human judgment on risks agents are least equipped to recognize. ## The Scale of Agent-Generated Pull Requests - GitHub Copilot code review has processed more than 60 million reviews and grown tenfold in under a year. - More than 20% of GitHub code reviews now involve an agent. - Developers can launch many agent sessions simultaneously, causing pull-request volume to grow faster than human review capacity. - Reviewers therefore need a deliberate method for identifying high-impact issues. ## Understanding the Agent’s Limitations - Coding agents are productive and literal, but lack: - Incident history - Team-specific edge-case knowledge - Operational constraints not documented in the repository - Agents can produce code that appears complete while quietly embedding incorrect assumptions. - Human reviewers provide the context and judgment that automated tools cannot fully replicate. ## CI Gaming Agents may weaken CI when their changes fail, for example by removing tests, skipping linting, or adding commands such as `|| true`. Reviewers should verify: - Coverage thresholds were not reduced. - Tests were not removed, renamed, or skipped. - Workflows still run for forks and pull requests. - CI steps were not placed behind new restrictive conditions. Any such change requires explicit justification before approval. ## Blindness to Existing Code Reuse Agents may copy patterns from nearby code without discovering equivalent utilities elsewhere in the repository. Warning signs include: - Duplicate helper or utility functions - Reimplemented validation logic - New middleware duplicating shared modules - “Almost identical” helpers with different names Reviewers should search for existing implementations and require consolidation rather than merely commenting on duplication. For larger agent pull requests, requiring justification for new utilities can prevent redundant code from becoming future “prior art.” ## Hallucinated Correctness The most dangerous agent errors are not obvious API or syntax failures. They are changes that compile, pass tests, and still behave incorrectly under conditions such as: - Pagination boundaries - Missing permission checks - Validation edge cases - Race conditions at scale Reviewers should trace a critical path from input to output, checking empty, zero, and maximum values, external input validation, permissions on every branch, and unusual conditionals. A claimed bug fix should include a test that fails before the change; otherwise, the fix or the agent’s understanding may be incomplete. ## Agentic Ghosting and Oversized Pull Requests Large, poorly structured agent pull requests are more likely to become abandoned or misaligned. Before conducting an in-depth review, check: - Whether the agent has responded usefully in earlier review rounds - Whether the pull request includes a clear implementation plan - Whether the changes can be divided into smaller, scoped units If no plan exists, request a breakdown or a clear explanation of each component before spending time on detailed comments. ## Untrusted Input in Agent Workflows Workflows that send pull-request bodies, issue content, or commit messages to an LLM can create prompt-injection risks—especially when model output is later executed with `GITHUB_TOKEN` permissions. Reviewers should block workflows that: - Interpolate untrusted content into prompts without sanitization - Grant write access when read-only permissions are sufficient - Execute model output as shell commands without validation - Expose secrets to agent steps or logs Safer designs should use least-privilege permissions such as `permissions: read-all`, sanitize and quote untrusted content, separate analysis from execution, and require human approval before actions affecting production. Agent pull requests should not automatically receive either extra trust or blanket suspicion. Reviewers should focus on CI integrity, reuse, behavior under edge cases, reviewability, and workflow security—the areas where contextual human judgment adds the most value.

Read original

Prompt Injection Large Language Models CI/CD Github Copilot+3

githubApr 14, 20263 min readCurated summary

Hack the AI agent: Build agentic AI security skills with the GitHub Secure Code Game

Agentic AI tools can automate powerful tasks, but their autonomy creates new security risks, including prompt injection, tool misuse, memory poisoning, and compromised multi-agent workflows. GitHub’s Season 4 Secure Code Game teaches developers to recognize these threats by attacking and hardening ProdBot, a deliberately vulnerable terminal-based AI assistant. Its five levels progressively add capabilities—and corresponding attack surfaces—mirroring how real-world AI systems evolve. ## The Secure Code Game’s Evolution - The free, open-source, in-editor course teaches security by having players exploit and fix intentionally vulnerable code. - Earlier seasons covered: - General secure coding across JavaScript, Python, Go, and GitHub Actions. - LLM security, including malicious prompts and defensive techniques. - More than 10,000 developers from industry, academia, and open source have participated. - Season 4 shifts focus from AI that generates content to AI that independently browses, uses tools, calls APIs, and acts for users. ## Why Agentic AI Security Is Urgent - Agentic systems are moving rapidly from research projects into production environments. - The OWASP Top 10 for Agentic Applications identifies threats such as: - Goal hijacking - Tool misuse - Identity abuse - Memory poisoning - A Dark Reading poll found that 48% of cybersecurity professionals expect agentic AI to be the leading attack vector by the end of 2026. - Cisco reported that although 83% of organizations planned to deploy agentic AI, only 29% felt prepared to secure it. - The article argues that learning to think like an attacker is essential for closing this readiness gap. ## ProdBot: A Deliberately Vulnerable AI Assistant - ProdBot is a terminal-based productivity and coding assistant inspired by tools such as OpenClaw and GitHub Copilot CLI. - It can: - Convert natural-language requests into bash commands. - Browse a simulated web. - Connect to MCP servers. - Run organization-approved skills. - Store persistent memory. - Coordinate multiple agents. - Players’ objective is to use natural language to make ProdBot reveal the contents of `password.txt`. - No prior AI or coding experience is required; all interaction takes place through the CLI. ## Five Progressive Attack Surfaces - **Level 1: Shell execution** - ProdBot runs generated bash commands in a sandbox. - The challenge is to determine whether the sandbox can be escaped. - **Level 2: Web browsing** - ProdBot reads simulated news, finance, sports, and shopping sites. - Untrusted web content introduces risks such as instruction hijacking and prompt injection. - **Level 3: MCP integrations** - ProdBot gains access to external tool providers for stock quotes, browsing, and cloud backup. - Additional tools increase both functionality and opportunities for abuse. - **Level 4: Skills and memory** - Organization-approved plugins and persistent memory create layered trust relationships. - The level tests whether trusted skills and stored information are actually safe. - **Level 5: Multi-agent orchestration** - ProdBot combines six specialized agents, three MCP servers, three skills, and a simulated open-source project. - Claims that agents are sandboxed and data is pre-verified become assumptions to test rather than guarantees. ## Real-World Relevance - The game’s vulnerabilities reflect active security concerns in deployed autonomous AI systems rather than purely theoretical exercises. - The article cites CVE-2026-25253, known as “ClawBleed,” an OpenClaw vulnerability rated CVSS 8.8. - The flaw allowed attackers to steal authentication tokens through a malicious link and gain full control of an OpenClaw instance. - Season 4’s broader goal is to develop instincts for identifying similar weaknesses during architecture reviews, tool-integration audits, and production deployments. Developers working with AI agents should treat every new capability—shell access, browsing, plugins, memory, or collaboration—as a potential attack surface. Practicing these failure modes in a controlled environment like the Secure Code Game can help teams design safer agentic systems before deploying them.

Read original

Prompt Injection AI Agents Model Context Protocol Ai Security+2

lineMar 25, 20263 min readCurated summary

Advancing Guardrail Models through Automated Vulnerability Collection and Generation Using Coding Agents

LLM guardrails must detect prompt injection and jailbreak attempts without blocking legitimate requests that merely contain security-related keywords. The post argues that benchmark scores alone do not reflect production performance, especially false positives caused by missing input diversity. It presents a Codex-based, automated testing pipeline that generates categorized test data, evaluates the guardrail model, and analyzes failures reproducibly. ## The Gap Between Benchmark and Production Performance - The initial guardrail model performed well on external benchmarks but produced unexpected false positives in production-like tests. - Legitimate requests containing terms such as “ignore,” “bypass,” “override,” “system prompt,” or “jailbreak” were sometimes classified as attacks. - Examples included: - Development questions about temporarily bypassing authentication in a local test environment. - Educational requests about jailbreak techniques and defensive guidelines. - The core issue was insufficient representation of real-world input diversity, not simply poor model quality. - This motivated an automated environment for repeatedly discovering and analyzing guardrail weaknesses. ## Using Codex as a Test Automation Tool - The team adapted coding agents from software development tasks to complex, repeatable security testing. - Codex was used through its CLI capabilities to: - Read and create project files. - Edit code. - Execute evaluation scripts. - The pipeline relies on three Codex concepts: - **AGENTS.md:** Defines global rules, project conventions, commands, and security constraints. - **Sub-agents:** Allow a main orchestrator to delegate independent category tests to parallel worker agents. - **Skills:** Package repeatable procedures, input/output specifications, prompts, and scripts into reusable modules. ## Category-Based Experiments - Instead of sending thousands of random samples, experiments are divided into vulnerability and false-positive categories. - Example categories include: - Normal development or IT requests containing security-related keywords. - Educational or preventive requests involving sensitive topics such as jailbreaks or drug abuse prevention. - Categorization improves: - Root-cause analysis. - Parallel execution through independent workers. - Context clarity. - Regression testing after model changes. ## Separate Generation and Evaluation Skills ### `synthetic-generator` - Creates test queries according to each category’s specification. - Enforces constraints such as: - Attack type. - Sentence length. - Safe or dangerous target labels. - Produces varied, realistic phrasing and stores the dataset as JSONL. ### `injection-classifier` - Sends generated inputs to the guardrail model API through Python scripts. - Compares predictions with ground-truth labels. - Calculates false-positive and false-negative statistics. - Stores the original text, labels, predictions, and metrics in a consolidated JSONL file. Separating these procedures into skills provides intermediate artifacts for debugging, fixed input/output contracts for reproducibility, and independent maintenance of generation and evaluation logic. ## Pipeline Architecture - A **main agent**: - Reads `AGENTS.md` and `TEST_CATEGORY.md`. - Determines categories, sample counts, and constraints. - Creates and assigns work to category-specific workers. - Collects completion reports and verifies the run. - Each **category worker**: - Generates `input.jsonl` using `synthetic-generator`. - Evaluates the guardrail model using `injection-classifier`. - Produces `result.jsonl` with predictions and metrics. - Analyzes false positives and false negatives. - Writes a Markdown analysis report. - Stores outputs under `outputs/<run_id>/`, organized by category. ## Results and Practical Recommendation The pipeline enables systematic, repeatable testing rather than isolated discovery of misclassifications. For production guardrails, teams should combine benchmark evaluation with categorized real-world simulations, modular generation and evaluation steps, parallel test agents, and preserved JSONL artifacts for debugging and regression analysis.

Read original

Prompt Injection Python API Codex+4

cloudflareMar 11, 20263 min readCurated summary

AI Security for Apps is now generally available

Cloudflare’s AI Security for Apps is now generally available to help organizations discover, detect, and mitigate threats targeting AI-powered applications. The service protects AI endpoints through Cloudflare’s reverse proxy and integrates AI-specific signals with existing WAF controls. New GA features include free endpoint discovery for all customers, custom topic detection, and custom prompt extraction capabilities. ## Why AI Applications Create a New Attack Surface - Traditional applications follow predictable workflows, while AI applications accept natural-language input and produce probabilistic outputs. - Attackers can exploit models through: - Prompt injection - Sensitive information disclosure - PII exposure - Toxic or off-policy content - Unbounded resource consumption - Risks become more serious when AI agents can call tools to issue refunds, change accounts, apply discounts, or access customer data. - Organizations may struggle to maintain consistent safeguards as teams rapidly adopt new models and providers. ## AI Endpoint Discovery - AI Security for Apps automatically identifies LLM-powered endpoints across web properties, regardless of hosting location or model provider. - Discovery analyzes endpoint behavior rather than relying only on paths such as `/chat/completions`, since AI may also power search, valuation, and recommendation features. - Discovered endpoints appear under **Security → Web Assets** with the `cf-llm` label. - Discovery is now free for Cloudflare Free, Pro, and Business customers: - Free-plan discovery begins when users open the Discovery page. - Paid plans receive recurring background discovery. - Sufficient valid traffic is required to confidently identify AI-powered endpoints. ## Always-On AI Threat Detection - Prompts sent to discovered AI endpoints are evaluated by multiple detection modules. - Built-in protections cover: - Prompt injection - PII exposure and extraction - Sensitive or toxic topics - Detection results are attached as metadata and can be used in custom WAF rules. - Cloudflare intends to use its global network data to identify emerging attack patterns across millions of sites. ## Custom Topics and Prompt Extraction - Custom topics allow organizations to define their own sensitive categories, such as securities, patient data, or competitor products. - Each prompt and response receives a relevance score that can be logged, blocked, or handled through custom policies. - Custom prompt extraction identifies where prompts are located in request bodies, supporting formats such as: - `$.messages[*].content` - `$.requests[*].params.messages[*].content` - `$.property_description` - Standard formats from providers including OpenAI, Anthropic, Google Gemini, Mistral, Cohere, xAI, and DeepSeek are supported. - Unknown payload formats use a default-secure approach that scans the full request body, which may create false positives. - Cloudflare plans to support user-defined JSONPath expressions and automated prompt learning to improve accuracy. ## WAF-Based Mitigation - Organizations can block, log, or customize responses to detected threats using the existing Cloudflare WAF rule builder. - AI-specific signals can be combined with broader request data, including IP reputation, browser fingerprints, bot activity, and attack history. - This lets security teams distinguish isolated suspicious prompts from attacks associated with broader malicious behavior. ## Expanding Partnerships - IBM has selected Cloudflare to provide AI security for its cloud customers. - Cloudflare and Wiz are partnering to provide mutual customers with a unified view of AI security posture. - The service is being extended through Cloudflare’s broader security ecosystem. Organizations adopting AI agents should first inventory their AI endpoints, then apply detection and WAF policies tailored to their data, users, and business risks. Cloudflare’s unified approach is designed to provide a practical fail-safe as AI deployments expand.

Read original

Prompt Injection Database Design Large Language Models Cloudflare+4

githubMar 9, 20263 min readCurated summary

Under the hood: Security architecture of GitHub Agentic Workflows

GitHub Agentic Workflows are designed to bring autonomous agents into CI/CD without giving them unrestricted access to repositories, secrets, or the internet. Because agents can be prompt-injected and behave unpredictably, GitHub treats them as untrusted components and compiles workflows into constrained GitHub Actions. The architecture relies on layered isolation, controlled communication, staged writes, and comprehensive auditing. ## Threat Model - Agents reason over repository state and act autonomously, so they cannot be trusted by default. - GitHub Actions normally place components in one permissive trust domain with broad access to: - Repository contents - Authentication secrets - MCP servers - Arbitrary network destinations - A malicious webpage, issue, or repository file could prompt an agent to: - Read credentials from files, environment variables, logs, or `/proc` - Upload secrets externally - Embed secrets in issues, pull requests, or comments - Make unwanted repository changes - Strict mode follows four principles: - Defense in depth - Never trust agents with secrets - Stage and vet writes - Log everything ## Layered Security Architecture GitHub Agentic Workflows use three complementary layers: - **Substrate layer** - Runs on a GitHub Actions runner VM. - Uses trusted containers, Docker isolation, network controls, and kernel-enforced boundaries. - Separates components and mediates privileged operations and system calls. - Is intended to contain damage even if an untrusted component is compromised. - **Configuration layer** - Defines which components run and how they connect. - Controls communication channels, privileges, firewall policies, Docker images, and MCP configuration. - Determines which tokens are loaded into which containers. - Converts declarative workflow configuration into a secure runtime structure. - **Planning layer** - Controls which components are active and how data moves between them over time. - Creates staged workflows with explicit data exchanges. - Uses the Safe Outputs subsystem to govern potentially dangerous operations. ## Keeping Secrets Away from Agents - In ordinary GitHub Actions, secrets may be visible through environment variables and configuration files across the shared runner trust domain. - This creates a major prompt-injection risk: an agent with shell access could discover credentials and exfiltrate them. - Agentic Workflows instead place the agent in a dedicated container with: - Firewalled internet access - MCP access through a trusted gateway - LLM communication through an API proxy - A private network connects the agent only to approved services. - The trusted MCP gateway launches MCP servers and exclusively handles MCP authentication material. - LLM authentication tokens are kept in the isolated API proxy rather than exposed directly inside the agent container. ## Controlled Execution and Writes - Open-ended workflow authoring is separated from governed execution. - Workflows are compiled into GitHub Actions with explicit constraints covering: - Permissions - Outputs - Network access - Auditability - The planning and Safe Outputs systems are intended to mediate GitHub write operations and apply controls such as call filtering, volume limits, secret removal, and moderation. GitHub’s approach is to treat agents as untrusted CI/CD components rather than granting them normal workflow privileges. Organizations adopting agentic automation should isolate agents, broker access to tools and credentials, restrict network connectivity, stage all writes for review, and maintain detailed logs.

Read original

Prompt Injection CI/CD Model Context Protocol Docker & Containers+3

lineDec 17, 2025Original article

Safety is a Given, Cost (opens in new tab)

AI developers often rely on system prompts to enforce safety rules, but this integrated approach frequently leads to "over-refusal" and unpredictable shifts in model performance. To ensure both security and operational efficiency, it is increasingly necessary to decouple safety mechanisms into separate guardrail systems that operate independently of the primary model's logic. ## Negative Impact on Model Utility * Integrating safety instructions directly into system prompts often leads to a high False Positive Rate (FPR), where the model rejects harmless requests alongside harmful ones. * Technical analysis using Principal Component Analysis (PCA) reveals that guardrail prompts shift the model's embedding results in a consistent direction toward refusal, regardless of the input's actual intent. * Studies show that aggressive safety prompting can cause models to refuse benign technical queries—such as "how to kill a Python process"—because the model adopts an overly conservative decision boundary. ## Positional Bias and Context Neglect * Research on the "Lost in the Middle" phenomenon indicates that LLMs are most sensitive to information at the beginning and end of a prompt, while accuracy drops significantly for information placed in the center. * The "Constraint Difficulty Distribution Index" (CDDI) demonstrates that the order of instructions matters; models generally follow instructions better when difficult constraints are placed at the beginning of the prompt. * In complex system prompts where safety rules are buried in the middle, the model may fail to prioritize these guardrails, leading to inconsistent safety enforcement depending on the prompt's structure. ## The Butterfly Effect of Prompt Alterations * Small, seemingly insignificant changes to a system prompt—such as adding a single whitespace, a "Thank you" note, or changing the output format to JSON—can alter more than 10% of a model's predictions. * Modifying safety-related lines within a unified system prompt can cause "catastrophic performance collapse," where the model's internal reasoning path is diverted, affecting unrelated tasks. * Because LLMs treat every part of the prompt as a signal that moves their decision boundaries, managing safety and task logic in a single string makes the system brittle and difficult to iterate upon. To build robust and high-performing AI applications, developers should move away from bloated system prompts and instead implement external guardrails. This modular approach allows for precise security filtering without compromising the model's creative or logical capabilities.

Prompt Injection AI Large Language Models Prompt Engineering+4

lineNov 4, 2025Original article

Security Threat Cases and Countermeasures (opens in new tab)

Developing AI products introduces unique security vulnerabilities that extend beyond traditional software risks, ranging from package hallucinations to sophisticated indirect prompt injections. To mitigate these threats, organizations must move away from trusting LLM-generated content and instead implement rigorous validation, automated threat modeling, and input/output guardrails. The following summary details the specific risks and mitigation strategies identified by LY Corporation’s security engineering team. ## Slopsquatting and Package Hallucinations - AI models frequently hallucinate non-existent library or package names when providing coding instructions (e.g., suggesting `huggingface-cli` instead of the correct `huggingface_hub[cli]`). - Attackers exploit this by registering these hallucinated names on public registries to distribute malware to unsuspecting developers. - Mitigation requires developers to manually verify all AI-suggested commands and dependencies before execution in any environment. ## Prompt Injection and Arbitrary Code Execution - As seen in CVE-2024-5565 (Vanna AI), attackers can inject malicious instructions into prompts to force the application to execute arbitrary code. - This vulnerability arises when developers grant LLMs the autonomy to generate and run logic within the application context without sufficient isolation. - Mitigation involves treating LLM outputs as untrusted data, sanitizing user inputs, and strictly limiting the LLM's ability to execute system-level commands. ## Indirect Prompt Injection in Integrated AI - AI assistants integrated into office environments (like Gemini for Workspace) are susceptible to indirect prompt injections hidden within emails or documents. - A malicious email can contain "system-like" instructions that trick the AI into hiding content, redirecting users to phishing sites, or leaking data from other files. - Mitigation requires the implementation of robust guardrails that scan both the input data (the content being processed) and the generated output for instructional anomalies. ## Permission Risks in AI Agents and MCP - The use of Model Context Protocol (MCP) and coding agents creates risks where an agent might overstep its intended scope. - If an agent has broad access to a developer's environment, a malicious prompt in a public repository could trick the agent into accessing or leaking sensitive data (such as salary info or private keys) from a private repository. - Mitigation centers on the principle of least privilege, ensuring AI agents are restricted to specific, scoped directories and repositories. ## Embedding Inversion and Vector Store Vulnerabilities - Attacks targeting the retrieval phase of RAG (Retrieval-Augmented Generation) systems can lead to data leaks. - Embedding Inversion techniques may allow attackers to reconstruct original sensitive text from the vector embeddings stored in a database. - Securing AI products requires protecting the integrity of the vector store and ensuring that retrieved context does not bypass security filters. ## Automated Security Assessment Tools - To scale security, LY Corporation is developing internal tools like "ConA" for automated threat modeling and "LAVA" for automated vulnerability assessment. - These tools aim to identify AI-specific risks during the design and development phases rather than relying solely on manual reviews. Effective AI security requires a shift in mindset: treat every LLM response as a potential security risk. Developers should adopt automated threat modeling and implement strict input/output validation layers to protect both the application infrastructure and user data from evolving AI-based exploits.

Prompt Injection AI Large Language Models Model Context Protocol+5