Malicious Code Detection

datadogMar 9, 20263 min readCurated summary

When an AI agent came knocking: Catching malicious contributions in Datadog’s open source repos

Datadog describes how AI-powered attackers targeted its open-source repositories through malicious issues, pull requests, and comments. The campaign, attributed to the “hackerbot-claw” agent, focused on weaknesses in GitHub Actions and LLM-powered workflows. Datadog’s LLM-based review system and layered CI security controls detected the activity and helped limit its impact, while prompting further hardening. ## Why Open-Source Repositories Attract Attackers - Public repositories are attractive targets because automated CI/CD pipelines often build and execute code from external contributions. - Common attack techniques include: - Injecting user-controlled values, such as PR titles, into workflow scripts. - Using indirect poisoned pipeline execution to introduce malicious dependencies or build instructions. - Abusing `pull_request_target` workflows, which may run untrusted code with elevated permissions. - Prompt-injecting LLM-powered GitHub Actions used for issue triage, labeling, or code assistance. - Attackers may also disguise malicious changes through: - Large or obfuscated diffs. - Invisible Unicode characters. - Malicious libraries. - Imposter commits that resemble legitimate dependency references. ## Datadog’s LLM-Based Contribution Detection - Datadog receives dozens of external PRs each week across projects such as the Agent, tracers, SDKs, Vector, chaos-controller, and Stratus Red Team. - Its BewAIre system monitors GitHub events and selects security-relevant activity, including PRs and pushes. - BewAIre: - Extracts, normalizes, and enriches code diffs. - Sends them through a two-stage LLM pipeline. - Classifies changes as benign or malicious. - Produces a structured explanation for each verdict. - Malicious verdicts are forwarded to Datadog Cloud SIEM, where detection rules create enriched signals for the Security Incident Response Team to investigate. ## Hardening CI and Development Workflows - Datadog reduces the potential impact of successful attacks through multiple preventive controls: - Its `dd-octo-sts-action` generates minimally scoped, short-lived GitHub credentials using OIDC. - Long-lived and overly broad personal access tokens and GitHub Apps are being replaced. - Unused GitHub Actions secrets are identified and removed across thousands of repositories. - Organization-wide controls enforce branch protection, mandatory PR approval, commit signing, and lower-privilege default `GITHUB_TOKEN` permissions. - Engineers are provided with documented best practices and secure “golden paths” for CI development. ## The Hackerbot-Claw Campaign - Modern AI models are increasingly capable of offensive security tasks, especially when given tools, feedback loops, and autonomy. - StepSecurity reported an AI agent attacking open-source CI systems on March 1. - Between February 27 and March 2, the actor: - Opened 16 pull requests. - Created two issues and eight comments. - Targeted nine repositories across six organizations. - The activity was later linked to the hackerbot-claw agent, whose GitHub account was removed. - Datadog’s investigation began after BewAIre alerted the team to a suspicious contribution in the newly public `datadog-iac-scanner` repository on February 27. ## Practical Takeaway Organizations that accept public contributions should combine automated, AI-assisted review with least-privilege credentials, strict workflow permissions, secret management, mandatory approvals, and human incident response. Detection alone is insufficient; CI pipelines should be designed so that a malicious contribution has limited access and minimal opportunity to compromise secrets or production systems.

Read original

Malicious Code Detection Database Design Large Language Models CI/CD+4

datadogOct 21, 20253 min readCurated summary

Detecting malicious pull requests at scale with LLMs

Datadog built an LLM-powered security reviewer, BewAIre, to examine every pull request for malicious intent as AI-assisted development increases code volume and reviewer fatigue. Unlike traditional static analysis, the system evaluates the design and likely purpose of changes, not just known syntactic patterns. In testing, it achieved over 99.3% accuracy with a 0.03% false-positive rate and now operates in production. ## Why Conventional Review Is Struggling - Datadog processes nearly 10,000 pull requests per week across internal and external repositories, with volume growing as coding assistants become commonplace. - More changes increase the attack surface and make subtle attacks harder to notice. - Reviewers, linters, and scanners can be overwhelmed by verbose or repetitive AI-generated code. - Static analysis is effective at identifying known bad patterns but generally lacks understanding of developer intent. - Legitimate security-sensitive changes, such as permission-model updates, can resemble malicious behavior. ## The tj-actions/changed-files Attack - An attacker used a compromised personal access token belonging to `tj-actions-bot` to modify `index.js`. - The injected Python code extracted secrets and other sensitive information from a GitHub Actions runner’s process memory. - Base64 encoding concealed both the malicious code reference and output written to build logs. - The commit was made to look like a legitimate `renovate[bot]` update. - Version tags were redirected to the malicious commit, allowing the compromise to spread quickly. - Even protected branches, two-person reviews, and other standard controls could miss an attack disguised as a routine dependency or maintenance change. ## BewAIre’s LLM-Based Review Pipeline - **Ingestion:** Collects pull requests merged into the default branch across Datadog repositories. - **Preprocessing:** Normalizes each pull request, extracts its diff, and adds metadata such as the author and repository type. - **Inference:** Sends the code changes and contextual information to an LLM to reason about intent and classify the change as benign or malicious. - **Security signals:** Converts each verdict into a Datadog security signal that appears in internal dashboards and can trigger alerts or pages for security engineers. - Each finding includes a written explanation of why the change may be malicious. ## Improving Accuracy and Limiting Noise - Datadog used prompt engineering and data tuning to improve the model’s ability to distinguish malicious behavior from legitimate changes. - Suppression rules were added for known-safe patterns. - Testing on a curated, balanced dataset of hundreds of malicious and benign pull requests produced: - More than 99.3% overall accuracy. - More than 93.5% balanced accuracy. - A 0.03% false-positive rate. - The team also designed the system to address model degradation caused by context limits and continuously evaluated it against real-world attacks, including the tj-actions and Nx incidents. ## Production Deployment - BewAIre reviews Datadog pull requests in real time and is already deployed across the company’s repositories. - The system is available in Preview to Static Code Analysis customers. - Its purpose is to add scalable, intent-focused detection without imposing stricter manual review requirements that could slow development. Datadog’s approach suggests that LLMs can complement—not replace—static analysis and human investigation by providing continuous, intent-aware security review at large scale.

Read original

Malicious Code Detection Large Language Models Prompt Engineering Code Review+4