How to scan for vulnerabilities with GitHub Security Lab’s open source AI-powered framework
GitHub Security Lab’s open-source Taskflow Agent uses AI-driven, multi-step auditing workflows to find high-impact vulnerabilities in web applications and open-source projects. The authors report more than 80 vulnerabilities, including authorization bypasses and private-data disclosures, with about 20 already disclosed. They argue that carefully designed taskflows and prompts can give LLMs enough freedom to discover vulnerabilities while reducing hallucinations and false positives. ## Running the Audits - The taskflows are available in the [`seclab-taskflows`](https://github.com/GitHubSecurityLab/seclab-taskflows) repository. - To run an audit: 1. Start a Codespace for the repository. 2. Wait for initialization. 3. Run `./scripts/audit/run_audit.sh myorg/myrepo`. - Audits may take one or two hours on a medium-sized repository. - Results are stored in SQLite and can be inspected in the `audit_results` table. - Rows marked with a check in `has_vulnerability` indicate potential findings. - A GitHub Copilot license and premium model requests are required. - The same repository should be audited multiple times because LLM results are nondeterministic; using different models may reveal different vulnerabilities. - Private repositories require changes to the Codespace configuration to grant access. ## How Taskflows Work - Taskflows are YAML files defining ordered tasks and dependencies for an LLM. - The `seclab-taskflow-agent` runs tasks sequentially and passes their results between stages. - Repository audits begin by dividing the codebase into functional components. - For each component, context is gathered, including: - Untrusted-input entry points - Intended privilege levels - Component purposes and behavior - This context is stored in a database for later auditing tasks. - Separate tasks can: - Suggest generic security issues - Carefully verify each suggested issue - Focus on specific vulnerability classes - Tasks can be reused across many components asynchronously through templated prompts and component-specific substitutions. ## Why Use Multiple Tasks - A single large prompt is less reliable because LLMs may omit steps in complex, multi-stage investigations. - Taskflows help control, debug, and structure the process even when models provide large context windows. - Breaking work into stages allows each result to be reviewed and reused as context for subsequent analysis. - Repeated task execution across components makes the approach scalable for large repositories. ## General Security Auditing - The team initially used the framework to triage CodeQL alerts, where strict instructions and predefined criteria helped limit false positives. - General auditing is more difficult because the LLM must search broadly for vulnerabilities rather than evaluate known alerts. - Greater freedom increases the risk of hallucinations and unexploitable findings. - The authors’ approach uses taskflow design and prompt engineering to preserve a high true-positive rate while allowing the model to investigate diverse security issues. ## Reported Vulnerabilities - The taskflows have found more than 80 vulnerabilities in open-source projects. - Many reported issues are high-impact, including: - Authorization bypasses - Information disclosure - Logging in as another user - Accessing private user data - Examples include exposing personally identifiable information in ecommerce shopping carts and authenticating to a chat application with arbitrary passwords. - The authors manually verify findings before reporting them and maintain an advisories page as disclosures become public. The practical recommendation is to run the open-source taskflows on your own projects, repeat audits with different models, and manually validate every result. The framework is intended to improve through shared taskflows, prompts, and findings across the security community.
Read original(opens in new tab)