GitHub/AI Agents

21 posts

githubAug 12, 20262 min readCurated summary

Write your first prompt with the GitHub Copilot app

Starting with GitHub Copilot does not require a perfect prompt or specialized syntax. The article recommends connecting Copilot to a repository or local folder, describing a task in plain English, and refining the request interactively. Users can gradually adjust the AI model, input method, and session settings as their needs become more complex. ## Start with Project Context - Connect an agent session to: - An existing GitHub repository - A local folder on your computer - Selecting a project gives Copilot access to the code and files needed for the task. - Once the project is connected, you can submit a prompt. ## Describe Tasks in Plain English - Prompts only need to explain the desired outcome. - Example: `Add a most-funded sort option to the games list.` - Copilot can inspect the codebase and identify relevant files. - If the result is incomplete or incorrect, provide more details and ask for revisions. - Prompting is iterative, so the initial request does not need to include every requirement. ## Select an Appropriate AI Model - The app supports multiple AI models with different strengths. - More capable reasoning models may help with complex tasks, while simpler models can be faster for straightforward changes. - Beginners can use the default model without understanding every model difference. - Models can be switched later if the task becomes more demanding or the initial result is unsatisfactory. ## Use Voice Input - Built-in voice input lets users describe tasks verbally. - Speech is converted into editable text before submission. - This can be useful for explaining lengthy or complicated ideas more naturally. ## Customize Agents and Sessions - Session settings allow users to select different agents for different types of work. - Remote control enables access to a session through the web. - Users can start work locally, leave their computer, and resume the same session from another device. - These options are available when needed but are not required for a first task. ## Start Small and Iterate - Begin with a modest change in a familiar project. - Review Copilot’s work and refine the prompt as necessary. - Experiment with different models or session configurations only when the task requires them. The practical recommendation is to choose a project, describe one small task in ordinary language, and begin. Experience with iterative prompting will make model selection and other Copilot settings easier to use over time.

AI Agents Github Copilot Prompt Engineering Ai Coding Assistant+2

githubAug 11, 20262 min readCurated summary

From coder to orchestrator: How agents shift the role of a developer

AI agents can generate impressive one-prompt demos, but reliable software delivery requires more than isolated outputs. Developers increasingly need to design workflows that define how code is proposed, tested, reviewed, and shipped. The article argues that this shifts developers from primarily writing code to orchestrating agents within controlled, repeatable systems. ## From One-Off Prompts to Reliable Workflows - A single prompt can quickly produce a demo, such as a simple game. - Production development requires repeatable delivery with: - Appropriate context - Validation and testing - Security controls - Review processes - Clear permissions and handoffs - GitHub Copilot is presented as a control plane for connecting these parts. ## An Agentic Development Flow - Familiar repository events can trigger agent work, including: - Adding a label to an issue - Running a scheduled workflow - Starting a GitHub Actions process - The agent’s changes are captured in a pull request. - Deterministic checks then validate the work through: - Linting - Tests - Security scans - Build verification - CODEOWNERS, required reviews, and branch protection rules control what can be merged. - Agents handle ambiguous, context-heavy tasks, while predictable automation provides the safety boundary. - Developers decide: - What agents can access - How tasks are scoped - Where workflows hand off - When human judgment is required ## GitHub’s Implementation Options - Copilot cloud agent workflows support event-driven automations. - Copilot CLI can run AI-powered steps inside GitHub Actions. - Model Context Protocol (MCP) can extend agents with additional tools and external context. - These options represent different stages of building an agent-enabled development workflow. ## Starting Small - Teams should begin with one bounded, low-risk workflow. - Suitable examples include: - Issue triage - Synchronizing documentation and tests - Routine maintenance updates - The recommended approach is to integrate Copilot into existing development infrastructure rather than redesigning everything at once. Developers should treat AI agents as components within an engineered delivery system, not as replacements for that system. Start with a limited workflow, surround agent output with automated checks and review controls, and gradually expand as the process proves reliable.

AI Agents CI/CD Model Context Protocol Github Copilot+4

githubJul 27, 20263 min readCurated summary

The harness is all you need (mostly)

The post argues that effective AI-assisted development depends less on discovering elaborate prompts, tools, or configurations and more on learning the underlying agent harness. A simple workflow—choose a consistent tool, grant agents autonomy in a safe environment, and prototype before implementing—can produce major productivity gains. The author concludes that the harness is “all you need,” with only a few practical habits required. ## Choose a Tool and Learn the Harness - GitHub Copilot is available through the CLI, Copilot app, VS Code, Visual Studio, JetBrains, and other environments. - Although interfaces differ, these tools increasingly share the same underlying harness and workflow. - Beginners may benefit from starting with the CLI because its text-based interface exposes the agent’s behavior directly. - Learning the harness once makes it easier to work across different Copilot environments. ## Enable Autonomous Agent Operation Safely - “YOLO mode,” also called `/allow-all` or “Allow All,” lets the agent execute commands without requesting approval each time. - Repeated approval prompts slow development and encourage users to approve actions without reading them. - Agents should be given autonomy, but unrestricted execution is risky on local or production systems. - Use isolated environments such as GitHub Codespaces or development containers, especially when working with private organizational data. ## Prototype Before Implementing - AI makes it inexpensive to explore multiple designs before committing to an implementation. - For a date picker, the author recommends generating many HTML mockups at once—for example, 20 variations—to compare layouts and interaction models. - Reviewing prototypes can reveal ideas that are difficult to anticipate from text alone, such as navigating from year to month to day. - Visual and tangible representations help people understand complex concepts faster than dense descriptions. ## Use Visual Models for Non-Visual Work - Prototyping is also useful for backend tasks and API design. - For an analytics-download endpoint, the author suggests asking the agent to create several visual implementation options. - Mermaid diagrams can map out alternatives directly in Markdown. - Exploring these options early exposes requirements and trade-offs, reducing rework and wasted tokens. ## Keep the Model and Reasoning Settings Consistent - The author recommends using a medium-sized model with medium reasoning for most tasks. - Keep the same model and reasoning level throughout a feature, bug fix, or enhancement. - Consistency allows previous conversations to remain cached, reducing token costs on later requests. The practical recommendation is to focus less on collecting AI tools and clever prompts and more on mastering the agent workflow: give it controlled autonomy, prototype alternatives early, and maintain a consistent working context.

AI Agents Github Copilot Github Codespaces Development Containers+2

githubJul 27, 20262 min readCurated summary

GitHub Copilot app for Beginners: Getting started

The GitHub Copilot app is designed as a development workspace rather than a single AI chat window. It connects agent sessions to projects, supports parallel tasks, provides an interactive browser canvas for UI work, and helps manage pull requests through Agent Merge. Together, these features aim to support the full workflow from exploration to shipping. ## Project-Based Agent Sessions - Each session is connected to a specific project and its repository context. - Projects can be selected from GitHub or added from a local machine. - Copilot can inspect the codebase, identify relevant files, implement changes, and run tests. - This reduces the setup required before beginning a development task. ## Managing Multiple Work Threads - Users can create separate sessions for different tasks without interrupting ongoing work. - **Quick Chat** provides a lightweight way to: - Ask questions about Copilot or the codebase - Explore implementation options - Investigate unfamiliar parts of an application - Gather context before making changes - Returning to an existing session preserves its history and allows work to continue from where it stopped. ## Interactive UI Work with Canvas - The app includes a browser canvas for previewing applications alongside the AI conversation. - Canvas can be created with the `/create-canvas` slash command. - **Enable Canvas Dev Mode** and **Pick & Polish** allow users to select page elements directly and use them as context for refinement requests. - This supports an iterative workflow in which developers can inspect the visual result, identify problems, and ask Copilot to adjust specific UI elements. ## Pull Request Assistance with Agent Merge - **Agent Merge** extends Copilot’s role beyond implementation into code review and delivery. - It can be enabled from a pull request’s options in the Copilot app. - Developers choose which actions it may perform, including: - Addressing review feedback - Helping resolve CI failures - Handling merge conflicts - Agent Merge monitors the pull request while checks and reviews are in progress, preparing it for merge once requirements are satisfied. The Copilot app is intended to centralize development activities in one workspace: start with a project, separate work into focused sessions, visually refine applications through canvas, and use Agent Merge to help complete the pull request process. Developers can learn the workflow by applying it to an existing backlog task.

AI Agents CI/CD Github Github Copilot+3

githubJul 21, 20262 min readCurated summary

How to build interactive experiences with canvases

Canvases extend GitHub Copilot beyond text-based conversations by providing shared, interactive workspaces where developers and agents can visualize information and take action together. They are useful for tasks such as triaging issues, exploring code architecture, managing worktrees, and searching organizational knowledge. The article concludes that canvases make AI workflows more engaging, practical, and easier to iterate on. ## How canvases work - Canvas extensions are interactive surfaces within the GitHub Copilot app. - Agents can update the canvas while users interact through clicks, edits, swipes, and other controls. - User actions may be sent back to the agent or handled locally. - Developers create one by running `/create-canvas` in a Copilot agent session and describing the desired interface and capabilities. - Canvases can evolve through follow-up prompts that add features or refine existing behavior. ## Visual issue triage - A card-based interface displays GitHub Issues one at a time. - Users swipe right to ship an issue or left to reject it. - The canvas updates immediately, organizing issues into decision-based categories. - This replaces a slower, text-heavy review process with direct visual interaction. ## Interactive codebase diagrams - A generated diagram represents components of a project as connected nodes. - Users can hover over, drag, and filter elements to explore relationships and architectural layers. - The interface turns static codebase documentation into an explorable model. ## Managing sessions and worktrees - A canvas can display active GitHub Copilot sessions and their associated Git worktrees. - It distinguishes active, stale, and orphaned worktrees. - Cleanup actions are available directly from the interface, reducing manual maintenance. ## Improving agent prompts - The prompt coach reviews previous agent interactions. - It identifies issues such as unclear context, spelling mistakes, and syntax problems. - It suggests ways to make prompts more precise and improve future agent results. ## Finding organizational knowledge - A knowledge finder can search sources such as Slack, Teams, email, and documentation. - It identifies people connected to a file or topic. - Results explain both who has relevant context and where that connection was found, making it easier to locate the right expert. ## Getting started Canvases are best suited to workflows where visual understanding and direct manipulation are more effective than sequential prompts. Developers can try them in the GitHub Copilot app by using `/create-canvas`, then iterating with the agent to build a workspace tailored to their task.

AI Agents Slack Github Copilot Github Issues+3

githubJul 10, 20263 min readCurated summary

Better tools made Copilot code review worse. Here’s how we actually improved it.

Copilot code review became more expensive and less effective after GitHub replaced its specialized exploration tools with shared `grep`, `glob`, and `view` tools. The tools themselves worked correctly, but their general-purpose instructions encouraged broad repository browsing rather than focused pull request investigation. After rewriting the instructions around diff-first review workflows, GitHub achieved roughly 20% lower average review cost without reducing review quality. ## Why the Tool Migration Regressed - Copilot code review previously used specialized tools for: - Listing directories - Searching files and directories - Reading code - These tools often returned matching lines along with surrounding context, which suited earlier models that made fewer tool calls and needed more context per request. - GitHub migrated to the shared Unix-inspired tools used by Copilot CLI and other products: - `glob` replaced `list_dir` - `grep` replaced `search_file` and `search_dir` - `view` replaced `read_code` - The migration aimed to reduce duplicated implementations and let improvements benefit multiple Copilot products. - Offline benchmarks showed higher review costs and fewer useful comments after the migration. ## Repository Browsing Instead of Pull Request Review - Execution traces showed the agent: - Searching broadly - Guessing file paths - Reading large sections of code - Finding more things to search - Carrying unnecessary context into later reasoning - This workflow is reasonable for a coding assistant asked to understand an unfamiliar repository. - It is inefficient for code review, where the agent should begin with the pull request diff and investigate a specific potential problem. - Excessive tool output increases token usage because returned file contents remain in the agent’s context window. - Broad exploration can also make the review less focused by mixing relevant evidence with unrelated code. ## The Difference Between Coding and Reviewing - A coding assistant may need to map a large area of a repository before editing code safely. - A reviewer typically asks targeted questions based on the diff, such as: - Where is the changed function called? - Is a modified configuration key used elsewhere? - Does a similar test or helper already exist? - What is the smallest code range needed to understand the behavior? - Copilot code review has a narrower objective: - Start from the pull request diff - Determine whether the change introduced a real issue - Gather only the evidence needed to confirm or dismiss that issue - The shared tools were designed for broader interactive workflows, so their instructions unintentionally encouraged the wrong behavior in the review agent. ## Instructions Were the Real Fix - GitHub concluded that changing tools was not enough; the agent’s workflow instructions also had to change. - The revised instructions emphasized: - Diff-first investigation - Targeted searches - Minimal surrounding context - Narrow evidence gathering - Avoiding unnecessary repository-wide exploration - With these workflow changes, the shared tools became more effective for review rather than merely reproducing their coding-assistant behavior. - The result was approximately 20% lower average review cost while preserving review quality. The practical lesson is that tool quality cannot be evaluated separately from the instructions and workflow guiding an agent. Shared tools can work well across products, but each use case needs instructions that match its task—in this case, focused, evidence-driven pull request review rather than broad repository exploration.

AI Agents Github Copilot Code Review Pull Requests+2

githubJul 8, 20263 min readCurated summary

Automating cross-repo documentation with GitHub Agentic Workflows

The Aspire team used GitHub Agentic Workflows to automate documentation across its product and documentation repositories. For versions 13.3 and 13.4, the system produced 82 documentation pull requests, typically within 44.8 hours of the corresponding product change, with review from the engineer who shipped the feature. The approach combines AI-generated drafts with deterministic branch selection and tightly controlled write permissions. ## The Cross-Repository Documentation Problem - Product code lives in `microsoft/aspire`, while documentation lives in `microsoft/aspire.dev`. - The old process depended on writers discovering changes weeks later and reconstructing intent from closed pull requests. - Engineers often had limited context by the time clarification was requested. - Broad repository tokens were unacceptable, making secure cross-repository automation difficult. ## How GitHub Agentic Workflows Work - Workflows are authored as Markdown files with YAML-style frontmatter and natural-language instructions. - A compiler generates a standard GitHub Actions `.lock.yml` workflow. - An agent analyzes repository data and produces proposed actions as JSON rather than writing directly to GitHub. - A separate “safe-outputs” handler executes only explicitly permitted actions through a narrowly scoped GitHub App. - This separation provides AI flexibility while preserving security controls and auditability. ## The Automated Documentation Pipeline - The `pr-docs-check.md` workflow runs when a pull request is merged into `main` or a `release/*` branch. - A deterministic Bash script resolves the documentation target branch before the agent runs: - Product pull request milestone, such as `13.4`, maps to `release/13.4`. - Linked issue milestones are checked next. - The pull request’s base branch is used if it matches a release pattern. - Otherwise, documentation targets `main`. - The agent: - Reviews the product diff and linked issues. - Determines whether documentation is necessary. - Checks out `microsoft/aspire.dev`. - Writes documentation using the project’s existing writing conventions and Starlight/MDX components. - The workflow creates a draft documentation pull request with: - A `[docs]` title prefix. - The `docs-from-code` label. - A restricted base branch. - The documentation repository as the target. - The subject-matter expert who reviewed the original product pull request as reviewer. - A comment containing the documentation pull request link is posted back to the source pull request, while older workflow comments are minimized on reruns. ## Security Through Safe Outputs - The agent receives constrained GitHub tools and read access. - Repository access is limited through allowed repositories and a dedicated GitHub App. - Actions must use pinned, integrity-checked components through `min-integrity: approved`. - Write operations are restricted to declared safe outputs, such as creating pull requests. - Documentation changes remain drafts and are never auto-merged. ## Results and Broader Fit - The process eliminated the need for additional staff or major process training. - Documentation drafts arrive shortly after the related code is merged, while the implementation context is still fresh. - The workflow preserves human review by routing drafts to the engineer or SME who approved the feature. - Both the automation documentation and `aspire.dev` use Astro and Starlight, making the tooling and publishing environment closely aligned. The practical recommendation is to use agentic automation for drafting and routing documentation, but keep branch resolution, permissions, and final review deterministic and human-controlled. This provides much of the speed of autonomous workflows without granting an AI agent unrestricted repository write access.

AI Agents GitHub Actions Astro Github Agentic Workflows+3

githubJun 25, 20263 min readCurated summary

Evaluating performance and efficiency of the GitHub Copilot agentic harness across models and tasks

GitHub argues that an agentic harness—the system coordinating tools, context, and workflow—strongly influences how effectively an AI model solves software tasks. Its shared Copilot harness aims to match model-vendor harnesses in task completion while using fewer tokens. Benchmark results across four models suggest near-parity overall, with performance varying by model and task. ## Benchmarking Approach - GitHub evaluates the harness using public and internal benchmarks, plus real-world metrics and online experiments. - Comparisons hold the following constant: - The same underlying model - The same task - Context-window size - Reasoning effort - Tool selection - MCP servers - Copilot CLI is compared with: - Claude Code for Claude Sonnet 4.6 and Claude Opus 4.7 - Codex CLI for GPT-5.4 and GPT-5.5 - Benchmarks include: - **SWE-bench Verified:** 500 human-validated Python bug fixes - **SWE-bench Pro:** More complex, multi-step engineering tasks - **SkillsBench:** Skill usage and triggering - **TerminalBench:** Command-line workflows - **Win-Hill:** Software tasks in Windows containers ## Token Efficiency - Copilot’s harness generally completes tasks at rates comparable to model-vendor harnesses while consuming fewer tokens. - It performed better across the reported tests for Sonnet 4.6 and Opus 4.7. - For GPT-5.4 and GPT-5.5, Copilot performed better on nearly every benchmark. - The main exception was SWE-bench Verified, where Copilot was 7% worse with GPT-5.4 and 4% worse with GPT-5.5. ## Task Resolution - GitHub emphasizes that lower token use matters only when the agent successfully completes the work. - Overall task-resolution rates were considered on par with vendor harnesses. - Results varied by benchmark: - Copilot generally did better on SWE-bench Pro, especially with GPT models. - It outperformed vendor harnesses on Win-Hill or matched them for every model. - It performed better on TerminalBench with Sonnet and Opus, matched GPT-5.5, and trailed GPT-5.4. - SkillsBench results favored GPT models on Copilot but favored vendor harnesses for Claude models. - Differences are described as statistically comparable because model behavior is stochastic and run-to-run variation can explain many gaps. ## TerminalBench Variance Analysis - GitHub uses TerminalBench 2.0 to study both cost and completion rate over repeated runs. - The preferred outcome is higher resolution with lower cost. - The analysis illustrates that benchmark results should account for variance rather than relying on a single run. - GitHub presents Copilot as equal to or better than the vendor harnesses on this cost-versus-success comparison. The practical conclusion is that harness design is a reusable performance multiplier across Copilot products. GitHub’s results support using its shared harness when developers need broad model choice, efficient token usage, and comparable task-completion performance, while recognizing that the best harness can still vary by model and workload.

AI Agents Large Language Models Model Context Protocol Github Copilot+3

githubJun 19, 20263 min readCurated summary

How we built an internal data analytics agent

Qubot is GitHub’s internal, Copilot-powered analytics agent, designed to make warehouse data accessible without requiring an analyst. Employees ask natural-language questions through Slack, VS Code, or the Copilot CLI, while Qubot uses curated context and selects between Kusto and Trino to produce answers. GitHub’s experience shows that structured context is central to improving both accuracy and response speed. ## The Problem Qubot Addresses - Product teams often struggle to identify the right data model, grain, filters, and query. - Limited analyst availability leaves many teams to investigate telemetry independently. - Qubot targets exploratory questions rather than replacing dashboards or formal reporting. - It enables employees to investigate unfamiliar datasets with minimal setup and maintenance. ## Multiple Ways to Ask Questions - **Slack:** Users ask questions in a shared channel, receive answers in threads, and collaborate on follow-up questions. - Each result is saved as a Markdown report in a pull request, allowing users to refine queries or reuse them in dashboards. - **VS Code and Copilot CLI:** Qubot is installed as a plugin and operates alongside other agents, skills, and tools. - Offering both zero-configuration and developer-focused interfaces increased adoption among different user groups. ## A Federated Context Layer Qubot’s context is tailored to GitHub’s warehouse layers: - **Bronze:** Product teams provide telemetry schemas and metadata for raw events. - **Silver:** Data teams maintain query examples, usage guidance, and required filters for conformed data. - **Gold:** Dataset owners contribute business rules and metric definitions. - ETL pipelines add derived metadata and other signals automatically. - Context is fetched at runtime through the GitHub MCP Server. A dedicated context agent standardizes contributions from different teams. It ingests Markdown documentation and repository references, then organizes and normalizes them into a structure that Qubot can use effectively. ## Evaluation Before Deployment Every change to Qubot’s instructions or context layer is tested through an offline evaluation framework. - Test cases contain prompts, expected answers, ground-truth SQL, domains, and difficulty levels. - Automated orchestration launches multiple parallel agent trials using `gh agent-task create`. - Results are saved as JSON and aggregated by completion rate, accuracy, and duration. - Teams compare configurations and detect regressions before releasing changes. ## Choosing Between Kusto and Trino Qubot connects to both query engines through MCP servers: - **Kusto** is used for fast exploration of recent event data. - **Trino** handles complex joins and deeper historical analysis. - Qubot defaults to Kusto and switches to Trino when the question requires more advanced or historical querying. - This hides query-engine selection from users. ## Adoption and Lessons Learned - Hundreds of GitHub employees have run thousands of queries through Qubot. - Routine questions in analytics support channels declined as teams gained more autonomy. - The agent helped employees who previously avoided the warehouse access data for decision-making. - GitHub found that well-structured, carefully maintained context improved accuracy and made Qubot approximately three times faster at finding the right answer. The main recommendation is to treat analytics context as a maintained engineering asset. A capable agent depends not only on model intelligence, but also on accurate metadata, documented business rules, evaluation-driven iteration, and interfaces that fit users’ existing workflows.

AI Agents Model Context Protocol Slack Github Copilot+4

githubJun 17, 20263 min readCurated summary

Getting more from each token: How Copilot improves context handling and model routing

GitHub is improving Copilot’s efficiency by optimizing both how context is handled and which model performs each task. Prompt caching and deferred tool loading reduce repeated overhead in long agentic sessions, while Auto selects models based on task complexity and real-time system conditions. The goal is to preserve quality while using tokens, compute, and cost more intelligently. ## Prompt Caching and Deferred Tools - Copilot sessions repeatedly include instructions, repository context, conversation history, tool definitions, and task state. - Prompt caching allows repeated prompt prefixes to reuse model state instead of being recomputed on every request. - Tool search loads tool definitions only when needed, rather than placing every full schema into context on every turn. - This is especially valuable for sessions involving MCP tools, terminal commands, file operations, workspace search, and other actions. - The approach reduces fixed per-turn overhead while preserving access to a broad toolset. ## Auto Model Selection - Copilot Auto chooses a model based on the task rather than requiring developers to select one manually. - Quick explanations, focused edits, and complex multi-file changes often benefit from different levels of reasoning. - Evaluations found that no single model consistently performs best across all tasks. - Auto uses more efficient models when they can achieve comparable results and selects stronger reasoning models when the task demands them. - The objective is not simply lower cost, but matching model capability to the work. ## How Auto Routes Tasks Auto combines two signals: - **Real-time model health** - Considers availability, utilization, response speed, error rates, and cost. - Avoids selecting a model that is technically capable but currently overloaded or unreliable. - **Task-aware routing with HyDRA** - Evaluates reasoning depth, code complexity, debugging difficulty, and tool-orchestration requirements. - Identifies models capable of meeting the task’s quality threshold, then chooses the best fit among them. - The system can be tuned toward maximum quality or greater cost savings. ## Cache-Aware Routing - Switching models during every turn can reduce efficiency by invalidating cached prompt prefixes. - Auto therefore keeps the same model during a conversation so the cache can continue building. - It changes models at natural cache boundaries: - The first turn, when no cache exists yet. - After context compaction, when older turns are summarized and the prompt prefix resets. ## Multilingual Routing - Copilot’s routing system was trained on conversations spanning 16 language families, including CJK and European languages. - Evaluations covered production VS Code chat data across 19 languages. - Routing accuracy remained within four points of the English baseline across language groups. - No statistically significant quality gap was observed between the language groups. Copilot’s efficiency strategy combines smarter context management with adaptive model selection. Caching and on-demand tools reduce repeated work, while cache-aware Auto routing chooses an appropriate model without unnecessarily sacrificing quality or session efficiency.

AI Agents Model Context Protocol Github Copilot Visual Studio Code+3

githubJun 11, 20261 min readCurated summary

Making secret scanning more trustworthy: Reducing false positives at scale

Mariko is a Principal Applied Scientist at Microsoft who leads the development of agentic AI workflows for cybersecurity operations. Her work centers on using large language models and agentic systems to bring frontier AI research into practical products and operational environments. ## Professional Role - Principal Applied Scientist at Microsoft. - Leads agentic AI workflow development for cybersecurity operations. ## Research and Technical Focus - LLM-powered systems. - Agentic workflows. - Applying frontier AI research to real-world products and operations. Overall, Mariko’s work connects advanced AI research with practical cybersecurity and enterprise applications.

AI Agents Large Language Models Cybersecurity Ai Workflows+1

githubMay 6, 20263 min readCurated summary

Validating agentic behavior when “correct” isn’t deterministic

Agentic systems such as GitHub Copilot cloud agent can complete tasks through multiple valid action sequences, making traditional deterministic tests unreliable. Timing changes, loading screens, and UI differences often produce false negatives even when the agent achieves the correct result. The post proposes an independent “Trust Layer” that validates essential outcomes and convergent behavior rather than rigid step-by-step execution paths. ## Challenges of Agent-Driven Validation - An agent may adapt to network delays or changing UI conditions and still complete its task successfully. - Conventional CI tests can fail when execution no longer matches a recorded script or expected assertion timing. - This creates a trust gap: - **False negatives:** successful tasks are reported as failures. - **Fragile infrastructure:** rendering, timing, and environment noise affect test results. - **Compliance trap:** valid behavioral variation is mistaken for regression. - Agent correctness should focus on reliably achieving essential outcomes, not reproducing an identical sequence of actions. ## Why Traditional Testing Breaks Down - **Assertion-based tests** require manually specifying every expected check and often omit valid alternative paths. - **Record-and-replay tools** are highly sensitive to timing and rendering differences. - **Visual regression tests** compare screenshots without understanding semantic meaning or the broader workflow. - **ML-based oracles** need large training datasets and generally provide little explanation for their decisions. - All four approaches assume correctness means following a stable sequence of observable states, which does not fit autonomous agents. ## Essential, Optional, and Convergent Behavior The proposed approach distinguishes between behavior that determines success and behavior that merely reflects environmental variation: - **Essential states:** Required milestones, such as reaching a VS Code “Search Results” screen. - **Optional variations:** Incidental states, including loading spinners or decorative UI changes. - **Convergent paths:** Different action sequences—such as using a keyboard shortcut or a menu—that eventually reach the same result. - A loading screen may appear in one run and not another, but the appearance of search results is what establishes success. ## Dominator Analysis The post connects this model to **dominator relationships** from compiler theory: - In a control-flow graph, node A dominates node B when every path to B must pass through A. - Applying dominator analysis to agent execution traces can identify: - Mandatory states - Optional states - Points where different execution paths converge - This produces a minimal and explainable definition of correctness instead of relying on every recorded step. ## Graph-Based Execution Modeling - Agent behavior should be represented as a graph rather than a linear script. - Graphs capture branching paths, optional states, and convergence points. - This structure provides a foundation for lightweight, explainable validation in GitHub Actions and other CI environments. A reliable validation system for agents should test whether essential outcomes occurred and whether critical invariants held, while ignoring harmless differences in timing, rendering, and execution order. This outcome-oriented Trust Layer can reduce false failures and make agentic workflows more dependable in production CI pipelines.

AI Agents CI/CD Github Copilot GitHub Actions+3

githubApr 14, 20263 min readCurated summary

Hack the AI agent: Build agentic AI security skills with the GitHub Secure Code Game

Agentic AI tools can automate powerful tasks, but their autonomy creates new security risks, including prompt injection, tool misuse, memory poisoning, and compromised multi-agent workflows. GitHub’s Season 4 Secure Code Game teaches developers to recognize these threats by attacking and hardening ProdBot, a deliberately vulnerable terminal-based AI assistant. Its five levels progressively add capabilities—and corresponding attack surfaces—mirroring how real-world AI systems evolve. ## The Secure Code Game’s Evolution - The free, open-source, in-editor course teaches security by having players exploit and fix intentionally vulnerable code. - Earlier seasons covered: - General secure coding across JavaScript, Python, Go, and GitHub Actions. - LLM security, including malicious prompts and defensive techniques. - More than 10,000 developers from industry, academia, and open source have participated. - Season 4 shifts focus from AI that generates content to AI that independently browses, uses tools, calls APIs, and acts for users. ## Why Agentic AI Security Is Urgent - Agentic systems are moving rapidly from research projects into production environments. - The OWASP Top 10 for Agentic Applications identifies threats such as: - Goal hijacking - Tool misuse - Identity abuse - Memory poisoning - A Dark Reading poll found that 48% of cybersecurity professionals expect agentic AI to be the leading attack vector by the end of 2026. - Cisco reported that although 83% of organizations planned to deploy agentic AI, only 29% felt prepared to secure it. - The article argues that learning to think like an attacker is essential for closing this readiness gap. ## ProdBot: A Deliberately Vulnerable AI Assistant - ProdBot is a terminal-based productivity and coding assistant inspired by tools such as OpenClaw and GitHub Copilot CLI. - It can: - Convert natural-language requests into bash commands. - Browse a simulated web. - Connect to MCP servers. - Run organization-approved skills. - Store persistent memory. - Coordinate multiple agents. - Players’ objective is to use natural language to make ProdBot reveal the contents of `password.txt`. - No prior AI or coding experience is required; all interaction takes place through the CLI. ## Five Progressive Attack Surfaces - **Level 1: Shell execution** - ProdBot runs generated bash commands in a sandbox. - The challenge is to determine whether the sandbox can be escaped. - **Level 2: Web browsing** - ProdBot reads simulated news, finance, sports, and shopping sites. - Untrusted web content introduces risks such as instruction hijacking and prompt injection. - **Level 3: MCP integrations** - ProdBot gains access to external tool providers for stock quotes, browsing, and cloud backup. - Additional tools increase both functionality and opportunities for abuse. - **Level 4: Skills and memory** - Organization-approved plugins and persistent memory create layered trust relationships. - The level tests whether trusted skills and stored information are actually safe. - **Level 5: Multi-agent orchestration** - ProdBot combines six specialized agents, three MCP servers, three skills, and a simulated open-source project. - Claims that agents are sandboxed and data is pre-verified become assumptions to test rather than guarantees. ## Real-World Relevance - The game’s vulnerabilities reflect active security concerns in deployed autonomous AI systems rather than purely theoretical exercises. - The article cites CVE-2026-25253, known as “ClawBleed,” an OpenClaw vulnerability rated CVSS 8.8. - The flaw allowed attackers to steal authentication tokens through a malicious link and gain full control of an OpenClaw instance. - Season 4’s broader goal is to develop instincts for identifying similar weaknesses during architecture reviews, tool-integration audits, and production deployments. Developers working with AI agents should treat every new capability—shell access, browsing, plugins, memory, or collaboration—as a potential attack surface. Practicing these failure modes in a controlled environment like the Secure Code Game can help teams design safer agentic systems before deploying them.

AI Agents Model Context Protocol Ai Security Prompt Injection+2

githubApr 6, 20263 min readCurated summary

GitHub Copilot CLI combines model families for a second opinion

GitHub Copilot CLI’s experimental Rubber Duck feature adds an independent reviewer from a different AI model family to catch mistakes before they compound. When Claude models orchestrate a task, GPT-5.4 reviews plans, implementations, and tests at key checkpoints. On SWE-Bench Pro, Claude Sonnet 4.6 with Rubber Duck closed 74.7% of the performance gap with Claude Opus 4.6 alone, particularly on complex, multi-file tasks. ## The Problem with Self-Review - Coding agents typically assess a task, plan, implement, test, and iterate. - Early assumptions can create downstream dependencies and make small mistakes expensive to fix. - Self-reflection helps, but a model reviewing its own work may retain the same training biases and blind spots. ## Cross-Family Review with Rubber Duck - Rubber Duck is a focused review agent powered by a complementary model family. - Claude orchestrators currently use GPT-5.4 as the reviewer. - It produces a short list of high-value concerns, including: - Missed details - Questionable assumptions - Architectural risks - Relevant edge cases ## Evaluation Results - On SWE-Bench Pro, Sonnet 4.6 plus Rubber Duck approached the resolution rate of Opus 4.6 running alone. - Benefits were strongest for problems involving at least three files and 70 or more steps. - Sonnet plus Rubber Duck scored: - 3.8% above the Sonnet baseline on difficult tasks - 4.8% higher on the hardest tasks across three trials - Examples included detecting: - A scheduler that would start and immediately exit - A loop overwriting one dictionary key and dropping Solr facet categories - Cross-file Redis references that would silently break email confirmation flows ## When Reviews Happen Rubber Duck can be invoked automatically, reactively, or on request: - After a plan is drafted, to prevent flawed decisions from spreading. - After complex implementation work, to identify edge cases. - After tests are written but before they run, to expose coverage gaps or weak assertions. - When the primary agent is stuck or repeating an unproductive loop. - Any time the user asks Copilot to critique its work. Copilot incorporates the feedback and explains what changed. Reviews are intentionally infrequent and targeted at checkpoints where they provide the most value. ## Availability and Use Cases - Rubber Duck is available in Copilot CLI’s experimental mode through `/experimental`. - It works with Claude Opus, Sonnet, and Haiku as orchestrator models, provided the user has GPT-5.4 access. - It is especially suited to: - Complex refactors and architectural changes - High-stakes coding tasks - Test coverage review - Getting a second opinion before committing to a plan Rubber Duck is a practical way to reduce model-specific blind spots by combining different AI families. Developers can enable it experimentally in Copilot CLI and use automatic or on-demand critiques for difficult work.

AI Agents Code Review Github Copilot Cli Claude+3

githubMar 31, 20263 min readCurated summary

Agent-driven development in Copilot Applied Science

The post describes how Tyler McGoffin used GitHub Copilot to automate the intellectual work of analyzing coding-agent evaluation trajectories. This led to `eval-agents`, a tool designed to let researchers create, share, and run specialized agents. By making coding agents the primary contributors, the team rapidly added 11 agents, four skills, and workflow support while learning new approaches to prompting, architecture, and collaboration. ## The Motivation: Automating Evaluation Analysis - McGoffin analyzes coding-agent performance using benchmarks such as TerminalBench2 and SWEBench-Pro. - Each benchmark task produces a trajectory: a large JSON record of the agent’s thoughts and actions. - Reviewing hundreds or thousands of trajectories can involve hundreds of thousands of lines of data. - Copilot initially helped identify patterns, reducing the amount of material requiring manual inspection from hundreds of thousands of lines to a few hundred. - The repetitive nature of this process inspired `eval-agents`, which automates parts of the analysis itself. ## Project Goals The project was designed around three objectives: - Make agents easy for others to share and use. - Make authoring new agents straightforward. - Make coding agents the primary mechanism for contributing to the project. The third goal had the greatest architectural impact. Using Copilot to build the tool also made the repository easier for teammates to understand, extend, and collaborate on. ## An Agent-First Development Setup McGoffin’s development environment consisted of: - Copilot CLI as the coding agent. - Claude Opus 4.6 as the model. - VS Code as the IDE. - The Copilot SDK for creating agents, registering tools and skills, and accessing existing MCP servers. This setup allowed the project to reuse Copilot’s existing agent infrastructure instead of implementing those capabilities from scratch. ## Prompting Strategies - Agents perform best when treated like capable engineers rather than simple code generators. - Effective prompts are conversational, detailed, and explicit about assumptions. - Planning mode should be used before implementation mode, especially for complex tasks. - McGoffin used stream-of-consciousness descriptions to explain problems and collaborate with Copilot on possible solutions. - For example, a discussion about preventing agents from weakening regression tests led to protected test areas and human-controlled contract-test-like guardrails. - The broader lesson is that agents benefit from many of the same practices as human engineers: context, dialogue, planning, and clear constraints. ## Architectural Strategies An agent-first codebase makes maintainability work especially valuable: - Refactoring names and file structures improves the repository’s understandability. - Documentation gives agents the context needed to implement features consistently. - Additional tests expose and prevent recurring mistakes. - Removing dead code helps keep agents from copying outdated or irrelevant patterns. - Work that was traditionally postponed—cleanup, documentation, and test improvements—becomes foundational when agents are responsible for much of the implementation. ## Rapid Team Collaboration Applying these principles enabled substantial development in a short period: - Five people contributed to the project for the first time. - The team created 11 agents and four skills. - They introduced eval-agent workflows for structured streams of scientific reasoning. - In under three days, the changes amounted to approximately 28,858 added and 2,884 removed lines across 345 files. ## Practical Recommendation Teams adopting agent-driven development should invest first in clear architecture, documentation, tests, and conversational planning practices. Agents become substantially more effective when the repository provides strong context and guardrails, allowing developers to focus less on repetitive implementation and more on directing, reviewing, and improving the overall system.

AI Agents Model Context Protocol Github Copilot Copilot Cli+2