githubAug 11, 2026

From coder to orchestrator: How agents shift the role of a developer (opens in new tab)

AI agents can generate impressive one-prompt demos, but reliable software delivery requires more than isolated outputs. Developers increasingly need to design workflows that define how code is proposed, tested, reviewed, and shipped. The article argues that this shifts developers from primarily writing code to orchestrating agents within controlled, repeatable systems. ## From One-Off Prompts to Reliable Workflows - A single prompt can quickly produce a demo, such as a simple game. - Production development requires repeatable delivery with: - Appropriate context - Validation and testing - Security controls - Review processes - Clear permissions and handoffs - GitHub Copilot is presented as a control plane for connecting these parts. ## An Agentic Development Flow - Familiar repository events can trigger agent work, including: - Adding a label to an issue - Running a scheduled workflow - Starting a GitHub Actions process - The agent’s changes are captured in a pull request. - Deterministic checks then validate the work through: - Linting - Tests - Security scans - Build verification - CODEOWNERS, required reviews, and branch protection rules control what can be merged. - Agents handle ambiguous, context-heavy tasks, while predictable automation provides the safety boundary. - Developers decide: - What agents can access - How tasks are scoped - Where workflows hand off - When human judgment is required ## GitHub’s Implementation Options - Copilot cloud agent workflows support event-driven automations. - Copilot CLI can run AI-powered steps inside GitHub Actions. - Model Context Protocol (MCP) can extend agents with additional tools and external context. - These options represent different stages of building an agent-enabled development workflow. ## Starting Small - Teams should begin with one bounded, low-risk workflow. - Suitable examples include: - Issue triage - Synchronizing documentation and tests - Routine maintenance updates - The recommended approach is to integrate Copilot into existing development infrastructure rather than redesigning everything at once. Developers should treat AI agents as components within an engineered delivery system, not as replacements for that system. Start with a limited workflow, surround agent output with automated checks and review controls, and gradually expand as the process proves reliable.

model-context-protocolai-agentgithub-copilotcode-review+4

githubJun 19, 2026

How we built an internal data analytics agent (opens in new tab)

Qubot is GitHub’s internal, Copilot-powered analytics agent, designed to make warehouse data accessible without requiring an analyst. Employees ask natural-language questions through Slack, VS Code, or the Copilot CLI, while Qubot uses curated context and selects between Kusto and Trino to produce answers. GitHub’s experience shows that structured context is central to improving both accuracy and response speed. ## The Problem Qubot Addresses - Product teams often struggle to identify the right data model, grain, filters, and query. - Limited analyst availability leaves many teams to investigate telemetry independently. - Qubot targets exploratory questions rather than replacing dashboards or formal reporting. - It enables employees to investigate unfamiliar datasets with minimal setup and maintenance. ## Multiple Ways to Ask Questions - **Slack:** Users ask questions in a shared channel, receive answers in threads, and collaborate on follow-up questions. - Each result is saved as a Markdown report in a pull request, allowing users to refine queries or reuse them in dashboards. - **VS Code and Copilot CLI:** Qubot is installed as a plugin and operates alongside other agents, skills, and tools. - Offering both zero-configuration and developer-focused interfaces increased adoption among different user groups. ## A Federated Context Layer Qubot’s context is tailored to GitHub’s warehouse layers: - **Bronze:** Product teams provide telemetry schemas and metadata for raw events. - **Silver:** Data teams maintain query examples, usage guidance, and required filters for conformed data. - **Gold:** Dataset owners contribute business rules and metric definitions. - ETL pipelines add derived metadata and other signals automatically. - Context is fetched at runtime through the GitHub MCP Server. A dedicated context agent standardizes contributions from different teams. It ingests Markdown documentation and repository references, then organizes and normalizes them into a structure that Qubot can use effectively. ## Evaluation Before Deployment Every change to Qubot’s instructions or context layer is tested through an offline evaluation framework. - Test cases contain prompts, expected answers, ground-truth SQL, domains, and difficulty levels. - Automated orchestration launches multiple parallel agent trials using `gh agent-task create`. - Results are saved as JSON and aggregated by completion rate, accuracy, and duration. - Teams compare configurations and detect regressions before releasing changes. ## Choosing Between Kusto and Trino Qubot connects to both query engines through MCP servers: - **Kusto** is used for fast exploration of recent event data. - **Trino** handles complex joins and deeper historical analysis. - Qubot defaults to Kusto and switches to Trino when the question requires more advanced or historical querying. - This hides query-engine selection from users. ## Adoption and Lessons Learned - Hundreds of GitHub employees have run thousands of queries through Qubot. - Routine questions in analytics support channels declined as teams gained more autonomy. - The agent helped employees who previously avoided the warehouse access data for decision-making. - GitHub found that well-structured, carefully maintained context improved accuracy and made Qubot approximately three times faster at finding the right answer. The main recommendation is to treat analytics context as a maintained engineering asset. A capable agent depends not only on model intelligence, but also on accurate metadata, documented business rules, evaluation-driven iteration, and interfaces that fit users’ existing workflows.

model-context-protocolai-agentslackgithub-copilot+4

githubMay 7, 2026

Improving token efficiency in GitHub Agentic Workflows (opens in new tab)

GitHub’s Agentic Workflows can quietly accumulate substantial token costs because they run automatically in CI. GitHub improved efficiency by instrumenting token usage, auditing workflows, pruning unused MCP tools, and replacing many MCP data-fetching calls with deterministic GitHub CLI commands. Early results show that reducing context and removing unnecessary LLM reasoning can save thousands of tokens per run, though measuring true efficiency requires accounting for model choice and workload quality. ## Logging Token Usage - GitHub runs hundreds of agentic workflows against real GitHub Actions limits. - Different agent frameworks produced incompatible usage logs, so GitHub used its API proxy to normalize data across Claude CLI, Copilot CLI, and Codex CLI. - Each workflow now emits a `token-usage.jsonl` artifact containing: - Input, output, cache-read, and cache-write tokens - Model and provider - Timestamps - One record per API call - These records make it possible to compare historical runs and identify recurring sources of waste. ## Automated Auditing and Optimization - A daily **Token Usage Auditor** aggregates recent usage by workflow and reports: - Significant increases in token consumption - The most expensive workflows - Anomalous runs, such as a workflow taking 18 LLM turns instead of its usual four - A daily **Token Optimizer** examines flagged workflows, their source YAML, and recent logs. - It creates GitHub Issues with concrete inefficiencies and recommended fixes. - The auditing workflows also consume tokens, creating a feedback loop in which their own costs are monitored. ## Removing Unused MCP Tools - MCP tool names and JSON schemas are typically included in every stateless LLM request. - A GitHub MCP server with roughly 40 tools can add 10–15 KB of schema to every turn. - If a workflow uses only two tools, the other 38 create repeated overhead without adding value. - GitHub compares configured tools with actual tool calls and recommends removing unused registrations. - In smoke tests, pruning tools reduced each call’s context by 8–12 KB and saved several thousand tokens per run without changing behavior. ## Replacing MCP Calls with GitHub CLI - GitHub found larger savings by replacing MCP calls for predictable data retrieval—such as pull request diffs, file contents, and review comments—with `gh` commands. - MCP calls require an additional reasoning cycle: the model chooses a tool, constructs arguments, and processes the response. - Commands such as `gh pr diff` make deterministic API requests without involving the LLM in the retrieval step. Two migration patterns were used: - **Pre-agentic downloads** - Workflow setup steps run `gh` commands before the agent starts. - Results such as diffs and changed-file lists are saved to workspace files. - The agent reads the files directly, eliminating MCP round trips. - **In-agent CLI proxy substitution** - When data must be selected dynamically, the agent runs commands such as `gh pr view --json`. - A transparent proxy routes CLI requests to GitHub’s API without exposing credentials. - This preserves the zero-secrets security model while avoiding MCP overhead. ## Measuring Efficiency - Lower token counts do not necessarily mean better workflows; a workflow may simply be doing less work. - Model selection also affects cost. Claude Haiku and Sonnet may use similar numbers of tokens, but Haiku is substantially cheaper. - GitHub therefore uses an **Effective Tokens (ET)** metric that weights usage by token type and model cost: ```text ET = m × (1.0 × I + 0.1 × C + 4.0 × O) ``` - `m` represents the model multiplier: Haiku `0.25×`, Sonnet `1.0×`, and Opus `5.0×`. - `I` is newly processed input, `C` is cache-read tokens, and `O` is output tokens. - Output tokens receive greater weight because they are typically the most expensive component. GitHub’s experience suggests that agentic workflow authors should measure usage continuously, remove tools that workflows do not actually use, and move routine API retrieval outside the LLM reasoning loop wherever possible.

model-context-protocolllmgithub-actionsyaml+3

githubApr 14, 2026

Hack the AI agent: Build agentic AI security skills with the GitHub Secure Code Game (opens in new tab)

Agentic AI tools can automate powerful tasks, but their autonomy creates new security risks, including prompt injection, tool misuse, memory poisoning, and compromised multi-agent workflows. GitHub’s Season 4 Secure Code Game teaches developers to recognize these threats by attacking and hardening ProdBot, a deliberately vulnerable terminal-based AI assistant. Its five levels progressively add capabilities—and corresponding attack surfaces—mirroring how real-world AI systems evolve. ## The Secure Code Game’s Evolution - The free, open-source, in-editor course teaches security by having players exploit and fix intentionally vulnerable code. - Earlier seasons covered: - General secure coding across JavaScript, Python, Go, and GitHub Actions. - LLM security, including malicious prompts and defensive techniques. - More than 10,000 developers from industry, academia, and open source have participated. - Season 4 shifts focus from AI that generates content to AI that independently browses, uses tools, calls APIs, and acts for users. ## Why Agentic AI Security Is Urgent - Agentic systems are moving rapidly from research projects into production environments. - The OWASP Top 10 for Agentic Applications identifies threats such as: - Goal hijacking - Tool misuse - Identity abuse - Memory poisoning - A Dark Reading poll found that 48% of cybersecurity professionals expect agentic AI to be the leading attack vector by the end of 2026. - Cisco reported that although 83% of organizations planned to deploy agentic AI, only 29% felt prepared to secure it. - The article argues that learning to think like an attacker is essential for closing this readiness gap. ## ProdBot: A Deliberately Vulnerable AI Assistant - ProdBot is a terminal-based productivity and coding assistant inspired by tools such as OpenClaw and GitHub Copilot CLI. - It can: - Convert natural-language requests into bash commands. - Browse a simulated web. - Connect to MCP servers. - Run organization-approved skills. - Store persistent memory. - Coordinate multiple agents. - Players’ objective is to use natural language to make ProdBot reveal the contents of `password.txt`. - No prior AI or coding experience is required; all interaction takes place through the CLI. ## Five Progressive Attack Surfaces - **Level 1: Shell execution** - ProdBot runs generated bash commands in a sandbox. - The challenge is to determine whether the sandbox can be escaped. - **Level 2: Web browsing** - ProdBot reads simulated news, finance, sports, and shopping sites. - Untrusted web content introduces risks such as instruction hijacking and prompt injection. - **Level 3: MCP integrations** - ProdBot gains access to external tool providers for stock quotes, browsing, and cloud backup. - Additional tools increase both functionality and opportunities for abuse. - **Level 4: Skills and memory** - Organization-approved plugins and persistent memory create layered trust relationships. - The level tests whether trusted skills and stored information are actually safe. - **Level 5: Multi-agent orchestration** - ProdBot combines six specialized agents, three MCP servers, three skills, and a simulated open-source project. - Claims that agents are sandboxed and data is pre-verified become assumptions to test rather than guarantees. ## Real-World Relevance - The game’s vulnerabilities reflect active security concerns in deployed autonomous AI systems rather than purely theoretical exercises. - The article cites CVE-2026-25253, known as “ClawBleed,” an OpenClaw vulnerability rated CVSS 8.8. - The flaw allowed attackers to steal authentication tokens through a malicious link and gain full control of an OpenClaw instance. - Season 4’s broader goal is to develop instincts for identifying similar weaknesses during architecture reviews, tool-integration audits, and production deployments. Developers working with AI agents should treat every new capability—shell access, browsing, plugins, memory, or collaboration—as a potential attack surface. Practicing these failure modes in a controlled environment like the Secure Code Game can help teams design safer agentic systems before deploying them.

model-context-protocolagentic-aiai-securityprompt-injection+2

githubMar 31, 2026

Agent-driven development in Copilot Applied Science (opens in new tab)

The post describes how Tyler McGoffin used GitHub Copilot to automate the intellectual work of analyzing coding-agent evaluation trajectories. This led to `eval-agents`, a tool designed to let researchers create, share, and run specialized agents. By making coding agents the primary contributors, the team rapidly added 11 agents, four skills, and workflow support while learning new approaches to prompting, architecture, and collaboration. ## The Motivation: Automating Evaluation Analysis - McGoffin analyzes coding-agent performance using benchmarks such as TerminalBench2 and SWEBench-Pro. - Each benchmark task produces a trajectory: a large JSON record of the agent’s thoughts and actions. - Reviewing hundreds or thousands of trajectories can involve hundreds of thousands of lines of data. - Copilot initially helped identify patterns, reducing the amount of material requiring manual inspection from hundreds of thousands of lines to a few hundred. - The repetitive nature of this process inspired `eval-agents`, which automates parts of the analysis itself. ## Project Goals The project was designed around three objectives: - Make agents easy for others to share and use. - Make authoring new agents straightforward. - Make coding agents the primary mechanism for contributing to the project. The third goal had the greatest architectural impact. Using Copilot to build the tool also made the repository easier for teammates to understand, extend, and collaborate on. ## An Agent-First Development Setup McGoffin’s development environment consisted of: - Copilot CLI as the coding agent. - Claude Opus 4.6 as the model. - VS Code as the IDE. - The Copilot SDK for creating agents, registering tools and skills, and accessing existing MCP servers. This setup allowed the project to reuse Copilot’s existing agent infrastructure instead of implementing those capabilities from scratch. ## Prompting Strategies - Agents perform best when treated like capable engineers rather than simple code generators. - Effective prompts are conversational, detailed, and explicit about assumptions. - Planning mode should be used before implementation mode, especially for complex tasks. - McGoffin used stream-of-consciousness descriptions to explain problems and collaborate with Copilot on possible solutions. - For example, a discussion about preventing agents from weakening regression tests led to protected test areas and human-controlled contract-test-like guardrails. - The broader lesson is that agents benefit from many of the same practices as human engineers: context, dialogue, planning, and clear constraints. ## Architectural Strategies An agent-first codebase makes maintainability work especially valuable: - Refactoring names and file structures improves the repository’s understandability. - Documentation gives agents the context needed to implement features consistently. - Additional tests expose and prevent recurring mistakes. - Removing dead code helps keep agents from copying outdated or irrelevant patterns. - Work that was traditionally postponed—cleanup, documentation, and test improvements—becomes foundational when agents are responsible for much of the implementation. ## Rapid Team Collaboration Applying these principles enabled substantial development in a short period: - Five people contributed to the project for the first time. - The team created 11 agents and four skills. - They introduced eval-agent workflows for structured streams of scientific reasoning. - In under three days, the changes amounted to approximately 28,858 added and 2,884 removed lines across 345 files. ## Practical Recommendation Teams adopting agent-driven development should invest first in clear architecture, documentation, tests, and conversational planning practices. Agents become substantially more effective when the repository provides strong context and guardrails, allowing developers to focus less on repetitive implementation and more on directing, reviewing, and improving the overall system.

model-context-protocolai-agentgithub-copilotcopilot-cli+2

githubMar 10, 2026

The era of “AI as text” is over. Execution is the new interface. (opens in new tab)

The post argues that AI is moving beyond text-based question-and-answer interactions toward embedded execution. The GitHub Copilot SDK lets applications use Copilot’s planning, tool use, file modification, command execution, and error recovery capabilities directly. This enables teams to build adaptable AI workflows without creating their own orchestration infrastructure. ## Delegating Multi-Step Work to Agents - Applications can express intent and constraints instead of hard-coding every workflow step. - For a task such as “Prepare this repository for release,” an agent can: - Explore the repository - Plan the necessary work - Modify files - Run commands - Recover and adapt when failures occur - This approach is more flexible than scripts, which become brittle when workflows depend on changing context or unexpected errors. - Teams can use agentic execution while maintaining defined boundaries and observability. ## Using Structured Runtime Context - Relying on prompts to contain system logic makes workflows difficult to test, maintain, and evolve. - The Copilot SDK supports structured, composable context through: - Domain-specific tools and agent skills - Model Context Protocol (MCP) - Runtime retrieval of relevant data - Agents can directly access systems such as: - Service ownership records - Historical decisions - Dependency graphs - Internal APIs - Permissioned tools and real-time data provide more reliable grounding than embedding organizational knowledge in prompts. ## Embedding Agents Beyond the IDE - Agentic capabilities can be integrated into: - Desktop applications - Internal operational tools - Background services - SaaS products - Event-driven systems - Applications can invoke Copilot in response to events such as file changes, deployments, or user actions. - Execution happens within the product itself rather than in a separate IDE or terminal interface. - This turns AI from an auxiliary developer tool into application infrastructure available wherever the software operates. ## Execution as a New Interface - Agentic workflows are programmable planning-and-execution loops that: - Integrate with real systems - Operate under constraints - Adapt during runtime - Use tools to complete tasks - The Copilot SDK provides this execution layer so teams can focus on defining outcomes instead of rebuilding orchestration systems. The practical recommendation is to treat AI as an executable application capability rather than merely a text interface. Teams can start by identifying multi-step workflows or event-driven tasks where structured tools, runtime context, and adaptive execution would provide more value than fixed scripts.

model-context-protocolai-agentgithub-copilotagentic-workflows+2

githubMar 9, 2026

Under the hood: Security architecture of GitHub Agentic Workflows (opens in new tab)

GitHub Agentic Workflows are designed to bring autonomous agents into CI/CD without giving them unrestricted access to repositories, secrets, or the internet. Because agents can be prompt-injected and behave unpredictably, GitHub treats them as untrusted components and compiles workflows into constrained GitHub Actions. The architecture relies on layered isolation, controlled communication, staged writes, and comprehensive auditing. ## Threat Model - Agents reason over repository state and act autonomously, so they cannot be trusted by default. - GitHub Actions normally place components in one permissive trust domain with broad access to: - Repository contents - Authentication secrets - MCP servers - Arbitrary network destinations - A malicious webpage, issue, or repository file could prompt an agent to: - Read credentials from files, environment variables, logs, or `/proc` - Upload secrets externally - Embed secrets in issues, pull requests, or comments - Make unwanted repository changes - Strict mode follows four principles: - Defense in depth - Never trust agents with secrets - Stage and vet writes - Log everything ## Layered Security Architecture GitHub Agentic Workflows use three complementary layers: - **Substrate layer** - Runs on a GitHub Actions runner VM. - Uses trusted containers, Docker isolation, network controls, and kernel-enforced boundaries. - Separates components and mediates privileged operations and system calls. - Is intended to contain damage even if an untrusted component is compromised. - **Configuration layer** - Defines which components run and how they connect. - Controls communication channels, privileges, firewall policies, Docker images, and MCP configuration. - Determines which tokens are loaded into which containers. - Converts declarative workflow configuration into a secure runtime structure. - **Planning layer** - Controls which components are active and how data moves between them over time. - Creates staged workflows with explicit data exchanges. - Uses the Safe Outputs subsystem to govern potentially dangerous operations. ## Keeping Secrets Away from Agents - In ordinary GitHub Actions, secrets may be visible through environment variables and configuration files across the shared runner trust domain. - This creates a major prompt-injection risk: an agent with shell access could discover credentials and exfiltrate them. - Agentic Workflows instead place the agent in a dedicated container with: - Firewalled internet access - MCP access through a trusted gateway - LLM communication through an API proxy - A private network connects the agent only to approved services. - The trusted MCP gateway launches MCP servers and exclusively handles MCP authentication material. - LLM authentication tokens are kept in the isolated API proxy rather than exposed directly inside the agent container. ## Controlled Execution and Writes - Open-ended workflow authoring is separated from governed execution. - Workflows are compiled into GitHub Actions with explicit constraints covering: - Permissions - Outputs - Network access - Auditability - The planning and Safe Outputs systems are intended to mediate GitHub write operations and apply controls such as call filtering, volume limits, secret removal, and moderation. GitHub’s approach is to treat agents as untrusted CI/CD components rather than granting them normal workflow privileges. Organizations adopting agentic automation should isolate agents, broker access to tools and credentials, restrict network connectivity, stage all writes for review, and maintain detailed logs.

model-context-protocolci-cdgithub-actionsdocker+3

githubFeb 24, 2026

Multi-agent workflows often fail. Here’s how to engineer ones that don’t. (opens in new tab)

Multi-agent workflows often fail because agents make implicit assumptions about state, ordering, and intended actions. The post argues that these systems should be engineered like distributed software rather than treated as chat interfaces. Typed schemas, explicit action definitions, and MCP-enforced interfaces make agent behavior more predictable and failures easier to contain. ## Typed Schemas Prevent Data Drift - Natural-language exchanges and inconsistent JSON lead to changing field names, mismatched types, and ambiguous payloads. - Typed interfaces define machine-checkable contracts, such as a `UserProfile` with fixed fields and allowed plan values. - Schema violations can fail fast, triggering retries, repairs, or escalation before invalid state spreads. - Debugging becomes contract-based instead of dependent on inspecting logs and guessing. ## Action Schemas Clarify Intent - Agents cannot reliably infer what “take action” means; they may assign, close, escalate, or do nothing. - Action schemas restrict outcomes to explicit, valid choices such as: - Requesting more information - Assigning an issue - Closing an issue as a duplicate - Taking no action - A discriminated union or similar structure ensures every agent returns one recognized action. - Invalid or ambiguous actions can be rejected, retried, or escalated. ## MCP Enforces Agent Interfaces - Schemas and action definitions are only conventions unless consistently enforced. - Model Context Protocol (MCP) provides explicit input and output schemas for tools and resources. - Calls are validated before execution, preventing agents from inventing fields, omitting required inputs, or drifting between interfaces. - MCP therefore acts as the enforcement layer for both data structure and intended behavior. Reliable multi-agent systems require explicit contracts at every boundary. Engineers should treat agents like code components: define their data and actions precisely, enforce interfaces with mechanisms such as MCP, and prevent invalid state from propagating.

model-context-protocolagentic-aigithub-copilotdistributed-systems+2