Agent-driven development in Copilot Applied Science
The post describes how Tyler McGoffin used GitHub Copilot to automate the intellectual work of analyzing coding-agent evaluation trajectories. This led to `eval-agents`, a tool designed to let researchers create, share, and run specialized agents. By making coding agents the primary contributors, the team rapidly added 11 agents, four skills, and workflow support while learning new approaches to prompting, architecture, and collaboration. ## The Motivation: Automating Evaluation Analysis - McGoffin analyzes coding-agent performance using benchmarks such as TerminalBench2 and SWEBench-Pro. - Each benchmark task produces a trajectory: a large JSON record of the agent’s thoughts and actions. - Reviewing hundreds or thousands of trajectories can involve hundreds of thousands of lines of data. - Copilot initially helped identify patterns, reducing the amount of material requiring manual inspection from hundreds of thousands of lines to a few hundred. - The repetitive nature of this process inspired `eval-agents`, which automates parts of the analysis itself. ## Project Goals The project was designed around three objectives: - Make agents easy for others to share and use. - Make authoring new agents straightforward. - Make coding agents the primary mechanism for contributing to the project. The third goal had the greatest architectural impact. Using Copilot to build the tool also made the repository easier for teammates to understand, extend, and collaborate on. ## An Agent-First Development Setup McGoffin’s development environment consisted of: - Copilot CLI as the coding agent. - Claude Opus 4.6 as the model. - VS Code as the IDE. - The Copilot SDK for creating agents, registering tools and skills, and accessing existing MCP servers. This setup allowed the project to reuse Copilot’s existing agent infrastructure instead of implementing those capabilities from scratch. ## Prompting Strategies - Agents perform best when treated like capable engineers rather than simple code generators. - Effective prompts are conversational, detailed, and explicit about assumptions. - Planning mode should be used before implementation mode, especially for complex tasks. - McGoffin used stream-of-consciousness descriptions to explain problems and collaborate with Copilot on possible solutions. - For example, a discussion about preventing agents from weakening regression tests led to protected test areas and human-controlled contract-test-like guardrails. - The broader lesson is that agents benefit from many of the same practices as human engineers: context, dialogue, planning, and clear constraints. ## Architectural Strategies An agent-first codebase makes maintainability work especially valuable: - Refactoring names and file structures improves the repository’s understandability. - Documentation gives agents the context needed to implement features consistently. - Additional tests expose and prevent recurring mistakes. - Removing dead code helps keep agents from copying outdated or irrelevant patterns. - Work that was traditionally postponed—cleanup, documentation, and test improvements—becomes foundational when agents are responsible for much of the implementation. ## Rapid Team Collaboration Applying these principles enabled substantial development in a short period: - Five people contributed to the project for the first time. - The team created 11 agents and four skills. - They introduced eval-agent workflows for structured streams of scientific reasoning. - In under three days, the changes amounted to approximately 28,858 added and 2,884 removed lines across 345 files. ## Practical Recommendation Teams adopting agent-driven development should invest first in clear architecture, documentation, tests, and conversational planning practices. Agents become substantially more effective when the repository provides strong context and guardrails, allowing developers to focus less on repetitive implementation and more on directing, reviewing, and improving the overall system.
Read original(opens in new tab)
