Evaluating performance and efficiency of the GitHub Copilot agentic harness across models and tasks (opens in new tab)
GitHub argues that an agentic harness—the system coordinating tools, context, and workflow—strongly influences how effectively an AI model solves software tasks. Its shared Copilot harness aims to match model-vendor harnesses in task completion while using fewer tokens. Benchmark results across four models suggest near-parity overall, with performance varying by model and task. ## Benchmarking Approach - GitHub evaluates the harness using public and internal benchmarks, plus real-world metrics and online experiments. - Comparisons hold the following constant: - The same underlying model - The same task - Context-window size - Reasoning effort - Tool selection - MCP servers - Copilot CLI is compared with: - Claude Code for Claude Sonnet 4.6 and Claude Opus 4.7 - Codex CLI for GPT-5.4 and GPT-5.5 - Benchmarks include: - **SWE-bench Verified:** 500 human-validated Python bug fixes - **SWE-bench Pro:** More complex, multi-step engineering tasks - **SkillsBench:** Skill usage and triggering - **TerminalBench:** Command-line workflows - **Win-Hill:** Software tasks in Windows containers ## Token Efficiency - Copilot’s harness generally completes tasks at rates comparable to model-vendor harnesses while consuming fewer tokens. - It performed better across the reported tests for Sonnet 4.6 and Opus 4.7. - For GPT-5.4 and GPT-5.5, Copilot performed better on nearly every benchmark. - The main exception was SWE-bench Verified, where Copilot was 7% worse with GPT-5.4 and 4% worse with GPT-5.5. ## Task Resolution - GitHub emphasizes that lower token use matters only when the agent successfully completes the work. - Overall task-resolution rates were considered on par with vendor harnesses. - Results varied by benchmark: - Copilot generally did better on SWE-bench Pro, especially with GPT models. - It outperformed vendor harnesses on Win-Hill or matched them for every model. - It performed better on TerminalBench with Sonnet and Opus, matched GPT-5.5, and trailed GPT-5.4. - SkillsBench results favored GPT models on Copilot but favored vendor harnesses for Claude models. - Differences are described as statistically comparable because model behavior is stochastic and run-to-run variation can explain many gaps. ## TerminalBench Variance Analysis - GitHub uses TerminalBench 2.0 to study both cost and completion rate over repeated runs. - The preferred outcome is higher resolution with lower cost. - The analysis illustrates that benchmark results should account for variance rather than relying on a single run. - GitHub presents Copilot as equal to or better than the vendor harnesses on this cost-versus-success comparison. The practical conclusion is that harness design is a reusable performance multiplier across Copilot products. GitHub’s results support using its shared harness when developers need broad model choice, efficient token usage, and comparable task-completion performance, while recognizing that the best harness can still vary by model and workload.