Whenever developers modify custom skills, register Model Context Protocol (MCP) servers, or tune prompt instructions (AGENTS.md), they confront a familiar software engineering anxiety: "Did this patch break existing agent workflows?"
While traditional software relies on unit testing and CI gates, probabilistic agentic workflows have largely operated without standardized regression harnesses. GitHub's November 17 release of Copilot Evals (Public Preview) alongside the local evaluation engine Vally formalizes automated regression testing for coding agents.
How the /eval Workflow Operates
GitHub Copilot Evals executes test suites declared in repository manifests (copilot-evals.json). Developers run evaluations directly from their terminal:
# Run regression tests for a specific skill
gh copilot eval --suite ./evals/git-commit-skill.json
# Perform A/B regression comparisons across branches
gh copilot eval --baseline main --target feature/new-mcp-tool
The structural innovation is the Vally local evaluation engine:
- Zero-API Local Replay: Vally utilizes recorded interaction trajectories and deterministic response stubs. It validates agent control flow locally without incurring external token costs or network latency.
- Deterministic Pass/Fail Assertion: Rather than vague semantic text similarity, Vally verifies exact tool dispatches, argument schema validity, and resulting filesystem mutations against expected outcomes.
- Statistical Regression Bounds: By running test scenarios across multiple seeds (5–10 iterations), Vally surfaces regressions in success probability, token usage spikes, and irregular reasoning loops.
Copilot Benchmarks: Pitting Harnesses Against Each Other
In tandem with Copilot Evals, GitHub introduced a "Harnesses" comparison view to its official Copilot Benchmarks portal.
Historically, benchmarks evaluated models in isolation. GitHub's new dashboard pins the identical foundation model and Medium reasoning tier, evaluating the execution harnesses of Copilot CLI, Claude Code, and Codex CLI side-by-side.
[Identical Foundation Model & Medium Reasoning Pinned]
│
┌───────────┼───────────┐
▼ ▼ ▼
[Copilot CLI] [Claude Code] [Codex CLI]
(Microsoft SDK) (Anthropic Harness) (Codex Pipeline)
│ │ │
└───────────┼───────────┘
▼
[Terminal-Bench-Science 0.1 Tasks]
- Pass Rate
- Mean Cost per Task
- 95% Confidence Intervals
Utilizing Terminal-Bench-Science 0.1, the data demonstrates that harness-level context management alone accounts for an 8–14 percentage point variance in task completion on identical model weights. Furthermore, variance distributions reveal distinct stability profiles across different orchestrators.
Implementing an Agent CI Pipeline
Engineering teams managing production agent skills and MCP servers should enforce structured evaluation gates:
| Lifecycle Stage | Verification Target | Mechanism |
|---|---|---|
| Pull Request | MCP tool schemas and skill prompt syntax | gh copilot eval --local (Vally engine) |
| Tool Updates | Baseline pass rate preservation | Branch A/B regression test against main |
| Pre-Release | Anomaly detection in token consumption | Hard cost and latency threshold boundaries |
Moving away from subjective intuition toward deterministic, test-driven evaluation marks a critical milestone for reliable agent development.