Copilot Automations: When Higher Acceptance Makes a Worse Reviewer

A review agent's comment acceptance rate rises from 50% to 100%. Has it improved, or has it stopped reporting difficult defects? That distinction matters when operational feedback becomes the input to the agent's next configuration.

GitHub's October 28 announcement describes Copilot Automations proposing improvements from outcomes such as accepted review comments, corrected issue labels, and merged pull requests. This October 29 article reviews official documentation and runs a synthetic calculation. We did not enable Automations in a repository.

Evaluate the worker from a separate loop

GitHub's example separates the automation reviewing pull requests from a weekly automation evaluating responses to those reviews. Changes to prompts, tools, or triggers go to a person for acceptance or rejection. This is a configuration feedback loop; the announcement does not establish that model weights are retrained.

Here is our architectural interpretation, not a reconstruction of GitHub's internal implementation:

Config v1 → reviews → developer responses and later outcomes
    ↑                                  ↓
Approved config v2 ← human review ← proposed improvement

Unlike revising one response during a retry, this changes the behavior of subsequent tasks. A poorly chosen metric can therefore influence more than one unsuccessful run.

Accepted comments omit the defects never reported

A developer may accept a style suggestion for different reasons than a security finding. A valid finding can also be deferred or moved into another issue. Acceptance is useful evidence of a response, rather than an independent verdict on correctness.

Save the following as feedback.mjs and run node feedback.mjs with Node.js 18 or newer. It requires no packages, keys, or network. The ten cases have predetermined defects and acceptance outcomes: synthetic teaching data, not product measurements.

const cases = Array.from({ length: 10 }, (_, i) => ({
  id: i + 1, defect: true, accepted: i < 5
}));
const policies = {
  broad: () => true,
  easyOnly: row => row.id <= 5
};
for (const [name, emits] of Object.entries(policies)) {
  const comments = cases.filter(emits);
  const accepted = comments.filter(row => row.accepted).length;
  const found = comments.filter(row => row.defect).length;
  const total = cases.filter(row => row.defect).length;
  const rate = comments.length ? `${100 * accepted / comments.length}%` : 'n/a';
  console.log(`${name}: accepted=${accepted}/${comments.length} (${rate}), defects=${found}/${total}`);
}

Local execution output:

broad: accepted=5/10 (50%), defects=10/10
easyOnly: accepted=5/5 (100%), defects=5/10

The second policy reports fewer findings and gets a better acceptance rate while missing half the defects. This does not show that Copilot chooses such a policy. It establishes that acceptance alone cannot distinguish these outcomes. Real evaluation also needs clean code, false positives, severity, and delayed observations, all absent from this fixture. With no comments, acceptance is undefined rather than 0%.

Give each proposal an evidence record

The following is our review protocol, not a description of the product's automatic scoring rules.

Start by comparing the existing and proposed configurations on historical cases with known outcomes. Then observe a narrow operational cohort. Keep some approval cases separate from the improvement process: repeatedly tuning against the same examples can specialize the configuration to that set. This is a proposed procedure; no product A/B experiment was run here.

For label triage, pair correction rates with misses by class. For performance pull requests, pair merges with regression checks and later reversions. Different automations need different evidence. A merged PR can be reverted later, so distinguish success at merge time from success in operation.

Configuration storage changes the review process

The official Automations documentation says configurations live separately from repository contents and are not versioned through Git commits and pull requests. Retain before-and-after settings and proposal rationale in your team's change record. Do not assume the versioning flow of Agentic Workflows' Markdown and lock files applies.

As checked on October 29, Automations require a paid Copilot plan, a private or internal repository, cloud-agent availability, and applicable organization policies. Runs consume AI Credits and separately billed Actions minutes. Daily, weekly, and monthly caps do not stop an already running task; consider a per-run limit too. Budget for the evaluator's executions as well as the worker's. We measured no bills.

Tool changes also deserve more scrutiny than wording changes. MCP tools can reach external systems allowed by their server credentials. If a proposal adds a write tool, assess its access scope separately from the evidence of better acceptance.

Before adopting outcome-driven improvement, establish what counts as success. Record denominators, misses, and observation windows before asking for weekly proposals. The practical contribution of loop engineering is the evidence supporting the next change, rather than the number of iterations.

The published paid course below connects to agent execution, verification, and feedback-loop design. It is not an Automations-specific course. The link is promotional.