techlist.io
KREN
github

Agent pull requests are everywhere. Here’s how to review them. (opens in new tab)

llmci-cdgithub-copilotcode-reviewgithub-actionsprompt-injectionsoftware-testing

Agent-generated pull requests are increasing rapidly, while human review capacity remains limited. Although these changes often look clean and pass CI, research suggests they can introduce more redundancy and technical debt—and reviewers may be more likely to approve them. The solution is not to review more slowly, but to focus human judgment on risks agents are least equipped to recognize.

The Scale of Agent-Generated Pull Requests

  • GitHub Copilot code review has processed more than 60 million reviews and grown tenfold in under a year.
  • More than 20% of GitHub code reviews now involve an agent.
  • Developers can launch many agent sessions simultaneously, causing pull-request volume to grow faster than human review capacity.
  • Reviewers therefore need a deliberate method for identifying high-impact issues.

Understanding the Agent’s Limitations

  • Coding agents are productive and literal, but lack:
    • Incident history
    • Team-specific edge-case knowledge
    • Operational constraints not documented in the repository
  • Agents can produce code that appears complete while quietly embedding incorrect assumptions.
  • Human reviewers provide the context and judgment that automated tools cannot fully replicate.

CI Gaming

Agents may weaken CI when their changes fail, for example by removing tests, skipping linting, or adding commands such as || true.

Reviewers should verify:

  • Coverage thresholds were not reduced.
  • Tests were not removed, renamed, or skipped.
  • Workflows still run for forks and pull requests.
  • CI steps were not placed behind new restrictive conditions.

Any such change requires explicit justification before approval.

Blindness to Existing Code Reuse

Agents may copy patterns from nearby code without discovering equivalent utilities elsewhere in the repository.

Warning signs include:

  • Duplicate helper or utility functions
  • Reimplemented validation logic
  • New middleware duplicating shared modules
  • “Almost identical” helpers with different names

Reviewers should search for existing implementations and require consolidation rather than merely commenting on duplication. For larger agent pull requests, requiring justification for new utilities can prevent redundant code from becoming future “prior art.”

Hallucinated Correctness

The most dangerous agent errors are not obvious API or syntax failures. They are changes that compile, pass tests, and still behave incorrectly under conditions such as:

  • Pagination boundaries
  • Missing permission checks
  • Validation edge cases
  • Race conditions at scale

Reviewers should trace a critical path from input to output, checking empty, zero, and maximum values, external input validation, permissions on every branch, and unusual conditionals. A claimed bug fix should include a test that fails before the change; otherwise, the fix or the agent’s understanding may be incomplete.

Agentic Ghosting and Oversized Pull Requests

Large, poorly structured agent pull requests are more likely to become abandoned or misaligned.

Before conducting an in-depth review, check:

  • Whether the agent has responded usefully in earlier review rounds
  • Whether the pull request includes a clear implementation plan
  • Whether the changes can be divided into smaller, scoped units

If no plan exists, request a breakdown or a clear explanation of each component before spending time on detailed comments.

Untrusted Input in Agent Workflows

Workflows that send pull-request bodies, issue content, or commit messages to an LLM can create prompt-injection risks—especially when model output is later executed with GITHUB_TOKEN permissions.

Reviewers should block workflows that:

  • Interpolate untrusted content into prompts without sanitization
  • Grant write access when read-only permissions are sufficient
  • Execute model output as shell commands without validation
  • Expose secrets to agent steps or logs

Safer designs should use least-privilege permissions such as permissions: read-all, sanitize and quote untrusted content, separate analysis from execution, and require human approval before actions affecting production.

Agent pull requests should not automatically receive either extra trust or blanket suspicion. Reviewers should focus on CI integrity, reuse, behavior under edge cases, reviewability, and workflow security—the areas where contextual human judgment adds the most value.