What 114,134 Pull Requests Reveal About Coding Agents and Tests

TL;DR: A new study of 114,134 merged pull requests across 391 Java repositories found that, within the same repository and for similarly sized changes, agent-supported pull requests were more likely to touch tests and add new ones, but less likely to modify or delete existing tests. Pooled test-change rates were nearly identical. This is not evidence that agents produce better suites. Our recommendation, not a finding measured by the study, is to review test-suite expansion and maintenance separately, then check whether each changed production risk has a new or updated oracle and an independent outcome check.

Coding agents are contributing tests that human teams will maintain. The harder question is whether each accepted pull request expands the suite for new risk and maintains the tests the codebase already depends on.

A new empirical manuscript, Testing with Coding Agents, compares agent-supported and primarily human-authored pull requests inside the same repositories. The study covers 114,134 merged pull requests from 391 Java repositories between January 2025 and April 2026.

Its most useful finding is not that agents “test more.” It is that agent-supported pull requests show a different mix of testing work:

• they change test code more often after repository and change size are considered;

• they add new tests more often;

• they modify and delete existing tests less often;

• they show a small detected increase in whether tests co-evolve with the production code changed in the same pull request;

• the relationship between agent participation and test quality remains uncertain.

For QA leads, engineering managers, and code reviewers, that creates a practical merge question. Did the pull request strengthen regression protection, or did it add fresh tests while leaving an existing oracle, fixture, or negative case behind?

This is source-backed analysis, not an executed experiment.

The Pooled Result Hides the Important Difference

At first glance, the two groups look almost identical. Pooled across all repositories, 60.1% of agent-supported pull requests and 59.4% of human-authored pull requests changed test code.

That pooled comparison mixes together projects with different testing cultures and levels of agent adoption. The paper instead compares pull requests within the same repository and controls for change size.

Under that model, the predicted share of pull requests changing test code was 55.0% for agent-supported work and 45.8% for human-authored work in a typical repository. The odds were 1.65 times higher for the agent-supported group.

This distinction matters because a pooled, cross-project headline can hide the local behavior a reviewer actually sees. The right question is not “Do agents add tests across GitHub?” It is “In this repository, for a change of this size, did the test suite respond to the changed risk?”

The result also varied substantially between repositories. In percentage-point terms, the difference was largest where testing was common but not systematic, and smaller near the low and high ends. The authors note that part of this pattern comes from the percentage scale. That makes repository context part of the finding, not noise to ignore.

Agent-Supported Pull Requests Add Tests More Often and Maintain Existing Tests Less Often

The study classified test work at the method and fixture level. It separated three jobs:

1. adding a test;

2. modifying an existing test;

3. deleting a test that no longer belongs.

Within the same repository and for similarly sized pull requests, agent-supported changes added tests in roughly 7 to 10 more out of every 100 pull requests. They modified existing tests in roughly 5 to 8 fewer and deleted tests in roughly 1 to 4 fewer. These results describe how often a pull request performed each task, not how many tests it added or how quickly a suite grew.

That pattern appeared whether the test was linked to the production change, accompanied unrelated production work, or appeared in a test-only pull request. The deletion gap was smaller for tests linked to production changes and appeared mainly in larger pull requests.

Adding a test is valuable, but it is not interchangeable with maintaining the suite. New production behavior can require both:

• a new test for a new branch, state, or permission boundary;

• an updated assertion for behavior that intentionally changed;

• a revised fixture because the valid starting state changed;

• removal of a test whose behavior no longer exists;

• preservation of a negative case that the new implementation can accidentally bypass.

A pull request can therefore look active in the test directory while still leaving the suite inconsistent with the product.

More Test Activity Is Not the Same as Better Test Quality

The paper does not show that coding agents create better or worse test suites overall.

The paper detected no difference in most structural characteristics of newly added tests. Agent-supported tests more often had an explicit oracle, with an odds ratio of 1.71, although predicted oracle rates were already near 100% in both groups and helper-method checks may affect the measurement. When agent-supported pull requests modified tests, they also added assertions more often.

Those are encouraging signals, but an assertion can still check the wrong consequence. For example, in a browser workflow, a test can prove that a button became enabled while missing that the server rejected the update. It can prove that a route changed while missing that the user’s permission was never applied. These are illustrative review risks, not browser findings measured by the paper.

The test-smell evidence was weaker and pulled in both directions. At the pull-request level, agent-supported additions showed a smaller net increase in smells, while modifications showed a larger one. Both estimates were borderline, and neither pattern held at the individual-method level. The authors correctly stop short of treating this as a clear quality advantage.

Coverage needs the same restraint. In the 95 repositories with analyzable pull requests, the researchers detected no difference in how well test pull requests covered the production changes they accompanied. That excludes production-only pull requests, and a non-significant result does not establish equivalence. In test-only pull requests, agent-supported tests covered 11.1 percentage points less of the classes they exercised, but that result rested on only 11 repositories containing both groups.

The useful conclusion is not “agents write weak tests.” It is “test count, test presence, and test quality are separate review decisions.”

Read the Study With Its Limits Visible

The study is a large, careful observational preprint that has not yet completed peer review.

Agent involvement was inferred from visible signals such as branch names, labels, accounts, authorship, and co-authorship. Agent use that left no detectable trace could remain in the comparison group, and the authors do not assume that this uncertainty biases the result in a favorable direction. Agent-supported pull requests were human-agent collaborations, so the study cannot separate an agent's behavior from what developers delegated, how they specified the task, or what reviewers changed before merge. Only merged pull requests were analyzed, so the results describe work projects accepted after review. The dataset is Java-only, the repositories were mostly small and young, and Codex, Copilot, and Claude Code accounted for nearly all agent-supported pull requests.

An independently published, machine-generated Pith review raises an additional concern. Agent-supported pull requests cluster later in the study period, including 40% in the final two months, but the central models do not include calendar time. The review argues that the comparison may partly reflect later, more mature repository conditions rather than agent participation alone. Pith labels its reviews as machine-made and fallible, so this is a critique to test, not an established correction to the paper.

That critique does not erase the paper’s contribution. It changes the confidence level. Reviewers should use the findings to sharpen their questions, not to declare a universal law about coding agents.

A Review Contract for Agent-Supported Test Changes

The paper's implications already recommend reviewing and updating existing tests linked to changed code and reporting which relevant tests were updated. The contract below builds on that recommendation, adds explicit risk ownership and an independent outcome check, and applies it to pull-request review. It is our practical extrapolation beyond the paper's Java evidence, not a measured result from the study.

Use this contract before accepting a test delta as regression protection.

1. Production Delta

State the behavior that changed in product terms.

• Which user, role, state, or integration is affected?

• Which branch or permission boundary is new?

• What behavior intentionally changed?

• What old behavior must remain stable?

A file diff is not enough. Reviewers need the risk boundary.

2. Test-Suite Expansion

List what the pull request added.

• new happy-path test;

• new negative case;

• new boundary value;

• new role or permission test;

• new state-transition test;

• new integration or persistence check.

Then ask whether each new test has an oracle that can fail for the consequential outcome, not only an intermediate UI signal.

3. Existing-Suite Maintenance

List the tests that should change because the product changed.

• Which existing assertions encode the old behavior?

• Which fixtures now start from an invalid state?

• Which helpers or mocks hide the new contract?

• Which obsolete tests should be removed?

• Which regression test should remain untouched to protect neighboring behavior?

An empty maintenance list needs an explanation when production behavior changed materially.

4. Coverage Gap

Record what remains untested and why.

Separate a deliberate exclusion from an accidental omission. Name the owner and follow-up date for every material gap. “The agent did not generate it” is not a risk decision.

5. Independent Outcome Check

Verify the result outside the agent’s own completion message.

For a database change, read the durable record. For an API change, inspect the actual response and side effect. For a browser workflow, reload or reopen the relevant state and confirm the expected result survives.

The test runner’s green status proves that its assertions passed. It does not prove that the assertions cover the right consequence.

A Worked Browser Example

Suppose an agent changes account settings so an administrator can downgrade a member from editor to viewer.

The production delta affects permissions, persisted state, and what the downgraded member can do after refresh.

A weak test delta adds one new test that clicks “Viewer” and checks for a success message. It never revisits the member’s permissions. It also leaves an older role-management test unchanged even though that test assumes only editors and administrators appear in the role menu.

The review contract exposes both failures:

• Expansion gap: no negative test proves the viewer cannot edit.

• Maintenance gap: the existing role-management test still assumes viewers cannot appear in the menu.

• Oracle gap: the success message is not the authoritative outcome.

• Independent check: reload as the downgraded member and verify the forbidden action is unavailable and rejected by the server.

The right repair may require one new test and one updated test. Counting only the new test would miss half the work.

How We Think Review Should Change

As coding agents contribute more production and test changes, we think review should shift from counting newly generated tests to checking whether the suite both expands for new risk and maintains existing oracles at the changed behavior boundary. This is our recommendation, not a forecast measured by the study.

For every agent-supported pull request, record the production delta, new tests, existing tests that should change, oracle changes, uncovered negative cases, the independent outcome check, and the owner of any remaining gap.

This creates a review object that a human or a later solving agent can inspect. It also helps keep new test activity from hiding a suite that is drifting away from the product.

Where Samelogic Fits

Samelogic does not analyze pull requests or generate test suites, and this paper provides no evidence about browser recording. In our latest fake-data QA run, the current product deliberately recorded a browser path, reopened it locally, and played it back, but immediate outcomes could be attached to the wrong action. Do not use the recording as merge evidence until that attribution defect is fixed and the same path is reverified.

The browser-agent regression test workflow shows the intended, waitlisted pilot pattern with synthetic data. Code review, the test suite, and an independent final-state check remain responsible for correctness.

Sources

• Testing with Coding Agents, Milanese et al., submitted October 26, 2026

• Pith review of Testing with Coding Agents, observed October 29, 2026