How to Measure AI-Assisted Pull Requests

Measure evidence, not authorship

An AI-assisted pull request is a work item with evidence that one or more AI coding sessions contributed to its changes. That is different from saying the AI authored the pull request, that every changed line came from a model, or that the result is good. A useful measurement system keeps those questions separate: assistance, estimated contribution, evidence confidence, and software outcome are distinct fields.

The measurement problem is a join across two imperfect timelines. Provider hooks observe sessions, repositories, file writes, and generated diffs on the developer machine. Git providers expose pull requests, branches, commits, files, additions, deletions, and authors. The system must connect them without uploading source code and without pretending that timing alone proves causation.

Step 1: identify candidate sessions

Start with direct evidence captured during the AI session. A provider may expose the working branch, a commit SHA, or a command that creates a pull request. When the pull request later contains that branch or commit, it can carry the matching session identifier forward. Direct commit evidence is the strongest candidate link because it connects an observed session to an immutable Git object. Branch evidence is useful but weaker because branches move and names can be reused.

Direct evidence is not always available. The v1 approach also considers sessions in the same managed repository during a bounded time window, from seven days before the pull request was created through merge time or the current time for an open pull request. This time-window fallback improves coverage but receives low confidence until file evidence also matches. It must never be presented as proof that every session in the window contributed.

Step 2: match changed files privately

A candidate session becomes useful only when its generated-change evidence matches a file in the pull request. MainLayer represents a repository file with a one-way identifier derived from the repository's normalized remote fingerprint and the relative path. The same calculation runs for endpoint events and Git-provider file metadata. This permits equality matching without sending the relative path or file contents in the telemetry event.

A generated-diff event contributes evidence when its file identifier equals a pull-request file identifier and the event reports added lines greater than zero. For each file, the estimate sums added lines from matching events and caps the value at that file's additions. The cap prevents the measured AI lines from exceeding the pull request's observable added lines when a file is edited repeatedly during several sessions.

Where hunk-level matching fits

Hunk-level matching is the more precise form of the same idea. The endpoint computes a fingerprint for each generated diff hunk. The pull-request side computes fingerprints for its patch hunks. Equal fingerprints provide evidence that a particular generated change survived into the pull request, even when the file also contains human edits. The comparison uses fingerprints rather than raw patch text.

That distinction should remain visible in the product. A file-level result means an AI session generated additions for the same file that appears in the pull request. It does not prove that the exact generated lines survived unchanged. A future hunk-level result can narrow the claim to matching change regions, but even that does not establish semantic authorship or quality.

Step 3: calculate a contribution estimate

For every pull-request file, calculate estimated AI lines as the smaller of the summed matching generated additions and the file's additions. Add those values to obtain matched lines. Add all pull-request file additions to obtain total lines. Estimated AI contribution is matched lines divided by total lines. When total additions are zero, the percentage is null rather than zero because the denominator does not support a meaningful result.

ai_lines(file) = min(sum(matching generated additions), pull_request_file.additions)
matched_lines = sum(ai_lines(file))
total_lines = sum(pull_request_file.additions)
contribution_pct = matched_lines / total_lines

Keep the numerator and denominator next to the percentage. A display of 38 percent is less informative than 214 of 560 added lines, medium confidence, across two matching sessions. Counts let reviewers see scale and make null or small-denominator cases obvious. The provider list and session count explain whether the pull request involved one assistant or several.

Step 4: assign confidence from the link

Confidence describes the strength of the session-to-pull-request link, not confidence that each line is correct. The v1 model is high when at least one matching session is linked through a commit SHA, medium when the best link is a branch, low when candidates come only from the repository and time window, and none when no generated-change evidence matches. The system should expose this tier beside every estimate.

Organization summaries should preserve the evidence boundary. Low-confidence estimates may be explored, but they should not quietly inflate headline contribution metrics. A repository or daily rollup should retain matched and total lines so a reader can understand the aggregate. The same rule applies when a pull request is open and its files are still changing: mark the result as an estimate recomputed from current evidence.

What this approach can and cannot say

• It can say that supported session evidence and pull-request files matched under a documented algorithm.
• It can estimate the share of added lines associated with matching generated-diff events.
• It cannot prove intellectual authorship, developer effort saved, correctness, originality, or review quality.
• It does not attribute deletions in the v1 percentage because additions form the denominator.
• It may miss work from unsupported providers, disabled hooks, offline devices, rebases, squashes, copied patches, or generated code that was heavily rewritten.
• File-level matching may overstate survival when AI and human changes touch the same file, while strict matching may understate assistance that influenced a human rewrite.

The estimate also depends on repository identity. The endpoint normalizes a Git remote and compares its fingerprint with the managed inventory. Local-only repositories and unmatched remotes cannot be joined to a managed pull request with the same assurance. Path renames and provider-specific file metadata require consistent normalization. These are engineering constraints, not reasons to hide uncertainty.

Use the metric in an operating review

Review AI-assisted pull-request rate, matched lines, total lines, evidence mix, cycle time, review outcomes, and rework together. Segment by repository or team purpose before interpreting changes. A higher contribution estimate can mean effective adoption, a large generated migration, or an unstable workflow that produces extensive rewrites. A lower estimate can mean unsupported tools or deliberate use for planning and debugging rather than code generation.

Do not turn the number into an individual productivity score. It measures observable assistance in changed files, not the value of design, review, testing, incident response, mentoring, or decisions not to ship. The best use is to test questions: where does AI-generated code survive review, which repositories need stronger controls, and where do tool cost and engineering outcomes move together? Transparent evidence makes those questions answerable without pretending to have a perfect authorship detector.