AI21’s 98% Code-Review Saving Is a Test to Reproduce

2026-10-23

Maestro’s take. Measure expensive fallbacks before swapping models. AI21’s experiment makes that a useful test. It does not establish what your production reviews will cost.

TL;DR

In its October 22 experiment, AI21 adapted GitLab Duo’s review workflow on 50 public pull requests. Its final setup used DeepSeek V4 Flash and trimmed context around changed code. Reported token cost fell from $0.589 to $0.0135 per review, about 98%.

The headline has limits. This was not GitLab production traffic. AI21 tuned and scored on the same 50 requests. Quality used model judges: F1 rose from 46.4% to 47.2%, while recall fell from 52.0% to 47.6%. The cheaper setup missed more of the benchmark’s reference findings.

Why Maestro users care

Oversized prompts triggered expensive Sonnet fallbacks. Trimming removed those fallbacks. Model routing and context size can matter together.

The dollar figures use published token prices, including a public inference rate for the self-hosted model. They do not establish actual hosting costs. Maestro would treat this as a promising experiment to reproduce, not a budget forecast.

One thing to try

Record cost by review stage, including retries and fallbacks. Then compare a cheaper model and smaller context on held-out pull requests. Track missed bugs and your full operating bill alongside the token bill.

Published October 23, 2026 · Brief and Maestro commentary · Tags: AI coding, costs, benchmarks

Original source: AI21’s experiment, methods and results.