33 scored configurations, priced per accepted task.

Current recovery measurement

Checking for a completed, validated recovery measurement. No new score is implied by this status check.

The August 22 historical comparison below retains its original models and date.

A separate GPT-6 Luna pilot through OpenAI's direct API: 32 authored short tasks, three repetitions each. A single-model measurement cannot establish a cross-model ranking or a baseline-relative Octane score.

Current candidates and benchmark status

Monthly model check

Checking monthly refresh status. No monthly measurement has been published.

The historical pilot below remains dated 22 August 2026.

Claude Opus 5.5 and GPT-6.1 Sol · native APIs · 32 authored short tasks, three repetitions each. Weighted API price-card estimates per saved accepted output; no full production-agent evaluation.

Refresh status

Current model candidates and benchmark status · Candidate listings are not new benchmark scores. The historical pilot below keeps its original models and date.

Workload calculator

The benchmark measures what an attempt costs and how often it is accepted. Everything else that decides your bill is yours: volume, review time, what a wrong answer costs when it reaches a customer. Supply those and the ranking can change — which is the point.

Retries assume a failed task fails again, because that is what we observed in repeated temperature-zero answers. The calculator treats this as a scenario assumption; independent retry success on production work is unmeasured. Method and limits →

Which model won each kind of work

These cards reflect performance on the specific benchmark tasks listed under “What we test” — they are starting points for a model evaluation, not blanket recommendations for an entire industry. The badge is the observed success rate on those tasks, with the attempt counts beneath it. The pilot does not measure production reliability. Each card leads with the configuration with the highest observed acceptance above the 80% display threshold, then names a lower-cost option that also cleared it. Where no configuration reached the threshold, the card says so instead of recommending one.

Sticker price is not the price

A pricing page sells you a rate per million tokens. That rate is one of three terms, and it is the only one published. A model can advertise the lowest rate, write two or three times as many tokens to answer the same question, fail a share of the time — and finish as the most expensive option here.

Task cost per success = Cost of that task's attempts ÷ Accepted attempts

Each bar adds one thing the price page does not tell you. The faintest is the advertised rate on this exact workload, priced as if the model were as concise as the leanest one measured. Then the tokens it really wrote. Then the attempts you paid for and could not use. The final bar uses the declared benchmark weights: a weighted ratio of cost to successes within each vertical, then a weighted mean of those vertical rates. It represents this basket's mix, rather than an estimate for your own workload.

Overall

The leading model in each panel sets 100; every other grade is its share of that leader's successful work per dollar. Log axis — a real panel spans orders of magnitude. Whiskers are 95% clustered bootstrap intervals, resampled by vertical, then task, then repetition. Raw baseline-relative scores are in the CSV and JSON exports.

By vertical

Every panel shares one axis, so they are directly comparable. This is the cut that matters — cost-effectiveness is a property of a model on a kind of work, not of a model alone.

A hollow, struck-through dot marks a model that recorded below 80% observed success on that vertical. It is excluded from the recommendation, not judged unusable — cost per successful task already prices retries, but it assumes a failure is free to detect and that retrying eventually works. Neither held in these measurements: most model-task pairs returned byte-identical text on every repetition, so a failed attempt repeated rather than varied.

The frontier

Observed success against cost per successful task. Bottom-right is the good corner: high observed success at low cost. Position carries the reading, not colour — every point is directly labelled.

Methodology

Per task: cost per success = Cost of all attempts ÷ Accepted attempts

Total billed cost includes input tokens, output tokens, cached tokens where applicable, reasoning tokens where separately billed, failed attempts, and retries where benchmarked. Work you paid for and could not use stays in the numerator.

Within each vertical, the index uses a weighted ratio of task costs to task successes. The overall figure is a weighted mean of the available vertical cost-per-success rates; verticals with no successes are excluded from that mean and its weight denominator. It is not pooled total dollars divided by pooled successes.

Full methodology, grader definitions, confidence rules and limitations →

What we test

Every task behind these numbers, with how it is graded and what it weighs. Per-task snapshot rows are in the CSV and JSON exports. They do not contain all raw responses or grading records; a full grading audit also needs the underlying store.

Benchmark your own workload

This pilot basket contains 32 authored short, single-shot tasks. It does not evaluate a full production agent or workflow. The same machinery can run against representative samples of your own workload — extraction from your documents, your support tickets, your code — and returns model comparison, total-cost analysis under your review and error costs, observed success by task type, routing recommendations, and the caveats that apply.

Useful for model selection, vendor evaluation, cost optimisation, routing design and procurement support. Get in touch to benchmark your workload →

All figures

Task-level rows are omitted here for length. The CSV and JSON exports carry them in full, including the excluded configuration for audit.

Pricing versions vary by configuration. The table and downloads retain each result's recorded version; these are snapshot costs rather than current-price quotes.

Download CSV · Download JSON · Download publication metadata