20 models, priced per finished task.

Workload calculator

The benchmark measures what an attempt costs and how often it is accepted. Everything else that decides your bill is yours: volume, review time, what a wrong answer costs when it reaches a customer. Supply those and the ranking can change — which is the point.

Retries assume a failed task fails again, because that is what we measured — asking a second time usually returns the same wrong answer at twice the price. Why that is, and when it is not true →

Which model won each kind of work

These cards reflect performance on the specific benchmark tasks listed under “What we test” — they are starting points for a model evaluation, not blanket recommendations for an entire industry. The badge is the observed success rate on those tasks, with the run counts beneath it; not a claim about production reliability. Each card leads with the highest-scoring model above the 80% threshold, then names a lower-cost option that also cleared it. Where no model reached the threshold, the card says so instead of recommending one.

Sticker price is not the price

A pricing page sells you a rate per million tokens. That rate is one of three terms, and it is the only one published. A model can advertise the lowest rate, write two or three times as many tokens to answer the same question, fail a share of the time — and finish as the most expensive option here.

Cost per Successful Task = Total Billed API Cost Across All Attempts ÷ Successful Outputs

Each bar adds one thing the price page does not tell you. The faintest is the advertised rate on this exact workload, priced as if the model were as concise as the leanest one measured. Then the tokens it really wrote. Then the attempts you paid for and could not use. Then your actual work mix, since the index weights every vertical equally instead of letting a model's weak vertical be diluted by its strong one. Only the faintest bar is visible when you pick a model.

Overall

The leading model in each panel sets 100; every other grade is its share of that leader's successful work per dollar. Log axis — a real panel spans orders of magnitude. Whiskers are 95% clustered bootstrap intervals, resampled by vertical, then task, then repetition. Raw baseline-relative scores are in the CSV and JSON exports.

By vertical

Every panel shares one axis, so they are directly comparable. This is the cut that matters — cost-effectiveness is a property of a model on a kind of work, not of a model alone.

A hollow, struck-through dot marks a model that recorded below 80% observed success on that vertical. It is excluded from the recommendation, not judged unusable — cost per successful task already prices retries, but it assumes a failure is free to detect and that retrying eventually works. Neither held in these measurements: most model-task pairs returned byte-identical text on every repetition, so a failed attempt repeated rather than varied.

The frontier

Observed success against cost per successful task. Bottom-right is the good corner: high observed success at low cost. Position carries the reading, not colour — every point is directly labelled.

Methodology

Cost per Successful Task = Total Billed API Cost Across All Attempts ÷ Successful Outputs

Total billed cost includes input tokens, output tokens, cached tokens where applicable, reasoning tokens where separately billed, failed attempts, and retries where benchmarked. Work you paid for and could not use stays in the numerator.

Full methodology, grader definitions, confidence rules and limitations →

What we test

Every task behind these numbers, with how it is graded and what it weighs. Full prompts are deliberately unpublished — printed prompts end up in training data and the benchmark decays — but per-task results are in the CSV and JSON exports, so any figure on this page traces to graded attempts.

Benchmark your own workload

This public basket is 23 short, single-shot tasks. Your work is not these tasks. The same machinery runs against representative samples of your own workload — extraction from your documents, your support tickets, your code — and returns model comparison, total-cost analysis under your review and error costs, observed success by task type, routing recommendations, and the caveats that apply.

Useful for model selection, vendor evaluation, cost optimisation, routing design and procurement support. Get in touch to benchmark your workload →

All figures

Task-level rows are omitted here for length. The CSV and JSON exports carry them in full.