What the AI Octane Index measures, how, and what it cannot tell you. Written to be checked.
Two things, and only two: what one attempt at a task costs, and whether a grader accepted the result. Everything else on the site — total cost, cost per completed task, recommendations — is those two measurements combined with assumptions that are labelled as assumptions.
Current scale: 20 models, 32 tasks, 5 verticals, 1,920 graded runs.
Per attempt, from the provider's reported token usage and a dated price card. Input and output are priced separately, and cached reads, cache writes and reasoning tokens are each priced at their own rate where the provider bills one:
Where a provider does not bill a line separately, that rate falls back to the standard input rate (for cache) or output rate (for reasoning) — which is how such providers actually charge. Reasoning tokens are handled per provider convention: Anthropic bills extended thinking as output and it is reported as zero rather than counted twice; OpenAI reports reasoning inside completion tokens and it is subtracted; Google reports thoughts alongside candidates and both are counted.
Cost per successful task is then computed directly, not inferred:
Total billed cost includes input tokens, output tokens, cached tokens where applicable, reasoning tokens where separately billed, failed attempts, and retries where benchmarked.
This keeps the cost of failed attempts in the numerator, which is the whole point — work you paid for and could not use is still work you paid for.
Three of the four graders are deterministic and involve no model judging another model.
| Grader | How it works | Tasks |
|---|---|---|
| Exact match | Deterministic. The extracted answer must equal the expected string after normalisation. No model judges another model. | 16 |
| Structured output | Deterministic. The response is parsed as JSON and checked field by field against a declared schema. | 10 |
| Executed tests | Deterministic. The generated code runs in a sandboxed subprocess with network access blocked, against tests written before the model saw the task. | 6 |
| Blinded model judge | Not deterministic. Used only where an answer is genuinely open-ended. Judged tasks are excluded from every published run until a judge-agreement audit exists. | 0 |
Graders extract the answer rather than requiring the whole response to be the answer. That distinction matters: an earlier version marked a model wrong for explaining its reasoning before answering correctly, which produced a confidently false headline.
Pass rates carry Wilson score intervals rather than the normal approximation, because at these sample sizes the normal interval produces impossible bounds — at 12 of 12 it claims certainty, which no run of twelve can establish.
Evidence tiers depend on how many distinct tasks stand behind a figure, not how many API calls were made. Repeating one task ten times at temperature 0 returns near-identical text, so repetitions do not buy independent evidence:
| Tier | Requires |
|---|---|
| Strong Evidence | 60+ distinct tasks |
| Moderate Evidence | 30+ distinct tasks |
| Directional | 12+ distinct tasks |
| Early Signal | 3+ distinct tasks |
Where two models' intervals overlap and their costs are within 10%, the site reports them as effectively tied rather than ranking noise.
A refused, malformed, timed-out or wrong answer is a failure, and its cost stays in the total. Infrastructure faults — rate limits, server errors, timeouts — are retried and tagged separately from a wrong answer, which is never silently re-rolled.
The workload calculator defaults to assuming failures repeat. That is what was measured: at temperature 0, most model-task pairs return byte-identical text on every repetition, so a retry usually buys the same wrong answer at twice the price. The optimistic assumption — that attempts fail independently — is available, and the interface always states which is in force.
Every attempt is stored with the model version the provider actually served, the prompt and task version, the grader and its version, temperature, seed, token counts, latency, the price card it was costed against, and the batch it belonged to. Snapshots are append-only: re-scoring adds a generation rather than overwriting one, so a published figure can always be traced to the attempts behind it. The full per-task results are downloadable as CSV and JSON from the main page.
Task prompts are deliberately not published. Printed prompts end up in training data and the benchmark decays.
Prices: daily. An automated job reads the provider catalogue at 06:00 UTC, appends a new dated card for anything that moved, repoints the model and republishes. Cards are never edited, so a historical result keeps the prices it was costed against. A price moving does not re-run the benchmark — it changes what the same measured token counts would cost today.
Benchmark runs: on demand, and stated on every figure. Results are re-run when models change, tasks are added, or a grader is corrected. Every published number carries the timestamp it was computed at and the benchmark version it belongs to. There is no fixed cadence yet, and claiming one would be a promise this has not earned.
Specific circumstances in which these numbers should not drive a decision:
Every task here is one prompt, one response. If your workload is agentic, long context, multi-turn, or depends on tool use across steps, this basket does not describe it.
This measures observed success on a fixed set of benchmark tasks. It is not a prediction of behaviour on your inputs, and the gap between the two is unmeasured.
Where the evidence is marked limited, one changed answer moves the rate by tens of points. Use those panels to shortlist, not to choose.
Graders check whether an answer is right, not whether it is well written, appropriately hedged, or safe to send unedited.
All figures use list, on-demand rates. Committed-use or batch pricing changes the ranking, and the calculator's totals with it.
A benchmark result is evidence for a shortlist. Run your own representative sample before putting a model on legal, medical, financial or safety-critical work.
These are the reasons to discount what you have read. They are listed because a benchmark that hides them is not worth reading.
32 tasks across 5 verticals. Several verticals rest on three or four tasks, where one flipped answer moves a rate by 25 points. Any figure drawn from fewer than 12 distinct tasks is marked as a limited sample.
Every task is one prompt and one response. Nothing here measures agentic work, long context, tool use across many steps, or multi-turn conversation. A model that trails on this basket may lead on work that looks nothing like it, and the reverse is equally true.
Two answer-key errors have already been found and corrected by re-reading. Until a second reviewer has audited the basket independently, treat the results as a well-instrumented pilot rather than citable research.
List, on-demand, standard tier, USD — not batch, committed-use or negotiated rates. Models reached through OpenRouter are priced at OpenRouter's rates, which include its margin, and may run on different hardware or quantisation than the same model bought direct.
Graders check whether an answer is correct, not whether it is well written, appropriately hedged, or safe to send to a customer unedited.
One model's success rate moved seven points between identical runs days apart. The published intervals are computed within a single run and cannot see that drift, so they understate true uncertainty.
Tasks may already appear in training data, and that exposure differs by model. A public sample plus a rotating private holdout is required before these numbers can be trusted over time.