BEP ResearchResearch tools
What the AI Octane Index measures, how, and what it cannot tell you. Written to be checked.
Pilot snapshot: 22 August 2026. 32 authored short, single-prompt tasks. 33 scored configurations · 3,168 benchmark attempts; 34 attempted configurations · 3,264 attempts including exclusions. Excluded: Muse Spark 1.2 (96 attempts), with no successful or billable response recorded; capability is unavailable. Full production agents were not evaluated. Rankings use saved acceptance grades (benchmark 0.3.0), not the answer-present diagnostic. Overall cost is a weighted mean of vertical cost-per-success rates. API costs exclude review, tools, integration and downstream errors. CSV · Snapshot JSON · Counts and headline calculation.
Checking for a completed, validated recovery measurement. No new score is implied by this status check.
Checking monthly refresh status. No monthly measurement has been published.
Two things, and only two: what one attempt at a task costs, and whether a grader accepted the result. Everything else on the site — total cost, cost per completed task, recommendations — is those two measurements combined with assumptions that are labelled as assumptions.
Scored scale: 33 configurations, 32 authored tasks, 5 verticals, 3,168 benchmark attempts. An attempt may fail before a grader runs, so attempts are not a count of saved grades.
Per attempt, from the provider's reported token usage and a dated price card. Input and output are priced separately, and cached reads, cache writes and reasoning tokens are each priced at their own rate where the provider bills one:
Where a provider does not bill a line separately, that rate falls back to the standard input rate (for cache) or output rate (for reasoning) — which is how such providers actually charge. Reasoning tokens are handled per provider convention: Anthropic bills extended thinking as output and it is reported as zero rather than counted twice; OpenAI reports reasoning inside completion tokens and it is subtracted; Google reports thoughts alongside candidates and both are counted.
For each task, cost per success is computed directly:
Total billed cost includes input tokens, output tokens, cached tokens where applicable, reasoning tokens where separately billed, failed attempts, and retries where benchmarked.
This keeps the cost of failed attempts in the numerator, which is the whole point — work you paid for and could not use is still work you paid for.
Aggregation uses declared weights at two further levels. Within a vertical, it is a weighted ratio of costs to successes; zero-success tasks keep their cost in the numerator. Overall, it is a weighted mean of the vertical rates:
The overall figure is not total pooled dollars divided by total pooled successes. A vertical without any successes has an unavailable cost-per-success rate and is excluded from that mean and its weight denominator; the unavailable value is not imputed. API-only costs exclude human review, tools, integration and downstream losses. Multiplying a benchmark rate by a workload volume is a scenario, not a measurement of completed production work.
Published acceptance and cost rankings use the saved
GradeResult.success decisions. An API failure without a successful grade
counts as a failed attempt. The deterministic graders in the current implementation
work as follows; they do not require the whole response to be only the answer.
| Grader | How it works | Tasks |
|---|---|---|
| Exact match | Deterministic. Compare the normalised response to the expected string; if it differs, compare the extracted final answer line. No model judge. | 16 |
| Structured output | Deterministic. Parse the response as JSON, or extract its last JSON object, then check it against the task's declared schema and success threshold. | 10 |
| Executed tests | Deterministic. The generated code runs in a sandboxed subprocess with network access blocked, against tests written before the model saw the task. | 6 |
| Blinded model judge | Not deterministic. Used only where an answer is genuinely open-ended. Judged tasks are excluded from every published run until a judge-agreement audit exists. | 0 |
Exact-match and JSON graders can extract a final answer line or JSON object before applying the task rubric. Extra explanation does not automatically fail a response; the extracted result must still satisfy the rubric. The 22 August 2026 pilot's exact-match and JSON grades agree with a read-only re-evaluation of its saved responses using the current implementation. The saved data carries rubric version identifiers but no separate grader implementation version; a benchmark version does not establish the exact grader code revision used historically.
The exported content_successes field is an answer-present
diagnostic: it also counts expected answer tokens found in rejected responses. A
response may mention an expected token and still conclude incorrectly. This field is
not corrected accuracy, is not an alternative correctness grade, and does not set
the cost ranking.
Pass rates carry Wilson score intervals rather than the normal approximation, because at these sample sizes the normal interval produces impossible bounds — at 12 of 12 it claims certainty, which no run of twelve can establish.
Evidence tiers depend on how many distinct tasks stand behind a figure, not how many API calls were made. Repeating one task ten times at temperature 0 returns near-identical text, so repetitions do not buy independent evidence:
| Tier | Requires |
|---|---|
| Strong Evidence | 60+ distinct tasks |
| Moderate Evidence | 30+ distinct tasks |
| Directional | 12+ distinct tasks |
| Early Signal | 3+ distinct tasks |
The display treats overlapping intervals and costs within 10% as a tie heuristic. Interval overlap does not establish that two configurations have equivalent quality, nor is this a formal paired equivalence test.
A refused, malformed, timed-out or wrong answer is a failure, and its cost stays in the total. Infrastructure faults — rate limits, server errors, timeouts — are retried and tagged separately from a wrong answer, which is never silently re-rolled.
The workload calculator defaults to assuming failures repeat. This is a scenario assumption, informed by repeated answers observed at temperature 0; the pilot does not establish independent retry success on a production workload. The alternative assumption — that attempts fail independently — is available, and the interface states which is in force.
The store preserves prompts, task and rubric versions, raw responses, provider usage, temperature and seed, latency, batch identifiers and pricing references. Resolved provider model versions are recorded where returned. Historical observations are append-only; re-scoring creates a new snapshot generation. The saved grader implementation version is a remaining provenance gap.
The public CSV and JSON contain the full per-task snapshot rows, including the excluded configuration. They do not include every raw prompt, response or grade; checking each historical grading decision therefore requires the underlying store.
Download CSV · Download JSON · Download publication metadata
Published prices are snapshot prices. The repository configures a daily 06:00 UTC catalogue refresh that appends dated cards. That configuration does not prove that a refresh ran, and it does not reprice the committed snapshots used by this site. Each result retains its recorded pricing version; it is not a current-price quote.
Benchmark dates are stated on the figures. The repository contains an optional scheduled benchmark workflow, but the date on this page is the saved snapshot's date. A configured schedule does not make a historical pilot current. Re-scoring saved observations and purchasing a new run are separate operations.
Specific circumstances in which these numbers should not drive a decision:
Every task here is one prompt, one response. If your workload is agentic, long context, multi-turn, or depends on tool use across steps, this basket does not describe it.
This measures observed success on a fixed set of benchmark tasks. It is not a prediction of behaviour on your inputs, and the gap between the two is unmeasured.
Where the evidence is marked limited, one changed answer moves the rate by tens of points. Use those panels to shortlist, not to choose.
Graders check whether an answer is right, not whether it is well written, appropriately hedged, or safe to send unedited.
All figures use list, on-demand rates. Committed-use or batch pricing changes the ranking, and the calculator's totals with it.
A benchmark result is evidence for a shortlist. Run your own representative sample before putting a model on legal, medical, financial or safety-critical work.
These are the reasons to discount what you have read. They are listed because a benchmark that hides them is not worth reading.
32 authored tasks across 5 verticals. A vertical with four tasks can move by 25 percentage points when one task verdict changes. Any figure drawn from fewer than 12 distinct tasks is marked as a limited sample.
Every task is one prompt and one response. This is a pilot, not a full production agent or workflow evaluation. It does not measure long context, tool use across many steps, or multi-turn conversation. A model that trails on this basket may lead on work that looks nothing like it, and the reverse is equally true.
The authored task basket and its answer keys have not had a documented independent audit. A single author's choices can affect the result; treat it as pilot evidence for further evaluation.
List, on-demand, standard tier, USD — not batch, committed-use or negotiated rates. Models reached through OpenRouter are priced at OpenRouter's rates, which include its margin, and may run on different hardware or quantisation than the same model bought direct.
Graders check whether an answer is correct, not whether it is well written, appropriately hedged, or safe to send to a customer unedited.
The published intervals describe resampling within the saved run. They do not quantify provider changes or variation between runs, so they are not a full account of uncertainty over time.
Tasks may already appear in training data, and that exposure differs by model. A public sample plus a rotating private holdout is required before these numbers can be trusted over time.