AI Octane Index · BEP Research

What an accepted benchmark task costs.

This pilot compares API spending per output a grader accepted. Failures remain in the bill. The highest and lowest published overall weighted API costs span 272.8× per saved accepted output on this basket.

Pilot snapshot: 22 August 2026. 32 authored short, single-prompt tasks. 33 scored configurations · 3,168 benchmark attempts; 34 attempted configurations · 3,264 attempts including exclusions. Excluded: Muse Spark 1.2 (96 attempts), with no successful or billable response recorded; capability is unavailable. Full production agents were not evaluated. Rankings use saved acceptance grades (benchmark 0.3.0), not the answer-present diagnostic. Overall cost is a weighted mean of vertical cost-per-success rates. API costs exclude review, tools, integration and downstream errors. CSV · Snapshot JSON · Counts and headline calculation.

Tap or hover any dot to see which model it is.

Each dot is one scored configuration. Left is lower observed acceptance; up is higher weighted API cost per accepted output. Anything inside the shaded region is beaten on both counts at once by something else on this board.

Lowest cost above the observed threshold, by task group

In 3 of 5 task groups the lowest-cost candidate differs from the board's overall candidate. Each industry is a slice of the same basket, so these rest on 15–27 attempts against the 96 behind the board above — a steer, not a ranking.

What one of the 32 tasks looks like

Calculate the interest accrued.

LOAN TERMS:
  - Principal: $1,000,000
  - Rate: 6.00% per annum
  - Day-count convention: Actual/360
  - Accrual period: 45 days

Respond with the interest in dollars, rounded to two decimal places, as a number
only — no commas, no currency symbol.

Every attempted configuration

Rankings use the saved grader acceptance decisions, which allow final-answer extraction. The answer-present diagnostic also counts expected answer tokens found anywhere in rejected text; it does not prove correctness or identify formatting as the only failure. 7 of 33 configurations have higher diagnostic counts. For Llama 4 Maverick, this is 81/96 versus 56/96 accepted (+25). This diagnostic does not change the published costs.