Cost showed $0 for every streamed request: the X-Cortiq-Cost-USD header is
written before generation, when the cost cannot be known. The gateway now
ships the final number in a closing SSE event (`cortiq_cost_usd`, alongside
the token estimate when the provider sent no usage), and the console prefers
it over the header. Money renders with 4 decimals, the last two in a smaller
font — $0.0006 stays readable instead of collapsing to $0.
Timeouts, both ends: the router default was 800 ms, silently degrading every
prompt the hosted router escalates to its oracle (~10 s) — now 15 s. Provider
clients had NO timeout at all (a dead OpenRouter host hung requests forever) —
now a 10 s connect timeout, with no overall cap so long generations and
streams are never cut.
Benchmark history: per-run × and a Clear-all button. Logs view: a clear
button backed by DELETE /admin/api/logs.
The benchmark now leads with the product's actual feature. Two explicit modes:
🧭 Smart routing (default) — every probe goes through cortiq-auto; each row
shows the router's task label judged against the expected one (✓/✗ with
"expected: …"), the complexity tier, the chosen model with what that
request cost, and a task-rule badge when a routing rule fired. KPIs:
answer quality, classification accuracy, TTFT, tok/s and money spent,
plus a "requests went to" distribution with per-model cost.
🎯 Individual models — the old head-to-head comparison, now as the clearly
separate option (model checkboxes appear only here).
The suite covers all 8 taxonomy labels (added summarization, creative-writing
and chitchat probes; expected labels annotated per test). Results are LIVE:
the table grows a row after every test (with a small fade-in), the progress
card shows the prompt being asked, and a 🏁 verdict toast lands at the end
with the score and the total spend.
12 new strings in all 7 languages.
The old standalone benchmark.html (Russian-only, Tailwind+Chart.js from CDNs,
never shipped in the crate) becomes a proper sidebar view. Same scoring ideas,
reimplemented: keyword checks with |-synonyms, think-tag stripping, tail-loop
detection, pass/partial/fail — plus what the old page never had: runs go
through the LIVE pipeline (pinned per model, streamed), so every test also
measures tok/s and time-to-first-token with the gateway's real token counts.
A built-in suite of 12 language-neutral probes (math, logic, code, QA,
instruction-following, extraction, translation, JSON formatting); the ⚡ quick
mode runs the 6 short ones — about a minute on a local model. Results: score
bars per model sorted best-first, a per-test grid (click a cell for the full
answer), previous runs and custom tests kept in the browser.
22 new strings in all 7 languages.
infosave2007