Every stream used to carry two chunks with an empty choices array: usage,
which the upstream emits BEFORE the stop reason, and a cortiq_cost_usd chunk
the gateway added itself. OpenAI ships the usage chunk only when the request
sets stream_options: {"include_usage": true}, and chat UIs do not set it -
which is why the same tool, in the same Open WebUI, sees neither chunk from
Groq and two from us.
Honour the flag. With no stream_options a stream is now exactly
role -> content... -> tool_calls -> finish_reason -> [DONE], with no
choice-less chunks at all. When usage is requested it ships once, after the
stop reason, with cortiq_cost_usd folded into the same chunk rather than
trailing behind it.
Token accounting, cost and the X-Cortiq-* headers are untouched: those numbers
were never read back off the wire.
Verified: the default stream matches the reference shape byte for byte, the
opt-in stream matches OpenAI's, the official OpenAI SDK reassembles the
streamed tool call, and the full Open WebUI 0.11.0 tool loop still runs the
tool and answers from its result.
Qwen-family templates write a function call into the generated text as
<tool_call>{...}</tool_call>. Runtimes disagree on what to do with it: some
parse it into structured tool_calls AND still stream the raw markup as
content - the client then gets the same call twice, once as prose it must not
show and once as the real thing - while others never parse it at all, so a
perfectly good call arrives as text no tool loop will run.
Settle it in the gateway. A streaming-aware filter strips the markup from
content (handling a marker split across SSE chunks, leaving ordinary text
containing "<" alone, and handing an unterminated block back as text), and
when nothing structured came with it the markup is promoted into a real
tool_calls entry with a generated call_... id. Markup that is not a JSON
object with a name is ignored so a stray marker cannot become a bogus call.
Streamed tool_calls are also brought to the OpenAI shape: index on every
entry, id and type exactly once per call so continuation deltas carrying only
an argument fragment stay legal, and function.arguments always a string. A
reply that carries tool calls now always ends with finish_reason "tool_calls",
whatever the upstream called it. The same cleanup runs on the non-streaming
path, so every protocol adapter benefits.
Verified end to end: Open WebUI 0.11.0 -> gateway -> managed CMF/Qwen with
native function calling. The model returns get_weather({"city":"Paris"}), Open
WebUI executes the tool, feeds the result back, and the assistant answers from
it - no markup in content, finish_reason tool_calls, stream closed with [DONE].
router.allaigate.com now resolves and serves a valid certificate, so the
hosted routing API no longer has to be reached by bare IP with certificate
verification switched off. Make https://router.allaigate.com the default
[router].url with verify_tls = true.
Existing configs still pointing at 138.226.222.209 (with or without a scheme
or a trailing slash) are moved to the hostname on load and have verification
restored, then saved: it is the same machine, and this is the difference
between a verified connection and a blind one. A self-hosted router URL is
left untouched.
README, docs/ROUTER.md (both languages), config/gateway.example.toml and
bench/accuracy.py follow.
Verified live: the migration rewrites a legacy config on startup, admin
health reports the hostname reachable over verified TLS (389 ms), and real
requests route through it - "math"/medium 0.416 and "translation"/low 0.268
with route_source=router.
Verified against a real Open WebUI container: it attaches its ~26 built-in
tools (knowledge bases, notes, calendar, ...) to every single message, and
resolve() treated that offer as a hard capability requirement. On a pool of
tools-less local models a plain "hello" therefore failed with "No healthy
model available that satisfies required capabilities", and a pinned local
model was reported as "not available - not in the pool (deleted?)".
In the OpenAI contract `tools` is an offer; only tool_choice: "required" (or
a named function) makes the call impossible without them. Keep preferring a
tool-capable model, but fall back to serving the request with the tools
stripped - together with tool_choice/parallel_tool_calls, which a provider
rejects on their own. A pin whose only shortcoming is tool support is now
honoured; an unknown pin and tool_choice: "required" still fail loudly.
The downgrade is never silent: X-Cortiq-Tools-Dropped: true, plus
cortiq.tools_dropped in the echo block.
/v1/chat/completions answered ordinary client payloads with a bare HTTP 422
(a plain-text body no OpenAI client can parse), so Open WebUI showed neither
an answer nor an error — it just hung. Parse the body leniently instead:
content as string/null/content-part array, tools: null, stream: null, a
string reasoning_effort, max_completion_tokens, messages[].name and
tool_call_id, and a missing Content-Type. Unreadable bodies now come back as
an OpenAI error envelope with 400.
Responses are strict OpenAI JSON: no "tool_calls": null, finish_reason mapped
to the spec's closed set (cortiq serve's max_tokens -> length), never an empty
id, and `model` echoes the id the client requested — the model that actually
answered stays in cortiq.selected_model and X-Cortiq-Selected-Model.
Streaming is re-normalised rather than forwarded verbatim: a leading role
delta, message-shaped chunks converted to delta, one terminal finish_reason,
always a trailing data: [DONE], and an SSE error event when the upstream dies
mid-flight. Adds X-Accel-Buffering: no so a fronting proxy cannot buffer the
stream into a hang.
GET /v1/models now lists managed [[cmf.servers]] models with their live
running/healthy state, so clients can pin a local .cmf directly.
Also strips client bookkeeping keys (chat_id, session_id, metadata, features,
...) from the upstream call and drops the leftover [to_wire] debug prints.
default_config_path treated any directory holding Cargo.toml plus a
config/ folder as a gateway checkout, so running cortiq-gateway inside
an unrelated project died with "cannot read config config/gateway.toml"
instead of using the per-user home. Now ./config/gateway.toml counts
only when the file is actually readable, the repo-checkout heuristic
requires Cargo.toml to be cortiq-gateway's own, and everything else
falls through to $CORTIQ_GATEWAY_HOME / ~/.cortiq-gateway — typing
cortiq-gateway in any directory picks up the same configuration.
Settings gains a second service button next to restart: stop every managed
local model server and exit the gateway process (nothing is respawned; in
Docker the restart policy decides). The SPA replaces itself with a static
"gateway is off" screen since the API is gone. The restart button is
relabeled "Restart everything" to say what it always did — the gateway
re-execs itself and managed servers come back with it. Translated in all
seven languages.
cortiq serve >=0.5.62 advertises the loaded model's capabilities in
/healthz ({"capabilities":{"tools":true}} when the chat template has a
tools branch). The gateway now reads that handshake when the managed
server becomes healthy and treats detected tool support as equivalent
to declared caps in the candidate filter — so a low-tier chain made of
a local model no longer widens agent (tools) traffic to cloud models
once the local server can actually drive tools.
Also adds an optional manual "caps" override to [[cmf.servers]] for
older serve builds that do not advertise capabilities.
OpenCode (like every coding agent) sends tool definitions with each request.
The router rated the prompt low → the low-tier chain contained only the
local model → the tools filter emptied the chain → hard failure: "No
healthy model available that satisfies required capabilities", with the
agent stuck in a retry loop.
On an AUTO route whose filtered chain comes up empty, the pipeline now
widens to any pool model satisfying the required capabilities — preferring
the routing table's own medium→high order (an agentic request deserves a
capable model), then the rest of the pool — and logs the widening. Pinned
requests still fail loudly as before. Verified with OpenCode's exact
request shape: cortiq-auto + tools at tier low now answers from
deepseek in 1.2s instead of erroring.
The copyable text opened with a "# hint" line — but default interactive
zsh has interactivecomments off, so the line was parsed as a command, and
the "!" inside «(перезапишет!)» fired history expansion: pasting the
OpenCode snippet died with "zsh: event not found: )" before doing
anything. Hints now live in the UI above the snippet; the clipboard gets
commands only. Verified by piping the actual rendered snippet through an
interactive zsh: the config file lands intact.
The chip used var(--panel-2, #23252c) — a variable that doesn't exist with
a hard dark fallback, so in the light theme the install command was a dark
box with dark text. Same class of bug in the bands slider track and handle
labels (invented --panel-2/--fg vars). All moved to the real theme
variables (--panel/--border/--text) via a .cmd-chip class; verified by
computed styles in the light theme (text rgb(15,23,42) on white) and
screenshots of both views.
Next to the copy button: a key input and an «Insert key» button (Enter works
too). The pasted key substitutes into every harness/SDK snippet immediately
and survives tab and OS switches; a freshly created key pre-fills the field.
No more copying a snippet and then hand-editing the placeholder. 2 strings
×7 languages.
The harness auto-setup scripts were bash-only (heredocs, mkdir -p) — useless
on Windows. Every scripty snippet now exists in two flavors behind a
🐧🍎/🪟 switch (auto-detected from the browser): bash heredocs for
Linux/macOS and PowerShell here-strings ($env:, New-Item, @'..'@,
curl.exe with backtick continuations) for Windows. SDK snippets and GUI
instructions stay OS-independent.
And when detection says a harness is NOT installed on the gateway machine,
the status line now offers its one-line install command for the selected OS
(claude.ai install.sh / install.ps1, npm -g for Codex and OpenCode-on-
Windows, code --install-extension for Continue/Cline) with a copy button.
1 string ×7 languages.
The office-readiness audit found the foundation missing: configured API keys
were decorative. Any bearer — or none — passed every model endpoint, per-key
rate limits existed only in the form, and all real traffic was recorded as
"anonymous", leaving the per-key spend table permanently blank.
A middleware on the model surface (/v1/*, /route, /mcp) now enforces keys:
- no keys configured → open mode as before (the console already warns);
- keys exist → Bearer or x-api-key must match (401 invalid_api_key);
- per-key rate_per_min → fixed one-minute window, 429 with Retry-After;
- NEW per-key monthly_budget_usd → 429 insufficient_quota once the account's
month-to-date spend crosses the ceiling;
- the matched account rides the request into stats — the keys table's spend
columns finally see real traffic.
Month-to-date spend is tracked per account in stats and persisted in the
totals sidecar with a watermark (no double counting across restarts). The
key dialog gains a budget field; the table shows "budget $N/mo" in red once
exhausted. All verified live: 401 wrong/missing key, 200 + attribution with
the right key, 429 + Retry-After over the limit, 429 insufficient_quota on
the second request past a $0.000001 budget.
Also: the dashboard shows the OpenRouter credit balance (GET
/api/v1/credits via the stored key, cached 5 min, hidden without a key, red
under $5 — live: $16.18), the server drains connections on Ctrl-C instead
of dropping them, request bodies are capped at 32 MB, and docs/DEPLOY(.ru)
cover office deployment: systemd/launchd/Windows service, nginx/caddy TLS,
security checklist, backups. 10 strings ×7 languages.
Full audit of the semantic cache against a live instance (mock embedder,
identical requests): the mechanism works — miss at 3.1s, then hits at 2ms
with entries/hits/saved-$ counting. Two honest gaps fixed:
- the threshold round-tripped through f32 and rendered as
0.9200000166893005 in both the TOML and the settings input — the config
field is f64 now (writes exactly what you typed) and the input rounds
legacy values to 2 decimals;
- the card now shows proof of life (entries / hits / hit-rate / saved $)
and, when the cache is enabled with NO embedding model available, a
warning that it is silently doing nothing and how to fix it — previously
that state looked identical to a working cache.
Also audited in the same pass, no changes needed: Prometheus /metrics
(toggle on/off applies live, 404 when off), and all 7 protocol switches
(each gates its endpoint instantly both ways). 4 strings ×7 languages.
Benchmark (routing mode) and the dashboard's recent-requests list now print
the router's raw 0..1 score under/next to the tier badge — people can see
what different prompts actually weigh and tune the boundary sliders from
real data. Tooltip explains the link to the Routing sliders (7 languages).
Caught before commit: naming the field row.score collided with the answer-
correctness fraction from evaluate() and dragged the benchmark's quality KPI
down to the average complexity — renamed to cscore, KPI verified back at
100%. Dashboard tier badges also render translated names now, and the
Russian locale calls the dashboard «Рабочий стол».
At 35% opacity over the dark background the amber medium zone and the
light-red high zone read as the same color right of the medium handle.
Saturation stepped up per zone (45/55/65%) and high uses a deeper red —
green / amber / red now read at a glance.
low/medium/high leaked as raw English into the bands legend, the tier badges
on the Routing page, the pair complexity selects, the benchmark's tier column
and the Playground decision panel. New trTier() helper renders the human name
(ru: лёгкий/средний/сложный, de: leicht/mittel/schwer, fr/es/zh/tr likewise);
custom tier names pass through untouched, badge colors stay keyed by the raw
tier. Verified in the browser in Russian and German.
The tier boundaries lived on the router and were global for every customer
of a hosted router; per-gateway tuning meant hand-editing score ranges. The
Routing page now has one track with two draggable handles (the low|medium
and medium|high boundaries) over three colored zones: handles push each
other (low can't cross medium — it shoves it along, and vice versa), the
numbers ride on the handles and the legend updates live, so nobody types
ranges by hand. Unchecked = the router's own bands, untouched behavior.
Under the hood the gateway re-maps the router's CONTINUOUS complexity score
with the user's boundaries ([routing.bands] low/medium in gateway.toml,
validated 0 < low < medium < 1) before tier chains, escalation and stats —
verified live: with bands at 0.70/0.72 a score-0.27 request that used to be
medium routed low, and back after restoring the defaults.
2 new strings in all 7 languages.
Difficulty is judged in the background: the first-ever occurrence of a hard
prompt routes by prior (often low/medium), the oracle's judgement lands in
the cache a couple of seconds later. The route-judged retry fired instantly
with a FRESH template — which has the same first-encounter fate — so the
test could go red exactly when the system was working as designed. The retry
now re-asks the same prompt after 4s (steady-state answer from the cache),
and only then tries a fresh template.
route_source reported "+task-rule" whenever a rule existed for the task —
so a math prompt routed low by the ordinary tier chain still wore the badge
next to the local model, reading as "the rule picked this" when it hadn't
(the rule only defines math@high). matched now means the head of the chain
came from the rule's tier-specific or any-tier list; verified live: the same
low math request now reports route_source "router".
The single per-task «Complexity» pin was the wrong shape — what you actually
decide is WHICH model serves a task at WHICH complexity. A task rule is now a
per-tier map, edited in the console as rows of “model ▾ + complexity ▾” pairs
(plus ×/add) inside each task:
[routing.tasks.code]
low = ["cheap-coder"]
high = ["best-coder"]
any = ["generic-coder"] # leads at every tier after the tier pick
Chain order: the task's tier-specific list → (at high: the global high chain,
so escalation still beats any-tier pins) → the task's any-tier list → the
global tier chain → default as the last resort; deduplicated. Plain-array
rules keep meaning "any tier" unchanged, and the short-lived {models, tier}
form from the previous commit is normalized into the map on load.
Verified live: legacy config migrated visibly in the editor, pairs survive
save+reload, a math prompt at low ignored the math@high pair, per-tier leads
confirmed by unit tests (19 passing).
API keys page: the table now answers "which department spends what" — per-key
requests, tokens in/out and cost for a selectable window (24h/7d/30d) plus a
cumulative all-time cost column, joined from stats by the key's account.
The stats log stays bounded while the accounting survives: once stats.jsonl
exceeds max_file_mb (default 16, 0 disables), records older than the
retention window fold into cumulative per-account totals in a small sidecar
file (tmp+rename on both) and leave the log. alltime = folded + in-file, a
record is never counted twice, verified end-to-end on synthetic data
(1.2 MB → 301 KB, totals identical before/after compaction and restarts).
Routing: a task rule can now PIN the complexity tier —
`[routing.tasks.legal] models=[...] tier="high"` (old plain-array rules parse
unchanged). The pinned tier replaces the router's verdict before the
escalation check and reports itself as route_source "router+task-rule@high".
The Routing page grows a per-task «Complexity» selector (Auto/low/medium/
high); verified live: a code prompt with a pinned high went
router+task-rule@high → the expensive model.
13 new strings in all 7 languages; docs/ROUTING(.ru) updated.
The router deliberately scores familiar text as less novel, so every
benchmark run slightly burns the hard-prompt pool — with 3 templates per
test a variant would drift below the high band within a handful of runs
and flip the test red for no real reason.
Two-part fix: pools doubled to 6 interdisciplinary families each (science:
+ quantum error correction, soil-carbon MRV, cryo-EM; ops: + ED triage +
staffing, port berth/tides/rail, agri-drone fleets — all probed high against
the live router), and a route-judged test that lands non-high gets ONE retry
with a freshly generated template before failing. A single burnt wording no
longer fails the assertion; a genuinely unreachable high tier still does.
Recurring confusion all day: the SPA lives in an open tab until reload, so
after a gateway rebuild/update users kept seeing (and benchmarking with!) an
old interface — "where is the feature", "why are the test ids old". The meta
endpoint now carries the process start timestamp; the existing 20s health
poll compares it, and when the gateway has restarted the tab shows a fixed
banner: "The gateway was updated — this page is running an older interface"
with a one-click Reload. 2 strings in all 7 languages.
Routing semantics fix observed live: a task rule ("code → cheap model") was
pinning even high-complexity prompts to the cheap model — the tier system
never got a say. At tier=high the tier chain now leads and the task rule
demotes to fallback; low/medium keep task-rule-first. Unit-tested both ways.
The benchmark's hard probes are now GENERATED per run from two pools
(science: protein folding / telescope scheduling / materials discovery;
ops: epidemic logistics / grid markets / climate risk) with rotating domains
and numbers. Two findings against the live router drove this: ambiguity
carries the biggest weight in the complexity score, so interdisciplinary
prompts rate high reliably (0.52-0.66) while single-domain "hard" prompts
classify too confidently and sink to medium; and any fixed wording loses
novelty on every repeat. Their labels are null — straddling task labels is
precisely what makes them high, so classification is not judged.
Verified end-to-end: full run 100% answers, 13/13 classification, both hard
probes high → routed to the expensive model (one of them over a code task
rule), at $0.0018 each thanks to the tiny route-judge token budget.
Two "hard" probes (full suite only): a compiler-backend design and a security
protocol design, each dense enough for the router's complexity model to rate
them high. They are judged on the ROUTER's decision (judge: "route") — pass
means tier=high was reached and the request went to the big model; the answer
itself is not scored (open-ended essays aren't keyword-judgeable, and
reasoning models happily burn 3000+ thinking tokens on them — measured). With
max_tokens kept tiny the probes cost cents even on the priciest model, while
the cost column tells the whole story: $0.0148 for the high-tier request next
to $0.00005 for the easy ones.
Each run picks a random variant of the prompt: the router deliberately scores
familiar text as less novel, so a fixed wording would decay to medium over
repeated runs. In pinned-models mode route-judged tests are skipped.
Also: creative-writing budget 96→512 (deepseek was spending the whole cap
thinking and returning an empty poem), and when a model does burn its entire
budget inside <think> the table now says so ("budget spent on reasoning")
instead of the misleading "empty answer".
Server-side (cortiq-router prod, both replicas rolling-restarted): the high
band started at 0.66 while the complexity model tops out ~0.65 on the hardest
single prompts — high was unreachable. Band boundary moved to 0.50 based on
measured distribution (easy 0.25-0.36, medium 0.37-0.49, hard 0.50-0.65);
config backed up as config.prod.toml.bak-tiers.
Three false signals in benchmark results, all seen on a live run:
- Reasoning models were scored ❌ because the per-test max_tokens cap cut
them mid-<think> before the actual answer. Every run now gets +256 tokens
of headroom; non-thinking models stop at EOS as before, thinking ones get
to finish (math-mult on nanbeige: false 0/1 → honest 1/1).
- When the router degraded (timeout → fallback chain) the table showed
"✗ degraded" and counted it against classification accuracy. There was no
classification to judge — such rows now show a muted "⚠ router degraded"
with an explanatory tooltip and are excluded from the accuracy KPI.
- Costs below $0.0001 (cheap flash models) collapsed to "$0.0000". money
formatting now grows precision until two significant digits survive
($0.000035 instead of $0.0000).
2 new strings in all 7 languages.
Creating a key now asks only account, rate limit and optional custom value —
allow_models is sent empty (= all models). Per-key model restrictions stay
editable in Settings for whoever actually needs them.
The one-click model benchmark showed "error" even though cortiq bench ran
fine: the parser expected the JSON result on a single line, while newer
cortiq releases pretty-print it across many. Extraction now finds the last
balanced JSON object anywhere in the mixed CLI output (both formats work).
Also hardened tok/s selection: steady decode speed (decode_tok_s_steady)
is preferred over incl-prefill, and the loose "contains tok+per" fallback —
which would happily report allocs_per_token=769 as tok/s — now only matches
*tok_s* / *tokens_per* fields.
The "connect your app" card in the API-keys view now leads with auto-setup
scripts for popular coding harnesses, each pre-filled with this gateway's URL
and the freshly created key:
Claude Code — env exports (the gateway speaks Anthropic /v1/messages
natively; the protocol is now ON by default, and when a config has it
off the tab shows a warning with a one-click Enable button)
Codex — appends a [model_providers.cortiq] block to ~/.codex/config.toml
OpenCode — writes the provider into opencode.json
Continue — appends the model to ~/.continue/config.yaml
Cline — the exact values to paste into the extension settings
The gateway also detects which harnesses are already present on its machine
(binaries across PATH + common install dirs, config files under home) via
GET /admin/api/harnesses: installed tabs get a ✓, and the status line under
the snippet says "installed" or "already configured for this gateway" when
the harness config mentions cortiq. Works on Linux/macOS/Windows.
curl / python / javascript SDK snippets stay as before. 16 new strings in
all 7 languages.
Cost showed $0 for every streamed request: the X-Cortiq-Cost-USD header is
written before generation, when the cost cannot be known. The gateway now
ships the final number in a closing SSE event (`cortiq_cost_usd`, alongside
the token estimate when the provider sent no usage), and the console prefers
it over the header. Money renders with 4 decimals, the last two in a smaller
font — $0.0006 stays readable instead of collapsing to $0.
Timeouts, both ends: the router default was 800 ms, silently degrading every
prompt the hosted router escalates to its oracle (~10 s) — now 15 s. Provider
clients had NO timeout at all (a dead OpenRouter host hung requests forever) —
now a 10 s connect timeout, with no overall cap so long generations and
streams are never cut.
Benchmark history: per-run × and a Clear-all button. Logs view: a clear
button backed by DELETE /admin/api/logs.
The benchmark now leads with the product's actual feature. Two explicit modes:
🧭 Smart routing (default) — every probe goes through cortiq-auto; each row
shows the router's task label judged against the expected one (✓/✗ with
"expected: …"), the complexity tier, the chosen model with what that
request cost, and a task-rule badge when a routing rule fired. KPIs:
answer quality, classification accuracy, TTFT, tok/s and money spent,
plus a "requests went to" distribution with per-model cost.
🎯 Individual models — the old head-to-head comparison, now as the clearly
separate option (model checkboxes appear only here).
The suite covers all 8 taxonomy labels (added summarization, creative-writing
and chitchat probes; expected labels annotated per test). Results are LIVE:
the table grows a row after every test (with a small fade-in), the progress
card shows the prompt being asked, and a 🏁 verdict toast lands at the end
with the score and the total spend.
12 new strings in all 7 languages.
The old standalone benchmark.html (Russian-only, Tailwind+Chart.js from CDNs,
never shipped in the crate) becomes a proper sidebar view. Same scoring ideas,
reimplemented: keyword checks with |-synonyms, think-tag stripping, tail-loop
detection, pass/partial/fail — plus what the old page never had: runs go
through the LIVE pipeline (pinned per model, streamed), so every test also
measures tok/s and time-to-first-token with the gateway's real token counts.
A built-in suite of 12 language-neutral probes (math, logic, code, QA,
instruction-following, extraction, translation, JSON formatting); the ⚡ quick
mode runs the 6 short ones — about a minute on a local model. Results: score
bars per model sorted best-first, a per-test grid (click a cell for the full
answer), previous runs and custom tests kept in the browser.
22 new strings in all 7 languages.
The router has always returned two signals — task label and complexity tier —
but only the tier picked models. [routing.tasks] adds `label → ordered models`:
a matching rule's models lead the candidate chain and the tier chain follows as
fallback, so pinning translations to a local model never weakens failover.
Requests whose chain was headed by a task rule report
route_source "router+task-rule".
The Routing page grows a per-task editor built from the account's LIVE
taxonomy (GET /v1/taxonomies via the new /admin/api/routing/labels proxy;
built-in label set when the router is unreachable), with localized task names,
each task's share of 7-day traffic, "by complexity (no rule)" as the explicit
empty state, and custom labels for other taxonomies. Deleting a model scrubs
it from task rules like it always did from tiers.
Also fixes the default config pointing at a placeholder localhost:8080 router
with a nonexistent taxonomy — a fresh install now targets the hosted allaigate
router (IP endpoint, verify_tls off, data-assistant, 12 s oracle timeout) and
degrades gracefully until a key is set. Verified end-to-end against the live
router: translation → task-rule model, math → tier path.
16 new strings in all 7 languages.
Fifteen changes aimed at the first day of real use:
- Import and media jobs persist to <data>/jobs-*.json and reload on start
(running ones settle as errors); the Register button no longer vanishes
because the gateway restarted.
- Docker ships the cortiq binary (built in the builder stage) — local .cmf
chat models and image/video generation now work in containers, where
auto-install never could (no cargo in the runtime image). libvulkan1 added
for optional GPU offload.
- Open admin console on 0.0.0.0 without a token now warns in the log and
shows a dismissible banner — the VPS footgun is at least labeled.
- "Restart gateway" button in Settings: stops managed model servers, spawns
a fresh copy (bind retried while the port frees), Docker relies on the
container restart policy instead.
- Dashboard "System" panel: CPU sparkline + load, RAM (with the gateway's
own RSS), free disk at the models dir — sampled by sysinfo every 2 s.
- Featured model cards warn "not enough disk" / "may not fit in RAM" before
the download, from the same live snapshot.
- Log viewer in the sidebar: a 500-line in-memory ring fed by a second
ANSI-free tracing layer; filter, level, pause, copy.
- Update check: crates.io polled twice a day, the console shows an
"update available" notice.
- Media generations queue up (with position shown) instead of being
rejected; progress gained an honest ETA from measured throughput.
- Generation gallery under the image/video tabs: lazy PPM thumbnails,
click to reopen with seed/steps, delete.
- Playground chat is a real dialogue now: history is resent, bubbles,
Clear, Cmd/Ctrl+Enter.
- "Connect your app" snippets (curl / Python / JS) on the keys page, with
the real key inlined right after creation.
- One-click benchmark on the Models page via `cortiq bench --json`.
- Config export now downloads the actual gateway.toml (comments intact);
import accepts TOML (validated, hot-reloaded) and the legacy JSON.
- Streaming token counts are exact when the model server provides usage
(cortiq-server patch, shipped separately); the gateway fallback now
counts content chunks (1/token on cortiq) instead of chars/4.
Wizard's Finish lands on the Playground. 37 new strings in all 7 languages.
The infosave collection now ships media models (Lumina-Image-2.0, MiniMax-H3
Turbo) that cortiq runs via `imagine`/`animate` CLI, not `cortiq serve`. The
gateway grows a media API: local .cmf files are classified by `cortiq info`
(chat/image/video, cached), generations run as tracked jobs with step progress
parsed from stdout, one at a time, cancellable, outputs under <data>/media.
The playground becomes three tabs. Image: width/height/steps/CFG/seed, live
progress, result rendered from P6 PPM to a canvas in the browser (PNG export
via toBlob). Video: frames/steps/quality/seed plus optional first/last
keyframes (rasterized to PPM client-side); the MJPEG+PCM AVI is decoded in JS
and played on a canvas synced to the wav — no ffmpeg on either side (mp4 remux
happens opportunistically when ffmpeg exists). A metrics card shows resolution,
steps, seed, total time, time per step/frame and file size.
Chat metrics: the local cortiq server sends no usage chunk in streams, so
token counts showed 0/0. tap_stream now injects one OpenAI-shaped usage event
ahead of [DONE] with the gateway's ~4-chars/token estimate, flagged
`cortiq_estimated`; the UI renders it as "≈ N / M" and adds tok/s and
time-to-first-token. Registration skips media packs (a `cortiq serve` entry
for a diffusion model would only fail) in both Register and the wizard's
Finish.
36 new UI strings translated across all 7 languages.
`cargo fmt --all -- --check` and `cargo clippy --all-targets -- -D warnings` are
CI gates and both were failing on this release's code.
- rustfmt across the files touched by the release
- `.split('/').last()` walks the whole iterator on a DoubleEndedIterator; use
`.next_back()` (4 sites in the setup/featured + delete handlers)
- `set_progress` had two identical branches; the intent is "take the first
reading, then only move forward", which is one condition
release.yml published unconditionally on a version tag, so a version already on
crates.io (released by hand, or a re-run of the job) failed the workflow — a
version can never be re-uploaded. It now checks the registry and skips instead.
Startup no longer needs arguments: the config is created on first launch, a busy
port is swapped for a free one, the console is open by default, and the browser
lands on a setup wizard that downloads a ready-made .cmf model and wires it into
routing (managed server + routing.default) when you press Finish.
Download progress was reported from curl's own meter, which prints a full
0→100% bar for the -L redirect hop before the transfer starts — so every job
showed 100% and then hung. Progress now comes from bytes on disk against the
size from the HuggingFace tree (or a HEAD request).
The UI and the gateway also disagreed about the output file name, so a finished
download never marked its card as downloaded; the gateway now returns the exact
stem and the UI sends it back verbatim for download, progress and delete.
Featured models are read live from the infosave HF account in both the wizard
and Import, and all 432 UI strings now exist in every one of the 7 languages.
- pin builder and runtime to the same Debian release (trixie): the floating
rust:1-slim builder drifted to a newer glibc than the bookworm runtime, so
freshly built images crashed on start with 'GLIBC_2.39 not found'
- install curl in the runtime image — ready-.cmf imports download with it
- keep all gateway state under /app/data (CORTIQ_GATEWAY_HOME) and run as
root so bind-mounted host folders on Docker Desktop are writable; pass
--user 10001 to harden when not bind-mounting
- persist /app/data via a cortiq-data volume in docker-compose
- fail imports immediately with the path and reason when the models dir
cannot be created, and annotate curl exit codes 23/22 with readable hints
- exclude local models/*.cmf from the docker build context
Two bugs:
1. run_stream() never merged per-model/server config (think_budget,
temperature, etc.) into the request — unlike run_inner(). Now both
paths apply the same config merge.
2. to_wire() was missing the enable_thinking bool conversion line
(lost in a prior revert). Restored: think_budget=0 → enable_thinking=false.
- New Advanced controls on the import page: O(1) attention spec (off / all /
deep N / explicit layer list) plus the --o1-m / --o1-window / --o1-sink knobs
with validated defaults (32 / 128 / 4) and explanatory hints.
- Backend passes --o1* to the native `cortiq convert`; fails fast with clear
messages when combined with Python-only options, used on GGUF imports, or
when the installed cortiq predates 0.2.0 (the hint needs the new runtime).
- The hint is recorded in the .cmf header (weights byte-identical); the runtime
applies it at load. Verified end-to-end: converted a model with --o1 all and
the runtime logged "o1 nystrom attention: 2/2 layer(s)".
- Stats snapshot now computes the KPI totals AND the breakdown from the
in-window per-minute buckets (previously they used all-time accumulators, so
switching 1h/24h/7d changed only the chart, not the numbers). Buckets now
carry failovers + per-group breakdown; Prometheus keeps all-time counters.
- New DELETE /admin/api/stats clears all in-memory stats and truncates the
JSONL log; the dashboard gets a "Clear logs" button (with confirm).
- The model id is now auto-filled from the model name as you type, and if left
blank it's derived on save (slug of the last path segment, e.g.
"qwen/qwen3.7-max" → "qwen3-7-max") — saving no longer errors on a blank id.
- probe/test auth failures (401 / "User not found" / "No auth") now append a
hint to check the API key (for OpenRouter: use an inference key, not a
provisioning/management key).
- ui.js: mount() now wraps non-node children in text nodes (like h()), so
mount(el, "") no longer throws in appendChild — this is what broke the
"Add model" form (applyDefaults cleared a status via mount(x,"")).
- probe_model: resolve managed local models too (they are synthesized into the
registry, not [[models]]) — probing them no longer returns "model not found".
- models.js: await reload() after delete so the row is removed immediately.
- Providers: add openrouter (https://openrouter.ai/api/v1) and lmstudio
(http://localhost:1234/v1), both OpenAI-compatible. Selecting a provider now
auto-fills its verified base URL, cost tier and default capabilities.
- API key is typed directly in the form and stored securely as a secret under the
provider's env name — no manual env var to manage.
- New POST /admin/api/provider/models validates the key and lists the provider's
models; a "Test & load models" button fills a searchable dropdown. The model
field now sits below the key (some providers list models from the key).
- Fuller capability set: tools, vision, audio, reasoning, json, caching.
- New GET /admin/api/cmf/files lists the .cmf files (and tensor-dir .cmf) under
models_dir, with sizes.
- Settings → Local models: the per-row model field is now a dropdown of those
files instead of a hand-typed path, and auto-fills the id from the filename.
- The gateway can now run several local .cmf models at once: a [[cmf.servers]]
list, one supervised `cortiq serve` per entry on its own port, each registered
in the pool and independently routable/pinnable. The single legacy
local_model/local_port still works (back-compat via effective_servers()).
- Settings → Local models is now a rows editor (id · path · port + check · threads
· GPU) with add/remove and per-server serving status.
- Deleting a managed model stops just its server and removes its file — unless
another model still points at the same file (shared-file guard).
- A pinned playground model that is unavailable now errors clearly instead of
silently answering from the managed local model (route_source no longer lies).
- The managed CMF model is surfaced in GET /admin/api/models (marked "managed",
shown read-only in the UI) so the real running model is visible and pinnable
without a hand-maintained [[models]] entry; a fresh install still lists nothing.
- Deleting a model now removes its local .cmf physically (file or tensor-dir);
deleting the managed model also stops the server and clears the [cmf] config.
Routing is stripped of the deleted id so the config still validates.
- import.rs / promotion.rs: sort_by(|a,b| b.k.cmp(&a.k)) -> sort_by_key(Reverse(k))
- pipeline.rs: skip_while(!p).next() -> find(p)
Behaviour-preserving; completes the CI fix started by the cargo fmt commit.
- Settings: "?" help tooltips on every option; router key auto-probes the
connection right after saving (no more hanging "unavailable")
- Local models card reworked into 3 clear steps (install cortiq → add model →
run locally); removed the hand-entered .cmf path / pool-id fields — models
are added via the Import tab and the managed server registers them
- Fresh install ships an EMPTY model pool; the local .cmf is auto-registered by
the managed CMF server, so it needs no static [[models]] entry
- New: port-availability check — GET /admin/api/cmf/port + "Check port" button
with next-free-port suggestion. bind()+connect() probe so Docker-proxied /
dual-stack ports (e.g. 8081) are correctly detected as occupied
- /admin/api/cmf now reports the serving model + port; card shows "▶ serving …"
- On-demand runtime install: POST /admin/api/cmf/install
The crates.io/release build shipped a STALE frontend (committed web/js was behind
the working tree), so the admin console hung on load. Commit the current working
web/js (app, api, dashboard, login, import view, i18n de/es/fr/tr/zh, css) and the
backend WIP (import.rs, admin/assets.rs) so the released console matches the dev
build. Verified: clean `cargo build --release`, empty-pool start, working
dashboard served.
- config.rs: validate() no longer errors on zero models — an empty pool is
allowed (warns instead), so the admin console comes up on a fresh install and
the user adds models in /admin -> Models (or points [cmf] at a local .cmf).
- gateway.example.toml: ships an EMPTY pool (no seed cmf-local/local-embed, empty
routing), so `cp example -> gateway.toml && run` shows an empty list — what a
new user expects.
The v0.2.3 CMF-integration commit referenced crate::promotion (main.rs),
router_client::RouterLastStatus (state.rs) and router_client::classify_status
(admin), but those definitions were not part of the commit, so a clean build
failed (E0583/E0432/E0425). Add the two source files; cargo build --release now
compiles.
The example config pool now shows one local CMF model (served by the managed
cortiq server) instead of local-qwen/gpt-4o-mini/claude-opus placeholders, with
routing pointing at it, and a documented [cmf] section (manage_server, threads,
gpu, ...). Add cloud/local providers by copying the [[models]] shape.
Expose the two speed knobs in the /admin Settings panel instead of TOML-only:
- [cmf].gpu (new) -> exports CMF_GPU=1 to the managed cortiq serve (Metal/wgpu).
On Apple Silicon decode is memory-bandwidth bound so this is ~neutral for
generation (helps prefill); off by default.
- [cmf].threads (added earlier) now has a UI number input.
- config.rs: CmfCfg.gpu bool (serde default false).
- cmf_runtime.rs: spawn_serve sets CMF_GPU=1 when gpu is on.
- admin/mod.rs: put_settings flags needs_restart on threads/gpu change (the
server is spawned once at startup with those env vars).
- settings.js: threads input + gpu checkbox + save payload + speed hint.
- i18n en/ru: threads/gpu/speedNote keys (other locales fall back to en, as the
whole cmf section already does there).
The gateway spawned `cortiq serve` with no thread hint, so the runtime ran the
decode loop single-threaded (~0.25 tok/s on a 15B CPU model). Export CMF_THREADS
from a new [cmf].threads config field (default 8) in spawn_serve.
Measured: 0.25 -> 0.81 tok/s (~3.2x) on qwopus-vbit7 (15B). Scaling is sublinear
(GatedDeltaNet state recurrence is serial per token, Amdahl-bound); a real jump
needs the GPU path.
Run models in the CMF (.cmf) format locally, via the native `cortiq`
CLI from crates.io — no Python and no external model server.
- cmf_runtime: install/update `cortiq-cli` from crates.io, spawn and
health-check `cortiq serve` as a managed local backend.
- import: model-import jobs shell out to the native `cortiq convert`
(HF repo -> .cmf, quant mapped Q8_2F->q8_2f / Q8_ROW->q8 / Q4->q4);
the Python converter is used only for advanced/exotic options.
- pipeline: local-first routing — serve from the local CMF model when
the router is disabled/unreachable, with a "local only" hard switch.
- registry/state: register the local CMF server as an OpenAI provider
and track local candidates.
- config: data-dir resolution + CmfCfg (auto-install/update, managed
server, host/port/model) and relaxed validation for the managed model.
- admin + settings UI (en/ru): "Local models (CMF)" settings card.
- docs/ROUTER.md + docs/ROUTER.ru.md: what the semantic router does, hosted vs
self-hosted, config, API contract, pricing, and the LAUNCH1 $1 Starter promo
- README (EN+RU): "The router (allaigate)" section with the promo callout
- bench/tasks.jsonl (37 natural-language prompts, 7 task types) + bench/accuracy.py
- measured on the live allaigate router: 100% (37/37) vs keyword heuristic 32%
- BENCHMARKS.md accuracy section; README (EN+RU) perf+accuracy tables and test matrix
- run.sh runs accuracy when CORTIQ_ROUTER_KEY is set
- GET /v1/models: pool + virtual cortiq-auto
- POST /v1/completions (legacy): non-stream + streaming text_completion
- POST /route: native passthrough returning the routing decision
- all inbound protocol adapters now implemented; admin meta + README/CHANGELOG updated
- bump to 0.2.1
- POST /mcp (JSON-RPC 2.0): initialize, ping, tools/list, tools/call; notifications -> 202
- tools: cortiq_route (classify -> routing decision) and cortiq_chat (route -> answer)
- gated by protocols.mcp; admin meta marks it implemented
- docs: README/README.ru protocol tables, highlights and status updated for
streaming, Anthropic, embeddings, semantic cache and MCP
- cache.rs: cosine-similarity store (threshold, TTL, ring buffer), partitioned by
routing signature; tracks hit rate and model cost saved
- pipeline: embed routing text, look up before the model call (route_source=cache on
hit), insert successful responses; stats record cache hits
- config [cache] + admin settings card; dashboard shows hit-rate and savings
- embedding model resolved from the pool (or first kind="embedding")
infosave2007