Every stream used to carry two chunks with an empty choices array: usage,
which the upstream emits BEFORE the stop reason, and a cortiq_cost_usd chunk
the gateway added itself. OpenAI ships the usage chunk only when the request
sets stream_options: {"include_usage": true}, and chat UIs do not set it -
which is why the same tool, in the same Open WebUI, sees neither chunk from
Groq and two from us.
Honour the flag. With no stream_options a stream is now exactly
role -> content... -> tool_calls -> finish_reason -> [DONE], with no
choice-less chunks at all. When usage is requested it ships once, after the
stop reason, with cortiq_cost_usd folded into the same chunk rather than
trailing behind it.
Token accounting, cost and the X-Cortiq-* headers are untouched: those numbers
were never read back off the wire.
Verified: the default stream matches the reference shape byte for byte, the
opt-in stream matches OpenAI's, the official OpenAI SDK reassembles the
streamed tool call, and the full Open WebUI 0.11.0 tool loop still runs the
tool and answers from its result.
Qwen-family templates write a function call into the generated text as
<tool_call>{...}</tool_call>. Runtimes disagree on what to do with it: some
parse it into structured tool_calls AND still stream the raw markup as
content - the client then gets the same call twice, once as prose it must not
show and once as the real thing - while others never parse it at all, so a
perfectly good call arrives as text no tool loop will run.
Settle it in the gateway. A streaming-aware filter strips the markup from
content (handling a marker split across SSE chunks, leaving ordinary text
containing "<" alone, and handing an unterminated block back as text), and
when nothing structured came with it the markup is promoted into a real
tool_calls entry with a generated call_... id. Markup that is not a JSON
object with a name is ignored so a stray marker cannot become a bogus call.
Streamed tool_calls are also brought to the OpenAI shape: index on every
entry, id and type exactly once per call so continuation deltas carrying only
an argument fragment stay legal, and function.arguments always a string. A
reply that carries tool calls now always ends with finish_reason "tool_calls",
whatever the upstream called it. The same cleanup runs on the non-streaming
path, so every protocol adapter benefits.
Verified end to end: Open WebUI 0.11.0 -> gateway -> managed CMF/Qwen with
native function calling. The model returns get_weather({"city":"Paris"}), Open
WebUI executes the tool, feeds the result back, and the assistant answers from
it - no markup in content, finish_reason tool_calls, stream closed with [DONE].
router.allaigate.com now resolves and serves a valid certificate, so the
hosted routing API no longer has to be reached by bare IP with certificate
verification switched off. Make https://router.allaigate.com the default
[router].url with verify_tls = true.
Existing configs still pointing at 138.226.222.209 (with or without a scheme
or a trailing slash) are moved to the hostname on load and have verification
restored, then saved: it is the same machine, and this is the difference
between a verified connection and a blind one. A self-hosted router URL is
left untouched.
README, docs/ROUTER.md (both languages), config/gateway.example.toml and
bench/accuracy.py follow.
Verified live: the migration rewrites a legacy config on startup, admin
health reports the hostname reachable over verified TLS (389 ms), and real
requests route through it - "math"/medium 0.416 and "translation"/low 0.268
with route_source=router.
Verified against a real Open WebUI container: it attaches its ~26 built-in
tools (knowledge bases, notes, calendar, ...) to every single message, and
resolve() treated that offer as a hard capability requirement. On a pool of
tools-less local models a plain "hello" therefore failed with "No healthy
model available that satisfies required capabilities", and a pinned local
model was reported as "not available - not in the pool (deleted?)".
In the OpenAI contract `tools` is an offer; only tool_choice: "required" (or
a named function) makes the call impossible without them. Keep preferring a
tool-capable model, but fall back to serving the request with the tools
stripped - together with tool_choice/parallel_tool_calls, which a provider
rejects on their own. A pin whose only shortcoming is tool support is now
honoured; an unknown pin and tool_choice: "required" still fail loudly.
The downgrade is never silent: X-Cortiq-Tools-Dropped: true, plus
cortiq.tools_dropped in the echo block.
/v1/chat/completions answered ordinary client payloads with a bare HTTP 422
(a plain-text body no OpenAI client can parse), so Open WebUI showed neither
an answer nor an error — it just hung. Parse the body leniently instead:
content as string/null/content-part array, tools: null, stream: null, a
string reasoning_effort, max_completion_tokens, messages[].name and
tool_call_id, and a missing Content-Type. Unreadable bodies now come back as
an OpenAI error envelope with 400.
Responses are strict OpenAI JSON: no "tool_calls": null, finish_reason mapped
to the spec's closed set (cortiq serve's max_tokens -> length), never an empty
id, and `model` echoes the id the client requested — the model that actually
answered stays in cortiq.selected_model and X-Cortiq-Selected-Model.
Streaming is re-normalised rather than forwarded verbatim: a leading role
delta, message-shaped chunks converted to delta, one terminal finish_reason,
always a trailing data: [DONE], and an SSE error event when the upstream dies
mid-flight. Adds X-Accel-Buffering: no so a fronting proxy cannot buffer the
stream into a hang.
GET /v1/models now lists managed [[cmf.servers]] models with their live
running/healthy state, so clients can pin a local .cmf directly.
Also strips client bookkeeping keys (chat_id, session_id, metadata, features,
...) from the upstream call and drops the leftover [to_wire] debug prints.
The office-readiness audit found the foundation missing: configured API keys
were decorative. Any bearer — or none — passed every model endpoint, per-key
rate limits existed only in the form, and all real traffic was recorded as
"anonymous", leaving the per-key spend table permanently blank.
A middleware on the model surface (/v1/*, /route, /mcp) now enforces keys:
- no keys configured → open mode as before (the console already warns);
- keys exist → Bearer or x-api-key must match (401 invalid_api_key);
- per-key rate_per_min → fixed one-minute window, 429 with Retry-After;
- NEW per-key monthly_budget_usd → 429 insufficient_quota once the account's
month-to-date spend crosses the ceiling;
- the matched account rides the request into stats — the keys table's spend
columns finally see real traffic.
Month-to-date spend is tracked per account in stats and persisted in the
totals sidecar with a watermark (no double counting across restarts). The
key dialog gains a budget field; the table shows "budget $N/mo" in red once
exhausted. All verified live: 401 wrong/missing key, 200 + attribution with
the right key, 429 + Retry-After over the limit, 429 insufficient_quota on
the second request past a $0.000001 budget.
Also: the dashboard shows the OpenRouter credit balance (GET
/api/v1/credits via the stored key, cached 5 min, hidden without a key, red
under $5 — live: $16.18), the server drains connections on Ctrl-C instead
of dropping them, request bodies are capped at 32 MB, and docs/DEPLOY(.ru)
cover office deployment: systemd/launchd/Windows service, nginx/caddy TLS,
security checklist, backups. 10 strings ×7 languages.
The single per-task «Complexity» pin was the wrong shape — what you actually
decide is WHICH model serves a task at WHICH complexity. A task rule is now a
per-tier map, edited in the console as rows of “model ▾ + complexity ▾” pairs
(plus ×/add) inside each task:
[routing.tasks.code]
low = ["cheap-coder"]
high = ["best-coder"]
any = ["generic-coder"] # leads at every tier after the tier pick
Chain order: the task's tier-specific list → (at high: the global high chain,
so escalation still beats any-tier pins) → the task's any-tier list → the
global tier chain → default as the last resort; deduplicated. Plain-array
rules keep meaning "any tier" unchanged, and the short-lived {models, tier}
form from the previous commit is normalized into the map on load.
Verified live: legacy config migrated visibly in the editor, pairs survive
save+reload, a math prompt at low ignored the math@high pair, per-tier leads
confirmed by unit tests (19 passing).
API keys page: the table now answers "which department spends what" — per-key
requests, tokens in/out and cost for a selectable window (24h/7d/30d) plus a
cumulative all-time cost column, joined from stats by the key's account.
The stats log stays bounded while the accounting survives: once stats.jsonl
exceeds max_file_mb (default 16, 0 disables), records older than the
retention window fold into cumulative per-account totals in a small sidecar
file (tmp+rename on both) and leave the log. alltime = folded + in-file, a
record is never counted twice, verified end-to-end on synthetic data
(1.2 MB → 301 KB, totals identical before/after compaction and restarts).
Routing: a task rule can now PIN the complexity tier —
`[routing.tasks.legal] models=[...] tier="high"` (old plain-array rules parse
unchanged). The pinned tier replaces the router's verdict before the
escalation check and reports itself as route_source "router+task-rule@high".
The Routing page grows a per-task «Complexity» selector (Auto/low/medium/
high); verified live: a code prompt with a pinned high went
router+task-rule@high → the expensive model.
13 new strings in all 7 languages; docs/ROUTING(.ru) updated.
The router has always returned two signals — task label and complexity tier —
but only the tier picked models. [routing.tasks] adds `label → ordered models`:
a matching rule's models lead the candidate chain and the tier chain follows as
fallback, so pinning translations to a local model never weakens failover.
Requests whose chain was headed by a task rule report
route_source "router+task-rule".
The Routing page grows a per-task editor built from the account's LIVE
taxonomy (GET /v1/taxonomies via the new /admin/api/routing/labels proxy;
built-in label set when the router is unreachable), with localized task names,
each task's share of 7-day traffic, "by complexity (no rule)" as the explicit
empty state, and custom labels for other taxonomies. Deleting a model scrubs
it from task rules like it always did from tiers.
Also fixes the default config pointing at a placeholder localhost:8080 router
with a nonexistent taxonomy — a fresh install now targets the hosted allaigate
router (IP endpoint, verify_tls off, data-assistant, 12 s oracle timeout) and
degrades gracefully until a key is set. Verified end-to-end against the live
router: translation → task-rule model, math → tier path.
16 new strings in all 7 languages.
Startup no longer needs arguments: the config is created on first launch, a busy
port is swapped for a free one, the console is open by default, and the browser
lands on a setup wizard that downloads a ready-made .cmf model and wires it into
routing (managed server + routing.default) when you press Finish.
Download progress was reported from curl's own meter, which prints a full
0→100% bar for the -L redirect hop before the transfer starts — so every job
showed 100% and then hung. Progress now comes from bytes on disk against the
size from the HuggingFace tree (or a HEAD request).
The UI and the gateway also disagreed about the output file name, so a finished
download never marked its card as downloaded; the gateway now returns the exact
stem and the UI sends it back verbatim for download, progress and delete.
Featured models are read live from the infosave HF account in both the wizard
and Import, and all 432 UI strings now exist in every one of the 7 languages.
- docs/ROUTER.md + docs/ROUTER.ru.md: what the semantic router does, hosted vs
self-hosted, config, API contract, pricing, and the LAUNCH1 $1 Starter promo
- README (EN+RU): "The router (allaigate)" section with the promo callout
infosave2007