fix(stream): no choice-less chunks unless the client asks for usage
CI / lint (push) Canceled after 0s
CI / test (macos-14) (push) Canceled after 0s
CI / test (ubuntu-latest) (push) Canceled after 0s
CI / test (windows-latest) (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
Docker / publish (push) Canceled after 0s

Every stream used to carry two chunks with an empty choices array: usage,
which the upstream emits BEFORE the stop reason, and a cortiq_cost_usd chunk
the gateway added itself. OpenAI ships the usage chunk only when the request
sets stream_options: {"include_usage": true}, and chat UIs do not set it -
which is why the same tool, in the same Open WebUI, sees neither chunk from
Groq and two from us.

Honour the flag. With no stream_options a stream is now exactly
role -> content... -> tool_calls -> finish_reason -> [DONE], with no
choice-less chunks at all. When usage is requested it ships once, after the
stop reason, with cortiq_cost_usd folded into the same chunk rather than
trailing behind it.

Token accounting, cost and the X-Cortiq-* headers are untouched: those numbers
were never read back off the wire.

Verified: the default stream matches the reference shape byte for byte, the
opt-in stream matches OpenAI's, the official OpenAI SDK reassembles the
streamed tool call, and the full Open WebUI 0.11.0 tool loop still runs the
tool and answers from its result.
This commit is contained in:
infosave2007
2026-08-21 21:24:11 +03:00
parent 9fd4bdaa85
commit 0b071279ac
6 changed files with 123 additions and 15 deletions
+6
View File
@@ -121,6 +121,12 @@ opens with a `role` delta, carries exactly one terminal `finish_reason`, and end
failure arrives as an SSE `error` event. A client can therefore never be left waiting on a
half-formed stream.
Streaming carries no choice-less chunks unless asked: `usage` is emitted only when the
request sets `stream_options: {"include_usage": true}`, and then exactly once, after the
`finish_reason` chunk, with `cortiq_cost_usd` folded in. Without it a stream is
`role → content… → tool_calls → finish_reason → [DONE]` and nothing else — the shape the
hosted providers send.
### Tool calls
`tools` and `tool_calls` are proxied through, with the wire kept clean:
+6
View File
@@ -119,6 +119,12 @@ HTTP 400 — но не «голый» 422, который OpenAI-клиенты
`data: [DONE]` — даже если upstream оборвался или упал на середине (тогда ошибка приходит
SSE-событием `error`). Клиент никогда не остаётся ждать недосказанный поток.
В стриме нет чанков без `choices`, если их не просили: `usage` отдаётся, только когда
запрос содержит `stream_options: {"include_usage": true}`, и тогда ровно один раз — после
чанка с `finish_reason`, с вложенным `cortiq_cost_usd`. Без него поток выглядит как
`role → content… → tool_calls → finish_reason → [DONE]` и ничего больше — ровно так же,
как у хостовых провайдеров.
### Tool calls
`tools` и `tool_calls` проксируются, но провод держится чистым: