fix(stream): no choice-less chunks unless the client asks for usage
CI / lint (push) Canceled after 0s
CI / test (macos-14) (push) Canceled after 0s
CI / test (ubuntu-latest) (push) Canceled after 0s
CI / test (windows-latest) (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
Docker / publish (push) Canceled after 0s
CI / lint (push) Canceled after 0s
CI / test (macos-14) (push) Canceled after 0s
CI / test (ubuntu-latest) (push) Canceled after 0s
CI / test (windows-latest) (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
Docker / publish (push) Canceled after 0s
Every stream used to carry two chunks with an empty choices array: usage,
which the upstream emits BEFORE the stop reason, and a cortiq_cost_usd chunk
the gateway added itself. OpenAI ships the usage chunk only when the request
sets stream_options: {"include_usage": true}, and chat UIs do not set it -
which is why the same tool, in the same Open WebUI, sees neither chunk from
Groq and two from us.
Honour the flag. With no stream_options a stream is now exactly
role -> content... -> tool_calls -> finish_reason -> [DONE], with no
choice-less chunks at all. When usage is requested it ships once, after the
stop reason, with cortiq_cost_usd folded into the same chunk rather than
trailing behind it.
Token accounting, cost and the X-Cortiq-* headers are untouched: those numbers
were never read back off the wire.
Verified: the default stream matches the reference shape byte for byte, the
opt-in stream matches OpenAI's, the official OpenAI SDK reassembles the
streamed tool call, and the full Open WebUI 0.11.0 tool loop still runs the
tool and answers from its result.
This commit is contained in:
@@ -121,6 +121,12 @@ opens with a `role` delta, carries exactly one terminal `finish_reason`, and end
|
||||
failure arrives as an SSE `error` event. A client can therefore never be left waiting on a
|
||||
half-formed stream.
|
||||
|
||||
Streaming carries no choice-less chunks unless asked: `usage` is emitted only when the
|
||||
request sets `stream_options: {"include_usage": true}`, and then exactly once, after the
|
||||
`finish_reason` chunk, with `cortiq_cost_usd` folded in. Without it a stream is
|
||||
`role → content… → tool_calls → finish_reason → [DONE]` and nothing else — the shape the
|
||||
hosted providers send.
|
||||
|
||||
### Tool calls
|
||||
|
||||
`tools` and `tool_calls` are proxied through, with the wire kept clean:
|
||||
|
||||
@@ -119,6 +119,12 @@ HTTP 400 — но не «голый» 422, который OpenAI-клиенты
|
||||
`data: [DONE]` — даже если upstream оборвался или упал на середине (тогда ошибка приходит
|
||||
SSE-событием `error`). Клиент никогда не остаётся ждать недосказанный поток.
|
||||
|
||||
В стриме нет чанков без `choices`, если их не просили: `usage` отдаётся, только когда
|
||||
запрос содержит `stream_options: {"include_usage": true}`, и тогда ровно один раз — после
|
||||
чанка с `finish_reason`, с вложенным `cortiq_cost_usd`. Без него поток выглядит как
|
||||
`role → content… → tool_calls → finish_reason → [DONE]` и ничего больше — ровно так же,
|
||||
как у хостовых провайдеров.
|
||||
|
||||
### Tool calls
|
||||
|
||||
`tools` и `tool_calls` проксируются, но провод держится чистым:
|
||||
|
||||
Reference in New Issue
Block a user