fix(openai): strict OpenAI wire compatibility — Open WebUI can chat again
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s

/v1/chat/completions answered ordinary client payloads with a bare HTTP 422
(a plain-text body no OpenAI client can parse), so Open WebUI showed neither
an answer nor an error — it just hung. Parse the body leniently instead:
content as string/null/content-part array, tools: null, stream: null, a
string reasoning_effort, max_completion_tokens, messages[].name and
tool_call_id, and a missing Content-Type. Unreadable bodies now come back as
an OpenAI error envelope with 400.

Responses are strict OpenAI JSON: no "tool_calls": null, finish_reason mapped
to the spec's closed set (cortiq serve's max_tokens -> length), never an empty
id, and `model` echoes the id the client requested — the model that actually
answered stays in cortiq.selected_model and X-Cortiq-Selected-Model.

Streaming is re-normalised rather than forwarded verbatim: a leading role
delta, message-shaped chunks converted to delta, one terminal finish_reason,
always a trailing data: [DONE], and an SSE error event when the upstream dies
mid-flight. Adds X-Accel-Buffering: no so a fronting proxy cannot buffer the
stream into a hang.

GET /v1/models now lists managed [[cmf.servers]] models with their live
running/healthy state, so clients can pin a local .cmf directly.

Also strips client bookkeeping keys (chat_id, session_id, metadata, features,
...) from the upstream call and drops the leftover [to_wire] debug prints.
This commit is contained in:
infosave2007
2026-08-21 09:10:31 +03:00
parent 34093330f6
commit b339c2b622
15 changed files with 660 additions and 93 deletions
+40 -9
View File
@@ -53,7 +53,17 @@ Routing rule common to all protocols:
```
Standard OpenAI fields are supported; unknown fields are proxied to the provider unchanged
(forward-compatible).
(forward-compatible), minus the bookkeeping keys chat UIs attach to the payload
(`chat_id`, `session_id`, `metadata`, `features`, …), which are dropped so strict upstreams
do not reject the call.
The parser is deliberately lenient about the shapes real clients send:
`content` may be a string, `null`, or an array of content parts (text parts are
concatenated, images and other non-text parts are skipped); `tools` may be `null`;
`reasoning_effort` may be a string (`low` | `medium` | `high`) or a token budget;
`max_completion_tokens` is accepted as an alias for `max_tokens`. A body that still
cannot be read comes back as an OpenAI error envelope with HTTP 400 — never a bare 422,
which OpenAI clients cannot parse (they show no answer at all).
### Response (non-streaming)
@@ -62,12 +72,12 @@ Standard OpenAI fields are supported; unknown fields are proxied to the provider
"id": "chatcmpl-9f...",
"object": "chat.completion",
"created": 1718800000,
"model": "local-qwen", // the ACTUALLY selected model
"model": "cortiq-auto", // the model the CLIENT asked for, echoed back
"choices": [
{
"index": 0,
"message": { "role": "assistant", "content": "x = 2 or x = 3" },
"finish_reason": "stop"
"finish_reason": "stop" // always one of stop|length|tool_calls|content_filter
}
],
"usage": {
@@ -86,10 +96,24 @@ Standard OpenAI fields are supported; unknown fields are proxied to the provider
}
```
> `model` echoes the id the client requested, exactly as OpenAI does — clients match the
> reply against the id they sent. The model that actually answered is in
> `cortiq.selected_model` and in the `X-Cortiq-Selected-Model` header.
> Optional fields are omitted rather than sent as `null`: absent tool calls mean no
> `tool_calls` key at all. `"tool_calls": null` is not in the OpenAI schema and trips
> strict clients.
> The `cortiq` field is an optional extension. OpenAI SDKs ignore it. If the client does
> not want extra fields, disable `cortiq.echo` — the metadata will remain in the response
> headers.
Streaming (`stream: true`) emits strict `chat.completion.chunk` events: the stream always
opens with a `role` delta, carries exactly one terminal `finish_reason`, and ends with
`data: [DONE]` — even when the upstream stops early or dies mid-flight, in which case the
failure arrives as an SSE `error` event. A client can therefore never be left waiting on a
half-formed stream.
### Metadata in response headers (always present)
```
@@ -164,20 +188,27 @@ single user message and follows the same pipeline. Response `object` is `"text_c
## 4. OpenAI Models
`GET /v1/models` — auto-discovery for clients. Returns the gateway's virtual models
(`cortiq-auto`, `cortiq-embed`) and all real models in the pool.
(`cortiq-auto`, `cortiq-embed`), every model in the `[[models]]` pool, and every managed
local `[[cmf.servers]]` model, so a client can pin a local `.cmf` directly.
```jsonc
{
"object": "list",
"data": [
{ "id": "cortiq-auto", "object": "model", "owned_by": "cortiq-gateway" },
{ "id": "local-qwen", "object": "model", "owned_by": "cortiq-gateway" },
{ "id": "gpt-4o-mini", "object": "model", "owned_by": "cortiq-gateway" },
{ "id": "claude-opus", "object": "model", "owned_by": "cortiq-gateway" }
{ "id": "cortiq-auto", "object": "model", "created": 1718800000, "owned_by": "cortiq" },
{ "id": "gpt-4o-mini", "object": "model", "created": 1718800000, "owned_by": "openai" },
{ "id": "claude-opus", "object": "model", "created": 1718800000, "owned_by": "anthropic" },
{ "id": "cmf-qwen3-4b", "object": "model", "created": 1718800000, "owned_by": "cortiq-cmf",
"cortiq": { "kind": "cmf", "local": true, "running": true, "healthy": true } }
]
}
```
Managed CMF models are listed as soon as they are configured, with their live state in the
non-standard `cortiq` field (clients ignore it). They are not filtered by health: a large
`.cmf` takes minutes to load, and a model that disappears from the list while it warms up
is worse than one marked "not ready yet".
---
## 5. Anthropic Messages
@@ -198,7 +229,7 @@ single user message and follows the same pipeline. Response `object` is `"text_c
"id": "msg_...",
"type": "message",
"role": "assistant",
"model": "local-qwen",
"model": "cortiq-auto",
"content": [ { "type": "text", "text": "x = 2 or x = 3" } ],
"stop_reason": "end_turn",
"usage": { "input_tokens": 24, "output_tokens": 11 }
+38 -8
View File
@@ -53,7 +53,17 @@
```
Поддерживаются стандартные поля OpenAI; неизвестные поля проксируются к провайдеру
без изменений (forward-compatible).
без изменений (forward-compatible) — кроме служебных ключей, которые чат-интерфейсы
добавляют в payload для собственных нужд (`chat_id`, `session_id`, `metadata`,
`features`, …): они отбрасываются, чтобы строгий upstream не отклонил вызов.
Парсер намеренно терпим к тем формам, которые реально шлют клиенты: `content` может быть
строкой, `null` или массивом content-частей (текстовые части склеиваются, изображения и
прочие нетекстовые пропускаются); `tools` может быть `null`; `reasoning_effort` — строкой
(`low` | `medium` | `high`) или бюджетом токенов; `max_completion_tokens` принимается как
синоним `max_tokens`. Если тело всё-таки не читается, возвращается OpenAI-конверт ошибки с
HTTP 400 — но не «голый» 422, который OpenAI-клиенты не умеют разбирать (для пользователя
это выглядит как отсутствие ответа).
### Ответ (без стриминга)
@@ -62,12 +72,12 @@
"id": "chatcmpl-9f...",
"object": "chat.completion",
"created": 1718800000,
"model": "local-qwen", // РЕАЛЬНО выбранная модель
"model": "cortiq-auto", // модель, которую ЗАПРОСИЛ клиент — возвращается как есть
"choices": [
{
"index": 0,
"message": { "role": "assistant", "content": "x = 2 или x = 3" },
"finish_reason": "stop"
"finish_reason": "stop" // всегда одно из stop|length|tool_calls|content_filter
}
],
"usage": {
@@ -86,9 +96,22 @@
}
```
> `model` возвращает тот id, который запросил клиент, — ровно как это делает OpenAI:
> клиенты сопоставляют ответ с отправленным id. Модель, которая реально ответила, — в
> `cortiq.selected_model` и в заголовке `X-Cortiq-Selected-Model`.
> Необязательные поля не сериализуются вместо того, чтобы приходить как `null`: если
> tool-вызовов нет, ключа `tool_calls` в ответе нет вообще. `"tool_calls": null` не входит
> в схему OpenAI и ломает строгие клиенты.
> Поле `cortiq` — необязательное расширение. OpenAI-SDK его игнорируют. Если клиент
> не хочет «лишних» полей — выключите `cortiq.echo`, метаданные останутся в заголовках.
Стриминг (`stream: true`) отдаёт строгие события `chat.completion.chunk`: поток всегда
открывается дельтой с `role`, содержит ровно один финальный `finish_reason` и завершается
`data: [DONE]` — даже если upstream оборвался или упал на середине (тогда ошибка приходит
SSE-событием `error`). Клиент никогда не остаётся ждать недосказанный поток.
### Метаданные в заголовках (всегда)
```
@@ -163,20 +186,27 @@ data: [DONE]
## 4. OpenAI Models
`GET /v1/models` — автодискавери для клиентов. Возвращает виртуальные модели шлюза
(`cortiq-auto`, `cortiq-embed`) и все реальные модели пула.
(`cortiq-auto`, `cortiq-embed`), все модели пула `[[models]]` и все управляемые локальные
модели `[[cmf.servers]]` — чтобы клиент мог напрямую закрепить локальный `.cmf`.
```jsonc
{
"object": "list",
"data": [
{ "id": "cortiq-auto", "object": "model", "owned_by": "cortiq-gateway" },
{ "id": "local-qwen", "object": "model", "owned_by": "cortiq-gateway" },
{ "id": "gpt-4o-mini", "object": "model", "owned_by": "cortiq-gateway" },
{ "id": "claude-opus", "object": "model", "owned_by": "cortiq-gateway" }
{ "id": "cortiq-auto", "object": "model", "created": 1718800000, "owned_by": "cortiq" },
{ "id": "gpt-4o-mini", "object": "model", "created": 1718800000, "owned_by": "openai" },
{ "id": "claude-opus", "object": "model", "created": 1718800000, "owned_by": "anthropic" },
{ "id": "cmf-qwen3-4b", "object": "model", "created": 1718800000, "owned_by": "cortiq-cmf",
"cortiq": { "kind": "cmf", "local": true, "running": true, "healthy": true } }
]
}
```
Управляемые CMF-модели попадают в список сразу, как только настроены; их живое состояние —
в нестандартном поле `cortiq` (клиенты его игнорируют). По здоровью они не фильтруются:
крупный `.cmf` грузится минутами, и модель, исчезающая из списка на время прогрева, хуже
модели, помеченной «ещё не готова».
---
## 5. Anthropic Messages