fix(openai): strict OpenAI wire compatibility — Open WebUI can chat again
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
/v1/chat/completions answered ordinary client payloads with a bare HTTP 422 (a plain-text body no OpenAI client can parse), so Open WebUI showed neither an answer nor an error — it just hung. Parse the body leniently instead: content as string/null/content-part array, tools: null, stream: null, a string reasoning_effort, max_completion_tokens, messages[].name and tool_call_id, and a missing Content-Type. Unreadable bodies now come back as an OpenAI error envelope with 400. Responses are strict OpenAI JSON: no "tool_calls": null, finish_reason mapped to the spec's closed set (cortiq serve's max_tokens -> length), never an empty id, and `model` echoes the id the client requested — the model that actually answered stays in cortiq.selected_model and X-Cortiq-Selected-Model. Streaming is re-normalised rather than forwarded verbatim: a leading role delta, message-shaped chunks converted to delta, one terminal finish_reason, always a trailing data: [DONE], and an SSE error event when the upstream dies mid-flight. Adds X-Accel-Buffering: no so a fronting proxy cannot buffer the stream into a hang. GET /v1/models now lists managed [[cmf.servers]] models with their live running/healthy state, so clients can pin a local .cmf directly. Also strips client bookkeeping keys (chat_id, session_id, metadata, features, ...) from the upstream call and drops the leftover [to_wire] debug prints.
This commit is contained in:
+40
-9
@@ -53,7 +53,17 @@ Routing rule common to all protocols:
|
||||
```
|
||||
|
||||
Standard OpenAI fields are supported; unknown fields are proxied to the provider unchanged
|
||||
(forward-compatible).
|
||||
(forward-compatible), minus the bookkeeping keys chat UIs attach to the payload
|
||||
(`chat_id`, `session_id`, `metadata`, `features`, …), which are dropped so strict upstreams
|
||||
do not reject the call.
|
||||
|
||||
The parser is deliberately lenient about the shapes real clients send:
|
||||
`content` may be a string, `null`, or an array of content parts (text parts are
|
||||
concatenated, images and other non-text parts are skipped); `tools` may be `null`;
|
||||
`reasoning_effort` may be a string (`low` | `medium` | `high`) or a token budget;
|
||||
`max_completion_tokens` is accepted as an alias for `max_tokens`. A body that still
|
||||
cannot be read comes back as an OpenAI error envelope with HTTP 400 — never a bare 422,
|
||||
which OpenAI clients cannot parse (they show no answer at all).
|
||||
|
||||
### Response (non-streaming)
|
||||
|
||||
@@ -62,12 +72,12 @@ Standard OpenAI fields are supported; unknown fields are proxied to the provider
|
||||
"id": "chatcmpl-9f...",
|
||||
"object": "chat.completion",
|
||||
"created": 1718800000,
|
||||
"model": "local-qwen", // the ACTUALLY selected model
|
||||
"model": "cortiq-auto", // the model the CLIENT asked for, echoed back
|
||||
"choices": [
|
||||
{
|
||||
"index": 0,
|
||||
"message": { "role": "assistant", "content": "x = 2 or x = 3" },
|
||||
"finish_reason": "stop"
|
||||
"finish_reason": "stop" // always one of stop|length|tool_calls|content_filter
|
||||
}
|
||||
],
|
||||
"usage": {
|
||||
@@ -86,10 +96,24 @@ Standard OpenAI fields are supported; unknown fields are proxied to the provider
|
||||
}
|
||||
```
|
||||
|
||||
> `model` echoes the id the client requested, exactly as OpenAI does — clients match the
|
||||
> reply against the id they sent. The model that actually answered is in
|
||||
> `cortiq.selected_model` and in the `X-Cortiq-Selected-Model` header.
|
||||
|
||||
> Optional fields are omitted rather than sent as `null`: absent tool calls mean no
|
||||
> `tool_calls` key at all. `"tool_calls": null` is not in the OpenAI schema and trips
|
||||
> strict clients.
|
||||
|
||||
> The `cortiq` field is an optional extension. OpenAI SDKs ignore it. If the client does
|
||||
> not want extra fields, disable `cortiq.echo` — the metadata will remain in the response
|
||||
> headers.
|
||||
|
||||
Streaming (`stream: true`) emits strict `chat.completion.chunk` events: the stream always
|
||||
opens with a `role` delta, carries exactly one terminal `finish_reason`, and ends with
|
||||
`data: [DONE]` — even when the upstream stops early or dies mid-flight, in which case the
|
||||
failure arrives as an SSE `error` event. A client can therefore never be left waiting on a
|
||||
half-formed stream.
|
||||
|
||||
### Metadata in response headers (always present)
|
||||
|
||||
```
|
||||
@@ -164,20 +188,27 @@ single user message and follows the same pipeline. Response `object` is `"text_c
|
||||
## 4. OpenAI Models
|
||||
|
||||
`GET /v1/models` — auto-discovery for clients. Returns the gateway's virtual models
|
||||
(`cortiq-auto`, `cortiq-embed`) and all real models in the pool.
|
||||
(`cortiq-auto`, `cortiq-embed`), every model in the `[[models]]` pool, and every managed
|
||||
local `[[cmf.servers]]` model, so a client can pin a local `.cmf` directly.
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"object": "list",
|
||||
"data": [
|
||||
{ "id": "cortiq-auto", "object": "model", "owned_by": "cortiq-gateway" },
|
||||
{ "id": "local-qwen", "object": "model", "owned_by": "cortiq-gateway" },
|
||||
{ "id": "gpt-4o-mini", "object": "model", "owned_by": "cortiq-gateway" },
|
||||
{ "id": "claude-opus", "object": "model", "owned_by": "cortiq-gateway" }
|
||||
{ "id": "cortiq-auto", "object": "model", "created": 1718800000, "owned_by": "cortiq" },
|
||||
{ "id": "gpt-4o-mini", "object": "model", "created": 1718800000, "owned_by": "openai" },
|
||||
{ "id": "claude-opus", "object": "model", "created": 1718800000, "owned_by": "anthropic" },
|
||||
{ "id": "cmf-qwen3-4b", "object": "model", "created": 1718800000, "owned_by": "cortiq-cmf",
|
||||
"cortiq": { "kind": "cmf", "local": true, "running": true, "healthy": true } }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Managed CMF models are listed as soon as they are configured, with their live state in the
|
||||
non-standard `cortiq` field (clients ignore it). They are not filtered by health: a large
|
||||
`.cmf` takes minutes to load, and a model that disappears from the list while it warms up
|
||||
is worse than one marked "not ready yet".
|
||||
|
||||
---
|
||||
|
||||
## 5. Anthropic Messages
|
||||
@@ -198,7 +229,7 @@ single user message and follows the same pipeline. Response `object` is `"text_c
|
||||
"id": "msg_...",
|
||||
"type": "message",
|
||||
"role": "assistant",
|
||||
"model": "local-qwen",
|
||||
"model": "cortiq-auto",
|
||||
"content": [ { "type": "text", "text": "x = 2 or x = 3" } ],
|
||||
"stop_reason": "end_turn",
|
||||
"usage": { "input_tokens": 24, "output_tokens": 11 }
|
||||
|
||||
+38
-8
@@ -53,7 +53,17 @@
|
||||
```
|
||||
|
||||
Поддерживаются стандартные поля OpenAI; неизвестные поля проксируются к провайдеру
|
||||
без изменений (forward-compatible).
|
||||
без изменений (forward-compatible) — кроме служебных ключей, которые чат-интерфейсы
|
||||
добавляют в payload для собственных нужд (`chat_id`, `session_id`, `metadata`,
|
||||
`features`, …): они отбрасываются, чтобы строгий upstream не отклонил вызов.
|
||||
|
||||
Парсер намеренно терпим к тем формам, которые реально шлют клиенты: `content` может быть
|
||||
строкой, `null` или массивом content-частей (текстовые части склеиваются, изображения и
|
||||
прочие нетекстовые пропускаются); `tools` может быть `null`; `reasoning_effort` — строкой
|
||||
(`low` | `medium` | `high`) или бюджетом токенов; `max_completion_tokens` принимается как
|
||||
синоним `max_tokens`. Если тело всё-таки не читается, возвращается OpenAI-конверт ошибки с
|
||||
HTTP 400 — но не «голый» 422, который OpenAI-клиенты не умеют разбирать (для пользователя
|
||||
это выглядит как отсутствие ответа).
|
||||
|
||||
### Ответ (без стриминга)
|
||||
|
||||
@@ -62,12 +72,12 @@
|
||||
"id": "chatcmpl-9f...",
|
||||
"object": "chat.completion",
|
||||
"created": 1718800000,
|
||||
"model": "local-qwen", // РЕАЛЬНО выбранная модель
|
||||
"model": "cortiq-auto", // модель, которую ЗАПРОСИЛ клиент — возвращается как есть
|
||||
"choices": [
|
||||
{
|
||||
"index": 0,
|
||||
"message": { "role": "assistant", "content": "x = 2 или x = 3" },
|
||||
"finish_reason": "stop"
|
||||
"finish_reason": "stop" // всегда одно из stop|length|tool_calls|content_filter
|
||||
}
|
||||
],
|
||||
"usage": {
|
||||
@@ -86,9 +96,22 @@
|
||||
}
|
||||
```
|
||||
|
||||
> `model` возвращает тот id, который запросил клиент, — ровно как это делает OpenAI:
|
||||
> клиенты сопоставляют ответ с отправленным id. Модель, которая реально ответила, — в
|
||||
> `cortiq.selected_model` и в заголовке `X-Cortiq-Selected-Model`.
|
||||
|
||||
> Необязательные поля не сериализуются вместо того, чтобы приходить как `null`: если
|
||||
> tool-вызовов нет, ключа `tool_calls` в ответе нет вообще. `"tool_calls": null` не входит
|
||||
> в схему OpenAI и ломает строгие клиенты.
|
||||
|
||||
> Поле `cortiq` — необязательное расширение. OpenAI-SDK его игнорируют. Если клиент
|
||||
> не хочет «лишних» полей — выключите `cortiq.echo`, метаданные останутся в заголовках.
|
||||
|
||||
Стриминг (`stream: true`) отдаёт строгие события `chat.completion.chunk`: поток всегда
|
||||
открывается дельтой с `role`, содержит ровно один финальный `finish_reason` и завершается
|
||||
`data: [DONE]` — даже если upstream оборвался или упал на середине (тогда ошибка приходит
|
||||
SSE-событием `error`). Клиент никогда не остаётся ждать недосказанный поток.
|
||||
|
||||
### Метаданные в заголовках (всегда)
|
||||
|
||||
```
|
||||
@@ -163,20 +186,27 @@ data: [DONE]
|
||||
## 4. OpenAI Models
|
||||
|
||||
`GET /v1/models` — автодискавери для клиентов. Возвращает виртуальные модели шлюза
|
||||
(`cortiq-auto`, `cortiq-embed`) и все реальные модели пула.
|
||||
(`cortiq-auto`, `cortiq-embed`), все модели пула `[[models]]` и все управляемые локальные
|
||||
модели `[[cmf.servers]]` — чтобы клиент мог напрямую закрепить локальный `.cmf`.
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"object": "list",
|
||||
"data": [
|
||||
{ "id": "cortiq-auto", "object": "model", "owned_by": "cortiq-gateway" },
|
||||
{ "id": "local-qwen", "object": "model", "owned_by": "cortiq-gateway" },
|
||||
{ "id": "gpt-4o-mini", "object": "model", "owned_by": "cortiq-gateway" },
|
||||
{ "id": "claude-opus", "object": "model", "owned_by": "cortiq-gateway" }
|
||||
{ "id": "cortiq-auto", "object": "model", "created": 1718800000, "owned_by": "cortiq" },
|
||||
{ "id": "gpt-4o-mini", "object": "model", "created": 1718800000, "owned_by": "openai" },
|
||||
{ "id": "claude-opus", "object": "model", "created": 1718800000, "owned_by": "anthropic" },
|
||||
{ "id": "cmf-qwen3-4b", "object": "model", "created": 1718800000, "owned_by": "cortiq-cmf",
|
||||
"cortiq": { "kind": "cmf", "local": true, "running": true, "healthy": true } }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Управляемые CMF-модели попадают в список сразу, как только настроены; их живое состояние —
|
||||
в нестандартном поле `cortiq` (клиенты его игнорируют). По здоровью они не фильтруются:
|
||||
крупный `.cmf` грузится минутами, и модель, исчезающая из списка на время прогрева, хуже
|
||||
модели, помеченной «ещё не готова».
|
||||
|
||||
---
|
||||
|
||||
## 5. Anthropic Messages
|
||||
|
||||
Reference in New Issue
Block a user