fix(openai): strict OpenAI wire compatibility — Open WebUI can chat again
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
/v1/chat/completions answered ordinary client payloads with a bare HTTP 422 (a plain-text body no OpenAI client can parse), so Open WebUI showed neither an answer nor an error — it just hung. Parse the body leniently instead: content as string/null/content-part array, tools: null, stream: null, a string reasoning_effort, max_completion_tokens, messages[].name and tool_call_id, and a missing Content-Type. Unreadable bodies now come back as an OpenAI error envelope with 400. Responses are strict OpenAI JSON: no "tool_calls": null, finish_reason mapped to the spec's closed set (cortiq serve's max_tokens -> length), never an empty id, and `model` echoes the id the client requested — the model that actually answered stays in cortiq.selected_model and X-Cortiq-Selected-Model. Streaming is re-normalised rather than forwarded verbatim: a leading role delta, message-shaped chunks converted to delta, one terminal finish_reason, always a trailing data: [DONE], and an SSE error event when the upstream dies mid-flight. Adds X-Accel-Buffering: no so a fronting proxy cannot buffer the stream into a hang. GET /v1/models now lists managed [[cmf.servers]] models with their live running/healthy state, so clients can pin a local .cmf directly. Also strips client bookkeeping keys (chat_id, session_id, metadata, features, ...) from the upstream call and drops the leftover [to_wire] debug prints.
This commit is contained in:
@@ -1,5 +1,10 @@
|
||||
//! Inbound OpenAI Models adapter: `GET /v1/models`.
|
||||
//! Lists the configured pool (plus the virtual `cortiq-auto`) for client discovery.
|
||||
//! Lists everything a client can address: the virtual `cortiq-auto`, the
|
||||
//! configured `[[models]]` pool, and the managed local `[[cmf.servers]]`.
|
||||
//!
|
||||
//! The managed CMF servers belong here because they are routable model ids like
|
||||
//! any other — omitting them left OpenAI clients (Open WebUI among them) unable
|
||||
//! to pin a local model even though the gateway would happily serve it.
|
||||
|
||||
use crate::error::{GatewayError, Result};
|
||||
use crate::state::SharedState;
|
||||
@@ -9,6 +14,13 @@ pub fn routes() -> Router<SharedState> {
|
||||
Router::new().route("/v1/models", get(handler))
|
||||
}
|
||||
|
||||
fn now_secs() -> u64 {
|
||||
std::time::SystemTime::now()
|
||||
.duration_since(std::time::UNIX_EPOCH)
|
||||
.unwrap_or_default()
|
||||
.as_secs()
|
||||
}
|
||||
|
||||
async fn handler(State(state): State<SharedState>) -> Result<impl IntoResponse> {
|
||||
let live = state.live();
|
||||
if !live.cfg.protocols.openai_models {
|
||||
@@ -16,17 +28,51 @@ async fn handler(State(state): State<SharedState>) -> Result<impl IntoResponse>
|
||||
"openai_models protocol is disabled".into(),
|
||||
));
|
||||
}
|
||||
let created = now_secs();
|
||||
let mut data = vec![serde_json::json!({
|
||||
"id": "cortiq-auto",
|
||||
"object": "model",
|
||||
"created": created,
|
||||
"owned_by": "cortiq",
|
||||
})];
|
||||
let mut seen: std::collections::HashSet<String> =
|
||||
std::collections::HashSet::from(["cortiq-auto".to_string()]);
|
||||
|
||||
for m in &live.cfg.models {
|
||||
if !seen.insert(m.id.clone()) {
|
||||
continue;
|
||||
}
|
||||
data.push(serde_json::json!({
|
||||
"id": m.id,
|
||||
"object": "model",
|
||||
"created": created,
|
||||
"owned_by": m.provider,
|
||||
}));
|
||||
}
|
||||
|
||||
// Managed local CMF servers. Health is reported alongside (a non-standard
|
||||
// field clients ignore) rather than used as a filter: a large `.cmf` takes
|
||||
// minutes to load, and a model that vanishes from the list while it warms up
|
||||
// is worse than one marked "not ready yet".
|
||||
let cmf = state.cmf.status();
|
||||
for s in live.cfg.cmf.effective_servers() {
|
||||
if !seen.insert(s.id.clone()) {
|
||||
continue; // a static [[models]] entry with the same id wins
|
||||
}
|
||||
let status = cmf.servers.iter().find(|st| st.id == s.id);
|
||||
data.push(serde_json::json!({
|
||||
"id": s.id,
|
||||
"object": "model",
|
||||
"created": created,
|
||||
"owned_by": "cortiq-cmf",
|
||||
"cortiq": {
|
||||
"kind": "cmf",
|
||||
"local": true,
|
||||
"running": status.map(|st| st.running).unwrap_or(false),
|
||||
"healthy": status.map(|st| st.healthy).unwrap_or(false),
|
||||
},
|
||||
}));
|
||||
}
|
||||
|
||||
Ok(Json(serde_json::json!({ "object": "list", "data": data })))
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user