github/cortiq-gateway
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: honest costs in streams, oracle-sized timeouts, history & log cleanup

Browse Source 
Cost showed $0 for every streamed request: the X-Cortiq-Cost-USD header is
written before generation, when the cost cannot be known. The gateway now
ships the final number in a closing SSE event (`cortiq_cost_usd`, alongside
the token estimate when the provider sent no usage), and the console prefers
it over the header. Money renders with 4 decimals, the last two in a smaller
font — $0.0006 stays readable instead of collapsing to $0.

Timeouts, both ends: the router default was 800 ms, silently degrading every
prompt the hosted router escalates to its oracle (~10 s) — now 15 s. Provider
clients had NO timeout at all (a dead OpenRouter host hung requests forever) —
now a 10 s connect timeout, with no overall cap so long generations and
streams are never cut.

Benchmark history: per-run × and a Clear-all button. Logs view: a clear
button backed by DELETE /admin/api/logs.
This commit is contained in:
infosave2007
2026-08-07 19:15:27 +03:00
parent ec92bdddb7
commit 54b73f1365
20 changed files with 120 additions and 38 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/pipeline.rs
+24 -17
View File
@@ -798,28 +798,35 @@ fn tap_stream(
// inject one final OpenAI-shaped usage event with the gateway's
// estimate (~4 chars/token), flagged `cortiq_estimated` so nobody
// mistakes it for the tokenizer's truth.
if done_marker && !got_usage {
let est_completion = if delta_chunks > 0 {
delta_chunks
} else {
(delta_chars / 4) as u32
};
let est_prompt = (prompt_chars / 4) as u32;
let evt = serde_json::json!({
if done_marker {
// The initial X-Cortiq-Cost-USD header is always 0 — cost is
// only known after generation. Ship the final number (and the
// token estimate when the provider sent none) ahead of [DONE].
if !got_usage {
prompt_tokens = (prompt_chars / 4) as u32;
completion_tokens = if delta_chunks > 0 {
delta_chunks
} else {
(delta_chars / 4) as u32
};
}
let cost = provider.price(prompt_tokens, completion_tokens);
let mut evt = serde_json::json!({
"object": "chat.completion.chunk",
"model": info.selected_model,
"choices": [],
"usage": {
"prompt_tokens": est_prompt,
"completion_tokens": est_completion,
"total_tokens": est_prompt + est_completion,
},
"cortiq_estimated": true,
"cortiq_cost_usd": cost,
});
if !got_usage {
evt["usage"] = serde_json::json!({
"prompt_tokens": prompt_tokens,
"completion_tokens": completion_tokens,
"total_tokens": prompt_tokens + completion_tokens,
});
evt["cortiq_estimated"] = serde_json::json!(true);
got_usage = true;
}
yield Ok(bytes::Bytes::from(format!("data: {evt}\n\n")));
prompt_tokens = est_prompt;
completion_tokens = est_completion;
got_usage = true;
}
yield item;
}