Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
 infosave2007
|
0b071279ac | ||
|
infosave2007
|
9fd4bdaa85 | ||
|
infosave2007
|
9c15109f26 |
@@ -3,6 +3,65 @@
|
||||
All notable changes to this project are documented here.
|
||||
The format is based on [Keep a Changelog](https://keepachangelog.com/).
|
||||
|
||||
## [0.2.49] - 2026-08-21
|
||||
|
||||
### Fixed
|
||||
- **The SSE stream now has the same shape every hosted provider sends.** Two choice-less
|
||||
chunks used to ride along on every stream: `usage` (emitted by the upstream *before* the
|
||||
stop reason) and a `cortiq_cost_usd` chunk of the gateway's own. OpenAI ships the usage
|
||||
chunk only when the client sets `stream_options: {"include_usage": true}` — and chat UIs
|
||||
do not set it, which is why a Groq stream carries neither. The gateway now honours that
|
||||
flag: with no `stream_options`, a stream is exactly
|
||||
`role → content… → tool_calls → finish_reason → [DONE]` with no choice-less chunks at
|
||||
all. When usage *is* requested it ships once, **after** the stop reason, with
|
||||
`cortiq_cost_usd` folded into the same chunk instead of trailing behind it.
|
||||
|
||||
Token accounting, cost and the `X-Cortiq-*` headers are unchanged — the numbers were
|
||||
never taken from those chunks.
|
||||
|
||||
## [0.2.48] - 2026-08-21
|
||||
|
||||
Verified end to end: Open WebUI 0.11.0 → gateway → managed CMF/Qwen, native
|
||||
function calling, tool executed and its result fed back into the reply.
|
||||
|
||||
### Fixed
|
||||
- **A tool call is never delivered twice.** Qwen-family templates write the call into the
|
||||
generated text as `<tool_call>{…}</tool_call>`; some runtimes parse it into structured
|
||||
`tool_calls` *and* still stream the raw markup as `content`, so the client sees the call
|
||||
once as prose it must not show and once as the real thing. The gateway now strips the
|
||||
markup from `content` — including when the marker is split across SSE chunks — while
|
||||
passing the structured call through untouched. Ordinary text containing `<` is not
|
||||
affected, and an unterminated block is handed back as text rather than swallowed.
|
||||
- **`finish_reason` agrees with the reply.** A response carrying tool calls now always ends
|
||||
with `finish_reason: "tool_calls"`, whatever the upstream called it — clients branch on
|
||||
that field to decide whether to run the tool.
|
||||
- **Streamed `tool_calls` are brought to the OpenAI shape**: `index` on every entry, `id`
|
||||
and `type` exactly once per call (continuation deltas that carry only an argument
|
||||
fragment are left alone, as the spec requires), and `function.arguments` always a string
|
||||
even when the runtime sends an object.
|
||||
|
||||
### Added
|
||||
- **Unparsed `<tool_call>` markup is promoted into a real call.** When the upstream never
|
||||
parsed it, the gateway turns it into a proper `tool_calls` entry with a generated
|
||||
`call_…` id, so the call is executable instead of decorative. Markup that is not a JSON
|
||||
object with a `name` is ignored — a stray marker must not become a bogus call.
|
||||
|
||||
## [0.2.47] - 2026-08-21
|
||||
|
||||
### Changed
|
||||
- **The hosted allaigate router is addressed by hostname:
|
||||
`https://router.allaigate.com`** is now the default `[router].url`, and
|
||||
`verify_tls` defaults to `true`. The endpoint used to be reachable only by bare IP,
|
||||
whose certificate is issued for the hostname — so every hosted setup had to run with
|
||||
certificate verification switched off. It no longer does.
|
||||
- **Existing configs are migrated on load.** A `[router].url` still pointing at the old
|
||||
`138.226.222.209` (with or without a scheme or trailing slash) is rewritten to the
|
||||
hostname, `verify_tls` is turned back on, and the file is saved — the endpoint is the
|
||||
same machine, and this is the difference between a verified connection and a blind one.
|
||||
A self-hosted router URL is never touched.
|
||||
- README, `docs/ROUTER.md` (both languages), `config/gateway.example.toml` and
|
||||
`bench/accuracy.py` updated to the hostname.
|
||||
|
||||
## [0.2.46] - 2026-08-21
|
||||
|
||||
Verified end to end against a real Open WebUI container talking to the gateway.
|
||||
|
||||
Generated
+1
-1
@@ -285,7 +285,7 @@ checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570"
|
||||
|
||||
[[package]]
|
||||
name = "cortiq-gateway"
|
||||
version = "0.2.46"
|
||||
version = "0.2.49"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"arc-swap",
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
[package]
|
||||
name = "cortiq-gateway"
|
||||
version = "0.2.46"
|
||||
version = "0.2.49"
|
||||
edition = "2021"
|
||||
rust-version = "1.75"
|
||||
license = "Apache-2.0"
|
||||
|
||||
@@ -175,9 +175,9 @@ print(resp.choices[0].message.content)
|
||||
`model: "cortiq-auto"` enables routing. Any **real** model name from the config
|
||||
(`"gpt-4o-mini"`, `"local-qwen"`) is a direct passthrough, without routing.
|
||||
|
||||
> **Using the hosted allaigate router?** Set `url = "https://138.226.222.209"`,
|
||||
> `verify_tls = false`, `taxonomy_id = "data-assistant"`, and a `cortiq_…` key in
|
||||
> `CORTIQ_ROUTER_KEY`. On hard prompts the router escalates to an oracle (~10 s), so
|
||||
> **Using the hosted allaigate router?** It is the default: `url =
|
||||
> "https://router.allaigate.com"` with `verify_tls = true`. Set `taxonomy_id =
|
||||
> "data-assistant"` and a `cortiq_…` key in `CORTIQ_ROUTER_KEY`. On hard prompts the router escalates to an oracle (~10 s), so
|
||||
> set `timeout_ms = 12000+` — otherwise the gateway gracefully degrades to the default model.
|
||||
|
||||
---
|
||||
|
||||
+3
-3
@@ -167,9 +167,9 @@ print(resp.choices[0].message.content)
|
||||
`model: "cortiq-auto"` включает роутинг. Любое **реальное** имя модели из конфига
|
||||
(`"gpt-4o-mini"`, `"local-qwen"`) — это passthrough напрямую, без роутинга.
|
||||
|
||||
> **Используете хостовый allaigate-роутер?** Укажите `url = "https://138.226.222.209"`,
|
||||
> `verify_tls = false`, `taxonomy_id = "data-assistant"` и ключ `cortiq_…` в
|
||||
> `CORTIQ_ROUTER_KEY`. На сложных запросах роутер эскалирует к oracle (~10 с), поэтому
|
||||
> **Используете хостовый allaigate-роутер?** Он стоит по умолчанию: `url =
|
||||
> "https://router.allaigate.com"`, `verify_tls = true`. Задайте `taxonomy_id =
|
||||
> "data-assistant"` и ключ `cortiq_…` в `CORTIQ_ROUTER_KEY`. На сложных запросах роутер эскалирует к oracle (~10 с), поэтому
|
||||
> ставьте `timeout_ms = 12000+` — иначе шлюз мягко деградирует на default-модель.
|
||||
|
||||
---
|
||||
|
||||
+2
-2
@@ -4,12 +4,12 @@ heuristic (what you get without a semantic classifier, e.g. with LiteLLM/Portkey
|
||||
|
||||
CORTIQ_ROUTER_KEY=cortiq_... python3 bench/accuracy.py
|
||||
|
||||
Env: CORTIQ_ROUTER_KEY (required), ROUTER_URL (default https://138.226.222.209),
|
||||
Env: CORTIQ_ROUTER_KEY (required), ROUTER_URL (default https://router.allaigate.com),
|
||||
TAXONOMY (default data-assistant).
|
||||
"""
|
||||
import json, os, ssl, sys, urllib.request
|
||||
|
||||
ROUTER_URL = os.environ.get("ROUTER_URL", "https://138.226.222.209").rstrip("/")
|
||||
ROUTER_URL = os.environ.get("ROUTER_URL", "https://router.allaigate.com").rstrip("/")
|
||||
TAXONOMY = os.environ.get("TAXONOMY", "data-assistant")
|
||||
KEY = os.environ.get("CORTIQ_ROUTER_KEY", "")
|
||||
CTX = ssl._create_unverified_context()
|
||||
|
||||
@@ -8,13 +8,13 @@
|
||||
listen = "0.0.0.0:9000"
|
||||
|
||||
# --- connection to cortiq-router (decides task type / complexity) -----------
|
||||
# Hosted allaigate router: url = "https://138.226.222.209", verify_tls = false
|
||||
# Hosted allaigate router: url = "https://router.allaigate.com" (the default)
|
||||
# (cert is bound to an IP), taxonomy_id = "data-assistant", a cortiq_… key in
|
||||
# CORTIQ_ROUTER_KEY.
|
||||
# NOTE: on hard prompts the router escalates to an oracle LLM (~10 s) — for the
|
||||
# hosted router set timeout_ms = 12000+, otherwise the gateway degrades to default.
|
||||
[router]
|
||||
url = "https://138.226.222.209" # hosted allaigate router (or your own cortiq-router URL)
|
||||
url = "https://router.allaigate.com" # hosted allaigate router (or your own cortiq-router URL)
|
||||
api_key_env = "CORTIQ_ROUTER_KEY" # name of the env var holding the router key
|
||||
timeout_ms = 12000 # hard prompts escalate to an oracle (~10 s)
|
||||
taxonomy_id = "data-assistant" # the hosted router's task taxonomy
|
||||
|
||||
@@ -121,6 +121,26 @@ opens with a `role` delta, carries exactly one terminal `finish_reason`, and end
|
||||
failure arrives as an SSE `error` event. A client can therefore never be left waiting on a
|
||||
half-formed stream.
|
||||
|
||||
Streaming carries no choice-less chunks unless asked: `usage` is emitted only when the
|
||||
request sets `stream_options: {"include_usage": true}`, and then exactly once, after the
|
||||
`finish_reason` chunk, with `cortiq_cost_usd` folded in. Without it a stream is
|
||||
`role → content… → tool_calls → finish_reason → [DONE]` and nothing else — the shape the
|
||||
hosted providers send.
|
||||
|
||||
### Tool calls
|
||||
|
||||
`tools` and `tool_calls` are proxied through, with the wire kept clean:
|
||||
|
||||
* raw `<tool_call>…</tool_call>` markup (how Qwen-family templates express a call in the
|
||||
generated text) is stripped from `content`, so a client never receives the same call
|
||||
twice — once as prose and once structured. Markers split across SSE chunks are handled;
|
||||
ordinary text containing `<` is not touched;
|
||||
* when the upstream never parsed that markup, it is promoted into a real `tool_calls`
|
||||
entry with a generated `call_…` id, so the call is executable rather than decorative;
|
||||
* streamed entries carry `index` always, `id`/`type` once per call, and
|
||||
`function.arguments` as a string even if the runtime sent an object;
|
||||
* a reply carrying tool calls always ends with `finish_reason: "tool_calls"`.
|
||||
|
||||
### Metadata in response headers (always present)
|
||||
|
||||
```
|
||||
|
||||
@@ -119,6 +119,27 @@ HTTP 400 — но не «голый» 422, который OpenAI-клиенты
|
||||
`data: [DONE]` — даже если upstream оборвался или упал на середине (тогда ошибка приходит
|
||||
SSE-событием `error`). Клиент никогда не остаётся ждать недосказанный поток.
|
||||
|
||||
В стриме нет чанков без `choices`, если их не просили: `usage` отдаётся, только когда
|
||||
запрос содержит `stream_options: {"include_usage": true}`, и тогда ровно один раз — после
|
||||
чанка с `finish_reason`, с вложенным `cortiq_cost_usd`. Без него поток выглядит как
|
||||
`role → content… → tool_calls → finish_reason → [DONE]` и ничего больше — ровно так же,
|
||||
как у хостовых провайдеров.
|
||||
|
||||
### Tool calls
|
||||
|
||||
`tools` и `tool_calls` проксируются, но провод держится чистым:
|
||||
|
||||
* сырая разметка `<tool_call>…</tool_call>` (так Qwen-подобные шаблоны выражают вызов
|
||||
прямо в тексте) вырезается из `content` — клиент никогда не получает один и тот же
|
||||
вызов дважды, прозой и структурой. Маркер, разорванный между SSE-чанками, обрабатывается
|
||||
корректно; обычный текст с символом `<` не трогается;
|
||||
* если апстрим эту разметку не распарсил, она повышается до настоящего элемента
|
||||
`tool_calls` со сгенерированным `call_…` id — вызов становится исполнимым, а не
|
||||
декоративным;
|
||||
* в стриме у каждого элемента всегда есть `index`, `id`/`type` — ровно один раз на вызов,
|
||||
а `function.arguments` — всегда строка, даже если рантайм прислал объект;
|
||||
* ответ с tool-вызовами всегда завершается `finish_reason: "tool_calls"`.
|
||||
|
||||
### Метаданные в заголовках (всегда)
|
||||
|
||||
```
|
||||
|
||||
+4
-4
@@ -41,11 +41,11 @@ minted and shown to you.
|
||||
|
||||
```toml
|
||||
[router]
|
||||
url = "https://138.226.222.209" # allaigate routing API
|
||||
verify_tls = false # cert is bound to an IP
|
||||
url = "https://router.allaigate.com" # allaigate routing API (the default)
|
||||
verify_tls = true # valid certificate — keep verification on
|
||||
taxonomy_id = "data-assistant"
|
||||
api_key_env = "CORTIQ_ROUTER_KEY" # your cortiq_… key
|
||||
timeout_ms = 15000 # allow for oracle escalation on hard prompts
|
||||
api_key_env = "CORTIQ_ROUTER_KEY" # your cortiq_… key
|
||||
timeout_ms = 15000 # allow for oracle escalation on hard prompts
|
||||
```
|
||||
|
||||
```bash
|
||||
|
||||
+4
-4
@@ -40,11 +40,11 @@ Cortiq Gateway — тонкий и быстрый прокси; **интелле
|
||||
|
||||
```toml
|
||||
[router]
|
||||
url = "https://138.226.222.209" # routing API allaigate
|
||||
verify_tls = false # серт привязан к IP
|
||||
url = "https://router.allaigate.com" # routing API allaigate (значение по умолчанию)
|
||||
verify_tls = true # валидный сертификат — проверку не выключаем
|
||||
taxonomy_id = "data-assistant"
|
||||
api_key_env = "CORTIQ_ROUTER_KEY" # ваш ключ cortiq_…
|
||||
timeout_ms = 15000 # запас под oracle-эскалацию на сложных запросах
|
||||
api_key_env = "CORTIQ_ROUTER_KEY" # ваш ключ cortiq_…
|
||||
timeout_ms = 15000 # запас под oracle-эскалацию на сложных запросах
|
||||
```
|
||||
|
||||
```bash
|
||||
|
||||
+71
-3
@@ -318,6 +318,34 @@ fn default_verify_tls() -> bool {
|
||||
true
|
||||
}
|
||||
|
||||
/// The hosted allaigate routing API.
|
||||
pub const DEFAULT_ROUTER_URL: &str = "https://router.allaigate.com";
|
||||
|
||||
/// The bare IP the hosted router used to be addressed by. It served a
|
||||
/// certificate for `router.allaigate.com`, so every config pointing at it had to
|
||||
/// turn TLS verification off. The hostname now resolves, so configs written
|
||||
/// before it existed are moved over on load — the endpoint is the same machine,
|
||||
/// and this is the difference between a verified connection and a blind one.
|
||||
const LEGACY_ROUTER_HOSTS: &[&str] = &[
|
||||
"https://138.226.222.209",
|
||||
"http://138.226.222.209",
|
||||
"138.226.222.209",
|
||||
];
|
||||
|
||||
impl RouterCfg {
|
||||
/// Point a legacy IP-addressed router config at the hostname and restore TLS
|
||||
/// verification. Returns true when something changed.
|
||||
fn migrate_hosted_endpoint(&mut self) -> bool {
|
||||
let url = self.url.trim().trim_end_matches('/');
|
||||
if !LEGACY_ROUTER_HOSTS.contains(&url) {
|
||||
return false;
|
||||
}
|
||||
self.url = DEFAULT_ROUTER_URL.to_string();
|
||||
self.verify_tls = true;
|
||||
true
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Serialize, Deserialize)]
|
||||
pub struct RouteCfg {
|
||||
#[serde(default = "default_text_strategy")]
|
||||
@@ -882,9 +910,9 @@ impl Default for Config {
|
||||
// models and the console asks for one; nothing breaks.
|
||||
router: RouterCfg {
|
||||
enabled: true,
|
||||
url: "https://138.226.222.209".to_string(),
|
||||
url: DEFAULT_ROUTER_URL.to_string(),
|
||||
timeout_ms: 15_000, // hard prompts escalate to an oracle (~10 s)
|
||||
verify_tls: false, // the hosted endpoint is IP-bound
|
||||
verify_tls: true,
|
||||
api_key_env: Some("CORTIQ_ROUTER_KEY".to_string()),
|
||||
taxonomy_id: Some("data-assistant".to_string()),
|
||||
},
|
||||
@@ -954,7 +982,14 @@ impl Config {
|
||||
pub fn load(path: &str) -> anyhow::Result<Self> {
|
||||
let raw = std::fs::read_to_string(path)
|
||||
.map_err(|e| anyhow::anyhow!("cannot read config {path}: {e}"))?;
|
||||
let cfg: Config = toml::from_str(&raw)?;
|
||||
let mut cfg: Config = toml::from_str(&raw)?;
|
||||
if cfg.router.migrate_hosted_endpoint() {
|
||||
tracing::info!(
|
||||
url = DEFAULT_ROUTER_URL,
|
||||
"router endpoint moved from the bare IP to the hostname; TLS verification is on"
|
||||
);
|
||||
let _ = cfg.save(path);
|
||||
}
|
||||
cfg.validate()?;
|
||||
Ok(cfg)
|
||||
}
|
||||
@@ -1091,6 +1126,39 @@ impl Config {
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn the_hosted_router_defaults_to_the_hostname_with_tls_on() {
|
||||
let cfg = Config::default();
|
||||
assert_eq!(cfg.router.url, "https://router.allaigate.com");
|
||||
assert!(cfg.router.verify_tls);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_legacy_ip_router_config_moves_to_the_hostname() {
|
||||
for legacy in [
|
||||
"https://138.226.222.209",
|
||||
"https://138.226.222.209/",
|
||||
"138.226.222.209",
|
||||
] {
|
||||
let mut r = Config::default().router;
|
||||
r.url = legacy.to_string();
|
||||
r.verify_tls = false;
|
||||
assert!(r.migrate_hosted_endpoint(), "{legacy} must migrate");
|
||||
assert_eq!(r.url, DEFAULT_ROUTER_URL);
|
||||
assert!(r.verify_tls, "TLS verification must be restored");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_self_hosted_router_url_is_left_alone() {
|
||||
let mut r = Config::default().router;
|
||||
r.url = "http://127.0.0.1:8081".to_string();
|
||||
r.verify_tls = false;
|
||||
assert!(!r.migrate_hosted_endpoint());
|
||||
assert_eq!(r.url, "http://127.0.0.1:8081");
|
||||
assert!(!r.verify_tls);
|
||||
}
|
||||
|
||||
const SAMPLE: &str = r#"
|
||||
listen = "0.0.0.0:9000"
|
||||
|
||||
|
||||
@@ -29,6 +29,7 @@ mod secrets;
|
||||
mod state;
|
||||
mod stats;
|
||||
mod sysmon;
|
||||
mod toolcalls;
|
||||
|
||||
use axum::{routing::get, Router};
|
||||
use clap::Parser;
|
||||
|
||||
+245
-16
@@ -233,6 +233,16 @@ async fn handler(
|
||||
.unwrap_or("cortiq-auto")
|
||||
.to_string();
|
||||
let stream = req.stream.unwrap_or(false);
|
||||
// OpenAI only emits the trailing usage chunk when the client asks for it,
|
||||
// and chat UIs do not ask. Mirroring that keeps our stream byte-shaped like
|
||||
// every other provider's instead of carrying two extra choice-less chunks
|
||||
// that each client has to know to ignore.
|
||||
let include_usage = req
|
||||
.rest
|
||||
.get("stream_options")
|
||||
.and_then(|o| o.get("include_usage"))
|
||||
.and_then(|v| v.as_bool())
|
||||
.unwrap_or(false);
|
||||
|
||||
let canonical = ChatRequest {
|
||||
routing: parse_routing(&requested_model),
|
||||
@@ -260,7 +270,7 @@ async fn handler(
|
||||
if stream {
|
||||
let (info, raw) = state.pipeline.run_stream(canonical, &state).await?;
|
||||
let headers = sse_headers(&info);
|
||||
let normalized = normalize_sse(raw, requested_model);
|
||||
let normalized = normalize_sse(raw, requested_model, include_usage);
|
||||
return Ok((headers, axum::body::Body::from_stream(normalized)).into_response());
|
||||
}
|
||||
|
||||
@@ -341,9 +351,16 @@ async fn handler(
|
||||
/// to end with `data: [DONE]` — even when the upstream dies mid-flight, in
|
||||
/// which case the failure is delivered as an SSE `error` event so the client
|
||||
/// shows a message instead of waiting forever.
|
||||
///
|
||||
/// Tool calls are settled here too (see [`crate::toolcalls`]): raw
|
||||
/// `<tool_call>` markup never reaches `content`, structured deltas are brought
|
||||
/// to the OpenAI shape, markup is promoted to a real call when the upstream
|
||||
/// produced none, and a stream that emitted tool calls always finishes with
|
||||
/// `finish_reason: "tool_calls"`.
|
||||
fn normalize_sse(
|
||||
stream: crate::providers::ChatStream,
|
||||
model: String,
|
||||
include_usage: bool,
|
||||
) -> impl futures::Stream<Item = Result<bytes::Bytes>> + Send + 'static {
|
||||
use futures::StreamExt;
|
||||
async_stream::stream! {
|
||||
@@ -355,6 +372,16 @@ fn normalize_sse(
|
||||
let mut roled: std::collections::HashSet<u64> = std::collections::HashSet::new();
|
||||
let mut sent_finish = false;
|
||||
let mut failure: Option<String> = None;
|
||||
// Tool-call hygiene: one markup filter per choice index, the set of
|
||||
// tool-call indices already opened, and whether anything structured
|
||||
// was emitted at all.
|
||||
let mut markup: std::collections::HashMap<u64, crate::toolcalls::ToolMarkupFilter> =
|
||||
std::collections::HashMap::new();
|
||||
let mut tool_indices: std::collections::HashSet<u64> = std::collections::HashSet::new();
|
||||
let mut emitted_tool_calls = false;
|
||||
// `usage` / `cortiq_*` collected along the way, shipped after the finish.
|
||||
let mut tail_extras: std::collections::BTreeMap<&str, serde_json::Value> =
|
||||
std::collections::BTreeMap::new();
|
||||
|
||||
// Emit one normalised event; `None` means "nothing worth sending".
|
||||
macro_rules! chunk {
|
||||
@@ -417,8 +444,15 @@ fn normalize_sse(
|
||||
delta.insert("role".into(), serde_json::json!("assistant"));
|
||||
}
|
||||
if let Some(text) = src["content"].as_str() {
|
||||
if !text.is_empty() {
|
||||
delta.insert("content".into(), serde_json::json!(text));
|
||||
// Strip any `<tool_call>` markup before it can
|
||||
// reach the client as prose — the structured
|
||||
// call is the only form a tool loop may see.
|
||||
let visible = markup
|
||||
.entry(index)
|
||||
.or_default()
|
||||
.push(text);
|
||||
if !visible.is_empty() {
|
||||
delta.insert("content".into(), serde_json::json!(visible));
|
||||
}
|
||||
}
|
||||
// Reasoning traces and citations are not in the base
|
||||
@@ -430,10 +464,45 @@ fn normalize_sse(
|
||||
}
|
||||
}
|
||||
if let Some(tc) = src["tool_calls"].as_array().filter(|a| !a.is_empty()) {
|
||||
delta.insert("tool_calls".into(), serde_json::Value::Array(tc.clone()));
|
||||
let normalized =
|
||||
crate::toolcalls::normalize_delta_tool_calls(tc, &mut tool_indices);
|
||||
if !normalized.is_empty() {
|
||||
emitted_tool_calls = true;
|
||||
delta.insert(
|
||||
"tool_calls".into(),
|
||||
serde_json::Value::Array(normalized),
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
let finish = c["finish_reason"].as_str().map(normalize_finish_reason);
|
||||
|
||||
// Last chance to promote `<tool_call>` markup into a
|
||||
// real call: an upstream that never parsed it would
|
||||
// otherwise hand the client prose no tool loop runs.
|
||||
if finish.is_some() && !emitted_tool_calls {
|
||||
if let Some(filter) = markup.get_mut(&index) {
|
||||
let leftover = filter.finish();
|
||||
if !leftover.is_empty() {
|
||||
let merged = match delta.get("content").and_then(|v| v.as_str()) {
|
||||
Some(prev) => format!("{prev}{leftover}"),
|
||||
None => leftover,
|
||||
};
|
||||
delta.insert("content".into(), serde_json::json!(merged));
|
||||
}
|
||||
let calls = crate::toolcalls::markup_to_tool_calls(filter.captured(), 0);
|
||||
if !calls.is_empty() {
|
||||
emitted_tool_calls = true;
|
||||
let tc = serde_json::json!({
|
||||
"index": index,
|
||||
"delta": { "tool_calls": calls },
|
||||
});
|
||||
let extra: Vec<(&str, serde_json::Value)> = Vec::new();
|
||||
yield Ok(chunk!(vec![tc], extra));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if finish.is_none() && delta.is_empty() {
|
||||
continue;
|
||||
}
|
||||
@@ -442,6 +511,10 @@ fn normalize_sse(
|
||||
"delta": serde_json::Value::Object(delta),
|
||||
});
|
||||
if let Some(f) = finish {
|
||||
// A reply that carries tool calls stops for that
|
||||
// reason, whatever the upstream called it —
|
||||
// clients branch on this field to run the tool.
|
||||
let f = if emitted_tool_calls { "tool_calls" } else { f };
|
||||
choice["finish_reason"] = serde_json::json!(f);
|
||||
sent_finish = true;
|
||||
}
|
||||
@@ -449,20 +522,20 @@ fn normalize_sse(
|
||||
}
|
||||
}
|
||||
|
||||
// Terminal metadata rides on a choice-less chunk, exactly
|
||||
// as OpenAI ships `usage` when `stream_options` asks for it.
|
||||
let mut extra: Vec<(&str, serde_json::Value)> = Vec::new();
|
||||
if let Some(u) = v.get("usage").filter(|u| !u.is_null()) {
|
||||
extra.push(("usage", u.clone()));
|
||||
}
|
||||
for key in ["cortiq_cost_usd", "cortiq_estimated"] {
|
||||
// Terminal metadata is held back and shipped once, after the
|
||||
// stop reason — the order OpenAI uses. Emitting it mid-stream
|
||||
// (as several runtimes do) puts choice-less chunks between the
|
||||
// deltas and the finish, which is legal but unlike every
|
||||
// reference implementation.
|
||||
for key in ["usage", "cortiq_cost_usd", "cortiq_estimated"] {
|
||||
if let Some(val) = v.get(key).filter(|v| !v.is_null()) {
|
||||
extra.push((key, val.clone()));
|
||||
tail_extras.insert(key, val.clone());
|
||||
}
|
||||
}
|
||||
|
||||
if !choices.is_empty() || !extra.is_empty() {
|
||||
yield Ok(chunk!(choices, extra));
|
||||
if !choices.is_empty() {
|
||||
let none: Vec<(&str, serde_json::Value)> = Vec::new();
|
||||
yield Ok(chunk!(choices, none));
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -482,14 +555,37 @@ fn normalize_sse(
|
||||
if roled.insert(0) {
|
||||
delta.insert("role".into(), serde_json::json!("assistant"));
|
||||
}
|
||||
// Held-back text and unparsed markup must not be lost with the stream.
|
||||
for (index, filter) in markup.iter_mut() {
|
||||
let leftover = filter.finish();
|
||||
if !leftover.is_empty() && *index == 0 {
|
||||
delta.insert("content".into(), serde_json::json!(leftover));
|
||||
}
|
||||
if !emitted_tool_calls {
|
||||
let calls = crate::toolcalls::markup_to_tool_calls(filter.captured(), 0);
|
||||
if !calls.is_empty() {
|
||||
emitted_tool_calls = true;
|
||||
delta.insert("tool_calls".into(), serde_json::Value::Array(calls));
|
||||
}
|
||||
}
|
||||
}
|
||||
let choices = vec![serde_json::json!({
|
||||
"index": 0,
|
||||
"delta": serde_json::Value::Object(delta),
|
||||
"finish_reason": "stop",
|
||||
"finish_reason": if emitted_tool_calls { "tool_calls" } else { "stop" },
|
||||
})];
|
||||
let extra: Vec<(&str, serde_json::Value)> = Vec::new();
|
||||
yield Ok(chunk!(choices, extra));
|
||||
}
|
||||
// One trailing chunk, and only when the client asked for usage.
|
||||
if include_usage {
|
||||
if let Some(usage) = tail_extras.remove("usage") {
|
||||
let mut extra: Vec<(&str, serde_json::Value)> = vec![("usage", usage)];
|
||||
extra.extend(tail_extras.into_iter());
|
||||
let no_choices: Vec<serde_json::Value> = Vec::new();
|
||||
yield Ok(chunk!(no_choices, extra));
|
||||
}
|
||||
}
|
||||
yield Ok(bytes::Bytes::from("data: [DONE]\n\n"));
|
||||
}
|
||||
}
|
||||
@@ -530,13 +626,146 @@ mod tests {
|
||||
assert_eq!(normalize_finish_reason("tool_use"), "tool_calls");
|
||||
}
|
||||
|
||||
/// Collect a normalised stream into one string for assertions.
|
||||
async fn drain(chunks: Vec<&'static str>) -> String {
|
||||
use futures::StreamExt;
|
||||
let upstream = futures::stream::iter(
|
||||
chunks
|
||||
.into_iter()
|
||||
.map(|c| Ok(bytes::Bytes::from(c)))
|
||||
.collect::<Vec<_>>(),
|
||||
);
|
||||
normalize_sse(Box::pin(upstream), "cortiq-auto".into(), false)
|
||||
.map(|b| String::from_utf8_lossy(&b.unwrap()).to_string())
|
||||
.collect::<Vec<_>>()
|
||||
.await
|
||||
.join("")
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn tool_markup_is_not_duplicated_into_content() {
|
||||
// The reported shape: the upstream streams the call as prose AND then
|
||||
// sends it properly structured.
|
||||
let out = drain(vec![
|
||||
"data: {\"choices\":[{\"index\":0,\"delta\":{\"content\":\"Sure. <tool_call>\"}}]}\n\n",
|
||||
"data: {\"choices\":[{\"index\":0,\"delta\":{\"content\":\"{\\\"name\\\":\\\"get_weather\\\",\\\"arguments\\\":{\\\"city\\\":\\\"Paris\\\"}}\"}}]}\n\n",
|
||||
"data: {\"choices\":[{\"index\":0,\"delta\":{\"content\":\"</tool_call>\"}}]}\n\n",
|
||||
"data: {\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"id\":\"call_1\",\"type\":\"function\",\"function\":{\"name\":\"get_weather\",\"arguments\":\"{\\\"city\\\":\\\"Paris\\\"}\"}}]}}]}\n\n",
|
||||
"data: {\"choices\":[{\"index\":0,\"delta\":{},\"finish_reason\":\"stop\"}]}\n\n",
|
||||
"data: [DONE]\n\n",
|
||||
])
|
||||
.await;
|
||||
assert!(
|
||||
!out.contains("tool_call>"),
|
||||
"markup leaked into content: {out}"
|
||||
);
|
||||
assert!(
|
||||
out.contains("\"id\":\"call_1\""),
|
||||
"structured call missing: {out}"
|
||||
);
|
||||
// exactly one structured call, and the stop reason agrees with it
|
||||
assert_eq!(out.matches("\"tool_calls\":[").count(), 1);
|
||||
assert!(out.contains("\"finish_reason\":\"tool_calls\""));
|
||||
assert!(out.trim_end().ends_with("data: [DONE]"));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn unparsed_markup_is_promoted_to_a_real_tool_call() {
|
||||
// An upstream that never parses the markup: the call must still be
|
||||
// executable by the client rather than arriving as prose.
|
||||
let out = drain(vec![
|
||||
"data: {\"choices\":[{\"index\":0,\"delta\":{\"content\":\"<tool_call>{\\\"name\\\":\\\"get_weather\\\",\\\"arguments\\\":{\\\"city\\\":\\\"Paris\\\"}}</tool_call>\"}}]}\n\n",
|
||||
"data: {\"choices\":[{\"index\":0,\"delta\":{},\"finish_reason\":\"stop\"}]}\n\n",
|
||||
"data: [DONE]\n\n",
|
||||
])
|
||||
.await;
|
||||
assert!(!out.contains("tool_call>"), "markup leaked: {out}");
|
||||
assert!(
|
||||
out.contains("\"name\":\"get_weather\""),
|
||||
"call not promoted: {out}"
|
||||
);
|
||||
assert!(
|
||||
out.contains("\"finish_reason\":\"tool_calls\""),
|
||||
"wrong stop reason: {out}"
|
||||
);
|
||||
}
|
||||
|
||||
async fn drain_with_usage(chunks: Vec<&'static str>, include_usage: bool) -> String {
|
||||
use futures::StreamExt;
|
||||
let upstream = futures::stream::iter(
|
||||
chunks
|
||||
.into_iter()
|
||||
.map(|c| Ok(bytes::Bytes::from(c)))
|
||||
.collect::<Vec<_>>(),
|
||||
);
|
||||
normalize_sse(Box::pin(upstream), "cortiq-auto".into(), include_usage)
|
||||
.map(|b| String::from_utf8_lossy(&b.unwrap()).to_string())
|
||||
.collect::<Vec<_>>()
|
||||
.await
|
||||
.join("")
|
||||
}
|
||||
|
||||
/// The upstream ships usage BEFORE the stop reason and the gateway adds a
|
||||
/// cost chunk of its own. A client that did not ask for usage must see
|
||||
/// neither — the same stream shape every hosted provider sends.
|
||||
const USAGE_STREAM: &[&str] = &[
|
||||
"data: {\"choices\":[{\"index\":0,\"delta\":{\"content\":\"hi\"}}]}\n\n",
|
||||
"data: {\"choices\":[],\"usage\":{\"prompt_tokens\":3,\"completion_tokens\":1,\"total_tokens\":4}}\n\n",
|
||||
"data: {\"choices\":[{\"index\":0,\"delta\":{},\"finish_reason\":\"stop\"}]}\n\n",
|
||||
"data: {\"choices\":[],\"cortiq_cost_usd\":0.0}\n\n",
|
||||
"data: [DONE]\n\n",
|
||||
];
|
||||
|
||||
#[tokio::test]
|
||||
async fn without_stream_options_no_choice_less_chunks_are_sent() {
|
||||
let out = drain_with_usage(USAGE_STREAM.to_vec(), false).await;
|
||||
assert!(!out.contains("\"usage\""), "unrequested usage chunk: {out}");
|
||||
assert!(!out.contains("cortiq_cost_usd"), "stray cost chunk: {out}");
|
||||
assert!(
|
||||
!out.contains("\"choices\":[]"),
|
||||
"choice-less chunk leaked: {out}"
|
||||
);
|
||||
assert!(out.contains("\"finish_reason\":\"stop\""));
|
||||
assert!(out.trim_end().ends_with("data: [DONE]"));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn requested_usage_is_one_chunk_after_the_finish_reason() {
|
||||
let out = drain_with_usage(USAGE_STREAM.to_vec(), true).await;
|
||||
assert_eq!(
|
||||
out.matches("\"usage\"").count(),
|
||||
1,
|
||||
"usage must ship once: {out}"
|
||||
);
|
||||
let finish = out.find("\"finish_reason\"").expect("finish chunk");
|
||||
let usage = out.find("\"usage\"").expect("usage chunk");
|
||||
assert!(usage > finish, "usage must follow the stop reason: {out}");
|
||||
assert!(
|
||||
out.contains("cortiq_cost_usd"),
|
||||
"cost folded into usage chunk: {out}"
|
||||
);
|
||||
assert!(out.trim_end().ends_with("data: [DONE]"));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn ordinary_text_is_untouched_by_the_tool_filter() {
|
||||
let out = drain(vec![
|
||||
"data: {\"choices\":[{\"index\":0,\"delta\":{\"content\":\"a < b and 2<3\"}}]}\n\n",
|
||||
"data: {\"choices\":[{\"index\":0,\"delta\":{},\"finish_reason\":\"stop\"}]}\n\n",
|
||||
"data: [DONE]\n\n",
|
||||
])
|
||||
.await;
|
||||
assert!(out.contains("a < b and 2<3"), "text mangled: {out}");
|
||||
assert!(out.contains("\"finish_reason\":\"stop\""));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn stream_is_closed_even_when_the_upstream_stops_early() {
|
||||
use futures::StreamExt;
|
||||
let upstream = futures::stream::iter(vec![Ok(bytes::Bytes::from(
|
||||
"data: {\"choices\":[{\"index\":0,\"delta\":{\"content\":\"hi\"}}]}\n\n",
|
||||
))]);
|
||||
let out: Vec<String> = normalize_sse(Box::pin(upstream), "cortiq-auto".into())
|
||||
let out: Vec<String> = normalize_sse(Box::pin(upstream), "cortiq-auto".into(), false)
|
||||
.map(|b| String::from_utf8_lossy(&b.unwrap()).to_string())
|
||||
.collect()
|
||||
.await;
|
||||
|
||||
+17
-3
@@ -122,13 +122,23 @@ impl Provider for OpenAiProvider {
|
||||
for (i, c) in choices_arr.iter().enumerate() {
|
||||
let msg_val = &c["message"];
|
||||
let role = msg_val["role"].as_str().unwrap_or("assistant").to_string();
|
||||
let content = msg_val["content"].as_str().unwrap_or("").to_string();
|
||||
let raw_content = msg_val["content"].as_str().unwrap_or("");
|
||||
// Qwen-style templates put the call in the text as `<tool_call>…`.
|
||||
// Strip it: the structured array is the only form a client may act
|
||||
// on, and prose that merely looks like a call is worse than none.
|
||||
let (content, markup) = crate::toolcalls::split_tool_markup(raw_content);
|
||||
|
||||
let tool_calls = msg_val["tool_calls"]
|
||||
let mut tool_calls = msg_val["tool_calls"]
|
||||
.as_array()
|
||||
.cloned()
|
||||
.unwrap_or_default();
|
||||
if tool_calls.is_empty() {
|
||||
// Upstream never parsed the markup — promote it, so the call is
|
||||
// executable instead of decorative.
|
||||
tool_calls = crate::toolcalls::markup_to_tool_calls(&markup, 0);
|
||||
}
|
||||
|
||||
let tool_calls_present = !tool_calls.is_empty();
|
||||
choices.push(crate::model::Choice {
|
||||
index: c["index"].as_u64().unwrap_or(i as u64) as u32,
|
||||
message: crate::model::Message {
|
||||
@@ -137,7 +147,11 @@ impl Provider for OpenAiProvider {
|
||||
tool_calls,
|
||||
..Default::default()
|
||||
},
|
||||
finish_reason: c["finish_reason"].as_str().unwrap_or("stop").to_string(),
|
||||
finish_reason: if tool_calls_present {
|
||||
"tool_calls".to_string()
|
||||
} else {
|
||||
c["finish_reason"].as_str().unwrap_or("stop").to_string()
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,300 @@
|
||||
//! Tool-call hygiene for OpenAI-compatible responses.
|
||||
//!
|
||||
//! Qwen-family chat templates express a function call as literal markup in the
|
||||
//! generated text:
|
||||
//!
|
||||
//! ```text
|
||||
//! <tool_call>
|
||||
//! {"name": "get_weather", "arguments": {"city": "Paris"}}
|
||||
//! </tool_call>
|
||||
//! ```
|
||||
//!
|
||||
//! Runtimes handle that inconsistently. Some parse it into structured
|
||||
//! `tool_calls` **and** still stream the raw markup as `content` — the client
|
||||
//! then sees the call twice, once as prose it must not show and once as the real
|
||||
//! thing, and tool-loop implementations (Open WebUI's among them) get confused.
|
||||
//! Others never parse it at all, so a perfectly good call reaches the client as
|
||||
//! text no tool loop will ever execute.
|
||||
//!
|
||||
//! This module makes the gateway the place where that is settled: markup is
|
||||
//! stripped from the text, and when nothing structured came with it the markup
|
||||
//! is promoted into a real `tool_calls` array.
|
||||
|
||||
const OPEN: &str = "<tool_call>";
|
||||
const CLOSE: &str = "</tool_call>";
|
||||
|
||||
/// Longest suffix of `s` that is a proper prefix of `pat`. Used to hold back the
|
||||
/// tail of a streamed chunk that might be the beginning of a marker split across
|
||||
/// chunk boundaries (`"<tool"` + `"_call>"`).
|
||||
fn partial_marker_len(s: &str, pat: &str) -> usize {
|
||||
let max = pat.len().saturating_sub(1).min(s.len());
|
||||
(1..=max)
|
||||
.rev()
|
||||
.find(|&n| {
|
||||
s.is_char_boundary(s.len() - n)
|
||||
&& pat.as_bytes().starts_with(&s.as_bytes()[s.len() - n..])
|
||||
})
|
||||
.unwrap_or(0)
|
||||
}
|
||||
|
||||
/// Incremental `<tool_call>` filter for SSE deltas.
|
||||
///
|
||||
/// Feed it content fragments in order; it returns the text that is safe to
|
||||
/// forward and keeps the markup payloads for [`Self::captured`].
|
||||
#[derive(Default)]
|
||||
pub struct ToolMarkupFilter {
|
||||
/// Text held back: either an unterminated marker prefix or the inside of an
|
||||
/// open `<tool_call>` block.
|
||||
pending: String,
|
||||
inside: bool,
|
||||
captured: Vec<String>,
|
||||
}
|
||||
|
||||
impl ToolMarkupFilter {
|
||||
pub fn new() -> Self {
|
||||
Self::default()
|
||||
}
|
||||
|
||||
/// Consume one content fragment; returns the part that may be forwarded now.
|
||||
pub fn push(&mut self, chunk: &str) -> String {
|
||||
self.pending.push_str(chunk);
|
||||
let mut out = String::new();
|
||||
loop {
|
||||
if self.inside {
|
||||
let Some(end) = self.pending.find(CLOSE) else {
|
||||
// still inside the block — nothing to forward
|
||||
return out;
|
||||
};
|
||||
let payload: String = self.pending[..end].to_string();
|
||||
self.captured.push(payload);
|
||||
self.pending = self.pending[end + CLOSE.len()..].to_string();
|
||||
self.inside = false;
|
||||
continue;
|
||||
}
|
||||
match self.pending.find(OPEN) {
|
||||
Some(start) => {
|
||||
out.push_str(&self.pending[..start]);
|
||||
self.pending = self.pending[start + OPEN.len()..].to_string();
|
||||
self.inside = true;
|
||||
}
|
||||
None => {
|
||||
// Hold back a trailing partial marker so a split like
|
||||
// "…<tool" / "_call>…" is not forwarded as visible text.
|
||||
let hold = partial_marker_len(&self.pending, OPEN);
|
||||
let cut = self.pending.len() - hold;
|
||||
out.push_str(&self.pending[..cut]);
|
||||
self.pending = self.pending[cut..].to_string();
|
||||
return out;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// End of stream: an unterminated block was never a tool call after all, so
|
||||
/// give the text back rather than swallowing output.
|
||||
pub fn finish(&mut self) -> String {
|
||||
let mut out = String::new();
|
||||
if self.inside {
|
||||
out.push_str(OPEN);
|
||||
self.inside = false;
|
||||
}
|
||||
out.push_str(&std::mem::take(&mut self.pending));
|
||||
out
|
||||
}
|
||||
|
||||
/// Payloads found between `<tool_call>` and `</tool_call>`.
|
||||
pub fn captured(&self) -> &[String] {
|
||||
&self.captured
|
||||
}
|
||||
}
|
||||
|
||||
/// Strip every complete `<tool_call>` block from a finished string.
|
||||
/// Returns the cleaned text and the payloads that were removed.
|
||||
pub fn split_tool_markup(text: &str) -> (String, Vec<String>) {
|
||||
let mut f = ToolMarkupFilter::new();
|
||||
let mut out = f.push(text);
|
||||
out.push_str(&f.finish());
|
||||
(out, f.captured)
|
||||
}
|
||||
|
||||
/// Turn `<tool_call>` payloads into OpenAI `tool_calls` entries, numbered from
|
||||
/// `start_index`. Payloads that are not a JSON object with a `name` are skipped —
|
||||
/// a stray marker must not become a bogus call.
|
||||
pub fn markup_to_tool_calls(payloads: &[String], start_index: usize) -> Vec<serde_json::Value> {
|
||||
let mut out = Vec::new();
|
||||
for payload in payloads {
|
||||
let Ok(v) = serde_json::from_str::<serde_json::Value>(payload.trim()) else {
|
||||
continue;
|
||||
};
|
||||
let Some(name) = v.get("name").and_then(|n| n.as_str()) else {
|
||||
continue;
|
||||
};
|
||||
// `arguments` is a JSON string on the wire; templates emit an object.
|
||||
let arguments = match v.get("arguments").or_else(|| v.get("parameters")) {
|
||||
Some(serde_json::Value::String(s)) => s.clone(),
|
||||
Some(other) => other.to_string(),
|
||||
None => "{}".to_string(),
|
||||
};
|
||||
out.push(serde_json::json!({
|
||||
"index": start_index + out.len(),
|
||||
"id": new_call_id(),
|
||||
"type": "function",
|
||||
"function": { "name": name, "arguments": arguments },
|
||||
}));
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
pub fn new_call_id() -> String {
|
||||
format!("call_{}", crate::admin::random_token(12))
|
||||
}
|
||||
|
||||
/// Bring one streamed `tool_calls` array up to the OpenAI shape.
|
||||
///
|
||||
/// `seen` tracks which choice-local indices already had their opening delta, so
|
||||
/// the identity fields (`id`, `type`) are added once and continuation deltas —
|
||||
/// which legitimately carry nothing but an argument fragment — are left alone.
|
||||
pub fn normalize_delta_tool_calls(
|
||||
raw: &[serde_json::Value],
|
||||
seen: &mut std::collections::HashSet<u64>,
|
||||
) -> Vec<serde_json::Value> {
|
||||
let mut out = Vec::new();
|
||||
for (i, tc) in raw.iter().enumerate() {
|
||||
let index = tc.get("index").and_then(|v| v.as_u64()).unwrap_or(i as u64);
|
||||
let mut entry = serde_json::Map::new();
|
||||
entry.insert("index".into(), serde_json::json!(index));
|
||||
|
||||
let first = seen.insert(index);
|
||||
if let Some(id) = tc
|
||||
.get("id")
|
||||
.and_then(|v| v.as_str())
|
||||
.filter(|s| !s.is_empty())
|
||||
{
|
||||
entry.insert("id".into(), serde_json::json!(id));
|
||||
} else if first {
|
||||
entry.insert("id".into(), serde_json::json!(new_call_id()));
|
||||
}
|
||||
if first {
|
||||
let kind = tc
|
||||
.get("type")
|
||||
.and_then(|v| v.as_str())
|
||||
.unwrap_or("function");
|
||||
entry.insert("type".into(), serde_json::json!(kind));
|
||||
}
|
||||
|
||||
if let Some(f) = tc.get("function") {
|
||||
let mut func = serde_json::Map::new();
|
||||
if let Some(name) = f.get("name").and_then(|v| v.as_str()) {
|
||||
func.insert("name".into(), serde_json::json!(name));
|
||||
} else if first {
|
||||
func.insert("name".into(), serde_json::json!(""));
|
||||
}
|
||||
match f.get("arguments") {
|
||||
// Arguments are a string on the wire; some runtimes send the object.
|
||||
Some(serde_json::Value::String(s)) => {
|
||||
func.insert("arguments".into(), serde_json::json!(s));
|
||||
}
|
||||
Some(serde_json::Value::Null) | None => {
|
||||
if first {
|
||||
func.insert("arguments".into(), serde_json::json!(""));
|
||||
}
|
||||
}
|
||||
Some(other) => {
|
||||
func.insert("arguments".into(), serde_json::json!(other.to_string()));
|
||||
}
|
||||
}
|
||||
entry.insert("function".into(), serde_json::Value::Object(func));
|
||||
}
|
||||
out.push(serde_json::Value::Object(entry));
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
const CALL: &str = r#"<tool_call>
|
||||
{"name": "get_weather", "arguments": {"city": "Paris"}}
|
||||
</tool_call>"#;
|
||||
|
||||
#[test]
|
||||
fn markup_is_stripped_from_finished_text() {
|
||||
let (clean, payloads) = split_tool_markup(&format!("Sure!{CALL}"));
|
||||
assert_eq!(clean, "Sure!");
|
||||
assert_eq!(payloads.len(), 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn markup_split_across_stream_chunks_never_leaks() {
|
||||
let mut f = ToolMarkupFilter::new();
|
||||
let mut seen = String::new();
|
||||
// the marker itself is broken across three fragments
|
||||
for chunk in [
|
||||
"Let me check. <tool",
|
||||
"_call>",
|
||||
"{\"name\":\"get_weather\",",
|
||||
"\"arguments\":{\"city\":\"Paris\"}}",
|
||||
"</tool",
|
||||
"_call>",
|
||||
" done",
|
||||
] {
|
||||
seen.push_str(&f.push(chunk));
|
||||
}
|
||||
seen.push_str(&f.finish());
|
||||
assert_eq!(seen, "Let me check. done");
|
||||
assert!(!seen.contains("tool_call"));
|
||||
assert_eq!(f.captured().len(), 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn captured_markup_becomes_a_real_tool_call() {
|
||||
let (_, payloads) = split_tool_markup(CALL);
|
||||
let calls = markup_to_tool_calls(&payloads, 0);
|
||||
assert_eq!(calls.len(), 1);
|
||||
assert_eq!(calls[0]["type"], "function");
|
||||
assert_eq!(calls[0]["index"], 0);
|
||||
assert_eq!(calls[0]["function"]["name"], "get_weather");
|
||||
// arguments must be a STRING, not an object
|
||||
assert_eq!(
|
||||
calls[0]["function"]["arguments"].as_str().unwrap(),
|
||||
r#"{"city":"Paris"}"#
|
||||
);
|
||||
assert!(calls[0]["id"].as_str().unwrap().starts_with("call_"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_stray_marker_does_not_become_a_call() {
|
||||
let calls = markup_to_tool_calls(&["not json at all".into(), "{\"no_name\":1}".into()], 0);
|
||||
assert!(calls.is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_unterminated_block_is_given_back_as_text() {
|
||||
let mut f = ToolMarkupFilter::new();
|
||||
let mut out = f.push("thinking <tool_call>{\"name\":\"x\"");
|
||||
out.push_str(&f.finish());
|
||||
assert_eq!(out, "thinking <tool_call>{\"name\":\"x\"");
|
||||
assert!(f.captured().is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn delta_tool_calls_get_identity_once_and_object_arguments_become_strings() {
|
||||
let mut seen = std::collections::HashSet::new();
|
||||
let opening = normalize_delta_tool_calls(
|
||||
&[serde_json::json!({"index": 0, "function": {"name": "f", "arguments": {"a": 1}}})],
|
||||
&mut seen,
|
||||
);
|
||||
assert_eq!(opening[0]["type"], "function");
|
||||
assert!(opening[0]["id"].as_str().unwrap().starts_with("call_"));
|
||||
assert_eq!(opening[0]["function"]["arguments"], r#"{"a":1}"#);
|
||||
|
||||
// a continuation delta must not be handed a second id
|
||||
let cont = normalize_delta_tool_calls(
|
||||
&[serde_json::json!({"index": 0, "function": {"arguments": "\"}"}})],
|
||||
&mut seen,
|
||||
);
|
||||
assert!(cont[0].get("id").is_none());
|
||||
assert!(cont[0].get("type").is_none());
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user