cortiq-gateway/docs/PROTOCOLS.md

Read this in Русский.

Cortiq Gateway Protocols

Contracts for inbound protocols and cross-standard mechanisms. Each protocol is enabled by a flag in [protocols]. Internally, all requests are normalized to the canonical model (see ARCHITECTURE.md).

Routing rule common to all protocols:

• model = "cortiq-auto" → enable intelligent routing via cortiq-router.
• model = "cortiq-auto:quality-first" → routing with the specified policy profile (cost-saver | balanced | quality-first).
• model = "<real id from config>" → passthrough to that model without routing.

---

Table of Contents

1. OpenAI Chat Completions
2. OpenAI Completions (legacy)
3. OpenAI Embeddings
4. OpenAI Models
5. Anthropic Messages
6. MCP (Model Context Protocol)
7. Native passthrough
8. Cross-standard mechanisms

---

1. OpenAI Chat Completions

POST /v1/chat/completions — the primary endpoint; the de facto standard for agents.

Request

{
  "model": "cortiq-auto",                 // routing; or a real id for passthrough
  "messages": [
    { "role": "system", "content": "You are a helpful assistant." },
    { "role": "user", "content": "Solve x^2 - 5x + 6 = 0" }
  ],
  "temperature": 0.7,
  "max_tokens": 1024,
  "stream": false,
  "tools": [ /* function-calling, proxied to the provider as-is */ ],
  "tool_choice": "auto",
  "stop": ["\n\n"],
  "top_p": 1.0,
  "response_format": { "type": "json_object" }
}

Standard OpenAI fields are supported; unknown fields are proxied to the provider unchanged (forward-compatible), minus the bookkeeping keys chat UIs attach to the payload (chat_id, session_id, metadata, features, …), which are dropped so strict upstreams do not reject the call.

The parser is deliberately lenient about the shapes real clients send: content may be a string, null, or an array of content parts (text parts are concatenated, images and other non-text parts are skipped); tools may be null; reasoning_effort may be a string (low | medium | high) or a token budget; max_completion_tokens is accepted as an alias for max_tokens. A body that still cannot be read comes back as an OpenAI error envelope with HTTP 400 — never a bare 422, which OpenAI clients cannot parse (they show no answer at all).

tools is treated as an offer, not a requirement. The gateway prefers a tool-capable model, but when none is available it answers without the tools rather than failing, and says so via X-Cortiq-Tools-Dropped: true (chat UIs attach their built-in tools to every message, so the strict reading made ordinary chat impossible on a tools-less local pool). tool_choice: "required" — or a named function — is a real requirement and still fails with 502 no_healthy_model when no model can honour it.

Response (non-streaming)

{
  "id": "chatcmpl-9f...",
  "object": "chat.completion",
  "created": 1718800000,
  "model": "cortiq-auto",                 // the model the CLIENT asked for, echoed back
  "choices": [
    {
      "index": 0,
      "message": { "role": "assistant", "content": "x = 2 or x = 3" },
      "finish_reason": "stop"               // always one of stop|length|tool_calls|content_filter
    }
  ],
  "usage": {
    "prompt_tokens": 24,
    "completion_tokens": 11,
    "total_tokens": 35
  },
  "cortiq": {                             // extension (enabled with cortiq.echo=true)
    "task_label": "math",
    "complexity": { "score": 0.22, "tier": "low" },
    "selected_model": "local-qwen",
    "route_source": "router",             // router | cache | fallback | pinned
    "router_request_id": "req_...",
    "cost_usd": 0.0
  }
}

model echoes the id the client requested, exactly as OpenAI does — clients match the reply against the id they sent. The model that actually answered is in cortiq.selected_model and in the X-Cortiq-Selected-Model header.

Optional fields are omitted rather than sent as null: absent tool calls mean no tool_calls key at all. "tool_calls": null is not in the OpenAI schema and trips strict clients.

The cortiq field is an optional extension. OpenAI SDKs ignore it. If the client does not want extra fields, disable cortiq.echo — the metadata will remain in the response headers.

Streaming (stream: true) emits strict chat.completion.chunk events: the stream always opens with a role delta, carries exactly one terminal finish_reason, and ends with data: [DONE] — even when the upstream stops early or dies mid-flight, in which case the failure arrives as an SSE error event. A client can therefore never be left waiting on a half-formed stream.

Streaming carries no choice-less chunks unless asked: usage is emitted only when the request sets stream_options: {"include_usage": true}, and then exactly once, after the finish_reason chunk, with cortiq_cost_usd folded in. Without it a stream is role → content… → tool_calls → finish_reason → [DONE] and nothing else — the shape the hosted providers send.

Tool calls

tools and tool_calls are proxied through, with the wire kept clean:

• raw <tool_call>…</tool_call> markup (how Qwen-family templates express a call in the generated text) is stripped from content, so a client never receives the same call twice — once as prose and once structured. Markers split across SSE chunks are handled; ordinary text containing < is not touched;
• when the upstream never parsed that markup, it is promoted into a real tool_calls entry with a generated call_… id, so the call is executable rather than decorative;
• streamed entries carry index always, id/type once per call, and function.arguments as a string even if the runtime sent an object;
• a reply carrying tool calls always ends with finish_reason: "tool_calls".

Metadata in response headers (always present)

X-Cortiq-Request-Id: req_...
X-Cortiq-Task-Label: math
X-Cortiq-Complexity-Score: 0.22
X-Cortiq-Complexity-Tier: low
X-Cortiq-Selected-Model: local-qwen
X-Cortiq-Route-Source: router
X-Cortiq-Cost-Usd: 0.000000

Response (streaming, stream: true)

SSE in OpenAI format: a series of chat.completion.chunk events, terminated by data: [DONE]. The gateway proxies provider chunks one-to-one (without buffering), adding routing metadata to HTTP headers before the stream begins.

data: {"id":"chatcmpl-9f","object":"chat.completion.chunk","choices":[{"delta":{"role":"assistant"},"index":0}]}

data: {"id":"chatcmpl-9f","object":"chat.completion.chunk","choices":[{"delta":{"content":"x = 2"},"index":0}]}

data: {"id":"chatcmpl-9f","object":"chat.completion.chunk","choices":[{"delta":{},"finish_reason":"stop","index":0}]}

data: [DONE]

---

2. OpenAI Completions (legacy)

POST /v1/completions — for legacy integrations. Internally, prompt is wrapped into a single user message and follows the same pipeline. Response object is "text_completion".

// request
{ "model": "cortiq-auto", "prompt": "Translate to French: hello", "max_tokens": 64 }
// response
{ "object": "text_completion", "model": "local-qwen",
  "choices": [ { "text": "bonjour", "index": 0, "finish_reason": "stop" } ],
  "usage": { "prompt_tokens": 6, "completion_tokens": 2, "total_tokens": 8 } }

---

3. OpenAI Embeddings

POST /v1/embeddings. Two modes (configurable):

• provider — proxy to an embedding model from the pool;
• router — return an embedding from the cortiq-router encoder (the same vector the router uses for classification — useful for consistent RAG).

// request
{ "model": "cortiq-embed", "input": ["first text", "second text"] }
// response
{
  "object": "list",
  "data": [
    { "object": "embedding", "index": 0, "embedding": [0.01, -0.02, /* ... */] },
    { "object": "embedding", "index": 1, "embedding": [0.03,  0.00, /* ... */] }
  ],
  "model": "cortiq-embed",
  "usage": { "prompt_tokens": 4, "total_tokens": 4 }
}

---

4. OpenAI Models

GET /v1/models — auto-discovery for clients. Returns the gateway's virtual models (cortiq-auto, cortiq-embed), every model in the [[models]] pool, and every managed local [[cmf.servers]] model, so a client can pin a local .cmf directly.

{
  "object": "list",
  "data": [
    { "id": "cortiq-auto", "object": "model", "created": 1718800000, "owned_by": "cortiq" },
    { "id": "gpt-4o-mini", "object": "model", "created": 1718800000, "owned_by": "openai" },
    { "id": "claude-opus", "object": "model", "created": 1718800000, "owned_by": "anthropic" },
    { "id": "cmf-qwen3-4b", "object": "model", "created": 1718800000, "owned_by": "cortiq-cmf",
      "cortiq": { "kind": "cmf", "local": true, "running": true, "healthy": true } }
  ]
}

Managed CMF models are listed as soon as they are configured, with their live state in the non-standard cortiq field (clients ignore it). They are not filtered by health: a large .cmf takes minutes to load, and a model that disappears from the list while it warms up is worse than one marked "not ready yet".

---

5. Anthropic Messages

POST /v1/messages — for Claude-native agents and the Anthropic SDK.

// request
{
  "model": "cortiq-auto",
  "max_tokens": 1024,
  "system": "You are a helpful assistant.",
  "messages": [ { "role": "user", "content": "Solve x^2 - 5x + 6 = 0" } ],
  "stream": false
}
// response
{
  "id": "msg_...",
  "type": "message",
  "role": "assistant",
  "model": "cortiq-auto",
  "content": [ { "type": "text", "text": "x = 2 or x = 3" } ],
  "stop_reason": "end_turn",
  "usage": { "input_tokens": 24, "output_tokens": 11 }
}

Streaming uses native Anthropic events (message_start, content_block_delta, message_delta, message_stop). If the selected model is a non-Anthropic provider, the gateway translates through the canonical model in both directions.

---

6. MCP (Model Context Protocol)

The gateway runs an MCP server (JSON-RPC 2.0 over HTTP+SSE or stdio) and exposes routing as tools for MCP-native orchestrators.

Published tools:

Tool	Purpose
route_and_complete	classify + execute on the selected model, return the response
classify_task	classification only (task type + complexity), no execution

tools/list returns JSON schemas. Example tools/call invocation:

// request (JSON-RPC)
{
  "jsonrpc": "2.0", "id": 1, "method": "tools/call",
  "params": {
    "name": "route_and_complete",
    "arguments": { "prompt": "Solve x^2 - 5x + 6 = 0", "profile": "balanced" }
  }
}
// response
{
  "jsonrpc": "2.0", "id": 1,
  "result": {
    "content": [ { "type": "text", "text": "x = 2 or x = 3" } ],
    "isError": false,
    "_meta": { "task_label": "math", "tier": "low", "selected_model": "local-qwen" }
  }
}

---

7. Native passthrough

POST /route — a thin wrapper over cortiq-router /v1/route for clients that need only the routing decision (without execution). The request body and response match the router contract (see cortiq-router/docs/ФОРМАТ_ЗАПРОСА_И_ОТВЕТА.md). Useful for debugging and for integrators who want to manage the LLM call themselves.

---

8. Cross-standard mechanisms

Common to all HTTP protocols.

8.1 Authentication

Authorization: Bearer sk-gw-...

Virtual gateway key. Mapped to an account ([[api_keys]]). If no keys are configured, the gateway runs in open mode (for local development). Provider keys are never exposed.

8.2 Idempotency

Idempotency-Key: <unique-operation-id>

A repeat request with the same key within the idempotency.ttl window returns the stored response (safe agent retries, without double-invoking the LLM or double billing).

8.3 Rate limiting

On 429, the following headers are returned:

Retry-After: 2
X-RateLimit-Limit: 600
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1718800060

8.4 Tracing

W3C traceparent is accepted and forwarded; OpenTelemetry spans cover inbound→route→provider. The response includes X-Request-Id. This enables an end-to-end trace from agent → gateway → provider.

8.5 Error format (OpenAI-compatible)

{
  "error": {
    "message": "no healthy model available for tier 'high'",
    "type": "upstream_unavailable",      // invalid_request_error | authentication_error | rate_limit_error | upstream_unavailable | internal_error
    "code": "no_healthy_model",
    "param": null
  }
}

OpenAI/Anthropic SDKs parse this envelope natively. Status code mapping:

Type	HTTP	retriable
invalid_request_error	400	no
authentication_error	401	no
rate_limit_error	429	yes
upstream_unavailable	502/503	yes
internal_error	500	yes

8.6 Cost tracking

usage is returned in the protocol format (OpenAI/Anthropic). Additionally, cost is included in the X-Cortiq-Cost-Usd header and in cortiq.cost_usd (if cortiq.echo is enabled). The price is calculated from the model's price_in/price_out in the config multiplied by the token counts.