Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
 infosave2007
|
04580196ad | ||
|
infosave2007
|
7a3d6a7bb4 | ||
|
infosave2007
|
07d011263d | ||
|
infosave2007
|
0040efc948 | ||
|
infosave2007
|
16a3e209b6 | ||
|
infosave2007
|
c19edee7bb | ||
|
infosave2007andClaude Opus 4.8
|
7af90a5eef | ||
|
infosave2007andClaude Opus 4.8
|
8d7e5fe653 | ||
|
infosave2007andClaude Opus 4.8
|
2605eccb64 | ||
|
infosave2007andClaude Opus 4.8
|
1317b8024b | ||
|
infosave2007andClaude Opus 4.8
|
9242806795 | ||
|
infosave2007andClaude Opus 4.8
|
6714c4f556 | ||
|
infosave2007andClaude Opus 4.8
|
a5966dd479 |
@@ -0,0 +1,10 @@
|
||||
target
|
||||
.git
|
||||
.github
|
||||
docs
|
||||
bench
|
||||
config/gateway.toml
|
||||
config/secrets.toml
|
||||
config/stats.jsonl
|
||||
*.log
|
||||
.DS_Store
|
||||
@@ -0,0 +1,47 @@
|
||||
name: Docker
|
||||
|
||||
# Publishes container images to GitHub Container Registry (ghcr.io).
|
||||
# master push -> :edge, :sha (amd64, fast)
|
||||
# tag v* -> :X.Y.Z, :X.Y, :latest, :sha (linux/amd64 + linux/arm64)
|
||||
on:
|
||||
push:
|
||||
branches: [master]
|
||||
tags: ["v*"]
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
packages: write
|
||||
|
||||
jobs:
|
||||
publish:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: docker/setup-qemu-action@v3
|
||||
- uses: docker/setup-buildx-action@v3
|
||||
- uses: docker/login-action@v3
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.GITHUB_TOKEN }}
|
||||
- id: meta
|
||||
uses: docker/metadata-action@v5
|
||||
with:
|
||||
images: ghcr.io/${{ github.repository }}
|
||||
tags: |
|
||||
type=semver,pattern={{version}}
|
||||
type=semver,pattern={{major}}.{{minor}}
|
||||
type=raw,value=latest,enable=${{ startsWith(github.ref, 'refs/tags/v') }}
|
||||
type=raw,value=edge,enable={{is_default_branch}}
|
||||
type=sha,format=short
|
||||
- uses: docker/build-push-action@v6
|
||||
with:
|
||||
context: .
|
||||
# multi-arch only for release tags (arm64 under emulation is slow)
|
||||
platforms: ${{ startsWith(github.ref, 'refs/tags/v') && 'linux/amd64,linux/arm64' || 'linux/amd64' }}
|
||||
push: true
|
||||
tags: ${{ steps.meta.outputs.tags }}
|
||||
labels: ${{ steps.meta.outputs.labels }}
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
+109
@@ -0,0 +1,109 @@
|
||||
# Gateway latency benchmark
|
||||
|
||||
How much latency does the **gateway itself** add? For an LLM proxy the model call
|
||||
dominates wall-clock time, but the gateway's own overhead is what differs between
|
||||
products — and it compounds across the many calls an agent makes. This benchmark
|
||||
isolates that overhead and compares Cortiq Gateway with **LiteLLM** and **Portkey**.
|
||||
|
||||
> Reproduce on your own hardware: `bash bench/run.sh` (see [Methodology](#methodology)).
|
||||
|
||||
## TL;DR
|
||||
|
||||
Every gateway proxies the **same instant mock backend** in passthrough mode, under
|
||||
identical load (`ab -k -r -c 20 -n 5000`). Latency is in milliseconds.
|
||||
|
||||
| Gateway | Throughput (req/s) | p50 | p90 | p99 | mean |
|
||||
|---|--:|--:|--:|--:|--:|
|
||||
| **Cortiq Gateway** (Rust) | **~57,300** | 0 ms | 1 ms | **1 ms** | 0.35 ms |
|
||||
| Portkey Gateway (Node) | ~5,800 | 3 ms | 5 ms | 9 ms | 3.5 ms |
|
||||
| LiteLLM (Python, 4 workers) | ~1,200 | 9 ms | 37 ms | 59 ms | 16.6 ms |
|
||||
| _mock-direct (no gateway)_ | _~29,000_ | _1 ms_ | _1 ms_ | _3 ms_ | _0.7 ms_ |
|
||||
|
||||
On this machine Cortiq adds **sub-millisecond p99** overhead and sustains **~10×**
|
||||
the throughput of Portkey and **~47×** of LiteLLM. The gap is architectural: a Rust
|
||||
/ Tokio gateway uses every core in one process, with no interpreter or GC in the hot
|
||||
path.
|
||||
|
||||
> Absolute numbers depend on hardware; the **ratios** between gateways are the point.
|
||||
> Run `bench/run.sh` to get numbers for your environment.
|
||||
|
||||
## Accuracy: task-type routing
|
||||
|
||||
Speed is only half the story. Cortiq's other edge is **routing by task type** — sending
|
||||
each prompt to the right model — via the allaigate semantic classifier. LiteLLM and
|
||||
Portkey have **no semantic task router**: you either pin a model, write brittle rules,
|
||||
or pay for an extra LLM classification call.
|
||||
|
||||
On a 37-prompt, 7-task-type set of **natural-language prompts** (no explicit trigger
|
||||
words like "translate" / "summarize" — [`bench/tasks.jsonl`](bench/tasks.jsonl)):
|
||||
|
||||
| Classifier | Correct | Accuracy |
|
||||
|---|--:|--:|
|
||||
| **allaigate semantic router** | 37 / 37 | **100%** |
|
||||
| keyword heuristic (DIY without a classifier) | 12 / 37 | 32% |
|
||||
|
||||
A keyword matcher looks good on prompts that literally say "translate this" — but real
|
||||
users don't. On natural phrasing (`"'Guten Tag' — what does that mean in English?"`,
|
||||
`"boil this down to a few key points"`, `"which companies are named in this report?"`)
|
||||
surface matching collapses while the semantic router still routes correctly.
|
||||
|
||||
- **Methodology.** [`bench/accuracy.py`](bench/accuracy.py) sends each prompt to the live
|
||||
allaigate router (`taxonomy = data-assistant`) and compares the predicted `task_label`
|
||||
to the ground truth; a keyword heuristic is scored on the same set as a lower bound for
|
||||
"route without a classifier."
|
||||
- **Caveats.** Small, hand-labeled set; phrasing is deliberately keyword-light to test
|
||||
semantic understanding vs surface matching. Expand `tasks.jsonl` and re-run. allaigate's
|
||||
own published figure is **94% at separating close task types** (a harder eval than this one).
|
||||
- **Reproduce.** `CORTIQ_ROUTER_KEY=cortiq_… python3 bench/accuracy.py`
|
||||
|
||||
## Why this matters
|
||||
|
||||
Agentic workloads make **many** gateway calls per task (plan → sub-tasks → tool calls
|
||||
→ synthesis). Overhead that looks small per call (tens of ms) multiplies across a run
|
||||
and caps the throughput a single instance can serve. Lower, more predictable tail
|
||||
latency (p99) is the developer-facing differentiator when the model time is equal.
|
||||
|
||||
## Methodology
|
||||
|
||||
- **Isolate gateway overhead.** A tiny, dependency-free Node mock ([`bench/mock.js`](bench/mock.js))
|
||||
returns a fixed OpenAI-compatible completion **instantly**. So measured latency is
|
||||
the gateway's own cost, not model time. Every gateway proxies this same mock.
|
||||
- **Apples-to-apples.** Each gateway runs in **passthrough / pinned** mode (no routing
|
||||
or model-selection logic) so we compare raw proxy cost. Cortiq's intelligent routing
|
||||
(`cortiq-auto`) adds one classifier round-trip — that's a feature, measured separately,
|
||||
not folded into this overhead number.
|
||||
- **Load.** Apache Bench, keep-alive, tolerant of length variance:
|
||||
`ab -k -r -c 20 -n 5000 -p bench/body.json -T application/json`.
|
||||
- **Fair to each runtime.** Cortiq = one process (Tokio, multi-core). Portkey = one Node
|
||||
process (event loop). LiteLLM = **4 uvicorn workers** (its recommended multi-process
|
||||
setup; a single Python worker is GIL-bound to one core and scores ~860 req/s).
|
||||
- **Same box, localhost**, warm processes, results are representative of one run.
|
||||
|
||||
### Caveats (read me)
|
||||
|
||||
- This measures **gateway overhead only**. Real requests are dominated by model latency,
|
||||
which is identical regardless of gateway — but overhead and tail latency are not.
|
||||
- Localhost + mock backend removes network/model variance on purpose. Your absolute
|
||||
numbers will differ; reproduce with `bench/run.sh`.
|
||||
- `ab` reports LiteLLM "Failed requests" as a non-zero count — these are **response
|
||||
content-length variations, not HTTP errors** (all responses are `200`, verified).
|
||||
- Tested versions: Cortiq Gateway (this repo, `--release`), LiteLLM proxy via
|
||||
`uvicorn litellm.proxy.proxy_server:app --workers 4`, Portkey `@portkey-ai/gateway`,
|
||||
Apache Bench. Hardware: Apple Silicon, macOS.
|
||||
|
||||
## Reproduce
|
||||
|
||||
```bash
|
||||
# prerequisites: node, ab (Apache Bench), a release build of the gateway
|
||||
cargo build --release
|
||||
|
||||
# optional competitors (the harness auto-detects them):
|
||||
pip install 'litellm[proxy]' uvicorn
|
||||
npm install -g @portkey-ai/gateway
|
||||
|
||||
# run (tune load via env)
|
||||
CONC=20 REQ=5000 bash bench/run.sh
|
||||
```
|
||||
|
||||
The harness ([`bench/run.sh`](bench/run.sh)) starts the mock, then benchmarks the mock
|
||||
directly and each available gateway, printing a throughput / p50 / p90 / p99 table.
|
||||
@@ -3,6 +3,27 @@
|
||||
All notable changes to this project are documented here.
|
||||
The format is based on [Keep a Changelog](https://keepachangelog.com/).
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Added
|
||||
- **Docker Compose** quickstart (`docker-compose.yml` + `config/gateway.compose.toml`):
|
||||
gateway + a local Ollama model in one command. GHCR badge in the READMEs.
|
||||
|
||||
## [0.2.2] - 2026-07-01
|
||||
|
||||
### Added
|
||||
- **Latency benchmark** vs LiteLLM and Portkey: a reproducible harness (`bench/`,
|
||||
instant mock backend + Apache Bench) and `BENCHMARKS.md`. Measured gateway overhead:
|
||||
Cortiq ~57k req/s / p99 1 ms vs Portkey ~5.8k / p99 9 ms vs LiteLLM ~1.2k / p99 59 ms.
|
||||
- **Accuracy benchmark**: task-type routing on natural-language prompts (`bench/accuracy.py`,
|
||||
`bench/tasks.jsonl`) — allaigate semantic router 100% (37/37) vs a keyword heuristic 32%
|
||||
(competitors have no semantic task router).
|
||||
- **Router docs** (`docs/ROUTER.md` + `.ru.md`): what the allaigate/cortiq-router does, how
|
||||
to point the gateway at it, API contract, pricing, and the current `LAUNCH1` $1 promo.
|
||||
- **Docker image**: multi-stage `Dockerfile` (non-root, ~160 MB) + `docker.yml` workflow
|
||||
publishing to `ghcr.io/infosave2007/cortiq-gateway` (`:latest`, `:X.Y.Z`, `:edge`;
|
||||
linux/amd64 + arm64 on release tags). `docker run` works out of the box.
|
||||
|
||||
## [0.2.1] - 2026-06-30
|
||||
|
||||
### Added
|
||||
|
||||
Generated
+1
-1
@@ -285,7 +285,7 @@ checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570"
|
||||
|
||||
[[package]]
|
||||
name = "cortiq-gateway"
|
||||
version = "0.2.1"
|
||||
version = "0.2.3"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"arc-swap",
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
[package]
|
||||
name = "cortiq-gateway"
|
||||
version = "0.2.1"
|
||||
version = "0.2.3"
|
||||
edition = "2021"
|
||||
rust-version = "1.75"
|
||||
license = "Apache-2.0"
|
||||
|
||||
+25
@@ -0,0 +1,25 @@
|
||||
# ---- build stage ------------------------------------------------------------
|
||||
FROM rust:1-slim AS builder
|
||||
WORKDIR /app
|
||||
# the admin SPA (web/) is embedded at compile time via rust-embed, so it must be present
|
||||
COPY Cargo.toml Cargo.lock ./
|
||||
COPY src ./src
|
||||
COPY web ./web
|
||||
RUN cargo build --release
|
||||
|
||||
# ---- runtime stage ----------------------------------------------------------
|
||||
FROM debian:bookworm-slim
|
||||
RUN apt-get update \
|
||||
&& apt-get install -y --no-install-recommends ca-certificates \
|
||||
&& rm -rf /var/lib/apt/lists/* \
|
||||
&& useradd -r -u 10001 cortiq \
|
||||
&& mkdir -p /app/config
|
||||
COPY --from=builder /app/target/release/cortiq-gateway /usr/local/bin/cortiq-gateway
|
||||
# ship the example as the default config so `docker run` works out of the box
|
||||
COPY config/gateway.example.toml /app/config/gateway.toml
|
||||
RUN chown -R 10001:10001 /app
|
||||
WORKDIR /app
|
||||
USER cortiq
|
||||
EXPOSE 9000
|
||||
ENTRYPOINT ["cortiq-gateway"]
|
||||
CMD ["--config", "/app/config/gateway.toml"]
|
||||
@@ -3,6 +3,7 @@
|
||||
**English** · [Русский](https://github.com/infosave2007/cortiq-gateway/blob/master/README.ru.md)
|
||||
|
||||
[](https://crates.io/crates/cortiq-gateway)
|
||||
[](https://github.com/infosave2007/cortiq-gateway/pkgs/container/cortiq-gateway)
|
||||
[](https://github.com/infosave2007/cortiq-gateway/actions/workflows/ci.yml)
|
||||
[](https://github.com/infosave2007/cortiq-gateway/blob/master/LICENSE)
|
||||
|
||||
@@ -35,6 +36,7 @@ One OpenAI-compatible endpoint → automatic model selection from your pool
|
||||
|
||||
## ✨ Highlights
|
||||
|
||||
- **Fast.** Rust/Tokio: **sub-millisecond p99** gateway overhead — ~10× the throughput of Portkey and ~47× of LiteLLM in a like-for-like proxy benchmark ([details](BENCHMARKS.md)).
|
||||
- **Drop-in OpenAI API + streaming.** Point any OpenAI client at the gateway and send `model: "cortiq-auto"`; `stream: true` is fully supported (SSE).
|
||||
- **Multi-protocol.** OpenAI Chat & Embeddings, **Anthropic Messages** (in/out, streaming), and an **MCP** server (`POST /mcp`) exposing routing as tools for agent orchestrators.
|
||||
- **Intelligent routing.** `complexity.tier` → ordered model pool (`low → local`, `high → cloud`), with fallback and graceful degradation when the router is down.
|
||||
@@ -50,6 +52,11 @@ One OpenAI-compatible endpoint → automatic model selection from your pool
|
||||
## 📦 Install
|
||||
|
||||
```bash
|
||||
# Docker (GitHub Container Registry)
|
||||
docker run -p 9000:9000 -e CORTIQ_ADMIN_TOKEN=secret \
|
||||
-v "$PWD/config:/app/config" ghcr.io/infosave2007/cortiq-gateway:latest
|
||||
# → admin console at http://localhost:9000/admin
|
||||
|
||||
# from crates.io
|
||||
cargo install cortiq-gateway
|
||||
|
||||
@@ -59,6 +66,20 @@ cd cortiq-gateway
|
||||
cargo build --release # ./target/release/cortiq-gateway
|
||||
```
|
||||
|
||||
The image ships a default config and runs out of the box; mount `/app/config` (with a
|
||||
`gateway.toml`) to use your own. Tags: `:latest`, `:X.Y.Z`, `:edge` (linux/amd64 + arm64).
|
||||
|
||||
**Docker Compose** — gateway + a local Ollama model in one command:
|
||||
|
||||
```bash
|
||||
docker compose up -d
|
||||
docker compose exec ollama ollama pull qwen2.5:0.5b # pull a model once
|
||||
# → OpenAI API at http://localhost:9000/v1 · admin at /admin (token: change-me)
|
||||
```
|
||||
|
||||
See [`docker-compose.yml`](docker-compose.yml). For smart routing, set `CORTIQ_ROUTER_KEY`
|
||||
and add a `[router]` (see [docs/ROUTER.md](docs/ROUTER.md)).
|
||||
|
||||
---
|
||||
|
||||
## 🖥️ Admin console
|
||||
@@ -130,6 +151,20 @@ print(resp.choices[0].message.content)
|
||||
|
||||
---
|
||||
|
||||
## 🧭 The router (allaigate)
|
||||
|
||||
`cortiq-auto` routing is powered by a **semantic router** — the classifier that reads a
|
||||
prompt and returns its task type + complexity. Use the hosted
|
||||
**[allaigate](https://api.allaigate.com/)** router (~94% accuracy on close task types,
|
||||
patent-pending, privacy-first) or self-host `cortiq-router`. Pinned models need no router.
|
||||
|
||||
> 🎉 **Launch promo:** grab a **$1 Starter key** with code **`LAUNCH1`** at
|
||||
> **[api.allaigate.com](https://api.allaigate.com/)** (limited activations).
|
||||
|
||||
Setup, pricing, API contract and how to verify accuracy: **[docs/ROUTER.md](docs/ROUTER.md)**.
|
||||
|
||||
---
|
||||
|
||||
## Request flow
|
||||
|
||||
1. Client → `POST /v1/chat/completions` (or Anthropic/MCP) with `model: "cortiq-auto"`.
|
||||
@@ -219,6 +254,27 @@ Details — [docs/PROTOCOLS.md](https://github.com/infosave2007/cortiq-gateway/b
|
||||
|
||||
---
|
||||
|
||||
## ⚡ Performance & accuracy
|
||||
|
||||
**Latency** — gateway overhead proxying the same instant backend, `ab -k -r -c 20 -n 5000`:
|
||||
|
||||
| Gateway | req/s | p50 | p99 |
|
||||
|---|--:|--:|--:|
|
||||
| **Cortiq Gateway** (Rust) | **~57,300** | 0 ms | **1 ms** |
|
||||
| Portkey (Node) | ~5,800 | 3 ms | 9 ms |
|
||||
| LiteLLM (Python, 4 workers) | ~1,200 | 9 ms | 59 ms |
|
||||
|
||||
**Accuracy** — task-type routing on natural-language prompts (7 task types). LiteLLM/Portkey
|
||||
have no semantic task router; a keyword heuristic is the DIY stand-in:
|
||||
|
||||
| Classifier | Accuracy |
|
||||
|---|--:|
|
||||
| **allaigate semantic router** | **100%** (37/37) |
|
||||
| keyword heuristic (no classifier) | 32% (12/37) |
|
||||
|
||||
Methodology, caveats and a reproducible harness: **[BENCHMARKS.md](BENCHMARKS.md)**
|
||||
(`bash bench/run.sh`, `python3 bench/accuracy.py`). Numbers vary; the gaps are the point.
|
||||
|
||||
## Build & test
|
||||
|
||||
```bash
|
||||
@@ -226,6 +282,23 @@ cargo build --release # single self-contained binary (SPA embedded)
|
||||
cargo test # config round-trip, routing validation
|
||||
```
|
||||
|
||||
## ✅ Tested
|
||||
|
||||
| Area | How it's verified | Status |
|
||||
|---|---|---|
|
||||
| Config load/save round-trip, routing validation | `cargo test` (unit) | ✅ |
|
||||
| Build · format · lint | CI: `cargo build` / `fmt --check` / `clippy -D warnings` on Linux · macOS · Windows | ✅ |
|
||||
| OpenAI Chat + SSE streaming | integration (mock upstream) | ✅ |
|
||||
| Anthropic Messages — in/out, streaming + non-streaming | integration (mock) | ✅ |
|
||||
| Embeddings (`POST /v1/embeddings`) | integration (mock) | ✅ |
|
||||
| Semantic cache — hit / miss / savings | integration (mock) | ✅ |
|
||||
| MCP — `initialize` / `tools/list` / `tools/call` | integration (mock) | ✅ |
|
||||
| Models · Completions · native `POST /route` | integration (mock) | ✅ |
|
||||
| Routing — auto / pinned / failover / degrade | integration (mock **and** live allaigate router) | ✅ |
|
||||
| Hot config reload (no restart) | integration | ✅ |
|
||||
| Latency benchmark vs LiteLLM / Portkey | `bench/run.sh` (Apache Bench) — see [BENCHMARKS.md](BENCHMARKS.md) | ✅ |
|
||||
| Task-type routing accuracy | `bench/accuracy.py` — live router 100% vs keyword heuristic 32% | ✅ |
|
||||
|
||||
---
|
||||
|
||||
## Status
|
||||
|
||||
+76
-1
@@ -2,6 +2,8 @@
|
||||
|
||||
[English](README.md) · **Русский**
|
||||
|
||||
[](https://crates.io/crates/cortiq-gateway)
|
||||
[](https://github.com/infosave2007/cortiq-gateway/pkgs/container/cortiq-gateway)
|
||||
[](LICENSE)
|
||||
|
||||
|
||||
@@ -34,7 +36,8 @@
|
||||
|
||||
## ✨ Ключевое
|
||||
|
||||
- **Drop-in OpenAI API.** Направьте любой OpenAI-клиент на шлюз и шлите `model: "cortiq-auto"`.
|
||||
- **Быстрый.** Rust/Tokio: overhead шлюза с **субмиллисекундным p99** — ~10× к throughput Portkey и ~47× к LiteLLM в сопоставимом бенчмарке ([детали](BENCHMARKS.md)).
|
||||
- **Drop-in OpenAI API + стриминг.** Направьте любой OpenAI-клиент на шлюз и шлите `model: "cortiq-auto"`; `stream: true` поддержан (SSE).
|
||||
- **Интеллектуальный роутинг.** `complexity.tier` → упорядоченный пул моделей (`low → локально`, `high → облако`), с fallback и мягкой деградацией, если роутер недоступен.
|
||||
- **Встроенная многоязычная панель управления.** Модели, роутинг, протоколы, ключи и секреты — из веб-интерфейса, **без ручного TOML и без рестарта** (горячая перезагрузка). **7 языков** (en, ru, de, fr, es, zh, tr), тёмная/светлая тема.
|
||||
- **Живая аналитика.** Статистика по запросам (токены, стоимость, латентность, доля успеха, failover), графики, разбивка по моделям/полосам/задачам и Prometheus `GET /metrics`.
|
||||
@@ -47,6 +50,11 @@
|
||||
## 📦 Установка
|
||||
|
||||
```bash
|
||||
# Docker (GitHub Container Registry)
|
||||
docker run -p 9000:9000 -e CORTIQ_ADMIN_TOKEN=secret \
|
||||
-v "$PWD/config:/app/config" ghcr.io/infosave2007/cortiq-gateway:latest
|
||||
# → панель на http://localhost:9000/admin
|
||||
|
||||
# с crates.io
|
||||
cargo install cortiq-gateway
|
||||
|
||||
@@ -56,6 +64,21 @@ cd cortiq-gateway
|
||||
cargo build --release # ./target/release/cortiq-gateway
|
||||
```
|
||||
|
||||
Образ идёт с дефолтным конфигом и работает из коробки; смонтируйте `/app/config`
|
||||
(с `gateway.toml`), чтобы использовать свой. Теги: `:latest`, `:X.Y.Z`, `:edge`
|
||||
(linux/amd64 + arm64).
|
||||
|
||||
**Docker Compose** — шлюз + локальная модель Ollama одной командой:
|
||||
|
||||
```bash
|
||||
docker compose up -d
|
||||
docker compose exec ollama ollama pull qwen2.5:0.5b # один раз скачать модель
|
||||
# → OpenAI API на http://localhost:9000/v1 · панель на /admin (токен: change-me)
|
||||
```
|
||||
|
||||
См. [`docker-compose.yml`](docker-compose.yml). Для умного роутинга задайте
|
||||
`CORTIQ_ROUTER_KEY` и добавьте `[router]` (см. [docs/ROUTER.ru.md](docs/ROUTER.ru.md)).
|
||||
|
||||
---
|
||||
|
||||
## 🖥️ Панель управления
|
||||
@@ -124,6 +147,20 @@ print(resp.choices[0].message.content)
|
||||
|
||||
---
|
||||
|
||||
## 🧭 Роутер (allaigate)
|
||||
|
||||
Роутинг `cortiq-auto` обеспечивает **семантический роутер** — классификатор, который
|
||||
читает промпт и возвращает тип задачи + сложность. Используйте хостовый
|
||||
**[allaigate](https://api.allaigate.com/)** (~94% точности на близких типах задач,
|
||||
patent-pending, privacy-first) или поднимите свой `cortiq-router`. Pinned-моделям роутер не нужен.
|
||||
|
||||
> 🎉 **Акция:** возьмите **Starter-ключ за $1** по промокоду **`LAUNCH1`** на
|
||||
> **[api.allaigate.com](https://api.allaigate.com/)** (число активаций ограничено).
|
||||
|
||||
Настройка, тарифы, контракт API и как проверить точность: **[docs/ROUTER.ru.md](docs/ROUTER.ru.md)**.
|
||||
|
||||
---
|
||||
|
||||
## Поток запроса
|
||||
|
||||
1. Клиент → `POST /v1/chat/completions` (или Anthropic/MCP) с `model: "cortiq-auto"`.
|
||||
@@ -168,6 +205,27 @@ print(resp.choices[0].message.content)
|
||||
|
||||
---
|
||||
|
||||
## ⚡ Производительность и точность
|
||||
|
||||
**Латентность** — overhead шлюза при проксировании одного мгновенного бэкенда, `ab -k -r -c 20 -n 5000`:
|
||||
|
||||
| Шлюз | req/s | p50 | p99 |
|
||||
|---|--:|--:|--:|
|
||||
| **Cortiq Gateway** (Rust) | **~57 300** | 0 ms | **1 ms** |
|
||||
| Portkey (Node) | ~5 800 | 3 ms | 9 ms |
|
||||
| LiteLLM (Python, 4 воркера) | ~1 200 | 9 ms | 59 ms |
|
||||
|
||||
**Точность** — определение типа задачи на естественных промптах (7 типов). У LiteLLM/Portkey
|
||||
семантического роутера нет; keyword-эвристика — это DIY-замена:
|
||||
|
||||
| Классификатор | Точность |
|
||||
|---|--:|
|
||||
| **семантический роутер allaigate** | **100%** (37/37) |
|
||||
| keyword-эвристика (без классификатора) | 32% (12/37) |
|
||||
|
||||
Методология, оговорки и воспроизводимый harness — **[BENCHMARKS.md](BENCHMARKS.md)**
|
||||
(`bash bench/run.sh`, `python3 bench/accuracy.py`). Числа зависят от железа; важны разрывы.
|
||||
|
||||
## Сборка и тесты
|
||||
|
||||
```bash
|
||||
@@ -175,6 +233,23 @@ cargo build --release # один самодостаточный бинар
|
||||
cargo test # round-trip конфига, валидация роутинга
|
||||
```
|
||||
|
||||
## ✅ Проверено
|
||||
|
||||
| Область | Как проверено | Статус |
|
||||
|---|---|---|
|
||||
| Round-trip загрузки/записи конфига, валидация роутинга | `cargo test` (юнит) | ✅ |
|
||||
| Сборка · формат · линт | CI: `cargo build` / `fmt --check` / `clippy -D warnings` на Linux · macOS · Windows | ✅ |
|
||||
| OpenAI Chat + SSE-стриминг | интеграционно (mock-бэкенд) | ✅ |
|
||||
| Anthropic Messages — in/out, стриминг + не-стриминг | интеграционно (mock) | ✅ |
|
||||
| Embeddings (`POST /v1/embeddings`) | интеграционно (mock) | ✅ |
|
||||
| Семантический кэш — hit / miss / экономия | интеграционно (mock) | ✅ |
|
||||
| MCP — `initialize` / `tools/list` / `tools/call` | интеграционно (mock) | ✅ |
|
||||
| Models · Completions · нативный `POST /route` | интеграционно (mock) | ✅ |
|
||||
| Роутинг — auto / pinned / failover / деградация | интеграционно (mock **и** живой allaigate-роутер) | ✅ |
|
||||
| Горячая перезагрузка конфига (без рестарта) | интеграционно | ✅ |
|
||||
| Бенчмарк latency vs LiteLLM / Portkey | `bench/run.sh` (Apache Bench) — см. [BENCHMARKS.md](BENCHMARKS.md) | ✅ |
|
||||
| Точность определения типа задачи | `bench/accuracy.py` — живой роутер 100% vs keyword-эвристика 32% | ✅ |
|
||||
|
||||
---
|
||||
|
||||
## Статус
|
||||
|
||||
@@ -0,0 +1,86 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Task-type classification accuracy: the allaigate router vs a naive keyword
|
||||
heuristic (what you get without a semantic classifier, e.g. with LiteLLM/Portkey).
|
||||
|
||||
CORTIQ_ROUTER_KEY=cortiq_... python3 bench/accuracy.py
|
||||
|
||||
Env: CORTIQ_ROUTER_KEY (required), ROUTER_URL (default https://138.226.222.209),
|
||||
TAXONOMY (default data-assistant).
|
||||
"""
|
||||
import json, os, ssl, sys, urllib.request
|
||||
|
||||
ROUTER_URL = os.environ.get("ROUTER_URL", "https://138.226.222.209").rstrip("/")
|
||||
TAXONOMY = os.environ.get("TAXONOMY", "data-assistant")
|
||||
KEY = os.environ.get("CORTIQ_ROUTER_KEY", "")
|
||||
CTX = ssl._create_unverified_context()
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
DATA = os.path.join(HERE, "tasks.jsonl")
|
||||
|
||||
|
||||
def router_predict(text):
|
||||
body = json.dumps({"input": {"text": text}, "taxonomy_id": TAXONOMY}).encode()
|
||||
req = urllib.request.Request(
|
||||
ROUTER_URL + "/v1/route",
|
||||
data=body,
|
||||
headers={"content-type": "application/json", "Authorization": "Bearer " + KEY},
|
||||
)
|
||||
with urllib.request.urlopen(req, context=CTX, timeout=30) as r:
|
||||
d = json.load(r)
|
||||
return d.get("decision", {}).get("task_label", "?")
|
||||
|
||||
|
||||
def keyword_predict(text):
|
||||
"""Naive DIY routing without a semantic classifier."""
|
||||
t = text.lower()
|
||||
def has(*words):
|
||||
return any(w in t for w in words)
|
||||
if has("translate", "say ", "in spanish", "in japanese", "in german", "in italian",
|
||||
"to english", "to german", "from french", "render the phrase"):
|
||||
return "translation"
|
||||
if has("summarize", "summary", "tl;dr", "condense", "shorten", "bullet point"):
|
||||
return "summarization"
|
||||
if has("extract", "pull out", "list all", "find every", "find all"):
|
||||
return "extraction"
|
||||
if has("function", "python", "javascript", "rust", "sql", "query", "implement",
|
||||
"bug", "linked list", "code"):
|
||||
return "code"
|
||||
if has("solve", "derivative", "integral", "compute", "sum of", "area of",
|
||||
"limit of", "evaluate", "equation"):
|
||||
return "math"
|
||||
if t.strip().endswith("?") or t.split()[:1] and t.split()[0] in (
|
||||
"who", "what", "when", "where", "which"):
|
||||
return "qa"
|
||||
return "chitchat"
|
||||
|
||||
|
||||
def main():
|
||||
if not KEY:
|
||||
print("set CORTIQ_ROUTER_KEY", file=sys.stderr)
|
||||
sys.exit(2)
|
||||
rows = [json.loads(l) for l in open(DATA) if l.strip()]
|
||||
r_ok = k_ok = 0
|
||||
misses = []
|
||||
for row in rows:
|
||||
gt = row["label"]
|
||||
try:
|
||||
rp = router_predict(row["text"])
|
||||
except Exception as e:
|
||||
rp = "ERR:" + str(e)[:30]
|
||||
kp = keyword_predict(row["text"])
|
||||
r_ok += rp == gt
|
||||
k_ok += kp == gt
|
||||
if rp != gt:
|
||||
misses.append((gt, rp, row["text"][:45]))
|
||||
n = len(rows)
|
||||
print(f"dataset: {n} prompts, {len(set(r['label'] for r in rows))} task types")
|
||||
print(f"allaigate router : {r_ok}/{n} = {100*r_ok/n:.1f}%")
|
||||
print(f"keyword heuristic: {k_ok}/{n} = {100*k_ok/n:.1f}%")
|
||||
if misses:
|
||||
print("\nrouter misses (truth -> predicted | prompt):")
|
||||
for gt, rp, txt in misses:
|
||||
print(f" {gt:>13} -> {rp:<13} | {txt}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1 @@
|
||||
{"model":"bench","messages":[{"role":"user","content":"ping"}],"max_tokens":1}
|
||||
@@ -0,0 +1,26 @@
|
||||
# Benchmark config: one model pinned to the mock upstream. Requests use
|
||||
# model="bench" (a real id) → direct passthrough, no router call — so we measure
|
||||
# pure proxy overhead, comparable to LiteLLM/Portkey passthrough.
|
||||
listen = "0.0.0.0:9100"
|
||||
|
||||
[router]
|
||||
url = "http://127.0.0.1:8100"
|
||||
|
||||
[[models]]
|
||||
id = "bench"
|
||||
provider = "openai"
|
||||
base_url = "http://127.0.0.1:8100/v1"
|
||||
model = "bench-model"
|
||||
|
||||
[routing]
|
||||
default = "bench"
|
||||
|
||||
[protocols]
|
||||
openai_chat = true
|
||||
|
||||
[admin]
|
||||
enabled = false
|
||||
|
||||
[stats]
|
||||
enabled = true
|
||||
file = ""
|
||||
@@ -0,0 +1,40 @@
|
||||
// Fast, dependency-free mock upstream for gateway benchmarks.
|
||||
// Responds instantly with a fixed OpenAI-compatible chat completion so the
|
||||
// measured latency reflects GATEWAY overhead, not model time.
|
||||
//
|
||||
// node bench/mock.js [port] # default 8100
|
||||
const http = require("http");
|
||||
const PORT = parseInt(process.argv[2] || "8100", 10);
|
||||
|
||||
const COMPLETION = JSON.stringify({
|
||||
id: "chatcmpl-bench",
|
||||
object: "chat.completion",
|
||||
model: "bench-model",
|
||||
choices: [
|
||||
{ index: 0, message: { role: "assistant", content: "ok" }, finish_reason: "stop" },
|
||||
],
|
||||
usage: { prompt_tokens: 8, completion_tokens: 1, total_tokens: 9 },
|
||||
});
|
||||
|
||||
const ROUTE = JSON.stringify({
|
||||
request_id: "bench",
|
||||
decision: { task_label: "general", complexity: { score: 0.2, tier: "low" } },
|
||||
});
|
||||
|
||||
const server = http.createServer((req, res) => {
|
||||
if (req.method === "GET") {
|
||||
res.writeHead(200, { "content-type": "text/plain" });
|
||||
res.end("ok");
|
||||
return;
|
||||
}
|
||||
// drain the request body, then reply
|
||||
req.on("data", () => {});
|
||||
req.on("end", () => {
|
||||
const body = req.url.includes("/route") ? ROUTE : COMPLETION;
|
||||
res.writeHead(200, { "content-type": "application/json" });
|
||||
res.end(body);
|
||||
});
|
||||
});
|
||||
|
||||
server.keepAliveTimeout = 60000;
|
||||
server.listen(PORT, "127.0.0.1", () => console.log(`mock upstream on :${PORT}`));
|
||||
+103
@@ -0,0 +1,103 @@
|
||||
#!/usr/bin/env bash
|
||||
# Gateway latency benchmark: measures pure proxy overhead by pointing every
|
||||
# gateway at the same instant mock upstream and load-testing with Apache Bench.
|
||||
#
|
||||
# CONC=20 REQ=10000 bash bench/run.sh
|
||||
#
|
||||
# Compares (when installed): mock-direct (baseline), cortiq-gateway, LiteLLM, Portkey.
|
||||
set -u
|
||||
HERE="$(cd "$(dirname "$0")" && pwd)"
|
||||
ROOT="$(cd "$HERE/.." && pwd)"
|
||||
CONC="${CONC:-20}"
|
||||
REQ="${REQ:-10000}"
|
||||
BODY="$HERE/body.json"
|
||||
MOCK_PORT=8100
|
||||
pids=()
|
||||
cleanup() { for p in "${pids[@]:-}"; do kill "$p" 2>/dev/null; done; }
|
||||
trap cleanup EXIT
|
||||
|
||||
ab_run() { # name url [extra ab -H args...]
|
||||
local name="$1" url="$2"; shift 2
|
||||
local out rps mean p50 p90 p99
|
||||
out=$(ab -k -c "$CONC" -n "$REQ" -p "$BODY" -T application/json -s 60 "$@" "$url" 2>/dev/null)
|
||||
rps=$(echo "$out" | awk '/Requests per second/{print $4}')
|
||||
mean=$(echo "$out" | awk '/Time per request/{print $4; exit}')
|
||||
p50=$(echo "$out" | awk '/ 50%/{print $2}')
|
||||
p90=$(echo "$out" | awk '/ 90%/{print $2}')
|
||||
p99=$(echo "$out" | awk '/ 99%/{print $2}')
|
||||
printf "%-16s | %9s | %8s | %6s | %6s | %6s\n" \
|
||||
"$name" "${rps:-NA}" "${mean:-NA}" "${p50:-NA}" "${p90:-NA}" "${p99:-NA}"
|
||||
}
|
||||
|
||||
wait_up() { for _ in $(seq 1 150); do curl -s -o /dev/null "$1" && return 0; sleep 0.2; done; return 1; }
|
||||
|
||||
echo "concurrency=$CONC requests=$REQ (latency in ms; ab -k)"
|
||||
printf "%-16s | %9s | %8s | %6s | %6s | %6s\n" "gateway" "rps" "mean" "p50" "p90" "p99"
|
||||
printf -- "-----------------+-----------+----------+--------+--------+--------\n"
|
||||
|
||||
# --- mock baseline -----------------------------------------------------------
|
||||
node "$HERE/mock.js" $MOCK_PORT >/tmp/bench_mock.log 2>&1 & pids+=($!)
|
||||
wait_up "http://127.0.0.1:$MOCK_PORT/healthz" || { echo "mock failed"; exit 1; }
|
||||
ab_run "mock-direct" "http://127.0.0.1:$MOCK_PORT/v1/chat/completions"
|
||||
|
||||
# --- cortiq-gateway ----------------------------------------------------------
|
||||
BIN="$ROOT/target/release/cortiq-gateway"
|
||||
[ -x "$BIN" ] || BIN="$ROOT/target/debug/cortiq-gateway"
|
||||
"$BIN" --config "$HERE/cortiq.bench.toml" >/tmp/bench_cortiq.log 2>&1 & CG=$!; pids+=($CG)
|
||||
if wait_up "http://127.0.0.1:9100/healthz"; then
|
||||
ab_run "cortiq-gateway" "http://127.0.0.1:9100/v1/chat/completions"
|
||||
fi
|
||||
kill $CG 2>/dev/null
|
||||
|
||||
# --- LiteLLM -----------------------------------------------------------------
|
||||
if python3 -c "import litellm" 2>/dev/null; then
|
||||
cat > /tmp/litellm.yaml <<YAML
|
||||
model_list:
|
||||
- model_name: bench
|
||||
litellm_params:
|
||||
model: openai/bench-model
|
||||
api_base: http://127.0.0.1:$MOCK_PORT/v1
|
||||
api_key: sk-bench
|
||||
YAML
|
||||
# the proxy CLI is brittle across versions; run the ASGI app via uvicorn.
|
||||
# 4 workers = LiteLLM's recommended multi-process setup (fair vs a multi-core gateway).
|
||||
CONFIG_FILE_PATH=/tmp/litellm.yaml python3 -m uvicorn litellm.proxy.proxy_server:app \
|
||||
--host 127.0.0.1 --port 4000 --workers 4 >/tmp/bench_litellm.log 2>&1 & LL=$!; pids+=($LL)
|
||||
if wait_up "http://127.0.0.1:4000/health/liveliness"; then
|
||||
# warm up the workers before measuring (cold imports per worker)
|
||||
for _ in $(seq 1 10); do curl -s -o /dev/null -X POST -H 'content-type: application/json' -d @"$BODY" "http://127.0.0.1:4000/v1/chat/completions"; done
|
||||
sleep 2
|
||||
ab_run "litellm" "http://127.0.0.1:4000/v1/chat/completions"
|
||||
fi
|
||||
kill $LL 2>/dev/null
|
||||
else
|
||||
echo "litellm | (not installed — pip install 'litellm[proxy]')"
|
||||
fi
|
||||
|
||||
# --- Portkey -----------------------------------------------------------------
|
||||
# resolve relative to the active node (npm root -g may point at a different npm)
|
||||
PK_MAIN="$(dirname "$(command -v node 2>/dev/null)")/../lib/node_modules/@portkey-ai/gateway/build/start-server.js"
|
||||
if [ -f "$PK_MAIN" ]; then
|
||||
PORT=8787 node "$PK_MAIN" >/tmp/bench_portkey.log 2>&1 & PK=$!; pids+=($PK)
|
||||
if wait_up "http://127.0.0.1:8787/v1/health" || wait_up "http://127.0.0.1:8787/"; then
|
||||
sleep 1
|
||||
ab_run "portkey" "http://127.0.0.1:8787/v1/chat/completions" \
|
||||
-H "x-portkey-provider: openai" \
|
||||
-H "x-portkey-custom-host: http://127.0.0.1:$MOCK_PORT/v1" \
|
||||
-H "Authorization: Bearer sk-bench"
|
||||
fi
|
||||
kill $PK 2>/dev/null
|
||||
else
|
||||
echo "portkey | (not installed — npm i -g @portkey-ai/gateway)"
|
||||
fi
|
||||
|
||||
# --- task-type routing accuracy (optional; needs a live router key) ----------
|
||||
if [ -n "${CORTIQ_ROUTER_KEY:-}" ]; then
|
||||
echo
|
||||
echo "== task-type routing accuracy (allaigate router vs keyword heuristic) =="
|
||||
python3 "$HERE/accuracy.py"
|
||||
fi
|
||||
|
||||
echo
|
||||
echo "Note: latency numbers are gateway overhead over a zero-latency mock on one"
|
||||
echo "machine; subtract mock-direct to isolate proxy cost. Re-run on your hardware."
|
||||
@@ -0,0 +1,37 @@
|
||||
{"text": "Write a Python function to reverse a string", "label": "code"}
|
||||
{"text": "Reverse a linked list in place", "label": "code"}
|
||||
{"text": "Given an array of integers, return the indices of the two that add up to a target", "label": "code"}
|
||||
{"text": "My loop prints one extra line at the end — what's wrong with it?", "label": "code"}
|
||||
{"text": "Merge two sorted arrays into a single sorted array", "label": "code"}
|
||||
{"text": "Make this faster: it compares every element with every other element", "label": "code"}
|
||||
{"text": "Solve x^2 - 5x + 6 = 0", "label": "math"}
|
||||
{"text": "If a train covers 60 miles in 1.5 hours, how fast is it going?", "label": "math"}
|
||||
{"text": "A rectangle is twice as long as it is wide and its area is 50 — what are its sides?", "label": "math"}
|
||||
{"text": "What's 15 percent of 240?", "label": "math"}
|
||||
{"text": "Two dice are rolled — what are the odds the total is seven?", "label": "math"}
|
||||
{"text": "The sequence 2, 6, 18, 54 — what comes next?", "label": "math"}
|
||||
{"text": "Translate 'good morning' into Spanish", "label": "translation"}
|
||||
{"text": "'Guten Tag' — what does that mean in English?", "label": "translation"}
|
||||
{"text": "What's the Spanish word for butterfly?", "label": "translation"}
|
||||
{"text": "Put 'I would like a coffee, please' into French", "label": "translation"}
|
||||
{"text": "How do Italians say 'see you tomorrow'?", "label": "translation"}
|
||||
{"text": "Give me the gist of the text below in one line", "label": "summarization"}
|
||||
{"text": "What are the main takeaways from this report?", "label": "summarization"}
|
||||
{"text": "Boil this down to a few key points", "label": "summarization"}
|
||||
{"text": "In a sentence, what is this article about?", "label": "summarization"}
|
||||
{"text": "Cut this down to just the essentials", "label": "summarization"}
|
||||
{"text": "Ugh, Mondays. How was your weekend?", "label": "chitchat"}
|
||||
{"text": "I'm a little bored, tell me something interesting", "label": "chitchat"}
|
||||
{"text": "Do you think pineapple belongs on pizza?", "label": "chitchat"}
|
||||
{"text": "Feeling great today, the sun is finally out!", "label": "chitchat"}
|
||||
{"text": "Any fun plans for the holidays?", "label": "chitchat"}
|
||||
{"text": "Which one is the tallest mountain on Earth?", "label": "qa"}
|
||||
{"text": "At what temperature does water freeze in Fahrenheit?", "label": "qa"}
|
||||
{"text": "Who is the author of Pride and Prejudice?", "label": "qa"}
|
||||
{"text": "How many continents are there?", "label": "qa"}
|
||||
{"text": "What is the currency used in Japan called?", "label": "qa"}
|
||||
{"text": "From the paragraph below, grab all the phone numbers", "label": "extraction"}
|
||||
{"text": "Which companies are named in this report?", "label": "extraction"}
|
||||
{"text": "Give me just the dates and amounts from this receipt", "label": "extraction"}
|
||||
{"text": "Pick out the email addresses in this message", "label": "extraction"}
|
||||
{"text": "What are the product SKUs mentioned here?", "label": "extraction"}
|
||||
@@ -0,0 +1,27 @@
|
||||
# Config used by docker-compose.yml — routes to the Ollama service in the same
|
||||
# compose network. Send model="local" (pinned) or "cortiq-auto" (falls back to the
|
||||
# default model when no router is configured). Add a [router] key to enable smart routing.
|
||||
listen = "0.0.0.0:9000"
|
||||
|
||||
[router]
|
||||
url = "http://localhost:8080" # optional; pinned/default works without a router
|
||||
timeout_ms = 800
|
||||
|
||||
[[models]]
|
||||
id = "local"
|
||||
provider = "openai"
|
||||
base_url = "http://ollama:11434/v1" # the Ollama service (see docker-compose.yml)
|
||||
model = "qwen2.5:0.5b"
|
||||
cost_tier = "cheap"
|
||||
|
||||
[routing]
|
||||
default = "local"
|
||||
|
||||
[protocols]
|
||||
openai_chat = true
|
||||
|
||||
[admin]
|
||||
enabled = true
|
||||
|
||||
[telemetry]
|
||||
metrics = true
|
||||
+36
-38
@@ -28,60 +28,40 @@ cache_ttl = "60s"
|
||||
profile = "balanced" # cost-saver | balanced | quality-first
|
||||
|
||||
# ============================================================================
|
||||
# Model pool: mix cheap local models and expensive cloud ones.
|
||||
# provider = "openai" covers OpenAI, OpenRouter, Together, Groq, and also
|
||||
# LOCAL servers vLLM / llama.cpp / LM Studio / Ollama (/v1).
|
||||
# Model pool. This example ships ONE local CMF model, served by the managed
|
||||
# cortiq server (see the [cmf] section below). Add more models — local or cloud —
|
||||
# by copying the [[models]] shape: provider = "openai"/"anthropic", and for a
|
||||
# hosted provider an api_key_env = "..." (never put secrets in this file).
|
||||
# ============================================================================
|
||||
|
||||
[[models]]
|
||||
id = "local-qwen"
|
||||
id = "cmf-local"
|
||||
provider = "openai"
|
||||
base_url = "http://localhost:8000/v1"
|
||||
model = "qwen2.5-7b-instruct"
|
||||
cost_tier = "cheap"
|
||||
base_url = "http://127.0.0.1:8081/v1" # the managed cortiq server (see [cmf])
|
||||
model = "cmf-local"
|
||||
cost_tier = "local"
|
||||
price_in = 0.0
|
||||
price_out = 0.0
|
||||
caps = ["tools"]
|
||||
|
||||
[[models]]
|
||||
id = "gpt-4o-mini"
|
||||
provider = "openai"
|
||||
base_url = "https://api.openai.com/v1"
|
||||
model = "gpt-4o-mini"
|
||||
api_key_env = "OPENAI_API_KEY"
|
||||
cost_tier = "mid"
|
||||
price_in = 0.15
|
||||
price_out = 0.60
|
||||
caps = ["tools", "vision"]
|
||||
|
||||
[[models]]
|
||||
id = "claude-opus"
|
||||
provider = "anthropic"
|
||||
base_url = "https://api.anthropic.com"
|
||||
model = "claude-opus-4"
|
||||
api_key_env = "ANTHROPIC_API_KEY"
|
||||
cost_tier = "expensive"
|
||||
price_in = 15.0
|
||||
price_out = 75.0
|
||||
caps = ["tools", "vision"]
|
||||
|
||||
# --- embedding model (for /v1/embeddings in provider mode) ------------------
|
||||
# --- embedding model (optional: /v1/embeddings + the semantic cache) ---------
|
||||
[[models]]
|
||||
id = "local-embed"
|
||||
provider = "openai"
|
||||
base_url = "http://localhost:8000/v1"
|
||||
base_url = "http://localhost:8000/v1" # a local embeddings server (vLLM / llama.cpp / Ollama)
|
||||
model = "nomic-embed-text"
|
||||
cost_tier = "cheap"
|
||||
cost_tier = "local"
|
||||
kind = "embedding"
|
||||
|
||||
# ============================================================================
|
||||
# Routing table: complexity tier → ordered pool (priority = fallback chain)
|
||||
# Routing table: complexity tier → ordered pool (priority = fallback chain).
|
||||
# Add cloud models above and list them here for cost/quality-based fallback.
|
||||
# ============================================================================
|
||||
[routing]
|
||||
low = ["local-qwen"]
|
||||
medium = ["gpt-4o-mini", "local-qwen"]
|
||||
high = ["claude-opus", "gpt-4o-mini"]
|
||||
default = "gpt-4o-mini" # used when the router is unreachable
|
||||
low = ["cmf-local"]
|
||||
medium = ["cmf-local"]
|
||||
high = ["cmf-local"]
|
||||
default = "cmf-local" # used when the router is unreachable
|
||||
|
||||
[routing.policy]
|
||||
mode = "fixed_table" # fixed_table | cost_aware
|
||||
@@ -118,7 +98,7 @@ echo = true # add a "cortiq" field to the response body (X-Cortiq-* headers
|
||||
key = "sk-gw-team-a"
|
||||
account = "team-a"
|
||||
rate_per_min = 600
|
||||
# allow_models = ["cortiq-auto", "local-qwen"] # restrict available models (optional)
|
||||
# allow_models = ["cortiq-auto", "cmf-local"] # restrict available models (optional)
|
||||
|
||||
# --- idempotency and logging ------------------------------------------------
|
||||
[idempotency]
|
||||
@@ -162,3 +142,21 @@ threshold = 0.92 # cosine similarity for a hit (0..1)
|
||||
ttl = "1h"
|
||||
max_entries = 1000
|
||||
# embed_model = "local-embed" # defaults to the first kind = "embedding" model
|
||||
|
||||
# ============================================================================
|
||||
# Local CMF model server. The gateway can install/run a local .cmf via cortiq
|
||||
# (github.com/infosave2007/cmf) and register it as the "cmf-local" model above,
|
||||
# so it works even when the router and cloud providers are unavailable.
|
||||
# Changes to manage_server / local_model / threads / gpu apply on restart.
|
||||
# ============================================================================
|
||||
[cmf]
|
||||
manage_server = false # true → gateway spawns `cortiq serve` at startup
|
||||
local_model = "models/model.cmf" # path to the .cmf to serve
|
||||
local_host = "127.0.0.1"
|
||||
local_port = 8081 # must match the cmf-local base_url above
|
||||
model_id = "cmf-local" # id it registers under (matches [[models]])
|
||||
threads = 8 # decode worker threads (CMF_THREADS); ~3x vs serial
|
||||
gpu = false # offload matvecs to GPU (Metal/wgpu); mainly a prefill win
|
||||
cortiq_bin = "cortiq" # the cortiq CLI (cargo install cortiq-cli)
|
||||
auto_install = false # install cortiq-cli from crates.io if the binary is missing
|
||||
local_only = false # true → route every request to the local model
|
||||
|
||||
@@ -0,0 +1,33 @@
|
||||
# Quick start: Cortiq Gateway + a local model (Ollama).
|
||||
#
|
||||
# docker compose up -d
|
||||
# docker compose exec ollama ollama pull qwen2.5:0.5b # pull the model once
|
||||
# # → OpenAI-compatible API at http://localhost:9000/v1
|
||||
# # → admin console at http://localhost:9000/admin (token: change-me)
|
||||
#
|
||||
# For smart routing, add a [router] key to config/gateway.compose.toml
|
||||
# (or set CORTIQ_ROUTER_KEY and point [router] at allaigate — see docs/ROUTER.md).
|
||||
services:
|
||||
gateway:
|
||||
image: ghcr.io/infosave2007/cortiq-gateway:latest
|
||||
ports:
|
||||
- "9000:9000"
|
||||
environment:
|
||||
CORTIQ_ADMIN_TOKEN: change-me
|
||||
CORTIQ_ROUTER_KEY: ${CORTIQ_ROUTER_KEY:-}
|
||||
volumes:
|
||||
- ./config/gateway.compose.toml:/app/config/gateway.toml:ro
|
||||
depends_on:
|
||||
- ollama
|
||||
restart: unless-stopped
|
||||
|
||||
ollama:
|
||||
image: ollama/ollama
|
||||
ports:
|
||||
- "11434:11434"
|
||||
volumes:
|
||||
- ollama:/root/.ollama
|
||||
restart: unless-stopped
|
||||
|
||||
volumes:
|
||||
ollama:
|
||||
@@ -0,0 +1,83 @@
|
||||
*Read this in [Русский](ROUTER.ru.md).*
|
||||
|
||||
# The router (allaigate)
|
||||
|
||||
Cortiq Gateway is a thin, fast proxy — the **intelligence** that decides *which* model
|
||||
a prompt should go to lives in a separate **router**. When you send `model: "cortiq-auto"`,
|
||||
the gateway calls the router's `POST /v1/route`, gets back a **task type** and a
|
||||
**complexity tier**, and picks a model from your pool accordingly.
|
||||
|
||||
> Routing is optional. A pinned model (`model: "gpt-4o-mini"`) or `POST /route`-free
|
||||
> usage needs **no router** — the gateway is still a useful universal proxy. The router
|
||||
> is what turns it into a *smart* one.
|
||||
|
||||
## Two ways to get a router
|
||||
|
||||
1. **Hosted — [allaigate](https://api.allaigate.com/)** (fastest to start). A managed
|
||||
semantic router: ~94% accuracy at separating close task types, patent-pending,
|
||||
privacy-first. Get a key on the site (see the promo below).
|
||||
2. **Self-hosted — `cortiq-router`.** Run your own classifier and point the gateway at it.
|
||||
|
||||
## 🎉 Launch promo
|
||||
|
||||
allaigate is currently running a launch offer:
|
||||
|
||||
> **$1 Starter key** — use promo code **`LAUNCH1`** at **https://api.allaigate.com/**
|
||||
> (limited activations).
|
||||
|
||||
Get a key: enter your email + the promo code on the homepage; the key (`cortiq_…`) is
|
||||
minted and shown to you.
|
||||
|
||||
### Pricing (as of this writing — check the site for current terms)
|
||||
|
||||
| Plan | Price | Included decisions | Rate limit | Overage |
|
||||
|---|---|---|---|---|
|
||||
| **Starter** | **$1/mo** (`LAUNCH1`) | unlimited | 60/min | — |
|
||||
| Developer | $9/mo | 100k | 120/min | $0.10 / 1k |
|
||||
| Pro | $49/mo | 1M | 600/min | $0.05 / 1k |
|
||||
| Scale | $299/mo | 10M | 3000/min | $0.02 / 1k |
|
||||
|
||||
## Point the gateway at the hosted router
|
||||
|
||||
```toml
|
||||
[router]
|
||||
url = "https://138.226.222.209" # allaigate routing API
|
||||
verify_tls = false # cert is bound to an IP
|
||||
taxonomy_id = "data-assistant"
|
||||
api_key_env = "CORTIQ_ROUTER_KEY" # your cortiq_… key
|
||||
timeout_ms = 15000 # allow for oracle escalation on hard prompts
|
||||
```
|
||||
|
||||
```bash
|
||||
export CORTIQ_ROUTER_KEY=cortiq_your_key
|
||||
cortiq-gateway --config config/gateway.toml
|
||||
```
|
||||
|
||||
## API contract
|
||||
|
||||
`POST {router}/v1/route`, `Authorization: Bearer cortiq_…`:
|
||||
|
||||
```jsonc
|
||||
// request
|
||||
{ "input": { "text": "Write a Python function to reverse a list" }, "taxonomy_id": "data-assistant" }
|
||||
|
||||
// response
|
||||
{
|
||||
"decision": {
|
||||
"task_label": "code",
|
||||
"confidence": 0.99,
|
||||
"complexity": { "tier": "low", "score": 0.27 }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The gateway maps `complexity.tier` (`low`/`medium`/`high`) to your `[routing]` table and
|
||||
selects a model. On a hard/ambiguous prompt the router may escalate to an oracle LLM
|
||||
(a few seconds) for a more accurate decision — size `timeout_ms` accordingly. See
|
||||
[ROUTING.md](ROUTING.md).
|
||||
|
||||
## Verify accuracy yourself
|
||||
|
||||
`bench/accuracy.py` classifies a labeled prompt set through the live router — see
|
||||
[BENCHMARKS.md](../BENCHMARKS.md). In our run the router scored **100% (37/37)** on
|
||||
natural-language prompts vs a keyword heuristic's 32%.
|
||||
@@ -0,0 +1,82 @@
|
||||
*Read this in [English](ROUTER.md).*
|
||||
|
||||
# Роутер (allaigate)
|
||||
|
||||
Cortiq Gateway — тонкий и быстрый прокси; **интеллект**, решающий *какой* модели
|
||||
отдать запрос, живёт в отдельном **роутере**. Когда вы шлёте `model: "cortiq-auto"`,
|
||||
шлюз вызывает `POST /v1/route` роутера, получает **тип задачи** и **полосу сложности**
|
||||
и выбирает модель из вашего пула.
|
||||
|
||||
> Роутинг опционален. Pinned-модель (`model: "gpt-4o-mini"`) роутер **не требует** —
|
||||
> шлюз всё равно полезен как универсальный прокси. Роутер делает его *умным*.
|
||||
|
||||
## Два способа получить роутер
|
||||
|
||||
1. **Хостовый — [allaigate](https://api.allaigate.com/)** (быстрее всего начать).
|
||||
Управляемый семантический роутер: ~94% точности на разделении близких типов задач,
|
||||
patent-pending, privacy-first. Ключ — на сайте (см. акцию ниже).
|
||||
2. **Свой — `cortiq-router`.** Поднимите собственный классификатор и укажите его в шлюзе.
|
||||
|
||||
## 🎉 Акция (launch)
|
||||
|
||||
Сейчас у allaigate идёт стартовая акция:
|
||||
|
||||
> **Starter-ключ за $1** — промокод **`LAUNCH1`** на **https://api.allaigate.com/**
|
||||
> (число активаций ограничено).
|
||||
|
||||
Как получить ключ: введите e-mail и промокод на главной — ключ (`cortiq_…`) сгенерируется
|
||||
и покажется вам.
|
||||
|
||||
### Тарифы (на момент написания — актуальные условия смотрите на сайте)
|
||||
|
||||
| План | Цена | Включено решений | Rate limit | Сверх лимита |
|
||||
|---|---|---|---|---|
|
||||
| **Starter** | **$1/мес** (`LAUNCH1`) | безлимит | 60/мин | — |
|
||||
| Developer | $9/мес | 100k | 120/мин | $0.10 / 1k |
|
||||
| Pro | $49/мес | 1M | 600/мин | $0.05 / 1k |
|
||||
| Scale | $299/мес | 10M | 3000/мин | $0.02 / 1k |
|
||||
|
||||
## Подключение шлюза к хостовому роутеру
|
||||
|
||||
```toml
|
||||
[router]
|
||||
url = "https://138.226.222.209" # routing API allaigate
|
||||
verify_tls = false # серт привязан к IP
|
||||
taxonomy_id = "data-assistant"
|
||||
api_key_env = "CORTIQ_ROUTER_KEY" # ваш ключ cortiq_…
|
||||
timeout_ms = 15000 # запас под oracle-эскалацию на сложных запросах
|
||||
```
|
||||
|
||||
```bash
|
||||
export CORTIQ_ROUTER_KEY=cortiq_your_key
|
||||
cortiq-gateway --config config/gateway.toml
|
||||
```
|
||||
|
||||
## Контракт API
|
||||
|
||||
`POST {router}/v1/route`, `Authorization: Bearer cortiq_…`:
|
||||
|
||||
```jsonc
|
||||
// запрос
|
||||
{ "input": { "text": "Write a Python function to reverse a list" }, "taxonomy_id": "data-assistant" }
|
||||
|
||||
// ответ
|
||||
{
|
||||
"decision": {
|
||||
"task_label": "code",
|
||||
"confidence": 0.99,
|
||||
"complexity": { "tier": "low", "score": 0.27 }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Шлюз сопоставляет `complexity.tier` (`low`/`medium`/`high`) с таблицей `[routing]` и
|
||||
выбирает модель. На сложном/неоднозначном запросе роутер может эскалировать к oracle-LLM
|
||||
(несколько секунд) для более точного решения — подберите `timeout_ms` соответственно.
|
||||
См. [ROUTING.ru.md](ROUTING.ru.md).
|
||||
|
||||
## Проверьте точность сами
|
||||
|
||||
`bench/accuracy.py` прогоняет размеченный набор промптов через живой роутер — см.
|
||||
[BENCHMARKS.md](../BENCHMARKS.md). В нашем прогоне роутер дал **100% (37/37)** на
|
||||
естественных промптах против 32% у keyword-эвристики.
|
||||
+267
-5
@@ -8,8 +8,9 @@
|
||||
pub mod assets;
|
||||
|
||||
use crate::config::{
|
||||
AdminCfg, ApiKeyCfg, BreakerCfg, CacheCfg, Config, CortiqCfg, LogCfg, ModelCfg, ProtocolsCfg,
|
||||
RouteCfg, RouterCfg, RoutingCfg, RoutingPolicy, StatsCfg, TelemetryCfg, TierTargets,
|
||||
AdminCfg, ApiKeyCfg, BreakerCfg, CacheCfg, CmfCfg, Config, CortiqCfg, LogCfg, ModelCfg,
|
||||
ProtocolsCfg, RouteCfg, RouterCfg, RoutingCfg, RoutingPolicy, StatsCfg, TelemetryCfg,
|
||||
TierTargets,
|
||||
};
|
||||
use crate::model::{ChatRequest, GenParams, Message, RequestMeta, RoutingDirective};
|
||||
use crate::state::SharedState;
|
||||
@@ -142,6 +143,7 @@ fn parse_routing(model: &str) -> RoutingDirective {
|
||||
pub fn api_routes(admin_token: String) -> Router<SharedState> {
|
||||
Router::new()
|
||||
.route("/admin/api/health", get(health))
|
||||
.route("/admin/api/router/probe", post(router_probe))
|
||||
.route("/admin/api/meta", get(meta))
|
||||
.route("/admin/api/config", get(get_config).put(put_config))
|
||||
.route("/admin/api/models", get(list_models).post(create_model))
|
||||
@@ -163,6 +165,12 @@ pub fn api_routes(admin_token: String) -> Router<SharedState> {
|
||||
get(list_secrets).put(set_secret).delete(clear_secret),
|
||||
)
|
||||
.route("/admin/api/stats", get(get_stats))
|
||||
.route("/admin/api/shadow", get(get_shadow))
|
||||
.route("/admin/api/hf/search", get(hf_search))
|
||||
.route("/admin/api/import", get(list_imports).post(start_import))
|
||||
.route("/admin/api/import/:job", get(import_status).delete(delete_import))
|
||||
.route("/admin/api/import/:job/cancel", post(cancel_import))
|
||||
.route("/admin/api/import/:job/register", post(register_import))
|
||||
.route("/admin/api/requests", get(get_requests))
|
||||
.route("/admin/api/test", post(run_test))
|
||||
.route("/admin/api/test/stream", post(run_test_stream))
|
||||
@@ -225,12 +233,14 @@ async fn probe_router(url: &str) -> (bool, u64) {
|
||||
Err(_) => return (false, 0),
|
||||
};
|
||||
let started = Instant::now();
|
||||
let target = format!("{}/healthz", url.trim_end_matches('/'));
|
||||
// cortiq-router exposes health at /v1/healthz (auth-exempt); a bare /healthz
|
||||
// hits the auth middleware and 401s, which would read as a false "reachable".
|
||||
let target = format!("{}/v1/healthz", url.trim_end_matches('/'));
|
||||
let ok = client
|
||||
.get(&target)
|
||||
.send()
|
||||
.await
|
||||
.map(|r| r.status().as_u16() < 500)
|
||||
.map(|r| r.status().is_success())
|
||||
.unwrap_or(false);
|
||||
(ok, started.elapsed().as_millis() as u64)
|
||||
}
|
||||
@@ -238,6 +248,18 @@ async fn probe_router(url: &str) -> (bool, u64) {
|
||||
async fn health(State(state): State<SharedState>) -> ApiResult {
|
||||
let live = state.live();
|
||||
let (router_ok, router_ms) = probe_router(&live.cfg.router.url).await;
|
||||
let key_source = match &live.cfg.router.api_key_env {
|
||||
Some(env) => state.secrets.source(env),
|
||||
None => "none",
|
||||
};
|
||||
let last = state
|
||||
.router_status
|
||||
.load(std::sync::atomic::Ordering::Relaxed);
|
||||
let last_error = if last == 0 {
|
||||
Value::Null
|
||||
} else {
|
||||
json!({ "kind": crate::router_client::classify_status(last), "http": last })
|
||||
};
|
||||
let models: Vec<Value> = live
|
||||
.cfg
|
||||
.models
|
||||
@@ -262,11 +284,105 @@ async fn health(State(state): State<SharedState>) -> ApiResult {
|
||||
"ok": true,
|
||||
"version": env!("CARGO_PKG_VERSION"),
|
||||
"listen": live.cfg.listen,
|
||||
"router": { "url": live.cfg.router.url, "reachable": router_ok, "latency_ms": router_ms },
|
||||
"router": {
|
||||
"url": live.cfg.router.url,
|
||||
"reachable": router_ok,
|
||||
"latency_ms": router_ms,
|
||||
"key_env": live.cfg.router.api_key_env,
|
||||
"key_source": key_source,
|
||||
"last_error": last_error,
|
||||
},
|
||||
"models": models,
|
||||
}))
|
||||
}
|
||||
|
||||
/// `POST /admin/api/router/probe` — a real (authenticated) `/v1/route` call so the
|
||||
/// panel can tell "key missing / rejected / out of quota" apart from "router down".
|
||||
/// User-triggered only: it consumes one routing decision. Also fetches
|
||||
/// `GET /v1/usage` best-effort — not part of the documented contract yet; the
|
||||
/// panel shows balance/usage automatically once the router starts serving it.
|
||||
async fn router_probe(State(state): State<SharedState>) -> ApiResult {
|
||||
let live = state.live();
|
||||
let rcfg = live.cfg.router.clone();
|
||||
let key = rcfg
|
||||
.api_key_env
|
||||
.as_deref()
|
||||
.and_then(|n| state.secrets.resolve(n));
|
||||
if rcfg.api_key_env.is_some() && key.is_none() {
|
||||
return ok(json!({ "ok": false, "status": "no_key" }));
|
||||
}
|
||||
|
||||
let mut builder = reqwest::Client::builder()
|
||||
.timeout(std::time::Duration::from_millis(rcfg.timeout_ms.max(5000)));
|
||||
if !rcfg.verify_tls {
|
||||
builder = builder
|
||||
.danger_accept_invalid_certs(true)
|
||||
.danger_accept_invalid_hostnames(true);
|
||||
}
|
||||
let client = builder.build().map_err(|e| ApiError::bad(e.to_string()))?;
|
||||
let url = rcfg.url.trim_end_matches('/').to_string();
|
||||
|
||||
let mut body = json!({
|
||||
"input": { "text": "ping" },
|
||||
"options": { "policy_profile": "balanced", "allow_oracle": false }
|
||||
});
|
||||
if let Some(tax) = &rcfg.taxonomy_id {
|
||||
body["taxonomy_id"] = json!(tax);
|
||||
}
|
||||
let mut req = client.post(format!("{url}/v1/route")).json(&body);
|
||||
if let Some(k) = &key {
|
||||
req = req.bearer_auth(k);
|
||||
}
|
||||
|
||||
let started = Instant::now();
|
||||
let resp = match req.send().await {
|
||||
Ok(r) => r,
|
||||
Err(e) => {
|
||||
let status = if e.is_timeout() {
|
||||
"timeout"
|
||||
} else {
|
||||
"unreachable"
|
||||
};
|
||||
return ok(json!({
|
||||
"ok": false,
|
||||
"status": status,
|
||||
"latency_ms": started.elapsed().as_millis() as u64,
|
||||
}));
|
||||
}
|
||||
};
|
||||
let latency_ms = started.elapsed().as_millis() as u64;
|
||||
let code = resp.status().as_u16();
|
||||
if !resp.status().is_success() {
|
||||
let message = resp
|
||||
.json::<Value>()
|
||||
.await
|
||||
.ok()
|
||||
.and_then(|v| v["error"]["message"].as_str().map(|s| s.to_string()));
|
||||
return ok(json!({
|
||||
"ok": false,
|
||||
"status": crate::router_client::classify_status(code),
|
||||
"http": code,
|
||||
"latency_ms": latency_ms,
|
||||
"message": message,
|
||||
}));
|
||||
}
|
||||
|
||||
let mut usage = Value::Null;
|
||||
if let Some(k) = &key {
|
||||
if let Ok(r) = client
|
||||
.get(format!("{url}/v1/usage"))
|
||||
.bearer_auth(k)
|
||||
.send()
|
||||
.await
|
||||
{
|
||||
if r.status().is_success() {
|
||||
usage = r.json::<Value>().await.unwrap_or(Value::Null);
|
||||
}
|
||||
}
|
||||
}
|
||||
ok(json!({ "ok": true, "status": "ok", "latency_ms": latency_ms, "usage": usage }))
|
||||
}
|
||||
|
||||
async fn meta() -> ApiResult {
|
||||
ok(json!({
|
||||
"version": env!("CARGO_PKG_VERSION"),
|
||||
@@ -478,6 +594,7 @@ async fn get_settings(State(state): State<SharedState>) -> ApiResult {
|
||||
"stats": c.stats,
|
||||
"admin": c.admin,
|
||||
"cache": c.cache,
|
||||
"cmf": c.cmf,
|
||||
}))
|
||||
}
|
||||
|
||||
@@ -493,6 +610,7 @@ struct SettingsBody {
|
||||
stats: Option<StatsCfg>,
|
||||
admin: Option<AdminCfg>,
|
||||
cache: Option<CacheCfg>,
|
||||
cmf: Option<CmfCfg>,
|
||||
}
|
||||
|
||||
async fn put_settings(State(state): State<SharedState>, Json(b): Json<SettingsBody>) -> ApiResult {
|
||||
@@ -536,6 +654,19 @@ async fn put_settings(State(state): State<SharedState>, Json(b): Json<SettingsBo
|
||||
needs_restart = true;
|
||||
cfg.cache = v;
|
||||
}
|
||||
if let Some(v) = b.cmf {
|
||||
// manage_server / local_model / threads / gpu apply on restart (the server
|
||||
// is spawned once at startup with those env vars); local_only / router
|
||||
// routing apply immediately.
|
||||
if v.manage_server != cfg.cmf.manage_server
|
||||
|| v.local_model != cfg.cmf.local_model
|
||||
|| v.threads != cfg.cmf.threads
|
||||
|| v.gpu != cfg.cmf.gpu
|
||||
{
|
||||
needs_restart = true;
|
||||
}
|
||||
cfg.cmf = v;
|
||||
}
|
||||
state.reload(cfg)?;
|
||||
ok(json!({ "ok": true, "needs_restart": needs_restart }))
|
||||
}
|
||||
@@ -678,6 +809,137 @@ async fn get_stats(State(state): State<SharedState>, Query(q): Query<StatsQuery>
|
||||
ok(snap)
|
||||
}
|
||||
|
||||
/// Self-warming shadow loop: per-task-type promotion state, Wilson-LB of the
|
||||
/// judged pass-rate, and whether the local model is serving it yet.
|
||||
async fn get_shadow(State(state): State<SharedState>) -> ApiResult {
|
||||
let labels: Vec<Value> = state
|
||||
.promotion
|
||||
.snapshot()
|
||||
.into_iter()
|
||||
.map(|(label, st, n, pass_rate, lb)| {
|
||||
json!({
|
||||
"label": label,
|
||||
"state": format!("{st:?}"),
|
||||
"n": n,
|
||||
"pass_rate": pass_rate,
|
||||
"wilson_lb": lb,
|
||||
"serves_local": state.promotion.serves_local(&label),
|
||||
})
|
||||
})
|
||||
.collect();
|
||||
ok(json!({ "enabled": state.promotion.enabled(), "labels": labels }))
|
||||
}
|
||||
|
||||
// ── CMF model factory: HF search → convert → register ──
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct HfQuery {
|
||||
q: Option<String>,
|
||||
limit: Option<usize>,
|
||||
}
|
||||
|
||||
/// Proxy HuggingFace model search (server-side: avoids CORS, adds token).
|
||||
async fn hf_search(State(state): State<SharedState>, Query(q): Query<HfQuery>) -> ApiResult {
|
||||
let live = state.live();
|
||||
let token = if live.cfg.cmf.hf_token_env.is_empty() {
|
||||
None
|
||||
} else {
|
||||
state.secrets.resolve(&live.cfg.cmf.hf_token_env)
|
||||
};
|
||||
let query = q.q.unwrap_or_default();
|
||||
let limit = q.limit.unwrap_or(24).min(50);
|
||||
let models = crate::import::hf_search(&query, limit, token.as_deref())
|
||||
.await
|
||||
.map_err(ApiError::bad)?;
|
||||
ok(json!({ "models": models }))
|
||||
}
|
||||
|
||||
async fn list_imports(State(state): State<SharedState>) -> ApiResult {
|
||||
ok(json!({ "jobs": state.imports.list() }))
|
||||
}
|
||||
|
||||
/// Start a conversion job (returns the job id; progress via `import/:job`).
|
||||
async fn start_import(
|
||||
State(state): State<SharedState>,
|
||||
Json(p): Json<crate::import::ImportParams>,
|
||||
) -> ApiResult {
|
||||
let live = state.live();
|
||||
let id = crate::import::start_import(state.imports.clone(), &live.cfg.cmf, p)
|
||||
.map_err(ApiError::bad)?;
|
||||
ok(json!({ "job": id }))
|
||||
}
|
||||
|
||||
async fn import_status(
|
||||
State(state): State<SharedState>,
|
||||
Path(job): Path<String>,
|
||||
) -> ApiResult {
|
||||
match state.imports.get(&job) {
|
||||
Some(j) => ok(serde_json::to_value(j).unwrap_or_else(|_| json!({}))),
|
||||
None => Err(ApiError::not_found(format!("job '{job}' not found"))),
|
||||
}
|
||||
}
|
||||
|
||||
/// Cancel a running conversion: kills the converter and removes partial output.
|
||||
async fn cancel_import(
|
||||
State(state): State<SharedState>,
|
||||
Path(job): Path<String>,
|
||||
) -> ApiResult {
|
||||
if state.imports.get(&job).is_none() {
|
||||
return Err(ApiError::not_found(format!("job '{job}' not found")));
|
||||
}
|
||||
let cancelled = state.imports.cancel(&job);
|
||||
ok(json!({ "ok": cancelled }))
|
||||
}
|
||||
|
||||
/// Delete a finished conversion and its converted `.cmf` file(s) from disk.
|
||||
async fn delete_import(
|
||||
State(state): State<SharedState>,
|
||||
Path(job): Path<String>,
|
||||
) -> ApiResult {
|
||||
if state.imports.get(&job).is_none() {
|
||||
return Err(ApiError::not_found(format!("job '{job}' not found")));
|
||||
}
|
||||
let removed = state.imports.delete(&job).map_err(ApiError::bad)?;
|
||||
ok(json!({ "ok": removed }))
|
||||
}
|
||||
|
||||
/// Register a finished `.cmf` as a local model (OpenAI provider → cortiq-server).
|
||||
async fn register_import(
|
||||
State(state): State<SharedState>,
|
||||
Path(job): Path<String>,
|
||||
) -> ApiResult {
|
||||
let j = state
|
||||
.imports
|
||||
.get(&job)
|
||||
.ok_or_else(|| ApiError::not_found(format!("job '{job}' not found")))?;
|
||||
if j.state != "done" {
|
||||
return Err(ApiError::bad("conversion not finished"));
|
||||
}
|
||||
let base = std::path::Path::new(&j.output)
|
||||
.file_stem()
|
||||
.map(|s| s.to_string_lossy().to_string())
|
||||
.unwrap_or_else(|| "cmf-local".into());
|
||||
let id = format!("cmf-{base}");
|
||||
let mut cfg = current_cfg(&state);
|
||||
if cfg.models.iter().any(|m| m.id == id) {
|
||||
return Err(ApiError::bad(format!("model '{id}' already registered")));
|
||||
}
|
||||
cfg.models.push(crate::config::ModelCfg {
|
||||
id: id.clone(),
|
||||
provider: "openai".into(),
|
||||
base_url: cfg.cmf.cortiq_server_url.clone(),
|
||||
model: base,
|
||||
cost_tier: "local".into(),
|
||||
price_in: 0.0,
|
||||
price_out: 0.0,
|
||||
kind: "chat".into(),
|
||||
api_key_env: None,
|
||||
caps: Vec::new(),
|
||||
});
|
||||
state.reload(cfg)?;
|
||||
ok(json!({ "ok": true, "model_id": id }))
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct RecentQuery {
|
||||
#[serde(default)]
|
||||
|
||||
@@ -0,0 +1,279 @@
|
||||
//! Local CMF model server lifecycle.
|
||||
//!
|
||||
//! Closes the loop with <https://github.com/infosave2007/cmf>: install the
|
||||
//! `cortiq` CLI from crates.io, keep it up to date, and run `cortiq serve` as a
|
||||
//! managed child process. The served model exposes an OpenAI-compatible API, so
|
||||
//! the registry registers it as an ordinary provider and the gateway routes to
|
||||
//! it like any other backend.
|
||||
|
||||
use crate::config::CmfCfg;
|
||||
use std::process::Stdio;
|
||||
use std::sync::{Arc, Mutex};
|
||||
use std::time::Duration;
|
||||
use tokio::process::{Child, Command};
|
||||
|
||||
/// Observable state of the managed local CMF server (surfaced to the admin API).
|
||||
#[derive(Clone, Debug, Default, serde::Serialize)]
|
||||
pub struct CmfStatus {
|
||||
/// `manage_server` is on and a `local_model` is configured.
|
||||
pub enabled: bool,
|
||||
/// Version reported by `cortiq --version` (None = not installed).
|
||||
pub installed_version: Option<String>,
|
||||
/// Newest `cortiq-cli` on crates.io (populated when `auto_update` runs).
|
||||
pub latest_version: Option<String>,
|
||||
/// The child `cortiq serve` process has been spawned.
|
||||
pub running: bool,
|
||||
/// `/healthz` on the local server answered 200.
|
||||
pub healthy: bool,
|
||||
/// OpenAI base URL the local server is registered under.
|
||||
pub base_url: String,
|
||||
/// Path of the `.cmf` being served.
|
||||
pub model: String,
|
||||
/// Last fatal error, if any.
|
||||
pub last_error: Option<String>,
|
||||
/// Recent lifecycle log lines (bounded).
|
||||
pub log: Vec<String>,
|
||||
}
|
||||
|
||||
/// Holds the managed child process and its live status.
|
||||
pub struct CmfRuntime {
|
||||
status: Mutex<CmfStatus>,
|
||||
child: Mutex<Option<Child>>,
|
||||
}
|
||||
|
||||
impl CmfRuntime {
|
||||
pub fn new() -> Arc<Self> {
|
||||
Arc::new(Self {
|
||||
status: Mutex::new(CmfStatus::default()),
|
||||
child: Mutex::new(None),
|
||||
})
|
||||
}
|
||||
|
||||
pub fn status(&self) -> CmfStatus {
|
||||
self.status.lock().unwrap().clone()
|
||||
}
|
||||
|
||||
fn log(&self, msg: impl Into<String>) {
|
||||
let msg = msg.into();
|
||||
tracing::info!(target: "cmf", "{msg}");
|
||||
let mut s = self.status.lock().unwrap();
|
||||
s.log.push(msg);
|
||||
let n = s.log.len();
|
||||
if n > 50 {
|
||||
s.log.drain(0..n - 50);
|
||||
}
|
||||
}
|
||||
|
||||
fn fail(&self, msg: impl Into<String>) {
|
||||
let msg = msg.into();
|
||||
tracing::warn!(target: "cmf", "{msg}");
|
||||
let mut s = self.status.lock().unwrap();
|
||||
s.last_error = Some(msg.clone());
|
||||
s.log.push(format!("✗ {msg}"));
|
||||
}
|
||||
|
||||
fn set_child(&self, child: Child) {
|
||||
*self.child.lock().unwrap() = Some(child);
|
||||
self.status.lock().unwrap().running = true;
|
||||
}
|
||||
|
||||
/// Stop the managed server (used on shutdown / reconfigure).
|
||||
pub async fn stop(&self) {
|
||||
if let Some(mut child) = self.child.lock().unwrap().take() {
|
||||
let _ = child.start_kill();
|
||||
}
|
||||
let mut s = self.status.lock().unwrap();
|
||||
s.running = false;
|
||||
s.healthy = false;
|
||||
}
|
||||
}
|
||||
|
||||
impl Default for CmfRuntime {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
status: Mutex::new(CmfStatus::default()),
|
||||
child: Mutex::new(None),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// crates.io `newest_version` for `cortiq-cli`.
|
||||
async fn latest_crates_version() -> Option<String> {
|
||||
let client = reqwest::Client::builder()
|
||||
.timeout(Duration::from_secs(10))
|
||||
.build()
|
||||
.ok()?;
|
||||
let body = client
|
||||
.get("https://crates.io/api/v1/crates/cortiq-cli")
|
||||
.header("User-Agent", "cortiq-gateway")
|
||||
.send()
|
||||
.await
|
||||
.ok()?
|
||||
.text()
|
||||
.await
|
||||
.ok()?;
|
||||
let v: serde_json::Value = serde_json::from_str(&body).ok()?;
|
||||
v["crate"]["newest_version"]
|
||||
.as_str()
|
||||
.map(|s| s.to_string())
|
||||
}
|
||||
|
||||
/// `<bin> --version` → the trailing version token (e.g. "cortiq 0.1.2" → "0.1.2").
|
||||
fn installed_version(bin: &str) -> Option<String> {
|
||||
let out = std::process::Command::new(bin).arg("--version").output().ok()?;
|
||||
if !out.status.success() {
|
||||
return None;
|
||||
}
|
||||
let s = String::from_utf8_lossy(&out.stdout);
|
||||
s.split_whitespace().last().map(|s| s.trim().to_string())
|
||||
}
|
||||
|
||||
fn parse_ver(s: &str) -> (u32, u32, u32) {
|
||||
let mut it = s.split('.').map(|x| x.trim().parse::<u32>().unwrap_or(0));
|
||||
(it.next().unwrap_or(0), it.next().unwrap_or(0), it.next().unwrap_or(0))
|
||||
}
|
||||
|
||||
fn version_lt(a: &str, b: &str) -> bool {
|
||||
parse_ver(a) < parse_ver(b)
|
||||
}
|
||||
|
||||
/// `cargo install cortiq-cli [--force]` — blocking compile, run off-thread.
|
||||
async fn cargo_install(force: bool) -> Result<(), String> {
|
||||
let mut cmd = Command::new("cargo");
|
||||
cmd.args(["install", "cortiq-cli", "--locked"]);
|
||||
if force {
|
||||
cmd.arg("--force");
|
||||
}
|
||||
let out = cmd.output().await.map_err(|e| e.to_string())?;
|
||||
if out.status.success() {
|
||||
Ok(())
|
||||
} else {
|
||||
Err(String::from_utf8_lossy(&out.stderr).trim().to_string())
|
||||
}
|
||||
}
|
||||
|
||||
/// Spawn `cortiq serve <model> --host <host> --port <port>`; the child is killed
|
||||
/// when its handle is dropped (i.e. when the gateway shuts down).
|
||||
fn spawn_serve(cfg: &CmfCfg) -> Result<Child, String> {
|
||||
let mut cmd = Command::new(&cfg.cortiq_bin);
|
||||
cmd.arg("serve")
|
||||
.arg(&cfg.local_model)
|
||||
.arg("--host")
|
||||
.arg(&cfg.local_host)
|
||||
.arg("--port")
|
||||
.arg(cfg.local_port.to_string());
|
||||
// Parallelise decode matvecs across cores (CMF_THREADS). 0 = leave the
|
||||
// runtime's own default; N>1 gave ~2.8x on a 15B CPU model.
|
||||
if cfg.threads > 1 {
|
||||
cmd.env("CMF_THREADS", cfg.threads.to_string());
|
||||
}
|
||||
// Offload large matvecs to the GPU (Metal on macOS). Memory-bandwidth-bound
|
||||
// on unified memory, so mostly a prefill win; opt-in.
|
||||
if cfg.gpu {
|
||||
cmd.env("CMF_GPU", "1");
|
||||
}
|
||||
cmd.stdout(Stdio::null())
|
||||
.stderr(Stdio::null())
|
||||
.kill_on_drop(true)
|
||||
.spawn()
|
||||
.map_err(|e| e.to_string())
|
||||
}
|
||||
|
||||
/// Poll `/healthz` until the local server answers 200 (or the deadline passes).
|
||||
async fn wait_healthy(host: &str, port: u16, tries: u32) -> bool {
|
||||
let url = format!("http://{host}:{port}/healthz");
|
||||
let client = reqwest::Client::new();
|
||||
for _ in 0..tries {
|
||||
if let Ok(r) = client
|
||||
.get(&url)
|
||||
.timeout(Duration::from_secs(2))
|
||||
.send()
|
||||
.await
|
||||
{
|
||||
if r.status().is_success() {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
tokio::time::sleep(Duration::from_millis(500)).await;
|
||||
}
|
||||
false
|
||||
}
|
||||
|
||||
/// Orchestrate the whole lifecycle: ensure installed → maybe update → spawn →
|
||||
/// wait healthy. Safe to call in a background task; it never panics.
|
||||
pub async fn manage(rt: Arc<CmfRuntime>, cfg: CmfCfg) {
|
||||
if !cfg.manage_server || cfg.local_model.trim().is_empty() {
|
||||
return;
|
||||
}
|
||||
{
|
||||
let mut s = rt.status.lock().unwrap();
|
||||
s.enabled = true;
|
||||
s.base_url = format!("http://{}:{}/v1", cfg.local_host, cfg.local_port);
|
||||
s.model = cfg.local_model.clone();
|
||||
}
|
||||
if !std::path::Path::new(&cfg.local_model).exists() {
|
||||
rt.fail(format!("local_model not found: {}", cfg.local_model));
|
||||
return;
|
||||
}
|
||||
|
||||
// 1. Ensure the `cortiq` binary is available.
|
||||
let mut ver = installed_version(&cfg.cortiq_bin);
|
||||
if ver.is_none() {
|
||||
if cfg.auto_install {
|
||||
rt.log("cortiq not found — installing cortiq-cli from crates.io (this compiles, may take a few minutes)…");
|
||||
if let Err(e) = cargo_install(false).await {
|
||||
rt.fail(format!("cargo install cortiq-cli failed: {e}"));
|
||||
return;
|
||||
}
|
||||
ver = installed_version(&cfg.cortiq_bin);
|
||||
if ver.is_some() {
|
||||
rt.log("installed cortiq-cli");
|
||||
}
|
||||
} else {
|
||||
rt.fail("cortiq binary not found (set [cmf].auto_install = true, or install cortiq-cli)");
|
||||
return;
|
||||
}
|
||||
}
|
||||
rt.status.lock().unwrap().installed_version = ver.clone();
|
||||
|
||||
// 2. Optionally update to the newest crates.io release.
|
||||
if cfg.auto_update {
|
||||
if let Some(latest) = latest_crates_version().await {
|
||||
rt.status.lock().unwrap().latest_version = Some(latest.clone());
|
||||
if ver.as_deref().map(|c| version_lt(c, &latest)).unwrap_or(false) {
|
||||
rt.log(format!("updating cortiq-cli {} → {latest}…", ver.as_deref().unwrap_or("?")));
|
||||
if let Err(e) = cargo_install(true).await {
|
||||
rt.log(format!("update failed (keeping current): {e}"));
|
||||
} else {
|
||||
ver = installed_version(&cfg.cortiq_bin);
|
||||
rt.status.lock().unwrap().installed_version = ver.clone();
|
||||
rt.log(format!("updated to {}", ver.as_deref().unwrap_or("?")));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 3. Spawn the local server.
|
||||
rt.log(format!(
|
||||
"starting: cortiq serve {} --host {} --port {}",
|
||||
cfg.local_model, cfg.local_host, cfg.local_port
|
||||
));
|
||||
match spawn_serve(&cfg) {
|
||||
Ok(child) => rt.set_child(child),
|
||||
Err(e) => {
|
||||
rt.fail(format!("failed to spawn cortiq serve: {e}"));
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
// 4. Wait until it is serving (model load can take a while for big models).
|
||||
if wait_healthy(&cfg.local_host, cfg.local_port, 240).await {
|
||||
rt.status.lock().unwrap().healthy = true;
|
||||
rt.log(format!(
|
||||
"local CMF server ready — registered as model '{}' at http://{}:{}/v1",
|
||||
cfg.model_id, cfg.local_host, cfg.local_port
|
||||
));
|
||||
} else {
|
||||
rt.fail("local CMF server did not become healthy in time");
|
||||
}
|
||||
}
|
||||
+232
-5
@@ -3,6 +3,26 @@
|
||||
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::collections::HashMap;
|
||||
use std::path::PathBuf;
|
||||
|
||||
/// Base directory for gateway-managed data (models, logs). Honors
|
||||
/// `$CORTIQ_GATEWAY_HOME`, then `$XDG_DATA_HOME/cortiq-gateway`, then
|
||||
/// `~/.cortiq-gateway` — so defaults work regardless of the process CWD
|
||||
/// (important once the gateway is installed rather than run from its repo).
|
||||
pub fn data_dir() -> PathBuf {
|
||||
for (var, sub) in [("CORTIQ_GATEWAY_HOME", ""), ("XDG_DATA_HOME", "cortiq-gateway"), ("HOME", ".cortiq-gateway")] {
|
||||
if let Ok(v) = std::env::var(var) {
|
||||
if !v.is_empty() {
|
||||
return if sub.is_empty() { PathBuf::from(v) } else { PathBuf::from(v).join(sub) };
|
||||
}
|
||||
}
|
||||
}
|
||||
PathBuf::from("data")
|
||||
}
|
||||
|
||||
fn data_path(name: &str) -> String {
|
||||
data_dir().join(name).to_string_lossy().into_owned()
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Serialize, Deserialize)]
|
||||
pub struct Config {
|
||||
@@ -33,10 +53,131 @@ pub struct Config {
|
||||
pub stats: StatsCfg,
|
||||
#[serde(default)]
|
||||
pub cache: CacheCfg,
|
||||
#[serde(default)]
|
||||
pub shadow: ShadowCfg,
|
||||
#[serde(default)]
|
||||
pub cmf: CmfCfg,
|
||||
}
|
||||
|
||||
/// CMF-format model factory: import a HuggingFace model, convert it to a
|
||||
/// local `.cmf` (quantized), and serve it via cortiq-server. Points the
|
||||
/// gateway at the Python converter + runtime on the host.
|
||||
#[derive(Clone, Debug, Serialize, Deserialize)]
|
||||
pub struct CmfCfg {
|
||||
/// Python interpreter that has torch + the converter deps.
|
||||
#[serde(default = "default_cmf_python")]
|
||||
pub python_bin: String,
|
||||
/// Path to `convert_dtgma_to_cmf.py`.
|
||||
#[serde(default = "default_cmf_converter")]
|
||||
pub converter: String,
|
||||
/// Directory where produced `.cmf` files are written.
|
||||
#[serde(default = "default_cmf_models_dir")]
|
||||
pub models_dir: String,
|
||||
/// Base URL of the cortiq-server that serves the `.cmf` (OpenAI /v1),
|
||||
/// used when registering a converted model as a provider.
|
||||
#[serde(default = "default_cmf_server_url")]
|
||||
pub cortiq_server_url: String,
|
||||
/// Secret name of a HuggingFace token (higher search rate limits +
|
||||
/// gated/private repos). Empty = anonymous public search.
|
||||
#[serde(default)]
|
||||
pub hf_token_env: String,
|
||||
/// `cortiq` CLI binary for the local server (from `cargo install cortiq-cli`).
|
||||
#[serde(default = "default_cortiq_bin")]
|
||||
pub cortiq_bin: String,
|
||||
/// Install cortiq-cli from crates.io automatically if the binary is missing.
|
||||
#[serde(default)]
|
||||
pub auto_install: bool,
|
||||
/// On startup, check crates.io and reinstall if a newer cortiq-cli exists.
|
||||
#[serde(default)]
|
||||
pub auto_update: bool,
|
||||
/// Spawn and manage a local `cortiq serve` for `local_model`, and register
|
||||
/// it in the model pool — a local `.cmf` backend the gateway can use even
|
||||
/// when no external provider (or the router) is available.
|
||||
#[serde(default)]
|
||||
pub manage_server: bool,
|
||||
/// Path to the local `.cmf` to serve (empty = disabled).
|
||||
#[serde(default)]
|
||||
pub local_model: String,
|
||||
/// Host the managed local server binds to.
|
||||
#[serde(default = "default_cmf_local_host")]
|
||||
pub local_host: String,
|
||||
/// Port the managed local server binds to.
|
||||
#[serde(default = "default_cmf_local_port")]
|
||||
pub local_port: u16,
|
||||
/// Model id the local server is registered under in the pool.
|
||||
#[serde(default = "default_cmf_model_id")]
|
||||
pub model_id: String,
|
||||
/// Use ONLY local models: route every request to the local model, ignoring
|
||||
/// the router and any external providers ("use only local models").
|
||||
#[serde(default)]
|
||||
pub local_only: bool,
|
||||
/// Worker threads for the managed `cortiq serve` (exported as `CMF_THREADS`).
|
||||
/// 0 = let the runtime decide; N>1 parallelises decode matvecs across N cores
|
||||
/// (measured ~2.8x faster than serial on a 15B model). Default 8.
|
||||
#[serde(default = "default_cmf_threads")]
|
||||
pub threads: usize,
|
||||
/// Offload large matvecs to the GPU (exports CMF_GPU=1). On Apple Silicon the
|
||||
/// decode loop is memory-bandwidth bound, so this is ~neutral for generation
|
||||
/// (helps prefill of long prompts). Off by default.
|
||||
#[serde(default)]
|
||||
pub gpu: bool,
|
||||
}
|
||||
impl Default for CmfCfg {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
python_bin: default_cmf_python(),
|
||||
converter: default_cmf_converter(),
|
||||
models_dir: default_cmf_models_dir(),
|
||||
cortiq_server_url: default_cmf_server_url(),
|
||||
hf_token_env: String::new(),
|
||||
cortiq_bin: default_cortiq_bin(),
|
||||
auto_install: false,
|
||||
auto_update: false,
|
||||
manage_server: false,
|
||||
local_model: String::new(),
|
||||
local_host: default_cmf_local_host(),
|
||||
local_port: default_cmf_local_port(),
|
||||
model_id: default_cmf_model_id(),
|
||||
local_only: false,
|
||||
threads: default_cmf_threads(),
|
||||
gpu: false,
|
||||
}
|
||||
}
|
||||
}
|
||||
fn default_cmf_threads() -> usize {
|
||||
8
|
||||
}
|
||||
fn default_cortiq_bin() -> String {
|
||||
"cortiq".into()
|
||||
}
|
||||
fn default_cmf_local_host() -> String {
|
||||
"127.0.0.1".into()
|
||||
}
|
||||
fn default_cmf_local_port() -> u16 {
|
||||
8081
|
||||
}
|
||||
fn default_cmf_model_id() -> String {
|
||||
"cmf-local".into()
|
||||
}
|
||||
fn default_cmf_python() -> String {
|
||||
"python3".into()
|
||||
}
|
||||
fn default_cmf_converter() -> String {
|
||||
"../cmf/converter/convert_dtgma_to_cmf.py".into()
|
||||
}
|
||||
fn default_cmf_models_dir() -> String {
|
||||
data_path("models")
|
||||
}
|
||||
fn default_cmf_server_url() -> String {
|
||||
"http://127.0.0.1:8081/v1".into()
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Serialize, Deserialize)]
|
||||
pub struct RouterCfg {
|
||||
/// Master switch — when false the router is never called; the gateway
|
||||
/// serves from the local model pool (local `.cmf` / configured default).
|
||||
#[serde(default = "default_router_enabled")]
|
||||
pub enabled: bool,
|
||||
pub url: String,
|
||||
#[serde(default = "default_router_timeout")]
|
||||
pub timeout_ms: u64,
|
||||
@@ -47,6 +188,9 @@ pub struct RouterCfg {
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub taxonomy_id: Option<String>,
|
||||
}
|
||||
fn default_router_enabled() -> bool {
|
||||
true
|
||||
}
|
||||
fn default_router_timeout() -> u64 {
|
||||
800
|
||||
}
|
||||
@@ -313,7 +457,7 @@ fn default_stats_enabled() -> bool {
|
||||
true
|
||||
}
|
||||
fn default_stats_file() -> String {
|
||||
"config/stats.jsonl".into()
|
||||
data_path("stats.jsonl")
|
||||
}
|
||||
fn default_stats_ring() -> usize {
|
||||
500
|
||||
@@ -322,6 +466,82 @@ fn default_stats_retention() -> String {
|
||||
"7d".into()
|
||||
}
|
||||
|
||||
/// Self-warming shadow loop (docs/SELF_WARMING_GATEWAY.md). While a task
|
||||
/// label is not yet promoted, the client is served the CLOUD answer; in
|
||||
/// parallel (sampled, non-blocking) the local `cmf-local` model answers and
|
||||
/// a cheap `judge` model scores it against the cloud answer. The per-label
|
||||
/// Wilson-LB of that pass stream gates promotion to local serving.
|
||||
#[derive(Clone, Debug, Serialize, Deserialize)]
|
||||
pub struct ShadowCfg {
|
||||
/// Master switch. Off = zero overhead, classic gateway behavior.
|
||||
#[serde(default)]
|
||||
pub enabled: bool,
|
||||
/// model_id (from `[[models]]`) of the local CMF model (OpenAI-compatible
|
||||
/// cortiq-server). Empty disables the loop.
|
||||
#[serde(default)]
|
||||
pub local_model_id: String,
|
||||
/// model_id of the cheap judge (different family than the answerer).
|
||||
#[serde(default)]
|
||||
pub judge_model_id: String,
|
||||
/// Fraction of eligible requests to shadow+judge (0..1). Bounds cost.
|
||||
#[serde(default = "default_shadow_sample")]
|
||||
pub sample_rate: f64,
|
||||
/// judge.jsonl append+replay path. Empty = in-memory only.
|
||||
#[serde(default = "default_shadow_file")]
|
||||
pub file: String,
|
||||
/// Wilson-95 lower bound of pass-rate required to promote.
|
||||
#[serde(default = "default_shadow_lb")]
|
||||
pub promote_lb: f64,
|
||||
/// Min judgments before promotion is considered.
|
||||
#[serde(default = "default_shadow_nmin")]
|
||||
pub n_min: usize,
|
||||
/// Rolling window of judgments per label.
|
||||
#[serde(default = "default_shadow_window")]
|
||||
pub window: usize,
|
||||
/// Extra canary judgments before full local serving.
|
||||
#[serde(default = "default_shadow_soak")]
|
||||
pub soak: usize,
|
||||
/// GATE past shadow-only: actually SERVE promoted labels from the local
|
||||
/// model (with failover to cloud on error + a complexity veto). Default
|
||||
/// OFF — the memo's 3 risks (§8) should be retired by measurement first.
|
||||
#[serde(default)]
|
||||
pub serve_when_promoted: bool,
|
||||
}
|
||||
impl Default for ShadowCfg {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
enabled: false,
|
||||
local_model_id: String::new(),
|
||||
judge_model_id: String::new(),
|
||||
sample_rate: default_shadow_sample(),
|
||||
file: default_shadow_file(),
|
||||
promote_lb: default_shadow_lb(),
|
||||
n_min: default_shadow_nmin(),
|
||||
window: default_shadow_window(),
|
||||
soak: default_shadow_soak(),
|
||||
serve_when_promoted: false,
|
||||
}
|
||||
}
|
||||
}
|
||||
fn default_shadow_sample() -> f64 {
|
||||
0.15
|
||||
}
|
||||
fn default_shadow_file() -> String {
|
||||
data_path("judge.jsonl")
|
||||
}
|
||||
fn default_shadow_lb() -> f64 {
|
||||
0.95
|
||||
}
|
||||
fn default_shadow_nmin() -> usize {
|
||||
200
|
||||
}
|
||||
fn default_shadow_window() -> usize {
|
||||
500
|
||||
}
|
||||
fn default_shadow_soak() -> usize {
|
||||
100
|
||||
}
|
||||
|
||||
/// Semantic (embedding-based) response cache — returns a cached answer for prompts
|
||||
/// that are semantically near a previous one, skipping the (expensive) model call.
|
||||
#[derive(Clone, Debug, Serialize, Deserialize)]
|
||||
@@ -374,8 +594,12 @@ impl Config {
|
||||
|
||||
/// Integrity validation: non-empty model pool, valid routing targets.
|
||||
pub fn validate(&self) -> anyhow::Result<()> {
|
||||
if self.models.is_empty() {
|
||||
anyhow::bail!("config: at least one [[models]] entry is required");
|
||||
// A managed local CMF model counts as an available model even though it
|
||||
// is registered dynamically rather than via a [[models]] entry — this
|
||||
// is what lets the gateway run on local models alone (no cloud, no router).
|
||||
let has_local = self.cmf.manage_server && !self.cmf.local_model.trim().is_empty();
|
||||
if self.models.is_empty() && !has_local {
|
||||
anyhow::bail!("config: at least one [[models]] entry (or a managed local CMF model) is required");
|
||||
}
|
||||
// uniqueness of model ids
|
||||
let mut seen = std::collections::HashSet::new();
|
||||
@@ -385,9 +609,12 @@ impl Config {
|
||||
}
|
||||
}
|
||||
// every target in [routing] must exist in the model pool
|
||||
let ids: std::collections::HashSet<&str> =
|
||||
let mut ids: std::collections::HashSet<&str> =
|
||||
self.models.iter().map(|m| m.id.as_str()).collect();
|
||||
if !ids.contains(self.routing.default.as_str()) {
|
||||
if has_local {
|
||||
ids.insert(self.cmf.model_id.as_str());
|
||||
}
|
||||
if !self.routing.default.is_empty() && !ids.contains(self.routing.default.as_str()) {
|
||||
anyhow::bail!(
|
||||
"config: routing.default '{}' is not a known model id",
|
||||
self.routing.default
|
||||
|
||||
+390
@@ -0,0 +1,390 @@
|
||||
//! CMF model factory: search HuggingFace, then convert a chosen repo to a
|
||||
//! local quantized `.cmf` via the Python converter — as a tracked, streamed
|
||||
//! background job so the admin UI can show live progress.
|
||||
|
||||
use crate::config::CmfCfg;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::collections::HashMap;
|
||||
use std::process::Stdio;
|
||||
use std::sync::{Arc, Mutex};
|
||||
use std::time::{SystemTime, UNIX_EPOCH};
|
||||
use tokio::io::{AsyncBufReadExt, BufReader};
|
||||
use tokio::sync::Notify;
|
||||
|
||||
/// Parse a `@PROGRESS <0..1> <phase>` marker emitted by the converter.
|
||||
fn parse_progress(line: &str) -> Option<(f32, String)> {
|
||||
let rest = line.strip_prefix("@PROGRESS ")?;
|
||||
let mut it = rest.splitn(2, ' ');
|
||||
let frac: f32 = it.next()?.trim().parse().ok()?;
|
||||
let phase = it.next().unwrap_or("").trim().to_string();
|
||||
Some((frac.clamp(0.0, 1.0), phase))
|
||||
}
|
||||
|
||||
/// Remove a partial/aborted output — `.cmf` may be a file or a tensor-dir.
|
||||
fn cleanup_output(path: &str) {
|
||||
let p = std::path::Path::new(path);
|
||||
if p.is_dir() {
|
||||
let _ = std::fs::remove_dir_all(p);
|
||||
} else if p.exists() {
|
||||
let _ = std::fs::remove_file(p);
|
||||
}
|
||||
}
|
||||
|
||||
fn now() -> u64 {
|
||||
SystemTime::now().duration_since(UNIX_EPOCH).unwrap_or_default().as_secs()
|
||||
}
|
||||
|
||||
/// Job id from a hash of (repo, nanos) — unique without a uuid dependency.
|
||||
fn gen_id(repo: &str) -> String {
|
||||
use std::hash::{Hash, Hasher};
|
||||
let nanos = SystemTime::now().duration_since(UNIX_EPOCH).unwrap_or_default().as_nanos();
|
||||
let mut h = std::collections::hash_map::DefaultHasher::new();
|
||||
repo.hash(&mut h);
|
||||
nanos.hash(&mut h);
|
||||
format!("{:016x}", h.finish())
|
||||
}
|
||||
|
||||
/// Parameters chosen in the import wizard.
|
||||
#[derive(Clone, Debug, Deserialize)]
|
||||
pub struct ImportParams {
|
||||
pub repo: String,
|
||||
#[serde(default = "default_quant")]
|
||||
pub quant: String,
|
||||
#[serde(default)]
|
||||
pub name: String, // output basename; empty → derived from repo
|
||||
// ── advanced ──
|
||||
#[serde(default)]
|
||||
pub linear_core: Option<String>, // gated_delta_net | vmf_phase
|
||||
#[serde(default)]
|
||||
pub nphase: Option<u32>,
|
||||
#[serde(default)]
|
||||
pub vbit_shape: Option<String>, // log2 | cubic
|
||||
#[serde(default)]
|
||||
pub mean_bits: Option<f32>,
|
||||
#[serde(default)]
|
||||
pub shard_max_gb: Option<f32>,
|
||||
#[serde(default)]
|
||||
pub skip_mtp: bool,
|
||||
}
|
||||
fn default_quant() -> String {
|
||||
"Q8_2F".into()
|
||||
}
|
||||
|
||||
/// Live state of one conversion job.
|
||||
#[derive(Clone, Debug, Serialize)]
|
||||
pub struct Job {
|
||||
pub id: String,
|
||||
pub repo: String,
|
||||
pub quant: String,
|
||||
pub output: String,
|
||||
pub state: String, // running | done | error | cancelled
|
||||
pub log: Vec<String>,
|
||||
pub started: u64,
|
||||
pub finished: Option<u64>,
|
||||
pub size_bytes: Option<u64>,
|
||||
pub progress: Option<f32>, // 0..1, parsed from converter @PROGRESS
|
||||
pub phase: Option<String>, // current phase label
|
||||
}
|
||||
|
||||
#[derive(Default)]
|
||||
pub struct JobStore {
|
||||
jobs: Mutex<HashMap<String, Job>>,
|
||||
/// Cancel handles, keyed by job id (not serialized). Notifying aborts the
|
||||
/// converter process and cleans up its partial output.
|
||||
cancels: Mutex<HashMap<String, Arc<Notify>>>,
|
||||
}
|
||||
|
||||
impl JobStore {
|
||||
pub fn new() -> Arc<Self> {
|
||||
Arc::new(Self::default())
|
||||
}
|
||||
fn insert(&self, job: Job) {
|
||||
self.jobs.lock().unwrap().insert(job.id.clone(), job);
|
||||
}
|
||||
fn set_cancel(&self, id: &str, n: Arc<Notify>) {
|
||||
self.cancels.lock().unwrap().insert(id.to_string(), n);
|
||||
}
|
||||
fn drop_cancel(&self, id: &str) {
|
||||
self.cancels.lock().unwrap().remove(id);
|
||||
}
|
||||
/// Signal a running job to abort. Returns false if the job isn't cancellable.
|
||||
pub fn cancel(&self, id: &str) -> bool {
|
||||
let running = matches!(
|
||||
self.jobs.lock().unwrap().get(id).map(|j| j.state.as_str()),
|
||||
Some("running")
|
||||
);
|
||||
if !running {
|
||||
return false;
|
||||
}
|
||||
if let Some(n) = self.cancels.lock().unwrap().get(id) {
|
||||
n.notify_one();
|
||||
true
|
||||
} else {
|
||||
false
|
||||
}
|
||||
}
|
||||
fn push_line(&self, id: &str, line: String) {
|
||||
if let Some(j) = self.jobs.lock().unwrap().get_mut(id) {
|
||||
j.log.push(line);
|
||||
let n = j.log.len();
|
||||
if n > 200 {
|
||||
j.log.drain(0..n - 200); // keep the tail
|
||||
}
|
||||
}
|
||||
}
|
||||
fn set_progress(&self, id: &str, frac: f32, phase: String) {
|
||||
if let Some(j) = self.jobs.lock().unwrap().get_mut(id) {
|
||||
j.progress = Some(frac);
|
||||
if !phase.is_empty() {
|
||||
j.phase = Some(phase);
|
||||
}
|
||||
}
|
||||
}
|
||||
fn set_state(&self, id: &str, state: &str) {
|
||||
if let Some(j) = self.jobs.lock().unwrap().get_mut(id) {
|
||||
j.state = state.into();
|
||||
j.finished = Some(now());
|
||||
}
|
||||
}
|
||||
fn finish(&self, id: &str, ok: bool) {
|
||||
if let Some(j) = self.jobs.lock().unwrap().get_mut(id) {
|
||||
if j.state == "cancelled" {
|
||||
return; // a concurrent cancel already settled this job
|
||||
}
|
||||
j.finished = Some(now());
|
||||
j.size_bytes = std::fs::metadata(&j.output).ok().map(|m| m.len());
|
||||
let done = ok && j.size_bytes.unwrap_or(0) > 0;
|
||||
j.state = if done { "done" } else { "error" }.into();
|
||||
if done {
|
||||
j.progress = Some(1.0);
|
||||
}
|
||||
}
|
||||
}
|
||||
pub fn get(&self, id: &str) -> Option<Job> {
|
||||
self.jobs.lock().unwrap().get(id).cloned()
|
||||
}
|
||||
/// Delete a finished job together with its converted output file(s).
|
||||
/// Refuses a running job (cancel it first). `Ok(false)` if unknown.
|
||||
pub fn delete(&self, id: &str) -> Result<bool, String> {
|
||||
let job = self.jobs.lock().unwrap().get(id).cloned();
|
||||
let Some(job) = job else { return Ok(false) };
|
||||
if job.state == "running" {
|
||||
return Err("cancel the running job before deleting".into());
|
||||
}
|
||||
cleanup_output(&job.output);
|
||||
self.cancels.lock().unwrap().remove(id);
|
||||
self.jobs.lock().unwrap().remove(id);
|
||||
Ok(true)
|
||||
}
|
||||
pub fn list(&self) -> Vec<Job> {
|
||||
let mut v: Vec<Job> = self.jobs.lock().unwrap().values().cloned().collect();
|
||||
v.sort_by(|a, b| b.started.cmp(&a.started));
|
||||
v.truncate(20);
|
||||
v
|
||||
}
|
||||
}
|
||||
|
||||
/// Proxy HuggingFace model search (avoids browser CORS + adds our token).
|
||||
pub async fn hf_search(query: &str, limit: usize, token: Option<&str>) -> Result<serde_json::Value, String> {
|
||||
let q: String = url_escape(query);
|
||||
let sort = if query.trim().is_empty() { "trendingScore" } else { "downloads" };
|
||||
let url = format!(
|
||||
"https://huggingface.co/api/models?search={q}&sort={sort}&direction=-1&limit={limit}&full=false"
|
||||
);
|
||||
let client = reqwest::Client::builder()
|
||||
.timeout(std::time::Duration::from_secs(15))
|
||||
.build()
|
||||
.map_err(|e| e.to_string())?;
|
||||
let mut req = client.get(&url).header("User-Agent", "cortiq-gateway");
|
||||
if let Some(t) = token {
|
||||
req = req.bearer_auth(t);
|
||||
}
|
||||
let resp = req.send().await.map_err(|e| e.to_string())?;
|
||||
if !resp.status().is_success() {
|
||||
return Err(format!("HuggingFace API {}", resp.status()));
|
||||
}
|
||||
resp.json().await.map_err(|e| e.to_string())
|
||||
}
|
||||
|
||||
fn url_escape(s: &str) -> String {
|
||||
s.chars()
|
||||
.map(|c| match c {
|
||||
'a'..='z' | 'A'..='Z' | '0'..='9' | '-' | '_' | '.' | '/' => c.to_string(),
|
||||
' ' => "+".to_string(),
|
||||
_ => format!("%{:02X}", c as u32),
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn sanitize(s: &str) -> String {
|
||||
s.chars()
|
||||
.map(|c| if c.is_ascii_alphanumeric() || c == '-' || c == '_' { c } else { '-' })
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Kick off a conversion. Returns the job id immediately; progress streams
|
||||
/// into the job log. Fails fast if the converter script is missing.
|
||||
pub fn start_import(store: Arc<JobStore>, cfg: &CmfCfg, p: ImportParams) -> Result<String, String> {
|
||||
if p.repo.trim().is_empty() {
|
||||
return Err("empty repo id".into());
|
||||
}
|
||||
// Native `cortiq convert` (from crates.io, no Python) handles standard + MoE
|
||||
// models; the bundled Python converter is used only for advanced options it
|
||||
// doesn't support yet (linear-attention folding, v-bit shaping).
|
||||
let use_python = p.linear_core.is_some() || p.vbit_shape.is_some() || p.mean_bits.is_some();
|
||||
if use_python && !std::path::Path::new(&cfg.converter).exists() {
|
||||
return Err(format!("advanced options need the Python converter, not found: {}", cfg.converter));
|
||||
}
|
||||
let _ = std::fs::create_dir_all(&cfg.models_dir);
|
||||
let base = if p.name.trim().is_empty() {
|
||||
sanitize(p.repo.rsplit('/').next().unwrap_or(&p.repo))
|
||||
} else {
|
||||
sanitize(&p.name)
|
||||
};
|
||||
let output = std::path::Path::new(&cfg.models_dir)
|
||||
.join(format!("{base}.cmf"))
|
||||
.to_string_lossy()
|
||||
.to_string();
|
||||
let output_abs = std::fs::canonicalize(&cfg.models_dir)
|
||||
.map(|d| d.join(format!("{base}.cmf")).to_string_lossy().to_string())
|
||||
.unwrap_or_else(|_| output.clone());
|
||||
|
||||
let id = gen_id(&p.repo);
|
||||
store.insert(Job {
|
||||
id: id.clone(),
|
||||
repo: p.repo.clone(),
|
||||
quant: p.quant.clone(),
|
||||
output: output_abs.clone(),
|
||||
state: "running".into(),
|
||||
log: vec![format!("→ converting {} to {} ({})", p.repo, base, p.quant)],
|
||||
started: now(),
|
||||
finished: None,
|
||||
size_bytes: None,
|
||||
progress: Some(0.0),
|
||||
phase: Some("starting".into()),
|
||||
});
|
||||
let cancel = Arc::new(Notify::new());
|
||||
store.set_cancel(&id, cancel.clone());
|
||||
|
||||
let hf_token = if cfg.hf_token_env.is_empty() {
|
||||
None
|
||||
} else {
|
||||
std::env::var(&cfg.hf_token_env).ok()
|
||||
};
|
||||
|
||||
// Program + args: `cortiq convert` by default, or the Python converter for
|
||||
// advanced options. Both stream `@PROGRESS <frac>` markers into the log.
|
||||
let (program, args, workdir): (String, Vec<String>, std::path::PathBuf) = if use_python {
|
||||
let conv = std::path::Path::new(&cfg.converter);
|
||||
let mut a = vec![
|
||||
std::fs::canonicalize(conv).map(|p| p.to_string_lossy().to_string()).unwrap_or_else(|_| cfg.converter.clone()),
|
||||
"--model".into(), p.repo.clone(),
|
||||
"--quant".into(), p.quant.clone(),
|
||||
"--output".into(), output_abs.clone(),
|
||||
];
|
||||
if let Some(lc) = &p.linear_core { a.push("--linear-core".into()); a.push(lc.clone()); }
|
||||
if let Some(n) = p.nphase { a.push("--nphase".into()); a.push(n.to_string()); }
|
||||
if let Some(vs) = &p.vbit_shape { a.push("--vbit-shape".into()); a.push(vs.clone()); }
|
||||
if let Some(mb) = p.mean_bits { a.push("--mean-bits".into()); a.push(mb.to_string()); }
|
||||
if let Some(g) = p.shard_max_gb { a.push("--shard-max-gb".into()); a.push(g.to_string()); }
|
||||
if p.skip_mtp { a.push("--skip-mtp".into()); }
|
||||
let wd = conv.parent().map(|d| d.to_path_buf()).unwrap_or_else(|| std::path::PathBuf::from("."));
|
||||
(cfg.python_bin.clone(), a, wd)
|
||||
} else {
|
||||
// native Rust converter — quant name maps to cortiq's scheme.
|
||||
let quant = match p.quant.to_ascii_uppercase().as_str() {
|
||||
s if s.starts_with("Q4") => "q4",
|
||||
"Q8_2F" => "q8_2f",
|
||||
"F16" | "FP16" => "f16",
|
||||
_ => "q8",
|
||||
};
|
||||
let mut a = vec![
|
||||
"convert".into(),
|
||||
"--model".into(), p.repo.clone(),
|
||||
"--quant".into(), quant.into(),
|
||||
"--output".into(), output_abs.clone(),
|
||||
];
|
||||
if let Some(t) = &hf_token { a.push("--hf-token".into()); a.push(t.clone()); }
|
||||
(cfg.cortiq_bin.clone(), a, std::env::current_dir().unwrap_or_else(|_| std::path::PathBuf::from(".")))
|
||||
};
|
||||
|
||||
let ret_id = id.clone();
|
||||
let output_for_task = output_abs.clone();
|
||||
tokio::spawn(async move {
|
||||
let mut cmd = tokio::process::Command::new(&program);
|
||||
cmd.args(&args)
|
||||
.current_dir(&workdir)
|
||||
.env("PYTHONUNBUFFERED", "1")
|
||||
.stdout(Stdio::piped())
|
||||
.stderr(Stdio::piped());
|
||||
// HF token for the Python converter (native cortiq gets --hf-token).
|
||||
if use_python {
|
||||
if let Some(tok) = hf_token {
|
||||
cmd.env("HF_TOKEN", &tok).env("HUGGING_FACE_HUB_TOKEN", tok);
|
||||
}
|
||||
}
|
||||
let mut child = match cmd.spawn() {
|
||||
Ok(c) => c,
|
||||
Err(e) => {
|
||||
store.push_line(&id, format!("✗ spawn failed: {e}"));
|
||||
store.finish(&id, false);
|
||||
store.drop_cancel(&id);
|
||||
return;
|
||||
}
|
||||
};
|
||||
let stdout = child.stdout.take();
|
||||
let stderr = child.stderr.take();
|
||||
// stdout: parse @PROGRESS markers into progress/phase; everything else logs.
|
||||
let s1 = store.clone();
|
||||
let id1 = id.clone();
|
||||
let t_out = tokio::spawn(async move {
|
||||
if let Some(o) = stdout {
|
||||
let mut lines = BufReader::new(o).lines();
|
||||
while let Ok(Some(l)) = lines.next_line().await {
|
||||
if let Some((frac, phase)) = parse_progress(&l) {
|
||||
s1.set_progress(&id1, frac, phase);
|
||||
} else {
|
||||
s1.push_line(&id1, l);
|
||||
}
|
||||
}
|
||||
}
|
||||
});
|
||||
let s2 = store.clone();
|
||||
let id2 = id.clone();
|
||||
let t_err = tokio::spawn(async move {
|
||||
if let Some(e) = stderr {
|
||||
let mut lines = BufReader::new(e).lines();
|
||||
while let Ok(Some(l)) = lines.next_line().await {
|
||||
s2.push_line(&id2, l);
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
tokio::select! {
|
||||
status = child.wait() => {
|
||||
let _ = tokio::join!(t_out, t_err);
|
||||
let ok = status.map(|s| s.success()).unwrap_or(false);
|
||||
if !ok {
|
||||
cleanup_output(&output_for_task); // drop partial/corrupt output
|
||||
}
|
||||
store.push_line(
|
||||
&id,
|
||||
if ok { "✓ done".into() } else { "✗ converter exited with error".into() },
|
||||
);
|
||||
store.finish(&id, ok);
|
||||
}
|
||||
_ = cancel.notified() => {
|
||||
let _ = child.start_kill();
|
||||
let _ = child.wait().await;
|
||||
t_out.abort();
|
||||
t_err.abort();
|
||||
cleanup_output(&output_for_task);
|
||||
store.push_line(&id, "✗ cancelled by user — partial output removed".into());
|
||||
store.set_state(&id, "cancelled");
|
||||
}
|
||||
}
|
||||
store.drop_cancel(&id);
|
||||
});
|
||||
|
||||
Ok(ret_id)
|
||||
}
|
||||
+13
@@ -10,10 +10,13 @@
|
||||
|
||||
mod admin;
|
||||
mod cache;
|
||||
mod cmf_runtime;
|
||||
mod config;
|
||||
mod error;
|
||||
mod import;
|
||||
mod model;
|
||||
mod pipeline;
|
||||
mod promotion;
|
||||
mod protocols;
|
||||
mod providers;
|
||||
mod registry;
|
||||
@@ -79,6 +82,16 @@ async fn main() -> anyhow::Result<()> {
|
||||
|
||||
let state = SharedState::build(cfg, args.config.clone())?;
|
||||
|
||||
// Close the loop with the CMF format (github.com/infosave2007/cmf): install
|
||||
// / update cortiq-cli and run a local `cortiq serve` if configured. Runs in
|
||||
// the background so startup is never blocked by a cargo install or a model
|
||||
// load; the served model is registered in the pool and usable offline.
|
||||
{
|
||||
let cmf_rt = state.cmf.clone();
|
||||
let cmf_cfg = state.live().cfg.cmf.clone();
|
||||
tokio::spawn(async move { cmf_runtime::manage(cmf_rt, cmf_cfg).await });
|
||||
}
|
||||
|
||||
let mut app = Router::new()
|
||||
.route("/healthz", get(|| async { "ok" }))
|
||||
.route("/readyz", get(|| async { "ready" }))
|
||||
|
||||
+248
-44
@@ -4,7 +4,8 @@
|
||||
|
||||
use crate::error::{GatewayError, Result};
|
||||
use crate::model::{
|
||||
ChatRequest, ChatResponse, Choice, Message, RouteDecision, RouteInfo, RoutingDirective, Usage,
|
||||
ChatRequest, ChatResponse, Choice, GenParams, Message, RequestMeta, RouteDecision, RouteInfo,
|
||||
RoutingDirective, Usage,
|
||||
};
|
||||
use crate::state::{Live, SharedState};
|
||||
use crate::stats::RequestRecord;
|
||||
@@ -67,6 +68,21 @@ impl Pipeline {
|
||||
}
|
||||
}
|
||||
|
||||
// Self-warming shadow loop: capture the prompt BEFORE `req` is moved,
|
||||
// so we can (non-blocking, sampled) judge a local answer after the
|
||||
// cloud reply. Client latency is unaffected. Nothing is served locally
|
||||
// here — this only feeds the promotion table.
|
||||
let shadow_cfg = state.live().cfg.shadow.clone();
|
||||
let shadow_prompt = if shadow_cfg.enabled
|
||||
&& !shadow_cfg.local_model_id.is_empty()
|
||||
&& !shadow_cfg.judge_model_id.is_empty()
|
||||
{
|
||||
let live = state.live();
|
||||
Some(self.extract_text(&req.messages, "last_user", live.cfg.route.max_chars))
|
||||
} else {
|
||||
None
|
||||
};
|
||||
|
||||
let result = self.run_inner(req, state).await;
|
||||
|
||||
// record statistics (for both success and error)
|
||||
@@ -135,6 +151,29 @@ impl Pipeline {
|
||||
}
|
||||
}
|
||||
|
||||
// Fire the shadow judgment for a sampled subset (non-blocking).
|
||||
if let (Some(prompt), Ok(resp)) = (shadow_prompt, &result) {
|
||||
let cloud_answer = resp
|
||||
.choices
|
||||
.first()
|
||||
.map(|c| c.message.content.clone())
|
||||
.unwrap_or_default();
|
||||
let label = resp.cortiq.task_label.clone();
|
||||
// Only judge CLOUD-served answers — the reference must be the cloud
|
||||
// answer, not a local one (a promoted label served locally would
|
||||
// otherwise judge local-vs-local). Canary auditing of served-local
|
||||
// answers (forcing a cloud reference for a fraction) is the next step.
|
||||
if !cloud_answer.is_empty()
|
||||
&& !label.is_empty()
|
||||
&& label != "cache"
|
||||
&& resp.cortiq.selected_model != shadow_cfg.local_model_id
|
||||
&& should_shadow(&prompt, shadow_cfg.sample_rate)
|
||||
{
|
||||
let st = state.0.clone();
|
||||
tokio::spawn(shadow_capture(st, shadow_cfg, prompt, cloud_answer, label));
|
||||
}
|
||||
}
|
||||
|
||||
result
|
||||
}
|
||||
|
||||
@@ -152,50 +191,48 @@ impl Pipeline {
|
||||
live.cfg.route.max_chars,
|
||||
);
|
||||
|
||||
// 2. Obtain the router decision (or use the pinned model)
|
||||
let (decision, route_source) = match &req.routing {
|
||||
RoutingDirective::Pinned { model_id: _ } => {
|
||||
let dec = RouteDecision {
|
||||
task_label: "pinned".to_string(),
|
||||
complexity_score: 0.0,
|
||||
complexity_tier: "pinned".to_string(),
|
||||
router_request_id: None,
|
||||
source: "pinned".to_string(),
|
||||
};
|
||||
(dec, "pinned".to_string())
|
||||
}
|
||||
RoutingDirective::Auto { profile } => {
|
||||
let prof = profile.as_deref().unwrap_or(&live.cfg.route.profile);
|
||||
match live.router.route(&text, prof).await {
|
||||
Ok(Some(dec)) => {
|
||||
let src = dec.source.clone();
|
||||
(dec, src)
|
||||
}
|
||||
Ok(None) | Err(_) => {
|
||||
tracing::warn!(
|
||||
"Router is unavailable. Gracefully degrading to default model."
|
||||
);
|
||||
let dec = RouteDecision {
|
||||
task_label: "degraded".to_string(),
|
||||
complexity_score: 0.5,
|
||||
complexity_tier: "degraded".to_string(),
|
||||
router_request_id: None,
|
||||
source: "fallback".to_string(),
|
||||
};
|
||||
(dec, "fallback".to_string())
|
||||
}
|
||||
}
|
||||
}
|
||||
// 2–3. Decide the route and pick candidate models.
|
||||
//
|
||||
// Local-first rules, so CMF models work with no router at all:
|
||||
// • `[cmf].local_only` → every request goes to the local pool,
|
||||
// ignoring the router, the pin, and external providers;
|
||||
// • `[router].enabled = false` → skip the router, use the local pool;
|
||||
// • router unavailable at runtime → graceful degrade to the local pool;
|
||||
// • an explicitly pinned model → used directly, no router call.
|
||||
let mk = |label: &str, source: &str| RouteDecision {
|
||||
task_label: label.to_string(),
|
||||
complexity_score: 0.0,
|
||||
complexity_tier: label.to_string(),
|
||||
router_request_id: None,
|
||||
source: source.to_string(),
|
||||
};
|
||||
|
||||
// 3. Select candidates by complexity tier / pinned model
|
||||
let candidates = match &req.routing {
|
||||
RoutingDirective::Pinned { model_id } => vec![model_id.clone()],
|
||||
RoutingDirective::Auto { .. } => {
|
||||
if route_source == "fallback" {
|
||||
vec![live.routing.default_model().to_string()]
|
||||
} else {
|
||||
live.routing.candidates(&decision.complexity_tier)
|
||||
let (decision, route_source, candidates) = if live.cfg.cmf.local_only {
|
||||
(mk("local-only", "local"), "local".to_string(), live.local_candidates())
|
||||
} else {
|
||||
match &req.routing {
|
||||
RoutingDirective::Pinned { model_id } => {
|
||||
(mk("pinned", "pinned"), "pinned".to_string(), vec![model_id.clone()])
|
||||
}
|
||||
RoutingDirective::Auto { profile } => {
|
||||
if !live.cfg.router.enabled {
|
||||
(mk("router-disabled", "local"), "local".to_string(), live.local_candidates())
|
||||
} else {
|
||||
let prof = profile.as_deref().unwrap_or(&live.cfg.route.profile);
|
||||
match live.router.route(&text, prof).await {
|
||||
Ok(Some(dec)) => {
|
||||
let src = dec.source.clone();
|
||||
let tier = dec.complexity_tier.clone();
|
||||
(dec, src, live.routing.candidates(&tier))
|
||||
}
|
||||
Ok(None) | Err(_) => {
|
||||
tracing::warn!(
|
||||
"Router is unavailable. Gracefully degrading to the local/default model."
|
||||
);
|
||||
(mk("degraded", "fallback"), "fallback".to_string(), live.local_candidates())
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
};
|
||||
@@ -216,6 +253,19 @@ impl Pipeline {
|
||||
}
|
||||
}
|
||||
|
||||
// Local-first safety net: if routing produced no usable candidate (e.g.
|
||||
// an unknown pinned model while the router is off), fall back to the
|
||||
// managed local model so a configured local backend still answers.
|
||||
if final_candidates.is_empty() && live.cfg.cmf.manage_server {
|
||||
for id in live.local_candidates() {
|
||||
if let Some(prov) = live.registry.get(&id) {
|
||||
if needs_tools && !prov.caps().tools {
|
||||
continue;
|
||||
}
|
||||
final_candidates.push(id);
|
||||
}
|
||||
}
|
||||
}
|
||||
if final_candidates.is_empty() {
|
||||
return Err(GatewayError::UpstreamUnavailable(
|
||||
"No healthy model available that satisfies required capabilities (e.g. tools)"
|
||||
@@ -227,7 +277,23 @@ impl Pipeline {
|
||||
|
||||
async fn run_inner(&self, req: ChatRequest, state: &SharedState) -> Result<ChatResponse> {
|
||||
let live = state.live();
|
||||
let (decision, route_source, final_candidates) = self.resolve(&req, &live).await?;
|
||||
let (decision, route_source, mut final_candidates) = self.resolve(&req, &live).await?;
|
||||
|
||||
// Serving-when-promoted (gated): a promoted task-label is served by the
|
||||
// local model FIRST (failover to cloud on error). Per-request VETO uses
|
||||
// the router's own complexity signal — a "high"-tier request escalates
|
||||
// to cloud even in a promoted label (the local skill is for the common
|
||||
// case, not the hard tail). Off by default.
|
||||
let sh = &live.cfg.shadow;
|
||||
if sh.serve_when_promoted
|
||||
&& !sh.local_model_id.is_empty()
|
||||
&& matches!(req.routing, RoutingDirective::Auto { .. })
|
||||
&& state.promotion.serves_local(&decision.task_label)
|
||||
&& !matches!(decision.complexity_tier.as_str(), "high" | "hard" | "complex")
|
||||
&& !final_candidates.iter().any(|m| m == &sh.local_model_id)
|
||||
{
|
||||
final_candidates.insert(0, sh.local_model_id.clone());
|
||||
}
|
||||
|
||||
// 4. Attempt provider call with failover (fall back to alternatives on error)
|
||||
let mut last_err = None;
|
||||
@@ -592,3 +658,141 @@ fn tap_stream(
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// ─────────────────────────── self-warming shadow loop ───────────────────────────
|
||||
|
||||
/// Deterministic sampling by prompt hash — no RNG dependency, and the same
|
||||
/// prompt is always (not) sampled, so repeated office templates don't skew
|
||||
/// the judged set request-by-request.
|
||||
fn should_shadow(prompt: &str, rate: f64) -> bool {
|
||||
if rate <= 0.0 {
|
||||
return false;
|
||||
}
|
||||
if rate >= 1.0 {
|
||||
return true;
|
||||
}
|
||||
use std::hash::{Hash, Hasher};
|
||||
let mut h = std::collections::hash_map::DefaultHasher::new();
|
||||
prompt.hash(&mut h);
|
||||
(h.finish() % 10_000) < (rate * 10_000.0) as u64
|
||||
}
|
||||
|
||||
fn clip(s: &str, max: usize) -> String {
|
||||
if s.len() <= max {
|
||||
s.to_string()
|
||||
} else {
|
||||
s.chars().take(max).collect()
|
||||
}
|
||||
}
|
||||
|
||||
fn shadow_req(model_id: &str, content: String, max_tokens: u32) -> ChatRequest {
|
||||
ChatRequest {
|
||||
routing: RoutingDirective::Pinned {
|
||||
model_id: model_id.to_string(),
|
||||
},
|
||||
messages: vec![Message {
|
||||
role: "user".to_string(),
|
||||
content,
|
||||
tool_calls: Vec::new(),
|
||||
}],
|
||||
tools: Vec::new(),
|
||||
params: GenParams {
|
||||
temperature: Some(0.0),
|
||||
max_tokens: Some(max_tokens),
|
||||
..Default::default()
|
||||
},
|
||||
stream: false,
|
||||
meta: RequestMeta::default(),
|
||||
}
|
||||
}
|
||||
|
||||
/// Extract `{"score":N,"hard_fail":bool}` from a judge reply, tolerant of
|
||||
/// surrounding prose. Missing/garbled → (0, true) (fail-closed: an
|
||||
/// unparseable judgment must never count as a pass).
|
||||
fn parse_judge(text: &str) -> (u8, bool) {
|
||||
let score = text
|
||||
.split("score")
|
||||
.nth(1)
|
||||
.and_then(|t| t.chars().skip_while(|c| !c.is_ascii_digit()).next())
|
||||
.and_then(|c| c.to_digit(10))
|
||||
.map(|d| d.min(4) as u8);
|
||||
let hard_fail = text
|
||||
.split("hard_fail")
|
||||
.nth(1)
|
||||
.map(|t| t.contains("true"))
|
||||
.unwrap_or(true);
|
||||
match score {
|
||||
Some(s) => (s, hard_fail),
|
||||
None => (0, true),
|
||||
}
|
||||
}
|
||||
|
||||
/// Non-blocking: answer locally, have the cheap judge score it against the
|
||||
/// already-paid cloud answer, and feed the per-label promotion table. Any
|
||||
/// failure (missing provider, upstream error, unparseable judgment) is
|
||||
/// silently dropped — the client already got the cloud answer.
|
||||
async fn shadow_capture(
|
||||
st: std::sync::Arc<crate::state::AppState>,
|
||||
cfg: crate::config::ShadowCfg,
|
||||
prompt: String,
|
||||
cloud_answer: String,
|
||||
label: String,
|
||||
) {
|
||||
let live = st.live.load_full();
|
||||
let (Some(local), Some(judge)) = (
|
||||
live.registry.get(&cfg.local_model_id),
|
||||
live.registry.get(&cfg.judge_model_id),
|
||||
) else {
|
||||
return;
|
||||
};
|
||||
|
||||
// 1. Local answer (free, CPU).
|
||||
let a_local = match local.chat(shadow_req(&cfg.local_model_id, clip(&prompt, 4000), 512)).await {
|
||||
Ok(r) => r
|
||||
.choices
|
||||
.into_iter()
|
||||
.next()
|
||||
.map(|c| c.message.content)
|
||||
.unwrap_or_default(),
|
||||
Err(_) => return,
|
||||
};
|
||||
if a_local.trim().is_empty() {
|
||||
return;
|
||||
}
|
||||
|
||||
// 2. Reference-based judgment (cheap model, different family).
|
||||
let jp = format!(
|
||||
"You grade a CANDIDATE answer against a REFERENCE answer for the same task.\n\
|
||||
Task type: {label}\n\n[TASK]\n{task}\n\n[REFERENCE]\n{reference}\n\n[CANDIDATE]\n{candidate}\n\n\
|
||||
Reply with ONLY compact JSON: {{\"score\":0-4,\"hard_fail\":true|false}}. \
|
||||
score>=3 means the candidate is acceptable (correct, complete, right format). \
|
||||
hard_fail=true for fabricated facts, broken format, or unsafe content.",
|
||||
label = label,
|
||||
task = clip(&prompt, 2000),
|
||||
reference = clip(&cloud_answer, 3000),
|
||||
candidate = clip(&a_local, 3000),
|
||||
);
|
||||
let Ok(jr) = judge.chat(shadow_req(&cfg.judge_model_id, jp, 32)).await else {
|
||||
return;
|
||||
};
|
||||
let judge_tokens = jr.usage.total_tokens;
|
||||
let jtext = jr
|
||||
.choices
|
||||
.into_iter()
|
||||
.next()
|
||||
.map(|c| c.message.content)
|
||||
.unwrap_or_default();
|
||||
let (score, hard_fail) = parse_judge(&jtext);
|
||||
|
||||
st.promotion.record(crate::promotion::JudgeRecord {
|
||||
ts: now_secs(),
|
||||
task_label: label,
|
||||
pass: score >= 3 && !hard_fail,
|
||||
hard_fail,
|
||||
score,
|
||||
source: "judge".to_string(),
|
||||
recon_e: None,
|
||||
born_conf: None,
|
||||
judge_tokens,
|
||||
});
|
||||
}
|
||||
|
||||
@@ -0,0 +1,381 @@
|
||||
//! Cloud-as-judge promotion table (self-warming local model, MVP).
|
||||
//!
|
||||
//! Per task-label we accumulate a judge/checker PASS stream and gate the
|
||||
//! per-label lifecycle on the **Wilson-95 LOWER bound** of the pass rate —
|
||||
//! never on the point estimate, so a lucky streak of easy requests can NOT
|
||||
//! promote a label (small n → wide interval → low bound → stays SHADOW).
|
||||
//! The full design + honest holes: docs/SELF_WARMING_GATEWAY.md.
|
||||
//!
|
||||
//! This module is pure bookkeeping (like `stats::Stats`): it decides the
|
||||
//! STATE; the pipeline consults the state to decide who serves. Nothing here
|
||||
//! serves a local answer — that is the caller's decision, guarded by these
|
||||
//! states plus a per-request veto.
|
||||
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::collections::{HashMap, VecDeque};
|
||||
use std::io::Write;
|
||||
use std::sync::Mutex;
|
||||
use std::time::{SystemTime, UNIX_EPOCH};
|
||||
|
||||
fn now_secs() -> u64 {
|
||||
SystemTime::now().duration_since(UNIX_EPOCH).unwrap_or_default().as_secs()
|
||||
}
|
||||
|
||||
/// One judgment of a local answer (JSONL line, replayed on startup).
|
||||
#[derive(Clone, Debug, Serialize, Deserialize)]
|
||||
pub struct JudgeRecord {
|
||||
pub ts: u64,
|
||||
pub task_label: String,
|
||||
/// 1 = local answer accepted (checker pass OR judge score≥bar & !hard_fail).
|
||||
pub pass: bool,
|
||||
/// Fact hallucination / broken format / unsafe — blocks promotion hard.
|
||||
#[serde(default)]
|
||||
pub hard_fail: bool,
|
||||
/// Raw judge score 0..4 (diagnostic only); 4 for a $0 programmatic checker.
|
||||
#[serde(default)]
|
||||
pub score: u8,
|
||||
/// "checker" ($0 ground-truth) or "judge" (cloud LLM-as-judge).
|
||||
#[serde(default)]
|
||||
pub source: String,
|
||||
/// Local recon-E (novelty) at judge time — diagnostic; NOT a correctness gate.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub recon_e: Option<f32>,
|
||||
/// Local calibrated Born-mass confidence — for the negative-control study.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub born_conf: Option<f32>,
|
||||
/// Cloud tokens spent judging (0 for a programmatic checker).
|
||||
#[serde(default)]
|
||||
pub judge_tokens: u32,
|
||||
}
|
||||
|
||||
/// Per-label lifecycle. Serving-locally is only ever Canary/LocalServed.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub enum PromotionState {
|
||||
CloudOnly,
|
||||
ShadowJudged,
|
||||
Canary,
|
||||
LocalServed,
|
||||
Demoted,
|
||||
}
|
||||
|
||||
/// Tuning (defaults from the memo; refit to measured ECE/traffic).
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct PromotionCfg {
|
||||
pub enabled: bool,
|
||||
pub window: usize, // rolling judgments per label (W)
|
||||
pub n_min: usize, // min judgments before promotion is even considered
|
||||
pub promote_lb: f64, // Wilson-95 LB of pass must clear this
|
||||
pub demote_pass: f64, // point pass-rate below this → demote a served label
|
||||
pub hardfail_ub_max: f64,// Wilson-95 UPPER bound of hard_fail must stay under this
|
||||
pub soak: usize, // extra Canary judgments before LocalServed
|
||||
pub z: f64, // 1.96 = 95%
|
||||
pub file: Option<String>,// judge.jsonl path (append + replay)
|
||||
}
|
||||
|
||||
impl Default for PromotionCfg {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
enabled: false,
|
||||
window: 500,
|
||||
n_min: 200,
|
||||
promote_lb: 0.95,
|
||||
demote_pass: 0.90,
|
||||
hardfail_ub_max: 0.015,
|
||||
soak: 100,
|
||||
z: 1.96,
|
||||
file: None,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
struct LabelState {
|
||||
passes: VecDeque<bool>, // rolling window of pass/fail
|
||||
hard_fails: VecDeque<bool>, // rolling window of hard_fail flags
|
||||
state: PromotionState,
|
||||
soaked: usize, // judgments accumulated while in Canary
|
||||
}
|
||||
|
||||
impl LabelState {
|
||||
fn new() -> Self {
|
||||
Self {
|
||||
passes: VecDeque::new(),
|
||||
hard_fails: VecDeque::new(),
|
||||
state: PromotionState::ShadowJudged,
|
||||
soaked: 0,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub struct Promotion {
|
||||
cfg: PromotionCfg,
|
||||
labels: Mutex<HashMap<String, LabelState>>,
|
||||
}
|
||||
|
||||
/// Wilson score interval bound for a Bernoulli rate. `upper=false` → lower.
|
||||
/// s successes of n trials, z the normal quantile. Distribution-aware: small
|
||||
/// n gives a wide interval (low lower bound / high upper bound) by construction.
|
||||
pub fn wilson_bound(s: usize, n: usize, z: f64, upper: bool) -> f64 {
|
||||
if n == 0 {
|
||||
return if upper { 1.0 } else { 0.0 };
|
||||
}
|
||||
let n = n as f64;
|
||||
let p = s as f64 / n;
|
||||
let z2 = z * z;
|
||||
let denom = 1.0 + z2 / n;
|
||||
let centre = p + z2 / (2.0 * n);
|
||||
let margin = z * ((p * (1.0 - p) / n) + z2 / (4.0 * n * n)).sqrt();
|
||||
let v = if upper { centre + margin } else { centre - margin };
|
||||
(v / denom).clamp(0.0, 1.0)
|
||||
}
|
||||
|
||||
impl Promotion {
|
||||
pub fn new(cfg: PromotionCfg) -> std::sync::Arc<Self> {
|
||||
let p = std::sync::Arc::new(Self { cfg, labels: Mutex::new(HashMap::new()) });
|
||||
p.replay();
|
||||
p
|
||||
}
|
||||
|
||||
fn replay(&self) {
|
||||
let Some(path) = &self.cfg.file else { return };
|
||||
let Ok(content) = std::fs::read_to_string(path) else { return };
|
||||
for line in content.lines() {
|
||||
if let Ok(rec) = serde_json::from_str::<JudgeRecord>(line) {
|
||||
self.apply(&rec);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Record a judgment: update the rolling windows, advance the state
|
||||
/// machine, append to judge.jsonl. Returns the (possibly new) state.
|
||||
pub fn record(&self, rec: JudgeRecord) -> PromotionState {
|
||||
let st = self.apply(&rec);
|
||||
if let Some(path) = &self.cfg.file {
|
||||
if let Ok(line) = serde_json::to_string(&rec) {
|
||||
if let Ok(mut f) =
|
||||
std::fs::OpenOptions::new().create(true).append(true).open(path)
|
||||
{
|
||||
let _ = writeln!(f, "{line}");
|
||||
}
|
||||
}
|
||||
}
|
||||
st
|
||||
}
|
||||
|
||||
fn apply(&self, rec: &JudgeRecord) -> PromotionState {
|
||||
let mut map = self.labels.lock().unwrap();
|
||||
let ls = map.entry(rec.task_label.clone()).or_insert_with(LabelState::new);
|
||||
push_cap(&mut ls.passes, rec.pass, self.cfg.window);
|
||||
push_cap(&mut ls.hard_fails, rec.hard_fail, self.cfg.window);
|
||||
self.advance(ls);
|
||||
ls.state
|
||||
}
|
||||
|
||||
fn advance(&self, ls: &mut LabelState) {
|
||||
let n = ls.passes.len();
|
||||
let s = ls.passes.iter().filter(|&&p| p).count();
|
||||
let hf = ls.hard_fails.iter().filter(|&&h| h).count();
|
||||
let lb = wilson_bound(s, n, self.cfg.z, false);
|
||||
let point = if n > 0 { s as f64 / n as f64 } else { 0.0 };
|
||||
let recent_hf = ls.hard_fails.iter().rev().take(50).filter(|&&h| h).count();
|
||||
|
||||
// Zero-tolerance on hard-fails in the window: a Wilson UPPER bound on
|
||||
// hard-fail rate ≤ 0.015 is UNREACHABLE at n=200 even with 0 fails
|
||||
// (rule-of-three: 0/200 → ~0.019) — the panel flagged this. Requiring
|
||||
// ZERO recent hard-fails is achievable AND safer (a recent
|
||||
// hallucination/unsafe answer → stay on cloud until it ages out).
|
||||
let bar = n >= self.cfg.n_min && lb >= self.cfg.promote_lb && hf == 0;
|
||||
let regressed = point < self.cfg.demote_pass || recent_hf >= 2;
|
||||
|
||||
ls.state = match ls.state {
|
||||
PromotionState::CloudOnly | PromotionState::ShadowJudged => {
|
||||
if bar { ls.soaked = 0; PromotionState::Canary } else { PromotionState::ShadowJudged }
|
||||
}
|
||||
PromotionState::Canary => {
|
||||
if regressed {
|
||||
PromotionState::Demoted
|
||||
} else {
|
||||
ls.soaked += 1;
|
||||
if ls.soaked >= self.cfg.soak && bar {
|
||||
PromotionState::LocalServed
|
||||
} else {
|
||||
PromotionState::Canary
|
||||
}
|
||||
}
|
||||
}
|
||||
PromotionState::LocalServed => {
|
||||
if regressed { PromotionState::Demoted } else { PromotionState::LocalServed }
|
||||
}
|
||||
// Demoted goes back to SHADOW to re-earn promotion (skill stays in .cmf).
|
||||
PromotionState::Demoted => PromotionState::ShadowJudged,
|
||||
};
|
||||
}
|
||||
|
||||
/// Current lifecycle state of a label.
|
||||
pub fn state(&self, label: &str) -> PromotionState {
|
||||
self.labels
|
||||
.lock()
|
||||
.unwrap()
|
||||
.get(label)
|
||||
.map(|l| l.state)
|
||||
.unwrap_or(PromotionState::CloudOnly)
|
||||
}
|
||||
|
||||
/// Should a request of this label be SERVED by the local model?
|
||||
/// (Canary/LocalServed; caller still applies the per-request veto.)
|
||||
pub fn serves_local(&self, label: &str) -> bool {
|
||||
matches!(self.state(label), PromotionState::Canary | PromotionState::LocalServed)
|
||||
}
|
||||
|
||||
/// Snapshot for /admin: (state, n, pass-rate, Wilson-LB) per label.
|
||||
pub fn snapshot(&self) -> Vec<(String, PromotionState, usize, f64, f64)> {
|
||||
let map = self.labels.lock().unwrap();
|
||||
let mut out: Vec<_> = map
|
||||
.iter()
|
||||
.map(|(k, ls)| {
|
||||
let n = ls.passes.len();
|
||||
let s = ls.passes.iter().filter(|&&p| p).count();
|
||||
let p = if n > 0 { s as f64 / n as f64 } else { 0.0 };
|
||||
(k.clone(), ls.state, n, p, wilson_bound(s, n, self.cfg.z, false))
|
||||
})
|
||||
.collect();
|
||||
out.sort_by(|a, b| b.2.cmp(&a.2));
|
||||
out
|
||||
}
|
||||
|
||||
pub fn enabled(&self) -> bool {
|
||||
self.cfg.enabled
|
||||
}
|
||||
}
|
||||
|
||||
fn push_cap<T>(q: &mut VecDeque<T>, v: T, cap: usize) {
|
||||
q.push_back(v);
|
||||
while q.len() > cap {
|
||||
q.pop_front();
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn cfg() -> PromotionCfg {
|
||||
PromotionCfg { enabled: true, n_min: 200, soak: 0, ..Default::default() }
|
||||
}
|
||||
fn rec(label: &str, pass: bool, hard_fail: bool) -> JudgeRecord {
|
||||
JudgeRecord {
|
||||
ts: 0, task_label: label.into(), pass, hard_fail, score: if pass { 4 } else { 0 },
|
||||
source: "checker".into(), recon_e: None, born_conf: None, judge_tokens: 0,
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn wilson_lb_penalizes_small_n() {
|
||||
// 5/5 perfect but tiny n → lower bound far below 0.95.
|
||||
assert!(wilson_bound(5, 5, 1.96, false) < 0.6);
|
||||
// 195/200 at 97.5% → lower bound clears 0.95.
|
||||
assert!(wilson_bound(195, 200, 1.96, false) >= 0.94);
|
||||
// upper bound of 0 hard-fails over 200 ≈ 0.019 (rule-of-three-ish).
|
||||
assert!(wilson_bound(0, 200, 1.96, true) < 0.025);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn no_promotion_on_a_lucky_streak() {
|
||||
let p = Promotion::new(cfg());
|
||||
// 50 straight passes: point-rate 100% but n < n_min AND LB < bar.
|
||||
for _ in 0..50 {
|
||||
p.record(rec("code", true, false));
|
||||
}
|
||||
assert_eq!(p.state("code"), PromotionState::ShadowJudged);
|
||||
assert!(!p.serves_local("code"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn promotes_only_with_enough_high_evidence() {
|
||||
let p = Promotion::new(cfg());
|
||||
// 198/200 (99%) + zero hard-fails → Wilson-LB ≈ 0.964 ≥ 0.95 → Canary.
|
||||
// (195/200 = 97.5% gives LB ≈ 0.943 < 0.95 and would NOT promote —
|
||||
// the bound is deliberately strict.)
|
||||
for i in 0..200 {
|
||||
p.record(rec("code", i % 100 != 0, false)); // 2 fails / 200
|
||||
}
|
||||
assert_eq!(p.state("code"), PromotionState::Canary); // soak=0 → straight to Canary
|
||||
assert!(p.serves_local("code"));
|
||||
}
|
||||
|
||||
/// Runnable narrative of the full loop:
|
||||
/// cargo test --release demo_full_lifecycle -- --nocapture
|
||||
/// Two labels side by side — a mature skill that promotes to local
|
||||
/// serving, and a task the local can't do (stays on cloud). Proves the
|
||||
/// state machine both PROMOTES on real evidence and REFUSES safely.
|
||||
#[test]
|
||||
fn demo_full_lifecycle() {
|
||||
let p = Promotion::new(PromotionCfg { soak: 50, ..cfg() });
|
||||
let show = |label: &str| {
|
||||
let snap = p.snapshot();
|
||||
let row = snap.iter().find(|r| r.0 == label);
|
||||
match row {
|
||||
Some((_, st, n, pr, lb)) => println!(
|
||||
" {label:<12} {:<13} n={n:<4} pass={:.0}% LB={:.3} serves_local={}",
|
||||
format!("{st:?}"), pr * 100.0, lb, p.serves_local(label)
|
||||
),
|
||||
None => println!(" {label:<12} (cloud-only, no judgments)"),
|
||||
}
|
||||
};
|
||||
|
||||
println!("\n── self-warming loop, live ──");
|
||||
// A mature skill (e.g. ru-tech): local answers pass the judge ~99%.
|
||||
println!(" 1) SHADOW→CANARY: судим локаль 'ru_tech' (сильный скилл ~99% pass), 210 судейств:");
|
||||
for i in 0..210 {
|
||||
p.record(rec("ru_tech", i % 100 != 0, false));
|
||||
}
|
||||
show("ru_tech");
|
||||
assert_eq!(p.state("ru_tech"), PromotionState::Canary, "n≥200 & LB≥0.95 → CANARY");
|
||||
|
||||
println!(" 2) CANARY soak (ещё 60 судейств держат планку) → LOCAL_SERVED:");
|
||||
for i in 0..60 {
|
||||
p.record(rec("ru_tech", i % 100 != 0, false));
|
||||
}
|
||||
show("ru_tech");
|
||||
assert_eq!(p.state("ru_tech"), PromotionState::LocalServed, "soak → LOCAL_SERVED");
|
||||
assert!(p.serves_local("ru_tech"), "теперь обслуживается локально");
|
||||
|
||||
// The code dataset: local fails the checker → never promotes.
|
||||
println!(" 3) параллельно 'code' (0.8B не тянет генерацию кода, 0% pass):");
|
||||
for _ in 0..250 {
|
||||
p.record(rec("code", false, false));
|
||||
}
|
||||
show("code");
|
||||
assert_eq!(p.state("code"), PromotionState::ShadowJudged, "остаётся на облаке — безопасно");
|
||||
assert!(!p.serves_local("code"));
|
||||
|
||||
// Drift on the promoted label → auto-demote back to cloud.
|
||||
println!(" 4) деградация 'ru_tech' (дрейф) → авто-DEMOTED, назад на облако:");
|
||||
for _ in 0..120 {
|
||||
p.record(rec("ru_tech", false, false));
|
||||
}
|
||||
show("ru_tech");
|
||||
assert!(!p.serves_local("ru_tech"), "демоушен снял с локали");
|
||||
println!(" ── петля: промоушен по доказательству, отказ и откат — безопасно ──\n");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn hard_fail_and_regression_block_or_demote() {
|
||||
// A hard-fail spike keeps hf upper bound high → no promotion.
|
||||
let p = Promotion::new(cfg());
|
||||
for i in 0..200 {
|
||||
p.record(rec("risky", true, i % 20 == 0)); // 5% hard_fail
|
||||
}
|
||||
assert_ne!(p.state("risky"), PromotionState::LocalServed);
|
||||
|
||||
// Promote a clean label, then feed a regression → demote.
|
||||
let p2 = Promotion::new(PromotionCfg { soak: 0, ..cfg() });
|
||||
for i in 0..200 {
|
||||
p2.record(rec("t", i % 100 != 0, false)); // 198/200, 0 hard-fails
|
||||
}
|
||||
assert_eq!(p2.state("t"), PromotionState::Canary);
|
||||
for _ in 0..300 {
|
||||
p2.record(rec("t", false, false)); // quality collapses
|
||||
}
|
||||
assert!(matches!(p2.state("t"), PromotionState::Demoted | PromotionState::ShadowJudged));
|
||||
}
|
||||
}
|
||||
+29
-1
@@ -1,7 +1,7 @@
|
||||
//! Model pool registry: builds providers from `[[models]]` and resolves
|
||||
//! `model_id → Arc<dyn Provider>`.
|
||||
|
||||
use crate::config::Config;
|
||||
use crate::config::{Config, ModelCfg};
|
||||
use crate::providers::{anthropic::AnthropicProvider, openai::OpenAiProvider, Provider};
|
||||
use crate::secrets::SecretStore;
|
||||
use std::collections::HashMap;
|
||||
@@ -27,9 +27,37 @@ impl Registry {
|
||||
};
|
||||
by_id.insert(m.id.clone(), provider);
|
||||
}
|
||||
|
||||
// Managed local CMF server: register it as an OpenAI-compatible provider
|
||||
// so it can be routed to like any other backend — including when the
|
||||
// router is unavailable or the gateway is in local-only mode.
|
||||
if cfg.cmf.manage_server
|
||||
&& !cfg.cmf.local_model.trim().is_empty()
|
||||
&& !by_id.contains_key(&cfg.cmf.model_id)
|
||||
{
|
||||
let m = ModelCfg {
|
||||
id: cfg.cmf.model_id.clone(),
|
||||
provider: "openai".into(),
|
||||
base_url: format!("http://{}:{}/v1", cfg.cmf.local_host, cfg.cmf.local_port),
|
||||
model: "cortiq".into(),
|
||||
cost_tier: "local".into(),
|
||||
price_in: 0.0,
|
||||
price_out: 0.0,
|
||||
kind: "chat".into(),
|
||||
api_key_env: None,
|
||||
caps: Vec::new(),
|
||||
};
|
||||
by_id.insert(m.id.clone(), Arc::new(OpenAiProvider::new(&m, None)?));
|
||||
}
|
||||
Ok(Self { by_id })
|
||||
}
|
||||
|
||||
/// Any model id, preferring a real registered one (used as a local-first
|
||||
/// fallback target when routing cannot pick a specific model).
|
||||
pub fn any_id(&self) -> Option<String> {
|
||||
self.by_id.keys().next().cloned()
|
||||
}
|
||||
|
||||
pub fn get(&self, model_id: &str) -> Option<Arc<dyn Provider>> {
|
||||
self.by_id.get(model_id).cloned()
|
||||
}
|
||||
|
||||
+47
-3
@@ -4,16 +4,48 @@
|
||||
|
||||
use crate::config::RouterCfg;
|
||||
use crate::model::RouteDecision;
|
||||
use crate::secrets::SecretStore;
|
||||
use std::sync::atomic::{AtomicU16, Ordering};
|
||||
use std::sync::Arc;
|
||||
|
||||
/// Outcome of the most recent real `/v1/route` call, shared with the admin health
|
||||
/// endpoint so the panel can tell "no money / bad key" apart from "router down".
|
||||
/// 0 = ok (or no calls yet); [`STATUS_NETWORK`] / [`STATUS_TIMEOUT`] for transport
|
||||
/// failures; otherwise the upstream HTTP status code.
|
||||
pub type RouterLastStatus = Arc<AtomicU16>;
|
||||
|
||||
pub const STATUS_NETWORK: u16 = 1;
|
||||
pub const STATUS_TIMEOUT: u16 = 2;
|
||||
|
||||
/// Error kind for the admin panel.
|
||||
pub fn classify_status(code: u16) -> &'static str {
|
||||
match code {
|
||||
STATUS_NETWORK => "unreachable",
|
||||
STATUS_TIMEOUT => "timeout",
|
||||
401 | 403 => "auth",
|
||||
402 => "payment",
|
||||
429 => "quota",
|
||||
_ => "error",
|
||||
}
|
||||
}
|
||||
|
||||
pub struct RouterClient {
|
||||
http: reqwest::Client,
|
||||
url: String,
|
||||
api_key: Option<String>,
|
||||
taxonomy_id: Option<String>,
|
||||
last_status: RouterLastStatus,
|
||||
}
|
||||
|
||||
impl RouterClient {
|
||||
pub fn new(cfg: &RouterCfg) -> anyhow::Result<Self> {
|
||||
/// The key is resolved through the secret store (admin panel value first,
|
||||
/// then the environment variable) so a key pasted in the console works
|
||||
/// without a restart.
|
||||
pub fn new(
|
||||
cfg: &RouterCfg,
|
||||
secrets: &SecretStore,
|
||||
last_status: RouterLastStatus,
|
||||
) -> anyhow::Result<Self> {
|
||||
let mut builder =
|
||||
reqwest::Client::builder().timeout(std::time::Duration::from_millis(cfg.timeout_ms));
|
||||
|
||||
@@ -27,12 +59,13 @@ impl RouterClient {
|
||||
let api_key = cfg
|
||||
.api_key_env
|
||||
.as_ref()
|
||||
.and_then(|name| std::env::var(name).ok());
|
||||
.and_then(|name| secrets.resolve(name));
|
||||
Ok(Self {
|
||||
http,
|
||||
url: cfg.url.trim_end_matches('/').to_string(),
|
||||
api_key,
|
||||
taxonomy_id: cfg.taxonomy_id.clone(),
|
||||
last_status,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -54,11 +87,22 @@ impl RouterClient {
|
||||
|
||||
let resp = match req.send().await {
|
||||
Ok(r) => r,
|
||||
Err(_) => return Ok(None), // degradation
|
||||
Err(e) => {
|
||||
let code = if e.is_timeout() {
|
||||
STATUS_TIMEOUT
|
||||
} else {
|
||||
STATUS_NETWORK
|
||||
};
|
||||
self.last_status.store(code, Ordering::Relaxed);
|
||||
return Ok(None); // degradation
|
||||
}
|
||||
};
|
||||
if !resp.status().is_success() {
|
||||
self.last_status
|
||||
.store(resp.status().as_u16(), Ordering::Relaxed);
|
||||
return Ok(None);
|
||||
}
|
||||
self.last_status.store(0, Ordering::Relaxed);
|
||||
let v: serde_json::Value = resp.json().await?;
|
||||
let d = &v["decision"];
|
||||
Ok(Some(RouteDecision {
|
||||
|
||||
+60
-6
@@ -8,7 +8,7 @@
|
||||
use crate::cache::SemanticCache;
|
||||
use crate::config::Config;
|
||||
use crate::registry::Registry;
|
||||
use crate::router_client::RouterClient;
|
||||
use crate::router_client::{RouterClient, RouterLastStatus};
|
||||
use crate::routing::RoutingTable;
|
||||
use crate::secrets::SecretStore;
|
||||
use crate::stats::Stats;
|
||||
@@ -27,8 +27,12 @@ pub struct Live {
|
||||
}
|
||||
|
||||
impl Live {
|
||||
pub fn build(cfg: Config, secrets: &SecretStore) -> anyhow::Result<Self> {
|
||||
let router = RouterClient::new(&cfg.router)?;
|
||||
pub fn build(
|
||||
cfg: Config,
|
||||
secrets: &SecretStore,
|
||||
router_status: RouterLastStatus,
|
||||
) -> anyhow::Result<Self> {
|
||||
let router = RouterClient::new(&cfg.router, secrets, router_status)?;
|
||||
let registry = Registry::from_config(&cfg, secrets)?;
|
||||
let routing = RoutingTable::from_config(&cfg);
|
||||
Ok(Self {
|
||||
@@ -38,15 +42,57 @@ impl Live {
|
||||
router,
|
||||
})
|
||||
}
|
||||
|
||||
/// Local-first fallback targets, in order: the managed local CMF model (if
|
||||
/// configured), then the routing default, then any registered model — only
|
||||
/// registered ids are returned. Used whenever the router is bypassed
|
||||
/// (local-only / disabled) or unavailable, so CMF models serve with no router.
|
||||
pub fn local_candidates(&self) -> Vec<String> {
|
||||
let mut v = Vec::new();
|
||||
if self.cfg.cmf.manage_server && !self.cfg.cmf.local_model.trim().is_empty() {
|
||||
v.push(self.cfg.cmf.model_id.clone());
|
||||
}
|
||||
let def = self.routing.default_model().to_string();
|
||||
if !def.is_empty() && !v.contains(&def) {
|
||||
v.push(def);
|
||||
}
|
||||
v.retain(|id| self.registry.get(id).is_some());
|
||||
if v.is_empty() {
|
||||
if let Some(a) = self.registry.any_id() {
|
||||
v.push(a);
|
||||
}
|
||||
}
|
||||
v
|
||||
}
|
||||
}
|
||||
|
||||
pub struct AppState {
|
||||
pub live: ArcSwap<Live>,
|
||||
pub stats: Arc<Stats>,
|
||||
pub promotion: Arc<crate::promotion::Promotion>,
|
||||
pub imports: Arc<crate::import::JobStore>,
|
||||
/// Managed local CMF model server (install/update/spawn lifecycle + status).
|
||||
pub cmf: Arc<crate::cmf_runtime::CmfRuntime>,
|
||||
pub cache: Arc<SemanticCache>,
|
||||
pub secrets: SecretStore,
|
||||
pub config_path: String,
|
||||
pub pipeline: crate::pipeline::Pipeline,
|
||||
/// Outcome of the most recent router call (survives config reloads) —
|
||||
/// lets the admin panel distinguish a bad/expired key from a down router.
|
||||
pub router_status: RouterLastStatus,
|
||||
}
|
||||
|
||||
/// Map the user-facing `[shadow]` config to the promotion-table tuning.
|
||||
fn promotion_cfg(s: &crate::config::ShadowCfg) -> crate::promotion::PromotionCfg {
|
||||
crate::promotion::PromotionCfg {
|
||||
enabled: s.enabled && !s.local_model_id.is_empty(),
|
||||
window: s.window,
|
||||
n_min: s.n_min,
|
||||
promote_lb: s.promote_lb,
|
||||
soak: s.soak,
|
||||
file: if s.file.is_empty() { None } else { Some(s.file.clone()) },
|
||||
..Default::default()
|
||||
}
|
||||
}
|
||||
|
||||
impl std::ops::Deref for SharedState {
|
||||
@@ -61,15 +107,23 @@ impl SharedState {
|
||||
let secrets_path = sibling(&config_path, "secrets.toml");
|
||||
let secrets = SecretStore::load(&secrets_path);
|
||||
let stats = Stats::new(&cfg.stats);
|
||||
let promotion = crate::promotion::Promotion::new(promotion_cfg(&cfg.shadow));
|
||||
let imports = crate::import::JobStore::new();
|
||||
let cmf = crate::cmf_runtime::CmfRuntime::new();
|
||||
let cache = SemanticCache::new(&cfg.cache);
|
||||
let live = Live::build(cfg, &secrets)?;
|
||||
let router_status = RouterLastStatus::default();
|
||||
let live = Live::build(cfg, &secrets, router_status.clone())?;
|
||||
Ok(SharedState(Arc::new(AppState {
|
||||
live: ArcSwap::from_pointee(live),
|
||||
stats,
|
||||
promotion,
|
||||
imports,
|
||||
cmf,
|
||||
cache,
|
||||
secrets,
|
||||
config_path,
|
||||
pipeline: crate::pipeline::Pipeline::new(),
|
||||
router_status,
|
||||
})))
|
||||
}
|
||||
|
||||
@@ -84,7 +138,7 @@ impl AppState {
|
||||
/// If any step fails, the active state and file are left unchanged.
|
||||
pub fn reload(&self, new_cfg: Config) -> anyhow::Result<()> {
|
||||
new_cfg.validate()?;
|
||||
let live = Live::build(new_cfg.clone(), &self.secrets)?;
|
||||
let live = Live::build(new_cfg.clone(), &self.secrets, self.router_status.clone())?;
|
||||
new_cfg.save(&self.config_path)?;
|
||||
self.live.store(Arc::new(live));
|
||||
Ok(())
|
||||
@@ -94,7 +148,7 @@ impl AppState {
|
||||
/// (e.g. after a secret change — only provider keys changed).
|
||||
pub fn rebuild(&self) -> anyhow::Result<()> {
|
||||
let cfg = self.live.load_full().cfg.clone();
|
||||
let live = Live::build(cfg, &self.secrets)?;
|
||||
let live = Live::build(cfg, &self.secrets, self.router_status.clone())?;
|
||||
self.live.store(Arc::new(live));
|
||||
Ok(())
|
||||
}
|
||||
|
||||
Vendored
+76
@@ -28,12 +28,14 @@ export default {
|
||||
// nav
|
||||
"nav.dashboard": "Dashboard",
|
||||
"nav.models": "Models",
|
||||
"nav.import": "Import model",
|
||||
"nav.routing": "Routing",
|
||||
"nav.protocols": "Protocols",
|
||||
"nav.keys": "API keys",
|
||||
"nav.playground": "Playground",
|
||||
"nav.settings": "Settings",
|
||||
"nav.foot": "Gateway · v{v}",
|
||||
"nav.billing": "Router key & billing",
|
||||
|
||||
// topbar / status
|
||||
"status.router": "Router",
|
||||
@@ -41,6 +43,7 @@ export default {
|
||||
"status.online": "online",
|
||||
"status.offline": "offline",
|
||||
"status.checking": "checking…",
|
||||
"status.noKey": "no key",
|
||||
"theme.toggle": "Theme",
|
||||
"logout": "Log out",
|
||||
|
||||
@@ -89,6 +92,7 @@ export default {
|
||||
"dash.health.title": "Health",
|
||||
"dash.health.reachable": "reachable",
|
||||
"dash.health.unreachable": "unreachable",
|
||||
"dash.health.getKey": "Get a router key",
|
||||
|
||||
// models
|
||||
"models.title": "Models",
|
||||
@@ -225,7 +229,37 @@ export default {
|
||||
"settings.router.keyEnv": "Router key env name",
|
||||
"settings.router.timeout": "Timeout (ms)",
|
||||
"settings.router.verifyTls": "Verify TLS certificate",
|
||||
"settings.router.enabled": "Use the router (off = local models only)",
|
||||
"settings.cmf": "Local models (CMF)",
|
||||
"settings.cmf.localOnly": "Use only local models",
|
||||
"settings.cmf.manage": "Run a local model server",
|
||||
"settings.cmf.autoInstall": "Auto-install cortiq (crates.io)",
|
||||
"settings.cmf.autoUpdate": "Auto-update cortiq",
|
||||
"settings.cmf.model": "Local .cmf model path",
|
||||
"settings.cmf.port": "Local server port",
|
||||
"settings.cmf.modelId": "Model id in the pool",
|
||||
"settings.cmf.note": "Serves a local .cmf via cortiq (github.com/infosave2007/cmf). Works even when the router is unavailable.",
|
||||
"settings.cmf.threads": "Worker threads",
|
||||
"settings.cmf.gpu": "Use GPU (Metal / wgpu)",
|
||||
"settings.cmf.speedNote": "Threads parallelise the decode matvecs (~3x faster; 0 = runtime default). GPU offload mainly helps prefill of long prompts — decode is memory-bandwidth-bound, so on Apple Silicon it is roughly neutral.",
|
||||
"settings.router.taxonomy": "Taxonomy id",
|
||||
"settings.router.getKey": "Get or top up a router key at",
|
||||
"settings.router.secret": "Router key (value)",
|
||||
"settings.router.saveKey": "Save key",
|
||||
"settings.router.keySaved": "Router key saved",
|
||||
"settings.router.keyEmpty": "Paste the key first",
|
||||
"settings.router.secretHelp": "Paste the key from your api.allaigate.com email, then Save key. This is NOT a “Keys” value — those are for clients calling your gateway.",
|
||||
"settings.router.keyStoredMask": "•••••••••••• — key saved (paste again to replace)",
|
||||
"settings.router.test": "Test router",
|
||||
"router.probe.ok": "Router OK ({ms} ms)",
|
||||
"router.probe.noKey": "Router key is not set",
|
||||
"router.probe.auth": "Key rejected — invalid or revoked",
|
||||
"router.probe.payment": "Payment required — plan expired or quota exhausted",
|
||||
"router.probe.quota": "Rate limit exceeded",
|
||||
"router.probe.timeout": "Router timed out",
|
||||
"router.probe.unreachable": "Router unreachable",
|
||||
"router.probe.error": "Router error",
|
||||
"router.usage": "Usage",
|
||||
"settings.route": "Routing input",
|
||||
"settings.route.strategy": "Text strategy",
|
||||
"settings.route.maxChars": "Max chars",
|
||||
@@ -266,4 +300,46 @@ export default {
|
||||
"settings.cache.ttl": "Entry TTL",
|
||||
"settings.cache.embed": "Embedding model",
|
||||
"settings.cache.note": "Cache settings take effect after a restart.",
|
||||
|
||||
// import (HuggingFace → CMF)
|
||||
"import.subtitle": "Search HuggingFace, convert to a local .cmf, and serve it.",
|
||||
"import.searchPlaceholder": "Search HuggingFace models (e.g. qwen3, llama)…",
|
||||
"import.searching": "Searching…",
|
||||
"import.noResults": "No models found.",
|
||||
"import.configure": "Configure conversion",
|
||||
"import.outName": "Output name",
|
||||
"import.outNameHint": "The .cmf file name (letters, digits, - and _).",
|
||||
"import.quant": "Quantization",
|
||||
"import.convert": "Convert & load",
|
||||
"import.converting": "Converting",
|
||||
"import.started": "Conversion started",
|
||||
"import.failed": "Conversion failed — see the log.",
|
||||
"import.register": "Register as local model",
|
||||
"import.registered": "Registered as '{id}'. Run cortiq-server to serve it.",
|
||||
"import.jobs": "Conversions",
|
||||
"import.cancel": "Cancel",
|
||||
"import.cancelling": "Cancelling — cleaning up partial output…",
|
||||
"import.delete": "Delete",
|
||||
"import.deleteConfirm": "Delete this model and its converted .cmf file from disk?",
|
||||
"import.deleted": "Deleted with its files",
|
||||
"import.state.running": "running",
|
||||
"import.state.done": "done",
|
||||
"import.state.error": "error",
|
||||
"import.state.cancelled": "cancelled",
|
||||
"import.q.q8_2f": "8-bit two-field (𝒲×θ) — recommended: best quality/size.",
|
||||
"import.q.q8_row": "8-bit per-row — simple, robust.",
|
||||
"import.q.q4": "4-bit block — smallest, lower quality.",
|
||||
"import.q.vbit": "Variable 3–8 bit — per-expert bit budgets.",
|
||||
"import.q.f16": "16-bit — no quantization.",
|
||||
"import.q.f32": "32-bit — full precision (large file).",
|
||||
"import.adv.title": "Advanced settings",
|
||||
"import.adv.auto": "auto",
|
||||
"import.adv.linearCore": "Linear core",
|
||||
"import.adv.linearHint": "How linear-attention layers are carried/folded.",
|
||||
"import.adv.nphase": "vmf_phase phases/head",
|
||||
"import.adv.vbitShape": "VBIT budget shape",
|
||||
"import.adv.meanBits": "VBIT mean bits",
|
||||
"import.adv.shard": "Shard max (GB)",
|
||||
"import.adv.noShard": "no sharding",
|
||||
"import.adv.skipMtp": "skip MTP head",
|
||||
};
|
||||
|
||||
Vendored
+76
@@ -27,12 +27,14 @@ export default {
|
||||
// nav
|
||||
"nav.dashboard": "Дашборд",
|
||||
"nav.models": "Модели",
|
||||
"nav.import": "Импорт модели",
|
||||
"nav.routing": "Роутинг",
|
||||
"nav.protocols": "Протоколы",
|
||||
"nav.keys": "API-ключи",
|
||||
"nav.playground": "Playground",
|
||||
"nav.settings": "Настройки",
|
||||
"nav.foot": "Шлюз · v{v}",
|
||||
"nav.billing": "Ключ роутера и оплата",
|
||||
|
||||
// topbar / status
|
||||
"status.router": "Роутер",
|
||||
@@ -40,6 +42,7 @@ export default {
|
||||
"status.online": "в сети",
|
||||
"status.offline": "недоступен",
|
||||
"status.checking": "проверка…",
|
||||
"status.noKey": "нет ключа",
|
||||
"theme.toggle": "Тема",
|
||||
"logout": "Выйти",
|
||||
|
||||
@@ -88,6 +91,7 @@ export default {
|
||||
"dash.health.title": "Состояние",
|
||||
"dash.health.reachable": "доступен",
|
||||
"dash.health.unreachable": "недоступен",
|
||||
"dash.health.getKey": "Получить ключ роутера",
|
||||
|
||||
// models
|
||||
"models.title": "Модели",
|
||||
@@ -224,7 +228,37 @@ export default {
|
||||
"settings.router.keyEnv": "Имя env-переменной с ключом роутера",
|
||||
"settings.router.timeout": "Таймаут (мс)",
|
||||
"settings.router.verifyTls": "Проверять TLS-сертификат",
|
||||
"settings.router.enabled": "Использовать роутер (выкл = только локальные модели)",
|
||||
"settings.cmf": "Локальные модели (CMF)",
|
||||
"settings.cmf.localOnly": "Использовать только локальные модели",
|
||||
"settings.cmf.manage": "Запускать локальный сервер моделей",
|
||||
"settings.cmf.autoInstall": "Автоустановка cortiq (crates.io)",
|
||||
"settings.cmf.autoUpdate": "Автообновление cortiq",
|
||||
"settings.cmf.model": "Путь к локальной .cmf-модели",
|
||||
"settings.cmf.port": "Порт локального сервера",
|
||||
"settings.cmf.modelId": "ID модели в пуле",
|
||||
"settings.cmf.note": "Обслуживает локальный .cmf через cortiq (github.com/infosave2007/cmf). Работает даже когда роутер недоступен.",
|
||||
"settings.cmf.threads": "Рабочие потоки",
|
||||
"settings.cmf.gpu": "Использовать GPU (Metal / wgpu)",
|
||||
"settings.cmf.speedNote": "Потоки распараллеливают декод-матвеки (~×3; 0 = по умолчанию рантайма). GPU-оффлоад помогает в основном на prefill длинных промптов — декод упирается в пропускную способность памяти, поэтому на Apple Silicon почти нейтрален.",
|
||||
"settings.router.taxonomy": "ID таксономии",
|
||||
"settings.router.getKey": "Получить или пополнить ключ роутера:",
|
||||
"settings.router.secret": "Ключ роутера (значение)",
|
||||
"settings.router.saveKey": "Сохранить ключ",
|
||||
"settings.router.keySaved": "Ключ роутера сохранён",
|
||||
"settings.router.keyEmpty": "Сначала вставьте ключ",
|
||||
"settings.router.secretHelp": "Вставьте ключ из письма с api.allaigate.com и нажмите «Сохранить ключ». Это НЕ ключ из раздела «Ключи» — те выдаются клиентам для доступа к вашему шлюзу.",
|
||||
"settings.router.keyStoredMask": "•••••••••••• — ключ сохранён (вставьте снова, чтобы заменить)",
|
||||
"settings.router.test": "Проверить роутер",
|
||||
"router.probe.ok": "Роутер работает ({ms} мс)",
|
||||
"router.probe.noKey": "Ключ роутера не задан",
|
||||
"router.probe.auth": "Ключ отклонён — недействителен или отозван",
|
||||
"router.probe.payment": "Требуется оплата — тариф истёк или квота исчерпана",
|
||||
"router.probe.quota": "Превышен лимит запросов",
|
||||
"router.probe.timeout": "Роутер не ответил вовремя",
|
||||
"router.probe.unreachable": "Роутер недоступен",
|
||||
"router.probe.error": "Ошибка роутера",
|
||||
"router.usage": "Использование",
|
||||
"settings.route": "Входные данные роутинга",
|
||||
"settings.route.strategy": "Стратегия текста",
|
||||
"settings.route.maxChars": "Макс. символов",
|
||||
@@ -265,4 +299,46 @@ export default {
|
||||
"settings.cache.ttl": "TTL записи",
|
||||
"settings.cache.embed": "Модель эмбеддингов",
|
||||
"settings.cache.note": "Настройки кэша применяются после перезапуска.",
|
||||
|
||||
// импорт (HuggingFace → CMF)
|
||||
"import.subtitle": "Найдите модель на HuggingFace, сконвертируйте в локальный .cmf и подключите.",
|
||||
"import.searchPlaceholder": "Поиск моделей на HuggingFace (напр. qwen3, llama)…",
|
||||
"import.searching": "Ищу…",
|
||||
"import.noResults": "Модели не найдены.",
|
||||
"import.configure": "Настройка конвертации",
|
||||
"import.outName": "Имя файла",
|
||||
"import.outNameHint": "Имя .cmf (буквы, цифры, - и _).",
|
||||
"import.quant": "Квантизация",
|
||||
"import.convert": "Сконвертировать и загрузить",
|
||||
"import.converting": "Конвертация",
|
||||
"import.started": "Конвертация запущена",
|
||||
"import.failed": "Конвертация не удалась — смотрите лог.",
|
||||
"import.register": "Подключить как локальную модель",
|
||||
"import.registered": "Подключено как '{id}'. Запустите cortiq-server для обслуживания.",
|
||||
"import.jobs": "Конвертации",
|
||||
"import.cancel": "Отменить",
|
||||
"import.cancelling": "Отмена — удаляю недописанный файл…",
|
||||
"import.delete": "Удалить",
|
||||
"import.deleteConfirm": "Удалить эту модель и сконвертированный .cmf-файл с диска?",
|
||||
"import.deleted": "Удалено вместе с файлами",
|
||||
"import.state.running": "идёт",
|
||||
"import.state.done": "готово",
|
||||
"import.state.error": "ошибка",
|
||||
"import.state.cancelled": "отменено",
|
||||
"import.q.q8_2f": "8-бит двухполевой (𝒲×θ) — рекомендуется: лучшее качество/размер.",
|
||||
"import.q.q8_row": "8-бит построчно — просто, надёжно.",
|
||||
"import.q.q4": "4-бит блочно — компактнее, ниже качество.",
|
||||
"import.q.vbit": "Переменная 3–8 бит — per-expert бюджеты.",
|
||||
"import.q.f16": "16-бит — без квантования.",
|
||||
"import.q.f32": "32-бит — полная точность (большой файл).",
|
||||
"import.adv.title": "Расширенные настройки",
|
||||
"import.adv.auto": "авто",
|
||||
"import.adv.linearCore": "Линейное ядро",
|
||||
"import.adv.linearHint": "Как переносятся/сворачиваются линейные слои внимания.",
|
||||
"import.adv.nphase": "vmf_phase фаз/голову",
|
||||
"import.adv.vbitShape": "Форма бюджета VBIT",
|
||||
"import.adv.meanBits": "VBIT средних бит",
|
||||
"import.adv.shard": "Шард макс (ГБ)",
|
||||
"import.adv.noShard": "без шардинга",
|
||||
"import.adv.skipMtp": "пропустить MTP-голову",
|
||||
};
|
||||
|
||||
Vendored
+184
-3
@@ -2,7 +2,7 @@
|
||||
import { h, mount, toast } from "../ui.js";
|
||||
import { t } from "../i18n.js";
|
||||
import { api } from "../api.js";
|
||||
import { appState } from "../app.js";
|
||||
import { appState, SITE_URL } from "../app.js";
|
||||
|
||||
function field(label, control, hint) {
|
||||
return h("label", { class: "field" }, h("span", {}, label), control, hint ? h("div", { class: "hint" }, hint) : null);
|
||||
@@ -12,9 +12,45 @@ function check(label, checked) {
|
||||
return { node: h("label", { class: "check" }, cb, label), cb };
|
||||
}
|
||||
|
||||
// Flatten router usage (nested account/usage objects) into readable, wrapping
|
||||
// key: value rows — a raw JSON.stringify dump overflows the card horizontally.
|
||||
function usageRows(obj, prefix) {
|
||||
const rows = [];
|
||||
for (const [k, v] of Object.entries(obj || {})) {
|
||||
const key = prefix ? prefix + "." + k : k;
|
||||
if (v && typeof v === "object" && !Array.isArray(v)) {
|
||||
rows.push(...usageRows(v, key));
|
||||
} else {
|
||||
rows.push(
|
||||
h("div", { class: "mono kv" },
|
||||
h("span", { class: "kv-k" }, key + ": "),
|
||||
h("span", { class: "kv-v" }, Array.isArray(v) ? JSON.stringify(v) : String(v)))
|
||||
);
|
||||
}
|
||||
}
|
||||
return rows;
|
||||
}
|
||||
|
||||
// Probe status → i18n key suffix (see /admin/api/router/probe).
|
||||
function probeMsgKey(status) {
|
||||
const map = {
|
||||
ok: "ok",
|
||||
no_key: "noKey",
|
||||
auth: "auth",
|
||||
payment: "payment",
|
||||
quota: "quota",
|
||||
timeout: "timeout",
|
||||
unreachable: "unreachable",
|
||||
};
|
||||
return "router.probe." + (map[status] || "error");
|
||||
}
|
||||
|
||||
// Statuses where the fix is on the site (get / pay for a key).
|
||||
const PROBE_SITE_STATUSES = ["no_key", "auth", "payment", "quota"];
|
||||
|
||||
export async function renderSettings() {
|
||||
const meta = appState.meta || (await api.meta());
|
||||
const s = await api.getSettings();
|
||||
const [s, secretsData] = await Promise.all([api.getSettings(), api.listSecrets()]);
|
||||
const root = h("div");
|
||||
|
||||
const listenIn = h("input", { value: s.listen || "" });
|
||||
@@ -24,6 +60,101 @@ export async function renderSettings() {
|
||||
const rTimeout = h("input", { type: "number", value: s.router?.timeout_ms ?? 800 });
|
||||
const rTax = h("input", { value: s.router?.taxonomy_id || "" });
|
||||
const rVerify = check(t("settings.router.verifyTls"), s.router?.verify_tls);
|
||||
const rEnabled = check(t("settings.router.enabled"), s.router?.enabled !== false);
|
||||
// router key value (write-only; stored in the secret store, like model keys)
|
||||
const rKeySource =
|
||||
(secretsData.secrets || []).find((x) => x.name === (s.router?.api_key_env || "CORTIQ_ROUTER_KEY"))?.source ||
|
||||
"missing";
|
||||
const keyStored = rKeySource === "store" || rKeySource === "env";
|
||||
// When a key is already saved, show a masked "saved" hint instead of an empty
|
||||
// field (an empty field reads as "nothing is set").
|
||||
const rSecret = h("input", {
|
||||
type: "password",
|
||||
placeholder: keyStored ? t("settings.router.keyStoredMask") : t("models.form.secretPlaceholder"),
|
||||
});
|
||||
const rKeyBadge = h(
|
||||
"span",
|
||||
{ class: "badge " + (rKeySource === "store" ? "store" : rKeySource === "env" ? "env" : "bad") },
|
||||
rKeySource === "store"
|
||||
? t("models.form.secretStored")
|
||||
: rKeySource === "env"
|
||||
? t("models.form.secretEnv")
|
||||
: t("models.form.secretMissing")
|
||||
);
|
||||
const rSecretHint = h(
|
||||
"div",
|
||||
{},
|
||||
h(
|
||||
"div",
|
||||
{},
|
||||
rKeyBadge,
|
||||
" · ",
|
||||
t("settings.router.getKey") + " ",
|
||||
h("a", { href: SITE_URL, target: "_blank", rel: "noopener" }, "api.allaigate.com")
|
||||
),
|
||||
h("div", { class: "muted", style: "margin-top:4px" }, t("settings.router.secretHelp"))
|
||||
);
|
||||
// The key lives in the secret store under this env name (config default: CORTIQ_ROUTER_KEY).
|
||||
const keyName = () => rKeyEnv.value.trim() || "CORTIQ_ROUTER_KEY";
|
||||
// Persist just the pasted router key (independent of the big Save at page top).
|
||||
async function saveKey() {
|
||||
const v = rSecret.value.trim();
|
||||
if (!v) return false;
|
||||
if (!rKeyEnv.value.trim()) rKeyEnv.value = "CORTIQ_ROUTER_KEY";
|
||||
await api.setSecret(keyName(), v);
|
||||
rSecret.value = "";
|
||||
rSecret.placeholder = t("settings.router.keyStoredMask"); // show "saved", not empty
|
||||
rKeyBadge.textContent = t("models.form.secretStored");
|
||||
rKeyBadge.className = "badge store";
|
||||
return true;
|
||||
}
|
||||
const saveKeyBtn = h("button", { class: "btn sm" }, "✓ " + t("settings.router.saveKey"));
|
||||
saveKeyBtn.addEventListener("click", async () => {
|
||||
saveKeyBtn.disabled = true;
|
||||
try {
|
||||
if (await saveKey()) toast(t("settings.router.keySaved"), "good");
|
||||
else toast(t("settings.router.keyEmpty"), "warn");
|
||||
} catch (e) {
|
||||
toast(String(e.message || e), "bad");
|
||||
}
|
||||
saveKeyBtn.disabled = false;
|
||||
});
|
||||
// "Test router" — real /v1/route call: tells key problems apart from a down router
|
||||
const probeBtn = h("button", { class: "btn" }, t("settings.router.test"));
|
||||
const probeOut = h("div", { class: "hint", style: "margin-top:8px" });
|
||||
probeBtn.addEventListener("click", async () => {
|
||||
probeBtn.disabled = true;
|
||||
mount(probeOut, h("span", { class: "spinner" }));
|
||||
try {
|
||||
// If a key was just pasted but not saved, persist it first so the test uses it.
|
||||
if (rSecret.value.trim()) {
|
||||
try {
|
||||
await saveKey();
|
||||
} catch (_) {
|
||||
/* fall through — probe will report no_key/auth */
|
||||
}
|
||||
}
|
||||
const r = await api.probeRouter();
|
||||
const parts = [t(probeMsgKey(r.status), { ms: r.latency_ms ?? "—" })];
|
||||
if (r.message) parts.push(" — " + r.message);
|
||||
const line = h("span", {}, ...parts);
|
||||
const out = [line];
|
||||
if (PROBE_SITE_STATUSES.includes(r.status)) {
|
||||
out.push(" · ", h("a", { href: SITE_URL, target: "_blank", rel: "noopener" }, t("dash.health.getKey") + " ↗"));
|
||||
}
|
||||
if (r.usage && typeof r.usage === "object") {
|
||||
out.push(
|
||||
h("div", { class: "usage-box", style: "margin-top:4px" },
|
||||
h("b", {}, t("router.usage")), ...usageRows(r.usage))
|
||||
);
|
||||
}
|
||||
mount(probeOut, h("div", {}, h("span", { class: "badge " + (r.ok ? "ok" : "bad") }, r.ok ? "OK" : "×"), " ", ...out));
|
||||
toast(t(probeMsgKey(r.status), { ms: r.latency_ms ?? "—" }), r.ok ? "good" : "bad");
|
||||
} catch (e) {
|
||||
mount(probeOut, String(e.message || e));
|
||||
}
|
||||
probeBtn.disabled = false;
|
||||
});
|
||||
// route
|
||||
const strategySel = h("select", {}, ...(meta.text_strategies || []).map((x) => h("option", { value: x, selected: x === s.route?.text_strategy }, x)));
|
||||
const maxChars = h("input", { type: "number", value: s.route?.max_chars ?? 4000 });
|
||||
@@ -54,13 +185,26 @@ export async function renderSettings() {
|
||||
h("option", { value: "" }, "auto"),
|
||||
...embedModels.map((id) => h("option", { value: id, selected: id === s.cache?.embed_model }, id))
|
||||
);
|
||||
// local models (CMF) — run a local .cmf server and/or force local-only routing
|
||||
const cmfLocalOnly = check(t("settings.cmf.localOnly"), s.cmf?.local_only);
|
||||
const cmfManage = check(t("settings.cmf.manage"), s.cmf?.manage_server);
|
||||
const cmfAutoInstall = check(t("settings.cmf.autoInstall"), s.cmf?.auto_install);
|
||||
const cmfAutoUpdate = check(t("settings.cmf.autoUpdate"), s.cmf?.auto_update);
|
||||
const cmfModel = h("input", { value: s.cmf?.local_model || "", placeholder: "models/model.cmf" });
|
||||
const cmfPort = h("input", { type: "number", value: s.cmf?.local_port ?? 8081 });
|
||||
const cmfModelId = h("input", { value: s.cmf?.model_id || "cmf-local" });
|
||||
const cmfThreads = h("input", { type: "number", min: 0, value: s.cmf?.threads ?? 8 });
|
||||
const cmfGpu = check(t("settings.cmf.gpu"), s.cmf?.gpu);
|
||||
|
||||
const saveBtn = h("button", { class: "btn primary" }, t("common.save"));
|
||||
saveBtn.addEventListener("click", async () => {
|
||||
saveBtn.disabled = true;
|
||||
// a pasted router key needs an env name to live under — default it
|
||||
if (rSecret.value.trim() && !rKeyEnv.value.trim()) rKeyEnv.value = "CORTIQ_ROUTER_KEY";
|
||||
const patch = {
|
||||
listen: listenIn.value.trim(),
|
||||
router: {
|
||||
enabled: rEnabled.cb.checked,
|
||||
url: rUrl.value.trim(),
|
||||
api_key_env: rKeyEnv.value.trim() || null,
|
||||
timeout_ms: parseInt(rTimeout.value) || 800,
|
||||
@@ -90,6 +234,18 @@ export async function renderSettings() {
|
||||
max_entries: s.cache?.max_entries ?? 1000,
|
||||
embed_model: cacheEmbedSel.value || null,
|
||||
},
|
||||
cmf: {
|
||||
...(s.cmf || {}),
|
||||
local_only: cmfLocalOnly.cb.checked,
|
||||
manage_server: cmfManage.cb.checked,
|
||||
auto_install: cmfAutoInstall.cb.checked,
|
||||
auto_update: cmfAutoUpdate.cb.checked,
|
||||
local_model: cmfModel.value.trim(),
|
||||
local_port: parseInt(cmfPort.value) || 8081,
|
||||
model_id: cmfModelId.value.trim() || "cmf-local",
|
||||
threads: parseInt(cmfThreads.value) || 0,
|
||||
gpu: cmfGpu.cb.checked,
|
||||
},
|
||||
};
|
||||
// serde: drop null optionals
|
||||
if (!patch.router.api_key_env) delete patch.router.api_key_env;
|
||||
@@ -98,6 +254,13 @@ export async function renderSettings() {
|
||||
if (!patch.cache.embed_model) delete patch.cache.embed_model;
|
||||
try {
|
||||
const r = await api.putSettings(patch);
|
||||
const secretVal = rSecret.value.trim();
|
||||
if (secretVal) {
|
||||
await api.setSecret(rKeyEnv.value.trim(), secretVal);
|
||||
rSecret.value = "";
|
||||
rKeyBadge.textContent = t("models.form.secretStored");
|
||||
rKeyBadge.className = "badge store";
|
||||
}
|
||||
toast(r.needs_restart ? t("settings.needsRestart") : t("toast.saved"), r.needs_restart ? "warn" : "good");
|
||||
} catch (e) {
|
||||
toast(String(e.message || e), "bad");
|
||||
@@ -150,8 +313,13 @@ export async function renderSettings() {
|
||||
h("div", { class: "card-head" }, h("h3", {}, t("settings.router"))),
|
||||
field(t("settings.router.url"), rUrl),
|
||||
field(t("settings.router.keyEnv"), rKeyEnv),
|
||||
field(t("settings.router.secret"), h("div", { class: "key-row" }, rSecret, saveKeyBtn), rSecretHint),
|
||||
h("div", { class: "row" }, field(t("settings.router.timeout"), rTimeout), field(t("settings.router.taxonomy"), rTax)),
|
||||
rVerify.node
|
||||
rEnabled.node,
|
||||
rVerify.node,
|
||||
h("div", { class: "divider" }),
|
||||
h("div", { class: "flex" }, probeBtn),
|
||||
probeOut
|
||||
),
|
||||
h(
|
||||
"div",
|
||||
@@ -199,6 +367,19 @@ export async function renderSettings() {
|
||||
h("div", { class: "row" }, field(t("settings.cache.threshold"), cacheThresh), field(t("settings.cache.ttl"), cacheTtlIn)),
|
||||
field(t("settings.cache.embed"), cacheEmbedSel),
|
||||
h("div", { class: "hint" }, t("settings.cache.note"))
|
||||
),
|
||||
h(
|
||||
"div",
|
||||
{ class: "card" },
|
||||
h("div", { class: "card-head" }, h("h3", {}, t("settings.cmf"))),
|
||||
cmfLocalOnly.node,
|
||||
cmfManage.node,
|
||||
h("div", { class: "row" }, cmfAutoInstall.node, cmfAutoUpdate.node),
|
||||
field(t("settings.cmf.model"), cmfModel),
|
||||
h("div", { class: "row" }, field(t("settings.cmf.port"), cmfPort), field(t("settings.cmf.modelId"), cmfModelId)),
|
||||
h("div", { class: "row" }, field(t("settings.cmf.threads"), cmfThreads), cmfGpu.node),
|
||||
h("div", { class: "hint" }, t("settings.cmf.note")),
|
||||
h("div", { class: "hint" }, t("settings.cmf.speedNote"))
|
||||
)
|
||||
),
|
||||
h(
|
||||
|
||||
Reference in New Issue
Block a user