Compare commits

...
21 Commits
Author SHA1 Message Date
infosave2007 9d2181e82b chore(release): v0.2.7 — honest pins, visible managed model, physical delete
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
- A pinned playground model that is unavailable now errors clearly instead of
  silently answering from the managed local model (route_source no longer lies).
- The managed CMF model is surfaced in GET /admin/api/models (marked "managed",
  shown read-only in the UI) so the real running model is visible and pinnable
  without a hand-maintained [[models]] entry; a fresh install still lists nothing.
- Deleting a model now removes its local .cmf physically (file or tensor-dir);
  deleting the managed model also stops the server and clears the [cmf] config.
  Routing is stripped of the deleted id so the config still validates.
2026-07-08 21:14:03 +03:00
infosave2007 30378f7b6a chore: fix clippy lints to green the lint CI
- import.rs / promotion.rs: sort_by(|a,b| b.k.cmp(&a.k)) -> sort_by_key(Reverse(k))
- pipeline.rs: skip_while(!p).next() -> find(p)
Behaviour-preserving; completes the CI fix started by the cargo fmt commit.
2026-07-08 20:09:54 +03:00
infosave2007 b4f78bd192 chore: cargo fmt — fix lint 'Format' CI job (red since v0.2.3)
Whole-repo rustfmt drift; behaviour unchanged (whitespace/layout only).
All test platforms already passed; this greens the lint job.
2026-07-08 19:32:35 +03:00
infosave2007 3c90dad9b9 chore(release): v0.2.6 — Settings UX, empty pool by default, CMF port check
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
- Settings: "?" help tooltips on every option; router key auto-probes the
  connection right after saving (no more hanging "unavailable")
- Local models card reworked into 3 clear steps (install cortiq → add model →
  run locally); removed the hand-entered .cmf path / pool-id fields — models
  are added via the Import tab and the managed server registers them
- Fresh install ships an EMPTY model pool; the local .cmf is auto-registered by
  the managed CMF server, so it needs no static [[models]] entry
- New: port-availability check — GET /admin/api/cmf/port + "Check port" button
  with next-free-port suggestion. bind()+connect() probe so Docker-proxied /
  dual-stack ports (e.g. 8081) are correctly detected as occupied
- /admin/api/cmf now reports the serving model + port; card shows "▶ serving …"
- On-demand runtime install: POST /admin/api/cmf/install
2026-07-08 19:07:24 +03:00
infosave2007 25f93d1a27 chore: ship the full working frontend + backend
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
The crates.io/release build shipped a STALE frontend (committed web/js was behind
the working tree), so the admin console hung on load. Commit the current working
web/js (app, api, dashboard, login, import view, i18n de/es/fr/tr/zh, css) and the
backend WIP (import.rs, admin/assets.rs) so the released console matches the dev
build. Verified: clean `cargo build --release`, empty-pool start, working
dashboard served.
2026-07-08 17:58:31 +03:00
infosave2007 0f89cdaa07 chore(release): v0.2.4
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
2026-07-08 17:39:43 +03:00
infosave2007 d211eb6150 feat: fresh install starts with an empty model pool
- config.rs: validate() no longer errors on zero models — an empty pool is
  allowed (warns instead), so the admin console comes up on a fresh install and
  the user adds models in /admin -> Models (or points [cmf] at a local .cmf).
- gateway.example.toml: ships an EMPTY pool (no seed cmf-local/local-embed, empty
  routing), so `cp example -> gateway.toml && run` shows an empty list — what a
  new user expects.
2026-07-08 17:39:43 +03:00
infosave2007 04580196ad fix: add promotion + router-status modules referenced by the CMF integration
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
The v0.2.3 CMF-integration commit referenced crate::promotion (main.rs),
router_client::RouterLastStatus (state.rs) and router_client::classify_status
(admin), but those definitions were not part of the commit, so a clean build
failed (E0583/E0432/E0425). Add the two source files; cargo build --release now
compiles.
2026-07-08 16:59:58 +03:00
infosave2007 7a3d6a7bb4 chore(release): v0.2.3 2026-07-08 16:39:30 +03:00
infosave2007 07d011263d config: example ships a local CMF model, drop cloud placeholders
The example config pool now shows one local CMF model (served by the managed
cortiq server) instead of local-qwen/gpt-4o-mini/claude-opus placeholders, with
routing pointing at it, and a documented [cmf] section (manage_server, threads,
gpu, ...). Add cloud/local providers by copying the [[models]] shape.
2026-07-08 16:39:30 +03:00
infosave2007 0040efc948 admin: surface CMF worker threads + GPU toggle in Settings
Expose the two speed knobs in the /admin Settings panel instead of TOML-only:
- [cmf].gpu (new) -> exports CMF_GPU=1 to the managed cortiq serve (Metal/wgpu).
  On Apple Silicon decode is memory-bandwidth bound so this is ~neutral for
  generation (helps prefill); off by default.
- [cmf].threads (added earlier) now has a UI number input.

- config.rs: CmfCfg.gpu bool (serde default false).
- cmf_runtime.rs: spawn_serve sets CMF_GPU=1 when gpu is on.
- admin/mod.rs: put_settings flags needs_restart on threads/gpu change (the
  server is spawned once at startup with those env vars).
- settings.js: threads input + gpu checkbox + save payload + speed hint.
- i18n en/ru: threads/gpu/speedNote keys (other locales fall back to en, as the
  whole cmf section already does there).
2026-07-08 16:18:31 +03:00
infosave2007 16a3e209b6 cmf: parallelise the managed local server (CMF_THREADS)
The gateway spawned `cortiq serve` with no thread hint, so the runtime ran the
decode loop single-threaded (~0.25 tok/s on a 15B CPU model). Export CMF_THREADS
from a new [cmf].threads config field (default 8) in spawn_serve.

Measured: 0.25 -> 0.81 tok/s (~3.2x) on qwopus-vbit7 (15B). Scaling is sublinear
(GatedDeltaNet state recurrence is serial per token, Amdahl-bound); a real jump
needs the GPU path.
2026-07-08 16:18:31 +03:00
infosave2007 c19edee7bb feat(cmf): integrate CMF as a managed local model server
Run models in the CMF (.cmf) format locally, via the native `cortiq`
CLI from crates.io — no Python and no external model server.

- cmf_runtime: install/update `cortiq-cli` from crates.io, spawn and
  health-check `cortiq serve` as a managed local backend.
- import: model-import jobs shell out to the native `cortiq convert`
  (HF repo -> .cmf, quant mapped Q8_2F->q8_2f / Q8_ROW->q8 / Q4->q4);
  the Python converter is used only for advanced/exotic options.
- pipeline: local-first routing — serve from the local CMF model when
  the router is disabled/unreachable, with a "local only" hard switch.
- registry/state: register the local CMF server as an OpenAI provider
  and track local candidates.
- config: data-dir resolution + CmfCfg (auto-install/update, managed
  server, host/port/model) and relaxed validation for the managed model.
- admin + settings UI (en/ru): "Local models (CMF)" settings card.
2026-07-08 16:18:31 +03:00
infosave2007andClaude Opus 4.8 7af90a5eef docs: add docker-compose quickstart (gateway + Ollama) and GHCR badge
- docker-compose.yml + config/gateway.compose.toml (routes to a local Ollama service)
- README (EN+RU): ghcr badge + Docker Compose section

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 13:40:24 +03:00
infosave2007andClaude Opus 4.8 8d7e5fe653 chore(release): v0.2.2
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
Benchmarks (latency + accuracy), router docs + LAUNCH1 promo, Docker image on ghcr.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 11:28:56 +03:00
infosave2007andClaude Opus 4.8 2605eccb64 feat: Docker image + ghcr publishing
- multi-stage Dockerfile (non-root, embeds SPA, default config, ~160MB) + .dockerignore
- .github/workflows/docker.yml: push to ghcr.io/infosave2007/cortiq-gateway
  (master -> :edge/:sha amd64; tags -> :X.Y.Z/:latest, linux/amd64+arm64)
- README (EN+RU): docker run instructions; CHANGELOG
- verified locally: image builds, container serves /healthz + /admin

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 11:23:09 +03:00
infosave2007andClaude Opus 4.8 1317b8024b docs: document the allaigate router + launch promo
- docs/ROUTER.md + docs/ROUTER.ru.md: what the semantic router does, hosted vs
  self-hosted, config, API contract, pricing, and the LAUNCH1 $1 Starter promo
- README (EN+RU): "The router (allaigate)" section with the promo callout

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 10:35:29 +03:00
infosave2007andClaude Opus 4.8 9242806795 docs: add task-type accuracy benchmark (router vs keyword heuristic)
- bench/tasks.jsonl (37 natural-language prompts, 7 task types) + bench/accuracy.py
- measured on the live allaigate router: 100% (37/37) vs keyword heuristic 32%
- BENCHMARKS.md accuracy section; README (EN+RU) perf+accuracy tables and test matrix
- run.sh runs accuracy when CORTIQ_ROUTER_KEY is set

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 07:45:16 +03:00
infosave2007andClaude Opus 4.8 6714c4f556 docs: add test/verification matrix to README (EN+RU)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 18:31:13 +03:00
infosave2007andClaude Opus 4.8 a5966dd479 docs: add latency benchmark vs LiteLLM/Portkey
- bench/ reproducible harness: instant mock backend + Apache Bench (ab -k -r)
- measures gateway overhead in passthrough mode (apples-to-apples)
- BENCHMARKS.md: methodology, caveats, results (cortiq ~57k rps p99 1ms vs
  Portkey ~5.8k p99 9ms vs LiteLLM 4-workers ~1.2k p99 59ms)
- README (EN+RU): Performance section + highlight

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 18:11:26 +03:00
infosave2007andClaude Opus 4.8 e015555dff feat: finish remaining inbound protocols (models, completions, native /route)
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
- GET /v1/models: pool + virtual cortiq-auto
- POST /v1/completions (legacy): non-stream + streaming text_completion
- POST /route: native passthrough returning the routing decision
- all inbound protocol adapters now implemented; admin meta + README/CHANGELOG updated
- bump to 0.2.1

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 17:42:45 +03:00
50 changed files with 4741 additions and 155 deletions
+10
View File
@@ -0,0 +1,10 @@
target
.git
.github
docs
bench
config/gateway.toml
config/secrets.toml
config/stats.jsonl
*.log
.DS_Store
+47
View File
@@ -0,0 +1,47 @@
name: Docker
# Publishes container images to GitHub Container Registry (ghcr.io).
# master push -> :edge, :sha (amd64, fast)
# tag v* -> :X.Y.Z, :X.Y, :latest, :sha (linux/amd64 + linux/arm64)
on:
push:
branches: [master]
tags: ["v*"]
workflow_dispatch:
permissions:
contents: read
packages: write
jobs:
publish:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: docker/setup-qemu-action@v3
- uses: docker/setup-buildx-action@v3
- uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- id: meta
uses: docker/metadata-action@v5
with:
images: ghcr.io/${{ github.repository }}
tags: |
type=semver,pattern={{version}}
type=semver,pattern={{major}}.{{minor}}
type=raw,value=latest,enable=${{ startsWith(github.ref, 'refs/tags/v') }}
type=raw,value=edge,enable={{is_default_branch}}
type=sha,format=short
- uses: docker/build-push-action@v6
with:
context: .
# multi-arch only for release tags (arm64 under emulation is slow)
platforms: ${{ startsWith(github.ref, 'refs/tags/v') && 'linux/amd64,linux/arm64' || 'linux/amd64' }}
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha
cache-to: type=gha,mode=max
+109
View File
@@ -0,0 +1,109 @@
# Gateway latency benchmark
How much latency does the **gateway itself** add? For an LLM proxy the model call
dominates wall-clock time, but the gateway's own overhead is what differs between
products — and it compounds across the many calls an agent makes. This benchmark
isolates that overhead and compares Cortiq Gateway with **LiteLLM** and **Portkey**.
> Reproduce on your own hardware: `bash bench/run.sh` (see [Methodology](#methodology)).
## TL;DR
Every gateway proxies the **same instant mock backend** in passthrough mode, under
identical load (`ab -k -r -c 20 -n 5000`). Latency is in milliseconds.
| Gateway | Throughput (req/s) | p50 | p90 | p99 | mean |
|---|--:|--:|--:|--:|--:|
| **Cortiq Gateway** (Rust) | **~57,300** | 0 ms | 1 ms | **1 ms** | 0.35 ms |
| Portkey Gateway (Node) | ~5,800 | 3 ms | 5 ms | 9 ms | 3.5 ms |
| LiteLLM (Python, 4 workers) | ~1,200 | 9 ms | 37 ms | 59 ms | 16.6 ms |
| _mock-direct (no gateway)_ | _~29,000_ | _1 ms_ | _1 ms_ | _3 ms_ | _0.7 ms_ |
On this machine Cortiq adds **sub-millisecond p99** overhead and sustains **~10×**
the throughput of Portkey and **~47×** of LiteLLM. The gap is architectural: a Rust
/ Tokio gateway uses every core in one process, with no interpreter or GC in the hot
path.
> Absolute numbers depend on hardware; the **ratios** between gateways are the point.
> Run `bench/run.sh` to get numbers for your environment.
## Accuracy: task-type routing
Speed is only half the story. Cortiq's other edge is **routing by task type** — sending
each prompt to the right model — via the allaigate semantic classifier. LiteLLM and
Portkey have **no semantic task router**: you either pin a model, write brittle rules,
or pay for an extra LLM classification call.
On a 37-prompt, 7-task-type set of **natural-language prompts** (no explicit trigger
words like "translate" / "summarize" — [`bench/tasks.jsonl`](bench/tasks.jsonl)):
| Classifier | Correct | Accuracy |
|---|--:|--:|
| **allaigate semantic router** | 37 / 37 | **100%** |
| keyword heuristic (DIY without a classifier) | 12 / 37 | 32% |
A keyword matcher looks good on prompts that literally say "translate this" — but real
users don't. On natural phrasing (`"'Guten Tag' — what does that mean in English?"`,
`"boil this down to a few key points"`, `"which companies are named in this report?"`)
surface matching collapses while the semantic router still routes correctly.
- **Methodology.** [`bench/accuracy.py`](bench/accuracy.py) sends each prompt to the live
allaigate router (`taxonomy = data-assistant`) and compares the predicted `task_label`
to the ground truth; a keyword heuristic is scored on the same set as a lower bound for
"route without a classifier."
- **Caveats.** Small, hand-labeled set; phrasing is deliberately keyword-light to test
semantic understanding vs surface matching. Expand `tasks.jsonl` and re-run. allaigate's
own published figure is **94% at separating close task types** (a harder eval than this one).
- **Reproduce.** `CORTIQ_ROUTER_KEY=cortiq_… python3 bench/accuracy.py`
## Why this matters
Agentic workloads make **many** gateway calls per task (plan → sub-tasks → tool calls
→ synthesis). Overhead that looks small per call (tens of ms) multiplies across a run
and caps the throughput a single instance can serve. Lower, more predictable tail
latency (p99) is the developer-facing differentiator when the model time is equal.
## Methodology
- **Isolate gateway overhead.** A tiny, dependency-free Node mock ([`bench/mock.js`](bench/mock.js))
returns a fixed OpenAI-compatible completion **instantly**. So measured latency is
the gateway's own cost, not model time. Every gateway proxies this same mock.
- **Apples-to-apples.** Each gateway runs in **passthrough / pinned** mode (no routing
or model-selection logic) so we compare raw proxy cost. Cortiq's intelligent routing
(`cortiq-auto`) adds one classifier round-trip — that's a feature, measured separately,
not folded into this overhead number.
- **Load.** Apache Bench, keep-alive, tolerant of length variance:
`ab -k -r -c 20 -n 5000 -p bench/body.json -T application/json`.
- **Fair to each runtime.** Cortiq = one process (Tokio, multi-core). Portkey = one Node
process (event loop). LiteLLM = **4 uvicorn workers** (its recommended multi-process
setup; a single Python worker is GIL-bound to one core and scores ~860 req/s).
- **Same box, localhost**, warm processes, results are representative of one run.
### Caveats (read me)
- This measures **gateway overhead only**. Real requests are dominated by model latency,
which is identical regardless of gateway — but overhead and tail latency are not.
- Localhost + mock backend removes network/model variance on purpose. Your absolute
numbers will differ; reproduce with `bench/run.sh`.
- `ab` reports LiteLLM "Failed requests" as a non-zero count — these are **response
content-length variations, not HTTP errors** (all responses are `200`, verified).
- Tested versions: Cortiq Gateway (this repo, `--release`), LiteLLM proxy via
`uvicorn litellm.proxy.proxy_server:app --workers 4`, Portkey `@portkey-ai/gateway`,
Apache Bench. Hardware: Apple Silicon, macOS.
## Reproduce
```bash
# prerequisites: node, ab (Apache Bench), a release build of the gateway
cargo build --release
# optional competitors (the harness auto-detects them):
pip install 'litellm[proxy]' uvicorn
npm install -g @portkey-ai/gateway
# run (tune load via env)
CONC=20 REQ=5000 bash bench/run.sh
```
The harness ([`bench/run.sh`](bench/run.sh)) starts the mock, then benchmarks the mock
directly and each available gateway, printing a throughput / p50 / p90 / p99 table.
+32
View File
@@ -3,6 +3,38 @@
All notable changes to this project are documented here.
The format is based on [Keep a Changelog](https://keepachangelog.com/).
## [Unreleased]
### Added
- **Docker Compose** quickstart (`docker-compose.yml` + `config/gateway.compose.toml`):
gateway + a local Ollama model in one command. GHCR badge in the READMEs.
## [0.2.2] - 2026-07-01
### Added
- **Latency benchmark** vs LiteLLM and Portkey: a reproducible harness (`bench/`,
instant mock backend + Apache Bench) and `BENCHMARKS.md`. Measured gateway overhead:
Cortiq ~57k req/s / p99 1 ms vs Portkey ~5.8k / p99 9 ms vs LiteLLM ~1.2k / p99 59 ms.
- **Accuracy benchmark**: task-type routing on natural-language prompts (`bench/accuracy.py`,
`bench/tasks.jsonl`) — allaigate semantic router 100% (37/37) vs a keyword heuristic 32%
(competitors have no semantic task router).
- **Router docs** (`docs/ROUTER.md` + `.ru.md`): what the allaigate/cortiq-router does, how
to point the gateway at it, API contract, pricing, and the current `LAUNCH1` $1 promo.
- **Docker image**: multi-stage `Dockerfile` (non-root, ~160 MB) + `docker.yml` workflow
publishing to `ghcr.io/infosave2007/cortiq-gateway` (`:latest`, `:X.Y.Z`, `:edge`;
linux/amd64 + arm64 on release tags). `docker run` works out of the box.
## [0.2.1] - 2026-06-30
### Added
- **OpenAI Models** (`GET /v1/models`): lists the pool + the virtual `cortiq-auto`.
- **OpenAI Completions (legacy)** (`POST /v1/completions`): non-streaming and streaming
(`text_completion`), routed through the same pipeline.
- **Native passthrough** (`POST /route`): returns the routing decision (task, complexity,
candidate models) without calling a model.
All inbound protocol adapters are now implemented.
## [0.2.0] - 2026-06-30
### Added
Generated
+1 -1
View File
@@ -285,7 +285,7 @@ checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570"
[[package]]
name = "cortiq-gateway"
version = "0.2.0"
version = "0.2.7"
dependencies = [
"anyhow",
"arc-swap",
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "cortiq-gateway"
version = "0.2.0"
version = "0.2.7"
edition = "2021"
rust-version = "1.75"
license = "Apache-2.0"
+25
View File
@@ -0,0 +1,25 @@
# ---- build stage ------------------------------------------------------------
FROM rust:1-slim AS builder
WORKDIR /app
# the admin SPA (web/) is embedded at compile time via rust-embed, so it must be present
COPY Cargo.toml Cargo.lock ./
COPY src ./src
COPY web ./web
RUN cargo build --release
# ---- runtime stage ----------------------------------------------------------
FROM debian:bookworm-slim
RUN apt-get update \
&& apt-get install -y --no-install-recommends ca-certificates \
&& rm -rf /var/lib/apt/lists/* \
&& useradd -r -u 10001 cortiq \
&& mkdir -p /app/config
COPY --from=builder /app/target/release/cortiq-gateway /usr/local/bin/cortiq-gateway
# ship the example as the default config so `docker run` works out of the box
COPY config/gateway.example.toml /app/config/gateway.toml
RUN chown -R 10001:10001 /app
WORKDIR /app
USER cortiq
EXPOSE 9000
ENTRYPOINT ["cortiq-gateway"]
CMD ["--config", "/app/config/gateway.toml"]
+77 -5
View File
@@ -3,6 +3,7 @@
**English** · [Русский](https://github.com/infosave2007/cortiq-gateway/blob/master/README.ru.md)
[![Crates.io](https://img.shields.io/crates/v/cortiq-gateway.svg)](https://crates.io/crates/cortiq-gateway)
[![Docker](https://img.shields.io/badge/ghcr.io-cortiq--gateway-2496ED?logo=docker&logoColor=white)](https://github.com/infosave2007/cortiq-gateway/pkgs/container/cortiq-gateway)
[![CI](https://github.com/infosave2007/cortiq-gateway/actions/workflows/ci.yml/badge.svg)](https://github.com/infosave2007/cortiq-gateway/actions/workflows/ci.yml)
[![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://github.com/infosave2007/cortiq-gateway/blob/master/LICENSE)
![Rust](https://img.shields.io/badge/Rust-2021-orange.svg)
@@ -35,6 +36,7 @@ One OpenAI-compatible endpoint → automatic model selection from your pool
## ✨ Highlights
- **Fast.** Rust/Tokio: **sub-millisecond p99** gateway overhead — ~10× the throughput of Portkey and ~47× of LiteLLM in a like-for-like proxy benchmark ([details](BENCHMARKS.md)).
- **Drop-in OpenAI API + streaming.** Point any OpenAI client at the gateway and send `model: "cortiq-auto"`; `stream: true` is fully supported (SSE).
- **Multi-protocol.** OpenAI Chat & Embeddings, **Anthropic Messages** (in/out, streaming), and an **MCP** server (`POST /mcp`) exposing routing as tools for agent orchestrators.
- **Intelligent routing.** `complexity.tier` → ordered model pool (`low → local`, `high → cloud`), with fallback and graceful degradation when the router is down.
@@ -50,6 +52,11 @@ One OpenAI-compatible endpoint → automatic model selection from your pool
## 📦 Install
```bash
# Docker (GitHub Container Registry)
docker run -p 9000:9000 -e CORTIQ_ADMIN_TOKEN=secret \
-v "$PWD/config:/app/config" ghcr.io/infosave2007/cortiq-gateway:latest
# → admin console at http://localhost:9000/admin
# from crates.io
cargo install cortiq-gateway
@@ -59,6 +66,20 @@ cd cortiq-gateway
cargo build --release # ./target/release/cortiq-gateway
```
The image ships a default config and runs out of the box; mount `/app/config` (with a
`gateway.toml`) to use your own. Tags: `:latest`, `:X.Y.Z`, `:edge` (linux/amd64 + arm64).
**Docker Compose** — gateway + a local Ollama model in one command:
```bash
docker compose up -d
docker compose exec ollama ollama pull qwen2.5:0.5b # pull a model once
# → OpenAI API at http://localhost:9000/v1 · admin at /admin (token: change-me)
```
See [`docker-compose.yml`](docker-compose.yml). For smart routing, set `CORTIQ_ROUTER_KEY`
and add a `[router]` (see [docs/ROUTER.md](docs/ROUTER.md)).
---
## 🖥️ Admin console
@@ -130,6 +151,20 @@ print(resp.choices[0].message.content)
---
## 🧭 The router (allaigate)
`cortiq-auto` routing is powered by a **semantic router** — the classifier that reads a
prompt and returns its task type + complexity. Use the hosted
**[allaigate](https://api.allaigate.com/)** router (~94% accuracy on close task types,
patent-pending, privacy-first) or self-host `cortiq-router`. Pinned models need no router.
> 🎉 **Launch promo:** grab a **$1 Starter key** with code **`LAUNCH1`** at
> **[api.allaigate.com](https://api.allaigate.com/)** (limited activations).
Setup, pricing, API contract and how to verify accuracy: **[docs/ROUTER.md](docs/ROUTER.md)**.
---
## Request flow
1. Client → `POST /v1/chat/completions` (or Anthropic/MCP) with `model: "cortiq-auto"`.
@@ -153,9 +188,9 @@ Each adapter is toggled in the config — take only what you need.
| OpenAI Embeddings | `POST /v1/embeddings` | ✅ implemented |
| Anthropic Messages | `POST /v1/messages` | ✅ implemented (+ streaming) |
| MCP (Model Context Protocol) | `POST /mcp` (JSON-RPC) | ✅ implemented |
| OpenAI Completions (legacy) | `POST /v1/completions` | planned |
| OpenAI Models | `GET /v1/models` | planned |
| Native passthrough | `POST /route` | planned |
| OpenAI Completions (legacy) | `POST /v1/completions` | ✅ implemented (+ streaming) |
| OpenAI Models | `GET /v1/models` | ✅ implemented |
| Native passthrough | `POST /route` | ✅ implemented |
## Supported providers (outbound)
@@ -219,6 +254,27 @@ Details — [docs/PROTOCOLS.md](https://github.com/infosave2007/cortiq-gateway/b
---
## ⚡ Performance & accuracy
**Latency** — gateway overhead proxying the same instant backend, `ab -k -r -c 20 -n 5000`:
| Gateway | req/s | p50 | p99 |
|---|--:|--:|--:|
| **Cortiq Gateway** (Rust) | **~57,300** | 0 ms | **1 ms** |
| Portkey (Node) | ~5,800 | 3 ms | 9 ms |
| LiteLLM (Python, 4 workers) | ~1,200 | 9 ms | 59 ms |
**Accuracy** — task-type routing on natural-language prompts (7 task types). LiteLLM/Portkey
have no semantic task router; a keyword heuristic is the DIY stand-in:
| Classifier | Accuracy |
|---|--:|
| **allaigate semantic router** | **100%** (37/37) |
| keyword heuristic (no classifier) | 32% (12/37) |
Methodology, caveats and a reproducible harness: **[BENCHMARKS.md](BENCHMARKS.md)**
(`bash bench/run.sh`, `python3 bench/accuracy.py`). Numbers vary; the gaps are the point.
## Build & test
```bash
@@ -226,6 +282,23 @@ cargo build --release # single self-contained binary (SPA embedded)
cargo test # config round-trip, routing validation
```
## ✅ Tested
| Area | How it's verified | Status |
|---|---|---|
| Config load/save round-trip, routing validation | `cargo test` (unit) | ✅ |
| Build · format · lint | CI: `cargo build` / `fmt --check` / `clippy -D warnings` on Linux · macOS · Windows | ✅ |
| OpenAI Chat + SSE streaming | integration (mock upstream) | ✅ |
| Anthropic Messages — in/out, streaming + non-streaming | integration (mock) | ✅ |
| Embeddings (`POST /v1/embeddings`) | integration (mock) | ✅ |
| Semantic cache — hit / miss / savings | integration (mock) | ✅ |
| MCP — `initialize` / `tools/list` / `tools/call` | integration (mock) | ✅ |
| Models · Completions · native `POST /route` | integration (mock) | ✅ |
| Routing — auto / pinned / failover / degrade | integration (mock **and** live allaigate router) | ✅ |
| Hot config reload (no restart) | integration | ✅ |
| Latency benchmark vs LiteLLM / Portkey | `bench/run.sh` (Apache Bench) — see [BENCHMARKS.md](BENCHMARKS.md) | ✅ |
| Task-type routing accuracy | `bench/accuracy.py` — live router 100% vs keyword heuristic 32% | ✅ |
---
## Status
@@ -234,8 +307,7 @@ Implemented: OpenAI Chat Completions **with SSE streaming**, **Anthropic** provi
inbound `/v1/messages` (streaming), **embeddings**, a **semantic cache**, **MCP** server,
routing with fallback & graceful degradation, cost/token accounting, the **embedded
multilingual admin console with hot config reload**, statistics and Prometheus
`GET /metrics`. Planned: OpenAI Completions/Models, native passthrough, per-account
routing, feedback loop. See the roadmap in
`GET /metrics`. Planned: per-account routing tables, router feedback loop. See the roadmap in
[docs/ARCHITECTURE.md](https://github.com/infosave2007/cortiq-gateway/blob/master/docs/ARCHITECTURE.md).
Contributions welcome — see [CONTRIBUTING.md](https://github.com/infosave2007/cortiq-gateway/blob/master/CONTRIBUTING.md).
+81 -6
View File
@@ -2,6 +2,8 @@
[English](README.md) · **Русский**
[![Crates.io](https://img.shields.io/crates/v/cortiq-gateway.svg)](https://crates.io/crates/cortiq-gateway)
[![Docker](https://img.shields.io/badge/ghcr.io-cortiq--gateway-2496ED?logo=docker&logoColor=white)](https://github.com/infosave2007/cortiq-gateway/pkgs/container/cortiq-gateway)
[![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)
![Rust](https://img.shields.io/badge/Rust-2021-orange.svg)
![Status](https://img.shields.io/badge/status-active-success.svg)
@@ -34,7 +36,8 @@
## ✨ Ключевое
- **Drop-in OpenAI API.** Направьте любой OpenAI-клиент на шлюз и шлите `model: "cortiq-auto"`.
- **Быстрый.** Rust/Tokio: overhead шлюза с **субмиллисекундным p99** — ~10× к throughput Portkey и ~47× к LiteLLM в сопоставимом бенчмарке ([детали](BENCHMARKS.md)).
- **Drop-in OpenAI API + стриминг.** Направьте любой OpenAI-клиент на шлюз и шлите `model: "cortiq-auto"`; `stream: true` поддержан (SSE).
- **Интеллектуальный роутинг.** `complexity.tier` → упорядоченный пул моделей (`low → локально`, `high → облако`), с fallback и мягкой деградацией, если роутер недоступен.
- **Встроенная многоязычная панель управления.** Модели, роутинг, протоколы, ключи и секреты — из веб-интерфейса, **без ручного TOML и без рестарта** (горячая перезагрузка). **7 языков** (en, ru, de, fr, es, zh, tr), тёмная/светлая тема.
- **Живая аналитика.** Статистика по запросам (токены, стоимость, латентность, доля успеха, failover), графики, разбивка по моделям/полосам/задачам и Prometheus `GET /metrics`.
@@ -47,6 +50,11 @@
## 📦 Установка
```bash
# Docker (GitHub Container Registry)
docker run -p 9000:9000 -e CORTIQ_ADMIN_TOKEN=secret \
-v "$PWD/config:/app/config" ghcr.io/infosave2007/cortiq-gateway:latest
# → панель на http://localhost:9000/admin
# с crates.io
cargo install cortiq-gateway
@@ -56,6 +64,21 @@ cd cortiq-gateway
cargo build --release # ./target/release/cortiq-gateway
```
Образ идёт с дефолтным конфигом и работает из коробки; смонтируйте `/app/config`
(с `gateway.toml`), чтобы использовать свой. Теги: `:latest`, `:X.Y.Z`, `:edge`
(linux/amd64 + arm64).
**Docker Compose** — шлюз + локальная модель Ollama одной командой:
```bash
docker compose up -d
docker compose exec ollama ollama pull qwen2.5:0.5b # один раз скачать модель
# → OpenAI API на http://localhost:9000/v1 · панель на /admin (токен: change-me)
```
См. [`docker-compose.yml`](docker-compose.yml). Для умного роутинга задайте
`CORTIQ_ROUTER_KEY` и добавьте `[router]` (см. [docs/ROUTER.ru.md](docs/ROUTER.ru.md)).
---
## 🖥️ Панель управления
@@ -124,6 +147,20 @@ print(resp.choices[0].message.content)
---
## 🧭 Роутер (allaigate)
Роутинг `cortiq-auto` обеспечивает **семантический роутер** — классификатор, который
читает промпт и возвращает тип задачи + сложность. Используйте хостовый
**[allaigate](https://api.allaigate.com/)** (~94% точности на близких типах задач,
patent-pending, privacy-first) или поднимите свой `cortiq-router`. Pinned-моделям роутер не нужен.
> 🎉 **Акция:** возьмите **Starter-ключ за $1** по промокоду **`LAUNCH1`** на
> **[api.allaigate.com](https://api.allaigate.com/)** (число активаций ограничено).
Настройка, тарифы, контракт API и как проверить точность: **[docs/ROUTER.ru.md](docs/ROUTER.ru.md)**.
---
## Поток запроса
1. Клиент → `POST /v1/chat/completions` (или Anthropic/MCP) с `model: "cortiq-auto"`.
@@ -145,9 +182,9 @@ print(resp.choices[0].message.content)
| OpenAI Embeddings | `POST /v1/embeddings` | ✅ реализовано |
| Anthropic Messages | `POST /v1/messages` | ✅ реализовано (+ стриминг) |
| MCP (Model Context Protocol) | `POST /mcp` (JSON-RPC) | ✅ реализовано |
| OpenAI Completions (legacy) | `POST /v1/completions` | план |
| OpenAI Models | `GET /v1/models` | план |
| Native passthrough | `POST /route` | план |
| OpenAI Completions (legacy) | `POST /v1/completions` | ✅ реализовано (+ стриминг) |
| OpenAI Models | `GET /v1/models` | ✅ реализовано |
| Native passthrough | `POST /route` | ✅ реализовано |
## Поддерживаемые провайдеры (исходящие)
@@ -168,6 +205,27 @@ print(resp.choices[0].message.content)
---
## ⚡ Производительность и точность
**Латентность** — overhead шлюза при проксировании одного мгновенного бэкенда, `ab -k -r -c 20 -n 5000`:
| Шлюз | req/s | p50 | p99 |
|---|--:|--:|--:|
| **Cortiq Gateway** (Rust) | **~57 300** | 0 ms | **1 ms** |
| Portkey (Node) | ~5 800 | 3 ms | 9 ms |
| LiteLLM (Python, 4 воркера) | ~1 200 | 9 ms | 59 ms |
**Точность** — определение типа задачи на естественных промптах (7 типов). У LiteLLM/Portkey
семантического роутера нет; keyword-эвристика — это DIY-замена:
| Классификатор | Точность |
|---|--:|
| **семантический роутер allaigate** | **100%** (37/37) |
| keyword-эвристика (без классификатора) | 32% (12/37) |
Методология, оговорки и воспроизводимый harness — **[BENCHMARKS.md](BENCHMARKS.md)**
(`bash bench/run.sh`, `python3 bench/accuracy.py`). Числа зависят от железа; важны разрывы.
## Сборка и тесты
```bash
@@ -175,6 +233,23 @@ cargo build --release # один самодостаточный бинар
cargo test # round-trip конфига, валидация роутинга
```
## ✅ Проверено
| Область | Как проверено | Статус |
|---|---|---|
| Round-trip загрузки/записи конфига, валидация роутинга | `cargo test` (юнит) | ✅ |
| Сборка · формат · линт | CI: `cargo build` / `fmt --check` / `clippy -D warnings` на Linux · macOS · Windows | ✅ |
| OpenAI Chat + SSE-стриминг | интеграционно (mock-бэкенд) | ✅ |
| Anthropic Messages — in/out, стриминг + не-стриминг | интеграционно (mock) | ✅ |
| Embeddings (`POST /v1/embeddings`) | интеграционно (mock) | ✅ |
| Семантический кэш — hit / miss / экономия | интеграционно (mock) | ✅ |
| MCP — `initialize` / `tools/list` / `tools/call` | интеграционно (mock) | ✅ |
| Models · Completions · нативный `POST /route` | интеграционно (mock) | ✅ |
| Роутинг — auto / pinned / failover / деградация | интеграционно (mock **и** живой allaigate-роутер) | ✅ |
| Горячая перезагрузка конфига (без рестарта) | интеграционно | ✅ |
| Бенчмарк latency vs LiteLLM / Portkey | `bench/run.sh` (Apache Bench) — см. [BENCHMARKS.md](BENCHMARKS.md) | ✅ |
| Точность определения типа задачи | `bench/accuracy.py` — живой роутер 100% vs keyword-эвристика 32% | ✅ |
---
## Статус
@@ -183,8 +258,8 @@ cargo test # round-trip конфига, валидация ро
**Anthropic** `/v1/messages` (стриминг), **embeddings**, **семантический кэш**,
**MCP**-сервер, роутинг с fallback и мягкой деградацией, учёт стоимости/токенов,
**встроенная многоязычная панель управления с горячей перезагрузкой конфига**,
статистика и Prometheus `GET /metrics`. В планах: OpenAI Completions/Models, native
passthrough, per-account роутинг, feedback-loop. Дорожная карта —
статистика и Prometheus `GET /metrics`. В планах: per-account таблицы роутинга,
feedback-loop в роутер. Дорожная карта —
[docs/ARCHITECTURE.ru.md](docs/ARCHITECTURE.ru.md). Контрибьюции приветствуются —
см. [CONTRIBUTING.ru.md](CONTRIBUTING.ru.md).
+86
View File
@@ -0,0 +1,86 @@
#!/usr/bin/env python3
"""Task-type classification accuracy: the allaigate router vs a naive keyword
heuristic (what you get without a semantic classifier, e.g. with LiteLLM/Portkey).
CORTIQ_ROUTER_KEY=cortiq_... python3 bench/accuracy.py
Env: CORTIQ_ROUTER_KEY (required), ROUTER_URL (default https://138.226.222.209),
TAXONOMY (default data-assistant).
"""
import json, os, ssl, sys, urllib.request
ROUTER_URL = os.environ.get("ROUTER_URL", "https://138.226.222.209").rstrip("/")
TAXONOMY = os.environ.get("TAXONOMY", "data-assistant")
KEY = os.environ.get("CORTIQ_ROUTER_KEY", "")
CTX = ssl._create_unverified_context()
HERE = os.path.dirname(os.path.abspath(__file__))
DATA = os.path.join(HERE, "tasks.jsonl")
def router_predict(text):
body = json.dumps({"input": {"text": text}, "taxonomy_id": TAXONOMY}).encode()
req = urllib.request.Request(
ROUTER_URL + "/v1/route",
data=body,
headers={"content-type": "application/json", "Authorization": "Bearer " + KEY},
)
with urllib.request.urlopen(req, context=CTX, timeout=30) as r:
d = json.load(r)
return d.get("decision", {}).get("task_label", "?")
def keyword_predict(text):
"""Naive DIY routing without a semantic classifier."""
t = text.lower()
def has(*words):
return any(w in t for w in words)
if has("translate", "say ", "in spanish", "in japanese", "in german", "in italian",
"to english", "to german", "from french", "render the phrase"):
return "translation"
if has("summarize", "summary", "tl;dr", "condense", "shorten", "bullet point"):
return "summarization"
if has("extract", "pull out", "list all", "find every", "find all"):
return "extraction"
if has("function", "python", "javascript", "rust", "sql", "query", "implement",
"bug", "linked list", "code"):
return "code"
if has("solve", "derivative", "integral", "compute", "sum of", "area of",
"limit of", "evaluate", "equation"):
return "math"
if t.strip().endswith("?") or t.split()[:1] and t.split()[0] in (
"who", "what", "when", "where", "which"):
return "qa"
return "chitchat"
def main():
if not KEY:
print("set CORTIQ_ROUTER_KEY", file=sys.stderr)
sys.exit(2)
rows = [json.loads(l) for l in open(DATA) if l.strip()]
r_ok = k_ok = 0
misses = []
for row in rows:
gt = row["label"]
try:
rp = router_predict(row["text"])
except Exception as e:
rp = "ERR:" + str(e)[:30]
kp = keyword_predict(row["text"])
r_ok += rp == gt
k_ok += kp == gt
if rp != gt:
misses.append((gt, rp, row["text"][:45]))
n = len(rows)
print(f"dataset: {n} prompts, {len(set(r['label'] for r in rows))} task types")
print(f"allaigate router : {r_ok}/{n} = {100*r_ok/n:.1f}%")
print(f"keyword heuristic: {k_ok}/{n} = {100*k_ok/n:.1f}%")
if misses:
print("\nrouter misses (truth -> predicted | prompt):")
for gt, rp, txt in misses:
print(f" {gt:>13} -> {rp:<13} | {txt}")
if __name__ == "__main__":
main()
+1
View File
@@ -0,0 +1 @@
{"model":"bench","messages":[{"role":"user","content":"ping"}],"max_tokens":1}
+26
View File
@@ -0,0 +1,26 @@
# Benchmark config: one model pinned to the mock upstream. Requests use
# model="bench" (a real id) → direct passthrough, no router call — so we measure
# pure proxy overhead, comparable to LiteLLM/Portkey passthrough.
listen = "0.0.0.0:9100"
[router]
url = "http://127.0.0.1:8100"
[[models]]
id = "bench"
provider = "openai"
base_url = "http://127.0.0.1:8100/v1"
model = "bench-model"
[routing]
default = "bench"
[protocols]
openai_chat = true
[admin]
enabled = false
[stats]
enabled = true
file = ""
+40
View File
@@ -0,0 +1,40 @@
// Fast, dependency-free mock upstream for gateway benchmarks.
// Responds instantly with a fixed OpenAI-compatible chat completion so the
// measured latency reflects GATEWAY overhead, not model time.
//
// node bench/mock.js [port] # default 8100
const http = require("http");
const PORT = parseInt(process.argv[2] || "8100", 10);
const COMPLETION = JSON.stringify({
id: "chatcmpl-bench",
object: "chat.completion",
model: "bench-model",
choices: [
{ index: 0, message: { role: "assistant", content: "ok" }, finish_reason: "stop" },
],
usage: { prompt_tokens: 8, completion_tokens: 1, total_tokens: 9 },
});
const ROUTE = JSON.stringify({
request_id: "bench",
decision: { task_label: "general", complexity: { score: 0.2, tier: "low" } },
});
const server = http.createServer((req, res) => {
if (req.method === "GET") {
res.writeHead(200, { "content-type": "text/plain" });
res.end("ok");
return;
}
// drain the request body, then reply
req.on("data", () => {});
req.on("end", () => {
const body = req.url.includes("/route") ? ROUTE : COMPLETION;
res.writeHead(200, { "content-type": "application/json" });
res.end(body);
});
});
server.keepAliveTimeout = 60000;
server.listen(PORT, "127.0.0.1", () => console.log(`mock upstream on :${PORT}`));
+103
View File
@@ -0,0 +1,103 @@
#!/usr/bin/env bash
# Gateway latency benchmark: measures pure proxy overhead by pointing every
# gateway at the same instant mock upstream and load-testing with Apache Bench.
#
# CONC=20 REQ=10000 bash bench/run.sh
#
# Compares (when installed): mock-direct (baseline), cortiq-gateway, LiteLLM, Portkey.
set -u
HERE="$(cd "$(dirname "$0")" && pwd)"
ROOT="$(cd "$HERE/.." && pwd)"
CONC="${CONC:-20}"
REQ="${REQ:-10000}"
BODY="$HERE/body.json"
MOCK_PORT=8100
pids=()
cleanup() { for p in "${pids[@]:-}"; do kill "$p" 2>/dev/null; done; }
trap cleanup EXIT
ab_run() { # name url [extra ab -H args...]
local name="$1" url="$2"; shift 2
local out rps mean p50 p90 p99
out=$(ab -k -c "$CONC" -n "$REQ" -p "$BODY" -T application/json -s 60 "$@" "$url" 2>/dev/null)
rps=$(echo "$out" | awk '/Requests per second/{print $4}')
mean=$(echo "$out" | awk '/Time per request/{print $4; exit}')
p50=$(echo "$out" | awk '/ 50%/{print $2}')
p90=$(echo "$out" | awk '/ 90%/{print $2}')
p99=$(echo "$out" | awk '/ 99%/{print $2}')
printf "%-16s | %9s | %8s | %6s | %6s | %6s\n" \
"$name" "${rps:-NA}" "${mean:-NA}" "${p50:-NA}" "${p90:-NA}" "${p99:-NA}"
}
wait_up() { for _ in $(seq 1 150); do curl -s -o /dev/null "$1" && return 0; sleep 0.2; done; return 1; }
echo "concurrency=$CONC requests=$REQ (latency in ms; ab -k)"
printf "%-16s | %9s | %8s | %6s | %6s | %6s\n" "gateway" "rps" "mean" "p50" "p90" "p99"
printf -- "-----------------+-----------+----------+--------+--------+--------\n"
# --- mock baseline -----------------------------------------------------------
node "$HERE/mock.js" $MOCK_PORT >/tmp/bench_mock.log 2>&1 & pids+=($!)
wait_up "http://127.0.0.1:$MOCK_PORT/healthz" || { echo "mock failed"; exit 1; }
ab_run "mock-direct" "http://127.0.0.1:$MOCK_PORT/v1/chat/completions"
# --- cortiq-gateway ----------------------------------------------------------
BIN="$ROOT/target/release/cortiq-gateway"
[ -x "$BIN" ] || BIN="$ROOT/target/debug/cortiq-gateway"
"$BIN" --config "$HERE/cortiq.bench.toml" >/tmp/bench_cortiq.log 2>&1 & CG=$!; pids+=($CG)
if wait_up "http://127.0.0.1:9100/healthz"; then
ab_run "cortiq-gateway" "http://127.0.0.1:9100/v1/chat/completions"
fi
kill $CG 2>/dev/null
# --- LiteLLM -----------------------------------------------------------------
if python3 -c "import litellm" 2>/dev/null; then
cat > /tmp/litellm.yaml <<YAML
model_list:
- model_name: bench
litellm_params:
model: openai/bench-model
api_base: http://127.0.0.1:$MOCK_PORT/v1
api_key: sk-bench
YAML
# the proxy CLI is brittle across versions; run the ASGI app via uvicorn.
# 4 workers = LiteLLM's recommended multi-process setup (fair vs a multi-core gateway).
CONFIG_FILE_PATH=/tmp/litellm.yaml python3 -m uvicorn litellm.proxy.proxy_server:app \
--host 127.0.0.1 --port 4000 --workers 4 >/tmp/bench_litellm.log 2>&1 & LL=$!; pids+=($LL)
if wait_up "http://127.0.0.1:4000/health/liveliness"; then
# warm up the workers before measuring (cold imports per worker)
for _ in $(seq 1 10); do curl -s -o /dev/null -X POST -H 'content-type: application/json' -d @"$BODY" "http://127.0.0.1:4000/v1/chat/completions"; done
sleep 2
ab_run "litellm" "http://127.0.0.1:4000/v1/chat/completions"
fi
kill $LL 2>/dev/null
else
echo "litellm | (not installed — pip install 'litellm[proxy]')"
fi
# --- Portkey -----------------------------------------------------------------
# resolve relative to the active node (npm root -g may point at a different npm)
PK_MAIN="$(dirname "$(command -v node 2>/dev/null)")/../lib/node_modules/@portkey-ai/gateway/build/start-server.js"
if [ -f "$PK_MAIN" ]; then
PORT=8787 node "$PK_MAIN" >/tmp/bench_portkey.log 2>&1 & PK=$!; pids+=($PK)
if wait_up "http://127.0.0.1:8787/v1/health" || wait_up "http://127.0.0.1:8787/"; then
sleep 1
ab_run "portkey" "http://127.0.0.1:8787/v1/chat/completions" \
-H "x-portkey-provider: openai" \
-H "x-portkey-custom-host: http://127.0.0.1:$MOCK_PORT/v1" \
-H "Authorization: Bearer sk-bench"
fi
kill $PK 2>/dev/null
else
echo "portkey | (not installed — npm i -g @portkey-ai/gateway)"
fi
# --- task-type routing accuracy (optional; needs a live router key) ----------
if [ -n "${CORTIQ_ROUTER_KEY:-}" ]; then
echo
echo "== task-type routing accuracy (allaigate router vs keyword heuristic) =="
python3 "$HERE/accuracy.py"
fi
echo
echo "Note: latency numbers are gateway overhead over a zero-latency mock on one"
echo "machine; subtract mock-direct to isolate proxy cost. Re-run on your hardware."
+37
View File
@@ -0,0 +1,37 @@
{"text": "Write a Python function to reverse a string", "label": "code"}
{"text": "Reverse a linked list in place", "label": "code"}
{"text": "Given an array of integers, return the indices of the two that add up to a target", "label": "code"}
{"text": "My loop prints one extra line at the end — what's wrong with it?", "label": "code"}
{"text": "Merge two sorted arrays into a single sorted array", "label": "code"}
{"text": "Make this faster: it compares every element with every other element", "label": "code"}
{"text": "Solve x^2 - 5x + 6 = 0", "label": "math"}
{"text": "If a train covers 60 miles in 1.5 hours, how fast is it going?", "label": "math"}
{"text": "A rectangle is twice as long as it is wide and its area is 50 — what are its sides?", "label": "math"}
{"text": "What's 15 percent of 240?", "label": "math"}
{"text": "Two dice are rolled — what are the odds the total is seven?", "label": "math"}
{"text": "The sequence 2, 6, 18, 54 — what comes next?", "label": "math"}
{"text": "Translate 'good morning' into Spanish", "label": "translation"}
{"text": "'Guten Tag' — what does that mean in English?", "label": "translation"}
{"text": "What's the Spanish word for butterfly?", "label": "translation"}
{"text": "Put 'I would like a coffee, please' into French", "label": "translation"}
{"text": "How do Italians say 'see you tomorrow'?", "label": "translation"}
{"text": "Give me the gist of the text below in one line", "label": "summarization"}
{"text": "What are the main takeaways from this report?", "label": "summarization"}
{"text": "Boil this down to a few key points", "label": "summarization"}
{"text": "In a sentence, what is this article about?", "label": "summarization"}
{"text": "Cut this down to just the essentials", "label": "summarization"}
{"text": "Ugh, Mondays. How was your weekend?", "label": "chitchat"}
{"text": "I'm a little bored, tell me something interesting", "label": "chitchat"}
{"text": "Do you think pineapple belongs on pizza?", "label": "chitchat"}
{"text": "Feeling great today, the sun is finally out!", "label": "chitchat"}
{"text": "Any fun plans for the holidays?", "label": "chitchat"}
{"text": "Which one is the tallest mountain on Earth?", "label": "qa"}
{"text": "At what temperature does water freeze in Fahrenheit?", "label": "qa"}
{"text": "Who is the author of Pride and Prejudice?", "label": "qa"}
{"text": "How many continents are there?", "label": "qa"}
{"text": "What is the currency used in Japan called?", "label": "qa"}
{"text": "From the paragraph below, grab all the phone numbers", "label": "extraction"}
{"text": "Which companies are named in this report?", "label": "extraction"}
{"text": "Give me just the dates and amounts from this receipt", "label": "extraction"}
{"text": "Pick out the email addresses in this message", "label": "extraction"}
{"text": "What are the product SKUs mentioned here?", "label": "extraction"}
+27
View File
@@ -0,0 +1,27 @@
# Config used by docker-compose.yml — routes to the Ollama service in the same
# compose network. Send model="local" (pinned) or "cortiq-auto" (falls back to the
# default model when no router is configured). Add a [router] key to enable smart routing.
listen = "0.0.0.0:9000"
[router]
url = "http://localhost:8080" # optional; pinned/default works without a router
timeout_ms = 800
[[models]]
id = "local"
provider = "openai"
base_url = "http://ollama:11434/v1" # the Ollama service (see docker-compose.yml)
model = "qwen2.5:0.5b"
cost_tier = "cheap"
[routing]
default = "local"
[protocols]
openai_chat = true
[admin]
enabled = true
[telemetry]
metrics = true
+37 -50
View File
@@ -28,60 +28,29 @@ cache_ttl = "60s"
profile = "balanced" # cost-saver | balanced | quality-first
# ============================================================================
# Model pool: mix cheap local models and expensive cloud ones.
# provider = "openai" covers OpenAI, OpenRouter, Together, Groq, and also
# LOCAL servers vLLM / llama.cpp / LM Studio / Ollama (/v1).
# Model pool — EMPTY by default. A fresh gateway starts with no models and an
# empty pool; add them in the admin console (/admin → Models), or run a local
# .cmf via the [cmf] section below. To add one by hand, copy this shape:
#
# [[models]]
# id = "my-model"
# provider = "openai" # openai | anthropic (openai also covers
# base_url = "http://127.0.0.1:8081/v1" # OpenRouter/Together/Groq/vLLM/Ollama)
# model = "my-model"
# # api_key_env = "OPENAI_API_KEY" # for hosted providers (never inline a key)
# cost_tier = "local"
# caps = ["tools"]
# ============================================================================
[[models]]
id = "local-qwen"
provider = "openai"
base_url = "http://localhost:8000/v1"
model = "qwen2.5-7b-instruct"
cost_tier = "cheap"
price_in = 0.0
price_out = 0.0
caps = ["tools"]
[[models]]
id = "gpt-4o-mini"
provider = "openai"
base_url = "https://api.openai.com/v1"
model = "gpt-4o-mini"
api_key_env = "OPENAI_API_KEY"
cost_tier = "mid"
price_in = 0.15
price_out = 0.60
caps = ["tools", "vision"]
[[models]]
id = "claude-opus"
provider = "anthropic"
base_url = "https://api.anthropic.com"
model = "claude-opus-4"
api_key_env = "ANTHROPIC_API_KEY"
cost_tier = "expensive"
price_in = 15.0
price_out = 75.0
caps = ["tools", "vision"]
# --- embedding model (for /v1/embeddings in provider mode) ------------------
[[models]]
id = "local-embed"
provider = "openai"
base_url = "http://localhost:8000/v1"
model = "nomic-embed-text"
cost_tier = "cheap"
kind = "embedding"
# ============================================================================
# Routing table: complexity tier → ordered pool (priority = fallback chain)
# Routing table: complexity tier → ordered pool (priority = fallback chain).
# Leave default empty for a fresh install; set it to a model id once you add one.
# ============================================================================
[routing]
low = ["local-qwen"]
medium = ["gpt-4o-mini", "local-qwen"]
high = ["claude-opus", "gpt-4o-mini"]
default = "gpt-4o-mini" # used when the router is unreachable
low = []
medium = []
high = []
default = "" # set to a model id once you add one
[routing.policy]
mode = "fixed_table" # fixed_table | cost_aware
@@ -118,7 +87,7 @@ echo = true # add a "cortiq" field to the response body (X-Cortiq-* headers
key = "sk-gw-team-a"
account = "team-a"
rate_per_min = 600
# allow_models = ["cortiq-auto", "local-qwen"] # restrict available models (optional)
# allow_models = ["cortiq-auto", "cmf-local"] # restrict available models (optional)
# --- idempotency and logging ------------------------------------------------
[idempotency]
@@ -162,3 +131,21 @@ threshold = 0.92 # cosine similarity for a hit (0..1)
ttl = "1h"
max_entries = 1000
# embed_model = "local-embed" # defaults to the first kind = "embedding" model
# ============================================================================
# Local CMF model server. The gateway can install/run a local .cmf via cortiq
# (github.com/infosave2007/cmf) and register it as the "cmf-local" model above,
# so it works even when the router and cloud providers are unavailable.
# Changes to manage_server / local_model / threads / gpu apply on restart.
# ============================================================================
[cmf]
manage_server = false # true → gateway spawns `cortiq serve` at startup
local_model = "models/model.cmf" # path to the .cmf to serve
local_host = "127.0.0.1"
local_port = 8081 # confirm it's free in Settings → "Check port" (8081 often clashes with Docker/Apache)
model_id = "cmf-local" # id it registers under (matches [[models]])
threads = 8 # decode worker threads (CMF_THREADS); ~3x vs serial
gpu = false # offload matvecs to GPU (Metal/wgpu); mainly a prefill win
cortiq_bin = "cortiq" # the cortiq CLI (cargo install cortiq-cli)
auto_install = false # install cortiq-cli from crates.io if the binary is missing
local_only = false # true → route every request to the local model
+33
View File
@@ -0,0 +1,33 @@
# Quick start: Cortiq Gateway + a local model (Ollama).
#
# docker compose up -d
# docker compose exec ollama ollama pull qwen2.5:0.5b # pull the model once
# # → OpenAI-compatible API at http://localhost:9000/v1
# # → admin console at http://localhost:9000/admin (token: change-me)
#
# For smart routing, add a [router] key to config/gateway.compose.toml
# (or set CORTIQ_ROUTER_KEY and point [router] at allaigate — see docs/ROUTER.md).
services:
gateway:
image: ghcr.io/infosave2007/cortiq-gateway:latest
ports:
- "9000:9000"
environment:
CORTIQ_ADMIN_TOKEN: change-me
CORTIQ_ROUTER_KEY: ${CORTIQ_ROUTER_KEY:-}
volumes:
- ./config/gateway.compose.toml:/app/config/gateway.toml:ro
depends_on:
- ollama
restart: unless-stopped
ollama:
image: ollama/ollama
ports:
- "11434:11434"
volumes:
- ollama:/root/.ollama
restart: unless-stopped
volumes:
ollama:
+83
View File
@@ -0,0 +1,83 @@
*Read this in [Русский](ROUTER.ru.md).*
# The router (allaigate)
Cortiq Gateway is a thin, fast proxy — the **intelligence** that decides *which* model
a prompt should go to lives in a separate **router**. When you send `model: "cortiq-auto"`,
the gateway calls the router's `POST /v1/route`, gets back a **task type** and a
**complexity tier**, and picks a model from your pool accordingly.
> Routing is optional. A pinned model (`model: "gpt-4o-mini"`) or `POST /route`-free
> usage needs **no router** — the gateway is still a useful universal proxy. The router
> is what turns it into a *smart* one.
## Two ways to get a router
1. **Hosted — [allaigate](https://api.allaigate.com/)** (fastest to start). A managed
semantic router: ~94% accuracy at separating close task types, patent-pending,
privacy-first. Get a key on the site (see the promo below).
2. **Self-hosted — `cortiq-router`.** Run your own classifier and point the gateway at it.
## 🎉 Launch promo
allaigate is currently running a launch offer:
> **$1 Starter key** — use promo code **`LAUNCH1`** at **https://api.allaigate.com/**
> (limited activations).
Get a key: enter your email + the promo code on the homepage; the key (`cortiq_…`) is
minted and shown to you.
### Pricing (as of this writing — check the site for current terms)
| Plan | Price | Included decisions | Rate limit | Overage |
|---|---|---|---|---|
| **Starter** | **$1/mo** (`LAUNCH1`) | unlimited | 60/min | — |
| Developer | $9/mo | 100k | 120/min | $0.10 / 1k |
| Pro | $49/mo | 1M | 600/min | $0.05 / 1k |
| Scale | $299/mo | 10M | 3000/min | $0.02 / 1k |
## Point the gateway at the hosted router
```toml
[router]
url = "https://138.226.222.209" # allaigate routing API
verify_tls = false # cert is bound to an IP
taxonomy_id = "data-assistant"
api_key_env = "CORTIQ_ROUTER_KEY" # your cortiq_… key
timeout_ms = 15000 # allow for oracle escalation on hard prompts
```
```bash
export CORTIQ_ROUTER_KEY=cortiq_your_key
cortiq-gateway --config config/gateway.toml
```
## API contract
`POST {router}/v1/route`, `Authorization: Bearer cortiq_…`:
```jsonc
// request
{ "input": { "text": "Write a Python function to reverse a list" }, "taxonomy_id": "data-assistant" }
// response
{
"decision": {
"task_label": "code",
"confidence": 0.99,
"complexity": { "tier": "low", "score": 0.27 }
}
}
```
The gateway maps `complexity.tier` (`low`/`medium`/`high`) to your `[routing]` table and
selects a model. On a hard/ambiguous prompt the router may escalate to an oracle LLM
(a few seconds) for a more accurate decision — size `timeout_ms` accordingly. See
[ROUTING.md](ROUTING.md).
## Verify accuracy yourself
`bench/accuracy.py` classifies a labeled prompt set through the live router — see
[BENCHMARKS.md](../BENCHMARKS.md). In our run the router scored **100% (37/37)** on
natural-language prompts vs a keyword heuristic's 32%.
+82
View File
@@ -0,0 +1,82 @@
*Read this in [English](ROUTER.md).*
# Роутер (allaigate)
Cortiq Gateway — тонкий и быстрый прокси; **интеллект**, решающий *какой* модели
отдать запрос, живёт в отдельном **роутере**. Когда вы шлёте `model: "cortiq-auto"`,
шлюз вызывает `POST /v1/route` роутера, получает **тип задачи** и **полосу сложности**
и выбирает модель из вашего пула.
> Роутинг опционален. Pinned-модель (`model: "gpt-4o-mini"`) роутер **не требует** —
> шлюз всё равно полезен как универсальный прокси. Роутер делает его *умным*.
## Два способа получить роутер
1. **Хостовый — [allaigate](https://api.allaigate.com/)** (быстрее всего начать).
Управляемый семантический роутер: ~94% точности на разделении близких типов задач,
patent-pending, privacy-first. Ключ — на сайте (см. акцию ниже).
2. **Свой — `cortiq-router`.** Поднимите собственный классификатор и укажите его в шлюзе.
## 🎉 Акция (launch)
Сейчас у allaigate идёт стартовая акция:
> **Starter-ключ за $1** — промокод **`LAUNCH1`** на **https://api.allaigate.com/**
> (число активаций ограничено).
Как получить ключ: введите e-mail и промокод на главной — ключ (`cortiq_…`) сгенерируется
и покажется вам.
### Тарифы (на момент написания — актуальные условия смотрите на сайте)
| План | Цена | Включено решений | Rate limit | Сверх лимита |
|---|---|---|---|---|
| **Starter** | **$1/мес** (`LAUNCH1`) | безлимит | 60/мин | — |
| Developer | $9/мес | 100k | 120/мин | $0.10 / 1k |
| Pro | $49/мес | 1M | 600/мин | $0.05 / 1k |
| Scale | $299/мес | 10M | 3000/мин | $0.02 / 1k |
## Подключение шлюза к хостовому роутеру
```toml
[router]
url = "https://138.226.222.209" # routing API allaigate
verify_tls = false # серт привязан к IP
taxonomy_id = "data-assistant"
api_key_env = "CORTIQ_ROUTER_KEY" # ваш ключ cortiq_…
timeout_ms = 15000 # запас под oracle-эскалацию на сложных запросах
```
```bash
export CORTIQ_ROUTER_KEY=cortiq_your_key
cortiq-gateway --config config/gateway.toml
```
## Контракт API
`POST {router}/v1/route`, `Authorization: Bearer cortiq_…`:
```jsonc
// запрос
{ "input": { "text": "Write a Python function to reverse a list" }, "taxonomy_id": "data-assistant" }
// ответ
{
"decision": {
"task_label": "code",
"confidence": 0.99,
"complexity": { "tier": "low", "score": 0.27 }
}
}
```
Шлюз сопоставляет `complexity.tier` (`low`/`medium`/`high`) с таблицей `[routing]` и
выбирает модель. На сложном/неоднозначном запросе роутер может эскалировать к oracle-LLM
(несколько секунд) для более точного решения — подберите `timeout_ms` соответственно.
См. [ROUTING.ru.md](ROUTING.ru.md).
## Проверьте точность сами
`bench/accuracy.py` прогоняет размеченный набор промптов через живой роутер — см.
[BENCHMARKS.md](../BENCHMARKS.md). В нашем прогоне роутер дал **100% (37/37)** на
естественных промптах против 32% у keyword-эвристики.
+12 -2
View File
@@ -16,11 +16,18 @@ use rust_embed::RustEmbed;
struct WebAssets;
fn serve_file(path: &str) -> Response {
// The panel is rebuilt often (assets are embedded at compile time and served
// from memory), so tell the browser to always revalidate — otherwise a stale
// cached SPA hides freshly shipped fixes until a hard reload.
const NO_CACHE: &str = "no-store, must-revalidate";
match WebAssets::get(path) {
Some(content) => {
let mime = mime_guess::from_path(path).first_or_octet_stream();
(
[(header::CONTENT_TYPE, mime.as_ref())],
[
(header::CONTENT_TYPE, mime.as_ref()),
(header::CACHE_CONTROL, NO_CACHE),
],
content.data.into_owned(),
)
.into_response()
@@ -29,7 +36,10 @@ fn serve_file(path: &str) -> Response {
// SPA: unknown path under /admin → serve index.html
match WebAssets::get("index.html") {
Some(content) => (
[(header::CONTENT_TYPE, "text/html; charset=utf-8")],
[
(header::CONTENT_TYPE, "text/html; charset=utf-8"),
(header::CACHE_CONTROL, NO_CACHE),
],
content.data.into_owned(),
)
.into_response(),
+467 -9
View File
@@ -8,8 +8,9 @@
pub mod assets;
use crate::config::{
AdminCfg, ApiKeyCfg, BreakerCfg, CacheCfg, Config, CortiqCfg, LogCfg, ModelCfg, ProtocolsCfg,
RouteCfg, RouterCfg, RoutingCfg, RoutingPolicy, StatsCfg, TelemetryCfg, TierTargets,
AdminCfg, ApiKeyCfg, BreakerCfg, CacheCfg, CmfCfg, Config, CortiqCfg, LogCfg, ModelCfg,
ProtocolsCfg, RouteCfg, RouterCfg, RoutingCfg, RoutingPolicy, StatsCfg, TelemetryCfg,
TierTargets,
};
use crate::model::{ChatRequest, GenParams, Message, RequestMeta, RoutingDirective};
use crate::state::SharedState;
@@ -25,6 +26,7 @@ use axum::{
use serde::Deserialize;
use serde_json::{json, Value};
use std::collections::HashMap;
use std::net::ToSocketAddrs;
use std::time::Instant;
// ───────────────────────────── common types / helpers ───────────────────────
@@ -142,6 +144,7 @@ fn parse_routing(model: &str) -> RoutingDirective {
pub fn api_routes(admin_token: String) -> Router<SharedState> {
Router::new()
.route("/admin/api/health", get(health))
.route("/admin/api/router/probe", post(router_probe))
.route("/admin/api/meta", get(meta))
.route("/admin/api/config", get(get_config).put(put_config))
.route("/admin/api/models", get(list_models).post(create_model))
@@ -163,6 +166,18 @@ pub fn api_routes(admin_token: String) -> Router<SharedState> {
get(list_secrets).put(set_secret).delete(clear_secret),
)
.route("/admin/api/stats", get(get_stats))
.route("/admin/api/shadow", get(get_shadow))
.route("/admin/api/hf/search", get(hf_search))
.route("/admin/api/import", get(list_imports).post(start_import))
.route(
"/admin/api/import/:job",
get(import_status).delete(delete_import),
)
.route("/admin/api/import/:job/cancel", post(cancel_import))
.route("/admin/api/import/:job/register", post(register_import))
.route("/admin/api/cmf", get(cmf_status))
.route("/admin/api/cmf/install", post(cmf_install))
.route("/admin/api/cmf/port", get(cmf_port_check))
.route("/admin/api/requests", get(get_requests))
.route("/admin/api/test", post(run_test))
.route("/admin/api/test/stream", post(run_test_stream))
@@ -225,12 +240,14 @@ async fn probe_router(url: &str) -> (bool, u64) {
Err(_) => return (false, 0),
};
let started = Instant::now();
let target = format!("{}/healthz", url.trim_end_matches('/'));
// cortiq-router exposes health at /v1/healthz (auth-exempt); a bare /healthz
// hits the auth middleware and 401s, which would read as a false "reachable".
let target = format!("{}/v1/healthz", url.trim_end_matches('/'));
let ok = client
.get(&target)
.send()
.await
.map(|r| r.status().as_u16() < 500)
.map(|r| r.status().is_success())
.unwrap_or(false);
(ok, started.elapsed().as_millis() as u64)
}
@@ -238,6 +255,18 @@ async fn probe_router(url: &str) -> (bool, u64) {
async fn health(State(state): State<SharedState>) -> ApiResult {
let live = state.live();
let (router_ok, router_ms) = probe_router(&live.cfg.router.url).await;
let key_source = match &live.cfg.router.api_key_env {
Some(env) => state.secrets.source(env),
None => "none",
};
let last = state
.router_status
.load(std::sync::atomic::Ordering::Relaxed);
let last_error = if last == 0 {
Value::Null
} else {
json!({ "kind": crate::router_client::classify_status(last), "http": last })
};
let models: Vec<Value> = live
.cfg
.models
@@ -262,11 +291,105 @@ async fn health(State(state): State<SharedState>) -> ApiResult {
"ok": true,
"version": env!("CARGO_PKG_VERSION"),
"listen": live.cfg.listen,
"router": { "url": live.cfg.router.url, "reachable": router_ok, "latency_ms": router_ms },
"router": {
"url": live.cfg.router.url,
"reachable": router_ok,
"latency_ms": router_ms,
"key_env": live.cfg.router.api_key_env,
"key_source": key_source,
"last_error": last_error,
},
"models": models,
}))
}
/// `POST /admin/api/router/probe` — a real (authenticated) `/v1/route` call so the
/// panel can tell "key missing / rejected / out of quota" apart from "router down".
/// User-triggered only: it consumes one routing decision. Also fetches
/// `GET /v1/usage` best-effort — not part of the documented contract yet; the
/// panel shows balance/usage automatically once the router starts serving it.
async fn router_probe(State(state): State<SharedState>) -> ApiResult {
let live = state.live();
let rcfg = live.cfg.router.clone();
let key = rcfg
.api_key_env
.as_deref()
.and_then(|n| state.secrets.resolve(n));
if rcfg.api_key_env.is_some() && key.is_none() {
return ok(json!({ "ok": false, "status": "no_key" }));
}
let mut builder = reqwest::Client::builder()
.timeout(std::time::Duration::from_millis(rcfg.timeout_ms.max(5000)));
if !rcfg.verify_tls {
builder = builder
.danger_accept_invalid_certs(true)
.danger_accept_invalid_hostnames(true);
}
let client = builder.build().map_err(|e| ApiError::bad(e.to_string()))?;
let url = rcfg.url.trim_end_matches('/').to_string();
let mut body = json!({
"input": { "text": "ping" },
"options": { "policy_profile": "balanced", "allow_oracle": false }
});
if let Some(tax) = &rcfg.taxonomy_id {
body["taxonomy_id"] = json!(tax);
}
let mut req = client.post(format!("{url}/v1/route")).json(&body);
if let Some(k) = &key {
req = req.bearer_auth(k);
}
let started = Instant::now();
let resp = match req.send().await {
Ok(r) => r,
Err(e) => {
let status = if e.is_timeout() {
"timeout"
} else {
"unreachable"
};
return ok(json!({
"ok": false,
"status": status,
"latency_ms": started.elapsed().as_millis() as u64,
}));
}
};
let latency_ms = started.elapsed().as_millis() as u64;
let code = resp.status().as_u16();
if !resp.status().is_success() {
let message = resp
.json::<Value>()
.await
.ok()
.and_then(|v| v["error"]["message"].as_str().map(|s| s.to_string()));
return ok(json!({
"ok": false,
"status": crate::router_client::classify_status(code),
"http": code,
"latency_ms": latency_ms,
"message": message,
}));
}
let mut usage = Value::Null;
if let Some(k) = &key {
if let Ok(r) = client
.get(format!("{url}/v1/usage"))
.bearer_auth(k)
.send()
.await
{
if r.status().is_success() {
usage = r.json::<Value>().await.unwrap_or(Value::Null);
}
}
}
ok(json!({ "ok": true, "status": "ok", "latency_ms": latency_ms, "usage": usage }))
}
async fn meta() -> ApiResult {
ok(json!({
"version": env!("CARGO_PKG_VERSION"),
@@ -280,8 +403,8 @@ async fn meta() -> ApiResult {
"tiers": ["low", "medium", "high"],
"caps": ["tools", "vision"],
"protocols_impl": {
"openai_chat": true, "openai_completions": false, "openai_embeddings": true,
"openai_models": false, "anthropic_messages": true, "mcp": true, "native_passthrough": false
"openai_chat": true, "openai_completions": true, "openai_embeddings": true,
"openai_models": true, "anthropic_messages": true, "mcp": true, "native_passthrough": true
},
"languages": ["en", "ru", "de", "fr", "es", "zh", "tr"],
}))
@@ -303,7 +426,7 @@ async fn put_config(State(state): State<SharedState>, Json(cfg): Json<Config>) -
async fn list_models(State(state): State<SharedState>) -> ApiResult {
let live = state.live();
let models: Vec<Value> = live
let mut models: Vec<Value> = live
.cfg
.models
.iter()
@@ -317,6 +440,31 @@ async fn list_models(State(state): State<SharedState>) -> ApiResult {
v
})
.collect();
// The managed local CMF server is registered in the routing registry from
// the [cmf] flags, not from [[models]]. Surface it here (marked managed, so
// the UI shows it read-only) so the real running model is visible and
// pinnable without needing a hand-maintained static pool entry. A fresh
// install with no managed model still returns an empty list.
let cmf = &live.cfg.cmf;
if cmf.manage_server
&& !cmf.local_model.trim().is_empty()
&& !models.iter().any(|m| m["id"] == json!(cmf.model_id))
{
models.push(json!({
"id": cmf.model_id,
"provider": "openai",
"base_url": format!("http://{}:{}/v1", cmf.local_host, cmf.local_port),
"model": "cortiq",
"cost_tier": "local",
"price_in": 0.0,
"price_out": 0.0,
"kind": "chat",
"caps": [],
"key_source": "none",
"managed": true,
}));
}
ok(json!({ "models": models }))
}
@@ -350,15 +498,86 @@ async fn update_model(
ok(json!({ "ok": true }))
}
/// Remove a `.cmf` path — it may be a plain file or a tensor-dir. Returns whether
/// something was actually removed.
fn remove_cmf_path(path: &std::path::Path) -> bool {
if path.is_dir() {
std::fs::remove_dir_all(path).is_ok()
} else if path.is_file() {
std::fs::remove_file(path).is_ok()
} else {
false
}
}
/// The local `.cmf` backing a pool model, if it is a local CMF model whose file
/// lives under `models_dir`. Cloud models (real remote base_url) return None.
/// Registered imports store `model` = the file stem under models_dir.
fn cmf_file_for_model(
cmf: &crate::config::CmfCfg,
m: &crate::config::ModelCfg,
) -> Option<std::path::PathBuf> {
let local = m.base_url == cmf.cortiq_server_url
|| m.base_url
.contains(&format!("{}:{}", cmf.local_host, cmf.local_port));
let stem = m.model.trim();
if !local || stem.is_empty() || stem.contains('/') || stem.contains("..") {
return None; // not local, or an unsafe/underivable path
}
Some(std::path::Path::new(&cmf.models_dir).join(format!("{stem}.cmf")))
}
/// Drop a model id from routing (default + every tier) so `validate()` still
/// passes after the model is gone.
fn strip_id_from_routing(routing: &mut RoutingCfg, id: &str) {
if routing.default == id {
routing.default = String::new();
}
for tt in routing.tiers.values_mut() {
match tt {
crate::config::TierTargets::List(v) => v.retain(|x| x != id),
crate::config::TierTargets::One(s) if s == id => {
*tt = crate::config::TierTargets::List(Vec::new());
}
crate::config::TierTargets::One(_) => {}
}
}
}
async fn delete_model(State(state): State<SharedState>, Path(id): Path<String>) -> ApiResult {
let mut cfg = current_cfg(&state);
// The managed local model lives in [cmf], not [[models]]. Deleting it stops
// the running server, clears the managed config, and removes its file.
if id == cfg.cmf.model_id
&& cfg.cmf.manage_server
&& !cfg.cmf.local_model.trim().is_empty()
&& !cfg.models.iter().any(|x| x.id == id)
{
let file = cfg.cmf.local_model.clone();
cfg.cmf.local_model = String::new();
cfg.cmf.manage_server = false;
strip_id_from_routing(&mut cfg.routing, &id);
state.cmf.stop().await;
state.reload(cfg)?;
let file_removed = remove_cmf_path(std::path::Path::new(&file));
return ok(json!({ "ok": true, "file_removed": file_removed }));
}
let victim = cfg.models.iter().find(|x| x.id == id).cloned();
let before = cfg.models.len();
cfg.models.retain(|x| x.id != id);
if cfg.models.len() == before {
return Err(ApiError::not_found(format!("model '{id}' not found")));
}
strip_id_from_routing(&mut cfg.routing, &id);
let file = victim
.as_ref()
.and_then(|m| cmf_file_for_model(&cfg.cmf, m));
state.reload(cfg)?;
ok(json!({ "ok": true }))
// Physically remove the local .cmf after the config is persisted.
let file_removed = file.map(|p| remove_cmf_path(&p)).unwrap_or(false);
ok(json!({ "ok": true, "file_removed": file_removed }))
}
async fn probe_model(State(state): State<SharedState>, Path(id): Path<String>) -> ApiResult {
@@ -478,6 +697,7 @@ async fn get_settings(State(state): State<SharedState>) -> ApiResult {
"stats": c.stats,
"admin": c.admin,
"cache": c.cache,
"cmf": c.cmf,
}))
}
@@ -493,6 +713,7 @@ struct SettingsBody {
stats: Option<StatsCfg>,
admin: Option<AdminCfg>,
cache: Option<CacheCfg>,
cmf: Option<CmfCfg>,
}
async fn put_settings(State(state): State<SharedState>, Json(b): Json<SettingsBody>) -> ApiResult {
@@ -536,10 +757,128 @@ async fn put_settings(State(state): State<SharedState>, Json(b): Json<SettingsBo
needs_restart = true;
cfg.cache = v;
}
if let Some(v) = b.cmf {
// manage_server / local_model / threads / gpu apply on restart (the server
// is spawned once at startup with those env vars); local_only / router
// routing apply immediately.
if v.manage_server != cfg.cmf.manage_server
|| v.local_model != cfg.cmf.local_model
|| v.threads != cfg.cmf.threads
|| v.gpu != cfg.cmf.gpu
{
needs_restart = true;
}
cfg.cmf = v;
}
state.reload(cfg)?;
ok(json!({ "ok": true, "needs_restart": needs_restart }))
}
// ───────────────────────────── cmf runtime ─────────────────────────────────
/// Status of the `cortiq` CLI (the CMF format runtime): is it installed, which
/// version, and whether a managed server is running. Drives the "Install cortiq"
/// button + status in the Settings → Local models card.
async fn cmf_status(State(state): State<SharedState>) -> ApiResult {
let cmf = state.live().cfg.cmf.clone();
let ver = crate::cmf_runtime::installed_version(&cmf.cortiq_bin);
let st = state.cmf.status();
ok(json!({
"installed": ver.is_some(),
"version": ver,
"running": st.running,
"healthy": st.healthy,
"last_error": st.last_error,
"log": st.log,
// serving info so the card can show "serving <model> on :<port>"
"manage_server": cmf.manage_server,
"local_model": cmf.local_model,
"model_id": cmf.model_id,
"local_host": cmf.local_host,
"local_port": cmf.local_port,
}))
}
/// Install/upgrade the `cortiq` CLI from crates.io on demand (spawned in the
/// background; poll `GET /admin/api/cmf` for `installed`/`version`).
async fn cmf_install(State(state): State<SharedState>) -> ApiResult {
let bin = state.live().cfg.cmf.cortiq_bin.clone();
tokio::spawn(crate::cmf_runtime::install_now(state.cmf.clone(), bin));
ok(json!({ "started": true }))
}
#[derive(Deserialize)]
struct PortQuery {
port: u16,
}
/// Check whether a TCP port is free to bind on the CMF local host, so a user can
/// pick an available port before enabling the managed server (avoids clashing
/// with something already on the default 8081). If the port is the one this
/// gateway's own managed server already runs on, that is reported distinctly.
/// When occupied, `suggested` holds the next free port so the UI can offer it.
async fn cmf_port_check(State(state): State<SharedState>, Query(q): Query<PortQuery>) -> ApiResult {
let cmf = state.live().cfg.cmf.clone();
let host = if cmf.local_host.trim().is_empty() {
"127.0.0.1".to_string()
} else {
cmf.local_host.clone()
};
let st = state.cmf.status();
let port = q.port;
let running_here = port == cmf.local_port && st.running;
// The bind+connect probe is blocking std::net; run it off the async runtime.
let (available, detail, suggested) = tokio::task::spawn_blocking(move || {
if running_here {
return (false, "in use by this gateway's managed CMF server", None);
}
if port_is_free(&host, port) {
(true, "free", None)
} else {
(
false,
"occupied by another process",
next_free_port(&host, port),
)
}
})
.await
.unwrap_or((false, "port check failed", None));
ok(json!({
"port": port,
"available": available,
"detail": detail,
"suggested": suggested,
}))
}
/// Whether `port` is truly free on `host`. A bind test alone is not enough: a
/// Docker-published / dual-stack listener can leave an IPv4 bind succeeding even
/// though connections to the port are forwarded to a container. So a port counts
/// as free only if it is BOTH bindable AND nothing answers a connect attempt.
fn port_is_free(host: &str, port: u16) -> bool {
if std::net::TcpListener::bind((host, port)).is_err() {
return false; // something is bound to exactly this address
}
// catch proxied/forwarded ports a bind test misses (e.g. Docker on macOS)
let reachable = (host, port)
.to_socket_addrs()
.ok()
.and_then(|mut it| it.next())
.map(|addr| {
std::net::TcpStream::connect_timeout(&addr, std::time::Duration::from_millis(200))
.is_ok()
})
.unwrap_or(false);
!reachable
}
/// First free port in (from, from+50], or None if all are taken.
fn next_free_port(host: &str, from: u16) -> Option<u16> {
(from.saturating_add(1)..=from.saturating_add(50)).find(|p| port_is_free(host, *p))
}
// ───────────────────────────── keys ────────────────────────────────────────
async fn list_keys(State(state): State<SharedState>) -> ApiResult {
@@ -678,6 +1017,125 @@ async fn get_stats(State(state): State<SharedState>, Query(q): Query<StatsQuery>
ok(snap)
}
/// Self-warming shadow loop: per-task-type promotion state, Wilson-LB of the
/// judged pass-rate, and whether the local model is serving it yet.
async fn get_shadow(State(state): State<SharedState>) -> ApiResult {
let labels: Vec<Value> = state
.promotion
.snapshot()
.into_iter()
.map(|(label, st, n, pass_rate, lb)| {
json!({
"label": label,
"state": format!("{st:?}"),
"n": n,
"pass_rate": pass_rate,
"wilson_lb": lb,
"serves_local": state.promotion.serves_local(&label),
})
})
.collect();
ok(json!({ "enabled": state.promotion.enabled(), "labels": labels }))
}
// ── CMF model factory: HF search → convert → register ──
#[derive(Deserialize)]
struct HfQuery {
q: Option<String>,
limit: Option<usize>,
}
/// Proxy HuggingFace model search (server-side: avoids CORS, adds token).
async fn hf_search(State(state): State<SharedState>, Query(q): Query<HfQuery>) -> ApiResult {
let live = state.live();
let token = if live.cfg.cmf.hf_token_env.is_empty() {
None
} else {
state.secrets.resolve(&live.cfg.cmf.hf_token_env)
};
let query = q.q.unwrap_or_default();
let limit = q.limit.unwrap_or(24).min(50);
let models = crate::import::hf_search(&query, limit, token.as_deref())
.await
.map_err(ApiError::bad)?;
ok(json!({ "models": models }))
}
async fn list_imports(State(state): State<SharedState>) -> ApiResult {
ok(json!({ "jobs": state.imports.list() }))
}
/// Start a conversion job (returns the job id; progress via `import/:job`).
async fn start_import(
State(state): State<SharedState>,
Json(p): Json<crate::import::ImportParams>,
) -> ApiResult {
let live = state.live();
let id = crate::import::start_import(state.imports.clone(), &live.cfg.cmf, p)
.map_err(ApiError::bad)?;
ok(json!({ "job": id }))
}
async fn import_status(State(state): State<SharedState>, Path(job): Path<String>) -> ApiResult {
match state.imports.get(&job) {
Some(j) => ok(serde_json::to_value(j).unwrap_or_else(|_| json!({}))),
None => Err(ApiError::not_found(format!("job '{job}' not found"))),
}
}
/// Cancel a running conversion: kills the converter and removes partial output.
async fn cancel_import(State(state): State<SharedState>, Path(job): Path<String>) -> ApiResult {
if state.imports.get(&job).is_none() {
return Err(ApiError::not_found(format!("job '{job}' not found")));
}
let cancelled = state.imports.cancel(&job);
ok(json!({ "ok": cancelled }))
}
/// Delete a finished conversion and its converted `.cmf` file(s) from disk.
async fn delete_import(State(state): State<SharedState>, Path(job): Path<String>) -> ApiResult {
if state.imports.get(&job).is_none() {
return Err(ApiError::not_found(format!("job '{job}' not found")));
}
let removed = state.imports.delete(&job).map_err(ApiError::bad)?;
ok(json!({ "ok": removed }))
}
/// Register a finished `.cmf` as a local model (OpenAI provider → cortiq-server).
async fn register_import(State(state): State<SharedState>, Path(job): Path<String>) -> ApiResult {
let j = state
.imports
.get(&job)
.ok_or_else(|| ApiError::not_found(format!("job '{job}' not found")))?;
if j.state != "done" {
return Err(ApiError::bad("conversion not finished"));
}
let base = std::path::Path::new(&j.output)
.file_stem()
.map(|s| s.to_string_lossy().to_string())
.unwrap_or_else(|| "cmf-local".into());
let id = format!("cmf-{base}");
let mut cfg = current_cfg(&state);
if cfg.models.iter().any(|m| m.id == id) {
return Err(ApiError::bad(format!("model '{id}' already registered")));
}
cfg.models.push(crate::config::ModelCfg {
id: id.clone(),
provider: "openai".into(),
base_url: cfg.cmf.cortiq_server_url.clone(),
model: base,
cost_tier: "local".into(),
price_in: 0.0,
price_out: 0.0,
kind: "chat".into(),
api_key_env: None,
caps: Vec::new(),
});
state.reload(cfg)?;
ok(json!({ "ok": true, "model_id": id }))
}
#[derive(Deserialize)]
struct RecentQuery {
#[serde(default)]
+311
View File
@@ -0,0 +1,311 @@
//! Local CMF model server lifecycle.
//!
//! Closes the loop with <https://github.com/infosave2007/cmf>: install the
//! `cortiq` CLI from crates.io, keep it up to date, and run `cortiq serve` as a
//! managed child process. The served model exposes an OpenAI-compatible API, so
//! the registry registers it as an ordinary provider and the gateway routes to
//! it like any other backend.
use crate::config::CmfCfg;
use std::process::Stdio;
use std::sync::{Arc, Mutex};
use std::time::Duration;
use tokio::process::{Child, Command};
/// Observable state of the managed local CMF server (surfaced to the admin API).
#[derive(Clone, Debug, Default, serde::Serialize)]
pub struct CmfStatus {
/// `manage_server` is on and a `local_model` is configured.
pub enabled: bool,
/// Version reported by `cortiq --version` (None = not installed).
pub installed_version: Option<String>,
/// Newest `cortiq-cli` on crates.io (populated when `auto_update` runs).
pub latest_version: Option<String>,
/// The child `cortiq serve` process has been spawned.
pub running: bool,
/// `/healthz` on the local server answered 200.
pub healthy: bool,
/// OpenAI base URL the local server is registered under.
pub base_url: String,
/// Path of the `.cmf` being served.
pub model: String,
/// Last fatal error, if any.
pub last_error: Option<String>,
/// Recent lifecycle log lines (bounded).
pub log: Vec<String>,
}
/// Holds the managed child process and its live status.
pub struct CmfRuntime {
status: Mutex<CmfStatus>,
child: Mutex<Option<Child>>,
}
impl CmfRuntime {
pub fn new() -> Arc<Self> {
Arc::new(Self {
status: Mutex::new(CmfStatus::default()),
child: Mutex::new(None),
})
}
pub fn status(&self) -> CmfStatus {
self.status.lock().unwrap().clone()
}
fn log(&self, msg: impl Into<String>) {
let msg = msg.into();
tracing::info!(target: "cmf", "{msg}");
let mut s = self.status.lock().unwrap();
s.log.push(msg);
let n = s.log.len();
if n > 50 {
s.log.drain(0..n - 50);
}
}
fn fail(&self, msg: impl Into<String>) {
let msg = msg.into();
tracing::warn!(target: "cmf", "{msg}");
let mut s = self.status.lock().unwrap();
s.last_error = Some(msg.clone());
s.log.push(format!("✗ {msg}"));
}
fn set_child(&self, child: Child) {
*self.child.lock().unwrap() = Some(child);
self.status.lock().unwrap().running = true;
}
/// Stop the managed server (used on shutdown / reconfigure).
pub async fn stop(&self) {
if let Some(mut child) = self.child.lock().unwrap().take() {
let _ = child.start_kill();
}
let mut s = self.status.lock().unwrap();
s.running = false;
s.healthy = false;
}
}
impl Default for CmfRuntime {
fn default() -> Self {
Self {
status: Mutex::new(CmfStatus::default()),
child: Mutex::new(None),
}
}
}
/// crates.io `newest_version` for `cortiq-cli`.
async fn latest_crates_version() -> Option<String> {
let client = reqwest::Client::builder()
.timeout(Duration::from_secs(10))
.build()
.ok()?;
let body = client
.get("https://crates.io/api/v1/crates/cortiq-cli")
.header("User-Agent", "cortiq-gateway")
.send()
.await
.ok()?
.text()
.await
.ok()?;
let v: serde_json::Value = serde_json::from_str(&body).ok()?;
v["crate"]["newest_version"].as_str().map(|s| s.to_string())
}
/// `<bin> --version` → the trailing version token (e.g. "cortiq 0.1.2" → "0.1.2").
pub(crate) fn installed_version(bin: &str) -> Option<String> {
let out = std::process::Command::new(bin)
.arg("--version")
.output()
.ok()?;
if !out.status.success() {
return None;
}
let s = String::from_utf8_lossy(&out.stdout);
s.split_whitespace().last().map(|s| s.trim().to_string())
}
fn parse_ver(s: &str) -> (u32, u32, u32) {
let mut it = s.split('.').map(|x| x.trim().parse::<u32>().unwrap_or(0));
(
it.next().unwrap_or(0),
it.next().unwrap_or(0),
it.next().unwrap_or(0),
)
}
fn version_lt(a: &str, b: &str) -> bool {
parse_ver(a) < parse_ver(b)
}
/// `cargo install cortiq-cli [--force]` — blocking compile, run off-thread.
async fn cargo_install(force: bool) -> Result<(), String> {
let mut cmd = Command::new("cargo");
cmd.args(["install", "cortiq-cli", "--locked"]);
if force {
cmd.arg("--force");
}
let out = cmd.output().await.map_err(|e| e.to_string())?;
if out.status.success() {
Ok(())
} else {
Err(String::from_utf8_lossy(&out.stderr).trim().to_string())
}
}
/// Install/upgrade `cortiq-cli` from crates.io on demand (the admin "Install
/// cortiq" button). Runs in the background; progress + result land in the CMF
/// status log, and `installed_version` is refreshed on success.
pub async fn install_now(rt: Arc<CmfRuntime>, bin: String) {
rt.log("installing cortiq-cli from crates.io (this compiles — a few minutes)…");
match cargo_install(true).await {
Ok(()) => {
let v = installed_version(&bin);
rt.status.lock().unwrap().installed_version = v.clone();
rt.log(format!(
"installed cortiq-cli {}",
v.as_deref().unwrap_or("?")
));
}
Err(e) => rt.fail(format!("cargo install cortiq-cli failed: {e}")),
}
}
/// Spawn `cortiq serve <model> --host <host> --port <port>`; the child is killed
/// when its handle is dropped (i.e. when the gateway shuts down).
fn spawn_serve(cfg: &CmfCfg) -> Result<Child, String> {
let mut cmd = Command::new(&cfg.cortiq_bin);
cmd.arg("serve")
.arg(&cfg.local_model)
.arg("--host")
.arg(&cfg.local_host)
.arg("--port")
.arg(cfg.local_port.to_string());
// Parallelise decode matvecs across cores (CMF_THREADS). 0 = leave the
// runtime's own default; N>1 gave ~2.8x on a 15B CPU model.
if cfg.threads > 1 {
cmd.env("CMF_THREADS", cfg.threads.to_string());
}
// Offload large matvecs to the GPU (Metal on macOS). Memory-bandwidth-bound
// on unified memory, so mostly a prefill win; opt-in.
if cfg.gpu {
cmd.env("CMF_GPU", "1");
}
cmd.stdout(Stdio::null())
.stderr(Stdio::null())
.kill_on_drop(true)
.spawn()
.map_err(|e| e.to_string())
}
/// Poll `/healthz` until the local server answers 200 (or the deadline passes).
async fn wait_healthy(host: &str, port: u16, tries: u32) -> bool {
let url = format!("http://{host}:{port}/healthz");
let client = reqwest::Client::new();
for _ in 0..tries {
if let Ok(r) = client
.get(&url)
.timeout(Duration::from_secs(2))
.send()
.await
{
if r.status().is_success() {
return true;
}
}
tokio::time::sleep(Duration::from_millis(500)).await;
}
false
}
/// Orchestrate the whole lifecycle: ensure installed → maybe update → spawn →
/// wait healthy. Safe to call in a background task; it never panics.
pub async fn manage(rt: Arc<CmfRuntime>, cfg: CmfCfg) {
if !cfg.manage_server || cfg.local_model.trim().is_empty() {
return;
}
{
let mut s = rt.status.lock().unwrap();
s.enabled = true;
s.base_url = format!("http://{}:{}/v1", cfg.local_host, cfg.local_port);
s.model = cfg.local_model.clone();
}
if !std::path::Path::new(&cfg.local_model).exists() {
rt.fail(format!("local_model not found: {}", cfg.local_model));
return;
}
// 1. Ensure the `cortiq` binary is available.
let mut ver = installed_version(&cfg.cortiq_bin);
if ver.is_none() {
if cfg.auto_install {
rt.log("cortiq not found — installing cortiq-cli from crates.io (this compiles, may take a few minutes)…");
if let Err(e) = cargo_install(false).await {
rt.fail(format!("cargo install cortiq-cli failed: {e}"));
return;
}
ver = installed_version(&cfg.cortiq_bin);
if ver.is_some() {
rt.log("installed cortiq-cli");
}
} else {
rt.fail(
"cortiq binary not found (set [cmf].auto_install = true, or install cortiq-cli)",
);
return;
}
}
rt.status.lock().unwrap().installed_version = ver.clone();
// 2. Optionally update to the newest crates.io release.
if cfg.auto_update {
if let Some(latest) = latest_crates_version().await {
rt.status.lock().unwrap().latest_version = Some(latest.clone());
if ver
.as_deref()
.map(|c| version_lt(c, &latest))
.unwrap_or(false)
{
rt.log(format!(
"updating cortiq-cli {} → {latest}…",
ver.as_deref().unwrap_or("?")
));
if let Err(e) = cargo_install(true).await {
rt.log(format!("update failed (keeping current): {e}"));
} else {
ver = installed_version(&cfg.cortiq_bin);
rt.status.lock().unwrap().installed_version = ver.clone();
rt.log(format!("updated to {}", ver.as_deref().unwrap_or("?")));
}
}
}
}
// 3. Spawn the local server.
rt.log(format!(
"starting: cortiq serve {} --host {} --port {}",
cfg.local_model, cfg.local_host, cfg.local_port
));
match spawn_serve(&cfg) {
Ok(child) => rt.set_child(child),
Err(e) => {
rt.fail(format!("failed to spawn cortiq serve: {e}"));
return;
}
}
// 4. Wait until it is serving (model load can take a while for big models).
if wait_healthy(&cfg.local_host, cfg.local_port, 240).await {
rt.status.lock().unwrap().healthy = true;
rt.log(format!(
"local CMF server ready — registered as model '{}' at http://{}:{}/v1",
cfg.model_id, cfg.local_host, cfg.local_port
));
} else {
rt.fail("local CMF server did not become healthy in time");
}
}
+245 -5
View File
@@ -3,6 +3,34 @@
use serde::{Deserialize, Serialize};
use std::collections::HashMap;
use std::path::PathBuf;
/// Base directory for gateway-managed data (models, logs). Honors
/// `$CORTIQ_GATEWAY_HOME`, then `$XDG_DATA_HOME/cortiq-gateway`, then
/// `~/.cortiq-gateway` — so defaults work regardless of the process CWD
/// (important once the gateway is installed rather than run from its repo).
pub fn data_dir() -> PathBuf {
for (var, sub) in [
("CORTIQ_GATEWAY_HOME", ""),
("XDG_DATA_HOME", "cortiq-gateway"),
("HOME", ".cortiq-gateway"),
] {
if let Ok(v) = std::env::var(var) {
if !v.is_empty() {
return if sub.is_empty() {
PathBuf::from(v)
} else {
PathBuf::from(v).join(sub)
};
}
}
}
PathBuf::from("data")
}
fn data_path(name: &str) -> String {
data_dir().join(name).to_string_lossy().into_owned()
}
#[derive(Clone, Debug, Serialize, Deserialize)]
pub struct Config {
@@ -33,10 +61,131 @@ pub struct Config {
pub stats: StatsCfg,
#[serde(default)]
pub cache: CacheCfg,
#[serde(default)]
pub shadow: ShadowCfg,
#[serde(default)]
pub cmf: CmfCfg,
}
/// CMF-format model factory: import a HuggingFace model, convert it to a
/// local `.cmf` (quantized), and serve it via cortiq-server. Points the
/// gateway at the Python converter + runtime on the host.
#[derive(Clone, Debug, Serialize, Deserialize)]
pub struct CmfCfg {
/// Python interpreter that has torch + the converter deps.
#[serde(default = "default_cmf_python")]
pub python_bin: String,
/// Path to `convert_dtgma_to_cmf.py`.
#[serde(default = "default_cmf_converter")]
pub converter: String,
/// Directory where produced `.cmf` files are written.
#[serde(default = "default_cmf_models_dir")]
pub models_dir: String,
/// Base URL of the cortiq-server that serves the `.cmf` (OpenAI /v1),
/// used when registering a converted model as a provider.
#[serde(default = "default_cmf_server_url")]
pub cortiq_server_url: String,
/// Secret name of a HuggingFace token (higher search rate limits +
/// gated/private repos). Empty = anonymous public search.
#[serde(default)]
pub hf_token_env: String,
/// `cortiq` CLI binary for the local server (from `cargo install cortiq-cli`).
#[serde(default = "default_cortiq_bin")]
pub cortiq_bin: String,
/// Install cortiq-cli from crates.io automatically if the binary is missing.
#[serde(default)]
pub auto_install: bool,
/// On startup, check crates.io and reinstall if a newer cortiq-cli exists.
#[serde(default)]
pub auto_update: bool,
/// Spawn and manage a local `cortiq serve` for `local_model`, and register
/// it in the model pool — a local `.cmf` backend the gateway can use even
/// when no external provider (or the router) is available.
#[serde(default)]
pub manage_server: bool,
/// Path to the local `.cmf` to serve (empty = disabled).
#[serde(default)]
pub local_model: String,
/// Host the managed local server binds to.
#[serde(default = "default_cmf_local_host")]
pub local_host: String,
/// Port the managed local server binds to.
#[serde(default = "default_cmf_local_port")]
pub local_port: u16,
/// Model id the local server is registered under in the pool.
#[serde(default = "default_cmf_model_id")]
pub model_id: String,
/// Use ONLY local models: route every request to the local model, ignoring
/// the router and any external providers ("use only local models").
#[serde(default)]
pub local_only: bool,
/// Worker threads for the managed `cortiq serve` (exported as `CMF_THREADS`).
/// 0 = let the runtime decide; N>1 parallelises decode matvecs across N cores
/// (measured ~2.8x faster than serial on a 15B model). Default 8.
#[serde(default = "default_cmf_threads")]
pub threads: usize,
/// Offload large matvecs to the GPU (exports CMF_GPU=1). On Apple Silicon the
/// decode loop is memory-bandwidth bound, so this is ~neutral for generation
/// (helps prefill of long prompts). Off by default.
#[serde(default)]
pub gpu: bool,
}
impl Default for CmfCfg {
fn default() -> Self {
Self {
python_bin: default_cmf_python(),
converter: default_cmf_converter(),
models_dir: default_cmf_models_dir(),
cortiq_server_url: default_cmf_server_url(),
hf_token_env: String::new(),
cortiq_bin: default_cortiq_bin(),
auto_install: false,
auto_update: false,
manage_server: false,
local_model: String::new(),
local_host: default_cmf_local_host(),
local_port: default_cmf_local_port(),
model_id: default_cmf_model_id(),
local_only: false,
threads: default_cmf_threads(),
gpu: false,
}
}
}
fn default_cmf_threads() -> usize {
8
}
fn default_cortiq_bin() -> String {
"cortiq".into()
}
fn default_cmf_local_host() -> String {
"127.0.0.1".into()
}
fn default_cmf_local_port() -> u16 {
8081
}
fn default_cmf_model_id() -> String {
"cmf-local".into()
}
fn default_cmf_python() -> String {
"python3".into()
}
fn default_cmf_converter() -> String {
"../cmf/converter/convert_dtgma_to_cmf.py".into()
}
fn default_cmf_models_dir() -> String {
data_path("models")
}
fn default_cmf_server_url() -> String {
"http://127.0.0.1:8081/v1".into()
}
#[derive(Clone, Debug, Serialize, Deserialize)]
pub struct RouterCfg {
/// Master switch — when false the router is never called; the gateway
/// serves from the local model pool (local `.cmf` / configured default).
#[serde(default = "default_router_enabled")]
pub enabled: bool,
pub url: String,
#[serde(default = "default_router_timeout")]
pub timeout_ms: u64,
@@ -47,6 +196,9 @@ pub struct RouterCfg {
#[serde(default, skip_serializing_if = "Option::is_none")]
pub taxonomy_id: Option<String>,
}
fn default_router_enabled() -> bool {
true
}
fn default_router_timeout() -> u64 {
800
}
@@ -313,7 +465,7 @@ fn default_stats_enabled() -> bool {
true
}
fn default_stats_file() -> String {
"config/stats.jsonl".into()
data_path("stats.jsonl")
}
fn default_stats_ring() -> usize {
500
@@ -322,6 +474,82 @@ fn default_stats_retention() -> String {
"7d".into()
}
/// Self-warming shadow loop (docs/SELF_WARMING_GATEWAY.md). While a task
/// label is not yet promoted, the client is served the CLOUD answer; in
/// parallel (sampled, non-blocking) the local `cmf-local` model answers and
/// a cheap `judge` model scores it against the cloud answer. The per-label
/// Wilson-LB of that pass stream gates promotion to local serving.
#[derive(Clone, Debug, Serialize, Deserialize)]
pub struct ShadowCfg {
/// Master switch. Off = zero overhead, classic gateway behavior.
#[serde(default)]
pub enabled: bool,
/// model_id (from `[[models]]`) of the local CMF model (OpenAI-compatible
/// cortiq-server). Empty disables the loop.
#[serde(default)]
pub local_model_id: String,
/// model_id of the cheap judge (different family than the answerer).
#[serde(default)]
pub judge_model_id: String,
/// Fraction of eligible requests to shadow+judge (0..1). Bounds cost.
#[serde(default = "default_shadow_sample")]
pub sample_rate: f64,
/// judge.jsonl append+replay path. Empty = in-memory only.
#[serde(default = "default_shadow_file")]
pub file: String,
/// Wilson-95 lower bound of pass-rate required to promote.
#[serde(default = "default_shadow_lb")]
pub promote_lb: f64,
/// Min judgments before promotion is considered.
#[serde(default = "default_shadow_nmin")]
pub n_min: usize,
/// Rolling window of judgments per label.
#[serde(default = "default_shadow_window")]
pub window: usize,
/// Extra canary judgments before full local serving.
#[serde(default = "default_shadow_soak")]
pub soak: usize,
/// GATE past shadow-only: actually SERVE promoted labels from the local
/// model (with failover to cloud on error + a complexity veto). Default
/// OFF — the memo's 3 risks (§8) should be retired by measurement first.
#[serde(default)]
pub serve_when_promoted: bool,
}
impl Default for ShadowCfg {
fn default() -> Self {
Self {
enabled: false,
local_model_id: String::new(),
judge_model_id: String::new(),
sample_rate: default_shadow_sample(),
file: default_shadow_file(),
promote_lb: default_shadow_lb(),
n_min: default_shadow_nmin(),
window: default_shadow_window(),
soak: default_shadow_soak(),
serve_when_promoted: false,
}
}
}
fn default_shadow_sample() -> f64 {
0.15
}
fn default_shadow_file() -> String {
data_path("judge.jsonl")
}
fn default_shadow_lb() -> f64 {
0.95
}
fn default_shadow_nmin() -> usize {
200
}
fn default_shadow_window() -> usize {
500
}
fn default_shadow_soak() -> usize {
100
}
/// Semantic (embedding-based) response cache — returns a cached answer for prompts
/// that are semantically near a previous one, skipping the (expensive) model call.
#[derive(Clone, Debug, Serialize, Deserialize)]
@@ -374,8 +602,17 @@ impl Config {
/// Integrity validation: non-empty model pool, valid routing targets.
pub fn validate(&self) -> anyhow::Result<()> {
if self.models.is_empty() {
anyhow::bail!("config: at least one [[models]] entry is required");
// A managed local CMF model counts as an available model even though it
// is registered dynamically rather than via a [[models]] entry — this
// is what lets the gateway run on local models alone (no cloud, no router).
let has_local = self.cmf.manage_server && !self.cmf.local_model.trim().is_empty();
if self.models.is_empty() && !has_local {
// Fresh install: start with an EMPTY model pool so the admin console
// comes up — the user adds models in /admin → Models (or points [cmf]
// at a local .cmf). Routing simply has nothing to select until then.
tracing::warn!(
"no models configured — starting with an empty pool; add one in /admin → Models"
);
}
// uniqueness of model ids
let mut seen = std::collections::HashSet::new();
@@ -385,9 +622,12 @@ impl Config {
}
}
// every target in [routing] must exist in the model pool
let ids: std::collections::HashSet<&str> =
let mut ids: std::collections::HashSet<&str> =
self.models.iter().map(|m| m.id.as_str()).collect();
if !ids.contains(self.routing.default.as_str()) {
if has_local {
ids.insert(self.cmf.model_id.as_str());
}
if !self.routing.default.is_empty() && !ids.contains(self.routing.default.as_str()) {
anyhow::bail!(
"config: routing.default '{}' is not a known model id",
self.routing.default
+473
View File
@@ -0,0 +1,473 @@
//! CMF model factory: search HuggingFace, then convert a chosen repo to a
//! local quantized `.cmf` via the Python converter — as a tracked, streamed
//! background job so the admin UI can show live progress.
use crate::config::CmfCfg;
use serde::{Deserialize, Serialize};
use std::collections::HashMap;
use std::process::Stdio;
use std::sync::{Arc, Mutex};
use std::time::{SystemTime, UNIX_EPOCH};
use tokio::io::{AsyncBufReadExt, BufReader};
use tokio::sync::Notify;
/// Parse a `@PROGRESS <0..1> <phase>` marker emitted by the converter.
fn parse_progress(line: &str) -> Option<(f32, String)> {
let rest = line.strip_prefix("@PROGRESS ")?;
let mut it = rest.splitn(2, ' ');
let frac: f32 = it.next()?.trim().parse().ok()?;
let phase = it.next().unwrap_or("").trim().to_string();
Some((frac.clamp(0.0, 1.0), phase))
}
/// Remove a partial/aborted output — `.cmf` may be a file or a tensor-dir.
fn cleanup_output(path: &str) {
let p = std::path::Path::new(path);
if p.is_dir() {
let _ = std::fs::remove_dir_all(p);
} else if p.exists() {
let _ = std::fs::remove_file(p);
}
}
fn now() -> u64 {
SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap_or_default()
.as_secs()
}
/// Job id from a hash of (repo, nanos) — unique without a uuid dependency.
fn gen_id(repo: &str) -> String {
use std::hash::{Hash, Hasher};
let nanos = SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap_or_default()
.as_nanos();
let mut h = std::collections::hash_map::DefaultHasher::new();
repo.hash(&mut h);
nanos.hash(&mut h);
format!("{:016x}", h.finish())
}
/// Parameters chosen in the import wizard.
#[derive(Clone, Debug, Deserialize)]
pub struct ImportParams {
pub repo: String,
#[serde(default = "default_quant")]
pub quant: String,
#[serde(default)]
pub name: String, // output basename; empty → derived from repo
// ── advanced ──
#[serde(default)]
pub linear_core: Option<String>, // gated_delta_net | vmf_phase
#[serde(default)]
pub nphase: Option<u32>,
#[serde(default)]
pub vbit_shape: Option<String>, // log2 | cubic
#[serde(default)]
pub mean_bits: Option<f32>,
#[serde(default)]
pub shard_max_gb: Option<f32>,
#[serde(default)]
pub skip_mtp: bool,
}
fn default_quant() -> String {
"Q8_2F".into()
}
/// Live state of one conversion job.
#[derive(Clone, Debug, Serialize)]
pub struct Job {
pub id: String,
pub repo: String,
pub quant: String,
pub output: String,
pub state: String, // running | done | error | cancelled
pub log: Vec<String>,
pub started: u64,
pub finished: Option<u64>,
pub size_bytes: Option<u64>,
pub progress: Option<f32>, // 0..1, parsed from converter @PROGRESS
pub phase: Option<String>, // current phase label
}
#[derive(Default)]
pub struct JobStore {
jobs: Mutex<HashMap<String, Job>>,
/// Cancel handles, keyed by job id (not serialized). Notifying aborts the
/// converter process and cleans up its partial output.
cancels: Mutex<HashMap<String, Arc<Notify>>>,
}
impl JobStore {
pub fn new() -> Arc<Self> {
Arc::new(Self::default())
}
fn insert(&self, job: Job) {
self.jobs.lock().unwrap().insert(job.id.clone(), job);
}
fn set_cancel(&self, id: &str, n: Arc<Notify>) {
self.cancels.lock().unwrap().insert(id.to_string(), n);
}
fn drop_cancel(&self, id: &str) {
self.cancels.lock().unwrap().remove(id);
}
/// Signal a running job to abort. Returns false if the job isn't cancellable.
pub fn cancel(&self, id: &str) -> bool {
let running = matches!(
self.jobs.lock().unwrap().get(id).map(|j| j.state.as_str()),
Some("running")
);
if !running {
return false;
}
if let Some(n) = self.cancels.lock().unwrap().get(id) {
n.notify_one();
true
} else {
false
}
}
fn push_line(&self, id: &str, line: String) {
if let Some(j) = self.jobs.lock().unwrap().get_mut(id) {
j.log.push(line);
let n = j.log.len();
if n > 200 {
j.log.drain(0..n - 200); // keep the tail
}
}
}
fn set_progress(&self, id: &str, frac: f32, phase: String) {
if let Some(j) = self.jobs.lock().unwrap().get_mut(id) {
j.progress = Some(frac);
if !phase.is_empty() {
j.phase = Some(phase);
}
}
}
fn set_state(&self, id: &str, state: &str) {
if let Some(j) = self.jobs.lock().unwrap().get_mut(id) {
j.state = state.into();
j.finished = Some(now());
}
}
fn finish(&self, id: &str, ok: bool) {
if let Some(j) = self.jobs.lock().unwrap().get_mut(id) {
if j.state == "cancelled" {
return; // a concurrent cancel already settled this job
}
j.finished = Some(now());
j.size_bytes = std::fs::metadata(&j.output).ok().map(|m| m.len());
let done = ok && j.size_bytes.unwrap_or(0) > 0;
j.state = if done { "done" } else { "error" }.into();
if done {
j.progress = Some(1.0);
}
}
}
pub fn get(&self, id: &str) -> Option<Job> {
self.jobs.lock().unwrap().get(id).cloned()
}
/// Delete a finished job together with its converted output file(s).
/// Refuses a running job (cancel it first). `Ok(false)` if unknown.
pub fn delete(&self, id: &str) -> Result<bool, String> {
let job = self.jobs.lock().unwrap().get(id).cloned();
let Some(job) = job else { return Ok(false) };
if job.state == "running" {
return Err("cancel the running job before deleting".into());
}
cleanup_output(&job.output);
self.cancels.lock().unwrap().remove(id);
self.jobs.lock().unwrap().remove(id);
Ok(true)
}
pub fn list(&self) -> Vec<Job> {
let mut v: Vec<Job> = self.jobs.lock().unwrap().values().cloned().collect();
v.sort_by_key(|x| std::cmp::Reverse(x.started));
v.truncate(20);
v
}
}
/// Proxy HuggingFace model search (avoids browser CORS + adds our token).
pub async fn hf_search(
query: &str,
limit: usize,
token: Option<&str>,
) -> Result<serde_json::Value, String> {
let q: String = url_escape(query);
let sort = if query.trim().is_empty() {
"trendingScore"
} else {
"downloads"
};
let url = format!(
"https://huggingface.co/api/models?search={q}&sort={sort}&direction=-1&limit={limit}&full=false"
);
let client = reqwest::Client::builder()
.timeout(std::time::Duration::from_secs(15))
.build()
.map_err(|e| e.to_string())?;
let mut req = client.get(&url).header("User-Agent", "cortiq-gateway");
if let Some(t) = token {
req = req.bearer_auth(t);
}
let resp = req.send().await.map_err(|e| e.to_string())?;
if !resp.status().is_success() {
return Err(format!("HuggingFace API {}", resp.status()));
}
resp.json().await.map_err(|e| e.to_string())
}
fn url_escape(s: &str) -> String {
s.chars()
.map(|c| match c {
'a'..='z' | 'A'..='Z' | '0'..='9' | '-' | '_' | '.' | '/' => c.to_string(),
' ' => "+".to_string(),
_ => format!("%{:02X}", c as u32),
})
.collect()
}
fn sanitize(s: &str) -> String {
s.chars()
.map(|c| {
if c.is_ascii_alphanumeric() || c == '-' || c == '_' {
c
} else {
'-'
}
})
.collect()
}
/// Kick off a conversion. Returns the job id immediately; progress streams
/// Map the UI quant label to the native `cortiq` quant flag.
fn map_quant(q: &str) -> &'static str {
match q.to_ascii_uppercase().as_str() {
s if s.starts_with("Q4") => "q4",
"Q8_2F" => "q8_2f",
"F16" | "FP16" => "f16",
"VBIT" => "vbit",
_ => "q8",
}
}
/// into the job log. Fails fast if the converter script is missing.
pub fn start_import(store: Arc<JobStore>, cfg: &CmfCfg, p: ImportParams) -> Result<String, String> {
if p.repo.trim().is_empty() {
return Err("empty repo id".into());
}
// Native `cortiq convert` (from crates.io, no Python) handles standard + MoE
// models; the bundled Python converter is used only for advanced options it
// doesn't support yet (linear-attention folding, v-bit shaping).
let use_python = p.linear_core.is_some() || p.vbit_shape.is_some() || p.mean_bits.is_some();
if use_python && !std::path::Path::new(&cfg.converter).exists() {
return Err(format!(
"advanced options need the Python converter, not found: {}",
cfg.converter
));
}
let _ = std::fs::create_dir_all(&cfg.models_dir);
let base = if p.name.trim().is_empty() {
sanitize(p.repo.rsplit('/').next().unwrap_or(&p.repo))
} else {
sanitize(&p.name)
};
let output = std::path::Path::new(&cfg.models_dir)
.join(format!("{base}.cmf"))
.to_string_lossy()
.to_string();
let output_abs = std::fs::canonicalize(&cfg.models_dir)
.map(|d| d.join(format!("{base}.cmf")).to_string_lossy().to_string())
.unwrap_or_else(|_| output.clone());
let id = gen_id(&p.repo);
store.insert(Job {
id: id.clone(),
repo: p.repo.clone(),
quant: p.quant.clone(),
output: output_abs.clone(),
state: "running".into(),
log: vec![format!("→ converting {} to {} ({})", p.repo, base, p.quant)],
started: now(),
finished: None,
size_bytes: None,
progress: Some(0.0),
phase: Some("starting".into()),
});
let cancel = Arc::new(Notify::new());
store.set_cancel(&id, cancel.clone());
let hf_token = if cfg.hf_token_env.is_empty() {
None
} else {
std::env::var(&cfg.hf_token_env).ok()
};
// Program + args: `cortiq convert` by default, or the Python converter for
// advanced options. Both stream `@PROGRESS <frac>` markers into the log.
let (program, args, workdir): (String, Vec<String>, std::path::PathBuf) = if use_python {
let conv = std::path::Path::new(&cfg.converter);
let mut a = vec![
std::fs::canonicalize(conv)
.map(|p| p.to_string_lossy().to_string())
.unwrap_or_else(|_| cfg.converter.clone()),
"--model".into(),
p.repo.clone(),
"--quant".into(),
p.quant.clone(),
"--output".into(),
output_abs.clone(),
];
if let Some(lc) = &p.linear_core {
a.push("--linear-core".into());
a.push(lc.clone());
}
if let Some(n) = p.nphase {
a.push("--nphase".into());
a.push(n.to_string());
}
if let Some(vs) = &p.vbit_shape {
a.push("--vbit-shape".into());
a.push(vs.clone());
}
if let Some(mb) = p.mean_bits {
a.push("--mean-bits".into());
a.push(mb.to_string());
}
if let Some(g) = p.shard_max_gb {
a.push("--shard-max-gb".into());
a.push(g.to_string());
}
if p.skip_mtp {
a.push("--skip-mtp".into());
}
let wd = conv
.parent()
.map(|d| d.to_path_buf())
.unwrap_or_else(|| std::path::PathBuf::from("."));
(cfg.python_bin.clone(), a, wd)
} else if p.repo.to_ascii_lowercase().contains("gguf") {
// GGUF repo → native `cortiq import-gguf` (downloads + dequantizes any
// common ggml quant type: Q4_0/1, Q5_0/1, Q8_0, Q2_K..Q6_K, IQ4_NL/XS).
let mut a = vec![
"import-gguf".into(),
p.repo.clone(),
"--quant".into(),
map_quant(&p.quant).into(),
"--output".into(),
output_abs.clone(),
];
if let Some(t) = &hf_token {
a.push("--hf-token".into());
a.push(t.clone());
}
(
cfg.cortiq_bin.clone(),
a,
std::env::current_dir().unwrap_or_else(|_| std::path::PathBuf::from(".")),
)
} else {
// safetensors → native `cortiq convert` (dense / MoE / GatedDeltaNet).
let mut a = vec![
"convert".into(),
"--model".into(),
p.repo.clone(),
"--quant".into(),
map_quant(&p.quant).into(),
"--output".into(),
output_abs.clone(),
];
if let Some(t) = &hf_token {
a.push("--hf-token".into());
a.push(t.clone());
}
(
cfg.cortiq_bin.clone(),
a,
std::env::current_dir().unwrap_or_else(|_| std::path::PathBuf::from(".")),
)
};
let ret_id = id.clone();
let output_for_task = output_abs.clone();
tokio::spawn(async move {
let mut cmd = tokio::process::Command::new(&program);
cmd.args(&args)
.current_dir(&workdir)
.env("PYTHONUNBUFFERED", "1")
.stdout(Stdio::piped())
.stderr(Stdio::piped());
// HF token for the Python converter (native cortiq gets --hf-token).
if use_python {
if let Some(tok) = hf_token {
cmd.env("HF_TOKEN", &tok).env("HUGGING_FACE_HUB_TOKEN", tok);
}
}
let mut child = match cmd.spawn() {
Ok(c) => c,
Err(e) => {
store.push_line(&id, format!("✗ spawn failed: {e}"));
store.finish(&id, false);
store.drop_cancel(&id);
return;
}
};
let stdout = child.stdout.take();
let stderr = child.stderr.take();
// stdout: parse @PROGRESS markers into progress/phase; everything else logs.
let s1 = store.clone();
let id1 = id.clone();
let t_out = tokio::spawn(async move {
if let Some(o) = stdout {
let mut lines = BufReader::new(o).lines();
while let Ok(Some(l)) = lines.next_line().await {
if let Some((frac, phase)) = parse_progress(&l) {
s1.set_progress(&id1, frac, phase);
} else {
s1.push_line(&id1, l);
}
}
}
});
let s2 = store.clone();
let id2 = id.clone();
let t_err = tokio::spawn(async move {
if let Some(e) = stderr {
let mut lines = BufReader::new(e).lines();
while let Ok(Some(l)) = lines.next_line().await {
s2.push_line(&id2, l);
}
}
});
tokio::select! {
status = child.wait() => {
let _ = tokio::join!(t_out, t_err);
let ok = status.map(|s| s.success()).unwrap_or(false);
if !ok {
cleanup_output(&output_for_task); // drop partial/corrupt output
}
store.push_line(
&id,
if ok { "✓ done".into() } else { "✗ converter exited with error".into() },
);
store.finish(&id, ok);
}
_ = cancel.notified() => {
let _ = child.start_kill();
let _ = child.wait().await;
t_out.abort();
t_err.abort();
cleanup_output(&output_for_task);
store.push_line(&id, "✗ cancelled by user — partial output removed".into());
store.set_state(&id, "cancelled");
}
}
store.drop_cancel(&id);
});
Ok(ret_id)
}
+13
View File
@@ -10,10 +10,13 @@
mod admin;
mod cache;
mod cmf_runtime;
mod config;
mod error;
mod import;
mod model;
mod pipeline;
mod promotion;
mod protocols;
mod providers;
mod registry;
@@ -79,6 +82,16 @@ async fn main() -> anyhow::Result<()> {
let state = SharedState::build(cfg, args.config.clone())?;
// Close the loop with the CMF format (github.com/infosave2007/cmf): install
// / update cortiq-cli and run a local `cortiq serve` if configured. Runs in
// the background so startup is never blocked by a cargo install or a model
// load; the served model is registered in the pool and usable offline.
{
let cmf_rt = state.cmf.clone();
let cmf_cfg = state.live().cfg.cmf.clone();
tokio::spawn(async move { cmf_runtime::manage(cmf_rt, cmf_cfg).await });
}
let mut app = Router::new()
.route("/healthz", get(|| async { "ok" }))
.route("/readyz", get(|| async { "ready" }))
+280 -44
View File
@@ -4,7 +4,8 @@
use crate::error::{GatewayError, Result};
use crate::model::{
ChatRequest, ChatResponse, Choice, Message, RouteDecision, RouteInfo, RoutingDirective, Usage,
ChatRequest, ChatResponse, Choice, GenParams, Message, RequestMeta, RouteDecision, RouteInfo,
RoutingDirective, Usage,
};
use crate::state::{Live, SharedState};
use crate::stats::RequestRecord;
@@ -67,6 +68,21 @@ impl Pipeline {
}
}
// Self-warming shadow loop: capture the prompt BEFORE `req` is moved,
// so we can (non-blocking, sampled) judge a local answer after the
// cloud reply. Client latency is unaffected. Nothing is served locally
// here — this only feeds the promotion table.
let shadow_cfg = state.live().cfg.shadow.clone();
let shadow_prompt = if shadow_cfg.enabled
&& !shadow_cfg.local_model_id.is_empty()
&& !shadow_cfg.judge_model_id.is_empty()
{
let live = state.live();
Some(self.extract_text(&req.messages, "last_user", live.cfg.route.max_chars))
} else {
None
};
let result = self.run_inner(req, state).await;
// record statistics (for both success and error)
@@ -135,6 +151,29 @@ impl Pipeline {
}
}
// Fire the shadow judgment for a sampled subset (non-blocking).
if let (Some(prompt), Ok(resp)) = (shadow_prompt, &result) {
let cloud_answer = resp
.choices
.first()
.map(|c| c.message.content.clone())
.unwrap_or_default();
let label = resp.cortiq.task_label.clone();
// Only judge CLOUD-served answers — the reference must be the cloud
// answer, not a local one (a promoted label served locally would
// otherwise judge local-vs-local). Canary auditing of served-local
// answers (forcing a cloud reference for a fraction) is the next step.
if !cloud_answer.is_empty()
&& !label.is_empty()
&& label != "cache"
&& resp.cortiq.selected_model != shadow_cfg.local_model_id
&& should_shadow(&prompt, shadow_cfg.sample_rate)
{
let st = state.0.clone();
tokio::spawn(shadow_capture(st, shadow_cfg, prompt, cloud_answer, label));
}
}
result
}
@@ -152,50 +191,62 @@ impl Pipeline {
live.cfg.route.max_chars,
);
// 2. Obtain the router decision (or use the pinned model)
let (decision, route_source) = match &req.routing {
RoutingDirective::Pinned { model_id: _ } => {
let dec = RouteDecision {
task_label: "pinned".to_string(),
complexity_score: 0.0,
complexity_tier: "pinned".to_string(),
router_request_id: None,
source: "pinned".to_string(),
};
(dec, "pinned".to_string())
}
RoutingDirective::Auto { profile } => {
let prof = profile.as_deref().unwrap_or(&live.cfg.route.profile);
match live.router.route(&text, prof).await {
Ok(Some(dec)) => {
let src = dec.source.clone();
(dec, src)
}
Ok(None) | Err(_) => {
tracing::warn!(
"Router is unavailable. Gracefully degrading to default model."
);
let dec = RouteDecision {
task_label: "degraded".to_string(),
complexity_score: 0.5,
complexity_tier: "degraded".to_string(),
router_request_id: None,
source: "fallback".to_string(),
};
(dec, "fallback".to_string())
}
}
}
// 2–3. Decide the route and pick candidate models.
//
// Local-first rules, so CMF models work with no router at all:
// • `[cmf].local_only` → every request goes to the local pool,
// ignoring the router, the pin, and external providers;
// • `[router].enabled = false` → skip the router, use the local pool;
// • router unavailable at runtime → graceful degrade to the local pool;
// • an explicitly pinned model → used directly, no router call.
let mk = |label: &str, source: &str| RouteDecision {
task_label: label.to_string(),
complexity_score: 0.0,
complexity_tier: label.to_string(),
router_request_id: None,
source: source.to_string(),
};
// 3. Select candidates by complexity tier / pinned model
let candidates = match &req.routing {
RoutingDirective::Pinned { model_id } => vec![model_id.clone()],
RoutingDirective::Auto { .. } => {
if route_source == "fallback" {
vec![live.routing.default_model().to_string()]
} else {
live.routing.candidates(&decision.complexity_tier)
let (decision, route_source, candidates) = if live.cfg.cmf.local_only {
(
mk("local-only", "local"),
"local".to_string(),
live.local_candidates(),
)
} else {
match &req.routing {
RoutingDirective::Pinned { model_id } => (
mk("pinned", "pinned"),
"pinned".to_string(),
vec![model_id.clone()],
),
RoutingDirective::Auto { profile } => {
if !live.cfg.router.enabled {
(
mk("router-disabled", "local"),
"local".to_string(),
live.local_candidates(),
)
} else {
let prof = profile.as_deref().unwrap_or(&live.cfg.route.profile);
match live.router.route(&text, prof).await {
Ok(Some(dec)) => {
let src = dec.source.clone();
let tier = dec.complexity_tier.clone();
(dec, src, live.routing.candidates(&tier))
}
Ok(None) | Err(_) => {
tracing::warn!(
"Router is unavailable. Gracefully degrading to the local/default model."
);
(
mk("degraded", "fallback"),
"fallback".to_string(),
live.local_candidates(),
)
}
}
}
}
}
};
@@ -216,6 +267,31 @@ impl Pipeline {
}
}
// An explicit pin must never silently answer from a different model: if
// the pinned id is unknown or was filtered out, fail loudly so the caller
// knows their choice was not honoured (instead of getting a stray reply
// from the managed local model via the safety net below).
if final_candidates.is_empty() {
if let RoutingDirective::Pinned { model_id } = &req.routing {
return Err(GatewayError::UpstreamUnavailable(format!(
"pinned model '{model_id}' is not available — it is not in the pool (deleted?) or lacks a required capability such as tools"
)));
}
}
// Local-first safety net: for AUTO/degraded routes only, if routing
// produced no usable candidate, fall back to the managed local model so a
// configured local backend still answers.
if final_candidates.is_empty() && live.cfg.cmf.manage_server {
for id in live.local_candidates() {
if let Some(prov) = live.registry.get(&id) {
if needs_tools && !prov.caps().tools {
continue;
}
final_candidates.push(id);
}
}
}
if final_candidates.is_empty() {
return Err(GatewayError::UpstreamUnavailable(
"No healthy model available that satisfies required capabilities (e.g. tools)"
@@ -227,7 +303,26 @@ impl Pipeline {
async fn run_inner(&self, req: ChatRequest, state: &SharedState) -> Result<ChatResponse> {
let live = state.live();
let (decision, route_source, final_candidates) = self.resolve(&req, &live).await?;
let (decision, route_source, mut final_candidates) = self.resolve(&req, &live).await?;
// Serving-when-promoted (gated): a promoted task-label is served by the
// local model FIRST (failover to cloud on error). Per-request VETO uses
// the router's own complexity signal — a "high"-tier request escalates
// to cloud even in a promoted label (the local skill is for the common
// case, not the hard tail). Off by default.
let sh = &live.cfg.shadow;
if sh.serve_when_promoted
&& !sh.local_model_id.is_empty()
&& matches!(req.routing, RoutingDirective::Auto { .. })
&& state.promotion.serves_local(&decision.task_label)
&& !matches!(
decision.complexity_tier.as_str(),
"high" | "hard" | "complex"
)
&& !final_candidates.iter().any(|m| m == &sh.local_model_id)
{
final_candidates.insert(0, sh.local_model_id.clone());
}
// 4. Attempt provider call with failover (fall back to alternatives on error)
let mut last_err = None;
@@ -592,3 +687,144 @@ fn tap_stream(
});
}
}
// ─────────────────────────── self-warming shadow loop ───────────────────────────
/// Deterministic sampling by prompt hash — no RNG dependency, and the same
/// prompt is always (not) sampled, so repeated office templates don't skew
/// the judged set request-by-request.
fn should_shadow(prompt: &str, rate: f64) -> bool {
if rate <= 0.0 {
return false;
}
if rate >= 1.0 {
return true;
}
use std::hash::{Hash, Hasher};
let mut h = std::collections::hash_map::DefaultHasher::new();
prompt.hash(&mut h);
(h.finish() % 10_000) < (rate * 10_000.0) as u64
}
fn clip(s: &str, max: usize) -> String {
if s.len() <= max {
s.to_string()
} else {
s.chars().take(max).collect()
}
}
fn shadow_req(model_id: &str, content: String, max_tokens: u32) -> ChatRequest {
ChatRequest {
routing: RoutingDirective::Pinned {
model_id: model_id.to_string(),
},
messages: vec![Message {
role: "user".to_string(),
content,
tool_calls: Vec::new(),
}],
tools: Vec::new(),
params: GenParams {
temperature: Some(0.0),
max_tokens: Some(max_tokens),
..Default::default()
},
stream: false,
meta: RequestMeta::default(),
}
}
/// Extract `{"score":N,"hard_fail":bool}` from a judge reply, tolerant of
/// surrounding prose. Missing/garbled → (0, true) (fail-closed: an
/// unparseable judgment must never count as a pass).
fn parse_judge(text: &str) -> (u8, bool) {
let score = text
.split("score")
.nth(1)
.and_then(|t| t.chars().find(|c| c.is_ascii_digit()))
.and_then(|c| c.to_digit(10))
.map(|d| d.min(4) as u8);
let hard_fail = text
.split("hard_fail")
.nth(1)
.map(|t| t.contains("true"))
.unwrap_or(true);
match score {
Some(s) => (s, hard_fail),
None => (0, true),
}
}
/// Non-blocking: answer locally, have the cheap judge score it against the
/// already-paid cloud answer, and feed the per-label promotion table. Any
/// failure (missing provider, upstream error, unparseable judgment) is
/// silently dropped — the client already got the cloud answer.
async fn shadow_capture(
st: std::sync::Arc<crate::state::AppState>,
cfg: crate::config::ShadowCfg,
prompt: String,
cloud_answer: String,
label: String,
) {
let live = st.live.load_full();
let (Some(local), Some(judge)) = (
live.registry.get(&cfg.local_model_id),
live.registry.get(&cfg.judge_model_id),
) else {
return;
};
// 1. Local answer (free, CPU).
let a_local = match local
.chat(shadow_req(&cfg.local_model_id, clip(&prompt, 4000), 512))
.await
{
Ok(r) => r
.choices
.into_iter()
.next()
.map(|c| c.message.content)
.unwrap_or_default(),
Err(_) => return,
};
if a_local.trim().is_empty() {
return;
}
// 2. Reference-based judgment (cheap model, different family).
let jp = format!(
"You grade a CANDIDATE answer against a REFERENCE answer for the same task.\n\
Task type: {label}\n\n[TASK]\n{task}\n\n[REFERENCE]\n{reference}\n\n[CANDIDATE]\n{candidate}\n\n\
Reply with ONLY compact JSON: {{\"score\":0-4,\"hard_fail\":true|false}}. \
score>=3 means the candidate is acceptable (correct, complete, right format). \
hard_fail=true for fabricated facts, broken format, or unsafe content.",
label = label,
task = clip(&prompt, 2000),
reference = clip(&cloud_answer, 3000),
candidate = clip(&a_local, 3000),
);
let Ok(jr) = judge.chat(shadow_req(&cfg.judge_model_id, jp, 32)).await else {
return;
};
let judge_tokens = jr.usage.total_tokens;
let jtext = jr
.choices
.into_iter()
.next()
.map(|c| c.message.content)
.unwrap_or_default();
let (score, hard_fail) = parse_judge(&jtext);
st.promotion.record(crate::promotion::JudgeRecord {
ts: now_secs(),
task_label: label,
pass: score >= 3 && !hard_fail,
hard_fail,
score,
source: "judge".to_string(),
recon_e: None,
born_conf: None,
judge_tokens,
});
}
+447
View File
@@ -0,0 +1,447 @@
//! Cloud-as-judge promotion table (self-warming local model, MVP).
//!
//! Per task-label we accumulate a judge/checker PASS stream and gate the
//! per-label lifecycle on the **Wilson-95 LOWER bound** of the pass rate —
//! never on the point estimate, so a lucky streak of easy requests can NOT
//! promote a label (small n → wide interval → low bound → stays SHADOW).
//! The full design + honest holes: docs/SELF_WARMING_GATEWAY.md.
//!
//! This module is pure bookkeeping (like `stats::Stats`): it decides the
//! STATE; the pipeline consults the state to decide who serves. Nothing here
//! serves a local answer — that is the caller's decision, guarded by these
//! states plus a per-request veto.
use serde::{Deserialize, Serialize};
use std::collections::{HashMap, VecDeque};
use std::io::Write;
use std::sync::Mutex;
use std::time::{SystemTime, UNIX_EPOCH};
fn now_secs() -> u64 {
SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap_or_default()
.as_secs()
}
/// One judgment of a local answer (JSONL line, replayed on startup).
#[derive(Clone, Debug, Serialize, Deserialize)]
pub struct JudgeRecord {
pub ts: u64,
pub task_label: String,
/// 1 = local answer accepted (checker pass OR judge score≥bar & !hard_fail).
pub pass: bool,
/// Fact hallucination / broken format / unsafe — blocks promotion hard.
#[serde(default)]
pub hard_fail: bool,
/// Raw judge score 0..4 (diagnostic only); 4 for a $0 programmatic checker.
#[serde(default)]
pub score: u8,
/// "checker" ($0 ground-truth) or "judge" (cloud LLM-as-judge).
#[serde(default)]
pub source: String,
/// Local recon-E (novelty) at judge time — diagnostic; NOT a correctness gate.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub recon_e: Option<f32>,
/// Local calibrated Born-mass confidence — for the negative-control study.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub born_conf: Option<f32>,
/// Cloud tokens spent judging (0 for a programmatic checker).
#[serde(default)]
pub judge_tokens: u32,
}
/// Per-label lifecycle. Serving-locally is only ever Canary/LocalServed.
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub enum PromotionState {
CloudOnly,
ShadowJudged,
Canary,
LocalServed,
Demoted,
}
/// Tuning (defaults from the memo; refit to measured ECE/traffic).
#[derive(Clone, Debug)]
pub struct PromotionCfg {
pub enabled: bool,
pub window: usize, // rolling judgments per label (W)
pub n_min: usize, // min judgments before promotion is even considered
pub promote_lb: f64, // Wilson-95 LB of pass must clear this
pub demote_pass: f64, // point pass-rate below this → demote a served label
pub hardfail_ub_max: f64, // Wilson-95 UPPER bound of hard_fail must stay under this
pub soak: usize, // extra Canary judgments before LocalServed
pub z: f64, // 1.96 = 95%
pub file: Option<String>, // judge.jsonl path (append + replay)
}
impl Default for PromotionCfg {
fn default() -> Self {
Self {
enabled: false,
window: 500,
n_min: 200,
promote_lb: 0.95,
demote_pass: 0.90,
hardfail_ub_max: 0.015,
soak: 100,
z: 1.96,
file: None,
}
}
}
struct LabelState {
passes: VecDeque<bool>, // rolling window of pass/fail
hard_fails: VecDeque<bool>, // rolling window of hard_fail flags
state: PromotionState,
soaked: usize, // judgments accumulated while in Canary
}
impl LabelState {
fn new() -> Self {
Self {
passes: VecDeque::new(),
hard_fails: VecDeque::new(),
state: PromotionState::ShadowJudged,
soaked: 0,
}
}
}
pub struct Promotion {
cfg: PromotionCfg,
labels: Mutex<HashMap<String, LabelState>>,
}
/// Wilson score interval bound for a Bernoulli rate. `upper=false` → lower.
/// s successes of n trials, z the normal quantile. Distribution-aware: small
/// n gives a wide interval (low lower bound / high upper bound) by construction.
pub fn wilson_bound(s: usize, n: usize, z: f64, upper: bool) -> f64 {
if n == 0 {
return if upper { 1.0 } else { 0.0 };
}
let n = n as f64;
let p = s as f64 / n;
let z2 = z * z;
let denom = 1.0 + z2 / n;
let centre = p + z2 / (2.0 * n);
let margin = z * ((p * (1.0 - p) / n) + z2 / (4.0 * n * n)).sqrt();
let v = if upper {
centre + margin
} else {
centre - margin
};
(v / denom).clamp(0.0, 1.0)
}
impl Promotion {
pub fn new(cfg: PromotionCfg) -> std::sync::Arc<Self> {
let p = std::sync::Arc::new(Self {
cfg,
labels: Mutex::new(HashMap::new()),
});
p.replay();
p
}
fn replay(&self) {
let Some(path) = &self.cfg.file else { return };
let Ok(content) = std::fs::read_to_string(path) else {
return;
};
for line in content.lines() {
if let Ok(rec) = serde_json::from_str::<JudgeRecord>(line) {
self.apply(&rec);
}
}
}
/// Record a judgment: update the rolling windows, advance the state
/// machine, append to judge.jsonl. Returns the (possibly new) state.
pub fn record(&self, rec: JudgeRecord) -> PromotionState {
let st = self.apply(&rec);
if let Some(path) = &self.cfg.file {
if let Ok(line) = serde_json::to_string(&rec) {
if let Ok(mut f) = std::fs::OpenOptions::new()
.create(true)
.append(true)
.open(path)
{
let _ = writeln!(f, "{line}");
}
}
}
st
}
fn apply(&self, rec: &JudgeRecord) -> PromotionState {
let mut map = self.labels.lock().unwrap();
let ls = map
.entry(rec.task_label.clone())
.or_insert_with(LabelState::new);
push_cap(&mut ls.passes, rec.pass, self.cfg.window);
push_cap(&mut ls.hard_fails, rec.hard_fail, self.cfg.window);
self.advance(ls);
ls.state
}
fn advance(&self, ls: &mut LabelState) {
let n = ls.passes.len();
let s = ls.passes.iter().filter(|&&p| p).count();
let hf = ls.hard_fails.iter().filter(|&&h| h).count();
let lb = wilson_bound(s, n, self.cfg.z, false);
let point = if n > 0 { s as f64 / n as f64 } else { 0.0 };
let recent_hf = ls.hard_fails.iter().rev().take(50).filter(|&&h| h).count();
// Zero-tolerance on hard-fails in the window: a Wilson UPPER bound on
// hard-fail rate ≤ 0.015 is UNREACHABLE at n=200 even with 0 fails
// (rule-of-three: 0/200 → ~0.019) — the panel flagged this. Requiring
// ZERO recent hard-fails is achievable AND safer (a recent
// hallucination/unsafe answer → stay on cloud until it ages out).
let bar = n >= self.cfg.n_min && lb >= self.cfg.promote_lb && hf == 0;
let regressed = point < self.cfg.demote_pass || recent_hf >= 2;
ls.state = match ls.state {
PromotionState::CloudOnly | PromotionState::ShadowJudged => {
if bar {
ls.soaked = 0;
PromotionState::Canary
} else {
PromotionState::ShadowJudged
}
}
PromotionState::Canary => {
if regressed {
PromotionState::Demoted
} else {
ls.soaked += 1;
if ls.soaked >= self.cfg.soak && bar {
PromotionState::LocalServed
} else {
PromotionState::Canary
}
}
}
PromotionState::LocalServed => {
if regressed {
PromotionState::Demoted
} else {
PromotionState::LocalServed
}
}
// Demoted goes back to SHADOW to re-earn promotion (skill stays in .cmf).
PromotionState::Demoted => PromotionState::ShadowJudged,
};
}
/// Current lifecycle state of a label.
pub fn state(&self, label: &str) -> PromotionState {
self.labels
.lock()
.unwrap()
.get(label)
.map(|l| l.state)
.unwrap_or(PromotionState::CloudOnly)
}
/// Should a request of this label be SERVED by the local model?
/// (Canary/LocalServed; caller still applies the per-request veto.)
pub fn serves_local(&self, label: &str) -> bool {
matches!(
self.state(label),
PromotionState::Canary | PromotionState::LocalServed
)
}
/// Snapshot for /admin: (state, n, pass-rate, Wilson-LB) per label.
pub fn snapshot(&self) -> Vec<(String, PromotionState, usize, f64, f64)> {
let map = self.labels.lock().unwrap();
let mut out: Vec<_> = map
.iter()
.map(|(k, ls)| {
let n = ls.passes.len();
let s = ls.passes.iter().filter(|&&p| p).count();
let p = if n > 0 { s as f64 / n as f64 } else { 0.0 };
(
k.clone(),
ls.state,
n,
p,
wilson_bound(s, n, self.cfg.z, false),
)
})
.collect();
out.sort_by_key(|x| std::cmp::Reverse(x.2));
out
}
pub fn enabled(&self) -> bool {
self.cfg.enabled
}
}
fn push_cap<T>(q: &mut VecDeque<T>, v: T, cap: usize) {
q.push_back(v);
while q.len() > cap {
q.pop_front();
}
}
#[cfg(test)]
mod tests {
use super::*;
fn cfg() -> PromotionCfg {
PromotionCfg {
enabled: true,
n_min: 200,
soak: 0,
..Default::default()
}
}
fn rec(label: &str, pass: bool, hard_fail: bool) -> JudgeRecord {
JudgeRecord {
ts: 0,
task_label: label.into(),
pass,
hard_fail,
score: if pass { 4 } else { 0 },
source: "checker".into(),
recon_e: None,
born_conf: None,
judge_tokens: 0,
}
}
#[test]
fn wilson_lb_penalizes_small_n() {
// 5/5 perfect but tiny n → lower bound far below 0.95.
assert!(wilson_bound(5, 5, 1.96, false) < 0.6);
// 195/200 at 97.5% → lower bound clears 0.95.
assert!(wilson_bound(195, 200, 1.96, false) >= 0.94);
// upper bound of 0 hard-fails over 200 ≈ 0.019 (rule-of-three-ish).
assert!(wilson_bound(0, 200, 1.96, true) < 0.025);
}
#[test]
fn no_promotion_on_a_lucky_streak() {
let p = Promotion::new(cfg());
// 50 straight passes: point-rate 100% but n < n_min AND LB < bar.
for _ in 0..50 {
p.record(rec("code", true, false));
}
assert_eq!(p.state("code"), PromotionState::ShadowJudged);
assert!(!p.serves_local("code"));
}
#[test]
fn promotes_only_with_enough_high_evidence() {
let p = Promotion::new(cfg());
// 198/200 (99%) + zero hard-fails → Wilson-LB ≈ 0.964 ≥ 0.95 → Canary.
// (195/200 = 97.5% gives LB ≈ 0.943 < 0.95 and would NOT promote —
// the bound is deliberately strict.)
for i in 0..200 {
p.record(rec("code", i % 100 != 0, false)); // 2 fails / 200
}
assert_eq!(p.state("code"), PromotionState::Canary); // soak=0 → straight to Canary
assert!(p.serves_local("code"));
}
/// Runnable narrative of the full loop:
/// cargo test --release demo_full_lifecycle -- --nocapture
/// Two labels side by side — a mature skill that promotes to local
/// serving, and a task the local can't do (stays on cloud). Proves the
/// state machine both PROMOTES on real evidence and REFUSES safely.
#[test]
fn demo_full_lifecycle() {
let p = Promotion::new(PromotionCfg { soak: 50, ..cfg() });
let show = |label: &str| {
let snap = p.snapshot();
let row = snap.iter().find(|r| r.0 == label);
match row {
Some((_, st, n, pr, lb)) => println!(
" {label:<12} {:<13} n={n:<4} pass={:.0}% LB={:.3} serves_local={}",
format!("{st:?}"),
pr * 100.0,
lb,
p.serves_local(label)
),
None => println!(" {label:<12} (cloud-only, no judgments)"),
}
};
println!("\n── self-warming loop, live ──");
// A mature skill (e.g. ru-tech): local answers pass the judge ~99%.
println!(
" 1) SHADOW→CANARY: судим локаль 'ru_tech' (сильный скилл ~99% pass), 210 судейств:"
);
for i in 0..210 {
p.record(rec("ru_tech", i % 100 != 0, false));
}
show("ru_tech");
assert_eq!(
p.state("ru_tech"),
PromotionState::Canary,
"n≥200 & LB≥0.95 → CANARY"
);
println!(" 2) CANARY soak (ещё 60 судейств держат планку) → LOCAL_SERVED:");
for i in 0..60 {
p.record(rec("ru_tech", i % 100 != 0, false));
}
show("ru_tech");
assert_eq!(
p.state("ru_tech"),
PromotionState::LocalServed,
"soak → LOCAL_SERVED"
);
assert!(p.serves_local("ru_tech"), "теперь обслуживается локально");
// The code dataset: local fails the checker → never promotes.
println!(" 3) параллельно 'code' (0.8B не тянет генерацию кода, 0% pass):");
for _ in 0..250 {
p.record(rec("code", false, false));
}
show("code");
assert_eq!(
p.state("code"),
PromotionState::ShadowJudged,
"остаётся на облаке — безопасно"
);
assert!(!p.serves_local("code"));
// Drift on the promoted label → auto-demote back to cloud.
println!(" 4) деградация 'ru_tech' (дрейф) → авто-DEMOTED, назад на облако:");
for _ in 0..120 {
p.record(rec("ru_tech", false, false));
}
show("ru_tech");
assert!(!p.serves_local("ru_tech"), "демоушен снял с локали");
println!(" ── петля: промоушен по доказательству, отказ и откат — безопасно ──\n");
}
#[test]
fn hard_fail_and_regression_block_or_demote() {
// A hard-fail spike keeps hf upper bound high → no promotion.
let p = Promotion::new(cfg());
for i in 0..200 {
p.record(rec("risky", true, i % 20 == 0)); // 5% hard_fail
}
assert_ne!(p.state("risky"), PromotionState::LocalServed);
// Promote a clean label, then feed a regression → demote.
let p2 = Promotion::new(PromotionCfg { soak: 0, ..cfg() });
for i in 0..200 {
p2.record(rec("t", i % 100 != 0, false)); // 198/200, 0 hard-fails
}
assert_eq!(p2.state("t"), PromotionState::Canary);
for _ in 0..300 {
p2.record(rec("t", false, false)); // quality collapses
}
assert!(matches!(
p2.state("t"),
PromotionState::Demoted | PromotionState::ShadowJudged
));
}
}
+6 -1
View File
@@ -3,8 +3,11 @@
pub mod anthropic_messages;
pub mod mcp;
pub mod native_passthrough;
pub mod openai_chat;
pub mod openai_completions;
pub mod openai_embeddings;
pub mod openai_models;
use crate::state::SharedState;
use axum::Router;
@@ -15,8 +18,10 @@ use axum::Router;
pub fn build_router() -> Router<SharedState> {
Router::new()
.merge(openai_chat::routes())
.merge(openai_completions::routes())
.merge(openai_embeddings::routes())
.merge(openai_models::routes())
.merge(anthropic_messages::routes())
.merge(mcp::routes())
// TODO: openai_completions, openai_models, native_passthrough.
.merge(native_passthrough::routes())
}
+86
View File
@@ -0,0 +1,86 @@
//! Native passthrough adapter: `POST /route`.
//! Returns the gateway's routing decision (task type, complexity, candidate models)
//! without calling a model — direct access for clients that route themselves.
use crate::error::{GatewayError, Result};
use crate::state::SharedState;
use axum::{extract::State, response::IntoResponse, routing::post, Json, Router};
use serde::Deserialize;
pub fn routes() -> Router<SharedState> {
Router::new().route("/route", post(handler))
}
#[derive(Deserialize)]
struct RouteRequest {
/// `{ "input": { "text": "..." } }` (router-style) — optional.
#[serde(default)]
input: serde_json::Value,
/// `{ "text": "..." }` — convenience shorthand.
#[serde(default)]
text: Option<String>,
#[serde(default)]
profile: Option<String>,
}
async fn handler(
State(state): State<SharedState>,
Json(req): Json<RouteRequest>,
) -> Result<impl IntoResponse> {
let live = state.live();
if !live.cfg.protocols.native_passthrough {
return Err(GatewayError::InvalidRequest(
"native_passthrough protocol is disabled".into(),
));
}
let text = req
.text
.clone()
.or_else(|| req.input["text"].as_str().map(|s| s.to_string()))
.or_else(|| req.input.as_str().map(|s| s.to_string()))
.unwrap_or_default();
if text.is_empty() {
return Err(GatewayError::InvalidRequest(
"missing input text (use {\"text\": ...} or {\"input\": {\"text\": ...}})".into(),
));
}
let profile = req
.profile
.clone()
.unwrap_or_else(|| live.cfg.route.profile.clone());
let (decision, source, candidates) = match live.router.route(&text, &profile).await {
Ok(Some(d)) => {
let cands = live.routing.candidates(&d.complexity_tier);
(Some(d), "router", cands)
}
_ => (
None,
"fallback",
vec![live.routing.default_model().to_string()],
),
};
let selected = candidates.first().cloned().unwrap_or_default();
let decision_json = match &decision {
Some(d) => serde_json::json!({
"task_label": d.task_label,
"complexity": { "score": d.complexity_score, "tier": d.complexity_tier },
"router_request_id": d.router_request_id,
}),
None => serde_json::json!({
"task_label": "degraded",
"complexity": { "score": 0.5, "tier": "degraded" },
"router_request_id": serde_json::Value::Null,
}),
};
Ok(Json(serde_json::json!({
"decision": decision_json,
"source": source,
"candidates": candidates,
"selected_model": selected,
})))
}
+183
View File
@@ -0,0 +1,183 @@
//! Inbound OpenAI Completions (legacy) adapter: `POST /v1/completions`.
//! Converts the legacy `prompt` into a single user message, runs the pipeline, and
//! returns the legacy `text_completion` shape (non-streaming JSON or streaming SSE).
use crate::error::{GatewayError, Result};
use crate::model::{ChatRequest, GenParams, Message, RequestMeta, RoutingDirective};
use crate::protocols::openai_chat::cortiq_headers;
use crate::state::SharedState;
use axum::response::{IntoResponse, Response};
use axum::{extract::State, routing::post, Json, Router};
use serde::Deserialize;
pub fn routes() -> Router<SharedState> {
Router::new().route("/v1/completions", post(handler))
}
#[derive(Deserialize)]
struct CompletionsRequest {
model: String,
prompt: serde_json::Value,
#[serde(default)]
temperature: Option<f32>,
#[serde(default)]
max_tokens: Option<u32>,
#[serde(default)]
top_p: Option<f32>,
#[serde(default)]
stream: bool,
}
fn parse_routing(model: &str) -> RoutingDirective {
if let Some(rest) = model.strip_prefix("cortiq-auto") {
RoutingDirective::Auto {
profile: rest.strip_prefix(':').map(|p| p.to_string()),
}
} else {
RoutingDirective::Pinned {
model_id: model.to_string(),
}
}
}
fn prompt_to_string(v: &serde_json::Value) -> String {
if let Some(s) = v.as_str() {
return s.to_string();
}
if let Some(arr) = v.as_array() {
return arr
.iter()
.filter_map(|x| x.as_str())
.collect::<Vec<_>>()
.join("\n");
}
String::new()
}
async fn handler(
State(state): State<SharedState>,
Json(req): Json<CompletionsRequest>,
) -> Result<Response> {
if !state.live().cfg.protocols.openai_completions {
return Err(GatewayError::InvalidRequest(
"openai_completions protocol is disabled".into(),
));
}
let prompt = prompt_to_string(&req.prompt);
if prompt.is_empty() {
return Err(GatewayError::InvalidRequest(
"prompt must not be empty".into(),
));
}
let canonical = ChatRequest {
routing: parse_routing(&req.model),
messages: vec![Message {
role: "user".into(),
content: prompt,
tool_calls: vec![],
}],
tools: vec![],
params: GenParams {
temperature: req.temperature,
max_tokens: req.max_tokens,
top_p: req.top_p,
stop: Vec::new(),
passthrough: Default::default(),
},
stream: req.stream,
meta: RequestMeta {
protocol: "openai_completions".into(),
..Default::default()
},
};
if req.stream {
let (info, stream) = state.pipeline.run_stream(canonical, &state).await?;
let mut headers = cortiq_headers(&info);
headers.insert(
axum::http::header::CONTENT_TYPE,
axum::http::HeaderValue::from_static("text/event-stream"),
);
return Ok((
headers,
axum::body::Body::from_stream(chat_to_completions_sse(stream)),
)
.into_response());
}
let resp = state.pipeline.run(canonical, &state).await?;
let answer = resp
.choices
.first()
.map(|c| c.message.content.clone())
.unwrap_or_default();
let finish = resp
.choices
.first()
.map(|c| c.finish_reason.clone())
.unwrap_or_else(|| "stop".into());
let headers = cortiq_headers(&resp.cortiq);
let body = serde_json::json!({
"id": resp.id,
"object": "text_completion",
"created": std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap_or_default()
.as_secs(),
"model": resp.model_used,
"choices": [{ "text": answer, "index": 0, "logprobs": serde_json::Value::Null, "finish_reason": finish }],
"usage": {
"prompt_tokens": resp.usage.prompt_tokens,
"completion_tokens": resp.usage.completion_tokens,
"total_tokens": resp.usage.total_tokens,
},
});
Ok((headers, Json(body)).into_response())
}
/// Translate chat `chat.completion.chunk` SSE into legacy `text_completion` SSE
/// (`delta.content` → `text`).
fn chat_to_completions_sse(
stream: crate::providers::ChatStream,
) -> impl futures::Stream<Item = Result<bytes::Bytes>> + Send + 'static {
use futures::StreamExt;
async_stream::stream! {
let mut buf = String::new();
futures::pin_mut!(stream);
while let Some(item) = stream.next().await {
let bytes = match item { Ok(b) => b, Err(e) => { yield Err(e); break; } };
if let Ok(s) = std::str::from_utf8(&bytes) {
buf.push_str(s);
}
while let Some(idx) = buf.find("\n\n") {
let chunk: String = buf.drain(..idx + 2).collect();
for line in chunk.lines() {
let Some(data) = line.trim_start().strip_prefix("data:") else { continue };
let data = data.trim();
if data.is_empty() { continue; }
if data == "[DONE]" {
yield Ok(bytes::Bytes::from("data: [DONE]\n\n"));
continue;
}
let Ok(v) = serde_json::from_str::<serde_json::Value>(data) else { continue };
if let Some(u) = v.get("usage").filter(|u| !u.is_null()) {
yield Ok(bytes::Bytes::from(format!(
"data: {}\n\n",
serde_json::json!({"object": "text_completion", "choices": [], "usage": u})
)));
}
if let Some(t) = v["choices"][0]["delta"]["content"].as_str() {
yield Ok(bytes::Bytes::from(format!(
"data: {}\n\n",
serde_json::json!({
"object": "text_completion",
"choices": [{"text": t, "index": 0, "finish_reason": null}]
})
)));
}
}
}
}
}
}
+32
View File
@@ -0,0 +1,32 @@
//! Inbound OpenAI Models adapter: `GET /v1/models`.
//! Lists the configured pool (plus the virtual `cortiq-auto`) for client discovery.
use crate::error::{GatewayError, Result};
use crate::state::SharedState;
use axum::{extract::State, response::IntoResponse, routing::get, Json, Router};
pub fn routes() -> Router<SharedState> {
Router::new().route("/v1/models", get(handler))
}
async fn handler(State(state): State<SharedState>) -> Result<impl IntoResponse> {
let live = state.live();
if !live.cfg.protocols.openai_models {
return Err(GatewayError::InvalidRequest(
"openai_models protocol is disabled".into(),
));
}
let mut data = vec![serde_json::json!({
"id": "cortiq-auto",
"object": "model",
"owned_by": "cortiq",
})];
for m in &live.cfg.models {
data.push(serde_json::json!({
"id": m.id,
"object": "model",
"owned_by": m.provider,
}));
}
Ok(Json(serde_json::json!({ "object": "list", "data": data })))
}
+29 -1
View File
@@ -1,7 +1,7 @@
//! Model pool registry: builds providers from `[[models]]` and resolves
//! `model_id → Arc<dyn Provider>`.
use crate::config::Config;
use crate::config::{Config, ModelCfg};
use crate::providers::{anthropic::AnthropicProvider, openai::OpenAiProvider, Provider};
use crate::secrets::SecretStore;
use std::collections::HashMap;
@@ -27,9 +27,37 @@ impl Registry {
};
by_id.insert(m.id.clone(), provider);
}
// Managed local CMF server: register it as an OpenAI-compatible provider
// so it can be routed to like any other backend — including when the
// router is unavailable or the gateway is in local-only mode.
if cfg.cmf.manage_server
&& !cfg.cmf.local_model.trim().is_empty()
&& !by_id.contains_key(&cfg.cmf.model_id)
{
let m = ModelCfg {
id: cfg.cmf.model_id.clone(),
provider: "openai".into(),
base_url: format!("http://{}:{}/v1", cfg.cmf.local_host, cfg.cmf.local_port),
model: "cortiq".into(),
cost_tier: "local".into(),
price_in: 0.0,
price_out: 0.0,
kind: "chat".into(),
api_key_env: None,
caps: Vec::new(),
};
by_id.insert(m.id.clone(), Arc::new(OpenAiProvider::new(&m, None)?));
}
Ok(Self { by_id })
}
/// Any model id, preferring a real registered one (used as a local-first
/// fallback target when routing cannot pick a specific model).
pub fn any_id(&self) -> Option<String> {
self.by_id.keys().next().cloned()
}
pub fn get(&self, model_id: &str) -> Option<Arc<dyn Provider>> {
self.by_id.get(model_id).cloned()
}
+47 -3
View File
@@ -4,16 +4,48 @@
use crate::config::RouterCfg;
use crate::model::RouteDecision;
use crate::secrets::SecretStore;
use std::sync::atomic::{AtomicU16, Ordering};
use std::sync::Arc;
/// Outcome of the most recent real `/v1/route` call, shared with the admin health
/// endpoint so the panel can tell "no money / bad key" apart from "router down".
/// 0 = ok (or no calls yet); [`STATUS_NETWORK`] / [`STATUS_TIMEOUT`] for transport
/// failures; otherwise the upstream HTTP status code.
pub type RouterLastStatus = Arc<AtomicU16>;
pub const STATUS_NETWORK: u16 = 1;
pub const STATUS_TIMEOUT: u16 = 2;
/// Error kind for the admin panel.
pub fn classify_status(code: u16) -> &'static str {
match code {
STATUS_NETWORK => "unreachable",
STATUS_TIMEOUT => "timeout",
401 | 403 => "auth",
402 => "payment",
429 => "quota",
_ => "error",
}
}
pub struct RouterClient {
http: reqwest::Client,
url: String,
api_key: Option<String>,
taxonomy_id: Option<String>,
last_status: RouterLastStatus,
}
impl RouterClient {
pub fn new(cfg: &RouterCfg) -> anyhow::Result<Self> {
/// The key is resolved through the secret store (admin panel value first,
/// then the environment variable) so a key pasted in the console works
/// without a restart.
pub fn new(
cfg: &RouterCfg,
secrets: &SecretStore,
last_status: RouterLastStatus,
) -> anyhow::Result<Self> {
let mut builder =
reqwest::Client::builder().timeout(std::time::Duration::from_millis(cfg.timeout_ms));
@@ -27,12 +59,13 @@ impl RouterClient {
let api_key = cfg
.api_key_env
.as_ref()
.and_then(|name| std::env::var(name).ok());
.and_then(|name| secrets.resolve(name));
Ok(Self {
http,
url: cfg.url.trim_end_matches('/').to_string(),
api_key,
taxonomy_id: cfg.taxonomy_id.clone(),
last_status,
})
}
@@ -54,11 +87,22 @@ impl RouterClient {
let resp = match req.send().await {
Ok(r) => r,
Err(_) => return Ok(None), // degradation
Err(e) => {
let code = if e.is_timeout() {
STATUS_TIMEOUT
} else {
STATUS_NETWORK
};
self.last_status.store(code, Ordering::Relaxed);
return Ok(None); // degradation
}
};
if !resp.status().is_success() {
self.last_status
.store(resp.status().as_u16(), Ordering::Relaxed);
return Ok(None);
}
self.last_status.store(0, Ordering::Relaxed);
let v: serde_json::Value = resp.json().await?;
let d = &v["decision"];
Ok(Some(RouteDecision {
+64 -6
View File
@@ -8,7 +8,7 @@
use crate::cache::SemanticCache;
use crate::config::Config;
use crate::registry::Registry;
use crate::router_client::RouterClient;
use crate::router_client::{RouterClient, RouterLastStatus};
use crate::routing::RoutingTable;
use crate::secrets::SecretStore;
use crate::stats::Stats;
@@ -27,8 +27,12 @@ pub struct Live {
}
impl Live {
pub fn build(cfg: Config, secrets: &SecretStore) -> anyhow::Result<Self> {
let router = RouterClient::new(&cfg.router)?;
pub fn build(
cfg: Config,
secrets: &SecretStore,
router_status: RouterLastStatus,
) -> anyhow::Result<Self> {
let router = RouterClient::new(&cfg.router, secrets, router_status)?;
let registry = Registry::from_config(&cfg, secrets)?;
let routing = RoutingTable::from_config(&cfg);
Ok(Self {
@@ -38,15 +42,61 @@ impl Live {
router,
})
}
/// Local-first fallback targets, in order: the managed local CMF model (if
/// configured), then the routing default, then any registered model — only
/// registered ids are returned. Used whenever the router is bypassed
/// (local-only / disabled) or unavailable, so CMF models serve with no router.
pub fn local_candidates(&self) -> Vec<String> {
let mut v = Vec::new();
if self.cfg.cmf.manage_server && !self.cfg.cmf.local_model.trim().is_empty() {
v.push(self.cfg.cmf.model_id.clone());
}
let def = self.routing.default_model().to_string();
if !def.is_empty() && !v.contains(&def) {
v.push(def);
}
v.retain(|id| self.registry.get(id).is_some());
if v.is_empty() {
if let Some(a) = self.registry.any_id() {
v.push(a);
}
}
v
}
}
pub struct AppState {
pub live: ArcSwap<Live>,
pub stats: Arc<Stats>,
pub promotion: Arc<crate::promotion::Promotion>,
pub imports: Arc<crate::import::JobStore>,
/// Managed local CMF model server (install/update/spawn lifecycle + status).
pub cmf: Arc<crate::cmf_runtime::CmfRuntime>,
pub cache: Arc<SemanticCache>,
pub secrets: SecretStore,
pub config_path: String,
pub pipeline: crate::pipeline::Pipeline,
/// Outcome of the most recent router call (survives config reloads) —
/// lets the admin panel distinguish a bad/expired key from a down router.
pub router_status: RouterLastStatus,
}
/// Map the user-facing `[shadow]` config to the promotion-table tuning.
fn promotion_cfg(s: &crate::config::ShadowCfg) -> crate::promotion::PromotionCfg {
crate::promotion::PromotionCfg {
enabled: s.enabled && !s.local_model_id.is_empty(),
window: s.window,
n_min: s.n_min,
promote_lb: s.promote_lb,
soak: s.soak,
file: if s.file.is_empty() {
None
} else {
Some(s.file.clone())
},
..Default::default()
}
}
impl std::ops::Deref for SharedState {
@@ -61,15 +111,23 @@ impl SharedState {
let secrets_path = sibling(&config_path, "secrets.toml");
let secrets = SecretStore::load(&secrets_path);
let stats = Stats::new(&cfg.stats);
let promotion = crate::promotion::Promotion::new(promotion_cfg(&cfg.shadow));
let imports = crate::import::JobStore::new();
let cmf = crate::cmf_runtime::CmfRuntime::new();
let cache = SemanticCache::new(&cfg.cache);
let live = Live::build(cfg, &secrets)?;
let router_status = RouterLastStatus::default();
let live = Live::build(cfg, &secrets, router_status.clone())?;
Ok(SharedState(Arc::new(AppState {
live: ArcSwap::from_pointee(live),
stats,
promotion,
imports,
cmf,
cache,
secrets,
config_path,
pipeline: crate::pipeline::Pipeline::new(),
router_status,
})))
}
@@ -84,7 +142,7 @@ impl AppState {
/// If any step fails, the active state and file are left unchanged.
pub fn reload(&self, new_cfg: Config) -> anyhow::Result<()> {
new_cfg.validate()?;
let live = Live::build(new_cfg.clone(), &self.secrets)?;
let live = Live::build(new_cfg.clone(), &self.secrets, self.router_status.clone())?;
new_cfg.save(&self.config_path)?;
self.live.store(Arc::new(live));
Ok(())
@@ -94,7 +152,7 @@ impl AppState {
/// (e.g. after a secret change — only provider keys changed).
pub fn rebuild(&self) -> anyhow::Result<()> {
let cfg = self.live.load_full().cfg.clone();
let live = Live::build(cfg, &self.secrets)?;
let live = Live::build(cfg, &self.secrets, self.router_status.clone())?;
self.live.store(Arc::new(live));
Ok(())
}
+90
View File
@@ -177,6 +177,18 @@ textarea { resize: vertical; min-height: 64px; }
.check input { width: auto; }
.hint { color: var(--faint); font-size: 12px; margin-top: 4px; }
/* "?" help badge — hover/focus shows the native title tooltip explaining a setting */
.help {
display: inline-flex; align-items: center; justify-content: center;
width: 15px; height: 15px; border-radius: 50%;
border: 1px solid var(--border); background: var(--panel);
color: var(--muted); font-size: 10px; font-weight: 700; line-height: 1;
cursor: help; vertical-align: middle; user-select: none;
transition: 0.12s;
}
.help:hover, .help:focus { color: var(--accent); border-color: var(--accent); outline: none; }
label.field > span .help { text-transform: none; }
/* ---- buttons ------------------------------------------------------------- */
.btn {
display: inline-flex; align-items: center; justify-content: center; gap: 8px;
@@ -307,3 +319,81 @@ tbody tr:hover td { background: var(--hover); }
.callout { border: 1px solid var(--border); border-left: 3px solid var(--warn); background: color-mix(in srgb, var(--warn) 6%, transparent); border-radius: 10px; padding: 13px 15px; color: var(--muted); margin-bottom: 16px; }
.callout.info { border-left-color: var(--accent-2); background: color-mix(in srgb, var(--accent-2) 6%, transparent); }
.callout b { color: var(--text); }
/* ── Import model (HuggingFace → CMF) ── */
.view-head { margin-bottom: 14px; }
.view-head h2 { margin: 0 0 2px; }
.card-title { font-weight: 600; margin-bottom: 10px; }
.actions { margin-top: 12px; display: flex; gap: 8px; }
.pad { padding: 18px; }
.err { color: #e5534b; }
.btn.sm { padding: 4px 10px; font-size: 12px; }
.hf-search {
width: 100%; box-sizing: border-box; margin-bottom: 12px;
padding: 10px 12px; font-size: 15px;
border: 1px solid var(--border, #2a2f3a); border-radius: 8px;
background: var(--bg-2, #12151c); color: inherit;
}
.hf-grid {
display: grid; grid-template-columns: repeat(auto-fill, minmax(230px, 1fr));
gap: 10px; max-height: 420px; overflow-y: auto;
}
.hf-card {
border: 1px solid var(--border, #2a2f3a); border-radius: 8px;
padding: 10px 12px; cursor: pointer; transition: border-color .12s, background .12s;
background: var(--bg-2, #12151c);
}
.hf-card:hover { border-color: var(--accent, #5b8def); }
.hf-card.sel { border-color: var(--accent, #5b8def); background: rgba(91,141,239,.10); }
.hf-id { font-weight: 600; font-size: 13px; word-break: break-all; margin-bottom: 6px; }
.hf-meta { display: flex; gap: 8px; align-items: center; flex-wrap: wrap; font-size: 12px; }
.badge.bad { background: rgba(229,83,75,.15); color: #e5534b; }
.import-form { margin-top: 14px; }
.adv { margin-top: 8px; }
.adv-body { margin-top: 10px; padding-top: 10px; border-top: 1px dashed var(--border, #2a2f3a); }
.pbar {
height: 8px; border-radius: 6px; overflow: hidden; margin: 10px 0;
background: var(--bg-2, #12151c); border: 1px solid var(--border, #2a2f3a);
}
.pbar-fill { height: 100%; width: 100%; border-radius: 6px; }
.pbar-fill.run {
width: 40%;
background: linear-gradient(90deg, transparent, var(--accent, #5b8def), transparent);
animation: pbar-slide 1.2s infinite ease-in-out;
}
.pbar-fill.ok { background: #3fb950; }
.pbar-fill.err { background: #e5534b; }
@keyframes pbar-slide { 0% { margin-left: -40%; } 100% { margin-left: 100%; } }
.conv-log {
margin-top: 10px; max-height: 260px; overflow-y: auto;
background: var(--bg-2, #0d1017); border: 1px solid var(--border, #2a2f3a);
border-radius: 8px; padding: 10px 12px;
font-family: ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 12px;
white-space: pre-wrap; word-break: break-word; color: var(--muted, #9aa4b2);
}
/* Router-key field: input grows, inline "Save key" button sits beside it */
.key-row { display: flex; gap: 8px; align-items: stretch; }
.key-row input { flex: 1 1 auto; min-width: 0; }
.key-row .btn { flex: 0 0 auto; white-space: nowrap; }
/* Determinate progress fill (known %); .run remains the indeterminate slider */
.pbar-fill.det {
background: linear-gradient(90deg, var(--accent, #5b8def), #7aa2f7);
transition: width .4s ease;
}
/* Import job cards (multiple conversions, live) */
.job { padding: 12px 0; border-top: 1px solid var(--border-soft, #232833); }
.job:first-of-type { border-top: 0; }
.job-head { display: flex; align-items: center; justify-content: space-between; gap: 12px; }
.job-head b { word-break: break-all; }
.small { font-size: 12px; }
/* Router usage: readable, wrapping key/value rows (no horizontal overflow) */
.usage-box { max-width: 100%; overflow-x: hidden; }
.mono.kv { display: flex; gap: 6px; padding: 1px 0; white-space: normal; overflow-wrap: anywhere; word-break: break-word; }
.mono.kv .kv-k { color: var(--muted, #9aa4b2); flex: 0 0 auto; }
.mono.kv .kv-v { font-weight: 600; overflow-wrap: anywhere; word-break: break-word; min-width: 0; }
+21
View File
@@ -50,6 +50,7 @@ export const api = {
// meta / health
meta: () => j("GET", "/meta"),
health: () => j("GET", "/health"),
probeRouter: () => j("POST", "/router/probe"),
// config
getConfig: () => j("GET", "/config"),
@@ -90,6 +91,26 @@ export const api = {
// playground
test: (b) => j("POST", "/test", b),
// shadow / self-warming loop
shadow: () => j("GET", "/shadow"),
// CMF runtime (the `cortiq` CLI that runs the .cmf format)
cmfStatus: () => j("GET", "/cmf"),
cmfInstall: () => j("POST", "/cmf/install"),
cmfPortCheck: (port) => j("GET", "/cmf/port?port=" + encodeURIComponent(port)),
// CMF model factory (HuggingFace → local .cmf)
hfSearch: (q, limit = 24) =>
j("GET", `/hf/search?q=${encodeURIComponent(q || "")}&limit=${limit}`),
startImport: (b) => j("POST", "/import", b),
listImports: () => j("GET", "/import"),
importStatus: (job) => j("GET", "/import/" + encodeURIComponent(job)),
cancelImport: (job) =>
j("POST", "/import/" + encodeURIComponent(job) + "/cancel"),
deleteImport: (job) => j("DELETE", "/import/" + encodeURIComponent(job)),
registerImport: (job) =>
j("POST", "/import/" + encodeURIComponent(job) + "/register"),
};
// Streaming playground: POST /test/stream, calls onDelta(text) per content chunk,
+24 -2
View File
@@ -5,6 +5,7 @@ import { api, getToken, setToken, setUnauthorizedHandler } from "./api.js";
import { renderLogin } from "./views/login.js";
import { renderDashboard } from "./views/dashboard.js";
import { renderModels } from "./views/models.js";
import { renderImport } from "./views/import.js";
import { renderRouting } from "./views/routing.js";
import { renderProtocols } from "./views/protocols.js";
import { renderKeys } from "./views/keys.js";
@@ -14,6 +15,7 @@ import { renderSettings } from "./views/settings.js";
const ROUTES = [
{ id: "dashboard", icon: "▤", view: renderDashboard },
{ id: "models", icon: "▦", view: renderModels },
{ id: "import", icon: "⬇", view: renderImport },
{ id: "routing", icon: "⇄", view: renderRouting },
{ id: "protocols", icon: "⇆", view: renderProtocols },
{ id: "keys", icon: "🔑", view: renderKeys },
@@ -23,6 +25,9 @@ const ROUTES = [
export const appState = { meta: null, health: null, version: "" };
// allaigate site — where router keys are issued and topped up
export const SITE_URL = "https://api.allaigate.com/";
function currentRoute() {
const id = location.hash.replace(/^#\/?/, "") || "dashboard";
return ROUTES.find((r) => r.id === id) || ROUTES[0];
@@ -53,6 +58,13 @@ function sidebar(activeId) {
)
),
h("div", { class: "nav-spacer" }),
h(
"a",
{ class: "nav-item", href: SITE_URL, target: "_blank", rel: "noopener" },
h("span", { class: "ico" }, "◈"),
t("nav.billing"),
h("span", { style: "margin-left:auto;opacity:.55" }, "↗")
),
h("div", { class: "nav-foot" }, t("nav.foot", { v: appState.version || "—" }))
);
}
@@ -61,6 +73,16 @@ function statusPills() {
const h_ = appState.health;
const gwOk = !!appState.meta;
const routerOk = h_?.router?.reachable;
// reachable but no key resolves → warn: routing silently degrades to default
const keyMissing = h_?.router?.key_source === "missing";
const routerDot = h_ ? (routerOk ? (keyMissing ? "warn" : "ok") : "bad") : "warn";
const routerText = h_
? routerOk
? keyMissing
? t("status.noKey")
: t("status.online")
: t("status.offline")
: t("status.checking");
return h(
"div",
{ class: "flex" },
@@ -73,8 +95,8 @@ function statusPills() {
h(
"span",
{ class: "pill", title: h_?.router?.url || "" },
h("span", { class: "dot " + (h_ ? (routerOk ? "ok" : "bad") : "warn") }),
t("status.router") + ": " + (h_ ? (routerOk ? t("status.online") : t("status.offline")) : t("status.checking"))
h("span", { class: "dot " + routerDot }),
t("status.router") + ": " + routerText
)
);
}
+15
View File
@@ -33,6 +33,7 @@ export default {
"nav.playground": "Playground",
"nav.settings": "Einstellungen",
"nav.foot": "Gateway · v{v}",
"nav.billing": "Router-Key & Abrechnung",
// topbar / status
"status.router": "Router",
@@ -40,6 +41,7 @@ export default {
"status.online": "online",
"status.offline": "offline",
"status.checking": "Prüfe…",
"status.noKey": "kein Key",
"theme.toggle": "Design",
"logout": "Abmelden",
@@ -88,6 +90,7 @@ export default {
"dash.health.title": "Status",
"dash.health.reachable": "erreichbar",
"dash.health.unreachable": "nicht erreichbar",
"dash.health.getKey": "Router-Key holen",
// models
"models.title": "Modelle",
@@ -225,6 +228,18 @@ export default {
"settings.router.timeout": "Timeout (ms)",
"settings.router.verifyTls": "TLS-Zertifikat prüfen",
"settings.router.taxonomy": "Taxonomie-ID",
"settings.router.getKey": "Router-Key holen oder aufladen:",
"settings.router.secret": "Router-Key (Wert)",
"settings.router.test": "Router testen",
"router.probe.ok": "Router OK ({ms} ms)",
"router.probe.noKey": "Router-Key ist nicht gesetzt",
"router.probe.auth": "Key abgelehnt — ungültig oder widerrufen",
"router.probe.payment": "Zahlung erforderlich — Tarif abgelaufen oder Kontingent erschöpft",
"router.probe.quota": "Ratenlimit überschritten",
"router.probe.timeout": "Router-Zeitüberschreitung",
"router.probe.unreachable": "Router nicht erreichbar",
"router.probe.error": "Router-Fehler",
"router.usage": "Verbrauch",
"settings.route": "Routing-Eingabe",
"settings.route.strategy": "Textstrategie",
"settings.route.maxChars": "Max. Zeichen",
+110
View File
@@ -28,12 +28,14 @@ export default {
// nav
"nav.dashboard": "Dashboard",
"nav.models": "Models",
"nav.import": "Import model",
"nav.routing": "Routing",
"nav.protocols": "Protocols",
"nav.keys": "API keys",
"nav.playground": "Playground",
"nav.settings": "Settings",
"nav.foot": "Gateway · v{v}",
"nav.billing": "Router key & billing",
// topbar / status
"status.router": "Router",
@@ -41,6 +43,7 @@ export default {
"status.online": "online",
"status.offline": "offline",
"status.checking": "checking…",
"status.noKey": "no key",
"theme.toggle": "Theme",
"logout": "Log out",
@@ -89,6 +92,7 @@ export default {
"dash.health.title": "Health",
"dash.health.reachable": "reachable",
"dash.health.unreachable": "unreachable",
"dash.health.getKey": "Get a router key",
// models
"models.title": "Models",
@@ -125,6 +129,8 @@ export default {
"models.form.secretMissing": "not set",
"models.form.secretPlaceholder": "set or replace key (write-only)",
"models.deleteConfirm": "Delete model \"{id}\"?",
"models.managed": "managed",
"models.managedHint": "Provided by the managed CMF server (Settings → Local models). Configure it there.",
// routing
"routing.title": "Routing",
@@ -225,7 +231,69 @@ export default {
"settings.router.keyEnv": "Router key env name",
"settings.router.timeout": "Timeout (ms)",
"settings.router.verifyTls": "Verify TLS certificate",
"settings.router.enabled": "Use the router (off = local models only)",
"settings.cmf": "Local models (CMF)",
"settings.cmf.localOnly": "Use only local models",
"settings.cmf.manage": "Run a local model server",
"settings.cmf.autoInstall": "Auto-install cortiq (crates.io)",
"settings.cmf.autoUpdate": "Auto-update cortiq",
"settings.cmf.model": "Local .cmf model path",
"settings.cmf.port": "Local server port",
"settings.cmf.modelId": "Model id in the pool",
"settings.cmf.note": "Serves a local .cmf via cortiq (github.com/infosave2007/cmf). Works even when the router is unavailable.",
"settings.cmf.threads": "Worker threads",
"settings.cmf.gpu": "Use GPU (Metal / wgpu)",
"settings.cmf.speedNote": "Threads parallelise the decode matvecs (~3x faster; 0 = runtime default). GPU offload mainly helps prefill of long prompts — decode is memory-bandwidth-bound, so on Apple Silicon it is roughly neutral.",
"settings.cmf.checkPort": "Check port",
"settings.cmf.portFree": "port is free",
"settings.cmf.portBusy": "port is in use",
"settings.cmf.useSuggested": "use {port}",
"settings.cmf.serving": "serving {model} on :{port}",
"settings.cmf.intro": "Run models locally with the cortiq engine — works even offline or when the router is down.",
"settings.cmf.runtime": "cortiq runtime",
"settings.cmf.runtimeHelp": "The `cortiq` command-line tool that loads and runs .cmf models. Install it once before starting a local server.",
"settings.cmf.install": "Install cortiq",
"settings.cmf.reinstall": "Reinstall / update",
"settings.cmf.installing": "installing… (compiles, a few minutes)",
"settings.cmf.installedOk": "cortiq installed",
"settings.cmf.notInstalled": "not installed",
"settings.cmf.addModel": "Add a model",
"settings.cmf.addModelHelp": "Models are downloaded and converted in the Import tab (HuggingFace → .cmf). Then point the local server at the .cmf file below.",
"settings.cmf.importNote": "No model yet? Import one:",
"settings.cmf.importLink": "Open Import",
"settings.cmf.modelHelp": "Path to the .cmf file to serve (e.g. models/qwen.cmf), produced by the Import tab. Running several models for different tasks? Import them and switch this path.",
"settings.cmf.portHelp": "TCP port the local model server listens on (default 8081). Change only if it clashes with another service.",
"settings.cmf.threadsHelp": "How many CPU cores to use for generation. More = faster (~3× vs 1 core); 0 lets the runtime decide.",
"settings.cmf.gpuHelp": "Offload matrix math to the GPU (Metal/wgpu). On Apple Silicon this mostly speeds up long prompts — decode is memory-bandwidth-bound.",
"settings.cmf.localOnlyHelp": "Route every request to the local model, ignoring the router and any cloud providers.",
"settings.cmf.manageHelp": "The gateway starts `cortiq serve` for the .cmf below at launch and routes to it. Needs cortiq installed (step 1). Applies on restart.",
"settings.cmf.autoInstallHelp": "If cortiq is missing at startup, install it automatically from crates.io.",
"settings.cmf.autoUpdateHelp": "On startup, upgrade cortiq to the newest release when a newer one exists.",
"settings.router.enabledHelp": "Use the cortiq-router to classify each request (task type + complexity) and pick a model. Off = use the local routing table only.",
"settings.router.verifyTlsHelp": "Verify the router's TLS certificate. Turn off only for a self-signed / IP-bound certificate.",
"settings.telemetry.metricsHelp": "Expose Prometheus metrics at GET /metrics.",
"settings.cortiq.echoHelp": "Add a `cortiq` field (task, model, cost) to each response body. Response headers are always sent.",
"settings.stats.enabledHelp": "Record recent requests to power the dashboard and charts.",
"settings.cache.enabledHelp": "Semantic cache: return a saved answer when a new prompt is very similar to a past one, skipping the model call. Needs an embedding model in the pool.",
"settings.log.bodiesHelp": "Log full request/response message bodies. Off by default for privacy.",
"settings.router.taxonomy": "Taxonomy id",
"settings.router.getKey": "Get or top up a router key at",
"settings.router.secret": "Router key (value)",
"settings.router.saveKey": "Save key",
"settings.router.keySaved": "Router key saved",
"settings.router.keyEmpty": "Paste the key first",
"settings.router.secretHelp": "Paste the key from your api.allaigate.com email, then Save key. This is NOT a “Keys” value — those are for clients calling your gateway.",
"settings.router.keyStoredMask": "•••••••••••• — key saved (paste again to replace)",
"settings.router.test": "Test router",
"router.probe.ok": "Router OK ({ms} ms)",
"router.probe.noKey": "Router key is not set",
"router.probe.auth": "Key rejected — invalid or revoked",
"router.probe.payment": "Payment required — plan expired or quota exhausted",
"router.probe.quota": "Rate limit exceeded",
"router.probe.timeout": "Router timed out",
"router.probe.unreachable": "Router unreachable",
"router.probe.error": "Router error",
"router.usage": "Usage",
"settings.route": "Routing input",
"settings.route.strategy": "Text strategy",
"settings.route.maxChars": "Max chars",
@@ -266,4 +334,46 @@ export default {
"settings.cache.ttl": "Entry TTL",
"settings.cache.embed": "Embedding model",
"settings.cache.note": "Cache settings take effect after a restart.",
// import (HuggingFace → CMF)
"import.subtitle": "Search HuggingFace, convert to a local .cmf, and serve it.",
"import.searchPlaceholder": "Search HuggingFace models (e.g. qwen3, llama)…",
"import.searching": "Searching…",
"import.noResults": "No models found.",
"import.configure": "Configure conversion",
"import.outName": "Output name",
"import.outNameHint": "The .cmf file name (letters, digits, - and _).",
"import.quant": "Quantization",
"import.convert": "Convert & load",
"import.converting": "Converting",
"import.started": "Conversion started",
"import.failed": "Conversion failed — see the log.",
"import.register": "Register as local model",
"import.registered": "Registered as '{id}'. Run cortiq-server to serve it.",
"import.jobs": "Conversions",
"import.cancel": "Cancel",
"import.cancelling": "Cancelling — cleaning up partial output…",
"import.delete": "Delete",
"import.deleteConfirm": "Delete this model and its converted .cmf file from disk?",
"import.deleted": "Deleted with its files",
"import.state.running": "running",
"import.state.done": "done",
"import.state.error": "error",
"import.state.cancelled": "cancelled",
"import.q.q8_2f": "8-bit two-field (𝒲×θ) — recommended: best quality/size.",
"import.q.q8_row": "8-bit per-row — simple, robust.",
"import.q.q4": "4-bit block — smallest, lower quality.",
"import.q.vbit": "Variable 3–8 bit — per-expert bit budgets.",
"import.q.f16": "16-bit — no quantization.",
"import.q.f32": "32-bit — full precision (large file).",
"import.adv.title": "Advanced settings",
"import.adv.auto": "auto",
"import.adv.linearCore": "Linear core",
"import.adv.linearHint": "How linear-attention layers are carried/folded.",
"import.adv.nphase": "vmf_phase phases/head",
"import.adv.vbitShape": "VBIT budget shape",
"import.adv.meanBits": "VBIT mean bits",
"import.adv.shard": "Shard max (GB)",
"import.adv.noShard": "no sharding",
"import.adv.skipMtp": "skip MTP head",
};
+15
View File
@@ -33,6 +33,7 @@ export default {
"nav.playground": "Playground",
"nav.settings": "Configuración",
"nav.foot": "Gateway · v{v}",
"nav.billing": "Clave del router y pagos",
// topbar / status
"status.router": "Router",
@@ -40,6 +41,7 @@ export default {
"status.online": "en línea",
"status.offline": "sin conexión",
"status.checking": "verificando…",
"status.noKey": "sin clave",
"theme.toggle": "Tema",
"logout": "Cerrar sesión",
@@ -88,6 +90,7 @@ export default {
"dash.health.title": "Estado",
"dash.health.reachable": "accesible",
"dash.health.unreachable": "inaccesible",
"dash.health.getKey": "Obtener una clave del router",
// models
"models.title": "Modelos",
@@ -225,6 +228,18 @@ export default {
"settings.router.timeout": "Tiempo de espera (ms)",
"settings.router.verifyTls": "Verificar certificado TLS",
"settings.router.taxonomy": "ID de taxonomía",
"settings.router.getKey": "Obtén o recarga una clave del router en",
"settings.router.secret": "Clave del router (valor)",
"settings.router.test": "Probar router",
"router.probe.ok": "Router OK ({ms} ms)",
"router.probe.noKey": "La clave del router no está configurada",
"router.probe.auth": "Clave rechazada — inválida o revocada",
"router.probe.payment": "Pago requerido — plan vencido o cuota agotada",
"router.probe.quota": "Límite de tasa excedido",
"router.probe.timeout": "Tiempo de espera del router agotado",
"router.probe.unreachable": "Router inaccesible",
"router.probe.error": "Error del router",
"router.usage": "Consumo",
"settings.route": "Entrada de enrutamiento",
"settings.route.strategy": "Estrategia de texto",
"settings.route.maxChars": "Máx. caracteres",
+15
View File
@@ -33,6 +33,7 @@ export default {
"nav.playground": "Playground",
"nav.settings": "Paramètres",
"nav.foot": "Gateway · v{v}",
"nav.billing": "Clé routeur & facturation",
// topbar / status
"status.router": "Routeur",
@@ -40,6 +41,7 @@ export default {
"status.online": "en ligne",
"status.offline": "hors ligne",
"status.checking": "vérification…",
"status.noKey": "pas de clé",
"theme.toggle": "Thème",
"logout": "Déconnexion",
@@ -88,6 +90,7 @@ export default {
"dash.health.title": "Santé",
"dash.health.reachable": "accessible",
"dash.health.unreachable": "inaccessible",
"dash.health.getKey": "Obtenir une clé routeur",
// models
"models.title": "Modèles",
@@ -225,6 +228,18 @@ export default {
"settings.router.timeout": "Délai d'expiration (ms)",
"settings.router.verifyTls": "Vérifier le certificat TLS",
"settings.router.taxonomy": "ID de taxonomie",
"settings.router.getKey": "Obtenir ou recharger une clé routeur sur",
"settings.router.secret": "Clé routeur (valeur)",
"settings.router.test": "Tester le routeur",
"router.probe.ok": "Routeur OK ({ms} ms)",
"router.probe.noKey": "La clé routeur n'est pas définie",
"router.probe.auth": "Clé rejetée — invalide ou révoquée",
"router.probe.payment": "Paiement requis — forfait expiré ou quota épuisé",
"router.probe.quota": "Limite de débit dépassée",
"router.probe.timeout": "Délai du routeur dépassé",
"router.probe.unreachable": "Routeur inaccessible",
"router.probe.error": "Erreur du routeur",
"router.usage": "Consommation",
"settings.route": "Entrée de routage",
"settings.route.strategy": "Stratégie de texte",
"settings.route.maxChars": "Caractères max",
+110
View File
@@ -27,12 +27,14 @@ export default {
// nav
"nav.dashboard": "Дашборд",
"nav.models": "Модели",
"nav.import": "Импорт модели",
"nav.routing": "Роутинг",
"nav.protocols": "Протоколы",
"nav.keys": "API-ключи",
"nav.playground": "Playground",
"nav.settings": "Настройки",
"nav.foot": "Шлюз · v{v}",
"nav.billing": "Ключ роутера и оплата",
// topbar / status
"status.router": "Роутер",
@@ -40,6 +42,7 @@ export default {
"status.online": "в сети",
"status.offline": "недоступен",
"status.checking": "проверка…",
"status.noKey": "нет ключа",
"theme.toggle": "Тема",
"logout": "Выйти",
@@ -88,6 +91,7 @@ export default {
"dash.health.title": "Состояние",
"dash.health.reachable": "доступен",
"dash.health.unreachable": "недоступен",
"dash.health.getKey": "Получить ключ роутера",
// models
"models.title": "Модели",
@@ -124,6 +128,8 @@ export default {
"models.form.secretMissing": "не задан",
"models.form.secretPlaceholder": "задать или заменить ключ (только запись)",
"models.deleteConfirm": "Удалить модель \"{id}\"?",
"models.managed": "управляемая",
"models.managedHint": "Предоставляется CMF-сервером (Настройки → Локальные модели). Настраивается там.",
// routing
"routing.title": "Роутинг",
@@ -224,7 +230,69 @@ export default {
"settings.router.keyEnv": "Имя env-переменной с ключом роутера",
"settings.router.timeout": "Таймаут (мс)",
"settings.router.verifyTls": "Проверять TLS-сертификат",
"settings.router.enabled": "Использовать роутер (выкл = только локальные модели)",
"settings.cmf": "Локальные модели (CMF)",
"settings.cmf.localOnly": "Использовать только локальные модели",
"settings.cmf.manage": "Запускать локальный сервер моделей",
"settings.cmf.autoInstall": "Автоустановка cortiq (crates.io)",
"settings.cmf.autoUpdate": "Автообновление cortiq",
"settings.cmf.model": "Путь к локальной .cmf-модели",
"settings.cmf.port": "Порт локального сервера",
"settings.cmf.modelId": "ID модели в пуле",
"settings.cmf.note": "Обслуживает локальный .cmf через cortiq (github.com/infosave2007/cmf). Работает даже когда роутер недоступен.",
"settings.cmf.threads": "Рабочие потоки",
"settings.cmf.gpu": "Использовать GPU (Metal / wgpu)",
"settings.cmf.speedNote": "Потоки распараллеливают декод-матвеки (~×3; 0 = по умолчанию рантайма). GPU-оффлоад помогает в основном на prefill длинных промптов — декод упирается в пропускную способность памяти, поэтому на Apple Silicon почти нейтрален.",
"settings.cmf.checkPort": "Проверить порт",
"settings.cmf.portFree": "порт свободен",
"settings.cmf.portBusy": "порт занят",
"settings.cmf.useSuggested": "занять {port}",
"settings.cmf.serving": "работает: {model} на :{port}",
"settings.cmf.intro": "Запускайте модели локально движком cortiq — работает даже офлайн или когда роутер недоступен.",
"settings.cmf.runtime": "Среда cortiq",
"settings.cmf.runtimeHelp": "Инструмент командной строки `cortiq`, который загружает и запускает .cmf-модели. Установите один раз перед запуском локального сервера.",
"settings.cmf.install": "Установить cortiq",
"settings.cmf.reinstall": "Переустановить / обновить",
"settings.cmf.installing": "устанавливается… (компилируется, пара минут)",
"settings.cmf.installedOk": "cortiq установлен",
"settings.cmf.notInstalled": "не установлен",
"settings.cmf.addModel": "Добавить модель",
"settings.cmf.addModelHelp": "Модели скачиваются и конвертируются во вкладке Импорт (HuggingFace → .cmf). Затем укажите путь к .cmf ниже.",
"settings.cmf.importNote": "Ещё нет модели? Импортируйте:",
"settings.cmf.importLink": "Открыть Импорт",
"settings.cmf.modelHelp": "Путь к .cmf-файлу для запуска (напр. models/qwen.cmf), создаётся во вкладке Импорт. Несколько моделей под разные задачи? Импортируйте их и меняйте этот путь.",
"settings.cmf.portHelp": "TCP-порт локального сервера моделей (по умолчанию 8081). Меняйте, только если конфликтует с другим сервисом.",
"settings.cmf.threadsHelp": "Сколько ядер CPU использовать для генерации. Больше = быстрее (~×3 против 1 ядра); 0 — на усмотрение рантайма.",
"settings.cmf.gpuHelp": "Выгрузить матричные операции на GPU (Metal/wgpu). На Apple Silicon ускоряет в основном длинные промпты — декод упирается в пропускную способность памяти.",
"settings.cmf.localOnlyHelp": "Направлять все запросы в локальную модель, игнорируя роутер и облачные провайдеры.",
"settings.cmf.manageHelp": "Шлюз запускает `cortiq serve` для .cmf ниже при старте и маршрутизирует в него. Нужен установленный cortiq (шаг 1). Применяется после перезапуска.",
"settings.cmf.autoInstallHelp": "Если cortiq не найден при старте — установить автоматически с crates.io.",
"settings.cmf.autoUpdateHelp": "При старте обновлять cortiq до новейшей версии, когда есть новее.",
"settings.router.enabledHelp": "Использовать cortiq-router для классификации запроса (тип задачи + сложность) и выбора модели. Выкл = только локальная таблица маршрутизации.",
"settings.router.verifyTlsHelp": "Проверять TLS-сертификат роутера. Выключайте только для самоподписанного / привязанного к IP сертификата.",
"settings.telemetry.metricsHelp": "Отдавать метрики Prometheus по GET /metrics.",
"settings.cortiq.echoHelp": "Добавлять поле `cortiq` (задача, модель, стоимость) в тело каждого ответа. Заголовки ответа шлются всегда.",
"settings.stats.enabledHelp": "Записывать недавние запросы для дашборда и графиков.",
"settings.cache.enabledHelp": "Семантический кэш: возвращать сохранённый ответ, когда новый промпт очень похож на прошлый, пропуская вызов модели. Нужна embedding-модель в пуле.",
"settings.log.bodiesHelp": "Логировать полные тела запросов/ответов. По умолчанию выкл ради приватности.",
"settings.router.taxonomy": "ID таксономии",
"settings.router.getKey": "Получить или пополнить ключ роутера:",
"settings.router.secret": "Ключ роутера (значение)",
"settings.router.saveKey": "Сохранить ключ",
"settings.router.keySaved": "Ключ роутера сохранён",
"settings.router.keyEmpty": "Сначала вставьте ключ",
"settings.router.secretHelp": "Вставьте ключ из письма с api.allaigate.com и нажмите «Сохранить ключ». Это НЕ ключ из раздела «Ключи» — те выдаются клиентам для доступа к вашему шлюзу.",
"settings.router.keyStoredMask": "•••••••••••• — ключ сохранён (вставьте снова, чтобы заменить)",
"settings.router.test": "Проверить роутер",
"router.probe.ok": "Роутер работает ({ms} мс)",
"router.probe.noKey": "Ключ роутера не задан",
"router.probe.auth": "Ключ отклонён — недействителен или отозван",
"router.probe.payment": "Требуется оплата — тариф истёк или квота исчерпана",
"router.probe.quota": "Превышен лимит запросов",
"router.probe.timeout": "Роутер не ответил вовремя",
"router.probe.unreachable": "Роутер недоступен",
"router.probe.error": "Ошибка роутера",
"router.usage": "Использование",
"settings.route": "Входные данные роутинга",
"settings.route.strategy": "Стратегия текста",
"settings.route.maxChars": "Макс. символов",
@@ -265,4 +333,46 @@ export default {
"settings.cache.ttl": "TTL записи",
"settings.cache.embed": "Модель эмбеддингов",
"settings.cache.note": "Настройки кэша применяются после перезапуска.",
// импорт (HuggingFace → CMF)
"import.subtitle": "Найдите модель на HuggingFace, сконвертируйте в локальный .cmf и подключите.",
"import.searchPlaceholder": "Поиск моделей на HuggingFace (напр. qwen3, llama)…",
"import.searching": "Ищу…",
"import.noResults": "Модели не найдены.",
"import.configure": "Настройка конвертации",
"import.outName": "Имя файла",
"import.outNameHint": "Имя .cmf (буквы, цифры, - и _).",
"import.quant": "Квантизация",
"import.convert": "Сконвертировать и загрузить",
"import.converting": "Конвертация",
"import.started": "Конвертация запущена",
"import.failed": "Конвертация не удалась — смотрите лог.",
"import.register": "Подключить как локальную модель",
"import.registered": "Подключено как '{id}'. Запустите cortiq-server для обслуживания.",
"import.jobs": "Конвертации",
"import.cancel": "Отменить",
"import.cancelling": "Отмена — удаляю недописанный файл…",
"import.delete": "Удалить",
"import.deleteConfirm": "Удалить эту модель и сконвертированный .cmf-файл с диска?",
"import.deleted": "Удалено вместе с файлами",
"import.state.running": "идёт",
"import.state.done": "готово",
"import.state.error": "ошибка",
"import.state.cancelled": "отменено",
"import.q.q8_2f": "8-бит двухполевой (𝒲×θ) — рекомендуется: лучшее качество/размер.",
"import.q.q8_row": "8-бит построчно — просто, надёжно.",
"import.q.q4": "4-бит блочно — компактнее, ниже качество.",
"import.q.vbit": "Переменная 3–8 бит — per-expert бюджеты.",
"import.q.f16": "16-бит — без квантования.",
"import.q.f32": "32-бит — полная точность (большой файл).",
"import.adv.title": "Расширенные настройки",
"import.adv.auto": "авто",
"import.adv.linearCore": "Линейное ядро",
"import.adv.linearHint": "Как переносятся/сворачиваются линейные слои внимания.",
"import.adv.nphase": "vmf_phase фаз/голову",
"import.adv.vbitShape": "Форма бюджета VBIT",
"import.adv.meanBits": "VBIT средних бит",
"import.adv.shard": "Шард макс (ГБ)",
"import.adv.noShard": "без шардинга",
"import.adv.skipMtp": "пропустить MTP-голову",
};
+15
View File
@@ -33,6 +33,7 @@ export default {
"nav.playground": "Oyun alanı",
"nav.settings": "Ayarlar",
"nav.foot": "Gateway · v{v}",
"nav.billing": "Router anahtarı ve ödeme",
// topbar / status
"status.router": "Yönlendirici",
@@ -40,6 +41,7 @@ export default {
"status.online": "çevrimiçi",
"status.offline": "çevrimdışı",
"status.checking": "kontrol ediliyor…",
"status.noKey": "anahtar yok",
"theme.toggle": "Tema",
"logout": "Çıkış yap",
@@ -88,6 +90,7 @@ export default {
"dash.health.title": "Sağlık",
"dash.health.reachable": "erişilebilir",
"dash.health.unreachable": "erişilemiyor",
"dash.health.getKey": "Router anahtarı al",
// models
"models.title": "Modeller",
@@ -225,6 +228,18 @@ export default {
"settings.router.timeout": "Zaman aşımı (ms)",
"settings.router.verifyTls": "TLS sertifikasını doğrula",
"settings.router.taxonomy": "Taksonomi id",
"settings.router.getKey": "Router anahtarı almak veya bakiye yüklemek için:",
"settings.router.secret": "Router anahtarı (değer)",
"settings.router.test": "Router'ı test et",
"router.probe.ok": "Router çalışıyor ({ms} ms)",
"router.probe.noKey": "Router anahtarı ayarlanmadı",
"router.probe.auth": "Anahtar reddedildi — geçersiz veya iptal edilmiş",
"router.probe.payment": "Ödeme gerekli — plan süresi doldu veya kota tükendi",
"router.probe.quota": "Hız sınırı aşıldı",
"router.probe.timeout": "Router zaman aşımına uğradı",
"router.probe.unreachable": "Router erişilemiyor",
"router.probe.error": "Router hatası",
"router.usage": "Kullanım",
"settings.route": "Yönlendirme girişi",
"settings.route.strategy": "Metin stratejisi",
"settings.route.maxChars": "Maks. karakter",
+15
View File
@@ -33,6 +33,7 @@ export default {
"nav.playground": "沙盒",
"nav.settings": "设置",
"nav.foot": "网关 · v{v}",
"nav.billing": "路由器密钥与充值",
// topbar / status
"status.router": "路由器",
@@ -40,6 +41,7 @@ export default {
"status.online": "在线",
"status.offline": "离线",
"status.checking": "检测中…",
"status.noKey": "无密钥",
"theme.toggle": "主题",
"logout": "退出",
@@ -88,6 +90,7 @@ export default {
"dash.health.title": "健康状态",
"dash.health.reachable": "可达",
"dash.health.unreachable": "不可达",
"dash.health.getKey": "获取路由器密钥",
// models
"models.title": "模型",
@@ -225,6 +228,18 @@ export default {
"settings.router.timeout": "超时(ms)",
"settings.router.verifyTls": "验证 TLS 证书",
"settings.router.taxonomy": "分类体系 ID",
"settings.router.getKey": "获取或充值路由器密钥:",
"settings.router.secret": "路由器密钥(值)",
"settings.router.test": "测试路由器",
"router.probe.ok": "路由器正常({ms} ms)",
"router.probe.noKey": "未设置路由器密钥",
"router.probe.auth": "密钥被拒绝——无效或已吊销",
"router.probe.payment": "需要付款——套餐到期或配额用尽",
"router.probe.quota": "超出速率限制",
"router.probe.timeout": "路由器响应超时",
"router.probe.unreachable": "路由器不可达",
"router.probe.error": "路由器错误",
"router.usage": "用量",
"settings.route": "路由输入",
"settings.route.strategy": "文本策略",
"settings.route.maxChars": "最大字符数",
+27 -3
View File
@@ -3,6 +3,7 @@ import { h, mount, money, num, ms, pct, timeAgo } from "../ui.js";
import { t } from "../i18n.js";
import { api } from "../api.js";
import { lineChart, barList } from "../charts.js";
import { SITE_URL } from "../app.js";
const RANGES = ["1h", "24h", "7d"];
const GROUPS = ["model", "tier", "label", "account"];
@@ -186,7 +187,19 @@ function recentTable(rows) {
);
}
function keySourceLabel(src) {
if (src === "store") return t("models.form.secretStored");
if (src === "env") return t("models.form.secretEnv");
return t("models.form.secretMissing");
}
function healthCard(hh) {
const r = hh.router || {};
const keySrc = r.key_source || "none";
const err = r.last_error;
// problems fixed on the site (get / pay for a key) get the top-up link
const needsSite =
!r.reachable || keySrc === "missing" || (err && ["auth", "payment", "quota"].includes(err.kind));
return h(
"div",
{ class: "card" },
@@ -196,9 +209,20 @@ function healthCard(hh) {
{ class: "flex wrap", style: "margin-bottom:12px" },
h(
"span",
{ class: "badge " + (hh.router?.reachable ? "ok" : "bad"), title: hh.router?.url || "" },
t("status.router") + ": " + (hh.router?.reachable ? t("dash.health.reachable") : t("dash.health.unreachable"))
)
{ class: "badge " + (r.reachable ? "ok" : "bad"), title: r.url || "" },
t("status.router") + ": " + (r.reachable ? t("dash.health.reachable") : t("dash.health.unreachable"))
),
keySrc !== "none"
? h(
"span",
{ class: "badge " + keyBadge(keySrc), title: r.key_env || "" },
t("models.col.key") + ": " + keySourceLabel(keySrc)
)
: null,
err ? h("span", { class: "badge bad" }, t("router.probe." + (err.kind || "error"))) : null,
needsSite
? h("a", { href: SITE_URL, target: "_blank", rel: "noopener" }, t("dash.health.getKey") + " ↗")
: null
),
h(
"div",
+332
View File
@@ -0,0 +1,332 @@
// Import a HuggingFace model → convert to a local .cmf (our format) with a
// live progress view. Search HF, pick quantization + advanced params, watch
// the conversion (multiple in parallel, resumable across tab switches), then
// register the result as a local model.
import { h, mount, toast } from "../ui.js";
import { t } from "../i18n.js";
import { api } from "../api.js";
const QUANTS = [
["Q8_2F", "import.q.q8_2f"],
["Q8_ROW", "import.q.q8_row"],
["Q4_BLOCK", "import.q.q4"],
["VBIT", "import.q.vbit"],
["F16", "import.q.f16"],
["F32", "import.q.f32"],
];
function field(label, control, hint) {
return h("label", { class: "field" }, h("span", {}, label), control,
hint ? h("div", { class: "hint" }, hint) : null);
}
function opt(v, label, sel) {
return h("option", { value: v, selected: sel ? true : null }, label || v);
}
function fmtNum(n) {
if (n == null) return "—";
if (n >= 1e6) return (n / 1e6).toFixed(1) + "M";
if (n >= 1e3) return (n / 1e3).toFixed(1) + "k";
return String(n);
}
function fmtBytes(n) {
if (!n) return "";
const g = n / 1e9;
return g >= 1 ? g.toFixed(2) + " GB" : (n / 1e6).toFixed(0) + " MB";
}
// state → localized label (falls back to the raw state)
function stateLabel(s) {
const k = "import.state." + s;
const v = t(k);
return v === k ? s : v;
}
export function renderImport() {
let selected = null;
let lastModels = []; // cache of the last search, so re-highlighting needs no refetch
let jobsTimer = null;
const searchIn = h("input", {
class: "hf-search",
placeholder: t("import.searchPlaceholder"),
autofocus: true,
});
const results = h("div", { class: "hf-grid" });
const formHost = h("div", {});
const jobsHost = h("div", {});
// ── live HF search (debounced) ──
let debounce = null;
function renderCards(models) {
if (!models.length) {
mount(results, h("div", { class: "muted pad" }, t("import.noResults")));
return;
}
mount(results, ...models.map(modelCard));
}
async function doSearch() {
const q = searchIn.value.trim();
mount(results, h("div", { class: "muted pad" }, t("import.searching")));
try {
const r = await api.hfSearch(q, 24);
lastModels = r.models || [];
renderCards(lastModels);
} catch (e) {
mount(results, h("div", { class: "err pad" }, "HF: " + e.message));
}
}
searchIn.addEventListener("input", () => {
clearTimeout(debounce);
debounce = setTimeout(doSearch, 300);
});
function modelCard(m) {
const card = h(
"div",
{ class: "hf-card" + (selected && selected.id === m.id ? " sel" : "") },
h("div", { class: "hf-id" }, m.id),
h(
"div",
{ class: "hf-meta" },
m.pipeline_tag ? h("span", { class: "badge" }, m.pipeline_tag) : null,
h("span", { class: "muted" }, "▼ " + fmtNum(m.downloads)),
h("span", { class: "muted" }, "♥ " + fmtNum(m.likes)),
(m.tags || []).some((x) => x === "gated") ? h("span", { class: "badge bad" }, "gated") : null
)
);
card.addEventListener("click", () => selectModel(m));
return card;
}
function selectModel(m) {
selected = m;
renderCards(lastModels); // re-highlight from cache — no refetch, no flash
renderForm();
formHost.scrollIntoView({ behavior: "smooth", block: "nearest" });
}
// ── conversion form ──
function renderForm() {
if (!selected) {
mount(formHost);
return;
}
const nameIn = h("input", { value: (selected.id.split("/").pop() || "model").toLowerCase() });
const quantSel = h("select", {}, ...QUANTS.map(([v], i) => opt(v, v, i === 0)));
const quantHint = h("div", { class: "hint" }, t("import.q.q8_2f"));
quantSel.addEventListener("change", () => {
const k = QUANTS.find(([v]) => v === quantSel.value);
quantHint.textContent = k ? t(k[1]) : "";
});
// advanced (collapsed)
const linearSel = h("select", {}, opt("", t("import.adv.auto")), opt("gated_delta_net", "gated_delta_net"), opt("vmf_phase", "vmf_phase"));
const nphaseIn = h("input", { type: "number", min: "1", placeholder: "16" });
const vbitSel = h("select", {}, opt("", t("import.adv.auto")), opt("log2", "log2"), opt("cubic", "cubic"));
const meanBitsIn = h("input", { type: "number", step: "0.1", placeholder: "5.0" });
const shardIn = h("input", { type: "number", step: "0.5", placeholder: t("import.adv.noShard") });
const skipMtp = h("input", { type: "checkbox" });
const advBody = h(
"div",
{ class: "adv-body", style: "display:none" },
h("div", { class: "row" },
field(t("import.adv.linearCore"), linearSel, t("import.adv.linearHint")),
field(t("import.adv.nphase"), nphaseIn)),
h("div", { class: "row" },
field(t("import.adv.vbitShape"), vbitSel),
field(t("import.adv.meanBits"), meanBitsIn)),
h("div", { class: "row" },
field(t("import.adv.shard"), shardIn),
field("MTP", h("label", { class: "check" }, skipMtp, t("import.adv.skipMtp")))),
);
const advToggle = h("button", { class: "btn ghost sm" }, "⚙ " + t("import.adv.title"));
advToggle.addEventListener("click", () => {
advBody.style.display = advBody.style.display === "none" ? "block" : "none";
});
const goBtn = h("button", { class: "btn primary" }, "⬇ " + t("import.convert"));
goBtn.addEventListener("click", async () => {
const params = {
repo: selected.id,
quant: quantSel.value,
name: nameIn.value.trim(),
skip_mtp: skipMtp.checked,
};
if (linearSel.value) params.linear_core = linearSel.value;
if (nphaseIn.value) params.nphase = parseInt(nphaseIn.value);
if (vbitSel.value) params.vbit_shape = vbitSel.value;
if (meanBitsIn.value) params.mean_bits = parseFloat(meanBitsIn.value);
if (shardIn.value) params.shard_max_gb = parseFloat(shardIn.value);
goBtn.disabled = true;
try {
await api.startImport(params);
toast(t("import.started"), "ok");
await refreshJobs(); // picks up the new job + (re)starts polling
jobsHost.scrollIntoView({ behavior: "smooth", block: "nearest" });
} catch (e) {
toast(e.message, "err");
}
goBtn.disabled = false;
});
mount(
formHost,
h("div", { class: "card import-form" },
h("div", { class: "card-title" }, t("import.configure") + ": " + selected.id),
h("div", { class: "row" },
field(t("import.outName"), nameIn, t("import.outNameHint")),
field(t("import.quant"), quantSel)),
quantHint,
h("div", { class: "adv" }, advToggle, advBody),
h("div", { class: "actions" }, goBtn),
)
);
}
// ── jobs panel: all conversions, live, resumable, cancellable ──
function ensurePolling() {
if (!jobsTimer) {
jobsTimer = setInterval(() => {
// stop once this view instance was replaced (navigated away)
if (!root.isConnected) {
stopPolling();
return;
}
refreshJobs();
}, 1500);
}
}
function stopPolling() {
if (jobsTimer) {
clearInterval(jobsTimer);
jobsTimer = null;
}
}
// Load + render all jobs. Runs on mount (before `root` is attached — so no
// isConnected guard here) and on each poll tick.
async function refreshJobs() {
let jobs = [];
try {
jobs = (await api.listImports()).jobs || [];
} catch {
return;
}
jobs = jobs.filter((j) => j.state !== "cancelled"); // hide user-cancelled jobs
renderJobs(jobs);
if (jobs.some((j) => j.state === "running")) ensurePolling();
else stopPolling();
}
function jobCard(j) {
const running = j.state === "running";
const pct = j.progress != null ? Math.round(j.progress * 100) : null;
// det = solid bar at a known %, run = indeterminate slider, ok/err = terminal.
let bar;
if (j.state === "done") {
bar = h("div", { class: "pbar-fill ok", style: "width:100%" });
} else if (running && pct != null) {
bar = h("div", { class: "pbar-fill det", style: `width:${pct}%` });
} else if (running) {
bar = h("div", { class: "pbar-fill run" }); // CSS drives width + animation
} else {
bar = h("div", { class: "pbar-fill err", style: "width:100%" }); // error / cancelled
}
const barWrap = h("div", { class: "pbar" }, bar);
const badge = h(
"span",
{ class: "badge " + (j.state === "done" ? "" : running ? "" : "bad") },
stateLabel(j.state) +
(running && pct != null ? " · " + pct + "%" : "") +
(j.size_bytes ? " · " + fmtBytes(j.size_bytes) : "")
);
const phase = h("div", { class: "muted small" }, j.phase || "");
const logEl = h("pre", { class: "conv-log" }, (j.log || []).slice(-10).join("\n"));
const actions = [];
if (running) {
const c = h("button", { class: "btn ghost sm" }, "✕ " + t("import.cancel"));
c.addEventListener("click", async () => {
c.disabled = true;
try {
await api.cancelImport(j.id);
toast(t("import.cancelling"), "warn");
refreshJobs();
} catch (e) {
toast(e.message, "err");
c.disabled = false;
}
});
actions.push(c);
}
if (j.state === "done") {
const r = h("button", { class: "btn primary sm" }, "✓ " + t("import.register"));
r.addEventListener("click", async () => {
r.disabled = true;
try {
const rr = await api.registerImport(j.id);
toast(t("import.registered", { id: rr.model_id }), "ok");
} catch (e) {
toast(e.message, "err");
r.disabled = false;
}
});
actions.push(r);
}
if (!running) {
// delete the job + its converted .cmf file(s) from disk
const d = h("button", { class: "btn ghost sm danger" }, "🗑 " + t("import.delete"));
d.addEventListener("click", async () => {
if (!window.confirm(t("import.deleteConfirm"))) return;
d.disabled = true;
try {
await api.deleteImport(j.id);
toast(t("import.deleted"), "ok");
refreshJobs();
} catch (e) {
toast(e.message, "err");
d.disabled = false;
}
});
actions.push(d);
}
return h(
"div",
{ class: "job" },
h("div", { class: "job-head" }, h("b", {}, j.repo || j.id), badge),
barWrap,
phase,
logEl,
actions.length ? h("div", { class: "actions" }, ...actions) : null
);
}
function renderJobs(jobs) {
if (!jobs.length) {
mount(jobsHost);
return;
}
mount(
jobsHost,
h(
"div",
{ class: "card" },
h("div", { class: "card-title" }, t("import.jobs")),
...jobs.map(jobCard)
)
);
}
const root = h(
"div",
{},
h("div", { class: "view-head" },
h("h2", {}, t("nav.import")),
h("div", { class: "muted" }, t("import.subtitle"))),
jobsHost, // progress panel on top, above the search
h("div", { class: "card" }, searchIn, results),
formHost
);
doSearch(); // initial trending list (mounts into `results`)
refreshJobs(); // resume: show any running/recent conversions immediately
return root;
}
+7 -1
View File
@@ -2,6 +2,7 @@
import { h } from "../ui.js";
import { t } from "../i18n.js";
import { setToken } from "../api.js";
import { SITE_URL } from "../app.js";
export function renderLogin(onOk) {
const input = h("input", {
@@ -34,7 +35,12 @@ export function renderLogin(onOk) {
h("p", { class: "sub" }, t("login.subtitle")),
h("label", { class: "field" }, h("span", {}, t("login.token")), input),
h("button", { class: "btn primary", style: "width:100%", onClick: submit }, t("login.submit")),
h("p", { class: "hint", style: "margin-top:16px" }, t("login.hint"))
h("p", { class: "hint", style: "margin-top:16px" }, t("login.hint")),
h(
"p",
{ class: "hint", style: "margin-top:8px" },
h("a", { href: SITE_URL, target: "_blank", rel: "noopener" }, t("nav.billing") + " · api.allaigate.com ↗")
)
)
);
}
+3 -2
View File
@@ -168,7 +168,8 @@ function row(m, meta, reload) {
return h(
"tr",
{},
h("td", { class: "mono" }, m.id),
h("td", { class: "mono" }, m.id,
m.managed ? h("span", { class: "badge", style: "margin-left:6px", title: t("models.managedHint") }, t("models.managed")) : null),
h("td", {}, m.provider),
h("td", { class: "mono" }, m.model),
h("td", {}, h("span", { class: "badge " + (m.cost_tier || "") }, m.cost_tier || "—")),
@@ -182,7 +183,7 @@ function row(m, meta, reload) {
"div",
{ class: "flex", style: "justify-content:flex-end" },
probeBtn,
h("button", { class: "btn sm ghost", onClick: () => openModal(meta, m, reload) }, t("common.edit")),
m.managed ? null : h("button", { class: "btn sm ghost", onClick: () => openModal(meta, m, reload) }, t("common.edit")),
h(
"button",
{
+309 -13
View File
@@ -2,19 +2,65 @@
import { h, mount, toast } from "../ui.js";
import { t } from "../i18n.js";
import { api } from "../api.js";
import { appState } from "../app.js";
import { appState, SITE_URL } from "../app.js";
function field(label, control, hint) {
return h("label", { class: "field" }, h("span", {}, label), control, hint ? h("div", { class: "hint" }, hint) : null);
// A small "?" badge with a hover/focus tooltip explaining a setting.
function help(text) {
return text ? h("span", { class: "help", title: text, tabindex: "0", role: "img", "aria-label": text }, "?") : null;
}
function check(label, checked) {
function field(label, control, hint, helpText) {
return h(
"label",
{ class: "field" },
h("span", {}, label, helpText ? " " : null, help(helpText)),
control,
hint ? h("div", { class: "hint" }, hint) : null
);
}
function check(label, checked, helpText) {
const cb = h("input", { type: "checkbox", checked: checked ? true : null });
return { node: h("label", { class: "check" }, cb, label), cb };
return { node: h("label", { class: "check" }, cb, label, helpText ? " " : null, help(helpText)), cb };
}
// Flatten router usage (nested account/usage objects) into readable, wrapping
// key: value rows — a raw JSON.stringify dump overflows the card horizontally.
function usageRows(obj, prefix) {
const rows = [];
for (const [k, v] of Object.entries(obj || {})) {
const key = prefix ? prefix + "." + k : k;
if (v && typeof v === "object" && !Array.isArray(v)) {
rows.push(...usageRows(v, key));
} else {
rows.push(
h("div", { class: "mono kv" },
h("span", { class: "kv-k" }, key + ": "),
h("span", { class: "kv-v" }, Array.isArray(v) ? JSON.stringify(v) : String(v)))
);
}
}
return rows;
}
// Probe status → i18n key suffix (see /admin/api/router/probe).
function probeMsgKey(status) {
const map = {
ok: "ok",
no_key: "noKey",
auth: "auth",
payment: "payment",
quota: "quota",
timeout: "timeout",
unreachable: "unreachable",
};
return "router.probe." + (map[status] || "error");
}
// Statuses where the fix is on the site (get / pay for a key).
const PROBE_SITE_STATUSES = ["no_key", "auth", "payment", "quota"];
export async function renderSettings() {
const meta = appState.meta || (await api.meta());
const s = await api.getSettings();
const [s, secretsData] = await Promise.all([api.getSettings(), api.listSecrets()]);
const root = h("div");
const listenIn = h("input", { value: s.listen || "" });
@@ -23,7 +69,109 @@ export async function renderSettings() {
const rKeyEnv = h("input", { value: s.router?.api_key_env || "", placeholder: "CORTIQ_ROUTER_KEY" });
const rTimeout = h("input", { type: "number", value: s.router?.timeout_ms ?? 800 });
const rTax = h("input", { value: s.router?.taxonomy_id || "" });
const rVerify = check(t("settings.router.verifyTls"), s.router?.verify_tls);
const rVerify = check(t("settings.router.verifyTls"), s.router?.verify_tls, t("settings.router.verifyTlsHelp"));
const rEnabled = check(t("settings.router.enabled"), s.router?.enabled !== false, t("settings.router.enabledHelp"));
// router key value (write-only; stored in the secret store, like model keys)
const rKeySource =
(secretsData.secrets || []).find((x) => x.name === (s.router?.api_key_env || "CORTIQ_ROUTER_KEY"))?.source ||
"missing";
const keyStored = rKeySource === "store" || rKeySource === "env";
// When a key is already saved, show a masked "saved" hint instead of an empty
// field (an empty field reads as "nothing is set").
const rSecret = h("input", {
type: "password",
placeholder: keyStored ? t("settings.router.keyStoredMask") : t("models.form.secretPlaceholder"),
});
const rKeyBadge = h(
"span",
{ class: "badge " + (rKeySource === "store" ? "store" : rKeySource === "env" ? "env" : "bad") },
rKeySource === "store"
? t("models.form.secretStored")
: rKeySource === "env"
? t("models.form.secretEnv")
: t("models.form.secretMissing")
);
const rSecretHint = h(
"div",
{},
h(
"div",
{},
rKeyBadge,
" · ",
t("settings.router.getKey") + " ",
h("a", { href: SITE_URL, target: "_blank", rel: "noopener" }, "api.allaigate.com")
),
h("div", { class: "muted", style: "margin-top:4px" }, t("settings.router.secretHelp"))
);
// The key lives in the secret store under this env name (config default: CORTIQ_ROUTER_KEY).
const keyName = () => rKeyEnv.value.trim() || "CORTIQ_ROUTER_KEY";
// Persist just the pasted router key (independent of the big Save at page top).
async function saveKey() {
const v = rSecret.value.trim();
if (!v) return false;
if (!rKeyEnv.value.trim()) rKeyEnv.value = "CORTIQ_ROUTER_KEY";
await api.setSecret(keyName(), v);
rSecret.value = "";
rSecret.placeholder = t("settings.router.keyStoredMask"); // show "saved", not empty
rKeyBadge.textContent = t("models.form.secretStored");
rKeyBadge.className = "badge store";
return true;
}
// Shared connection test (real /v1/route call) — tells key problems apart from
// a down router. Rendered into probeOut; also used right after saving the key.
const probeOut = h("div", { class: "hint", style: "margin-top:8px" });
async function runProbe() {
mount(probeOut, h("span", { class: "spinner" }));
try {
const r = await api.probeRouter();
const parts = [t(probeMsgKey(r.status), { ms: r.latency_ms ?? "—" })];
if (r.message) parts.push(" — " + r.message);
const out = [h("span", {}, ...parts)];
if (PROBE_SITE_STATUSES.includes(r.status)) {
out.push(" · ", h("a", { href: SITE_URL, target: "_blank", rel: "noopener" }, t("dash.health.getKey") + " ↗"));
}
if (r.usage && typeof r.usage === "object") {
out.push(
h("div", { class: "usage-box", style: "margin-top:4px" },
h("b", {}, t("router.usage")), ...usageRows(r.usage))
);
}
mount(probeOut, h("div", {}, h("span", { class: "badge " + (r.ok ? "ok" : "bad") }, r.ok ? "OK" : "×"), " ", ...out));
toast(t(probeMsgKey(r.status), { ms: r.latency_ms ?? "—" }), r.ok ? "good" : "bad");
return r.ok;
} catch (e) {
mount(probeOut, String(e.message || e));
return false;
}
}
const saveKeyBtn = h("button", { class: "btn sm" }, "✓ " + t("settings.router.saveKey"));
saveKeyBtn.addEventListener("click", async () => {
saveKeyBtn.disabled = true;
try {
if (await saveKey()) {
toast(t("settings.router.keySaved"), "good");
// Immediately test the connection so the user sees reachable/error at once,
// instead of a stale "unavailable" until something re-checks.
if (rEnabled.cb.checked) await runProbe();
} else {
toast(t("settings.router.keyEmpty"), "warn");
}
} catch (e) {
toast(String(e.message || e), "bad");
}
saveKeyBtn.disabled = false;
});
const probeBtn = h("button", { class: "btn" }, t("settings.router.test"));
probeBtn.addEventListener("click", async () => {
probeBtn.disabled = true;
// If a key was pasted but not saved, persist it first so the test uses it.
if (rSecret.value.trim()) {
try { await saveKey(); } catch (_) { /* probe will report no_key/auth */ }
}
await runProbe();
probeBtn.disabled = false;
});
// route
const strategySel = h("select", {}, ...(meta.text_strategies || []).map((x) => h("option", { value: x, selected: x === s.route?.text_strategy }, x)));
const maxChars = h("input", { type: "number", value: s.route?.max_chars ?? 4000 });
@@ -34,18 +182,18 @@ export async function renderSettings() {
const brCool = h("input", { value: s.breaker?.cooldown || "30s" });
// log
const logSel = h("select", {}, ...["error", "warn", "info", "debug", "trace"].map((x) => h("option", { value: x, selected: x === s.log?.level }, x)));
const logBodies = check(t("settings.log.bodies"), s.log?.bodies);
const logBodies = check(t("settings.log.bodies"), s.log?.bodies, t("settings.log.bodiesHelp"));
// telemetry / cortiq
const metricsCb = check(t("settings.telemetry.metrics"), s.telemetry?.metrics);
const echoCb = check(t("settings.cortiq.echo"), s.cortiq?.echo);
const metricsCb = check(t("settings.telemetry.metrics"), s.telemetry?.metrics, t("settings.telemetry.metricsHelp"));
const echoCb = check(t("settings.cortiq.echo"), s.cortiq?.echo, t("settings.cortiq.echoHelp"));
// stats
const statsEnabled = check(t("settings.stats.enabled"), s.stats?.enabled);
const statsEnabled = check(t("settings.stats.enabled"), s.stats?.enabled, t("settings.stats.enabledHelp"));
const statsFile = h("input", { value: s.stats?.file || "" });
const statsRet = h("input", { value: s.stats?.retention || "7d" });
// cache
const modelsData = await api.listModels();
const embedModels = (modelsData.models || []).filter((m) => m.kind === "embedding").map((m) => m.id);
const cacheEnabled = check(t("settings.cache.enabled"), s.cache?.enabled);
const cacheEnabled = check(t("settings.cache.enabled"), s.cache?.enabled, t("settings.cache.enabledHelp"));
const cacheThresh = h("input", { type: "number", step: "0.01", value: s.cache?.threshold ?? 0.92 });
const cacheTtlIn = h("input", { value: s.cache?.ttl || "1h" });
const cacheEmbedSel = h(
@@ -54,13 +202,104 @@ export async function renderSettings() {
h("option", { value: "" }, "auto"),
...embedModels.map((id) => h("option", { value: id, selected: id === s.cache?.embed_model }, id))
);
// local models (CMF) — run a local .cmf server and/or force local-only routing
const cmfLocalOnly = check(t("settings.cmf.localOnly"), s.cmf?.local_only, t("settings.cmf.localOnlyHelp"));
const cmfManage = check(t("settings.cmf.manage"), s.cmf?.manage_server, t("settings.cmf.manageHelp"));
const cmfAutoInstall = check(t("settings.cmf.autoInstall"), s.cmf?.auto_install, t("settings.cmf.autoInstallHelp"));
const cmfAutoUpdate = check(t("settings.cmf.autoUpdate"), s.cmf?.auto_update, t("settings.cmf.autoUpdateHelp"));
const cmfModel = h("input", { value: s.cmf?.local_model || "", placeholder: "models/my-model.cmf" });
const cmfPort = h("input", { type: "number", value: s.cmf?.local_port ?? 8081 });
const cmfThreads = h("input", { type: "number", min: 0, value: s.cmf?.threads ?? 8 });
const cmfGpu = check(t("settings.cmf.gpu"), s.cmf?.gpu, t("settings.cmf.gpuHelp"));
const cmfModelIdVal = s.cmf?.model_id || "cmf-local"; // auto — no manual "model id" field
// "Install cortiq" — the CLI that RUNS the .cmf format. Must be present before
// a local model server can start; this is step 1 for a new user.
const cmfStatusEl = h("span", { class: "muted" }, "…");
const cmfInstallBtn = h("button", { class: "btn" }, t("settings.cmf.install"));
// live "serving <model> on :<port>" indicator for the managed local server
const cmfServingEl = h("div", { class: "hint" });
async function refreshCmfStatus() {
try {
const st = await api.cmfStatus();
cmfInstallBtn.textContent = st.installed ? t("settings.cmf.reinstall") : t("settings.cmf.install");
mount(
cmfStatusEl,
st.installed
? h("span", {}, h("span", { class: "badge ok" }, "✓"), " cortiq " + (st.version || ""))
: h("span", {}, h("span", { class: "badge bad" }, "×"), " " + t("settings.cmf.notInstalled"))
);
// reflect the managed server state under the "run locally" toggle
if (st.manage_server && st.running) {
mount(cmfServingEl, h("span", { style: "color:var(--good)" },
"▶ " + t("settings.cmf.serving", { model: st.model_id || "", port: st.local_port }) + (st.healthy ? "" : " …")));
} else if (st.last_error) {
mount(cmfServingEl, h("span", { style: "color:var(--bad)" }, "✗ " + st.last_error));
} else {
mount(cmfServingEl, "");
}
} catch (_) {
mount(cmfStatusEl, "—");
}
}
cmfInstallBtn.addEventListener("click", async () => {
cmfInstallBtn.disabled = true;
mount(cmfStatusEl, h("span", {}, h("span", { class: "spinner" }), " " + t("settings.cmf.installing")));
try {
await api.cmfInstall();
// install compiles from crates.io — poll status for a few minutes
for (let i = 0; i < 120; i++) {
await new Promise((r) => setTimeout(r, 3000));
const st = await api.cmfStatus();
if (st.installed) { toast(t("settings.cmf.installedOk"), "good"); break; }
if (st.last_error) { toast(st.last_error, "bad"); break; }
}
} catch (e) {
toast(String(e.message || e), "bad");
}
await refreshCmfStatus();
cmfInstallBtn.disabled = false;
});
refreshCmfStatus();
// Port availability check — let the user confirm the chosen port is free
// before enabling the managed server (the default 8081 often clashes).
const cmfPortStatus = h("span", { class: "small", style: "margin-left:8px" });
const cmfPortCheckBtn = h("button", { class: "btn small", type: "button" }, t("settings.cmf.checkPort"));
async function checkPort() {
const p = parseInt(cmfPort.value, 10);
if (!p) { mount(cmfPortStatus, ""); return; }
cmfPortCheckBtn.disabled = true;
mount(cmfPortStatus, t("status.checking"));
try {
const r = await api.cmfPortCheck(p);
if (r.available) {
mount(cmfPortStatus, h("span", { style: "color:var(--good)" }, "✓ " + t("settings.cmf.portFree")));
} else {
const kids = [h("span", { style: "color:var(--bad)" }, "✗ " + t("settings.cmf.portBusy"))];
if (r.suggested) {
const useBtn = h("button", { class: "btn small", type: "button", style: "margin-left:8px" },
t("settings.cmf.useSuggested", { port: r.suggested }));
useBtn.addEventListener("click", () => { cmfPort.value = r.suggested; checkPort(); });
kids.push(useBtn);
}
mount(cmfPortStatus, h("span", {}, ...kids));
}
} catch (e) {
mount(cmfPortStatus, String(e.message || e));
}
cmfPortCheckBtn.disabled = false;
}
cmfPortCheckBtn.addEventListener("click", checkPort);
const saveBtn = h("button", { class: "btn primary" }, t("common.save"));
saveBtn.addEventListener("click", async () => {
saveBtn.disabled = true;
// a pasted router key needs an env name to live under — default it
if (rSecret.value.trim() && !rKeyEnv.value.trim()) rKeyEnv.value = "CORTIQ_ROUTER_KEY";
const patch = {
listen: listenIn.value.trim(),
router: {
enabled: rEnabled.cb.checked,
url: rUrl.value.trim(),
api_key_env: rKeyEnv.value.trim() || null,
timeout_ms: parseInt(rTimeout.value) || 800,
@@ -90,6 +329,18 @@ export async function renderSettings() {
max_entries: s.cache?.max_entries ?? 1000,
embed_model: cacheEmbedSel.value || null,
},
cmf: {
...(s.cmf || {}),
local_only: cmfLocalOnly.cb.checked,
manage_server: cmfManage.cb.checked,
auto_install: cmfAutoInstall.cb.checked,
auto_update: cmfAutoUpdate.cb.checked,
local_model: cmfModel.value.trim(),
local_port: parseInt(cmfPort.value) || 8081,
model_id: cmfModelIdVal,
threads: parseInt(cmfThreads.value) || 0,
gpu: cmfGpu.cb.checked,
},
};
// serde: drop null optionals
if (!patch.router.api_key_env) delete patch.router.api_key_env;
@@ -98,6 +349,13 @@ export async function renderSettings() {
if (!patch.cache.embed_model) delete patch.cache.embed_model;
try {
const r = await api.putSettings(patch);
const secretVal = rSecret.value.trim();
if (secretVal) {
await api.setSecret(rKeyEnv.value.trim(), secretVal);
rSecret.value = "";
rKeyBadge.textContent = t("models.form.secretStored");
rKeyBadge.className = "badge store";
}
toast(r.needs_restart ? t("settings.needsRestart") : t("toast.saved"), r.needs_restart ? "warn" : "good");
} catch (e) {
toast(String(e.message || e), "bad");
@@ -150,8 +408,13 @@ export async function renderSettings() {
h("div", { class: "card-head" }, h("h3", {}, t("settings.router"))),
field(t("settings.router.url"), rUrl),
field(t("settings.router.keyEnv"), rKeyEnv),
field(t("settings.router.secret"), h("div", { class: "key-row" }, rSecret, saveKeyBtn), rSecretHint),
h("div", { class: "row" }, field(t("settings.router.timeout"), rTimeout), field(t("settings.router.taxonomy"), rTax)),
rVerify.node
rEnabled.node,
rVerify.node,
h("div", { class: "divider" }),
h("div", { class: "flex" }, probeBtn),
probeOut
),
h(
"div",
@@ -199,6 +462,39 @@ export async function renderSettings() {
h("div", { class: "row" }, field(t("settings.cache.threshold"), cacheThresh), field(t("settings.cache.ttl"), cacheTtlIn)),
field(t("settings.cache.embed"), cacheEmbedSel),
h("div", { class: "hint" }, t("settings.cache.note"))
),
h(
"div",
{ class: "card" },
h("div", { class: "card-head" }, h("h3", {}, t("settings.cmf"))),
h("div", { class: "hint" }, t("settings.cmf.intro")),
// Step 1 — install the cortiq runtime (the CLI that runs .cmf models).
h(
"label",
{ class: "field" },
h("span", {}, "1. " + t("settings.cmf.runtime"), " ", help(t("settings.cmf.runtimeHelp"))),
h("div", { class: "flex", style: "gap:8px;align-items:center;flex-wrap:wrap" }, cmfInstallBtn, cmfStatusEl)
),
h("div", { class: "row" }, cmfAutoInstall.node, cmfAutoUpdate.node),
h("div", { class: "divider" }),
// Step 2 — add a model. Models are imported in the Models/Import tab; here
// you just point the local server at the .cmf file it produced.
h("div", { class: "field" }, h("span", {}, "2. " + t("settings.cmf.addModel"), " ", help(t("settings.cmf.addModelHelp"))),
h("div", { class: "hint" }, t("settings.cmf.importNote"),
" ", h("a", { href: "#/import" }, t("settings.cmf.importLink") + " ↗"))),
field(t("settings.cmf.model"), cmfModel, null, t("settings.cmf.modelHelp")),
h("div", { class: "divider" }),
// Step 3 — run it locally + performance.
cmfManage.node,
cmfServingEl,
h("div", { class: "row" },
field(t("settings.cmf.port"),
h("div", { class: "flex wrap" }, cmfPort, cmfPortCheckBtn, cmfPortStatus),
null, t("settings.cmf.portHelp")),
field(t("settings.cmf.threads"), cmfThreads, null, t("settings.cmf.threadsHelp"))),
cmfGpu.node,
cmfLocalOnly.node,
h("div", { class: "hint" }, t("settings.cmf.speedNote"))
)
),
h(