Compare commits

..
28 Commits
Author SHA1 Message Date
infosave2007 3c90dad9b9 chore(release): v0.2.6 — Settings UX, empty pool by default, CMF port check
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
- Settings: "?" help tooltips on every option; router key auto-probes the
  connection right after saving (no more hanging "unavailable")
- Local models card reworked into 3 clear steps (install cortiq → add model →
  run locally); removed the hand-entered .cmf path / pool-id fields — models
  are added via the Import tab and the managed server registers them
- Fresh install ships an EMPTY model pool; the local .cmf is auto-registered by
  the managed CMF server, so it needs no static [[models]] entry
- New: port-availability check — GET /admin/api/cmf/port + "Check port" button
  with next-free-port suggestion. bind()+connect() probe so Docker-proxied /
  dual-stack ports (e.g. 8081) are correctly detected as occupied
- /admin/api/cmf now reports the serving model + port; card shows "▶ serving …"
- On-demand runtime install: POST /admin/api/cmf/install
2026-07-08 19:07:24 +03:00
infosave2007 25f93d1a27 chore: ship the full working frontend + backend
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
The crates.io/release build shipped a STALE frontend (committed web/js was behind
the working tree), so the admin console hung on load. Commit the current working
web/js (app, api, dashboard, login, import view, i18n de/es/fr/tr/zh, css) and the
backend WIP (import.rs, admin/assets.rs) so the released console matches the dev
build. Verified: clean `cargo build --release`, empty-pool start, working
dashboard served.
2026-07-08 17:58:31 +03:00
infosave2007 0f89cdaa07 chore(release): v0.2.4
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
2026-07-08 17:39:43 +03:00
infosave2007 d211eb6150 feat: fresh install starts with an empty model pool
- config.rs: validate() no longer errors on zero models — an empty pool is
  allowed (warns instead), so the admin console comes up on a fresh install and
  the user adds models in /admin -> Models (or points [cmf] at a local .cmf).
- gateway.example.toml: ships an EMPTY pool (no seed cmf-local/local-embed, empty
  routing), so `cp example -> gateway.toml && run` shows an empty list — what a
  new user expects.
2026-07-08 17:39:43 +03:00
infosave2007 04580196ad fix: add promotion + router-status modules referenced by the CMF integration
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
The v0.2.3 CMF-integration commit referenced crate::promotion (main.rs),
router_client::RouterLastStatus (state.rs) and router_client::classify_status
(admin), but those definitions were not part of the commit, so a clean build
failed (E0583/E0432/E0425). Add the two source files; cargo build --release now
compiles.
2026-07-08 16:59:58 +03:00
infosave2007 7a3d6a7bb4 chore(release): v0.2.3 2026-07-08 16:39:30 +03:00
infosave2007 07d011263d config: example ships a local CMF model, drop cloud placeholders
The example config pool now shows one local CMF model (served by the managed
cortiq server) instead of local-qwen/gpt-4o-mini/claude-opus placeholders, with
routing pointing at it, and a documented [cmf] section (manage_server, threads,
gpu, ...). Add cloud/local providers by copying the [[models]] shape.
2026-07-08 16:39:30 +03:00
infosave2007 0040efc948 admin: surface CMF worker threads + GPU toggle in Settings
Expose the two speed knobs in the /admin Settings panel instead of TOML-only:
- [cmf].gpu (new) -> exports CMF_GPU=1 to the managed cortiq serve (Metal/wgpu).
  On Apple Silicon decode is memory-bandwidth bound so this is ~neutral for
  generation (helps prefill); off by default.
- [cmf].threads (added earlier) now has a UI number input.

- config.rs: CmfCfg.gpu bool (serde default false).
- cmf_runtime.rs: spawn_serve sets CMF_GPU=1 when gpu is on.
- admin/mod.rs: put_settings flags needs_restart on threads/gpu change (the
  server is spawned once at startup with those env vars).
- settings.js: threads input + gpu checkbox + save payload + speed hint.
- i18n en/ru: threads/gpu/speedNote keys (other locales fall back to en, as the
  whole cmf section already does there).
2026-07-08 16:18:31 +03:00
infosave2007 16a3e209b6 cmf: parallelise the managed local server (CMF_THREADS)
The gateway spawned `cortiq serve` with no thread hint, so the runtime ran the
decode loop single-threaded (~0.25 tok/s on a 15B CPU model). Export CMF_THREADS
from a new [cmf].threads config field (default 8) in spawn_serve.

Measured: 0.25 -> 0.81 tok/s (~3.2x) on qwopus-vbit7 (15B). Scaling is sublinear
(GatedDeltaNet state recurrence is serial per token, Amdahl-bound); a real jump
needs the GPU path.
2026-07-08 16:18:31 +03:00
infosave2007 c19edee7bb feat(cmf): integrate CMF as a managed local model server
Run models in the CMF (.cmf) format locally, via the native `cortiq`
CLI from crates.io — no Python and no external model server.

- cmf_runtime: install/update `cortiq-cli` from crates.io, spawn and
  health-check `cortiq serve` as a managed local backend.
- import: model-import jobs shell out to the native `cortiq convert`
  (HF repo -> .cmf, quant mapped Q8_2F->q8_2f / Q8_ROW->q8 / Q4->q4);
  the Python converter is used only for advanced/exotic options.
- pipeline: local-first routing — serve from the local CMF model when
  the router is disabled/unreachable, with a "local only" hard switch.
- registry/state: register the local CMF server as an OpenAI provider
  and track local candidates.
- config: data-dir resolution + CmfCfg (auto-install/update, managed
  server, host/port/model) and relaxed validation for the managed model.
- admin + settings UI (en/ru): "Local models (CMF)" settings card.
2026-07-08 16:18:31 +03:00
infosave2007andClaude Opus 4.8 7af90a5eef docs: add docker-compose quickstart (gateway + Ollama) and GHCR badge
- docker-compose.yml + config/gateway.compose.toml (routes to a local Ollama service)
- README (EN+RU): ghcr badge + Docker Compose section

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 13:40:24 +03:00
infosave2007andClaude Opus 4.8 8d7e5fe653 chore(release): v0.2.2
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
Benchmarks (latency + accuracy), router docs + LAUNCH1 promo, Docker image on ghcr.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 11:28:56 +03:00
infosave2007andClaude Opus 4.8 2605eccb64 feat: Docker image + ghcr publishing
- multi-stage Dockerfile (non-root, embeds SPA, default config, ~160MB) + .dockerignore
- .github/workflows/docker.yml: push to ghcr.io/infosave2007/cortiq-gateway
  (master -> :edge/:sha amd64; tags -> :X.Y.Z/:latest, linux/amd64+arm64)
- README (EN+RU): docker run instructions; CHANGELOG
- verified locally: image builds, container serves /healthz + /admin

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 11:23:09 +03:00
infosave2007andClaude Opus 4.8 1317b8024b docs: document the allaigate router + launch promo
- docs/ROUTER.md + docs/ROUTER.ru.md: what the semantic router does, hosted vs
  self-hosted, config, API contract, pricing, and the LAUNCH1 $1 Starter promo
- README (EN+RU): "The router (allaigate)" section with the promo callout

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 10:35:29 +03:00
infosave2007andClaude Opus 4.8 9242806795 docs: add task-type accuracy benchmark (router vs keyword heuristic)
- bench/tasks.jsonl (37 natural-language prompts, 7 task types) + bench/accuracy.py
- measured on the live allaigate router: 100% (37/37) vs keyword heuristic 32%
- BENCHMARKS.md accuracy section; README (EN+RU) perf+accuracy tables and test matrix
- run.sh runs accuracy when CORTIQ_ROUTER_KEY is set

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 07:45:16 +03:00
infosave2007andClaude Opus 4.8 6714c4f556 docs: add test/verification matrix to README (EN+RU)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 18:31:13 +03:00
infosave2007andClaude Opus 4.8 a5966dd479 docs: add latency benchmark vs LiteLLM/Portkey
- bench/ reproducible harness: instant mock backend + Apache Bench (ab -k -r)
- measures gateway overhead in passthrough mode (apples-to-apples)
- BENCHMARKS.md: methodology, caveats, results (cortiq ~57k rps p99 1ms vs
  Portkey ~5.8k p99 9ms vs LiteLLM 4-workers ~1.2k p99 59ms)
- README (EN+RU): Performance section + highlight

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 18:11:26 +03:00
infosave2007andClaude Opus 4.8 e015555dff feat: finish remaining inbound protocols (models, completions, native /route)
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
- GET /v1/models: pool + virtual cortiq-auto
- POST /v1/completions (legacy): non-stream + streaming text_completion
- POST /route: native passthrough returning the routing decision
- all inbound protocol adapters now implemented; admin meta + README/CHANGELOG updated
- bump to 0.2.1

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 17:42:45 +03:00
infosave2007andClaude Opus 4.8 f00bf8355a chore(release): v0.2.0
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
Streaming (SSE), Anthropic provider/inbound, embeddings, semantic cache, MCP server.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 17:17:04 +03:00
infosave2007andClaude Opus 4.8 ddca3a8363 feat: MCP (Model Context Protocol) server
- POST /mcp (JSON-RPC 2.0): initialize, ping, tools/list, tools/call; notifications -> 202
- tools: cortiq_route (classify -> routing decision) and cortiq_chat (route -> answer)
- gated by protocols.mcp; admin meta marks it implemented
- docs: README/README.ru protocol tables, highlights and status updated for
  streaming, Anthropic, embeddings, semantic cache and MCP

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 17:12:55 +03:00
infosave2007andClaude Opus 4.8 a67564bc24 feat: semantic (embedding-based) response cache
- cache.rs: cosine-similarity store (threshold, TTL, ring buffer), partitioned by
  routing signature; tracks hit rate and model cost saved
- pipeline: embed routing text, look up before the model call (route_source=cache on
  hit), insert successful responses; stats record cache hits
- config [cache] + admin settings card; dashboard shows hit-rate and savings
- embedding model resolved from the pool (or first kind="embedding")

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 17:04:53 +03:00
infosave2007andClaude Opus 4.8 7dbc39c6a6 feat: Anthropic provider/inbound + embeddings
- Anthropic outbound provider: /v1/messages, system extraction, x-api-key +
  anthropic-version; streaming translates Anthropic SSE -> unified OpenAI wire format
- Anthropic inbound adapter (POST /v1/messages): non-streaming JSON + streaming SSE
  (OpenAI -> Anthropic event translation)
- Embeddings: POST /v1/embeddings inbound + Provider::embed(); embedding models
  resolvable from the pool (registry no longer skips them)
- protocols mounted always, gated on live flags; admin meta marks them implemented

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 16:50:22 +03:00
infosave2007andClaude Opus 4.8 b6fdf23aa6 feat: streaming (SSE) end-to-end
- Provider::chat_stream; OpenAI provider streams via stream_options.include_usage
- pipeline.run_stream: shared route/select, failover before first byte
- /v1/chat/completions forwards SSE verbatim with X-Cortiq-* headers; usage tapped for stats
- admin /test/stream + live streaming in Playground (stream toggle, 7 langs)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 16:37:35 +03:00
infosave2007andClaude Opus 4.8 220df29888 ci: build Windows binaries; add cross-platform CI test matrix
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
- release.yml: add x86_64-pc-windows-msvc target, OS-aware packaging (.exe + .zip)
- ci.yml: split into lint (fmt/clippy) + test matrix (ubuntu/windows/macos)
- use getrandom for the admin/key token CSPRNG (secure on Windows too)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 14:56:54 +03:00
infosave2007andClaude Opus 4.8 74d0295181 docs: add crates.io install instructions to both READMEs
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 14:11:07 +03:00
infosave2007andClaude Opus 4.8 73a2bd7b42 ci(release): use env-mapped token guard for crates.io publish step
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 13:54:35 +03:00
infosave2007andClaude Opus 4.8 e97fcddf87 chore(release): prepare v0.1.0 for crates.io
- bump version to 0.1.0; add crates.io metadata (include list, homepage, rust-version)
- add .github/workflows/release.yml: on tag v* → publish to crates.io + GitHub Release with binaries
- README: absolute links/images and badges so it renders on crates.io; add install section
- CHANGELOG: 0.1.0

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 13:47:42 +03:00
infosave2007andClaude Opus 4.8 51487e6329 feat: universal LLM gateway with embedded multilingual admin console
- OpenAI-compatible gateway with intelligent routing via cortiq-router (allaigate)
- Embedded build-free admin SPA at /admin: 7 languages (en/ru/de/fr/es/zh/tr),
  light/dark themes, hot config reload (no restart)
- Admin API, per-request statistics + Prometheus /metrics, secret store, playground
- English documentation (primary) with Russian translations (*.ru.md)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 10:51:37 +03:00
29 changed files with 402 additions and 2548 deletions
-58
View File
@@ -3,66 +3,8 @@
All notable changes to this project are documented here. All notable changes to this project are documented here.
The format is based on [Keep a Changelog](https://keepachangelog.com/). The format is based on [Keep a Changelog](https://keepachangelog.com/).
## [0.2.35] - 2026-07-23
### Fixed
- **Streaming Pipeline System Prompt Merging**: Fixed system prompt injection in `run_stream` so configured model system prompts and reasoning suppression directives are properly passed into streaming requests.
- **CMF Reasoning Mode Compatibility**: Fully integrated with `cortiq-engine` v0.5.17 for sampler-level logit bias reasoning token suppression and zero-token thinking bypass.
## [0.2.34] - 2026-07-23
### Added
- **Responsive mobile admin console**: Added a hamburger navigation drawer, touch-friendly controls, phone-sized forms and cards, scrollable tables, compact toolbars, safe-area support, and mobile bottom-sheet dialogs across the admin interface.
- **Accessible mobile interactions**: Navigation and dialogs now support backdrop dismissal, Escape-key closing, reduced-motion preferences, and explicit accessibility attributes.
### Fixed
- **Add model dialog**: Restored the missing `opt` helper import that prevented the Add model button from opening its form.
- **Local model settings on phones**: Reflowed managed CMF server controls to avoid clipped fields and actions on narrow screens.
## [0.2.31] - 2026-07-22
### Added
- **Structured O(1) Nyström Attention & Reasoning Control**: Enhanced settings drawer with structured parameter sections for O(1) Nyström Streaming Attention (`o1_m`, `o1_window`, `o1_sink`), Generation (Temp, Top-P, MaxTokens, System Prompt), Reasoning ("Disable reasoning" toggle mapping to `think_budget = 0` for Qwen3/3.5 models), and Runtime (`skip_mtp` via `CMF_MTP=0`).
- **Persistent Admin Token Configuration**: Support for explicit `token` field in `config/gateway.toml` under `[admin]` block to prevent authorization invalidation across restarts.
### Fixed
- **JavaScript UI Helper Restoration**: Restored `opt(...)` helper in `web/js/ui.js` and `web/js/views/import.js` to ensure HF model import modal options render smoothly.
- **Engine Command Flag Accuracy**: Fixed `CMF_MTP=0` environment flag pass-through for `cortiq serve` processes.
## [0.2.30] - 2026-07-22
### Added
- **Per-Model Generation & Reasoning Parameters**: Full support in Web UI and Gateway core for per-model generation defaults: Temperature, Top-P, Max Tokens, Think Budget (Reasoning mode), O(1) Constant-Memory Streaming Attention (`o1`), MTP (Multi-Token Prediction toggle `skip_mtp`), and default System Prompt.
### Fixed
- **Duplicate Local Model Registration**: Fixed `duplicate managed model id` conflict in `register_import` by cleanly routing managed CMF models directly into `[[cmf.servers]]` with automatic free port allocation without pushing duplicate entries to `[[models]]`.
### Fixed
- **Quantization Badge & Title for Pre-Converted Models**: Automatically detect quantization (`Q1`, `Q1T`) for ready-to-run `.cmf` models (e.g. `infosave/Bonsai-27Bcmf`) from repo name and downloaded filename, resolving default `Q8_2F` badge fallback. Updated section title to "Загрузки / Конвертации" ("Downloads & Conversions").
### Fixed
- **Automatic CMF Local Server Lifecycle**: Registering an imported `.cmf` model automatically adds it to `[[cmf.servers]]` and reloads the runtime (`cmf_runtime::manage`), automatically spawning `cortiq serve` on port 8090 so chat completion requests route seamlessly without `upstream unavailable` errors.
### Fixed
- **Multi-GB Ready CMF Download Stream**: Replaced HTTP stream decoder with a streaming process downloader (`curl`) for pre-converted `.cmf` models to support multi-gigabyte Hugging Face LFS repositories without 30s/60s socket timeouts.
- **Ready CMF Quantization & UI Labels**: Set correct quantization badges (`Q1`, `Q1T`) for featured ready-to-run `.cmf` models and removed misleading "converting" log messages for direct downloads.
### Fixed
- **Direct Ready CMF Download**: Importing pre-converted `.cmf` model repositories (e.g. `infosave/Bonsai-8B_2bit_cmf`) directly streams the ready `.cmf` file from Hugging Face instead of attempting a safetensors conversion without `config.json`.
### Added
- **New Quantization Schemes**: Added support for `Q4_TILED` (q4t), `Q1T` (1.58-bit ternary), `Q1S` (1-bit sparse), `Q1P` (1-bit PTQ error diffusion), and `Q1` (pure binary) in converter backend and web admin UI.
- **Featured Ready-to-Run .cmf Models**: Added 1-tap download cards for pre-converted `.cmf` models (`infosave/Bonsai-1.7Bcmf`, `infosave/Bonsai-8B_2bit_cmf`, `infosave/Bonsai-27Bcmf`).
## [Unreleased] ## [Unreleased]
### Added ### Added
- **Docker Compose** quickstart (`docker-compose.yml` + `config/gateway.compose.toml`): - **Docker Compose** quickstart (`docker-compose.yml` + `config/gateway.compose.toml`):
gateway + a local Ollama model in one command. GHCR badge in the READMEs. gateway + a local Ollama model in one command. GHCR badge in the READMEs.
Generated
+1 -1
View File
@@ -285,7 +285,7 @@ checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570"
[[package]] [[package]]
name = "cortiq-gateway" name = "cortiq-gateway"
version = "0.2.34" version = "0.2.6"
dependencies = [ dependencies = [
"anyhow", "anyhow",
"arc-swap", "arc-swap",
+1 -1
View File
@@ -1,6 +1,6 @@
[package] [package]
name = "cortiq-gateway" name = "cortiq-gateway"
version = "0.2.35" version = "0.2.6"
edition = "2021" edition = "2021"
rust-version = "1.75" rust-version = "1.75"
license = "Apache-2.0" license = "Apache-2.0"
+54 -470
View File
@@ -165,21 +165,16 @@ pub fn api_routes(admin_token: String) -> Router<SharedState> {
"/admin/api/secrets", "/admin/api/secrets",
get(list_secrets).put(set_secret).delete(clear_secret), get(list_secrets).put(set_secret).delete(clear_secret),
) )
.route("/admin/api/stats", get(get_stats).delete(clear_stats)) .route("/admin/api/stats", get(get_stats))
.route("/admin/api/shadow", get(get_shadow)) .route("/admin/api/shadow", get(get_shadow))
.route("/admin/api/hf/search", get(hf_search)) .route("/admin/api/hf/search", get(hf_search))
.route("/admin/api/import", get(list_imports).post(start_import)) .route("/admin/api/import", get(list_imports).post(start_import))
.route( .route("/admin/api/import/:job", get(import_status).delete(delete_import))
"/admin/api/import/:job",
get(import_status).delete(delete_import),
)
.route("/admin/api/import/:job/cancel", post(cancel_import)) .route("/admin/api/import/:job/cancel", post(cancel_import))
.route("/admin/api/import/:job/register", post(register_import)) .route("/admin/api/import/:job/register", post(register_import))
.route("/admin/api/cmf", get(cmf_status)) .route("/admin/api/cmf", get(cmf_status))
.route("/admin/api/cmf/install", post(cmf_install)) .route("/admin/api/cmf/install", post(cmf_install))
.route("/admin/api/cmf/port", get(cmf_port_check)) .route("/admin/api/cmf/port", get(cmf_port_check))
.route("/admin/api/cmf/files", get(cmf_files))
.route("/admin/api/provider/models", post(provider_models))
.route("/admin/api/requests", get(get_requests)) .route("/admin/api/requests", get(get_requests))
.route("/admin/api/test", post(run_test)) .route("/admin/api/test", post(run_test))
.route("/admin/api/test/stream", post(run_test_stream)) .route("/admin/api/test/stream", post(run_test_stream))
@@ -392,200 +387,18 @@ async fn router_probe(State(state): State<SharedState>) -> ApiResult {
ok(json!({ "ok": true, "status": "ok", "latency_ms": latency_ms, "usage": usage })) ok(json!({ "ok": true, "status": "ok", "latency_ms": latency_ms, "usage": usage }))
} }
#[derive(Deserialize)]
struct ProviderModelsBody {
provider: String,
base_url: String,
/// Key typed into the form (takes priority)…
#[serde(default)]
api_key: Option<String>,
/// …or the name of a stored secret to reuse if no key was typed.
#[serde(default)]
api_key_env: Option<String>,
}
/// Concise error line from a provider's JSON error body.
fn provider_err(body: &str, status: u16) -> String {
if let Ok(v) = serde_json::from_str::<Value>(body) {
if let Some(m) = v["error"]["message"]
.as_str()
.or_else(|| v["error"].as_str())
.or_else(|| v["message"].as_str())
{
return format!("{status}: {m}");
}
}
format!("HTTP {status}")
}
/// Validate a provider API key and list its models by calling the provider's
/// `/models` endpoint — powers the "Test key & load models" button. The key is
/// used transiently (from the form) or resolved from a stored secret; it is not
/// persisted here.
async fn provider_models(
State(state): State<SharedState>,
Json(b): Json<ProviderModelsBody>,
) -> ApiResult {
let base = b.base_url.trim_end_matches('/').to_string();
if base.is_empty() {
return Err(ApiError::bad("base URL is required"));
}
let key = b.api_key.filter(|k| !k.trim().is_empty()).or_else(|| {
b.api_key_env
.as_deref()
.and_then(|e| state.secrets.resolve(e))
});
let key = key.as_deref();
let client = reqwest::Client::builder()
.timeout(std::time::Duration::from_secs(15))
.build()
.map_err(|e| ApiError::bad(e.to_string()))?;
// Anthropic uses x-api-key + anthropic-version and /v1/models; every other
// (OpenAI-compatible) provider uses a Bearer token and /models.
let resp = if b.provider == "anthropic" {
let mut r = client
.get(format!("{base}/v1/models"))
.header("anthropic-version", "2023-06-01");
if let Some(k) = key {
r = r.header("x-api-key", k);
}
r.send().await
} else {
let mut r = client.get(format!("{base}/models"));
if let Some(k) = key {
r = r.header("authorization", format!("Bearer {k}"));
}
r.send().await
};
let resp = resp.map_err(|e| ApiError::bad(format!("request failed: {e}")))?;
let status = resp.status().as_u16();
let body = resp.text().await.unwrap_or_default();
if !(200..300).contains(&status) {
return ok(json!({ "ok": false, "status": status, "error": provider_err(&body, status) }));
}
// OpenAI + Anthropic both return { "data": [ { "id": ... } ] }.
let v: Value = serde_json::from_str(&body).unwrap_or(json!({}));
let mut models: Vec<Value> = v["data"]
.as_array()
.map(|arr| {
arr.iter()
.filter_map(|m| {
let id = m["id"].as_str()?.to_string();
let mut obj = json!({ "id": id });
let mut caps = vec![];
if let Some(pricing) = m.get("pricing").filter(|v| !v.is_null()) {
if let Some(prompt) = pricing
.get("prompt")
.and_then(|s| s.as_str())
.and_then(|s| s.parse::<f64>().ok())
{
obj["price_in"] = json!(prompt * 1_000_000.0);
}
if let Some(comp) = pricing
.get("completion")
.and_then(|s| s.as_str())
.and_then(|s| s.parse::<f64>().ok())
{
obj["price_out"] = json!(comp * 1_000_000.0);
}
if pricing
.get("input_cache_read")
.filter(|v| !v.is_null())
.is_some()
|| pricing
.get("input_cache_write")
.filter(|v| !v.is_null())
.is_some()
{
caps.push("caching");
}
}
if let Some(params) = m.get("supported_parameters").and_then(|p| p.as_array()) {
let has_tools = params.iter().any(|p| {
p.as_str() == Some("tools") || p.as_str() == Some("tool_choice")
});
if has_tools {
caps.push("tools");
}
let has_json = params.iter().any(|p| p.as_str() == Some("response_format"));
if has_json {
caps.push("json");
}
let has_reasoning = params.iter().any(|p| {
p.as_str() == Some("reasoning")
|| p.as_str() == Some("include_reasoning")
});
if has_reasoning || m.get("reasoning").filter(|v| !v.is_null()).is_some() {
caps.push("reasoning");
}
}
if let Some(arch) = m.get("architecture").filter(|v| !v.is_null()) {
if let Some(inputs) =
arch.get("input_modalities").and_then(|i| i.as_array())
{
if inputs.iter().any(|i| i.as_str() == Some("image")) {
caps.push("vision");
}
if inputs.iter().any(|i| i.as_str() == Some("audio")) {
caps.push("audio");
}
}
}
if !caps.is_empty() {
obj["caps"] = json!(caps);
}
Some(obj)
})
.collect()
})
.unwrap_or_default();
models.sort_by(|a, b| a["id"].as_str().cmp(&b["id"].as_str()));
let mut resp = json!({ "ok": true, "count": models.len(), "models": models });
if b.provider == "openrouter" {
if let Some(k) = key {
if let Ok(cred_resp) = client
.get("https://openrouter.ai/api/v1/credits")
.header("authorization", format!("Bearer {k}"))
.send()
.await
{
if let Ok(cred_body) = cred_resp.text().await {
let cred_v: Value = serde_json::from_str(&cred_body).unwrap_or(json!({}));
if let Some(data) = cred_v.get("data") {
let total = data
.get("total_credits")
.and_then(|v| v.as_f64())
.unwrap_or(0.0);
let usage = data
.get("total_usage")
.and_then(|v| v.as_f64())
.unwrap_or(0.0);
resp["balance"] = json!(total - usage);
}
}
}
}
}
ok(resp)
}
async fn meta() -> ApiResult { async fn meta() -> ApiResult {
ok(json!({ ok(json!({
"version": env!("CARGO_PKG_VERSION"), "version": env!("CARGO_PKG_VERSION"),
"providers": ["openai", "anthropic", "openrouter", "lmstudio", "ollama", "http"], "providers": ["openai", "anthropic", "ollama", "http"],
"provider_impl": { "openai": true, "anthropic": true, "openrouter": true, "lmstudio": true, "ollama": true, "http": true }, "provider_impl": { "openai": true, "ollama": true, "http": true, "anthropic": true },
"cost_tiers": ["cheap", "mid", "expensive", "local"], "cost_tiers": ["cheap", "mid", "expensive"],
"kinds": ["chat", "embedding"], "kinds": ["chat", "embedding"],
"profiles": ["cost-saver", "balanced", "quality-first"], "profiles": ["cost-saver", "balanced", "quality-first"],
"text_strategies": ["last_user", "last_user_plus_system", "concat_all"], "text_strategies": ["last_user", "last_user_plus_system", "concat_all"],
"policy_modes": ["fixed_table", "cost_aware"], "policy_modes": ["fixed_table", "cost_aware"],
"tiers": ["low", "medium", "high"], "tiers": ["low", "medium", "high"],
"caps": ["tools", "vision", "audio", "reasoning", "json", "caching"], "caps": ["tools", "vision"],
"protocols_impl": { "protocols_impl": {
"openai_chat": true, "openai_completions": true, "openai_embeddings": true, "openai_chat": true, "openai_completions": true, "openai_embeddings": true,
"openai_models": true, "anthropic_messages": true, "mcp": true, "native_passthrough": true "openai_models": true, "anthropic_messages": true, "mcp": true, "native_passthrough": true
@@ -610,7 +423,7 @@ async fn put_config(State(state): State<SharedState>, Json(cfg): Json<Config>) -
async fn list_models(State(state): State<SharedState>) -> ApiResult { async fn list_models(State(state): State<SharedState>) -> ApiResult {
let live = state.live(); let live = state.live();
let mut models: Vec<Value> = live let models: Vec<Value> = live
.cfg .cfg
.models .models
.iter() .iter()
@@ -624,31 +437,6 @@ async fn list_models(State(state): State<SharedState>) -> ApiResult {
v v
}) })
.collect(); .collect();
// Managed local CMF servers are registered in the routing registry from the
// [cmf] config, not from [[models]]. Surface each here (marked managed, so
// the UI shows it read-only) so the real running models are visible and
// pinnable without a hand-maintained static pool entry. A fresh install with
// no managed model still returns an empty list.
let cmf = &live.cfg.cmf;
for s in cmf.effective_servers() {
if models.iter().any(|m| m["id"] == json!(s.id)) {
continue;
}
models.push(json!({
"id": s.id,
"provider": "openai",
"base_url": format!("http://{}:{}/v1", cmf.local_host, s.port),
"model": "cortiq",
"cost_tier": "local",
"price_in": 0.0,
"price_out": 0.0,
"kind": "chat",
"caps": [],
"key_source": "none",
"managed": true,
}));
}
ok(json!({ "models": models })) ok(json!({ "models": models }))
} }
@@ -682,132 +470,28 @@ async fn update_model(
ok(json!({ "ok": true })) ok(json!({ "ok": true }))
} }
/// Remove a `.cmf` path — it may be a plain file or a tensor-dir. Returns whether
/// something was actually removed.
fn remove_cmf_path(path: &std::path::Path) -> bool {
if path.is_dir() {
std::fs::remove_dir_all(path).is_ok()
} else if path.is_file() {
std::fs::remove_file(path).is_ok()
} else {
false
}
}
/// The local `.cmf` backing a pool model, if it is a local CMF model whose file
/// lives under `models_dir`. Cloud models (real remote base_url) return None.
/// Registered imports store `model` = the file stem under models_dir.
fn cmf_file_for_model(
cmf: &crate::config::CmfCfg,
m: &crate::config::ModelCfg,
) -> Option<std::path::PathBuf> {
let local = m.base_url == cmf.cortiq_server_url
|| m.base_url
.contains(&format!("{}:{}", cmf.local_host, cmf.local_port));
let stem = m.model.trim();
if !local || stem.is_empty() || stem.contains('/') || stem.contains("..") {
return None; // not local, or an unsafe/underivable path
}
Some(std::path::Path::new(&cmf.models_dir).join(format!("{stem}.cmf")))
}
/// Drop a model id from routing (default + every tier) so `validate()` still
/// passes after the model is gone.
fn strip_id_from_routing(routing: &mut RoutingCfg, id: &str) {
if routing.default == id {
routing.default = String::new();
}
for tt in routing.tiers.values_mut() {
match tt {
crate::config::TierTargets::List(v) => v.retain(|x| x != id),
crate::config::TierTargets::One(s) if s == id => {
*tt = crate::config::TierTargets::List(Vec::new());
}
crate::config::TierTargets::One(_) => {}
}
}
}
/// Whether any managed server or static model in `cfg` still uses `path` as its
/// local `.cmf`. Guards against deleting a file that another model shares.
fn cmf_path_referenced(cfg: &crate::config::Config, path: &std::path::Path) -> bool {
cfg.cmf
.effective_servers()
.iter()
.any(|s| std::path::Path::new(&s.model) == path)
|| cfg
.models
.iter()
.any(|m| cmf_file_for_model(&cfg.cmf, m).as_deref() == Some(path))
}
async fn delete_model(State(state): State<SharedState>, Path(id): Path<String>) -> ApiResult { async fn delete_model(State(state): State<SharedState>, Path(id): Path<String>) -> ApiResult {
let mut cfg = current_cfg(&state); let mut cfg = current_cfg(&state);
// Managed local models live in [cmf], not [[models]]. Deleting one stops its
// server, removes it from the managed config, and deletes its file — unless
// another model still points at the same file.
let managed = cfg.cmf.effective_servers().into_iter().find(|s| s.id == id);
if let Some(server) = managed {
if !cfg.models.iter().any(|x| x.id == id) {
if cfg.cmf.servers.iter().any(|s| s.id == id) {
cfg.cmf.servers.retain(|s| s.id != id); // multi-model list entry
} else {
cfg.cmf.local_model = String::new(); // legacy single model
cfg.cmf.manage_server = false;
}
strip_id_from_routing(&mut cfg.routing, &id);
let path = std::path::PathBuf::from(&server.model);
let shared = cmf_path_referenced(&cfg, &path);
state.cmf.stop_one(&id).await;
state.reload(cfg)?;
let file_removed = if shared {
false
} else {
remove_cmf_path(&path)
};
return ok(json!({ "ok": true, "file_removed": file_removed }));
}
}
let victim = cfg.models.iter().find(|x| x.id == id).cloned();
let before = cfg.models.len(); let before = cfg.models.len();
cfg.models.retain(|x| x.id != id); cfg.models.retain(|x| x.id != id);
if cfg.models.len() == before { if cfg.models.len() == before {
return Err(ApiError::not_found(format!("model '{id}' not found"))); return Err(ApiError::not_found(format!("model '{id}' not found")));
} }
strip_id_from_routing(&mut cfg.routing, &id);
let file = victim
.as_ref()
.and_then(|m| cmf_file_for_model(&cfg.cmf, m));
// Only remove the file if no remaining model still references it.
let file = file.filter(|p| !cmf_path_referenced(&cfg, p));
state.reload(cfg)?; state.reload(cfg)?;
let file_removed = file.map(|p| remove_cmf_path(&p)).unwrap_or(false); ok(json!({ "ok": true }))
ok(json!({ "ok": true, "file_removed": file_removed }))
} }
async fn probe_model(State(state): State<SharedState>, Path(id): Path<String>) -> ApiResult { async fn probe_model(State(state): State<SharedState>, Path(id): Path<String>) -> ApiResult {
let live = state.live(); let live = state.live();
// Provider string: from the static pool, or "openai" for a managed local let mcfg = live
// model (which is synthesized into the registry, not [[models]]).
let provider_str = live
.cfg .cfg
.models .models
.iter() .iter()
.find(|m| m.id == id) .find(|m| m.id == id)
.map(|m| m.provider.clone()) .ok_or_else(|| ApiError::not_found(format!("model '{id}' not found")))?
.or_else(|| { .clone();
live.cfg
.cmf
.effective_servers()
.iter()
.find(|s| s.id == id)
.map(|_| "openai".to_string())
})
.ok_or_else(|| ApiError::not_found(format!("model '{id}' not found")))?;
if provider_str == "anthropic" { if mcfg.provider == "anthropic" {
return ok(json!({ return ok(json!({
"ok": false, "ok": false,
"error": "anthropic provider not yet implemented (planned v0.2)", "error": "anthropic provider not yet implemented (planned v0.2)",
@@ -844,21 +528,7 @@ async fn probe_model(State(state): State<SharedState>, Path(id): Path<String>) -
let latency_ms = started.elapsed().as_millis() as u64; let latency_ms = started.elapsed().as_millis() as u64;
match res { match res {
Ok(_) => ok(json!({ "ok": true, "latency_ms": latency_ms })), Ok(_) => ok(json!({ "ok": true, "latency_ms": latency_ms })),
Err(e) => { Err(e) => ok(json!({ "ok": false, "latency_ms": latency_ms, "error": e.to_string() })),
let msg = e.to_string();
// Make auth failures actionable — a raw "User not found" (OpenRouter's
// 401) usually means the API key is wrong or the wrong type.
let hint = if msg.contains("401")
|| msg.contains("Unauthorized")
|| msg.contains("User not found")
|| msg.contains("No auth")
{
" — check the API key (for OpenRouter use an inference key, not a provisioning/management key)"
} else {
""
};
ok(json!({ "ok": false, "latency_ms": latency_ms, "error": format!("{msg}{hint}") }))
}
} }
} }
@@ -996,7 +666,6 @@ async fn put_settings(State(state): State<SharedState>, Json(b): Json<SettingsBo
|| v.local_model != cfg.cmf.local_model || v.local_model != cfg.cmf.local_model
|| v.threads != cfg.cmf.threads || v.threads != cfg.cmf.threads
|| v.gpu != cfg.cmf.gpu || v.gpu != cfg.cmf.gpu
|| v.servers != cfg.cmf.servers
{ {
needs_restart = true; needs_restart = true;
} }
@@ -1014,32 +683,19 @@ async fn cmf_status(State(state): State<SharedState>) -> ApiResult {
let cmf = state.live().cfg.cmf.clone(); let cmf = state.live().cfg.cmf.clone();
let ver = crate::cmf_runtime::installed_version(&cmf.cortiq_bin); let ver = crate::cmf_runtime::installed_version(&cmf.cortiq_bin);
let st = state.cmf.status(); let st = state.cmf.status();
// Each configured managed server, merged with its live runtime status.
let servers: Vec<Value> = cmf
.effective_servers()
.iter()
.map(|s| {
let live = st.servers.iter().find(|x| x.id == s.id);
json!({
"id": s.id,
"model": s.model,
"port": s.port,
"threads": s.threads,
"gpu": s.gpu,
"running": live.map(|l| l.running).unwrap_or(false),
"healthy": live.map(|l| l.healthy).unwrap_or(false),
"last_error": live.and_then(|l| l.last_error.clone()),
})
})
.collect();
ok(json!({ ok(json!({
"installed": ver.is_some(), "installed": ver.is_some(),
"version": ver, "version": ver,
"latest": st.latest_version, "running": st.running,
"manage_server": cmf.manage_server, "healthy": st.healthy,
"local_host": cmf.local_host, "last_error": st.last_error,
"servers": servers,
"log": st.log, "log": st.log,
// serving info so the card can show "serving <model> on :<port>"
"manage_server": cmf.manage_server,
"local_model": cmf.local_model,
"model_id": cmf.model_id,
"local_host": cmf.local_host,
"local_port": cmf.local_port,
})) }))
} }
@@ -1051,36 +707,6 @@ async fn cmf_install(State(state): State<SharedState>) -> ApiResult {
ok(json!({ "started": true })) ok(json!({ "started": true }))
} }
/// List the `.cmf` model files available under `models_dir` so the UI can offer
/// them as a dropdown instead of a hand-typed path. Includes tensor-dir `.cmf`.
async fn cmf_files(State(state): State<SharedState>) -> ApiResult {
let dir = state.live().cfg.cmf.models_dir.clone();
let mut entries: Vec<(String, String, Option<u64>, bool)> = Vec::new();
if let Ok(rd) = std::fs::read_dir(&dir) {
for e in rd.flatten() {
let name = e.file_name().to_string_lossy().to_string();
if !name.ends_with(".cmf") {
continue;
}
let p = e.path();
let is_dir = p.is_dir();
let size = if is_dir {
None
} else {
std::fs::metadata(&p).ok().map(|m| m.len())
};
let path = format!("{}/{}", dir.trim_end_matches('/'), name);
entries.push((name, path, size, is_dir));
}
}
entries.sort_by(|a, b| a.0.cmp(&b.0));
let files: Vec<Value> = entries
.into_iter()
.map(|(name, path, size, is_dir)| json!({ "name": name, "path": path, "size": size, "is_dir": is_dir }))
.collect();
ok(json!({ "dir": dir, "files": files }))
}
#[derive(Deserialize)] #[derive(Deserialize)]
struct PortQuery { struct PortQuery {
port: u16, port: u16,
@@ -1100,7 +726,7 @@ async fn cmf_port_check(State(state): State<SharedState>, Query(q): Query<PortQu
}; };
let st = state.cmf.status(); let st = state.cmf.status();
let port = q.port; let port = q.port;
let running_here = st.servers.iter().any(|s| s.port == port && s.running); let running_here = port == cmf.local_port && st.running;
// The bind+connect probe is blocking std::net; run it off the async runtime. // The bind+connect probe is blocking std::net; run it off the async runtime.
let (available, detail, suggested) = tokio::task::spawn_blocking(move || { let (available, detail, suggested) = tokio::task::spawn_blocking(move || {
@@ -1110,11 +736,7 @@ async fn cmf_port_check(State(state): State<SharedState>, Query(q): Query<PortQu
if port_is_free(&host, port) { if port_is_free(&host, port) {
(true, "free", None) (true, "free", None)
} else { } else {
( (false, "occupied by another process", next_free_port(&host, port))
false,
"occupied by another process",
next_free_port(&host, port),
)
} }
}) })
.await .await
@@ -1292,13 +914,6 @@ async fn get_stats(State(state): State<SharedState>, Query(q): Query<StatsQuery>
ok(snap) ok(snap)
} }
/// Clear all recorded stats/logs (in-memory aggregates, series, recent requests)
/// and truncate the JSONL log — the dashboard "Clear logs" button.
async fn clear_stats(State(state): State<SharedState>) -> ApiResult {
state.stats.clear();
ok(json!({ "ok": true }))
}
/// Self-warming shadow loop: per-task-type promotion state, Wilson-LB of the /// Self-warming shadow loop: per-task-type promotion state, Wilson-LB of the
/// judged pass-rate, and whether the local model is serving it yet. /// judged pass-rate, and whether the local model is serving it yet.
async fn get_shadow(State(state): State<SharedState>) -> ApiResult { async fn get_shadow(State(state): State<SharedState>) -> ApiResult {
@@ -1359,7 +974,10 @@ async fn start_import(
ok(json!({ "job": id })) ok(json!({ "job": id }))
} }
async fn import_status(State(state): State<SharedState>, Path(job): Path<String>) -> ApiResult { async fn import_status(
State(state): State<SharedState>,
Path(job): Path<String>,
) -> ApiResult {
match state.imports.get(&job) { match state.imports.get(&job) {
Some(j) => ok(serde_json::to_value(j).unwrap_or_else(|_| json!({}))), Some(j) => ok(serde_json::to_value(j).unwrap_or_else(|_| json!({}))),
None => Err(ApiError::not_found(format!("job '{job}' not found"))), None => Err(ApiError::not_found(format!("job '{job}' not found"))),
@@ -1367,7 +985,10 @@ async fn import_status(State(state): State<SharedState>, Path(job): Path<String>
} }
/// Cancel a running conversion: kills the converter and removes partial output. /// Cancel a running conversion: kills the converter and removes partial output.
async fn cancel_import(State(state): State<SharedState>, Path(job): Path<String>) -> ApiResult { async fn cancel_import(
State(state): State<SharedState>,
Path(job): Path<String>,
) -> ApiResult {
if state.imports.get(&job).is_none() { if state.imports.get(&job).is_none() {
return Err(ApiError::not_found(format!("job '{job}' not found"))); return Err(ApiError::not_found(format!("job '{job}' not found")));
} }
@@ -1376,7 +997,10 @@ async fn cancel_import(State(state): State<SharedState>, Path(job): Path<String>
} }
/// Delete a finished conversion and its converted `.cmf` file(s) from disk. /// Delete a finished conversion and its converted `.cmf` file(s) from disk.
async fn delete_import(State(state): State<SharedState>, Path(job): Path<String>) -> ApiResult { async fn delete_import(
State(state): State<SharedState>,
Path(job): Path<String>,
) -> ApiResult {
if state.imports.get(&job).is_none() { if state.imports.get(&job).is_none() {
return Err(ApiError::not_found(format!("job '{job}' not found"))); return Err(ApiError::not_found(format!("job '{job}' not found")));
} }
@@ -1385,7 +1009,10 @@ async fn delete_import(State(state): State<SharedState>, Path(job): Path<String>
} }
/// Register a finished `.cmf` as a local model (OpenAI provider → cortiq-server). /// Register a finished `.cmf` as a local model (OpenAI provider → cortiq-server).
async fn register_import(State(state): State<SharedState>, Path(job): Path<String>) -> ApiResult { async fn register_import(
State(state): State<SharedState>,
Path(job): Path<String>,
) -> ApiResult {
let j = state let j = state
.imports .imports
.get(&job) .get(&job)
@@ -1399,64 +1026,21 @@ async fn register_import(State(state): State<SharedState>, Path(job): Path<Strin
.unwrap_or_else(|| "cmf-local".into()); .unwrap_or_else(|| "cmf-local".into());
let id = format!("cmf-{base}"); let id = format!("cmf-{base}");
let mut cfg = current_cfg(&state); let mut cfg = current_cfg(&state);
if cfg.models.iter().any(|m| m.id == id) {
let is_in_models = cfg.models.iter().any(|m| m.id == id); return Err(ApiError::bad(format!("model '{id}' already registered")));
let is_in_servers = cfg
.cmf
.servers
.iter()
.any(|s| s.id == id || s.model == j.output);
if is_in_models || is_in_servers {
return Err(ApiError::bad(format!("model '{id}' is already registered")));
}
if cfg.cmf.manage_server {
let host = if cfg.cmf.local_host.trim().is_empty() {
"127.0.0.1"
} else {
&cfg.cmf.local_host
};
let base_port = cfg.cmf.servers.iter().map(|s| s.port).max().unwrap_or(8089);
let port = next_free_port(host, base_port).unwrap_or_else(|| base_port + 1);
cfg.cmf.servers.push(crate::config::CmfServer {
id: id.clone(),
model: j.output.clone(),
port,
threads: cfg.cmf.threads,
gpu: cfg.cmf.gpu,
temperature: None,
top_p: None,
max_tokens: None,
think_budget: None,
system_prompt: None,
o1: None,
o1_m: None,
o1_window: None,
o1_sink: None,
skip_mtp: false,
});
} else {
cfg.models.push(crate::config::ModelCfg {
id: id.clone(),
provider: "openai".into(),
base_url: cfg.cmf.cortiq_server_url.clone(),
model: base,
cost_tier: "local".into(),
price_in: 0.0,
price_out: 0.0,
kind: "chat".into(),
api_key_env: None,
caps: Vec::new(),
temperature: None,
top_p: None,
max_tokens: None,
think_budget: None,
system_prompt: None,
o1: None,
skip_mtp: false,
});
} }
cfg.models.push(crate::config::ModelCfg {
id: id.clone(),
provider: "openai".into(),
base_url: cfg.cmf.cortiq_server_url.clone(),
model: base,
cost_tier: "local".into(),
price_in: 0.0,
price_out: 0.0,
kind: "chat".into(),
api_key_env: None,
caps: Vec::new(),
});
state.reload(cfg)?; state.reload(cfg)?;
ok(json!({ "ok": true, "model_id": id })) ok(json!({ "ok": true, "model_id": id }))
} }
+87 -228
View File
@@ -1,68 +1,51 @@
//! Local CMF model server lifecycle. //! Local CMF model server lifecycle.
//! //!
//! Closes the loop with <https://github.com/infosave2007/cmf>: install the //! Closes the loop with <https://github.com/infosave2007/cmf>: install the
//! `cortiq` CLI from crates.io, keep it up to date, and run one `cortiq serve` //! `cortiq` CLI from crates.io, keep it up to date, and run `cortiq serve` as a
//! per configured local model as a managed child process. Each served model //! managed child process. The served model exposes an OpenAI-compatible API, so
//! exposes an OpenAI-compatible API, so the registry registers it as an ordinary //! the registry registers it as an ordinary provider and the gateway routes to
//! provider and the gateway routes to it like any other backend. //! it like any other backend.
use crate::config::{CmfCfg, CmfServer}; use crate::config::CmfCfg;
use std::process::Stdio; use std::process::Stdio;
use std::sync::{Arc, Mutex}; use std::sync::{Arc, Mutex};
use std::time::Duration; use std::time::Duration;
use tokio::process::{Child, Command}; use tokio::process::{Child, Command};
/// Observable state of one managed local model server. /// Observable state of the managed local CMF server (surfaced to the admin API).
#[derive(Clone, Debug, Default, serde::Serialize)]
pub struct CmfServerStatus {
/// Pool id it is registered under.
pub id: String,
/// Path of the `.cmf` being served.
pub model: String,
/// Port the server binds to.
pub port: u16,
/// OpenAI base URL the local server is registered under.
pub base_url: String,
/// The child `cortiq serve` process has been spawned.
pub running: bool,
/// `/healthz` on the local server answered 200.
pub healthy: bool,
/// Last fatal error for this server, if any.
pub last_error: Option<String>,
}
/// Observable state of the managed local CMF layer (surfaced to the admin API).
#[derive(Clone, Debug, Default, serde::Serialize)] #[derive(Clone, Debug, Default, serde::Serialize)]
pub struct CmfStatus { pub struct CmfStatus {
/// `manage_server` is on and at least one local model is configured. /// `manage_server` is on and a `local_model` is configured.
pub enabled: bool, pub enabled: bool,
/// Version reported by `cortiq --version` (None = not installed). /// Version reported by `cortiq --version` (None = not installed).
pub installed_version: Option<String>, pub installed_version: Option<String>,
/// Newest `cortiq-cli` on crates.io (populated when `auto_update` runs). /// Newest `cortiq-cli` on crates.io (populated when `auto_update` runs).
pub latest_version: Option<String>, pub latest_version: Option<String>,
/// Per-model server status. /// The child `cortiq serve` process has been spawned.
pub servers: Vec<CmfServerStatus>, pub running: bool,
/// `/healthz` on the local server answered 200.
pub healthy: bool,
/// OpenAI base URL the local server is registered under.
pub base_url: String,
/// Path of the `.cmf` being served.
pub model: String,
/// Last fatal error, if any.
pub last_error: Option<String>,
/// Recent lifecycle log lines (bounded). /// Recent lifecycle log lines (bounded).
pub log: Vec<String>, pub log: Vec<String>,
} }
/// A spawned server: its pool id and the child process handle. /// Holds the managed child process and its live status.
struct Managed {
id: String,
child: Child,
}
/// Holds the managed child processes and their live status.
pub struct CmfRuntime { pub struct CmfRuntime {
status: Mutex<CmfStatus>, status: Mutex<CmfStatus>,
children: Mutex<Vec<Managed>>, child: Mutex<Option<Child>>,
} }
impl CmfRuntime { impl CmfRuntime {
pub fn new() -> Arc<Self> { pub fn new() -> Arc<Self> {
Arc::new(Self { Arc::new(Self {
status: Mutex::new(CmfStatus::default()), status: Mutex::new(CmfStatus::default()),
children: Mutex::new(Vec::new()), child: Mutex::new(None),
}) })
} }
@@ -76,115 +59,32 @@ impl CmfRuntime {
let mut s = self.status.lock().unwrap(); let mut s = self.status.lock().unwrap();
s.log.push(msg); s.log.push(msg);
let n = s.log.len(); let n = s.log.len();
if n > 80 { if n > 50 {
s.log.drain(0..n - 80); s.log.drain(0..n - 50);
} }
} }
/// A global (install/update) failure — not tied to a single server.
fn fail(&self, msg: impl Into<String>) { fn fail(&self, msg: impl Into<String>) {
let msg = msg.into(); let msg = msg.into();
tracing::warn!(target: "cmf", "{msg}"); tracing::warn!(target: "cmf", "{msg}");
self.status.lock().unwrap().log.push(format!("✗ {msg}"));
}
/// Record a fatal error against a specific server (and the log).
fn fail_server(&self, id: &str, msg: impl Into<String>) {
let msg = msg.into();
tracing::warn!(target: "cmf", "[{id}] {msg}");
let mut s = self.status.lock().unwrap(); let mut s = self.status.lock().unwrap();
if let Some(sv) = s.servers.iter_mut().find(|x| x.id == id) { s.last_error = Some(msg.clone());
sv.last_error = Some(msg.clone()); s.log.push(format!("✗ {msg}"));
}
s.log.push(format!("✗ [{id}] {msg}"));
} }
/// Seed the per-server status slots from the effective server list. fn set_child(&self, child: Child) {
fn init_servers(&self, servers: &[CmfServer], host: &str) { *self.child.lock().unwrap() = Some(child);
let mut s = self.status.lock().unwrap(); self.status.lock().unwrap().running = true;
s.enabled = true;
s.servers = servers
.iter()
.map(|sv| CmfServerStatus {
id: sv.id.clone(),
model: sv.model.clone(),
port: sv.port,
base_url: format!("http://{host}:{}/v1", sv.port),
running: false,
healthy: false,
last_error: None,
})
.collect();
} }
fn set_running(&self, id: &str, running: bool) { /// Stop the managed server (used on shutdown / reconfigure).
if let Some(sv) = self
.status
.lock()
.unwrap()
.servers
.iter_mut()
.find(|x| x.id == id)
{
sv.running = running;
}
}
fn set_healthy(&self, id: &str, healthy: bool) {
if let Some(sv) = self
.status
.lock()
.unwrap()
.servers
.iter_mut()
.find(|x| x.id == id)
{
sv.healthy = healthy;
}
}
fn add_child(&self, id: String, child: Child) {
self.children.lock().unwrap().push(Managed {
id: id.clone(),
child,
});
self.set_running(&id, true);
}
fn is_running(&self, id: &str) -> bool {
self.status
.lock()
.unwrap()
.servers
.iter()
.any(|s| s.id == id && s.running)
}
/// Stop all managed servers (used on shutdown / reconfigure).
pub async fn stop(&self) { pub async fn stop(&self) {
let mut kids = std::mem::take(&mut *self.children.lock().unwrap()); if let Some(mut child) = self.child.lock().unwrap().take() {
for m in kids.iter_mut() { let _ = child.start_kill();
let _ = m.child.start_kill();
} }
let mut s = self.status.lock().unwrap(); let mut s = self.status.lock().unwrap();
for sv in s.servers.iter_mut() { s.running = false;
sv.running = false; s.healthy = false;
sv.healthy = false;
}
}
/// Stop and forget a single managed server (used when a model is deleted).
pub async fn stop_one(&self, id: &str) {
let removed = {
let mut kids = self.children.lock().unwrap();
kids.iter()
.position(|m| m.id == id)
.map(|pos| kids.remove(pos))
};
if let Some(mut m) = removed {
let _ = m.child.start_kill();
}
self.status.lock().unwrap().servers.retain(|sv| sv.id != id);
} }
} }
@@ -192,7 +92,7 @@ impl Default for CmfRuntime {
fn default() -> Self { fn default() -> Self {
Self { Self {
status: Mutex::new(CmfStatus::default()), status: Mutex::new(CmfStatus::default()),
children: Mutex::new(Vec::new()), child: Mutex::new(None),
} }
} }
} }
@@ -213,15 +113,14 @@ async fn latest_crates_version() -> Option<String> {
.await .await
.ok()?; .ok()?;
let v: serde_json::Value = serde_json::from_str(&body).ok()?; let v: serde_json::Value = serde_json::from_str(&body).ok()?;
v["crate"]["newest_version"].as_str().map(|s| s.to_string()) v["crate"]["newest_version"]
.as_str()
.map(|s| s.to_string())
} }
/// `<bin> --version` → the trailing version token (e.g. "cortiq 0.1.2" → "0.1.2"). /// `<bin> --version` → the trailing version token (e.g. "cortiq 0.1.2" → "0.1.2").
pub(crate) fn installed_version(bin: &str) -> Option<String> { pub(crate) fn installed_version(bin: &str) -> Option<String> {
let out = std::process::Command::new(bin) let out = std::process::Command::new(bin).arg("--version").output().ok()?;
.arg("--version")
.output()
.ok()?;
if !out.status.success() { if !out.status.success() {
return None; return None;
} }
@@ -231,14 +130,10 @@ pub(crate) fn installed_version(bin: &str) -> Option<String> {
fn parse_ver(s: &str) -> (u32, u32, u32) { fn parse_ver(s: &str) -> (u32, u32, u32) {
let mut it = s.split('.').map(|x| x.trim().parse::<u32>().unwrap_or(0)); let mut it = s.split('.').map(|x| x.trim().parse::<u32>().unwrap_or(0));
( (it.next().unwrap_or(0), it.next().unwrap_or(0), it.next().unwrap_or(0))
it.next().unwrap_or(0),
it.next().unwrap_or(0),
it.next().unwrap_or(0),
)
} }
pub(crate) fn version_lt(a: &str, b: &str) -> bool { fn version_lt(a: &str, b: &str) -> bool {
parse_ver(a) < parse_ver(b) parse_ver(a) < parse_ver(b)
} }
@@ -266,55 +161,32 @@ pub async fn install_now(rt: Arc<CmfRuntime>, bin: String) {
Ok(()) => { Ok(()) => {
let v = installed_version(&bin); let v = installed_version(&bin);
rt.status.lock().unwrap().installed_version = v.clone(); rt.status.lock().unwrap().installed_version = v.clone();
rt.log(format!( rt.log(format!("installed cortiq-cli {}", v.as_deref().unwrap_or("?")));
"installed cortiq-cli {}",
v.as_deref().unwrap_or("?")
));
} }
Err(e) => rt.fail(format!("cargo install cortiq-cli failed: {e}")), Err(e) => rt.fail(format!("cargo install cortiq-cli failed: {e}")),
} }
} }
/// Spawn `cortiq serve <model> --host <host> --port <port>` for one server; the /// Spawn `cortiq serve <model> --host <host> --port <port>`; the child is killed
/// child is killed when its handle is dropped (i.e. when the gateway shuts down). /// when its handle is dropped (i.e. when the gateway shuts down).
fn spawn_serve(server: &CmfServer, cfg: &CmfCfg) -> Result<Child, String> { fn spawn_serve(cfg: &CmfCfg) -> Result<Child, String> {
let mut cmd = Command::new(&cfg.cortiq_bin); let mut cmd = Command::new(&cfg.cortiq_bin);
cmd.arg("serve") cmd.arg("serve")
.arg(&server.model) .arg(&cfg.local_model)
.arg("--host") .arg("--host")
.arg(&cfg.local_host) .arg(&cfg.local_host)
.arg("--port") .arg("--port")
.arg(server.port.to_string()); .arg(cfg.local_port.to_string());
// Parallelise decode matvecs across cores (CMF_THREADS). 0/1 = leave the // Parallelise decode matvecs across cores (CMF_THREADS). 0 = leave the
// runtime's own default; N>1 gave ~2.8x on a 15B CPU model. // runtime's own default; N>1 gave ~2.8x on a 15B CPU model.
if server.threads > 1 { if cfg.threads > 1 {
cmd.env("CMF_THREADS", server.threads.to_string()); cmd.env("CMF_THREADS", cfg.threads.to_string());
} }
if server.gpu { // Offload large matvecs to the GPU (Metal on macOS). Memory-bandwidth-bound
// on unified memory, so mostly a prefill win; opt-in.
if cfg.gpu {
cmd.env("CMF_GPU", "1"); cmd.env("CMF_GPU", "1");
} }
// O(1) Nyström attention: layers spec + sub-parameters.
if let Some(o1) = &server.o1 {
let s = o1.trim();
if !s.is_empty() {
// Pass even "off" so it explicitly overrides a file converter hint.
cmd.arg("--o1").arg(s);
}
}
if let Some(m) = server.o1_m {
cmd.arg("--o1-m").arg(m.to_string());
}
if let Some(w) = server.o1_window {
cmd.arg("--o1-window").arg(w.to_string());
}
if let Some(sink) = server.o1_sink {
cmd.arg("--o1-sink").arg(sink.to_string());
}
// MTP (Multi-Token Prediction) speculative decoding: the engine reads
// CMF_MTP env; "0" disables. There is no CLI flag for this.
if server.skip_mtp {
cmd.env("CMF_MTP", "0");
}
cmd.stdout(Stdio::null()) cmd.stdout(Stdio::null())
.stderr(Stdio::null()) .stderr(Stdio::null())
.kill_on_drop(true) .kill_on_drop(true)
@@ -342,17 +214,24 @@ async fn wait_healthy(host: &str, port: u16, tries: u32) -> bool {
false false
} }
/// Orchestrate the whole lifecycle: ensure installed → maybe update → spawn each /// Orchestrate the whole lifecycle: ensure installed → maybe update → spawn →
/// configured local model → wait healthy. Safe to call in a background task; it /// wait healthy. Safe to call in a background task; it never panics.
/// never panics.
pub async fn manage(rt: Arc<CmfRuntime>, cfg: CmfCfg) { pub async fn manage(rt: Arc<CmfRuntime>, cfg: CmfCfg) {
let servers = cfg.effective_servers(); if !cfg.manage_server || cfg.local_model.trim().is_empty() {
if servers.is_empty() { return;
}
{
let mut s = rt.status.lock().unwrap();
s.enabled = true;
s.base_url = format!("http://{}:{}/v1", cfg.local_host, cfg.local_port);
s.model = cfg.local_model.clone();
}
if !std::path::Path::new(&cfg.local_model).exists() {
rt.fail(format!("local_model not found: {}", cfg.local_model));
return; return;
} }
rt.init_servers(&servers, &cfg.local_host);
// 1. Ensure the `cortiq` binary is available (once for all servers). // 1. Ensure the `cortiq` binary is available.
let mut ver = installed_version(&cfg.cortiq_bin); let mut ver = installed_version(&cfg.cortiq_bin);
if ver.is_none() { if ver.is_none() {
if cfg.auto_install { if cfg.auto_install {
@@ -366,9 +245,7 @@ pub async fn manage(rt: Arc<CmfRuntime>, cfg: CmfCfg) {
rt.log("installed cortiq-cli"); rt.log("installed cortiq-cli");
} }
} else { } else {
rt.fail( rt.fail("cortiq binary not found (set [cmf].auto_install = true, or install cortiq-cli)");
"cortiq binary not found (set [cmf].auto_install = true, or install cortiq-cli)",
);
return; return;
} }
} }
@@ -378,15 +255,8 @@ pub async fn manage(rt: Arc<CmfRuntime>, cfg: CmfCfg) {
if cfg.auto_update { if cfg.auto_update {
if let Some(latest) = latest_crates_version().await { if let Some(latest) = latest_crates_version().await {
rt.status.lock().unwrap().latest_version = Some(latest.clone()); rt.status.lock().unwrap().latest_version = Some(latest.clone());
if ver if ver.as_deref().map(|c| version_lt(c, &latest)).unwrap_or(false) {
.as_deref() rt.log(format!("updating cortiq-cli {} → {latest}…", ver.as_deref().unwrap_or("?")));
.map(|c| version_lt(c, &latest))
.unwrap_or(false)
{
rt.log(format!(
"updating cortiq-cli {} → {latest}…",
ver.as_deref().unwrap_or("?")
));
if let Err(e) = cargo_install(true).await { if let Err(e) = cargo_install(true).await {
rt.log(format!("update failed (keeping current): {e}")); rt.log(format!("update failed (keeping current): {e}"));
} else { } else {
@@ -398,38 +268,27 @@ pub async fn manage(rt: Arc<CmfRuntime>, cfg: CmfCfg) {
} }
} }
// 3. Spawn each server (fast). // 3. Spawn the local server.
for server in &servers { rt.log(format!(
if !std::path::Path::new(&server.model).exists() { "starting: cortiq serve {} --host {} --port {}",
rt.fail_server( cfg.local_model, cfg.local_host, cfg.local_port
&server.id, ));
format!("model file not found: {}", server.model), match spawn_serve(&cfg) {
); Ok(child) => rt.set_child(child),
continue; Err(e) => {
} rt.fail(format!("failed to spawn cortiq serve: {e}"));
rt.log(format!( return;
"starting: cortiq serve {} --host {} --port {}",
server.model, cfg.local_host, server.port
));
match spawn_serve(server, &cfg) {
Ok(child) => rt.add_child(server.id.clone(), child),
Err(e) => rt.fail_server(&server.id, format!("failed to spawn: {e}")),
} }
} }
// 4. Wait until each spawned server is serving (model load can be slow). // 4. Wait until it is serving (model load can take a while for big models).
for server in &servers { if wait_healthy(&cfg.local_host, cfg.local_port, 240).await {
if !rt.is_running(&server.id) { rt.status.lock().unwrap().healthy = true;
continue; // failed to spawn / file missing rt.log(format!(
} "local CMF server ready — registered as model '{}' at http://{}:{}/v1",
if wait_healthy(&cfg.local_host, server.port, 240).await { cfg.model_id, cfg.local_host, cfg.local_port
rt.set_healthy(&server.id, true); ));
rt.log(format!( } else {
"local CMF server ready — '{}' at http://{}:{}/v1", rt.fail("local CMF server did not become healthy in time");
server.id, cfg.local_host, server.port
));
} else {
rt.fail_server(&server.id, "did not become healthy in time");
}
} }
} }
+7 -135
View File
@@ -10,18 +10,10 @@ use std::path::PathBuf;
/// `~/.cortiq-gateway` — so defaults work regardless of the process CWD /// `~/.cortiq-gateway` — so defaults work regardless of the process CWD
/// (important once the gateway is installed rather than run from its repo). /// (important once the gateway is installed rather than run from its repo).
pub fn data_dir() -> PathBuf { pub fn data_dir() -> PathBuf {
for (var, sub) in [ for (var, sub) in [("CORTIQ_GATEWAY_HOME", ""), ("XDG_DATA_HOME", "cortiq-gateway"), ("HOME", ".cortiq-gateway")] {
("CORTIQ_GATEWAY_HOME", ""),
("XDG_DATA_HOME", "cortiq-gateway"),
("HOME", ".cortiq-gateway"),
] {
if let Ok(v) = std::env::var(var) { if let Ok(v) = std::env::var(var) {
if !v.is_empty() { if !v.is_empty() {
return if sub.is_empty() { return if sub.is_empty() { PathBuf::from(v) } else { PathBuf::from(v).join(sub) };
PathBuf::from(v)
} else {
PathBuf::from(v).join(sub)
};
} }
} }
} }
@@ -67,54 +59,6 @@ pub struct Config {
pub cmf: CmfCfg, pub cmf: CmfCfg,
} }
/// One managed local model: the gateway runs a dedicated `cortiq serve` for it
/// on its own port and registers it in the pool under `id`.
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
pub struct CmfServer {
/// Pool id it is registered under (e.g. "cmf-qwen3-5-2b").
pub id: String,
/// Path to the `.cmf` to serve.
pub model: String,
/// TCP port the dedicated `cortiq serve` binds to (must be unique).
pub port: u16,
/// Worker threads (CMF_THREADS). 0 = runtime default.
#[serde(default = "default_cmf_threads")]
pub threads: usize,
/// Offload matvecs to the GPU (CMF_GPU=1).
#[serde(default)]
pub gpu: bool,
/// Default generation temperature (0.0 .. 2.0).
#[serde(default, skip_serializing_if = "Option::is_none")]
pub temperature: Option<f32>,
/// Default nucleus sampling threshold (0.0 .. 1.0).
#[serde(default, skip_serializing_if = "Option::is_none")]
pub top_p: Option<f32>,
/// Default maximum completion tokens.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub max_tokens: Option<u32>,
/// Reasoning / thinking token budget.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub think_budget: Option<u32>,
/// Default system prompt prepended if request lacks one.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub system_prompt: Option<String>,
/// O(1) Nyström attention layer spec (all | deepN | i,j,k | off).
#[serde(default, skip_serializing_if = "Option::is_none")]
pub o1: Option<String>,
/// O(1) landmark budget (default 32; ≥4).
#[serde(default, skip_serializing_if = "Option::is_none")]
pub o1_m: Option<u32>,
/// O(1) exact-window width (default 128).
#[serde(default, skip_serializing_if = "Option::is_none")]
pub o1_window: Option<u32>,
/// O(1) permanent exact sink keys (default 4, StreamingLLM discipline).
#[serde(default, skip_serializing_if = "Option::is_none")]
pub o1_sink: Option<u32>,
/// Disable Multi-Token Prediction speculative decoding (CMF_MTP=0).
#[serde(default)]
pub skip_mtp: bool,
}
/// CMF-format model factory: import a HuggingFace model, convert it to a /// CMF-format model factory: import a HuggingFace model, convert it to a
/// local `.cmf` (quantized), and serve it via cortiq-server. Points the /// local `.cmf` (quantized), and serve it via cortiq-server. Points the
/// gateway at the Python converter + runtime on the host. /// gateway at the Python converter + runtime on the host.
@@ -177,45 +121,6 @@ pub struct CmfCfg {
/// (helps prefill of long prompts). Off by default. /// (helps prefill of long prompts). Off by default.
#[serde(default)] #[serde(default)]
pub gpu: bool, pub gpu: bool,
/// Managed local models. When non-empty (and `manage_server` is on), the
/// gateway spawns and supervises one `cortiq serve` per entry — each on its
/// own port — and registers each in the pool. Empty = fall back to the single
/// legacy `local_model` / `local_port` / `model_id` below (back-compat).
#[serde(default)]
pub servers: Vec<CmfServer>,
}
impl CmfCfg {
/// The managed local models actually in effect: the `servers` list when set,
/// otherwise the single legacy `local_model` (as a one-element list). Empty
/// when `manage_server` is off or nothing is configured.
pub fn effective_servers(&self) -> Vec<CmfServer> {
if !self.manage_server {
return Vec::new();
}
if !self.servers.is_empty() {
return self.servers.clone();
}
if !self.local_model.trim().is_empty() {
return vec![CmfServer {
id: self.model_id.clone(),
model: self.local_model.clone(),
port: self.local_port,
threads: self.threads,
gpu: self.gpu,
temperature: None,
top_p: None,
max_tokens: None,
think_budget: None,
system_prompt: None,
o1: None,
o1_m: None,
o1_window: None,
o1_sink: None,
skip_mtp: false,
}];
}
Vec::new()
}
} }
impl Default for CmfCfg { impl Default for CmfCfg {
fn default() -> Self { fn default() -> Self {
@@ -236,7 +141,6 @@ impl Default for CmfCfg {
local_only: false, local_only: false,
threads: default_cmf_threads(), threads: default_cmf_threads(),
gpu: false, gpu: false,
servers: Vec::new(),
} }
} }
} }
@@ -346,27 +250,6 @@ pub struct ModelCfg {
pub api_key_env: Option<String>, pub api_key_env: Option<String>,
#[serde(default, skip_serializing_if = "Vec::is_empty")] #[serde(default, skip_serializing_if = "Vec::is_empty")]
pub caps: Vec<String>, pub caps: Vec<String>,
/// Default generation temperature (0.0 .. 2.0).
#[serde(default, skip_serializing_if = "Option::is_none")]
pub temperature: Option<f32>,
/// Default nucleus sampling threshold (0.0 .. 1.0).
#[serde(default, skip_serializing_if = "Option::is_none")]
pub top_p: Option<f32>,
/// Default maximum completion tokens.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub max_tokens: Option<u32>,
/// Reasoning / thinking token budget.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub think_budget: Option<u32>,
/// Default system prompt prepended if request lacks one.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub system_prompt: Option<String>,
/// O(1) constant-memory streaming attention hint (all, deep, custom, off).
#[serde(default, skip_serializing_if = "Option::is_none")]
pub o1: Option<String>,
/// Skip Multi-Token Prediction (MTP) heads.
#[serde(default)]
pub skip_mtp: bool,
} }
fn default_kind() -> String { fn default_kind() -> String {
"chat".into() "chat".into()
@@ -522,12 +405,10 @@ pub struct AdminCfg {
/// Whether the admin panel and admin API (`/admin`, `/admin/api/*`) are enabled. /// Whether the admin panel and admin API (`/admin`, `/admin/api/*`) are enabled.
#[serde(default = "default_admin_enabled")] #[serde(default = "default_admin_enabled")]
pub enabled: bool, pub enabled: bool,
/// Name of the env variable holding the admin token. /// Name of the env variable holding the admin token. If unset (or empty),
/// a token is generated at startup and printed to the log.
#[serde(default, skip_serializing_if = "Option::is_none")] #[serde(default, skip_serializing_if = "Option::is_none")]
pub token_env: Option<String>, pub token_env: Option<String>,
/// Persistent admin token configured directly in toml.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub token: Option<String>,
/// Optional separate bind address for the admin surface (e.g. `127.0.0.1:9001`). /// Optional separate bind address for the admin surface (e.g. `127.0.0.1:9001`).
/// If unset, the admin interface lives on the main `listen` address under `/admin`. /// If unset, the admin interface lives on the main `listen` address under `/admin`.
#[serde(default, skip_serializing_if = "Option::is_none")] #[serde(default, skip_serializing_if = "Option::is_none")]
@@ -538,7 +419,6 @@ impl Default for AdminCfg {
Self { Self {
enabled: default_admin_enabled(), enabled: default_admin_enabled(),
token_env: None, token_env: None,
token: None,
listen: None, listen: None,
} }
} }
@@ -717,8 +597,7 @@ impl Config {
// A managed local CMF model counts as an available model even though it // A managed local CMF model counts as an available model even though it
// is registered dynamically rather than via a [[models]] entry — this // is registered dynamically rather than via a [[models]] entry — this
// is what lets the gateway run on local models alone (no cloud, no router). // is what lets the gateway run on local models alone (no cloud, no router).
let servers = self.cmf.effective_servers(); let has_local = self.cmf.manage_server && !self.cmf.local_model.trim().is_empty();
let has_local = !servers.is_empty();
if self.models.is_empty() && !has_local { if self.models.is_empty() && !has_local {
// Fresh install: start with an EMPTY model pool so the admin console // Fresh install: start with an EMPTY model pool so the admin console
// comes up — the user adds models in /admin → Models (or points [cmf] // comes up — the user adds models in /admin → Models (or points [cmf]
@@ -737,15 +616,8 @@ impl Config {
// every target in [routing] must exist in the model pool // every target in [routing] must exist in the model pool
let mut ids: std::collections::HashSet<&str> = let mut ids: std::collections::HashSet<&str> =
self.models.iter().map(|m| m.id.as_str()).collect(); self.models.iter().map(|m| m.id.as_str()).collect();
// managed local servers are known ids too; also enforce unique ports/ids if has_local {
let mut ports = std::collections::HashSet::new(); ids.insert(self.cmf.model_id.as_str());
for s in &servers {
if !ids.insert(s.id.as_str()) {
anyhow::bail!("config: duplicate managed model id '{}'", s.id);
}
if !ports.insert(s.port) {
anyhow::bail!("config: two managed models share port {}", s.port);
}
} }
if !self.routing.default.is_empty() && !ids.contains(self.routing.default.as_str()) { if !self.routing.default.is_empty() && !ids.contains(self.routing.default.as_str()) {
anyhow::bail!( anyhow::bail!(
+32 -383
View File
@@ -31,19 +31,13 @@ fn cleanup_output(path: &str) {
} }
fn now() -> u64 { fn now() -> u64 {
SystemTime::now() SystemTime::now().duration_since(UNIX_EPOCH).unwrap_or_default().as_secs()
.duration_since(UNIX_EPOCH)
.unwrap_or_default()
.as_secs()
} }
/// Job id from a hash of (repo, nanos) — unique without a uuid dependency. /// Job id from a hash of (repo, nanos) — unique without a uuid dependency.
fn gen_id(repo: &str) -> String { fn gen_id(repo: &str) -> String {
use std::hash::{Hash, Hasher}; use std::hash::{Hash, Hasher};
let nanos = SystemTime::now() let nanos = SystemTime::now().duration_since(UNIX_EPOCH).unwrap_or_default().as_nanos();
.duration_since(UNIX_EPOCH)
.unwrap_or_default()
.as_nanos();
let mut h = std::collections::hash_map::DefaultHasher::new(); let mut h = std::collections::hash_map::DefaultHasher::new();
repo.hash(&mut h); repo.hash(&mut h);
nanos.hash(&mut h); nanos.hash(&mut h);
@@ -71,20 +65,6 @@ pub struct ImportParams {
pub shard_max_gb: Option<f32>, pub shard_max_gb: Option<f32>,
#[serde(default)] #[serde(default)]
pub skip_mtp: bool, pub skip_mtp: bool,
/// O(1) Nyström attention hint (native converter, cortiq ≥ 0.2.0):
/// `all` | `deepN` | `i,j,k`. Weights pass through unchanged — the runtime
/// reads the hint at load. Empty/None/"off" = exact attention.
#[serde(default)]
pub o1: Option<String>,
/// Landmark budget for the o1 hint (validated default 32).
#[serde(default)]
pub o1_m: Option<usize>,
/// Exact-window width for the o1 hint (validated default 128).
#[serde(default)]
pub o1_window: Option<usize>,
/// Permanent exact sink keys for the o1 hint (validated default 4).
#[serde(default)]
pub o1_sink: Option<usize>,
} }
fn default_quant() -> String { fn default_quant() -> String {
"Q8_2F".into() "Q8_2F".into()
@@ -102,8 +82,8 @@ pub struct Job {
pub started: u64, pub started: u64,
pub finished: Option<u64>, pub finished: Option<u64>,
pub size_bytes: Option<u64>, pub size_bytes: Option<u64>,
pub progress: Option<f32>, // 0..1, parsed from converter @PROGRESS pub progress: Option<f32>, // 0..1, parsed from converter @PROGRESS
pub phase: Option<String>, // current phase label pub phase: Option<String>, // current phase label
} }
#[derive(Default)] #[derive(Default)]
@@ -160,14 +140,6 @@ impl JobStore {
} }
} }
} }
fn update_quant(&self, id: &str, quant: String) {
if let Some(j) = self.jobs.lock().unwrap().get_mut(id) {
j.quant = quant.clone();
if !j.log.is_empty() && j.log[0].starts_with("→ downloading ready .cmf model") {
j.log[0] = format!("→ downloading ready .cmf model {} ({})", j.repo, quant);
}
}
}
fn set_state(&self, id: &str, state: &str) { fn set_state(&self, id: &str, state: &str) {
if let Some(j) = self.jobs.lock().unwrap().get_mut(id) { if let Some(j) = self.jobs.lock().unwrap().get_mut(id) {
j.state = state.into(); j.state = state.into();
@@ -206,24 +178,16 @@ impl JobStore {
} }
pub fn list(&self) -> Vec<Job> { pub fn list(&self) -> Vec<Job> {
let mut v: Vec<Job> = self.jobs.lock().unwrap().values().cloned().collect(); let mut v: Vec<Job> = self.jobs.lock().unwrap().values().cloned().collect();
v.sort_by_key(|x| std::cmp::Reverse(x.started)); v.sort_by(|a, b| b.started.cmp(&a.started));
v.truncate(20); v.truncate(20);
v v
} }
} }
/// Proxy HuggingFace model search (avoids browser CORS + adds our token). /// Proxy HuggingFace model search (avoids browser CORS + adds our token).
pub async fn hf_search( pub async fn hf_search(query: &str, limit: usize, token: Option<&str>) -> Result<serde_json::Value, String> {
query: &str,
limit: usize,
token: Option<&str>,
) -> Result<serde_json::Value, String> {
let q: String = url_escape(query); let q: String = url_escape(query);
let sort = if query.trim().is_empty() { let sort = if query.trim().is_empty() { "trendingScore" } else { "downloads" };
"trendingScore"
} else {
"downloads"
};
let url = format!( let url = format!(
"https://huggingface.co/api/models?search={q}&sort={sort}&direction=-1&limit={limit}&full=false" "https://huggingface.co/api/models?search={q}&sort={sort}&direction=-1&limit={limit}&full=false"
); );
@@ -254,13 +218,7 @@ fn url_escape(s: &str) -> String {
fn sanitize(s: &str) -> String { fn sanitize(s: &str) -> String {
s.chars() s.chars()
.map(|c| { .map(|c| if c.is_ascii_alphanumeric() || c == '-' || c == '_' { c } else { '-' })
if c.is_ascii_alphanumeric() || c == '-' || c == '_' {
c
} else {
'-'
}
})
.collect() .collect()
} }
@@ -268,166 +226,14 @@ fn sanitize(s: &str) -> String {
/// Map the UI quant label to the native `cortiq` quant flag. /// Map the UI quant label to the native `cortiq` quant flag.
fn map_quant(q: &str) -> &'static str { fn map_quant(q: &str) -> &'static str {
match q.to_ascii_uppercase().as_str() { match q.to_ascii_uppercase().as_str() {
s if s.starts_with("Q4T") || s == "Q4_TILED" => "q4t",
s if s.starts_with("Q4") => "q4", s if s.starts_with("Q4") => "q4",
"Q8_2F" | "Q82F" => "q8_2f", "Q8_2F" => "q8_2f",
"F16" | "FP16" => "f16", "F16" | "FP16" => "f16",
"VBIT" => "vbit", "VBIT" => "vbit",
"Q1" => "q1",
"Q1P" | "Q1_PTQ" => "q1p",
"Q1S" | "Q1_MASK" => "q1s",
"Q1T" | "Q1_TERNARY" => "q1t",
_ => "q8", _ => "q8",
} }
} }
fn is_cmf_repo(repo: &str) -> bool {
let lower = repo.to_ascii_lowercase();
lower.contains("cmf") || lower.ends_with(".cmf")
}
async fn download_cmf_repo(
repo: String,
output_abs: String,
hf_token: Option<String>,
store: Arc<JobStore>,
id: String,
cancel: Arc<Notify>,
) {
store.set_progress(&id, 0.0, "listing files".into());
let tree_url = format!("https://huggingface.co/api/models/{repo}/tree/main?recursive=true");
let client = reqwest::Client::new();
let mut req = client.get(&tree_url).header("User-Agent", "cortiq-gateway");
if let Some(t) = &hf_token {
req = req.bearer_auth(t);
}
let mut cmf_rel_path = None;
if let Ok(resp) = req.send().await {
if resp.status().is_success() {
if let Ok(json) = resp.json::<serde_json::Value>().await {
if let Some(arr) = json.as_array() {
for item in arr {
if let Some(path) = item.get("path").and_then(|p| p.as_str()) {
if path.to_ascii_lowercase().ends_with(".cmf") {
cmf_rel_path = Some(path.to_string());
break;
}
}
}
}
}
}
}
let cmf_path = match cmf_rel_path {
Some(p) => p,
None => {
let repo_name = repo.rsplit('/').next().unwrap_or(&repo);
format!("{repo_name}.cmf")
}
};
let path_lower = cmf_path.to_ascii_lowercase();
let detected_quant =
if path_lower.contains("-q1t.") || repo.to_ascii_lowercase().contains("2bit") {
"Q1T".to_string()
} else if path_lower.contains("-q1.") || repo.to_ascii_lowercase().contains("cmf") {
"Q1".to_string()
} else if path_lower.contains("-q4.") {
"Q4".to_string()
} else if path_lower.contains("-q8.") {
"Q8".to_string()
} else {
"READY_CMF".to_string()
};
store.update_quant(&id, detected_quant);
store.push_line(&id, format!("→ downloading {cmf_path} from HuggingFace..."));
store.set_progress(&id, 0.05, "downloading".into());
let download_url = format!("https://huggingface.co/{repo}/resolve/main/{cmf_path}");
let mut cmd = tokio::process::Command::new("curl");
cmd.arg("-f").arg("-L").arg("-#").arg("-o").arg(&output_abs);
if let Some(t) = &hf_token {
cmd.arg("-H").arg(format!("Authorization: Bearer {t}"));
}
cmd.arg(&download_url);
cmd.stderr(std::process::Stdio::piped());
cmd.stdout(std::process::Stdio::null());
let mut child = match cmd.spawn() {
Ok(c) => c,
Err(e) => {
store.push_line(&id, format!("✗ failed to start curl download: {e}"));
store.finish(&id, false);
store.drop_cancel(&id);
return;
}
};
let stderr = child.stderr.take();
let mut t_err = None;
if let Some(err_stream) = stderr {
let store_clone = store.clone();
let id_clone = id.clone();
t_err = Some(tokio::spawn(async move {
use tokio::io::AsyncReadExt;
let mut reader = err_stream;
let mut buf = [0u8; 256];
while let Ok(n) = reader.read(&mut buf).await {
if n == 0 {
break;
}
let text = String::from_utf8_lossy(&buf[..n]);
for word in text.split_whitespace() {
if let Some(num_str) = word.strip_suffix('%') {
if let Ok(pct) = num_str.parse::<f32>() {
let frac = (pct / 100.0).clamp(0.0, 1.0);
store_clone.set_progress(&id_clone, frac, "downloading".into());
}
}
}
}
}));
}
tokio::select! {
status_res = child.wait() => {
if let Some(t) = t_err { t.abort(); }
match status_res {
Ok(status) if status.success() => {
store.push_line(&id, "✓ done".into());
store.finish(&id, true);
}
Ok(status) => {
cleanup_output(&output_abs);
store.push_line(&id, format!("✗ download failed: exit code {:?}", status.code()));
store.finish(&id, false);
}
Err(e) => {
cleanup_output(&output_abs);
store.push_line(&id, format!("✗ download process error: {e}"));
store.finish(&id, false);
}
}
}
_ = cancel.notified() => {
let _ = child.start_kill();
let _ = child.wait().await;
if let Some(t) = t_err { t.abort(); }
cleanup_output(&output_abs);
store.push_line(&id, "✗ cancelled by user — partial output removed".into());
store.set_state(&id, "cancelled");
}
}
store.drop_cancel(&id);
}
/// into the job log. Fails fast if the converter script is missing. /// into the job log. Fails fast if the converter script is missing.
pub fn start_import(store: Arc<JobStore>, cfg: &CmfCfg, p: ImportParams) -> Result<String, String> { pub fn start_import(store: Arc<JobStore>, cfg: &CmfCfg, p: ImportParams) -> Result<String, String> {
if p.repo.trim().is_empty() { if p.repo.trim().is_empty() {
@@ -438,38 +244,7 @@ pub fn start_import(store: Arc<JobStore>, cfg: &CmfCfg, p: ImportParams) -> Resu
// doesn't support yet (linear-attention folding, v-bit shaping). // doesn't support yet (linear-attention folding, v-bit shaping).
let use_python = p.linear_core.is_some() || p.vbit_shape.is_some() || p.mean_bits.is_some(); let use_python = p.linear_core.is_some() || p.vbit_shape.is_some() || p.mean_bits.is_some();
if use_python && !std::path::Path::new(&cfg.converter).exists() { if use_python && !std::path::Path::new(&cfg.converter).exists() {
return Err(format!( return Err(format!("advanced options need the Python converter, not found: {}", cfg.converter));
"advanced options need the Python converter, not found: {}",
cfg.converter
));
}
// O(1) attention hint — native-converter feature (cortiq ≥ 0.2.0). Fail fast
// with a clear message instead of a mid-job "unexpected argument" error.
let o1 =
p.o1.as_deref()
.map(str::trim)
.filter(|s| !s.is_empty() && *s != "off")
.map(str::to_string);
if o1.is_some() {
if use_python {
return Err(
"O(1) attention is supported by the native converter only — remove the \
linear-core / v-bit shape / mean-bits options to use it"
.into(),
);
}
if p.repo.to_ascii_lowercase().contains("gguf") {
return Err("O(1) attention is not supported for GGUF imports yet".into());
}
let ver = crate::cmf_runtime::installed_version(&cfg.cortiq_bin);
if let Some(v) = &ver {
if crate::cmf_runtime::version_lt(v, "0.2.0") {
return Err(format!(
"O(1) attention needs cortiq ≥ 0.2.0 (installed: {v}) — update the runtime \
in Settings → Local models"
));
}
}
} }
let _ = std::fs::create_dir_all(&cfg.models_dir); let _ = std::fs::create_dir_all(&cfg.models_dir);
let base = if p.name.trim().is_empty() { let base = if p.name.trim().is_empty() {
@@ -486,41 +261,13 @@ pub fn start_import(store: Arc<JobStore>, cfg: &CmfCfg, p: ImportParams) -> Resu
.unwrap_or_else(|_| output.clone()); .unwrap_or_else(|_| output.clone());
let id = gen_id(&p.repo); let id = gen_id(&p.repo);
let is_cmf = is_cmf_repo(&p.repo);
let quant_display = if is_cmf {
let r = p.repo.to_lowercase();
if r.contains("2bit") || r.contains("q1t") {
"Q1T".to_string()
} else if r.contains("1.7bcmf")
|| r.contains("27bcmf")
|| r.contains("cmf")
|| p.quant.is_empty()
|| p.quant == "auto"
|| p.quant == "Q8_2F"
{
"Q1".to_string()
} else {
p.quant.clone()
}
} else if p.quant.is_empty() || p.quant == "auto" {
"q8".to_string()
} else {
p.quant.clone()
};
store.insert(Job { store.insert(Job {
id: id.clone(), id: id.clone(),
repo: p.repo.clone(), repo: p.repo.clone(),
quant: quant_display.clone(), quant: p.quant.clone(),
output: output_abs.clone(), output: output_abs.clone(),
state: "running".into(), state: "running".into(),
log: if is_cmf { log: vec![format!("→ converting {} to {} ({})", p.repo, base, p.quant)],
vec![format!(
"→ downloading ready .cmf model {} ({})",
p.repo, quant_display
)]
} else {
vec![format!("→ converting {} to {} ({})", p.repo, base, p.quant)]
},
started: now(), started: now(),
finished: None, finished: None,
size_bytes: None, size_bytes: None,
@@ -536,56 +283,23 @@ pub fn start_import(store: Arc<JobStore>, cfg: &CmfCfg, p: ImportParams) -> Resu
std::env::var(&cfg.hf_token_env).ok() std::env::var(&cfg.hf_token_env).ok()
}; };
if is_cmf && !use_python {
let ret_id = id.clone();
tokio::spawn(async move {
download_cmf_repo(p.repo, output_abs, hf_token, store, id, cancel).await;
});
return Ok(ret_id);
}
// Program + args: `cortiq convert` by default, or the Python converter for // Program + args: `cortiq convert` by default, or the Python converter for
// advanced options. Both stream `@PROGRESS <frac>` markers into the log. // advanced options. Both stream `@PROGRESS <frac>` markers into the log.
let (program, args, workdir): (String, Vec<String>, std::path::PathBuf) = if use_python { let (program, args, workdir): (String, Vec<String>, std::path::PathBuf) = if use_python {
let conv = std::path::Path::new(&cfg.converter); let conv = std::path::Path::new(&cfg.converter);
let mut a = vec![ let mut a = vec![
std::fs::canonicalize(conv) std::fs::canonicalize(conv).map(|p| p.to_string_lossy().to_string()).unwrap_or_else(|_| cfg.converter.clone()),
.map(|p| p.to_string_lossy().to_string()) "--model".into(), p.repo.clone(),
.unwrap_or_else(|_| cfg.converter.clone()), "--quant".into(), p.quant.clone(),
"--model".into(), "--output".into(), output_abs.clone(),
p.repo.clone(),
"--quant".into(),
p.quant.clone(),
"--output".into(),
output_abs.clone(),
]; ];
if let Some(lc) = &p.linear_core { if let Some(lc) = &p.linear_core { a.push("--linear-core".into()); a.push(lc.clone()); }
a.push("--linear-core".into()); if let Some(n) = p.nphase { a.push("--nphase".into()); a.push(n.to_string()); }
a.push(lc.clone()); if let Some(vs) = &p.vbit_shape { a.push("--vbit-shape".into()); a.push(vs.clone()); }
} if let Some(mb) = p.mean_bits { a.push("--mean-bits".into()); a.push(mb.to_string()); }
if let Some(n) = p.nphase { if let Some(g) = p.shard_max_gb { a.push("--shard-max-gb".into()); a.push(g.to_string()); }
a.push("--nphase".into()); if p.skip_mtp { a.push("--skip-mtp".into()); }
a.push(n.to_string()); let wd = conv.parent().map(|d| d.to_path_buf()).unwrap_or_else(|| std::path::PathBuf::from("."));
}
if let Some(vs) = &p.vbit_shape {
a.push("--vbit-shape".into());
a.push(vs.clone());
}
if let Some(mb) = p.mean_bits {
a.push("--mean-bits".into());
a.push(mb.to_string());
}
if let Some(g) = p.shard_max_gb {
a.push("--shard-max-gb".into());
a.push(g.to_string());
}
if p.skip_mtp {
a.push("--skip-mtp".into());
}
let wd = conv
.parent()
.map(|d| d.to_path_buf())
.unwrap_or_else(|| std::path::PathBuf::from("."));
(cfg.python_bin.clone(), a, wd) (cfg.python_bin.clone(), a, wd)
} else if p.repo.to_ascii_lowercase().contains("gguf") { } else if p.repo.to_ascii_lowercase().contains("gguf") {
// GGUF repo → native `cortiq import-gguf` (downloads + dequantizes any // GGUF repo → native `cortiq import-gguf` (downloads + dequantizes any
@@ -593,57 +307,21 @@ pub fn start_import(store: Arc<JobStore>, cfg: &CmfCfg, p: ImportParams) -> Resu
let mut a = vec![ let mut a = vec![
"import-gguf".into(), "import-gguf".into(),
p.repo.clone(), p.repo.clone(),
"--quant".into(), "--quant".into(), map_quant(&p.quant).into(),
map_quant(&p.quant).into(), "--output".into(), output_abs.clone(),
"--output".into(),
output_abs.clone(),
]; ];
if let Some(t) = &hf_token { if let Some(t) = &hf_token { a.push("--hf-token".into()); a.push(t.clone()); }
a.push("--hf-token".into()); (cfg.cortiq_bin.clone(), a, std::env::current_dir().unwrap_or_else(|_| std::path::PathBuf::from(".")))
a.push(t.clone());
}
(
cfg.cortiq_bin.clone(),
a,
std::env::current_dir().unwrap_or_else(|_| std::path::PathBuf::from(".")),
)
} else { } else {
// safetensors → native `cortiq convert` (dense / MoE / GatedDeltaNet). // safetensors → native `cortiq convert` (dense / MoE / GatedDeltaNet).
let mut a = vec![ let mut a = vec![
"convert".into(), "convert".into(),
"--model".into(), "--model".into(), p.repo.clone(),
p.repo.clone(), "--quant".into(), map_quant(&p.quant).into(),
"--quant".into(), "--output".into(), output_abs.clone(),
map_quant(&p.quant).into(),
"--output".into(),
output_abs.clone(),
]; ];
if let Some(t) = &hf_token { if let Some(t) = &hf_token { a.push("--hf-token".into()); a.push(t.clone()); }
a.push("--hf-token".into()); (cfg.cortiq_bin.clone(), a, std::env::current_dir().unwrap_or_else(|_| std::path::PathBuf::from(".")))
a.push(t.clone());
}
// O(1) attention hint (weights unchanged; the runtime reads it at load)
if let Some(spec) = &o1 {
a.push("--o1".into());
a.push(spec.clone());
if let Some(m) = p.o1_m {
a.push("--o1-m".into());
a.push(m.to_string());
}
if let Some(w) = p.o1_window {
a.push("--o1-window".into());
a.push(w.to_string());
}
if let Some(s) = p.o1_sink {
a.push("--o1-sink".into());
a.push(s.to_string());
}
}
(
cfg.cortiq_bin.clone(),
a,
std::env::current_dir().unwrap_or_else(|_| std::path::PathBuf::from(".")),
)
}; };
let ret_id = id.clone(); let ret_id = id.clone();
@@ -726,32 +404,3 @@ pub fn start_import(store: Arc<JobStore>, cfg: &CmfCfg, p: ImportParams) -> Resu
Ok(ret_id) Ok(ret_id)
} }
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_map_quant_variants() {
assert_eq!(map_quant("Q8_2F"), "q8_2f");
assert_eq!(map_quant("q82f"), "q8_2f");
assert_eq!(map_quant("Q8_ROW"), "q8");
assert_eq!(map_quant("Q4_BLOCK"), "q4");
assert_eq!(map_quant("Q4_TILED"), "q4t");
assert_eq!(map_quant("q4t"), "q4t");
assert_eq!(map_quant("vbit"), "vbit");
assert_eq!(map_quant("q1"), "q1");
assert_eq!(map_quant("Q1P"), "q1p");
assert_eq!(map_quant("q1s"), "q1s");
assert_eq!(map_quant("Q1T"), "q1t");
assert_eq!(map_quant("F16"), "f16");
}
#[test]
fn test_is_cmf_repo_check() {
assert!(is_cmf_repo("infosave/Bonsai-8B_2bit_cmf"));
assert!(is_cmf_repo("infosave/Bonsai-1.7Bcmf"));
assert!(is_cmf_repo("user/model.cmf"));
assert!(!is_cmf_repo("Qwen/Qwen2.5-0.5B-Instruct"));
}
}
+1 -12
View File
@@ -61,11 +61,6 @@ fn resolve_admin_token(cfg: &Config, args: &Args) -> (String, bool) {
return (t.clone(), false); return (t.clone(), false);
} }
} }
if let Some(t) = &cfg.admin.token {
if !t.is_empty() {
return (t.clone(), false);
}
}
(admin::random_token(24), true) (admin::random_token(24), true)
} }
@@ -99,7 +94,6 @@ async fn main() -> anyhow::Result<()> {
let mut app = Router::new() let mut app = Router::new()
.route("/healthz", get(|| async { "ok" })) .route("/healthz", get(|| async { "ok" }))
.route("/health", get(|| async { "ok" }))
.route("/readyz", get(|| async { "ready" })) .route("/readyz", get(|| async { "ready" }))
.route("/metrics", get(admin::metrics)) .route("/metrics", get(admin::metrics))
.merge(protocols::build_router()); .merge(protocols::build_router());
@@ -110,12 +104,7 @@ async fn main() -> anyhow::Result<()> {
.fallback(admin::assets::fallback); .fallback(admin::assets::fallback);
} }
let cors = tower_http::cors::CorsLayer::new() let app = app.with_state(state);
.allow_origin(tower_http::cors::Any)
.allow_methods(tower_http::cors::Any)
.allow_headers(tower_http::cors::Any);
let app = app.with_state(state).layer(cors);
let listener = tokio::net::TcpListener::bind(&listen).await?; let listener = tokio::net::TcpListener::bind(&listen).await?;
tracing::info!("cortiq-gateway listening on {listen}"); tracing::info!("cortiq-gateway listening on {listen}");
-1
View File
@@ -29,7 +29,6 @@ pub struct GenParams {
pub temperature: Option<f32>, pub temperature: Option<f32>,
pub max_tokens: Option<u32>, pub max_tokens: Option<u32>,
pub top_p: Option<f32>, pub top_p: Option<f32>,
pub think_budget: Option<u32>,
pub stop: Vec<String>, pub stop: Vec<String>,
/// Fields we do not interpret but must proxy through to the provider. /// Fields we do not interpret but must proxy through to the provider.
pub passthrough: serde_json::Map<String, serde_json::Value>, pub passthrough: serde_json::Map<String, serde_json::Value>,
+14 -161
View File
@@ -208,25 +208,15 @@ impl Pipeline {
}; };
let (decision, route_source, candidates) = if live.cfg.cmf.local_only { let (decision, route_source, candidates) = if live.cfg.cmf.local_only {
( (mk("local-only", "local"), "local".to_string(), live.local_candidates())
mk("local-only", "local"),
"local".to_string(),
live.local_candidates(),
)
} else { } else {
match &req.routing { match &req.routing {
RoutingDirective::Pinned { model_id } => ( RoutingDirective::Pinned { model_id } => {
mk("pinned", "pinned"), (mk("pinned", "pinned"), "pinned".to_string(), vec![model_id.clone()])
"pinned".to_string(), }
vec![model_id.clone()],
),
RoutingDirective::Auto { profile } => { RoutingDirective::Auto { profile } => {
if !live.cfg.router.enabled { if !live.cfg.router.enabled {
( (mk("router-disabled", "local"), "local".to_string(), live.local_candidates())
mk("router-disabled", "local"),
"local".to_string(),
live.local_candidates(),
)
} else { } else {
let prof = profile.as_deref().unwrap_or(&live.cfg.route.profile); let prof = profile.as_deref().unwrap_or(&live.cfg.route.profile);
match live.router.route(&text, prof).await { match live.router.route(&text, prof).await {
@@ -239,11 +229,7 @@ impl Pipeline {
tracing::warn!( tracing::warn!(
"Router is unavailable. Gracefully degrading to the local/default model." "Router is unavailable. Gracefully degrading to the local/default model."
); );
( (mk("degraded", "fallback"), "fallback".to_string(), live.local_candidates())
mk("degraded", "fallback"),
"fallback".to_string(),
live.local_candidates(),
)
} }
} }
} }
@@ -267,21 +253,9 @@ impl Pipeline {
} }
} }
// An explicit pin must never silently answer from a different model: if // Local-first safety net: if routing produced no usable candidate (e.g.
// the pinned id is unknown or was filtered out, fail loudly so the caller // an unknown pinned model while the router is off), fall back to the
// knows their choice was not honoured (instead of getting a stray reply // managed local model so a configured local backend still answers.
// from the managed local model via the safety net below).
if final_candidates.is_empty() {
if let RoutingDirective::Pinned { model_id } = &req.routing {
return Err(GatewayError::UpstreamUnavailable(format!(
"pinned model '{model_id}' is not available — it is not in the pool (deleted?) or lacks a required capability such as tools"
)));
}
}
// Local-first safety net: for AUTO/degraded routes only, if routing
// produced no usable candidate, fall back to the managed local model so a
// configured local backend still answers.
if final_candidates.is_empty() && live.cfg.cmf.manage_server { if final_candidates.is_empty() && live.cfg.cmf.manage_server {
for id in live.local_candidates() { for id in live.local_candidates() {
if let Some(prov) = live.registry.get(&id) { if let Some(prov) = live.registry.get(&id) {
@@ -315,10 +289,7 @@ impl Pipeline {
&& !sh.local_model_id.is_empty() && !sh.local_model_id.is_empty()
&& matches!(req.routing, RoutingDirective::Auto { .. }) && matches!(req.routing, RoutingDirective::Auto { .. })
&& state.promotion.serves_local(&decision.task_label) && state.promotion.serves_local(&decision.task_label)
&& !matches!( && !matches!(decision.complexity_tier.as_str(), "high" | "hard" | "complex")
decision.complexity_tier.as_str(),
"high" | "hard" | "complex"
)
&& !final_candidates.iter().any(|m| m == &sh.local_model_id) && !final_candidates.iter().any(|m| m == &sh.local_model_id)
{ {
final_candidates.insert(0, sh.local_model_id.clone()); final_candidates.insert(0, sh.local_model_id.clone());
@@ -341,64 +312,7 @@ impl Pipeline {
route_source route_source
); );
let mut req_call = req.clone(); match provider.chat(req.clone()).await {
if let Some(m_cfg) = live.cfg.models.iter().find(|m| m.id == model_id) {
if req_call.params.temperature.is_none() {
req_call.params.temperature = m_cfg.temperature;
}
if req_call.params.top_p.is_none() {
req_call.params.top_p = m_cfg.top_p;
}
if req_call.params.max_tokens.is_none() {
req_call.params.max_tokens = m_cfg.max_tokens;
}
if req_call.params.think_budget.is_none() {
req_call.params.think_budget = m_cfg.think_budget;
}
if let Some(sys) = &m_cfg.system_prompt {
if !sys.trim().is_empty()
&& !req_call.messages.iter().any(|m| m.role == "system")
{
req_call.messages.insert(
0,
Message {
role: "system".into(),
content: sys.clone(),
tool_calls: Vec::new(),
},
);
}
}
} else if let Some(s_cfg) = live.cfg.cmf.servers.iter().find(|s| s.id == model_id) {
if req_call.params.temperature.is_none() {
req_call.params.temperature = s_cfg.temperature;
}
if req_call.params.top_p.is_none() {
req_call.params.top_p = s_cfg.top_p;
}
if req_call.params.max_tokens.is_none() {
req_call.params.max_tokens = s_cfg.max_tokens;
}
if req_call.params.think_budget.is_none() {
req_call.params.think_budget = s_cfg.think_budget;
}
if let Some(sys) = &s_cfg.system_prompt {
if !sys.trim().is_empty()
&& !req_call.messages.iter().any(|m| m.role == "system")
{
req_call.messages.insert(
0,
Message {
role: "system".into(),
content: sys.clone(),
tool_calls: Vec::new(),
},
);
}
}
}
match provider.chat(req_call).await {
Ok(mut resp) => { Ok(mut resp) => {
let cost = let cost =
provider.price(resp.usage.prompt_tokens, resp.usage.completion_tokens); provider.price(resp.usage.prompt_tokens, resp.usage.completion_tokens);
@@ -472,65 +386,7 @@ impl Pipeline {
model_id, model_id,
route_source route_source
); );
// Merge per-model/server config (same as run_inner). match provider.chat_stream(req.clone()).await {
let mut req_call = req.clone();
if let Some(m_cfg) = live.cfg.models.iter().find(|m| m.id == model_id) {
if req_call.params.temperature.is_none() {
req_call.params.temperature = m_cfg.temperature;
}
if req_call.params.top_p.is_none() {
req_call.params.top_p = m_cfg.top_p;
}
if req_call.params.max_tokens.is_none() {
req_call.params.max_tokens = m_cfg.max_tokens;
}
if req_call.params.think_budget.is_none() {
req_call.params.think_budget = m_cfg.think_budget;
}
if let Some(sys) = &m_cfg.system_prompt {
if !sys.trim().is_empty()
&& !req_call.messages.iter().any(|m| m.role == "system")
{
req_call.messages.insert(
0,
Message {
role: "system".into(),
content: sys.clone(),
tool_calls: Vec::new(),
},
);
}
}
} else if let Some(s_cfg) = live.cfg.cmf.servers.iter().find(|s| s.id == model_id) {
if req_call.params.temperature.is_none() {
req_call.params.temperature = s_cfg.temperature;
}
if req_call.params.top_p.is_none() {
req_call.params.top_p = s_cfg.top_p;
}
if req_call.params.max_tokens.is_none() {
req_call.params.max_tokens = s_cfg.max_tokens;
}
if req_call.params.think_budget.is_none() {
req_call.params.think_budget = s_cfg.think_budget;
}
if let Some(sys) = &s_cfg.system_prompt {
if !sys.trim().is_empty()
&& !req_call.messages.iter().any(|m| m.role == "system")
{
req_call.messages.insert(
0,
Message {
role: "system".into(),
content: sys.clone(),
tool_calls: Vec::new(),
},
);
}
}
}
match provider.chat_stream(req_call).await {
Ok(stream) => { Ok(stream) => {
let info = RouteInfo { let info = RouteInfo {
task_label: decision.task_label.clone(), task_label: decision.task_label.clone(),
@@ -857,7 +713,7 @@ fn parse_judge(text: &str) -> (u8, bool) {
let score = text let score = text
.split("score") .split("score")
.nth(1) .nth(1)
.and_then(|t| t.chars().find(|c| c.is_ascii_digit())) .and_then(|t| t.chars().skip_while(|c| !c.is_ascii_digit()).next())
.and_then(|c| c.to_digit(10)) .and_then(|c| c.to_digit(10))
.map(|d| d.min(4) as u8); .map(|d| d.min(4) as u8);
let hard_fail = text let hard_fail = text
@@ -891,10 +747,7 @@ async fn shadow_capture(
}; };
// 1. Local answer (free, CPU). // 1. Local answer (free, CPU).
let a_local = match local let a_local = match local.chat(shadow_req(&cfg.local_model_id, clip(&prompt, 4000), 512)).await {
.chat(shadow_req(&cfg.local_model_id, clip(&prompt, 4000), 512))
.await
{
Ok(r) => r Ok(r) => r
.choices .choices
.into_iter() .into_iter()
+30 -96
View File
@@ -18,10 +18,7 @@ use std::sync::Mutex;
use std::time::{SystemTime, UNIX_EPOCH}; use std::time::{SystemTime, UNIX_EPOCH};
fn now_secs() -> u64 { fn now_secs() -> u64 {
SystemTime::now() SystemTime::now().duration_since(UNIX_EPOCH).unwrap_or_default().as_secs()
.duration_since(UNIX_EPOCH)
.unwrap_or_default()
.as_secs()
} }
/// One judgment of a local answer (JSONL line, replayed on startup). /// One judgment of a local answer (JSONL line, replayed on startup).
@@ -65,14 +62,14 @@ pub enum PromotionState {
#[derive(Clone, Debug)] #[derive(Clone, Debug)]
pub struct PromotionCfg { pub struct PromotionCfg {
pub enabled: bool, pub enabled: bool,
pub window: usize, // rolling judgments per label (W) pub window: usize, // rolling judgments per label (W)
pub n_min: usize, // min judgments before promotion is even considered pub n_min: usize, // min judgments before promotion is even considered
pub promote_lb: f64, // Wilson-95 LB of pass must clear this pub promote_lb: f64, // Wilson-95 LB of pass must clear this
pub demote_pass: f64, // point pass-rate below this → demote a served label pub demote_pass: f64, // point pass-rate below this → demote a served label
pub hardfail_ub_max: f64, // Wilson-95 UPPER bound of hard_fail must stay under this pub hardfail_ub_max: f64,// Wilson-95 UPPER bound of hard_fail must stay under this
pub soak: usize, // extra Canary judgments before LocalServed pub soak: usize, // extra Canary judgments before LocalServed
pub z: f64, // 1.96 = 95% pub z: f64, // 1.96 = 95%
pub file: Option<String>, // judge.jsonl path (append + replay) pub file: Option<String>,// judge.jsonl path (append + replay)
} }
impl Default for PromotionCfg { impl Default for PromotionCfg {
@@ -95,7 +92,7 @@ struct LabelState {
passes: VecDeque<bool>, // rolling window of pass/fail passes: VecDeque<bool>, // rolling window of pass/fail
hard_fails: VecDeque<bool>, // rolling window of hard_fail flags hard_fails: VecDeque<bool>, // rolling window of hard_fail flags
state: PromotionState, state: PromotionState,
soaked: usize, // judgments accumulated while in Canary soaked: usize, // judgments accumulated while in Canary
} }
impl LabelState { impl LabelState {
@@ -127,29 +124,20 @@ pub fn wilson_bound(s: usize, n: usize, z: f64, upper: bool) -> f64 {
let denom = 1.0 + z2 / n; let denom = 1.0 + z2 / n;
let centre = p + z2 / (2.0 * n); let centre = p + z2 / (2.0 * n);
let margin = z * ((p * (1.0 - p) / n) + z2 / (4.0 * n * n)).sqrt(); let margin = z * ((p * (1.0 - p) / n) + z2 / (4.0 * n * n)).sqrt();
let v = if upper { let v = if upper { centre + margin } else { centre - margin };
centre + margin
} else {
centre - margin
};
(v / denom).clamp(0.0, 1.0) (v / denom).clamp(0.0, 1.0)
} }
impl Promotion { impl Promotion {
pub fn new(cfg: PromotionCfg) -> std::sync::Arc<Self> { pub fn new(cfg: PromotionCfg) -> std::sync::Arc<Self> {
let p = std::sync::Arc::new(Self { let p = std::sync::Arc::new(Self { cfg, labels: Mutex::new(HashMap::new()) });
cfg,
labels: Mutex::new(HashMap::new()),
});
p.replay(); p.replay();
p p
} }
fn replay(&self) { fn replay(&self) {
let Some(path) = &self.cfg.file else { return }; let Some(path) = &self.cfg.file else { return };
let Ok(content) = std::fs::read_to_string(path) else { let Ok(content) = std::fs::read_to_string(path) else { return };
return;
};
for line in content.lines() { for line in content.lines() {
if let Ok(rec) = serde_json::from_str::<JudgeRecord>(line) { if let Ok(rec) = serde_json::from_str::<JudgeRecord>(line) {
self.apply(&rec); self.apply(&rec);
@@ -163,10 +151,8 @@ impl Promotion {
let st = self.apply(&rec); let st = self.apply(&rec);
if let Some(path) = &self.cfg.file { if let Some(path) = &self.cfg.file {
if let Ok(line) = serde_json::to_string(&rec) { if let Ok(line) = serde_json::to_string(&rec) {
if let Ok(mut f) = std::fs::OpenOptions::new() if let Ok(mut f) =
.create(true) std::fs::OpenOptions::new().create(true).append(true).open(path)
.append(true)
.open(path)
{ {
let _ = writeln!(f, "{line}"); let _ = writeln!(f, "{line}");
} }
@@ -177,9 +163,7 @@ impl Promotion {
fn apply(&self, rec: &JudgeRecord) -> PromotionState { fn apply(&self, rec: &JudgeRecord) -> PromotionState {
let mut map = self.labels.lock().unwrap(); let mut map = self.labels.lock().unwrap();
let ls = map let ls = map.entry(rec.task_label.clone()).or_insert_with(LabelState::new);
.entry(rec.task_label.clone())
.or_insert_with(LabelState::new);
push_cap(&mut ls.passes, rec.pass, self.cfg.window); push_cap(&mut ls.passes, rec.pass, self.cfg.window);
push_cap(&mut ls.hard_fails, rec.hard_fail, self.cfg.window); push_cap(&mut ls.hard_fails, rec.hard_fail, self.cfg.window);
self.advance(ls); self.advance(ls);
@@ -204,12 +188,7 @@ impl Promotion {
ls.state = match ls.state { ls.state = match ls.state {
PromotionState::CloudOnly | PromotionState::ShadowJudged => { PromotionState::CloudOnly | PromotionState::ShadowJudged => {
if bar { if bar { ls.soaked = 0; PromotionState::Canary } else { PromotionState::ShadowJudged }
ls.soaked = 0;
PromotionState::Canary
} else {
PromotionState::ShadowJudged
}
} }
PromotionState::Canary => { PromotionState::Canary => {
if regressed { if regressed {
@@ -224,11 +203,7 @@ impl Promotion {
} }
} }
PromotionState::LocalServed => { PromotionState::LocalServed => {
if regressed { if regressed { PromotionState::Demoted } else { PromotionState::LocalServed }
PromotionState::Demoted
} else {
PromotionState::LocalServed
}
} }
// Demoted goes back to SHADOW to re-earn promotion (skill stays in .cmf). // Demoted goes back to SHADOW to re-earn promotion (skill stays in .cmf).
PromotionState::Demoted => PromotionState::ShadowJudged, PromotionState::Demoted => PromotionState::ShadowJudged,
@@ -248,10 +223,7 @@ impl Promotion {
/// Should a request of this label be SERVED by the local model? /// Should a request of this label be SERVED by the local model?
/// (Canary/LocalServed; caller still applies the per-request veto.) /// (Canary/LocalServed; caller still applies the per-request veto.)
pub fn serves_local(&self, label: &str) -> bool { pub fn serves_local(&self, label: &str) -> bool {
matches!( matches!(self.state(label), PromotionState::Canary | PromotionState::LocalServed)
self.state(label),
PromotionState::Canary | PromotionState::LocalServed
)
} }
/// Snapshot for /admin: (state, n, pass-rate, Wilson-LB) per label. /// Snapshot for /admin: (state, n, pass-rate, Wilson-LB) per label.
@@ -263,16 +235,10 @@ impl Promotion {
let n = ls.passes.len(); let n = ls.passes.len();
let s = ls.passes.iter().filter(|&&p| p).count(); let s = ls.passes.iter().filter(|&&p| p).count();
let p = if n > 0 { s as f64 / n as f64 } else { 0.0 }; let p = if n > 0 { s as f64 / n as f64 } else { 0.0 };
( (k.clone(), ls.state, n, p, wilson_bound(s, n, self.cfg.z, false))
k.clone(),
ls.state,
n,
p,
wilson_bound(s, n, self.cfg.z, false),
)
}) })
.collect(); .collect();
out.sort_by_key(|x| std::cmp::Reverse(x.2)); out.sort_by(|a, b| b.2.cmp(&a.2));
out out
} }
@@ -293,24 +259,12 @@ mod tests {
use super::*; use super::*;
fn cfg() -> PromotionCfg { fn cfg() -> PromotionCfg {
PromotionCfg { PromotionCfg { enabled: true, n_min: 200, soak: 0, ..Default::default() }
enabled: true,
n_min: 200,
soak: 0,
..Default::default()
}
} }
fn rec(label: &str, pass: bool, hard_fail: bool) -> JudgeRecord { fn rec(label: &str, pass: bool, hard_fail: bool) -> JudgeRecord {
JudgeRecord { JudgeRecord {
ts: 0, ts: 0, task_label: label.into(), pass, hard_fail, score: if pass { 4 } else { 0 },
task_label: label.into(), source: "checker".into(), recon_e: None, born_conf: None, judge_tokens: 0,
pass,
hard_fail,
score: if pass { 4 } else { 0 },
source: "checker".into(),
recon_e: None,
born_conf: None,
judge_tokens: 0,
} }
} }
@@ -362,10 +316,7 @@ mod tests {
match row { match row {
Some((_, st, n, pr, lb)) => println!( Some((_, st, n, pr, lb)) => println!(
" {label:<12} {:<13} n={n:<4} pass={:.0}% LB={:.3} serves_local={}", " {label:<12} {:<13} n={n:<4} pass={:.0}% LB={:.3} serves_local={}",
format!("{st:?}"), format!("{st:?}"), pr * 100.0, lb, p.serves_local(label)
pr * 100.0,
lb,
p.serves_local(label)
), ),
None => println!(" {label:<12} (cloud-only, no judgments)"), None => println!(" {label:<12} (cloud-only, no judgments)"),
} }
@@ -373,29 +324,19 @@ mod tests {
println!("\n── self-warming loop, live ──"); println!("\n── self-warming loop, live ──");
// A mature skill (e.g. ru-tech): local answers pass the judge ~99%. // A mature skill (e.g. ru-tech): local answers pass the judge ~99%.
println!( println!(" 1) SHADOW→CANARY: судим локаль 'ru_tech' (сильный скилл ~99% pass), 210 судейств:");
" 1) SHADOW→CANARY: судим локаль 'ru_tech' (сильный скилл ~99% pass), 210 судейств:"
);
for i in 0..210 { for i in 0..210 {
p.record(rec("ru_tech", i % 100 != 0, false)); p.record(rec("ru_tech", i % 100 != 0, false));
} }
show("ru_tech"); show("ru_tech");
assert_eq!( assert_eq!(p.state("ru_tech"), PromotionState::Canary, "n≥200 & LB≥0.95 → CANARY");
p.state("ru_tech"),
PromotionState::Canary,
"n≥200 & LB≥0.95 → CANARY"
);
println!(" 2) CANARY soak (ещё 60 судейств держат планку) → LOCAL_SERVED:"); println!(" 2) CANARY soak (ещё 60 судейств держат планку) → LOCAL_SERVED:");
for i in 0..60 { for i in 0..60 {
p.record(rec("ru_tech", i % 100 != 0, false)); p.record(rec("ru_tech", i % 100 != 0, false));
} }
show("ru_tech"); show("ru_tech");
assert_eq!( assert_eq!(p.state("ru_tech"), PromotionState::LocalServed, "soak → LOCAL_SERVED");
p.state("ru_tech"),
PromotionState::LocalServed,
"soak → LOCAL_SERVED"
);
assert!(p.serves_local("ru_tech"), "теперь обслуживается локально"); assert!(p.serves_local("ru_tech"), "теперь обслуживается локально");
// The code dataset: local fails the checker → never promotes. // The code dataset: local fails the checker → never promotes.
@@ -404,11 +345,7 @@ mod tests {
p.record(rec("code", false, false)); p.record(rec("code", false, false));
} }
show("code"); show("code");
assert_eq!( assert_eq!(p.state("code"), PromotionState::ShadowJudged, "остаётся на облаке — безопасно");
p.state("code"),
PromotionState::ShadowJudged,
"остаётся на облаке — безопасно"
);
assert!(!p.serves_local("code")); assert!(!p.serves_local("code"));
// Drift on the promoted label → auto-demote back to cloud. // Drift on the promoted label → auto-demote back to cloud.
@@ -439,9 +376,6 @@ mod tests {
for _ in 0..300 { for _ in 0..300 {
p2.record(rec("t", false, false)); // quality collapses p2.record(rec("t", false, false)); // quality collapses
} }
assert!(matches!( assert!(matches!(p2.state("t"), PromotionState::Demoted | PromotionState::ShadowJudged));
p2.state("t"),
PromotionState::Demoted | PromotionState::ShadowJudged
));
} }
} }
-1
View File
@@ -101,7 +101,6 @@ async fn handler(
temperature: req.temperature, temperature: req.temperature,
max_tokens: req.max_tokens, max_tokens: req.max_tokens,
top_p: req.top_p, top_p: req.top_p,
think_budget: None,
stop: Vec::new(), stop: Vec::new(),
passthrough: Default::default(), passthrough: Default::default(),
}, },
-5
View File
@@ -49,10 +49,6 @@ struct OpenAiChatRequest {
#[serde(default)] #[serde(default)]
top_p: Option<f32>, top_p: Option<f32>,
#[serde(default)] #[serde(default)]
think_budget: Option<u32>,
#[serde(default)]
reasoning_effort: Option<u32>,
#[serde(default)]
stream: bool, stream: bool,
#[serde(default)] #[serde(default)]
tools: Vec<serde_json::Value>, tools: Vec<serde_json::Value>,
@@ -97,7 +93,6 @@ async fn handler(
temperature: req.temperature, temperature: req.temperature,
max_tokens: req.max_tokens, max_tokens: req.max_tokens,
top_p: req.top_p, top_p: req.top_p,
think_budget: req.think_budget.or(req.reasoning_effort),
stop: Vec::new(), stop: Vec::new(),
passthrough: req.rest, passthrough: req.rest,
}, },
-1
View File
@@ -82,7 +82,6 @@ async fn handler(
temperature: req.temperature, temperature: req.temperature,
max_tokens: req.max_tokens, max_tokens: req.max_tokens,
top_p: req.top_p, top_p: req.top_p,
think_budget: None,
stop: Vec::new(), stop: Vec::new(),
passthrough: Default::default(), passthrough: Default::default(),
}, },
-15
View File
@@ -42,20 +42,9 @@ impl OpenAiProvider {
if let Some(t) = req.params.temperature { if let Some(t) = req.params.temperature {
body["temperature"] = t.into(); body["temperature"] = t.into();
} }
if let Some(p) = req.params.top_p {
body["top_p"] = p.into();
}
if let Some(m) = req.params.max_tokens { if let Some(m) = req.params.max_tokens {
body["max_tokens"] = m.into(); body["max_tokens"] = m.into();
} }
eprintln!("[to_wire] think_budget={:?}", req.params.think_budget);
if let Some(tb) = req.params.think_budget {
body["think_budget"] = tb.into();
body["reasoning_effort"] = tb.into();
// cortiq serve uses enable_thinking (bool): 0 = off, >0 = on
body["enable_thinking"] = (tb > 0).into();
eprintln!("[to_wire] SET enable_thinking={}", tb > 0);
}
if !req.tools.is_empty() { if !req.tools.is_empty() {
body["tools"] = serde_json::Value::Array(req.tools.clone()); body["tools"] = serde_json::Value::Array(req.tools.clone());
} }
@@ -63,10 +52,6 @@ impl OpenAiProvider {
for (k, v) in &req.params.passthrough { for (k, v) in &req.params.passthrough {
body[k] = v.clone(); body[k] = v.clone();
} }
eprintln!(
"[to_wire] enable_thinking={:?}",
body.get("enable_thinking")
);
body body
} }
} }
+10 -20
View File
@@ -21,27 +21,24 @@ impl Registry {
.as_ref() .as_ref()
.and_then(|name| secrets.resolve(name)); .and_then(|name| secrets.resolve(name));
let provider: Arc<dyn Provider> = match m.provider.as_str() { let provider: Arc<dyn Provider> = match m.provider.as_str() {
// openrouter + lmstudio are OpenAI-compatible (Bearer + /chat/completions) "openai" | "ollama" | "http" => Arc::new(OpenAiProvider::new(m, api_key)?),
"openai" | "ollama" | "http" | "openrouter" | "lmstudio" => {
Arc::new(OpenAiProvider::new(m, api_key)?)
}
"anthropic" => Arc::new(AnthropicProvider::new(m, api_key)?), "anthropic" => Arc::new(AnthropicProvider::new(m, api_key)?),
other => anyhow::bail!("unknown provider '{}' for model '{}'", other, m.id), other => anyhow::bail!("unknown provider '{}' for model '{}'", other, m.id),
}; };
by_id.insert(m.id.clone(), provider); by_id.insert(m.id.clone(), provider);
} }
// Managed local CMF servers: register each as an OpenAI-compatible // Managed local CMF server: register it as an OpenAI-compatible provider
// provider so it can be routed to like any other backend — including when // so it can be routed to like any other backend — including when the
// the router is unavailable or the gateway is in local-only mode. // router is unavailable or the gateway is in local-only mode.
for s in cfg.cmf.effective_servers() { if cfg.cmf.manage_server
if by_id.contains_key(&s.id) { && !cfg.cmf.local_model.trim().is_empty()
continue; // a static [[models]] entry with the same id wins && !by_id.contains_key(&cfg.cmf.model_id)
} {
let m = ModelCfg { let m = ModelCfg {
id: s.id.clone(), id: cfg.cmf.model_id.clone(),
provider: "openai".into(), provider: "openai".into(),
base_url: format!("http://{}:{}/v1", cfg.cmf.local_host, s.port), base_url: format!("http://{}:{}/v1", cfg.cmf.local_host, cfg.cmf.local_port),
model: "cortiq".into(), model: "cortiq".into(),
cost_tier: "local".into(), cost_tier: "local".into(),
price_in: 0.0, price_in: 0.0,
@@ -49,13 +46,6 @@ impl Registry {
kind: "chat".into(), kind: "chat".into(),
api_key_env: None, api_key_env: None,
caps: Vec::new(), caps: Vec::new(),
temperature: s.temperature,
top_p: s.top_p,
max_tokens: s.max_tokens,
think_budget: s.think_budget,
system_prompt: s.system_prompt.clone(),
o1: s.o1.clone(),
skip_mtp: s.skip_mtp,
}; };
by_id.insert(m.id.clone(), Arc::new(OpenAiProvider::new(&m, None)?)); by_id.insert(m.id.clone(), Arc::new(OpenAiProvider::new(&m, None)?));
} }
+3 -15
View File
@@ -49,8 +49,8 @@ impl Live {
/// (local-only / disabled) or unavailable, so CMF models serve with no router. /// (local-only / disabled) or unavailable, so CMF models serve with no router.
pub fn local_candidates(&self) -> Vec<String> { pub fn local_candidates(&self) -> Vec<String> {
let mut v = Vec::new(); let mut v = Vec::new();
for s in self.cfg.cmf.effective_servers() { if self.cfg.cmf.manage_server && !self.cfg.cmf.local_model.trim().is_empty() {
v.push(s.id); v.push(self.cfg.cmf.model_id.clone());
} }
let def = self.routing.default_model().to_string(); let def = self.routing.default_model().to_string();
if !def.is_empty() && !v.contains(&def) { if !def.is_empty() && !v.contains(&def) {
@@ -90,11 +90,7 @@ fn promotion_cfg(s: &crate::config::ShadowCfg) -> crate::promotion::PromotionCfg
n_min: s.n_min, n_min: s.n_min,
promote_lb: s.promote_lb, promote_lb: s.promote_lb,
soak: s.soak, soak: s.soak,
file: if s.file.is_empty() { file: if s.file.is_empty() { None } else { Some(s.file.clone()) },
None
} else {
Some(s.file.clone())
},
..Default::default() ..Default::default()
} }
} }
@@ -145,14 +141,6 @@ impl AppState {
let live = Live::build(new_cfg.clone(), &self.secrets, self.router_status.clone())?; let live = Live::build(new_cfg.clone(), &self.secrets, self.router_status.clone())?;
new_cfg.save(&self.config_path)?; new_cfg.save(&self.config_path)?;
self.live.store(Arc::new(live)); self.live.store(Arc::new(live));
let cmf_rt = self.cmf.clone();
let cmf_cfg = new_cfg.cmf.clone();
tokio::spawn(async move {
cmf_rt.stop().await;
crate::cmf_runtime::manage(cmf_rt, cmf_cfg).await;
});
Ok(()) Ok(())
} }
+41 -93
View File
@@ -83,23 +83,9 @@ impl Totals {
self.cost_usd += r.cost_usd; self.cost_usd += r.cost_usd;
self.latency_ms_sum += r.latency_ms; self.latency_ms_sum += r.latency_ms;
} }
/// Accumulate another Totals into this one (for windowed aggregation).
fn merge(&mut self, o: &Totals) {
self.requests += o.requests;
self.ok += o.ok;
self.errors += o.errors;
self.failovers += o.failovers;
self.prompt_tokens += o.prompt_tokens;
self.completion_tokens += o.completion_tokens;
self.cost_usd += o.cost_usd;
self.latency_ms_sum += o.latency_ms_sum;
}
} }
/// Per-minute bucket. Beyond the chart fields it carries the per-group breakdown /// Per-minute bucket for time-series charts.
/// and failovers so the admin snapshot can be computed for ANY time window by
/// summing the in-window buckets (the internal fields are not serialized into
/// the `series` payload).
#[derive(Clone, Debug, Default, Serialize)] #[derive(Clone, Debug, Default, Serialize)]
pub struct Bucket { pub struct Bucket {
pub minute: u64, // unix seconds, rounded down to the minute pub minute: u64, // unix seconds, rounded down to the minute
@@ -109,25 +95,15 @@ pub struct Bucket {
pub completion_tokens: u64, pub completion_tokens: u64,
pub cost_usd: f64, pub cost_usd: f64,
pub latency_ms_sum: u64, pub latency_ms_sum: u64,
#[serde(skip)]
pub failovers: u64,
#[serde(skip)]
pub by_model: HashMap<String, Totals>,
#[serde(skip)]
pub by_account: HashMap<String, Totals>,
#[serde(skip)]
pub by_tier: HashMap<String, Totals>,
#[serde(skip)]
pub by_label: HashMap<String, Totals>,
} }
#[derive(Default)] #[derive(Default)]
struct Inner { struct Inner {
/// All-time totals + per-model (used by the Prometheus counters, which must
/// be monotonic — the windowed admin snapshot is derived from the buckets).
total: Totals, total: Totals,
by_model: HashMap<String, Totals>, by_model: HashMap<String, Totals>,
/// Per-minute buckets — the single source for the windowed admin snapshot. by_account: HashMap<String, Totals>,
by_tier: HashMap<String, Totals>,
by_label: HashMap<String, Totals>,
buckets: VecDeque<Bucket>, buckets: VecDeque<Bucket>,
recent: VecDeque<RequestRecord>, recent: VecDeque<RequestRecord>,
} }
@@ -190,14 +166,31 @@ impl Stats {
} }
fn apply_aggregates(inner: &mut Inner, rec: &RequestRecord, retention_secs: u64) { fn apply_aggregates(inner: &mut Inner, rec: &RequestRecord, retention_secs: u64) {
// all-time counters for Prometheus; the windowed snapshot is derived
// from the buckets below.
inner.total.apply(rec); inner.total.apply(rec);
inner inner
.by_model .by_model
.entry(rec.model_id.clone()) .entry(rec.model_id.clone())
.or_default() .or_default()
.apply(rec); .apply(rec);
inner
.by_account
.entry(if rec.account.is_empty() {
"anonymous".to_string()
} else {
rec.account.clone()
})
.or_default()
.apply(rec);
inner
.by_tier
.entry(rec.tier.clone())
.or_default()
.apply(rec);
inner
.by_label
.entry(rec.task_label.clone())
.or_default()
.apply(rec);
let minute = rec.ts - (rec.ts % 60); let minute = rec.ts - (rec.ts % 60);
match inner.buckets.back_mut() { match inner.buckets.back_mut() {
@@ -257,31 +250,13 @@ impl Stats {
.filter(|b| b.minute >= cutoff) .filter(|b| b.minute >= cutoff)
.collect(); .collect();
// Windowed totals + breakdown = sum of the in-window buckets, so the KPIs let breakdown_src = match groupby {
// and the breakdown recompute whenever the range changes. "account" => &inner.by_account,
let mut win = Totals::default(); "tier" => &inner.by_tier,
let mut group: HashMap<String, Totals> = HashMap::new(); "label" => &inner.by_label,
for b in &series { _ => &inner.by_model,
win.requests += b.requests; };
win.errors += b.errors; let mut breakdown: Vec<serde_json::Value> = breakdown_src
win.ok += b.requests.saturating_sub(b.errors);
win.failovers += b.failovers;
win.prompt_tokens += b.prompt_tokens;
win.completion_tokens += b.completion_tokens;
win.cost_usd += b.cost_usd;
win.latency_ms_sum += b.latency_ms_sum;
let src = match groupby {
"account" => &b.by_account,
"tier" => &b.by_tier,
"label" => &b.by_label,
_ => &b.by_model,
};
for (k, t) in src {
group.entry(k.clone()).or_default().merge(t);
}
}
let mut breakdown: Vec<serde_json::Value> = group
.iter() .iter()
.map(|(k, t)| { .map(|(k, t)| {
serde_json::json!({ serde_json::json!({
@@ -305,17 +280,17 @@ impl Stats {
serde_json::json!({ serde_json::json!({
"totals": { "totals": {
"requests": win.requests, "requests": inner.total.requests,
"ok": win.ok, "ok": inner.total.ok,
"errors": win.errors, "errors": inner.total.errors,
"failovers": win.failovers, "failovers": inner.total.failovers,
"prompt_tokens": win.prompt_tokens, "prompt_tokens": inner.total.prompt_tokens,
"completion_tokens": win.completion_tokens, "completion_tokens": inner.total.completion_tokens,
"total_tokens": win.prompt_tokens + win.completion_tokens, "total_tokens": inner.total.prompt_tokens + inner.total.completion_tokens,
"cost_usd": win.cost_usd, "cost_usd": inner.total.cost_usd,
"avg_latency_ms": avg(win.latency_ms_sum, win.requests), "avg_latency_ms": avg(inner.total.latency_ms_sum, inner.total.requests),
"success_rate": if win.requests > 0 { "success_rate": if inner.total.requests > 0 {
win.ok as f64 / win.requests as f64 inner.total.ok as f64 / inner.total.requests as f64
} else { 0.0 }, } else { 0.0 },
}, },
"groupby": groupby, "groupby": groupby,
@@ -324,18 +299,6 @@ impl Stats {
}) })
} }
/// Reset all in-memory stats and truncate the JSONL log — the "clear logs"
/// admin action.
pub fn clear(&self) {
{
let mut inner = self.inner.lock().unwrap();
*inner = Inner::default();
}
if let Some(path) = &self.file {
let _ = std::fs::write(path, "");
}
}
/// Recent requests (newest first), with pagination. /// Recent requests (newest first), with pagination.
pub fn recent(&self, limit: usize, offset: usize) -> Vec<RequestRecord> { pub fn recent(&self, limit: usize, offset: usize) -> Vec<RequestRecord> {
let inner = self.inner.lock().unwrap(); let inner = self.inner.lock().unwrap();
@@ -395,25 +358,10 @@ fn fill_bucket(b: &mut Bucket, r: &RequestRecord) {
if r.outcome != "ok" { if r.outcome != "ok" {
b.errors += 1; b.errors += 1;
} }
if r.failover {
b.failovers += 1;
}
b.prompt_tokens += r.prompt_tokens as u64; b.prompt_tokens += r.prompt_tokens as u64;
b.completion_tokens += r.completion_tokens as u64; b.completion_tokens += r.completion_tokens as u64;
b.cost_usd += r.cost_usd; b.cost_usd += r.cost_usd;
b.latency_ms_sum += r.latency_ms; b.latency_ms_sum += r.latency_ms;
// per-group breakdown, so the snapshot can be windowed by summing buckets
b.by_model.entry(r.model_id.clone()).or_default().apply(r);
b.by_account
.entry(if r.account.is_empty() {
"anonymous".to_string()
} else {
r.account.clone()
})
.or_default()
.apply(r);
b.by_tier.entry(r.tier.clone()).or_default().apply(r);
b.by_label.entry(r.task_label.clone()).or_default().apply(r);
} }
fn avg(sum: u64, n: u64) -> u64 { fn avg(sum: u64, n: u64) -> u64 {
-167
View File
@@ -397,170 +397,3 @@ tbody tr:hover td { background: var(--hover); }
.mono.kv { display: flex; gap: 6px; padding: 1px 0; white-space: normal; overflow-wrap: anywhere; word-break: break-word; } .mono.kv { display: flex; gap: 6px; padding: 1px 0; white-space: normal; overflow-wrap: anywhere; word-break: break-word; }
.mono.kv .kv-k { color: var(--muted, #9aa4b2); flex: 0 0 auto; } .mono.kv .kv-k { color: var(--muted, #9aa4b2); flex: 0 0 auto; }
.mono.kv .kv-v { font-weight: 600; overflow-wrap: anywhere; word-break: break-word; min-width: 0; } .mono.kv .kv-v { font-weight: 600; overflow-wrap: anywhere; word-break: break-word; min-width: 0; }
/* ---- responsive shell + touch layout ----------------------------------- */
.menu-toggle, .nav-close, .mobile-nav-backdrop { display: none; }
.hamburger { width: 18px; display: flex; flex-direction: column; gap: 4px; }
.hamburger i { display: block; height: 2px; border-radius: 2px; background: currentColor; }
body.modal-open { overflow: hidden; }
@media (max-width: 980px) {
.app { display: block; min-height: 100dvh; }
.main { min-height: 100dvh; }
.menu-toggle { display: inline-flex; flex: 0 0 auto; }
.nav-close { display: inline-flex; margin-left: auto; flex: 0 0 auto; font-size: 24px; line-height: 1; }
.brand-copy { min-width: 0; }
.sidebar {
display: flex;
position: fixed;
inset: 0 auto 0 0;
width: min(84vw, 310px);
height: 100dvh;
padding: max(18px, env(safe-area-inset-top)) 14px max(14px, env(safe-area-inset-bottom));
z-index: 70;
box-shadow: 18px 0 45px rgba(0, 0, 0, .42);
transform: translateX(-104%);
visibility: hidden;
transition: transform .22s ease, visibility 0s linear .22s;
overflow-y: auto;
}
.app.nav-open .sidebar {
transform: translateX(0);
visibility: visible;
transition: transform .22s ease;
}
.mobile-nav-backdrop {
display: block;
position: fixed;
inset: 0;
z-index: 65;
width: 100%;
height: 100%;
border: 0;
background: rgba(3, 7, 18, .62);
opacity: 0;
visibility: hidden;
pointer-events: none;
transition: opacity .22s ease, visibility 0s linear .22s;
}
.app.nav-open .mobile-nav-backdrop {
opacity: 1;
visibility: visible;
pointer-events: auto;
transition: opacity .22s ease;
}
body.mobile-nav-open { overflow: hidden; }
.topbar { padding: 10px 18px; min-height: 58px; gap: 10px; }
.topbar .title { min-width: 0; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
.topbar .right { gap: 8px; min-width: 0; }
.content { max-width: none; padding: 22px 20px 52px; }
}
@media (max-width: 720px) {
body { font-size: 14px; }
.status-pills { display: none; }
.topbar { padding: 9px 12px; }
.content { padding: 18px 14px 42px; }
.page-head { margin-bottom: 16px; }
.page-head > .flex { align-items: stretch; flex-direction: column; }
.page-head > .flex > .btn { width: 100%; }
.page-head p { line-height: 1.45; }
.card { padding: 15px; margin-bottom: 14px; border-radius: 12px; }
.card .card-head { align-items: flex-start; flex-wrap: wrap; }
.card .card-head .right { width: 100%; margin-left: 0; flex-wrap: wrap; }
.grid { gap: 14px; }
.kpis { gap: 10px; margin-bottom: 14px; }
.kpi { padding: 12px; border-radius: 11px; }
.kpi .big { font-size: 21px; }
.row { display: grid; grid-template-columns: minmax(0, 1fr); gap: 0; }
.row > * { min-width: 0; width: 100%; }
input, select, textarea { min-height: 44px; font-size: 16px; }
input[type="checkbox"], input[type="radio"] { min-height: 0; }
.check { min-height: 36px; margin-bottom: 8px; }
.btn, .icon-btn { min-height: 44px; }
.btn { padding: 10px 14px; }
.btn.sm, .btn.small { min-height: 38px; padding: 8px 11px; }
.table-wrap {
width: 100%;
overflow-x: auto;
overscroll-behavior-x: contain;
-webkit-overflow-scrolling: touch;
scrollbar-width: thin;
}
.table-wrap table { min-width: max-content; }
th, td { padding: 10px; }
.tierrow { align-items: flex-start; flex-direction: column; gap: 8px; }
.tierrow .tname { width: auto; }
.chips { width: 100%; }
.togglerow { align-items: flex-start; }
.pg { gap: 14px; }
.routecard .kv { flex-wrap: wrap; }
.routecard .kv .v { max-width: 100%; overflow-wrap: anywhere; }
.answer { padding: 12px; }
.seg { max-width: 100%; overflow-x: auto; }
.seg button { min-height: 40px; white-space: nowrap; }
.hf-grid { grid-template-columns: 1fr; max-height: 56dvh; }
.job-head, .actions { align-items: stretch; flex-wrap: wrap; }
.actions .btn { flex: 1 1 150px; }
.key-row { flex-direction: column; }
.key-row .btn { width: 100%; }
.cmf-server-row { align-items: stretch !important; padding: 10px; border: 1px solid var(--border-soft); border-radius: 10px; }
.cmf-server-row > input, .cmf-server-row > select { width: 100% !important; min-width: 0 !important; flex: 1 1 100% !important; }
.cmf-server-row > .btn { flex: 1 1 calc(50% - 6px); }
.cmf-server-row > .small { flex: 1 1 100%; }
.modal-backdrop { align-items: flex-end; padding: 8px 0 0; }
.modal {
max-width: none;
max-height: calc(100dvh - 8px);
overflow-y: auto;
padding: 18px 15px max(14px, env(safe-area-inset-bottom));
border-radius: 18px 18px 0 0;
}
.modal .modal-foot {
position: sticky;
bottom: calc(-1 * max(14px, env(safe-area-inset-bottom)));
margin: 12px -15px 0;
padding: 12px 15px max(14px, env(safe-area-inset-bottom));
background: color-mix(in srgb, var(--card) 94%, transparent);
backdrop-filter: blur(10px);
border-top: 1px solid var(--border-soft);
}
.modal .modal-foot .btn { flex: 1; }
.toasts { left: 12px; right: 12px; bottom: max(12px, env(safe-area-inset-bottom)); }
.toast { min-width: 0; width: 100%; }
.login-wrap { align-items: flex-start; padding: max(20px, env(safe-area-inset-top)) 14px; }
.login-card { padding: 22px 18px; margin-top: 8dvh; }
}
@media (max-width: 480px) {
.topbar .title { font-size: 14px; }
.topbar .right { gap: 6px; }
.lang-select { width: 54px; min-height: 44px; padding: 6px; }
.icon-btn { width: 42px; height: 42px; }
.kpis { grid-template-columns: repeat(2, minmax(0, 1fr)); }
.kpi .lbl { line-height: 1.25; }
.barlist .b { grid-template-columns: minmax(0, 1fr) auto; gap: 6px 10px; }
.barlist .b .track { grid-column: 1 / -1; grid-row: 2; }
.barlist .b .val { min-width: 0; }
.empty { padding: 28px 12px; }
}
@media (max-width: 360px) {
.topbar .title { display: none; }
.kpis { grid-template-columns: 1fr; }
}
@media (prefers-reduced-motion: reduce) {
.sidebar, .mobile-nav-backdrop { transition: none; }
}
+1 -1
View File
@@ -19,6 +19,6 @@
<noscript>This console requires JavaScript.</noscript> <noscript>This console requires JavaScript.</noscript>
<div id="app"><div class="boot">Loading…</div></div> <div id="app"><div class="boot">Loading…</div></div>
<div id="toasts" class="toasts"></div> <div id="toasts" class="toasts"></div>
<script type="module" src="/admin/js/app.js?v=3"></script> <script type="module" src="/admin/js/app.js"></script>
</body> </body>
</html> </html>
-3
View File
@@ -87,7 +87,6 @@ export const api = {
// stats // stats
stats: (range, groupby) => j("GET", `/stats?range=${range}&groupby=${groupby}`), stats: (range, groupby) => j("GET", `/stats?range=${range}&groupby=${groupby}`),
clearStats: () => j("DELETE", "/stats"),
requests: (limit = 50, offset = 0) => j("GET", `/requests?limit=${limit}&offset=${offset}`), requests: (limit = 50, offset = 0) => j("GET", `/requests?limit=${limit}&offset=${offset}`),
// playground // playground
@@ -100,8 +99,6 @@ export const api = {
cmfStatus: () => j("GET", "/cmf"), cmfStatus: () => j("GET", "/cmf"),
cmfInstall: () => j("POST", "/cmf/install"), cmfInstall: () => j("POST", "/cmf/install"),
cmfPortCheck: (port) => j("GET", "/cmf/port?port=" + encodeURIComponent(port)), cmfPortCheck: (port) => j("GET", "/cmf/port?port=" + encodeURIComponent(port)),
cmfFiles: () => j("GET", "/cmf/files"),
providerModels: (b) => j("POST", "/provider/models", b),
// CMF model factory (HuggingFace → local .cmf) // CMF model factory (HuggingFace → local .cmf)
hfSearch: (q, limit = 24) => hfSearch: (q, limit = 24) =>
+5 -39
View File
@@ -4,7 +4,7 @@ import { t, getLang, setLang, onLangChange, LANGS } from "./i18n.js";
import { api, getToken, setToken, setUnauthorizedHandler } from "./api.js"; import { api, getToken, setToken, setUnauthorizedHandler } from "./api.js";
import { renderLogin } from "./views/login.js"; import { renderLogin } from "./views/login.js";
import { renderDashboard } from "./views/dashboard.js"; import { renderDashboard } from "./views/dashboard.js";
import { renderModels } from "./views/models.js?v=3"; import { renderModels } from "./views/models.js";
import { renderImport } from "./views/import.js"; import { renderImport } from "./views/import.js";
import { renderRouting } from "./views/routing.js"; import { renderRouting } from "./views/routing.js";
import { renderProtocols } from "./views/protocols.js"; import { renderProtocols } from "./views/protocols.js";
@@ -28,17 +28,6 @@ export const appState = { meta: null, health: null, version: "" };
// allaigate site — where router keys are issued and topped up // allaigate site — where router keys are issued and topped up
export const SITE_URL = "https://api.allaigate.com/"; export const SITE_URL = "https://api.allaigate.com/";
let mobileNavOpen = false;
function setMobileNav(open) {
mobileNavOpen = !!open;
const app = document.querySelector(".app");
if (app) app.classList.toggle("nav-open", mobileNavOpen);
document.body.classList.toggle("mobile-nav-open", mobileNavOpen);
const toggle = document.querySelector(".menu-toggle");
if (toggle) toggle.setAttribute("aria-expanded", String(mobileNavOpen));
}
function currentRoute() { function currentRoute() {
const id = location.hash.replace(/^#\/?/, "") || "dashboard"; const id = location.hash.replace(/^#\/?/, "") || "dashboard";
return ROUTES.find((r) => r.id === id) || ROUTES[0]; return ROUTES.find((r) => r.id === id) || ROUTES[0];
@@ -53,18 +42,17 @@ const FAVICON =
function sidebar(activeId) { function sidebar(activeId) {
return h( return h(
"aside", "aside",
{ class: "sidebar", id: "sidebar", "aria-label": "Main navigation" }, { class: "sidebar", id: "sidebar" },
h( h(
"div", "div",
{ class: "brand" }, { class: "brand" },
h("img", { src: FAVICON, alt: "" }), h("img", { src: FAVICON, alt: "" }),
h("div", { class: "brand-copy" }, h("b", {}, "allaigate"), h("div", { class: "sub" }, t("brand.sub"))), h("div", {}, h("b", {}, "allaigate"), h("div", { class: "sub" }, t("brand.sub")))
h("button", { class: "icon-btn nav-close", type: "button", title: "Close menu", "aria-label": "Close menu", onClick: () => setMobileNav(false) }, "×")
), ),
...ROUTES.map((r) => ...ROUTES.map((r) =>
h( h(
"a", "a",
{ class: "nav-item" + (r.id === activeId ? " active" : ""), href: "#/" + r.id, onClick: () => setMobileNav(false) }, { class: "nav-item" + (r.id === activeId ? " active" : ""), href: "#/" + r.id },
h("span", { class: "ico" }, r.icon), h("span", { class: "ico" }, r.icon),
t("nav." + r.id) t("nav." + r.id)
) )
@@ -97,7 +85,7 @@ function statusPills() {
: t("status.checking"); : t("status.checking");
return h( return h(
"div", "div",
{ class: "flex status-pills" }, { class: "flex" },
h( h(
"span", "span",
{ class: "pill" }, { class: "pill" },
@@ -148,19 +136,6 @@ function topbar(activeId) {
return h( return h(
"header", "header",
{ class: "topbar" }, { class: "topbar" },
h(
"button",
{
class: "icon-btn menu-toggle",
type: "button",
title: "Menu",
"aria-label": "Open menu",
"aria-controls": "sidebar",
"aria-expanded": String(mobileNavOpen),
onClick: () => setMobileNav(!mobileNavOpen),
},
h("span", { class: "hamburger", "aria-hidden": "true" }, h("i"), h("i"), h("i"))
),
h("div", { class: "title" }, t("nav." + activeId)), h("div", { class: "title" }, t("nav." + activeId)),
h( h(
"div", "div",
@@ -200,15 +175,12 @@ async function renderView() {
function renderShell() { function renderShell() {
const route = currentRoute(); const route = currentRoute();
mobileNavOpen = false;
document.body.classList.remove("mobile-nav-open");
mount( mount(
document.getElementById("app"), document.getElementById("app"),
h( h(
"div", "div",
{ class: "app" }, { class: "app" },
sidebar(route.id), sidebar(route.id),
h("button", { class: "mobile-nav-backdrop", type: "button", tabindex: "-1", "aria-label": "Close menu", onClick: () => setMobileNav(false) }),
h("main", { class: "main" }, topbar(route.id), h("div", { class: "content", id: "view" })) h("main", { class: "main" }, topbar(route.id), h("div", { class: "content", id: "view" }))
) )
); );
@@ -271,12 +243,6 @@ onLangChange(() => {
window.addEventListener("hashchange", () => { window.addEventListener("hashchange", () => {
if (document.querySelector(".app")) renderShell(); if (document.querySelector(".app")) renderShell();
}); });
window.addEventListener("keydown", (e) => {
if (e.key === "Escape" && mobileNavOpen) setMobileNav(false);
});
window.addEventListener("resize", () => {
if (window.innerWidth > 980 && mobileNavOpen) setMobileNav(false);
});
setInterval(() => { setInterval(() => {
if (document.querySelector(".app")) refreshHealth(); if (document.querySelector(".app")) refreshHealth();
}, 20000); }, 20000);
+1 -59
View File
@@ -59,9 +59,6 @@ export default {
// dashboard // dashboard
"dash.title": "Dashboard", "dash.title": "Dashboard",
"dash.subtitle": "Live traffic, cost and routing analytics for your model pool.", "dash.subtitle": "Live traffic, cost and routing analytics for your model pool.",
"dash.clear": "Clear logs",
"dash.clearConfirm": "Clear all recorded stats and logs? This cannot be undone.",
"dash.cleared": "Logs cleared",
"dash.range": "Range", "dash.range": "Range",
"dash.range.1h": "1h", "dash.range.1h": "1h",
"dash.range.24h": "24h", "dash.range.24h": "24h",
@@ -132,32 +129,6 @@ export default {
"models.form.secretMissing": "not set", "models.form.secretMissing": "not set",
"models.form.secretPlaceholder": "set or replace key (write-only)", "models.form.secretPlaceholder": "set or replace key (write-only)",
"models.deleteConfirm": "Delete model \"{id}\"?", "models.deleteConfirm": "Delete model \"{id}\"?",
"models.form.idOrModel": "Enter an id or a model name",
"models.form.apiKey": "API Key",
"models.form.apiKeyPlaceholder": "paste your API key",
"models.form.apiKeyNotNeeded": "not required (local server)",
"models.form.apiKeyHint": "Stored securely on the gateway; it is not shown again.",
"models.form.keyStored": "A key is saved — leave blank to keep it, or paste a new one.",
"models.form.keyFromEnv": "Using a key from an environment variable.",
"models.form.paramsTitle": "Generation & Thinking Parameters (O(1), MTP, Temperature)",
"models.form.temperature": "Temperature",
"models.form.temperatureHint": "Generation temperature (0.0 = deterministic, 1.0+ = creative).",
"models.form.topP": "Top-P (Nucleus Sampling)",
"models.form.topPHint": "Sampling threshold (0.0 .. 1.0).",
"models.form.maxTokens": "Max Tokens",
"models.form.maxTokensHint": "Maximum completion tokens to generate.",
"models.form.thinkBudget": "Thinking Mode / Think Budget",
"models.form.thinkBudgetHint": "Reasoning token budget (0 = disabled, N > 0 = max reasoning tokens).",
"models.form.systemPrompt": "Default System Prompt",
"models.form.systemPromptHint": "Prepend automatically if request lacks a system prompt.",
"models.form.systemPromptPlaceholder": "You are a helpful assistant...",
"models.form.testKey": "Test & load models",
"models.form.testing": "checking…",
"models.form.foundModels": "{n} models loaded",
"models.form.modelPlaceholder": "type to search…",
"models.form.modelHint": "Click “Test & load models” to pull the list from your key, then pick one.",
"models.managed": "managed",
"models.managedHint": "Provided by the managed CMF server (Settings → Local models). Configure it there.",
// routing // routing
"routing.title": "Routing", "routing.title": "Routing",
@@ -271,11 +242,6 @@ export default {
"settings.cmf.threads": "Worker threads", "settings.cmf.threads": "Worker threads",
"settings.cmf.gpu": "Use GPU (Metal / wgpu)", "settings.cmf.gpu": "Use GPU (Metal / wgpu)",
"settings.cmf.speedNote": "Threads parallelise the decode matvecs (~3x faster; 0 = runtime default). GPU offload mainly helps prefill of long prompts — decode is memory-bandwidth-bound, so on Apple Silicon it is roughly neutral.", "settings.cmf.speedNote": "Threads parallelise the decode matvecs (~3x faster; 0 = runtime default). GPU offload mainly helps prefill of long prompts — decode is memory-bandwidth-bound, so on Apple Silicon it is roughly neutral.",
"settings.cmf.pick": "select a model",
"settings.cmf.models": "Local models",
"settings.cmf.modelsHelp": "Each row is one model the gateway runs (its own cortiq serve on its own port) and registers in the pool. Add as many as you have .cmf files and free ports.",
"settings.cmf.rowHint": "id · .cmf file · port · threads · GPU. Changes apply on restart; deleting a row stops that server.",
"settings.cmf.addServer": "+ Add local model",
"settings.cmf.checkPort": "Check port", "settings.cmf.checkPort": "Check port",
"settings.cmf.portFree": "port is free", "settings.cmf.portFree": "port is free",
"settings.cmf.portBusy": "port is in use", "settings.cmf.portBusy": "port is in use",
@@ -331,10 +297,6 @@ export default {
"settings.route.maxChars": "Max chars", "settings.route.maxChars": "Max chars",
"settings.route.cacheTtl": "Cache TTL", "settings.route.cacheTtl": "Cache TTL",
"settings.route.profile": "Default profile", "settings.route.profile": "Default profile",
"settings.route.strategyHelp": "What text is sent to the router for classification: last_user = only the last user message; last_user_plus_system = last user message + system prompt; concat_all = the whole conversation.",
"settings.route.maxCharsHelp": "Max characters of that text sent to the router (longer is truncated). More = more accurate classification, but slower/pricier router calls.",
"settings.route.cacheTtlHelp": "How long a routing decision is cached for an identical request before the router is asked again (e.g. 60s, 5m).",
"settings.route.profileHelp": "Default routing profile when a request doesn't specify one: cost-saver (cheapest), balanced, or quality-first (best model). Biases which model the router picks.",
"settings.breaker": "Circuit breaker", "settings.breaker": "Circuit breaker",
"settings.breaker.threshold": "Error threshold", "settings.breaker.threshold": "Error threshold",
"settings.breaker.cooldown": "Cooldown", "settings.breaker.cooldown": "Cooldown",
@@ -386,7 +348,7 @@ export default {
"import.failed": "Conversion failed — see the log.", "import.failed": "Conversion failed — see the log.",
"import.register": "Register as local model", "import.register": "Register as local model",
"import.registered": "Registered as '{id}'. Run cortiq-server to serve it.", "import.registered": "Registered as '{id}'. Run cortiq-server to serve it.",
"import.jobs": "Downloads & Conversions", "import.jobs": "Conversions",
"import.cancel": "Cancel", "import.cancel": "Cancel",
"import.cancelling": "Cancelling — cleaning up partial output…", "import.cancelling": "Cancelling — cleaning up partial output…",
"import.delete": "Delete", "import.delete": "Delete",
@@ -396,21 +358,12 @@ export default {
"import.state.done": "done", "import.state.done": "done",
"import.state.error": "error", "import.state.error": "error",
"import.state.cancelled": "cancelled", "import.state.cancelled": "cancelled",
"import.featuredTitle": "Featured Ready-to-Run .cmf Models",
"import.readyCmfBadge": "Ready .cmf",
"import.directDownload": "Download",
"import.q.q8_2f": "8-bit two-field (𝒲×θ) — recommended: best quality/size.", "import.q.q8_2f": "8-bit two-field (𝒲×θ) — recommended: best quality/size.",
"import.q.q8_row": "8-bit per-row — simple, robust.", "import.q.q8_row": "8-bit per-row — simple, robust.",
"import.q.q4": "4-bit block — smallest, lower quality.", "import.q.q4": "4-bit block — smallest, lower quality.",
"import.q.q4_tiled": "4-bit tiled (Q4_TILED) — 1.66x faster on Metal GPU.",
"import.q.vbit": "Variable 3–8 bit — per-expert bit budgets.", "import.q.vbit": "Variable 3–8 bit — per-expert bit budgets.",
"import.q.q1t": "1.58-bit ternary (Q1T) — BitNet b1.58 with sparse outliers.",
"import.q.q1s": "1-bit sparse (Q1S) — 1-bit base with holographic outliers.",
"import.q.q1p": "1-bit PTQ (Q1P) — 1-bit with error diffusion.",
"import.q.q1": "1-bit (Q1) — pure binary format.",
"import.q.f16": "16-bit — no quantization.", "import.q.f16": "16-bit — no quantization.",
"import.q.f32": "32-bit — full precision (large file).", "import.q.f32": "32-bit — full precision (large file).",
"import.adv.title": "Advanced settings", "import.adv.title": "Advanced settings",
"import.adv.auto": "auto", "import.adv.auto": "auto",
"import.adv.linearCore": "Linear core", "import.adv.linearCore": "Linear core",
@@ -421,15 +374,4 @@ export default {
"import.adv.shard": "Shard max (GB)", "import.adv.shard": "Shard max (GB)",
"import.adv.noShard": "no sharding", "import.adv.noShard": "no sharding",
"import.adv.skipMtp": "skip MTP head", "import.adv.skipMtp": "skip MTP head",
"import.adv.o1": "O(1) attention",
"import.adv.o1Hint": "Constant-memory streaming attention: the KV cache is released after prefill, so long contexts don't grow memory. Weights stay byte-identical — only a load-time hint is recorded; you can switch back anytime (serve with --o1 off). Validated on Qwen3-0.6B: all layers ≈ ×1.18 perplexity.",
"import.adv.o1Off": "off — exact attention (default)",
"import.adv.o1All": "all — convert every softmax layer",
"import.adv.o1Deep": "deep N — only the N deepest layers",
"import.adv.o1Custom": "custom — explicit layer list",
"import.adv.o1DeepN": "N deep layers",
"import.adv.o1Layers": "Layers (comma-separated)",
"import.adv.o1mHint": "landmark budget (default 32)",
"import.adv.o1windowHint": "exact-window width (default 128)",
"import.adv.o1sinkHint": "permanent exact sink keys (default 4)",
}; };
+1 -59
View File
@@ -58,9 +58,6 @@ export default {
// dashboard // dashboard
"dash.title": "Дашборд", "dash.title": "Дашборд",
"dash.subtitle": "Трафик в реальном времени, расходы и аналитика роутинга по пулу моделей.", "dash.subtitle": "Трафик в реальном времени, расходы и аналитика роутинга по пулу моделей.",
"dash.clear": "Очистить логи",
"dash.clearConfirm": "Очистить всю статистику и логи? Отменить нельзя.",
"dash.cleared": "Логи очищены",
"dash.range": "Период", "dash.range": "Период",
"dash.range.1h": "1ч", "dash.range.1h": "1ч",
"dash.range.24h": "24ч", "dash.range.24h": "24ч",
@@ -131,32 +128,6 @@ export default {
"models.form.secretMissing": "не задан", "models.form.secretMissing": "не задан",
"models.form.secretPlaceholder": "задать или заменить ключ (только запись)", "models.form.secretPlaceholder": "задать или заменить ключ (только запись)",
"models.deleteConfirm": "Удалить модель \"{id}\"?", "models.deleteConfirm": "Удалить модель \"{id}\"?",
"models.form.idOrModel": "Укажите id или имя модели",
"models.form.apiKey": "API-ключ",
"models.form.apiKeyPlaceholder": "вставьте API-ключ",
"models.form.apiKeyNotNeeded": "не нужен (локальный сервер)",
"models.form.apiKeyHint": "Хранится безопасно на шлюзе; больше не показывается.",
"models.form.keyStored": "Ключ сохранён — оставьте пустым, чтобы не менять, или вставьте новый.",
"models.form.keyFromEnv": "Используется ключ из переменной окружения.",
"models.form.paramsTitle": "Параметры генерации, O(1) и режим размышления",
"models.form.temperature": "Температура",
"models.form.temperatureHint": "Температура генерации (0.0 — точные факты, 1.0+ — творчество).",
"models.form.topP": "Top-P (Nucleus Sampling)",
"models.form.topPHint": "Порог сэмплирования токенов (0.0 .. 1.0).",
"models.form.maxTokens": "Макс. токенов (Max Tokens)",
"models.form.maxTokensHint": "Максимальное количество генерируемых токенов.",
"models.form.thinkBudget": "Режим размышления / Think Budget",
"models.form.thinkBudgetHint": "Бюджет токенов мыслей (0 = выключен, N > 0 = лимит рассуждений).",
"models.form.systemPrompt": "Системный промпт по умолчанию",
"models.form.systemPromptHint": "Автоматически подставляется, если запрос не содержит системное сообщение.",
"models.form.systemPromptPlaceholder": "Вы — полезный ассистент...",
"models.form.testKey": "Проверить и загрузить модели",
"models.form.testing": "проверка…",
"models.form.foundModels": "загружено моделей: {n}",
"models.form.modelPlaceholder": "введите для поиска…",
"models.form.modelHint": "Нажмите «Проверить и загрузить модели», чтобы подтянуть список по ключу, затем выберите.",
"models.managed": "управляемая",
"models.managedHint": "Предоставляется CMF-сервером (Настройки → Локальные модели). Настраивается там.",
// routing // routing
"routing.title": "Роутинг", "routing.title": "Роутинг",
@@ -270,11 +241,6 @@ export default {
"settings.cmf.threads": "Рабочие потоки", "settings.cmf.threads": "Рабочие потоки",
"settings.cmf.gpu": "Использовать GPU (Metal / wgpu)", "settings.cmf.gpu": "Использовать GPU (Metal / wgpu)",
"settings.cmf.speedNote": "Потоки распараллеливают декод-матвеки (~×3; 0 = по умолчанию рантайма). GPU-оффлоад помогает в основном на prefill длинных промптов — декод упирается в пропускную способность памяти, поэтому на Apple Silicon почти нейтрален.", "settings.cmf.speedNote": "Потоки распараллеливают декод-матвеки (~×3; 0 = по умолчанию рантайма). GPU-оффлоад помогает в основном на prefill длинных промптов — декод упирается в пропускную способность памяти, поэтому на Apple Silicon почти нейтрален.",
"settings.cmf.pick": "выберите модель",
"settings.cmf.models": "Локальные модели",
"settings.cmf.modelsHelp": "Каждая строка — отдельная модель: шлюз поднимает свой cortiq serve на своём порту и регистрирует её в пуле. Добавляйте столько, сколько есть .cmf-файлов и свободных портов.",
"settings.cmf.rowHint": "id · файл .cmf · порт · потоки · GPU. Изменения применяются после перезапуска; удаление строки останавливает сервер.",
"settings.cmf.addServer": "+ Добавить модель",
"settings.cmf.checkPort": "Проверить порт", "settings.cmf.checkPort": "Проверить порт",
"settings.cmf.portFree": "порт свободен", "settings.cmf.portFree": "порт свободен",
"settings.cmf.portBusy": "порт занят", "settings.cmf.portBusy": "порт занят",
@@ -330,10 +296,6 @@ export default {
"settings.route.maxChars": "Макс. символов", "settings.route.maxChars": "Макс. символов",
"settings.route.cacheTtl": "TTL кэша", "settings.route.cacheTtl": "TTL кэша",
"settings.route.profile": "Профиль по умолчанию", "settings.route.profile": "Профиль по умолчанию",
"settings.route.strategyHelp": "Какой текст отправляется роутеру для классификации: last_user — только последнее сообщение пользователя; last_user_plus_system — последнее сообщение + системный промпт; concat_all — вся переписка.",
"settings.route.maxCharsHelp": "Максимум символов этого текста, отправляемого роутеру (длиннее — обрезается). Больше = точнее классификация, но медленнее/дороже вызовы роутера.",
"settings.route.cacheTtlHelp": "Сколько кэшируется решение роутинга для одинакового запроса, прежде чем роутер спрашивается снова (напр. 60s, 5m).",
"settings.route.profileHelp": "Профиль роутинга по умолчанию, если запрос его не задаёт: cost-saver (дешевле), balanced (баланс) или quality-first (качество). Влияет на выбор модели роутером.",
"settings.breaker": "Автоматический выключатель", "settings.breaker": "Автоматический выключатель",
"settings.breaker.threshold": "Порог ошибок", "settings.breaker.threshold": "Порог ошибок",
"settings.breaker.cooldown": "Время восстановления", "settings.breaker.cooldown": "Время восстановления",
@@ -385,7 +347,7 @@ export default {
"import.failed": "Конвертация не удалась — смотрите лог.", "import.failed": "Конвертация не удалась — смотрите лог.",
"import.register": "Подключить как локальную модель", "import.register": "Подключить как локальную модель",
"import.registered": "Подключено как '{id}'. Запустите cortiq-server для обслуживания.", "import.registered": "Подключено как '{id}'. Запустите cortiq-server для обслуживания.",
"import.jobs": "Загрузки / Конвертации", "import.jobs": "Конвертации",
"import.cancel": "Отменить", "import.cancel": "Отменить",
"import.cancelling": "Отмена — удаляю недописанный файл…", "import.cancelling": "Отмена — удаляю недописанный файл…",
"import.delete": "Удалить", "import.delete": "Удалить",
@@ -395,21 +357,12 @@ export default {
"import.state.done": "готово", "import.state.done": "готово",
"import.state.error": "ошибка", "import.state.error": "ошибка",
"import.state.cancelled": "отменено", "import.state.cancelled": "отменено",
"import.featuredTitle": "Рекомендуемые готовые .cmf модели",
"import.readyCmfBadge": "Готовая .cmf",
"import.directDownload": "Скачать",
"import.q.q8_2f": "8-бит двухполевой (𝒲×θ) — рекомендуется: лучшее качество/размер.", "import.q.q8_2f": "8-бит двухполевой (𝒲×θ) — рекомендуется: лучшее качество/размер.",
"import.q.q8_row": "8-бит построчно — просто, надёжно.", "import.q.q8_row": "8-бит построчно — просто, надёжно.",
"import.q.q4": "4-бит блочно — компактнее, ниже качество.", "import.q.q4": "4-бит блочно — компактнее, ниже качество.",
"import.q.q4_tiled": "4-бит тайловый (Q4_TILED) — х1.66 быстрее на Metal GPU.",
"import.q.vbit": "Переменная 3–8 бит — per-expert бюджеты.", "import.q.vbit": "Переменная 3–8 бит — per-expert бюджеты.",
"import.q.q1t": "1.58-бит тернарный (Q1T) — BitNet b1.58 с оутлайерами.",
"import.q.q1s": "1-бит спарс (Q1S) — 1-бит база с голографическими оутлайерами.",
"import.q.q1p": "1-бит PTQ (Q1P) — 1-бит с переносом ошибки (error diffusion).",
"import.q.q1": "1-бит (Q1) — чистый бинарный формат.",
"import.q.f16": "16-бит — без квантования.", "import.q.f16": "16-бит — без квантования.",
"import.q.f32": "32-бит — полная точность (большой файл).", "import.q.f32": "32-бит — полная точность (большой файл).",
"import.adv.title": "Расширенные настройки", "import.adv.title": "Расширенные настройки",
"import.adv.auto": "авто", "import.adv.auto": "авто",
"import.adv.linearCore": "Линейное ядро", "import.adv.linearCore": "Линейное ядро",
@@ -420,15 +373,4 @@ export default {
"import.adv.shard": "Шард макс (ГБ)", "import.adv.shard": "Шард макс (ГБ)",
"import.adv.noShard": "без шардинга", "import.adv.noShard": "без шардинга",
"import.adv.skipMtp": "пропустить MTP-голову", "import.adv.skipMtp": "пропустить MTP-голову",
"import.adv.o1": "O(1)-внимание",
"import.adv.o1Hint": "Потоковое внимание с постоянной памятью: KV-кэш освобождается после prefill, длинный контекст не раздувает память. Веса не меняются — записывается лишь подсказка на загрузку; вернуться можно всегда (запуск с --o1 off). Проверено на Qwen3-0.6B: все слои ≈ ×1.18 перплексии.",
"import.adv.o1Off": "выкл — точное внимание (по умолчанию)",
"import.adv.o1All": "all — конвертировать все softmax-слои",
"import.adv.o1Deep": "deep N — только N самых глубоких слоёв",
"import.adv.o1Custom": "вручную — явный список слоёв",
"import.adv.o1DeepN": "N глубоких слоёв",
"import.adv.o1Layers": "Слои (через запятую)",
"import.adv.o1mHint": "число опорных точек (по умолчанию 32)",
"import.adv.o1windowHint": "ширина точного окна (по умолчанию 128)",
"import.adv.o1sinkHint": "постоянные точные sink-ключи (по умолчанию 4)",
}; };
+2 -17
View File
@@ -23,15 +23,9 @@ export function clear(node) {
while (node && node.firstChild) node.removeChild(node.firstChild); while (node && node.firstChild) node.removeChild(node.firstChild);
} }
export function opt(v, label, sel) {
return h("option", { value: v, selected: sel ? true : null }, label || v);
}
export function mount(node, ...children) { export function mount(node, ...children) {
clear(node); clear(node);
for (const c of children.flat()) for (const c of children.flat()) if (c != null && c !== false) node.appendChild(c);
if (c != null && c !== false)
node.appendChild(typeof c === "object" ? c : document.createTextNode(String(c)));
return node; return node;
} }
@@ -50,14 +44,7 @@ export function toast(msg, kind = "info", ms = 3200) {
// Modal dialog. `body` is a DOM node; `onSave` returns a Promise; returns nothing. // Modal dialog. `body` is a DOM node; `onSave` returns a Promise; returns nothing.
export function modal(title, body, onSave, saveLabel) { export function modal(title, body, onSave, saveLabel) {
const backdrop = h("div", { class: "modal-backdrop" }); const backdrop = h("div", { class: "modal-backdrop" });
const onKeydown = (e) => { const close = () => backdrop.remove();
if (e.key === "Escape") close();
};
const close = () => {
backdrop.remove();
document.body.classList.remove("modal-open");
document.removeEventListener("keydown", onKeydown);
};
backdrop.addEventListener("click", (e) => { backdrop.addEventListener("click", (e) => {
if (e.target === backdrop) close(); if (e.target === backdrop) close();
}); });
@@ -84,8 +71,6 @@ export function modal(title, body, onSave, saveLabel) {
) )
); );
backdrop.appendChild(card); backdrop.appendChild(card);
document.body.classList.add("modal-open");
document.addEventListener("keydown", onKeydown);
document.body.appendChild(backdrop); document.body.appendChild(backdrop);
return { close }; return { close };
} }
+5 -19
View File
@@ -1,5 +1,5 @@
// Dashboard — KPIs, time-series charts, breakdown, recent requests, health. // Dashboard — KPIs, time-series charts, breakdown, recent requests, health.
import { h, mount, money, num, ms, pct, timeAgo, toast, confirmDialog } from "../ui.js"; import { h, mount, money, num, ms, pct, timeAgo } from "../ui.js";
import { t } from "../i18n.js"; import { t } from "../i18n.js";
import { api } from "../api.js"; import { api } from "../api.js";
import { lineChart, barList } from "../charts.js"; import { lineChart, barList } from "../charts.js";
@@ -53,24 +53,10 @@ export async function renderDashboard() {
"div", "div",
{ class: "flex" }, { class: "flex" },
h("div", { class: "grow" }, h("h2", {}, t("dash.title")), h("p", {}, t("dash.subtitle"))), h("div", { class: "grow" }, h("h2", {}, t("dash.title")), h("p", {}, t("dash.subtitle"))),
h("div", { class: "flex", style: "gap:8px;align-items:center" }, seg(RANGES, range, (r) => {
seg(RANGES, range, (r) => { range = r;
range = r; load();
load(); }, (r) => t("dash.range." + r))
}, (r) => t("dash.range." + r)),
h("button", {
class: "btn sm danger",
onClick: async () => {
if (!(await confirmDialog(t("dash.clearConfirm")))) return;
try {
await api.clearStats();
toast(t("dash.cleared"), "good");
await load();
} catch (e) {
toast(String(e.message || e), "bad");
}
},
}, t("dash.clear")))
) )
), ),
+4 -97
View File
@@ -2,7 +2,7 @@
// live progress view. Search HF, pick quantization + advanced params, watch // live progress view. Search HF, pick quantization + advanced params, watch
// the conversion (multiple in parallel, resumable across tab switches), then // the conversion (multiple in parallel, resumable across tab switches), then
// register the result as a local model. // register the result as a local model.
import { h, mount, toast, opt } from "../ui.js"; import { h, mount, toast } from "../ui.js";
import { t } from "../i18n.js"; import { t } from "../i18n.js";
import { api } from "../api.js"; import { api } from "../api.js";
@@ -10,27 +10,18 @@ const QUANTS = [
["Q8_2F", "import.q.q8_2f"], ["Q8_2F", "import.q.q8_2f"],
["Q8_ROW", "import.q.q8_row"], ["Q8_ROW", "import.q.q8_row"],
["Q4_BLOCK", "import.q.q4"], ["Q4_BLOCK", "import.q.q4"],
["Q4_TILED", "import.q.q4_tiled"],
["VBIT", "import.q.vbit"], ["VBIT", "import.q.vbit"],
["Q1T", "import.q.q1t"],
["Q1S", "import.q.q1s"],
["Q1P", "import.q.q1p"],
["Q1", "import.q.q1"],
["F16", "import.q.f16"], ["F16", "import.q.f16"],
["F32", "import.q.f32"], ["F32", "import.q.f32"],
]; ];
const FEATURED_REPOS = [
{ id: "infosave/Bonsai-1.7Bcmf", label: "Bonsai 1.7B CMF", badge: "1.7B · Q1", quant: "Q1" },
{ id: "infosave/Bonsai-8B_2bit_cmf", label: "Bonsai 8B 2-bit CMF", badge: "8B · Q1T", quant: "Q1T" },
{ id: "infosave/Bonsai-27Bcmf", label: "Bonsai 27B CMF", badge: "27B · Q1", quant: "Q1" },
];
function field(label, control, hint) { function field(label, control, hint) {
return h("label", { class: "field" }, h("span", {}, label), control, return h("label", { class: "field" }, h("span", {}, label), control,
hint ? h("div", { class: "hint" }, hint) : null); hint ? h("div", { class: "hint" }, hint) : null);
} }
function opt(v, label, sel) {
return h("option", { value: v, selected: sel ? true : null }, label || v);
}
function fmtNum(n) { function fmtNum(n) {
if (n == null) return "—"; if (n == null) return "—";
if (n >= 1e6) return (n / 1e6).toFixed(1) + "M"; if (n >= 1e6) return (n / 1e6).toFixed(1) + "M";
@@ -134,32 +125,6 @@ export function renderImport() {
const meanBitsIn = h("input", { type: "number", step: "0.1", placeholder: "5.0" }); const meanBitsIn = h("input", { type: "number", step: "0.1", placeholder: "5.0" });
const shardIn = h("input", { type: "number", step: "0.5", placeholder: t("import.adv.noShard") }); const shardIn = h("input", { type: "number", step: "0.5", placeholder: t("import.adv.noShard") });
const skipMtp = h("input", { type: "checkbox" }); const skipMtp = h("input", { type: "checkbox" });
// O(1) attention (cortiq ≥ 0.2.0): constant-memory streaming attention hint,
// weights byte-identical — the runtime applies it at load.
const o1Sel = h("select", {},
opt("", t("import.adv.o1Off")),
opt("all", t("import.adv.o1All")),
opt("deep", t("import.adv.o1Deep")),
opt("custom", t("import.adv.o1Custom")));
const o1DeepIn = h("input", { type: "number", min: "1", placeholder: "12" });
const o1CustomIn = h("input", { placeholder: "0,4,8" });
const o1M = h("input", { type: "number", min: "1", placeholder: "32" });
const o1Window = h("input", { type: "number", min: "0", placeholder: "128" });
const o1Sink = h("input", { type: "number", min: "0", placeholder: "4" });
const o1SpecRow = h("div", { class: "row", style: "display:none" },
field(t("import.adv.o1DeepN"), o1DeepIn),
field(t("import.adv.o1Layers"), o1CustomIn));
const o1Knobs = h("div", { class: "row", style: "display:none" },
field("--o1-m", o1M, t("import.adv.o1mHint")),
field("--o1-window", o1Window, t("import.adv.o1windowHint")),
field("--o1-sink", o1Sink, t("import.adv.o1sinkHint")));
o1Sel.addEventListener("change", () => {
const on = !!o1Sel.value;
o1Knobs.style.display = on ? "" : "none";
o1SpecRow.style.display = o1Sel.value === "deep" || o1Sel.value === "custom" ? "" : "none";
o1DeepIn.parentElement.style.display = o1Sel.value === "deep" ? "" : "none";
o1CustomIn.parentElement.style.display = o1Sel.value === "custom" ? "" : "none";
});
const advBody = h( const advBody = h(
"div", "div",
{ class: "adv-body", style: "display:none" }, { class: "adv-body", style: "display:none" },
@@ -172,9 +137,6 @@ export function renderImport() {
h("div", { class: "row" }, h("div", { class: "row" },
field(t("import.adv.shard"), shardIn), field(t("import.adv.shard"), shardIn),
field("MTP", h("label", { class: "check" }, skipMtp, t("import.adv.skipMtp")))), field("MTP", h("label", { class: "check" }, skipMtp, t("import.adv.skipMtp")))),
field(t("import.adv.o1"), o1Sel, t("import.adv.o1Hint")),
o1SpecRow,
o1Knobs,
); );
const advToggle = h("button", { class: "btn ghost sm" }, "⚙ " + t("import.adv.title")); const advToggle = h("button", { class: "btn ghost sm" }, "⚙ " + t("import.adv.title"));
advToggle.addEventListener("click", () => { advToggle.addEventListener("click", () => {
@@ -194,19 +156,6 @@ export function renderImport() {
if (vbitSel.value) params.vbit_shape = vbitSel.value; if (vbitSel.value) params.vbit_shape = vbitSel.value;
if (meanBitsIn.value) params.mean_bits = parseFloat(meanBitsIn.value); if (meanBitsIn.value) params.mean_bits = parseFloat(meanBitsIn.value);
if (shardIn.value) params.shard_max_gb = parseFloat(shardIn.value); if (shardIn.value) params.shard_max_gb = parseFloat(shardIn.value);
if (o1Sel.value) {
// spec: all | deepN | explicit layer list
params.o1 =
o1Sel.value === "deep" ? "deep" + (parseInt(o1DeepIn.value) || 12)
: o1Sel.value === "custom" ? o1CustomIn.value.trim()
: "all";
if (!params.o1) delete params.o1;
else {
if (o1M.value) params.o1_m = parseInt(o1M.value);
if (o1Window.value) params.o1_window = parseInt(o1Window.value);
if (o1Sink.value) params.o1_sink = parseInt(o1Sink.value);
}
}
goBtn.disabled = true; goBtn.disabled = true;
try { try {
await api.startImport(params); await api.startImport(params);
@@ -367,46 +316,6 @@ export function renderImport() {
); );
} }
function featuredCard(item) {
const card = h(
"div",
{ class: "hf-card featured-card" },
h("div", { class: "row", style: "align-items:center; justify-content:space-between;" },
h("div", {},
h("div", { class: "hf-id" }, item.label),
h("div", { class: "hf-meta" },
h("span", { class: "badge" }, item.badge),
h("span", { class: "muted" }, item.id)
)
),
h("button", { class: "btn primary sm" }, "⬇ " + t("import.directDownload"))
)
);
card.querySelector("button").addEventListener("click", async (e) => {
e.stopPropagation();
try {
await api.startImport({
repo: item.id,
quant: item.quant || "READY_CMF",
name: item.id.split("/").pop().toLowerCase(),
});
toast(t("import.started"), "ok");
refreshJobs();
jobsHost.scrollIntoView({ behavior: "smooth", block: "nearest" });
} catch (err) {
toast(err.message, "err");
}
});
return card;
}
const featuredHost = h(
"div",
{ class: "card featured-section" },
h("div", { class: "card-title" }, "⚡ " + t("import.featuredTitle")),
h("div", { class: "hf-grid" }, ...FEATURED_REPOS.map(featuredCard))
);
const root = h( const root = h(
"div", "div",
{}, {},
@@ -414,7 +323,6 @@ export function renderImport() {
h("h2", {}, t("nav.import")), h("h2", {}, t("nav.import")),
h("div", { class: "muted" }, t("import.subtitle"))), h("div", { class: "muted" }, t("import.subtitle"))),
jobsHost, // progress panel on top, above the search jobsHost, // progress panel on top, above the search
featuredHost, // featured ready-to-run models
h("div", { class: "card" }, searchIn, results), h("div", { class: "card" }, searchIn, results),
formHost formHost
); );
@@ -422,4 +330,3 @@ export function renderImport() {
refreshJobs(); // resume: show any running/recent conversions immediately refreshJobs(); // resume: show any running/recent conversions immediately
return root; return root;
} }
+37 -211
View File
@@ -1,5 +1,5 @@
// Models — pool table with add/edit modal, probe, and per-model secret entry. // Models — pool table with add/edit modal, probe, and per-model secret entry.
import { h, mount, modal, toast, confirmDialog, money, opt } from "../ui.js"; import { h, mount, modal, toast, confirmDialog, money } from "../ui.js";
import { t } from "../i18n.js"; import { t } from "../i18n.js";
import { api } from "../api.js"; import { api } from "../api.js";
import { appState } from "../app.js"; import { appState } from "../app.js";
@@ -7,232 +7,61 @@ import { appState } from "../app.js";
function field(label, control, hint) { function field(label, control, hint) {
return h("label", { class: "field" }, h("span", {}, label), control, hint ? h("div", { class: "hint" }, hint) : null); return h("label", { class: "field" }, h("span", {}, label), control, hint ? h("div", { class: "hint" }, hint) : null);
} }
function opt(v, label, sel) {
return h("option", { value: v, selected: sel ? true : null }, label || v);
}
function keyBadge(src) { function keyBadge(src) {
const map = { store: "store", env: "env", missing: "bad", none: "" }; const map = { store: "store", env: "env", missing: "bad", none: "" };
return h("span", { class: "badge " + (map[src] ?? "") }, src); return h("span", { class: "badge " + (map[src] ?? "") }, src);
} }
// Verified per-provider defaults (base URL, key env, whether a key is needed).
const PROVIDER_DEFAULTS = {
openai: { base: "https://api.openai.com/v1", keyEnv: "OPENAI_API_KEY", needsKey: true, tier: "mid", caps: ["tools", "vision"] },
anthropic: { base: "https://api.anthropic.com", keyEnv: "ANTHROPIC_API_KEY", needsKey: true, tier: "expensive", caps: ["tools", "vision"] },
openrouter: { base: "https://openrouter.ai/api/v1", keyEnv: "OPENROUTER_API_KEY", needsKey: true, tier: "mid", caps: ["tools", "vision"] },
lmstudio: { base: "http://localhost:1234/v1", keyEnv: "", needsKey: false, tier: "local", caps: ["tools", "vision"] },
ollama: { base: "http://localhost:11434/v1", keyEnv: "", needsKey: false, tier: "local", caps: ["tools"] },
http: { base: "", keyEnv: "", needsKey: false, tier: "cheap", caps: ["tools"] },
};
// Derive a clean model id from a model name (last path segment, slugified).
// e.g. "qwen/qwen3.7-max" → "qwen3-7-max".
function slugId(name) {
const base = (name || "").split("/").pop() || "";
return base.toLowerCase().replace(/[^a-z0-9]+/g, "-").replace(/^-+|-+$/g, "");
}
function modelForm(meta, existing) { function modelForm(meta, existing) {
const m = existing || { provider: "openai", kind: "chat", cost_tier: "mid", caps: ["tools"], price_in: 0, price_out: 0 }; const m = existing || { provider: "openai", kind: "chat", cost_tier: "cheap", caps: [], price_in: 0, price_out: 0 };
const idIn = h("input", { value: m.id || "", placeholder: "my-model", disabled: existing ? true : null }); const idIn = h("input", { value: m.id || "", placeholder: "local-qwen", disabled: existing ? true : null });
const providerSel = h("select", {}, ...(meta.providers || ["openai"]).map((p) => opt(p, p, p === m.provider))); const providerSel = h("select", {}, ...(meta.providers || ["openai"]).map((p) => opt(p, p, p === m.provider)));
const baseUrlIn = h("input", { value: m.base_url || "", placeholder: "https://api.openai.com/v1" }); const baseUrlIn = h("input", { value: m.base_url || "", placeholder: "http://localhost:8000/v1" });
const modelIn = h("input", { value: m.model || "", placeholder: "qwen2.5-7b-instruct" });
// API key — typed directly, stored securely as a secret under a provider env name.
const keyIn = h("input", { type: "password", placeholder: t("models.form.apiKeyPlaceholder"), autocomplete: "new-password" });
const testBtn = h("button", { class: "btn small", type: "button" }, t("models.form.testKey"));
const keyStatus = h("span", { class: "small", style: "margin-left:8px" });
// Model — below the key (some providers list models from the key); searchable datalist.
const dlId = "model-opts-" + (m.id || "new");
const dl = h("datalist", { id: dlId });
if (m.model) dl.appendChild(h("option", { value: m.model }));
const modelIn = h("input", { value: m.model || "", placeholder: t("models.form.modelPlaceholder"), list: dlId, autocomplete: "off" });
const kindSel = h("select", {}, ...(meta.kinds || ["chat"]).map((k) => opt(k, k, k === m.kind))); const kindSel = h("select", {}, ...(meta.kinds || ["chat"]).map((k) => opt(k, k, k === m.kind)));
const tierSel = h("select", {}, ...(meta.cost_tiers || ["cheap"]).map((c) => opt(c, c, c === m.cost_tier))); const tierSel = h("select", {}, ...(meta.cost_tiers || ["cheap"]).map((c) => opt(c, c, c === m.cost_tier)));
const priceInIn = h("input", { type: "number", step: "0.01", value: m.price_in ?? 0 }); const priceInIn = h("input", { type: "number", step: "0.01", value: m.price_in ?? 0 });
const priceOutIn = h("input", { type: "number", step: "0.01", value: m.price_out ?? 0 }); const priceOutIn = h("input", { type: "number", step: "0.01", value: m.price_out ?? 0 });
const CAP_EMOJIS = { const keyEnvIn = h("input", { value: m.api_key_env || "", placeholder: "OPENAI_API_KEY" });
tools: "🔧 tools", const secretIn = h("input", { type: "password", placeholder: t("models.form.secretPlaceholder") });
vision: "👁️ vision", const capInputs = (meta.caps || ["tools", "vision"]).map((c) => {
audio: "🎙️ audio",
reasoning: "🧠 reasoning",
json: "📄 json",
caching: "⚡ caching"
};
const capInputs = (meta.caps || Object.keys(CAP_EMOJIS)).map((c) => {
const cb = h("input", { type: "checkbox", checked: (m.caps || []).includes(c) ? true : null }); const cb = h("input", { type: "checkbox", checked: (m.caps || []).includes(c) ? true : null });
return { c, cb }; return { c, cb };
}); });
const envFor = () =>
(existing && existing.api_key_env) ||
(PROVIDER_DEFAULTS[providerSel.value] || {}).keyEnv ||
providerSel.value.toUpperCase().replace(/[^A-Z0-9]/g, "_") + "_API_KEY";
// Fill in a provider's verified defaults when it is selected.
function applyDefaults() {
const d = PROVIDER_DEFAULTS[providerSel.value];
if (!d) return;
baseUrlIn.value = d.base;
tierSel.value = d.tier;
capInputs.forEach(({ c, cb }) => { cb.checked = d.caps.includes(c); });
keyIn.placeholder = d.needsKey ? t("models.form.apiKeyPlaceholder") : t("models.form.apiKeyNotNeeded");
mount(keyStatus, "");
// Automatically pre-fetch models (especially useful for OpenRouter which doesn't need a key)
testKey(true);
}
providerSel.addEventListener("change", applyDefaults);
if (!existing) applyDefaults();
let cachedModels = [];
const autofillData = () => {
console.log("autofillData triggered. modelIn.value:", modelIn.value);
console.log("cachedModels length:", cachedModels.length);
if (!existing && (!idIn.value || idIn._auto)) {
idIn.value = slugId(modelIn.value);
idIn._auto = true;
console.log("auto-filled idIn:", idIn.value);
}
const found = cachedModels.find((m) => (m.id || m) === modelIn.value);
console.log("found model:", found);
if (found) {
if (found.price_in !== undefined) {
priceInIn.value = parseFloat(Number(found.price_in).toFixed(4));
priceOutIn.value = parseFloat(Number(found.price_out).toFixed(4));
console.log("Updated prices:", priceInIn.value, priceOutIn.value);
}
if (found.caps && Array.isArray(found.caps)) {
capInputs.forEach(({ c, cb }) => {
cb.checked = found.caps.includes(c);
});
console.log("Updated caps:", found.caps);
}
}
};
modelIn.addEventListener("input", autofillData);
modelIn.addEventListener("change", autofillData);
if (!existing) {
idIn.addEventListener("input", () => { idIn._auto = false; });
} else {
console.log("Editing existing model, fetching testKey(true)");
testKey(true);
}
async function testKey(silent = false) {
if (!baseUrlIn.value.trim()) {
if (!silent) mount(keyStatus, h("span", { style: "color:var(--bad)" }, t("models.form.baseUrl") + "?"));
return;
}
testBtn.disabled = true;
if (!silent) mount(keyStatus, t("models.form.testing"));
try {
const r = await api.providerModels({
provider: providerSel.value,
base_url: baseUrlIn.value.trim(),
api_key: keyIn.value.trim() || null,
api_key_env: envFor(),
});
if (r.ok) {
cachedModels = r.models || [];
dl.replaceChildren(...cachedModels.map((mm) => h("option", { value: mm.id || mm })));
let msg = "✓ " + t("models.form.foundModels", { n: r.count });
if (r.balance !== undefined) {
msg += ` (Balance: $${r.balance.toFixed(4)})`;
}
mount(keyStatus, h("span", { style: "color:var(--good)" }, msg));
if (!silent) modelIn.focus();
} else {
if (!silent) mount(keyStatus, h("span", { style: "color:var(--bad)" }, "✗ " + (r.error || t("common.error"))));
}
} catch (e) {
if (!silent) mount(keyStatus, h("span", { style: "color:var(--bad)" }, "✗ " + String(e.message || e)));
}
testBtn.disabled = false;
}
testBtn.addEventListener("click", () => testKey(false));
keyIn.addEventListener("change", () => testKey(true));
const keyHint =
existing && existing.key_source === "store" ? t("models.form.keyStored")
: existing && existing.key_source === "env" ? t("models.form.keyFromEnv")
: t("models.form.apiKeyHint");
// ⚙ Generation & Thinking Parameters
const tempIn = h("input", { type: "number", step: "0.05", min: "0", max: "2.0", placeholder: "0.7", value: m.temperature ?? "" });
const topPIn = h("input", { type: "number", step: "0.05", min: "0", max: "1.0", placeholder: "0.9", value: m.top_p ?? "" });
const maxTokIn = h("input", { type: "number", min: "1", placeholder: "2048", value: m.max_tokens ?? "" });
const thinkIn = h("input", { type: "number", min: "0", placeholder: "1024", value: m.think_budget ?? "" });
const o1Sel = h("select", {},
h("option", { value: "" }, t("import.adv.auto")),
h("option", { value: "all" }, "all — O(1) Attention"),
h("option", { value: "deep" }, "deep — O(1) Deep"),
h("option", { value: "off" }, "off — Standard"));
if (m.o1) o1Sel.value = m.o1;
const skipMtpCb = h("input", { type: "checkbox", checked: m.skip_mtp ? true : null });
const sysPromptIn = h("textarea", { rows: 2, placeholder: t("models.form.systemPromptPlaceholder"), value: m.system_prompt || "" });
const advParams = h(
"details",
{ class: "adv-params", style: "margin-top:12px;padding:10px;border:1px solid var(--border);border-radius:6px;" },
h("summary", { style: "font-weight:600;cursor:pointer;" }, "⚙ " + t("models.form.paramsTitle")),
h("div", { class: "row", style: "margin-top:8px" },
field(t("models.form.temperature"), tempIn, t("models.form.temperatureHint")),
field(t("models.form.topP"), topPIn, t("models.form.topPHint"))),
h("div", { class: "row" },
field(t("models.form.maxTokens"), maxTokIn, t("models.form.maxTokensHint")),
field(t("models.form.thinkBudget"), thinkIn, t("models.form.thinkBudgetHint"))),
h("div", { class: "row" },
field("O(1) Attention", o1Sel, t("import.adv.o1Hint")),
field("MTP", h("label", { class: "check" }, skipMtpCb, t("import.adv.skipMtp")))),
field(t("models.form.systemPrompt"), sysPromptIn, t("models.form.systemPromptHint"))
);
const node = h( const node = h(
"div", "div",
{}, {},
h("div", { class: "row" }, field(t("models.form.id"), idIn, existing ? null : t("models.form.idHint")), field(t("models.form.provider"), providerSel)), h("div", { class: "row" }, field(t("models.form.id"), idIn, existing ? null : t("models.form.idHint")), field(t("models.form.provider"), providerSel)),
field(t("models.form.baseUrl"), baseUrlIn), field(t("models.form.baseUrl"), baseUrlIn),
field(t("models.form.apiKey"), h("div", { class: "flex wrap", style: "gap:8px;align-items:center" }, keyIn, testBtn, keyStatus), keyHint), field(t("models.form.model"), modelIn),
field(t("models.form.model"), modelIn, t("models.form.modelHint")),
dl,
h("div", { class: "row" }, field(t("models.form.kind"), kindSel), field(t("models.form.costTier"), tierSel)), h("div", { class: "row" }, field(t("models.form.kind"), kindSel), field(t("models.form.costTier"), tierSel)),
h("div", { class: "row" }, field(t("models.form.priceIn"), priceInIn), field(t("models.form.priceOut"), priceOutIn)), h("div", { class: "row" }, field(t("models.form.priceIn"), priceInIn), field(t("models.form.priceOut"), priceOutIn)),
field(t("models.form.caps"), h("div", { class: "flex wrap", style: "gap:10px" }, ...capInputs.map(({ c, cb }) => h("label", { class: "check", style: "margin:0" }, cb, CAP_EMOJIS[c] || c)))), field(t("models.form.caps"), h("div", { class: "flex wrap" }, ...capInputs.map(({ c, cb }) => h("label", { class: "check" }, cb, c)))),
advParams field(t("models.form.apiKeyEnv"), keyEnvIn, t("models.form.apiKeyEnvHint")),
field(t("models.form.secret"), secretIn)
); );
const getValue = () => { const getValue = () => ({
const typedKey = keyIn.value.trim(); model: {
const hasKey = !!typedKey || !!(existing && existing.api_key_env); id: existing ? m.id : idIn.value.trim(),
const keyEnv = envFor(); provider: providerSel.value,
return { base_url: baseUrlIn.value.trim(),
model: { model: modelIn.value.trim(),
id: existing ? m.id : idIn.value.trim(), kind: kindSel.value,
provider: providerSel.value, cost_tier: tierSel.value,
base_url: baseUrlIn.value.trim(), price_in: parseFloat(priceInIn.value) || 0,
model: modelIn.value.trim(), price_out: parseFloat(priceOutIn.value) || 0,
kind: kindSel.value, api_key_env: keyEnvIn.value.trim() || null,
cost_tier: tierSel.value, caps: capInputs.filter(({ cb }) => cb.checked).map(({ c }) => c),
price_in: parseFloat(priceInIn.value) || 0, },
price_out: parseFloat(priceOutIn.value) || 0, secret: secretIn.value,
api_key_env: hasKey ? keyEnv : null, keyEnv: keyEnvIn.value.trim(),
caps: capInputs.filter(({ cb }) => cb.checked).map(({ c }) => c), });
temperature: tempIn.value !== "" ? parseFloat(tempIn.value) : null,
top_p: topPIn.value !== "" ? parseFloat(topPIn.value) : null,
max_tokens: maxTokIn.value !== "" ? parseInt(maxTokIn.value) : null,
think_budget: thinkIn.value !== "" ? parseInt(thinkIn.value) : null,
o1: o1Sel.value || null,
skip_mtp: skipMtpCb.checked,
system_prompt: sysPromptIn.value.trim() || null,
},
secret: typedKey,
keyEnv,
};
};
return { node, getValue }; return { node, getValue };
} }
@@ -241,14 +70,12 @@ async function openModal(meta, existing, reload) {
const { node, getValue } = modelForm(meta, existing); const { node, getValue } = modelForm(meta, existing);
modal(existing ? t("models.form.editTitle") : t("models.form.addTitle"), node, async () => { modal(existing ? t("models.form.editTitle") : t("models.form.addTitle"), node, async () => {
const { model, secret, keyEnv } = getValue(); const { model, secret, keyEnv } = getValue();
// Forgot the id? Derive it from the model name so saving never errors on a
// blank id.
if (!model.id) model.id = slugId(model.model);
if (!model.id) { if (!model.id) {
toast(t("models.form.idOrModel"), "bad"); toast(t("models.form.id") + "?", "bad");
return false; return false;
} }
if (!model.api_key_env) delete model.api_key_env; // keep serde field optional // strip null api_key_env so serde keeps it optional
if (!model.api_key_env) delete model.api_key_env;
try { try {
if (existing) await api.updateModel(existing.id, model); if (existing) await api.updateModel(existing.id, model);
else await api.createModel(model); else await api.createModel(model);
@@ -341,8 +168,7 @@ function row(m, meta, reload) {
return h( return h(
"tr", "tr",
{}, {},
h("td", { class: "mono" }, m.id, h("td", { class: "mono" }, m.id),
m.managed ? h("span", { class: "badge", style: "margin-left:6px", title: t("models.managedHint") }, t("models.managed")) : null),
h("td", {}, m.provider), h("td", {}, m.provider),
h("td", { class: "mono" }, m.model), h("td", { class: "mono" }, m.model),
h("td", {}, h("span", { class: "badge " + (m.cost_tier || "") }, m.cost_tier || "—")), h("td", {}, h("span", { class: "badge " + (m.cost_tier || "") }, m.cost_tier || "—")),
@@ -356,7 +182,7 @@ function row(m, meta, reload) {
"div", "div",
{ class: "flex", style: "justify-content:flex-end" }, { class: "flex", style: "justify-content:flex-end" },
probeBtn, probeBtn,
m.managed ? null : h("button", { class: "btn sm ghost", onClick: () => openModal(meta, m, reload) }, t("common.edit")), h("button", { class: "btn sm ghost", onClick: () => openModal(meta, m, reload) }, t("common.edit")),
h( h(
"button", "button",
{ {
@@ -366,7 +192,7 @@ function row(m, meta, reload) {
try { try {
await api.deleteModel(m.id); await api.deleteModel(m.id);
toast(t("toast.deleted"), "good"); toast(t("toast.deleted"), "good");
await reload(); reload();
} catch (e) { } catch (e) {
toast(String(e.message || e), "bad"); toast(String(e.message || e), "bad");
} }
+65 -180
View File
@@ -207,157 +207,11 @@ export async function renderSettings() {
const cmfManage = check(t("settings.cmf.manage"), s.cmf?.manage_server, t("settings.cmf.manageHelp")); const cmfManage = check(t("settings.cmf.manage"), s.cmf?.manage_server, t("settings.cmf.manageHelp"));
const cmfAutoInstall = check(t("settings.cmf.autoInstall"), s.cmf?.auto_install, t("settings.cmf.autoInstallHelp")); const cmfAutoInstall = check(t("settings.cmf.autoInstall"), s.cmf?.auto_install, t("settings.cmf.autoInstallHelp"));
const cmfAutoUpdate = check(t("settings.cmf.autoUpdate"), s.cmf?.auto_update, t("settings.cmf.autoUpdateHelp")); const cmfAutoUpdate = check(t("settings.cmf.autoUpdate"), s.cmf?.auto_update, t("settings.cmf.autoUpdateHelp"));
// ── managed local models: one supervised `cortiq serve` per row ── const cmfModel = h("input", { value: s.cmf?.local_model || "", placeholder: "models/my-model.cmf" });
const serverRows = []; const cmfPort = h("input", { type: "number", value: s.cmf?.local_port ?? 8081 });
const serversWrap = h("div", {}); const cmfThreads = h("input", { type: "number", min: 0, value: s.cmf?.threads ?? 8 });
const legacyServers = s.cmf?.local_model const cmfGpu = check(t("settings.cmf.gpu"), s.cmf?.gpu, t("settings.cmf.gpuHelp"));
? [{ id: s.cmf.model_id || "cmf-local", model: s.cmf.local_model, port: s.cmf.local_port ?? 8081, threads: s.cmf.threads ?? 8, gpu: !!s.cmf.gpu }] const cmfModelIdVal = s.cmf?.model_id || "cmf-local"; // auto — no manual "model id" field
: [];
const initServers = (s.cmf?.servers && s.cmf.servers.length) ? s.cmf.servers : legacyServers;
// available .cmf files under models_dir → offer as a dropdown, not a typed path
const cmfFiles = ((await api.cmfFiles().catch(() => ({ files: [] }))).files) || [];
const fileLabel = (f) => f.name + (f.size ? ` (${(f.size / 1073741824).toFixed(f.size >= 1073741824 ? 1 : 2)} GB)` : f.is_dir ? " (dir)" : "");
function autoId(path) {
const base = (path || "").split("/").pop().replace(/\.cmf$/i, "");
return base ? "cmf-" + base.replace(/[^a-zA-Z0-9_-]/g, "-") : "";
}
function makeServerRow(sv) {
sv = sv || { id: "", model: "", port: 8090, threads: 8, gpu: false };
const idIn = h("input", { value: sv.id || "", placeholder: "cmf-id", style: "width:130px" });
// pick from existing .cmf files rather than typing a path
const modelSel = h("select", { style: "flex:2;min-width:150px" });
const seen = new Set();
modelSel.appendChild(h("option", { value: "" }, "— " + t("settings.cmf.pick") + " —"));
cmfFiles.forEach((f) => { modelSel.appendChild(h("option", { value: f.path }, fileLabel(f))); seen.add(f.path); });
if (sv.model && !seen.has(sv.model)) modelSel.appendChild(h("option", { value: sv.model }, sv.model));
modelSel.value = sv.model || "";
const portIn = h("input", { type: "number", value: sv.port ?? 8090, style: "width:82px" });
const threadsIn = h("input", { type: "number", min: 0, value: sv.threads ?? 8, style: "width:60px", title: t("settings.cmf.threadsHelp") });
const gpuIn = h("input", { type: "checkbox", checked: sv.gpu ? true : null, title: t("settings.cmf.gpuHelp") });
const checkBtn = h("button", { class: "btn small", type: "button" }, t("settings.cmf.checkPort"));
const removeBtn = h("button", { class: "btn small danger", type: "button", title: t("common.delete") }, "✕");
const st = h("span", { class: "small" });
modelSel.addEventListener("change", () => { if (!idIn.value || idIn._auto) { idIn.value = autoId(modelSel.value); idIn._auto = true; } });
idIn.addEventListener("input", () => { idIn._auto = false; });
checkBtn.addEventListener("click", async () => {
const p = parseInt(portIn.value, 10); if (!p) return;
checkBtn.disabled = true; mount(st, t("status.checking"));
try {
const r = await api.cmfPortCheck(p);
if (r.available) mount(st, h("span", { style: "color:var(--good)" }, "✓ " + t("settings.cmf.portFree")));
else {
const kids = [h("span", { style: "color:var(--bad)" }, "✗ " + t("settings.cmf.portBusy"))];
if (r.suggested) { const b = h("button", { class: "btn small", type: "button" }, t("settings.cmf.useSuggested", { port: r.suggested })); b.addEventListener("click", () => { portIn.value = r.suggested; mount(st, ""); }); kids.push(b); }
mount(st, h("span", {}, ...kids));
}
} catch (e) { mount(st, String(e.message || e)); }
checkBtn.disabled = false;
});
// ── O(1) Nyström Attention ──
// NOT a "reasoning mode". This is a memory-efficient streaming attention
// kernel (Nyström/landmark) that replaces standard KV-cache attention,
// giving O(1) memory per decode step instead of O(n).
const o1Sel = h("select", { style: "width:120px", title: "O(1) Nyström attention layer spec: which layers replace KV-cache with streaming Nyström attention" },
h("option", { value: "" }, "O(1) авто"),
h("option", { value: "all" }, "O(1) все слои"),
h("option", { value: "deep" }, "O(1) глубокие"),
h("option", { value: "off" }, "O(1) выкл."));
if (sv.o1) o1Sel.value = sv.o1;
const o1mIn = h("input", { type: "number", min: "4", placeholder: "32", value: sv.o1_m ?? "", style: "width:55px", title: "Landmark budget (≥4, default 32). More landmarks ≠ better: m=64 measured worse." });
const o1wIn = h("input", { type: "number", min: "1", placeholder: "128", value: sv.o1_window ?? "", style: "width:60px", title: "Exact-window width — main quality lever (default 128)" });
const o1sinkIn = h("input", { type: "number", min: "0", placeholder: "4", value: sv.o1_sink ?? "", style: "width:50px", title: "Permanent exact sink keys, StreamingLLM discipline (default 4)" });
// ── Generation parameters ──
const tempIn = h("input", { type: "number", step: "0.05", min: "0", max: "2.0", placeholder: "0.7", value: sv.temperature ?? "", style: "width:60px" });
const topPIn = h("input", { type: "number", step: "0.05", min: "0", max: "1.0", placeholder: "0.9", value: sv.top_p ?? "", style: "width:60px" });
const maxTokIn = h("input", { type: "number", min: "1", placeholder: "2048", value: sv.max_tokens ?? "", style: "width:75px" });
// think_budget = 0 means reasoning disabled (enable_thinking=false);
// null = model decides; N>0 = token budget for the <think> block.
const noThinkCb = h("input", { type: "checkbox", checked: (sv.think_budget === 0) ? true : null, title: "Отключить режим размышлений (think_budget=0, enable_thinking=false для Qwen3/3.5)" });
const thinkIn = h("input", { type: "number", min: "1", placeholder: "авто", value: (sv.think_budget && sv.think_budget > 0) ? sv.think_budget : "", style: "width:75px", title: "Бюджет токенов на размышления (think_budget). Пустое = модель решает сама." });
if (sv.think_budget === 0) thinkIn.disabled = true;
noThinkCb.addEventListener("change", () => {
thinkIn.disabled = noThinkCb.checked;
if (noThinkCb.checked) thinkIn.value = "";
});
const sysIn = h("input", { placeholder: "System prompt (optional)", value: sv.system_prompt || "", style: "flex:1;min-width:200px" });
// ── MTP (Multi-Token Prediction): speculative decoding, NOT reasoning ──
const skipMtpCb = h("input", { type: "checkbox", checked: sv.skip_mtp ? true : null, title: "Disable MTP speculative decoding (env CMF_MTP=0)" });
// ── Settings drawer ──
const drawerStyle = "display:none;width:100%;margin-top:6px;padding:10px 12px;background:var(--bg-subtle);border-radius:6px;border:1px solid var(--border)";
const groupStyle = "display:flex;flex-wrap:wrap;gap:8px;align-items:center;padding:4px 0";
const labelStyle = "font-size:11px;color:var(--text-muted);font-weight:600;text-transform:uppercase;letter-spacing:0.5px;width:100%;margin-bottom:2px";
const drawer = h("div", { style: drawerStyle },
// Group 1: O(1) Nyström Attention
h("div", { style: labelStyle }, "O(1) Nyström Attention"),
h("div", { style: groupStyle },
h("span", { class: "small muted" }, "Слои:"), o1Sel,
h("span", { class: "small muted" }, "m:"), o1mIn,
h("span", { class: "small muted" }, "Window:"), o1wIn,
h("span", { class: "small muted" }, "Sink:"), o1sinkIn
),
h("hr", { style: "border:none;border-top:1px solid var(--border);margin:6px 0" }),
// Group 2: Generation
h("div", { style: labelStyle }, "Генерация"),
h("div", { style: groupStyle },
h("span", { class: "small muted" }, "Temp:"), tempIn,
h("span", { class: "small muted" }, "Top-P:"), topPIn,
h("span", { class: "small muted" }, "Max токенов:"), maxTokIn
),
h("div", { style: groupStyle },
h("span", { class: "small muted" }, "System prompt:"), sysIn
),
h("hr", { style: "border:none;border-top:1px solid var(--border);margin:6px 0" }),
// Group 3: Reasoning (thinking)
h("div", { style: labelStyle }, "Режим размышлений"),
h("div", { style: groupStyle },
h("label", { class: "check small", title: "Для моделей Qwen3/3.5: enable_thinking=false — модель отвечает напрямую без блока <think>" }, noThinkCb, "Отключить размышления"),
h("span", { class: "small muted" }, "Think budget:"), thinkIn
),
h("hr", { style: "border:none;border-top:1px solid var(--border);margin:6px 0" }),
// Group 4: Runtime
h("div", { style: labelStyle }, "Runtime"),
h("div", { style: groupStyle },
h("label", { class: "check small", title: "MTP — Multi-Token Prediction (спекулятивное декодирование). Отключение замедляет генерацию, но стабилизирует output." }, skipMtpCb, "Отключить MTP")
)
);
const gearBtn = h("button", { class: "btn small", type: "button", title: "⚙ Настройки модели" }, "⚙");
gearBtn.addEventListener("click", () => {
drawer.style.display = drawer.style.display === "none" ? "block" : "none";
});
const row = h("div", { class: "flex wrap cmf-server-row", style: "gap:6px;align-items:center;margin:6px 0" },
idIn, modelSel, portIn, checkBtn, threadsIn, h("label", { class: "flex", style: "gap:4px" }, gpuIn, "GPU"), gearBtn, removeBtn, st, drawer);
const entry = { node: row, read: () => ({
id: idIn.value.trim() || autoId(modelSel.value),
model: modelSel.value.trim(),
port: parseInt(portIn.value) || 8090,
threads: parseInt(threadsIn.value) || 0,
gpu: gpuIn.checked,
temperature: tempIn.value !== "" ? parseFloat(tempIn.value) : null,
top_p: topPIn.value !== "" ? parseFloat(topPIn.value) : null,
max_tokens: maxTokIn.value !== "" ? parseInt(maxTokIn.value) : null,
think_budget: noThinkCb.checked ? 0 : (thinkIn.value !== "" ? parseInt(thinkIn.value) : null),
o1: o1Sel.value || null,
o1_m: o1mIn.value !== "" ? parseInt(o1mIn.value) : null,
o1_window: o1wIn.value !== "" ? parseInt(o1wIn.value) : null,
o1_sink: o1sinkIn.value !== "" ? parseInt(o1sinkIn.value) : null,
skip_mtp: skipMtpCb.checked,
system_prompt: sysIn.value.trim() || null,
}) };
removeBtn.addEventListener("click", () => { const i = serverRows.indexOf(entry); if (i >= 0) serverRows.splice(i, 1); row.remove(); });
serverRows.push(entry);
serversWrap.appendChild(row);
return entry;
}
initServers.forEach((sv) => makeServerRow(sv));
const addServerBtn = h("button", { class: "btn small", type: "button" }, t("settings.cmf.addServer"));
addServerBtn.addEventListener("click", () => makeServerRow());
// "Install cortiq" — the CLI that RUNS the .cmf format. Must be present before // "Install cortiq" — the CLI that RUNS the .cmf format. Must be present before
// a local model server can start; this is step 1 for a new user. // a local model server can start; this is step 1 for a new user.
const cmfStatusEl = h("span", { class: "muted" }, "…"); const cmfStatusEl = h("span", { class: "muted" }, "…");
@@ -374,17 +228,14 @@ export async function renderSettings() {
? h("span", {}, h("span", { class: "badge ok" }, "✓"), " cortiq " + (st.version || "")) ? h("span", {}, h("span", { class: "badge ok" }, "✓"), " cortiq " + (st.version || ""))
: h("span", {}, h("span", { class: "badge bad" }, "×"), " " + t("settings.cmf.notInstalled")) : h("span", {}, h("span", { class: "badge bad" }, "×"), " " + t("settings.cmf.notInstalled"))
); );
// per-server serving indicators // reflect the managed server state under the "run locally" toggle
const servers = st.servers || []; if (st.manage_server && st.running) {
if (!servers.length) { mount(cmfServingEl, h("span", { style: "color:var(--good)" },
mount(cmfServingEl, ""); "▶ " + t("settings.cmf.serving", { model: st.model_id || "", port: st.local_port }) + (st.healthy ? "" : " …")));
} else if (st.last_error) {
mount(cmfServingEl, h("span", { style: "color:var(--bad)" }, "✗ " + st.last_error));
} else { } else {
mount(cmfServingEl, h("div", {}, ...servers.map((sv) => { mount(cmfServingEl, "");
if (sv.last_error) return h("div", { style: "color:var(--bad)" }, "✗ " + sv.id + ": " + sv.last_error);
if (sv.running) return h("div", { style: "color:var(--good)" },
"▶ " + t("settings.cmf.serving", { model: sv.id, port: sv.port }) + (sv.healthy ? "" : " …"));
return h("div", { class: "muted" }, "• " + sv.id + " :" + sv.port);
})));
} }
} catch (_) { } catch (_) {
mount(cmfStatusEl, "—"); mount(cmfStatusEl, "—");
@@ -400,6 +251,7 @@ export async function renderSettings() {
await new Promise((r) => setTimeout(r, 3000)); await new Promise((r) => setTimeout(r, 3000));
const st = await api.cmfStatus(); const st = await api.cmfStatus();
if (st.installed) { toast(t("settings.cmf.installedOk"), "good"); break; } if (st.installed) { toast(t("settings.cmf.installedOk"), "good"); break; }
if (st.last_error) { toast(st.last_error, "bad"); break; }
} }
} catch (e) { } catch (e) {
toast(String(e.message || e), "bad"); toast(String(e.message || e), "bad");
@@ -409,6 +261,36 @@ export async function renderSettings() {
}); });
refreshCmfStatus(); refreshCmfStatus();
// Port availability check — let the user confirm the chosen port is free
// before enabling the managed server (the default 8081 often clashes).
const cmfPortStatus = h("span", { class: "small", style: "margin-left:8px" });
const cmfPortCheckBtn = h("button", { class: "btn small", type: "button" }, t("settings.cmf.checkPort"));
async function checkPort() {
const p = parseInt(cmfPort.value, 10);
if (!p) { mount(cmfPortStatus, ""); return; }
cmfPortCheckBtn.disabled = true;
mount(cmfPortStatus, t("status.checking"));
try {
const r = await api.cmfPortCheck(p);
if (r.available) {
mount(cmfPortStatus, h("span", { style: "color:var(--good)" }, "✓ " + t("settings.cmf.portFree")));
} else {
const kids = [h("span", { style: "color:var(--bad)" }, "✗ " + t("settings.cmf.portBusy"))];
if (r.suggested) {
const useBtn = h("button", { class: "btn small", type: "button", style: "margin-left:8px" },
t("settings.cmf.useSuggested", { port: r.suggested }));
useBtn.addEventListener("click", () => { cmfPort.value = r.suggested; checkPort(); });
kids.push(useBtn);
}
mount(cmfPortStatus, h("span", {}, ...kids));
}
} catch (e) {
mount(cmfPortStatus, String(e.message || e));
}
cmfPortCheckBtn.disabled = false;
}
cmfPortCheckBtn.addEventListener("click", checkPort);
const saveBtn = h("button", { class: "btn primary" }, t("common.save")); const saveBtn = h("button", { class: "btn primary" }, t("common.save"));
saveBtn.addEventListener("click", async () => { saveBtn.addEventListener("click", async () => {
saveBtn.disabled = true; saveBtn.disabled = true;
@@ -453,9 +335,11 @@ export async function renderSettings() {
manage_server: cmfManage.cb.checked, manage_server: cmfManage.cb.checked,
auto_install: cmfAutoInstall.cb.checked, auto_install: cmfAutoInstall.cb.checked,
auto_update: cmfAutoUpdate.cb.checked, auto_update: cmfAutoUpdate.cb.checked,
// the managed-models list is the source of truth; clear the legacy single local_model: cmfModel.value.trim(),
servers: serverRows.map((r) => r.read()).filter((sv) => sv.model && sv.id), local_port: parseInt(cmfPort.value) || 8081,
local_model: "", model_id: cmfModelIdVal,
threads: parseInt(cmfThreads.value) || 0,
gpu: cmfGpu.cb.checked,
}, },
}; };
// serde: drop null optionals // serde: drop null optionals
@@ -536,11 +420,9 @@ export async function renderSettings() {
"div", "div",
{ class: "card" }, { class: "card" },
h("div", { class: "card-head" }, h("h3", {}, t("settings.route"))), h("div", { class: "card-head" }, h("h3", {}, t("settings.route"))),
field(t("settings.route.strategy"), strategySel, null, t("settings.route.strategyHelp")), field(t("settings.route.strategy"), strategySel),
h("div", { class: "row" }, h("div", { class: "row" }, field(t("settings.route.maxChars"), maxChars), field(t("settings.route.cacheTtl"), cacheTtl)),
field(t("settings.route.maxChars"), maxChars, null, t("settings.route.maxCharsHelp")), field(t("settings.route.profile"), profileSel)
field(t("settings.route.cacheTtl"), cacheTtl, null, t("settings.route.cacheTtlHelp"))),
field(t("settings.route.profile"), profileSel, null, t("settings.route.profileHelp"))
), ),
h( h(
"div", "div",
@@ -595,21 +477,24 @@ export async function renderSettings() {
), ),
h("div", { class: "row" }, cmfAutoInstall.node, cmfAutoUpdate.node), h("div", { class: "row" }, cmfAutoInstall.node, cmfAutoUpdate.node),
h("div", { class: "divider" }), h("div", { class: "divider" }),
// Step 2 — managed local models: one supervised cortiq serve per row, each // Step 2 — add a model. Models are imported in the Models/Import tab; here
// on its own port. Add as many as you have .cmf files + free ports. // you just point the local server at the .cmf file it produced.
h("div", { class: "field" }, h("div", { class: "field" }, h("span", {}, "2. " + t("settings.cmf.addModel"), " ", help(t("settings.cmf.addModelHelp"))),
h("span", {}, "2. " + t("settings.cmf.models"), " ", help(t("settings.cmf.modelsHelp"))),
h("div", { class: "hint" }, t("settings.cmf.rowHint")),
h("div", { class: "hint" }, t("settings.cmf.importNote"), h("div", { class: "hint" }, t("settings.cmf.importNote"),
" ", h("a", { href: "#/import" }, t("settings.cmf.importLink") + " ↗"))), " ", h("a", { href: "#/import" }, t("settings.cmf.importLink") + " ↗"))),
serversWrap, field(t("settings.cmf.model"), cmfModel, null, t("settings.cmf.modelHelp")),
addServerBtn,
cmfServingEl,
h("div", { class: "hint" }, t("settings.cmf.speedNote")),
h("div", { class: "divider" }), h("div", { class: "divider" }),
// Step 3 — how the gateway uses them. // Step 3 — run it locally + performance.
cmfManage.node, cmfManage.node,
cmfLocalOnly.node cmfServingEl,
h("div", { class: "row" },
field(t("settings.cmf.port"),
h("div", { class: "flex wrap" }, cmfPort, cmfPortCheckBtn, cmfPortStatus),
null, t("settings.cmf.portHelp")),
field(t("settings.cmf.threads"), cmfThreads, null, t("settings.cmf.threadsHelp"))),
cmfGpu.node,
cmfLocalOnly.node,
h("div", { class: "hint" }, t("settings.cmf.speedNote"))
) )
), ),
h( h(