Compare commits

...
14 Commits
Author SHA1 Message Date
infosave2007 d93ac537c8 docs(import): human-readable labels + hover help for the O(1) knobs
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
Landmarks / Exact window / Sink keys now carry a '?' tooltip explaining in
plain language what each knob does, its trade-off, and the validated default
(32 / 128 / 4), in EN + RU.
2026-07-14 11:19:00 +03:00
infosave2007 4220bc27c2 feat(models): ask whether to delete the model file from disk
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
- The delete button now opens a dialog; for local models it includes an "also
  delete the model file from disk" checkbox (checked by default), so a pool
  entry can be removed while keeping the .cmf for later.
- DELETE /admin/api/models/:id accepts ?delete_file=true|false (default true).
  The shared-file guard still applies: a file referenced by another model is
  never removed.
- cmf_file_for_model now treats any loopback base_url as local (managed servers
  can sit on any port, not just the legacy local_port).
2026-07-14 11:09:44 +03:00
infosave2007 6fdbe0e159 fix(settings): replacing a managed local model no longer breaks save
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
- put_settings now heals [routing] after the managed-servers list changes: a
  removed server whose port is taken over by a new one is treated as a
  replacement (old id → new id in default + every tier); truly vanished ids are
  stripped; an emptied default falls back to the first managed server / pool
  model. Previously the stale id failed validation and the save errored.
- /admin/api/cmf/port now returns managed:true when the port is held by this
  gateway's own managed server; the Settings port check shows that as info
  ("freed on restart, replacing is fine") instead of a red "port busy".
2026-07-14 10:30:39 +03:00
infosave2007 243d8f4c7d feat(import): O(1) attention options in Advanced (cortiq >= 0.2.0)
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
- New Advanced controls on the import page: O(1) attention spec (off / all /
  deep N / explicit layer list) plus the --o1-m / --o1-window / --o1-sink knobs
  with validated defaults (32 / 128 / 4) and explanatory hints.
- Backend passes --o1* to the native `cortiq convert`; fails fast with clear
  messages when combined with Python-only options, used on GGUF imports, or
  when the installed cortiq predates 0.2.0 (the hint needs the new runtime).
- The hint is recorded in the .cmf header (weights byte-identical); the runtime
  applies it at load. Verified end-to-end: converted a model with --o1 all and
  the runtime logged "o1 nystrom attention: 2/2 layer(s)".
2026-07-14 10:16:54 +03:00
infosave2007 4f72711a1e feat(dashboard): windowed stats per range + clear-logs button
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
- Stats snapshot now computes the KPI totals AND the breakdown from the
  in-window per-minute buckets (previously they used all-time accumulators, so
  switching 1h/24h/7d changed only the chart, not the numbers). Buckets now
  carry failovers + per-group breakdown; Prometheus keeps all-time counters.
- New DELETE /admin/api/stats clears all in-memory stats and truncates the
  JSONL log; the dashboard gets a "Clear logs" button (with confirm).
2026-07-09 17:07:05 +03:00
infosave2007 7855472499 docs(settings): add help tooltips to routing-input fields (strategy, max chars, cache TTL, profile)
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
2026-07-09 16:49:16 +03:00
infosave2007 a6ed974ece feat(models): auto-derive id from model name; actionable probe auth errors
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
- The model id is now auto-filled from the model name as you type, and if left
  blank it's derived on save (slug of the last path segment, e.g.
  "qwen/qwen3.7-max" → "qwen3-7-max") — saving no longer errors on a blank id.
- probe/test auth failures (401 / "User not found" / "No auth") now append a
  hint to check the API key (for OpenRouter: use an inference key, not a
  provisioning/management key).
2026-07-09 16:30:48 +03:00
infosave2007 4c11b97197 fix(models): Add button, probe of managed models, delete list refresh
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
- ui.js: mount() now wraps non-node children in text nodes (like h()), so
  mount(el, "") no longer throws in appendChild — this is what broke the
  "Add model" form (applyDefaults cleared a status via mount(x,"")).
- probe_model: resolve managed local models too (they are synthesized into the
  registry, not [[models]]) — probing them no longer returns "model not found".
- models.js: await reload() after delete so the row is removed immediately.
2026-07-09 16:13:07 +03:00
infosave2007 87f00ee502 feat(models): friendlier add-model form — OpenRouter/LM Studio, direct key, test & load
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
- Providers: add openrouter (https://openrouter.ai/api/v1) and lmstudio
  (http://localhost:1234/v1), both OpenAI-compatible. Selecting a provider now
  auto-fills its verified base URL, cost tier and default capabilities.
- API key is typed directly in the form and stored securely as a secret under the
  provider's env name — no manual env var to manage.
- New POST /admin/api/provider/models validates the key and lists the provider's
  models; a "Test & load models" button fills a searchable dropdown. The model
  field now sits below the key (some providers list models from the key).
- Fuller capability set: tools, vision, audio, reasoning, json, caching.
2026-07-09 15:45:21 +03:00
infosave2007 4fa1b8ea22 feat(settings): pick a local model from a dropdown of existing .cmf files
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
- New GET /admin/api/cmf/files lists the .cmf files (and tensor-dir .cmf) under
  models_dir, with sizes.
- Settings → Local models: the per-row model field is now a dropdown of those
  files instead of a hand-typed path, and auto-fills the id from the filename.
2026-07-09 15:25:30 +03:00
infosave2007 357441e0da chore(release): v0.2.8 — multiple managed local models
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
- The gateway can now run several local .cmf models at once: a [[cmf.servers]]
  list, one supervised `cortiq serve` per entry on its own port, each registered
  in the pool and independently routable/pinnable. The single legacy
  local_model/local_port still works (back-compat via effective_servers()).
- Settings → Local models is now a rows editor (id · path · port + check · threads
  · GPU) with add/remove and per-server serving status.
- Deleting a managed model stops just its server and removes its file — unless
  another model still points at the same file (shared-file guard).
2026-07-09 12:00:10 +03:00
infosave2007 9d2181e82b chore(release): v0.2.7 — honest pins, visible managed model, physical delete
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
- A pinned playground model that is unavailable now errors clearly instead of
  silently answering from the managed local model (route_source no longer lies).
- The managed CMF model is surfaced in GET /admin/api/models (marked "managed",
  shown read-only in the UI) so the real running model is visible and pinnable
  without a hand-maintained [[models]] entry; a fresh install still lists nothing.
- Deleting a model now removes its local .cmf physically (file or tensor-dir);
  deleting the managed model also stops the server and clears the [cmf] config.
  Routing is stripped of the deleted id so the config still validates.
2026-07-08 21:14:03 +03:00
infosave2007 30378f7b6a chore: fix clippy lints to green the lint CI
- import.rs / promotion.rs: sort_by(|a,b| b.k.cmp(&a.k)) -> sort_by_key(Reverse(k))
- pipeline.rs: skip_while(!p).next() -> find(p)
Behaviour-preserving; completes the CI fix started by the cargo fmt commit.
2026-07-08 20:09:54 +03:00
infosave2007 b4f78bd192 chore: cargo fmt — fix lint 'Format' CI job (red since v0.2.3)
Whole-repo rustfmt drift; behaviour unchanged (whitespace/layout only).
All test platforms already passed; this greens the lint job.
2026-07-08 19:32:35 +03:00
19 changed files with 1494 additions and 373 deletions
Generated
+1 -1
View File
@@ -285,7 +285,7 @@ checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570"
[[package]] [[package]]
name = "cortiq-gateway" name = "cortiq-gateway"
version = "0.2.6" version = "0.2.18"
dependencies = [ dependencies = [
"anyhow", "anyhow",
"arc-swap", "arc-swap",
+1 -1
View File
@@ -1,6 +1,6 @@
[package] [package]
name = "cortiq-gateway" name = "cortiq-gateway"
version = "0.2.6" version = "0.2.18"
edition = "2021" edition = "2021"
rust-version = "1.75" rust-version = "1.75"
license = "Apache-2.0" license = "Apache-2.0"
+399 -40
View File
@@ -165,16 +165,21 @@ pub fn api_routes(admin_token: String) -> Router<SharedState> {
"/admin/api/secrets", "/admin/api/secrets",
get(list_secrets).put(set_secret).delete(clear_secret), get(list_secrets).put(set_secret).delete(clear_secret),
) )
.route("/admin/api/stats", get(get_stats)) .route("/admin/api/stats", get(get_stats).delete(clear_stats))
.route("/admin/api/shadow", get(get_shadow)) .route("/admin/api/shadow", get(get_shadow))
.route("/admin/api/hf/search", get(hf_search)) .route("/admin/api/hf/search", get(hf_search))
.route("/admin/api/import", get(list_imports).post(start_import)) .route("/admin/api/import", get(list_imports).post(start_import))
.route("/admin/api/import/:job", get(import_status).delete(delete_import)) .route(
"/admin/api/import/:job",
get(import_status).delete(delete_import),
)
.route("/admin/api/import/:job/cancel", post(cancel_import)) .route("/admin/api/import/:job/cancel", post(cancel_import))
.route("/admin/api/import/:job/register", post(register_import)) .route("/admin/api/import/:job/register", post(register_import))
.route("/admin/api/cmf", get(cmf_status)) .route("/admin/api/cmf", get(cmf_status))
.route("/admin/api/cmf/install", post(cmf_install)) .route("/admin/api/cmf/install", post(cmf_install))
.route("/admin/api/cmf/port", get(cmf_port_check)) .route("/admin/api/cmf/port", get(cmf_port_check))
.route("/admin/api/cmf/files", get(cmf_files))
.route("/admin/api/provider/models", post(provider_models))
.route("/admin/api/requests", get(get_requests)) .route("/admin/api/requests", get(get_requests))
.route("/admin/api/test", post(run_test)) .route("/admin/api/test", post(run_test))
.route("/admin/api/test/stream", post(run_test_stream)) .route("/admin/api/test/stream", post(run_test_stream))
@@ -387,18 +392,105 @@ async fn router_probe(State(state): State<SharedState>) -> ApiResult {
ok(json!({ "ok": true, "status": "ok", "latency_ms": latency_ms, "usage": usage })) ok(json!({ "ok": true, "status": "ok", "latency_ms": latency_ms, "usage": usage }))
} }
#[derive(Deserialize)]
struct ProviderModelsBody {
provider: String,
base_url: String,
/// Key typed into the form (takes priority)…
#[serde(default)]
api_key: Option<String>,
/// …or the name of a stored secret to reuse if no key was typed.
#[serde(default)]
api_key_env: Option<String>,
}
/// Concise error line from a provider's JSON error body.
fn provider_err(body: &str, status: u16) -> String {
if let Ok(v) = serde_json::from_str::<Value>(body) {
if let Some(m) = v["error"]["message"]
.as_str()
.or_else(|| v["error"].as_str())
.or_else(|| v["message"].as_str())
{
return format!("{status}: {m}");
}
}
format!("HTTP {status}")
}
/// Validate a provider API key and list its models by calling the provider's
/// `/models` endpoint — powers the "Test key & load models" button. The key is
/// used transiently (from the form) or resolved from a stored secret; it is not
/// persisted here.
async fn provider_models(
State(state): State<SharedState>,
Json(b): Json<ProviderModelsBody>,
) -> ApiResult {
let base = b.base_url.trim_end_matches('/').to_string();
if base.is_empty() {
return Err(ApiError::bad("base URL is required"));
}
let key = b.api_key.filter(|k| !k.trim().is_empty()).or_else(|| {
b.api_key_env
.as_deref()
.and_then(|e| state.secrets.resolve(e))
});
let key = key.as_deref();
let client = reqwest::Client::builder()
.timeout(std::time::Duration::from_secs(15))
.build()
.map_err(|e| ApiError::bad(e.to_string()))?;
// Anthropic uses x-api-key + anthropic-version and /v1/models; every other
// (OpenAI-compatible) provider uses a Bearer token and /models.
let resp = if b.provider == "anthropic" {
let mut r = client
.get(format!("{base}/v1/models"))
.header("anthropic-version", "2023-06-01");
if let Some(k) = key {
r = r.header("x-api-key", k);
}
r.send().await
} else {
let mut r = client.get(format!("{base}/models"));
if let Some(k) = key {
r = r.header("authorization", format!("Bearer {k}"));
}
r.send().await
};
let resp = resp.map_err(|e| ApiError::bad(format!("request failed: {e}")))?;
let status = resp.status().as_u16();
let body = resp.text().await.unwrap_or_default();
if !(200..300).contains(&status) {
return ok(json!({ "ok": false, "status": status, "error": provider_err(&body, status) }));
}
// OpenAI + Anthropic both return { "data": [ { "id": ... } ] }.
let v: Value = serde_json::from_str(&body).unwrap_or(json!({}));
let mut models: Vec<String> = v["data"]
.as_array()
.map(|arr| {
arr.iter()
.filter_map(|m| m["id"].as_str().map(|s| s.to_string()))
.collect()
})
.unwrap_or_default();
models.sort();
ok(json!({ "ok": true, "count": models.len(), "models": models }))
}
async fn meta() -> ApiResult { async fn meta() -> ApiResult {
ok(json!({ ok(json!({
"version": env!("CARGO_PKG_VERSION"), "version": env!("CARGO_PKG_VERSION"),
"providers": ["openai", "anthropic", "ollama", "http"], "providers": ["openai", "anthropic", "openrouter", "lmstudio", "ollama", "http"],
"provider_impl": { "openai": true, "ollama": true, "http": true, "anthropic": true }, "provider_impl": { "openai": true, "anthropic": true, "openrouter": true, "lmstudio": true, "ollama": true, "http": true },
"cost_tiers": ["cheap", "mid", "expensive"], "cost_tiers": ["cheap", "mid", "expensive", "local"],
"kinds": ["chat", "embedding"], "kinds": ["chat", "embedding"],
"profiles": ["cost-saver", "balanced", "quality-first"], "profiles": ["cost-saver", "balanced", "quality-first"],
"text_strategies": ["last_user", "last_user_plus_system", "concat_all"], "text_strategies": ["last_user", "last_user_plus_system", "concat_all"],
"policy_modes": ["fixed_table", "cost_aware"], "policy_modes": ["fixed_table", "cost_aware"],
"tiers": ["low", "medium", "high"], "tiers": ["low", "medium", "high"],
"caps": ["tools", "vision"], "caps": ["tools", "vision", "audio", "reasoning", "json", "caching"],
"protocols_impl": { "protocols_impl": {
"openai_chat": true, "openai_completions": true, "openai_embeddings": true, "openai_chat": true, "openai_completions": true, "openai_embeddings": true,
"openai_models": true, "anthropic_messages": true, "mcp": true, "native_passthrough": true "openai_models": true, "anthropic_messages": true, "mcp": true, "native_passthrough": true
@@ -423,7 +515,7 @@ async fn put_config(State(state): State<SharedState>, Json(cfg): Json<Config>) -
async fn list_models(State(state): State<SharedState>) -> ApiResult { async fn list_models(State(state): State<SharedState>) -> ApiResult {
let live = state.live(); let live = state.live();
let models: Vec<Value> = live let mut models: Vec<Value> = live
.cfg .cfg
.models .models
.iter() .iter()
@@ -437,6 +529,31 @@ async fn list_models(State(state): State<SharedState>) -> ApiResult {
v v
}) })
.collect(); .collect();
// Managed local CMF servers are registered in the routing registry from the
// [cmf] config, not from [[models]]. Surface each here (marked managed, so
// the UI shows it read-only) so the real running models are visible and
// pinnable without a hand-maintained static pool entry. A fresh install with
// no managed model still returns an empty list.
let cmf = &live.cfg.cmf;
for s in cmf.effective_servers() {
if models.iter().any(|m| m["id"] == json!(s.id)) {
continue;
}
models.push(json!({
"id": s.id,
"provider": "openai",
"base_url": format!("http://{}:{}/v1", cmf.local_host, s.port),
"model": "cortiq",
"cost_tier": "local",
"price_in": 0.0,
"price_out": 0.0,
"kind": "chat",
"caps": [],
"key_source": "none",
"managed": true,
}));
}
ok(json!({ "models": models })) ok(json!({ "models": models }))
} }
@@ -470,28 +587,147 @@ async fn update_model(
ok(json!({ "ok": true })) ok(json!({ "ok": true }))
} }
async fn delete_model(State(state): State<SharedState>, Path(id): Path<String>) -> ApiResult { /// Remove a `.cmf` path — it may be a plain file or a tensor-dir. Returns whether
/// something was actually removed.
fn remove_cmf_path(path: &std::path::Path) -> bool {
if path.is_dir() {
std::fs::remove_dir_all(path).is_ok()
} else if path.is_file() {
std::fs::remove_file(path).is_ok()
} else {
false
}
}
/// The local `.cmf` backing a pool model, if it is a local CMF model whose file
/// lives under `models_dir`. Cloud models (real remote base_url) return None.
/// Registered imports store `model` = the file stem under models_dir.
fn cmf_file_for_model(
cmf: &crate::config::CmfCfg,
m: &crate::config::ModelCfg,
) -> Option<std::path::PathBuf> {
// Any loopback URL counts as local — managed servers may sit on any port.
let local = m.base_url == cmf.cortiq_server_url
|| m.base_url.contains("//127.0.0.1:")
|| m.base_url.contains("//localhost:")
|| m.base_url
.contains(&format!("{}:{}", cmf.local_host, cmf.local_port));
let stem = m.model.trim();
if !local || stem.is_empty() || stem.contains('/') || stem.contains("..") {
return None; // not local, or an unsafe/underivable path
}
Some(std::path::Path::new(&cmf.models_dir).join(format!("{stem}.cmf")))
}
/// Drop a model id from routing (default + every tier) so `validate()` still
/// passes after the model is gone.
fn strip_id_from_routing(routing: &mut RoutingCfg, id: &str) {
if routing.default == id {
routing.default = String::new();
}
for tt in routing.tiers.values_mut() {
match tt {
crate::config::TierTargets::List(v) => v.retain(|x| x != id),
crate::config::TierTargets::One(s) if s == id => {
*tt = crate::config::TierTargets::List(Vec::new());
}
crate::config::TierTargets::One(_) => {}
}
}
}
/// Whether any managed server or static model in `cfg` still uses `path` as its
/// local `.cmf`. Guards against deleting a file that another model shares.
fn cmf_path_referenced(cfg: &crate::config::Config, path: &std::path::Path) -> bool {
cfg.cmf
.effective_servers()
.iter()
.any(|s| std::path::Path::new(&s.model) == path)
|| cfg
.models
.iter()
.any(|m| cmf_file_for_model(&cfg.cmf, m).as_deref() == Some(path))
}
#[derive(Deserialize)]
struct DeleteModelQuery {
/// Also remove the local `.cmf` from disk. Defaults to true (the historical
/// behaviour); the UI asks the user and passes it explicitly.
delete_file: Option<bool>,
}
async fn delete_model(
State(state): State<SharedState>,
Path(id): Path<String>,
Query(q): Query<DeleteModelQuery>,
) -> ApiResult {
let mut cfg = current_cfg(&state); let mut cfg = current_cfg(&state);
let delete_file = q.delete_file.unwrap_or(true);
// Managed local models live in [cmf], not [[models]]. Deleting one stops its
// server, removes it from the managed config, and (when asked) deletes its
// file — unless another model still points at the same file.
let managed = cfg.cmf.effective_servers().into_iter().find(|s| s.id == id);
if let Some(server) = managed {
if !cfg.models.iter().any(|x| x.id == id) {
if cfg.cmf.servers.iter().any(|s| s.id == id) {
cfg.cmf.servers.retain(|s| s.id != id); // multi-model list entry
} else {
cfg.cmf.local_model = String::new(); // legacy single model
cfg.cmf.manage_server = false;
}
strip_id_from_routing(&mut cfg.routing, &id);
let path = std::path::PathBuf::from(&server.model);
let shared = cmf_path_referenced(&cfg, &path);
state.cmf.stop_one(&id).await;
state.reload(cfg)?;
let file_removed = if delete_file && !shared {
remove_cmf_path(&path)
} else {
false
};
return ok(json!({ "ok": true, "file_removed": file_removed }));
}
}
let victim = cfg.models.iter().find(|x| x.id == id).cloned();
let before = cfg.models.len(); let before = cfg.models.len();
cfg.models.retain(|x| x.id != id); cfg.models.retain(|x| x.id != id);
if cfg.models.len() == before { if cfg.models.len() == before {
return Err(ApiError::not_found(format!("model '{id}' not found"))); return Err(ApiError::not_found(format!("model '{id}' not found")));
} }
strip_id_from_routing(&mut cfg.routing, &id);
let file = victim
.as_ref()
.and_then(|m| cmf_file_for_model(&cfg.cmf, m));
// Only remove the file if asked to AND no remaining model still references it.
let file = file.filter(|p| delete_file && !cmf_path_referenced(&cfg, p));
state.reload(cfg)?; state.reload(cfg)?;
ok(json!({ "ok": true })) let file_removed = file.map(|p| remove_cmf_path(&p)).unwrap_or(false);
ok(json!({ "ok": true, "file_removed": file_removed }))
} }
async fn probe_model(State(state): State<SharedState>, Path(id): Path<String>) -> ApiResult { async fn probe_model(State(state): State<SharedState>, Path(id): Path<String>) -> ApiResult {
let live = state.live(); let live = state.live();
let mcfg = live // Provider string: from the static pool, or "openai" for a managed local
// model (which is synthesized into the registry, not [[models]]).
let provider_str = live
.cfg .cfg
.models .models
.iter() .iter()
.find(|m| m.id == id) .find(|m| m.id == id)
.ok_or_else(|| ApiError::not_found(format!("model '{id}' not found")))? .map(|m| m.provider.clone())
.clone(); .or_else(|| {
live.cfg
.cmf
.effective_servers()
.iter()
.find(|s| s.id == id)
.map(|_| "openai".to_string())
})
.ok_or_else(|| ApiError::not_found(format!("model '{id}' not found")))?;
if mcfg.provider == "anthropic" { if provider_str == "anthropic" {
return ok(json!({ return ok(json!({
"ok": false, "ok": false,
"error": "anthropic provider not yet implemented (planned v0.2)", "error": "anthropic provider not yet implemented (planned v0.2)",
@@ -528,7 +764,21 @@ async fn probe_model(State(state): State<SharedState>, Path(id): Path<String>) -
let latency_ms = started.elapsed().as_millis() as u64; let latency_ms = started.elapsed().as_millis() as u64;
match res { match res {
Ok(_) => ok(json!({ "ok": true, "latency_ms": latency_ms })), Ok(_) => ok(json!({ "ok": true, "latency_ms": latency_ms })),
Err(e) => ok(json!({ "ok": false, "latency_ms": latency_ms, "error": e.to_string() })), Err(e) => {
let msg = e.to_string();
// Make auth failures actionable — a raw "User not found" (OpenRouter's
// 401) usually means the API key is wrong or the wrong type.
let hint = if msg.contains("401")
|| msg.contains("Unauthorized")
|| msg.contains("User not found")
|| msg.contains("No auth")
{
" — check the API key (for OpenRouter use an inference key, not a provisioning/management key)"
} else {
""
};
ok(json!({ "ok": false, "latency_ms": latency_ms, "error": format!("{msg}{hint}") }))
}
} }
} }
@@ -666,15 +916,79 @@ async fn put_settings(State(state): State<SharedState>, Json(b): Json<SettingsBo
|| v.local_model != cfg.cmf.local_model || v.local_model != cfg.cmf.local_model
|| v.threads != cfg.cmf.threads || v.threads != cfg.cmf.threads
|| v.gpu != cfg.cmf.gpu || v.gpu != cfg.cmf.gpu
|| v.servers != cfg.cmf.servers
{ {
needs_restart = true; needs_restart = true;
} }
let old_servers = cfg.cmf.effective_servers();
cfg.cmf = v; cfg.cmf = v;
// Replacing or renaming a managed model must not brick the save: routing
// may still reference the old id. Map a vanished id to its replacement
// (the new server on the same port), strip ids that are truly gone, and
// repair the default so validate() passes.
heal_routing_after_cmf_change(&mut cfg, &old_servers);
} }
state.reload(cfg)?; state.reload(cfg)?;
ok(json!({ "ok": true, "needs_restart": needs_restart })) ok(json!({ "ok": true, "needs_restart": needs_restart }))
} }
/// After the managed-servers list changed, rewrite [routing] so it only
/// references ids that still exist. A removed server whose port is now held by
/// a NEW server is treated as a replacement (old id → new id); other vanished
/// ids are stripped. An emptied default falls back to the first managed server,
/// then the first pool model.
fn heal_routing_after_cmf_change(cfg: &mut Config, old_servers: &[crate::config::CmfServer]) {
let servers = cfg.cmf.effective_servers();
let known: std::collections::HashSet<String> = cfg
.models
.iter()
.map(|m| m.id.clone())
.chain(servers.iter().map(|s| s.id.clone()))
.collect();
let new_by_port: HashMap<u16, String> =
servers.iter().map(|s| (s.port, s.id.clone())).collect();
let mut rename: HashMap<String, String> = HashMap::new();
for old in old_servers {
if !known.contains(&old.id) {
if let Some(new_id) = new_by_port.get(&old.port) {
rename.insert(old.id.clone(), new_id.clone());
}
}
}
let fix = |id: &str| -> Option<String> {
if known.contains(id) {
Some(id.to_string())
} else {
rename.get(id).cloned()
}
};
if !cfg.routing.default.is_empty() {
cfg.routing.default = fix(&cfg.routing.default).unwrap_or_default();
}
if cfg.routing.default.is_empty() {
cfg.routing.default = servers
.first()
.map(|s| s.id.clone())
.or_else(|| cfg.models.first().map(|m| m.id.clone()))
.unwrap_or_default();
}
for tt in cfg.routing.tiers.values_mut() {
let list = match tt {
TierTargets::List(v) => v.clone(),
TierTargets::One(s) => vec![s.clone()],
};
let mut healed: Vec<String> = Vec::new();
for id in &list {
if let Some(new_id) = fix(id) {
if !healed.contains(&new_id) {
healed.push(new_id);
}
}
}
*tt = TierTargets::List(healed);
}
}
// ───────────────────────────── cmf runtime ───────────────────────────────── // ───────────────────────────── cmf runtime ─────────────────────────────────
/// Status of the `cortiq` CLI (the CMF format runtime): is it installed, which /// Status of the `cortiq` CLI (the CMF format runtime): is it installed, which
/// version, and whether a managed server is running. Drives the "Install cortiq" /// version, and whether a managed server is running. Drives the "Install cortiq"
@@ -683,19 +997,32 @@ async fn cmf_status(State(state): State<SharedState>) -> ApiResult {
let cmf = state.live().cfg.cmf.clone(); let cmf = state.live().cfg.cmf.clone();
let ver = crate::cmf_runtime::installed_version(&cmf.cortiq_bin); let ver = crate::cmf_runtime::installed_version(&cmf.cortiq_bin);
let st = state.cmf.status(); let st = state.cmf.status();
// Each configured managed server, merged with its live runtime status.
let servers: Vec<Value> = cmf
.effective_servers()
.iter()
.map(|s| {
let live = st.servers.iter().find(|x| x.id == s.id);
json!({
"id": s.id,
"model": s.model,
"port": s.port,
"threads": s.threads,
"gpu": s.gpu,
"running": live.map(|l| l.running).unwrap_or(false),
"healthy": live.map(|l| l.healthy).unwrap_or(false),
"last_error": live.and_then(|l| l.last_error.clone()),
})
})
.collect();
ok(json!({ ok(json!({
"installed": ver.is_some(), "installed": ver.is_some(),
"version": ver, "version": ver,
"running": st.running, "latest": st.latest_version,
"healthy": st.healthy,
"last_error": st.last_error,
"log": st.log,
// serving info so the card can show "serving <model> on :<port>"
"manage_server": cmf.manage_server, "manage_server": cmf.manage_server,
"local_model": cmf.local_model,
"model_id": cmf.model_id,
"local_host": cmf.local_host, "local_host": cmf.local_host,
"local_port": cmf.local_port, "servers": servers,
"log": st.log,
})) }))
} }
@@ -707,6 +1034,36 @@ async fn cmf_install(State(state): State<SharedState>) -> ApiResult {
ok(json!({ "started": true })) ok(json!({ "started": true }))
} }
/// List the `.cmf` model files available under `models_dir` so the UI can offer
/// them as a dropdown instead of a hand-typed path. Includes tensor-dir `.cmf`.
async fn cmf_files(State(state): State<SharedState>) -> ApiResult {
let dir = state.live().cfg.cmf.models_dir.clone();
let mut entries: Vec<(String, String, Option<u64>, bool)> = Vec::new();
if let Ok(rd) = std::fs::read_dir(&dir) {
for e in rd.flatten() {
let name = e.file_name().to_string_lossy().to_string();
if !name.ends_with(".cmf") {
continue;
}
let p = e.path();
let is_dir = p.is_dir();
let size = if is_dir {
None
} else {
std::fs::metadata(&p).ok().map(|m| m.len())
};
let path = format!("{}/{}", dir.trim_end_matches('/'), name);
entries.push((name, path, size, is_dir));
}
}
entries.sort_by(|a, b| a.0.cmp(&b.0));
let files: Vec<Value> = entries
.into_iter()
.map(|(name, path, size, is_dir)| json!({ "name": name, "path": path, "size": size, "is_dir": is_dir }))
.collect();
ok(json!({ "dir": dir, "files": files }))
}
#[derive(Deserialize)] #[derive(Deserialize)]
struct PortQuery { struct PortQuery {
port: u16, port: u16,
@@ -726,7 +1083,7 @@ async fn cmf_port_check(State(state): State<SharedState>, Query(q): Query<PortQu
}; };
let st = state.cmf.status(); let st = state.cmf.status();
let port = q.port; let port = q.port;
let running_here = port == cmf.local_port && st.running; let running_here = st.servers.iter().any(|s| s.port == port && s.running);
// The bind+connect probe is blocking std::net; run it off the async runtime. // The bind+connect probe is blocking std::net; run it off the async runtime.
let (available, detail, suggested) = tokio::task::spawn_blocking(move || { let (available, detail, suggested) = tokio::task::spawn_blocking(move || {
@@ -736,7 +1093,11 @@ async fn cmf_port_check(State(state): State<SharedState>, Query(q): Query<PortQu
if port_is_free(&host, port) { if port_is_free(&host, port) {
(true, "free", None) (true, "free", None)
} else { } else {
(false, "occupied by another process", next_free_port(&host, port)) (
false,
"occupied by another process",
next_free_port(&host, port),
)
} }
}) })
.await .await
@@ -745,6 +1106,9 @@ async fn cmf_port_check(State(state): State<SharedState>, Query(q): Query<PortQu
ok(json!({ ok(json!({
"port": port, "port": port,
"available": available, "available": available,
// true = held by this gateway's own managed server (a replaced/edited
// row frees it on restart) — the UI shows this as info, not an error
"managed": running_here,
"detail": detail, "detail": detail,
"suggested": suggested, "suggested": suggested,
})) }))
@@ -914,6 +1278,13 @@ async fn get_stats(State(state): State<SharedState>, Query(q): Query<StatsQuery>
ok(snap) ok(snap)
} }
/// Clear all recorded stats/logs (in-memory aggregates, series, recent requests)
/// and truncate the JSONL log — the dashboard "Clear logs" button.
async fn clear_stats(State(state): State<SharedState>) -> ApiResult {
state.stats.clear();
ok(json!({ "ok": true }))
}
/// Self-warming shadow loop: per-task-type promotion state, Wilson-LB of the /// Self-warming shadow loop: per-task-type promotion state, Wilson-LB of the
/// judged pass-rate, and whether the local model is serving it yet. /// judged pass-rate, and whether the local model is serving it yet.
async fn get_shadow(State(state): State<SharedState>) -> ApiResult { async fn get_shadow(State(state): State<SharedState>) -> ApiResult {
@@ -974,10 +1345,7 @@ async fn start_import(
ok(json!({ "job": id })) ok(json!({ "job": id }))
} }
async fn import_status( async fn import_status(State(state): State<SharedState>, Path(job): Path<String>) -> ApiResult {
State(state): State<SharedState>,
Path(job): Path<String>,
) -> ApiResult {
match state.imports.get(&job) { match state.imports.get(&job) {
Some(j) => ok(serde_json::to_value(j).unwrap_or_else(|_| json!({}))), Some(j) => ok(serde_json::to_value(j).unwrap_or_else(|_| json!({}))),
None => Err(ApiError::not_found(format!("job '{job}' not found"))), None => Err(ApiError::not_found(format!("job '{job}' not found"))),
@@ -985,10 +1353,7 @@ async fn import_status(
} }
/// Cancel a running conversion: kills the converter and removes partial output. /// Cancel a running conversion: kills the converter and removes partial output.
async fn cancel_import( async fn cancel_import(State(state): State<SharedState>, Path(job): Path<String>) -> ApiResult {
State(state): State<SharedState>,
Path(job): Path<String>,
) -> ApiResult {
if state.imports.get(&job).is_none() { if state.imports.get(&job).is_none() {
return Err(ApiError::not_found(format!("job '{job}' not found"))); return Err(ApiError::not_found(format!("job '{job}' not found")));
} }
@@ -997,10 +1362,7 @@ async fn cancel_import(
} }
/// Delete a finished conversion and its converted `.cmf` file(s) from disk. /// Delete a finished conversion and its converted `.cmf` file(s) from disk.
async fn delete_import( async fn delete_import(State(state): State<SharedState>, Path(job): Path<String>) -> ApiResult {
State(state): State<SharedState>,
Path(job): Path<String>,
) -> ApiResult {
if state.imports.get(&job).is_none() { if state.imports.get(&job).is_none() {
return Err(ApiError::not_found(format!("job '{job}' not found"))); return Err(ApiError::not_found(format!("job '{job}' not found")));
} }
@@ -1009,10 +1371,7 @@ async fn delete_import(
} }
/// Register a finished `.cmf` as a local model (OpenAI provider → cortiq-server). /// Register a finished `.cmf` as a local model (OpenAI provider → cortiq-server).
async fn register_import( async fn register_import(State(state): State<SharedState>, Path(job): Path<String>) -> ApiResult {
State(state): State<SharedState>,
Path(job): Path<String>,
) -> ApiResult {
let j = state let j = state
.imports .imports
.get(&job) .get(&job)
+206 -85
View File
@@ -1,51 +1,68 @@
//! Local CMF model server lifecycle. //! Local CMF model server lifecycle.
//! //!
//! Closes the loop with <https://github.com/infosave2007/cmf>: install the //! Closes the loop with <https://github.com/infosave2007/cmf>: install the
//! `cortiq` CLI from crates.io, keep it up to date, and run `cortiq serve` as a //! `cortiq` CLI from crates.io, keep it up to date, and run one `cortiq serve`
//! managed child process. The served model exposes an OpenAI-compatible API, so //! per configured local model as a managed child process. Each served model
//! the registry registers it as an ordinary provider and the gateway routes to //! exposes an OpenAI-compatible API, so the registry registers it as an ordinary
//! it like any other backend. //! provider and the gateway routes to it like any other backend.
use crate::config::CmfCfg; use crate::config::{CmfCfg, CmfServer};
use std::process::Stdio; use std::process::Stdio;
use std::sync::{Arc, Mutex}; use std::sync::{Arc, Mutex};
use std::time::Duration; use std::time::Duration;
use tokio::process::{Child, Command}; use tokio::process::{Child, Command};
/// Observable state of the managed local CMF server (surfaced to the admin API). /// Observable state of one managed local model server.
#[derive(Clone, Debug, Default, serde::Serialize)]
pub struct CmfServerStatus {
/// Pool id it is registered under.
pub id: String,
/// Path of the `.cmf` being served.
pub model: String,
/// Port the server binds to.
pub port: u16,
/// OpenAI base URL the local server is registered under.
pub base_url: String,
/// The child `cortiq serve` process has been spawned.
pub running: bool,
/// `/healthz` on the local server answered 200.
pub healthy: bool,
/// Last fatal error for this server, if any.
pub last_error: Option<String>,
}
/// Observable state of the managed local CMF layer (surfaced to the admin API).
#[derive(Clone, Debug, Default, serde::Serialize)] #[derive(Clone, Debug, Default, serde::Serialize)]
pub struct CmfStatus { pub struct CmfStatus {
/// `manage_server` is on and a `local_model` is configured. /// `manage_server` is on and at least one local model is configured.
pub enabled: bool, pub enabled: bool,
/// Version reported by `cortiq --version` (None = not installed). /// Version reported by `cortiq --version` (None = not installed).
pub installed_version: Option<String>, pub installed_version: Option<String>,
/// Newest `cortiq-cli` on crates.io (populated when `auto_update` runs). /// Newest `cortiq-cli` on crates.io (populated when `auto_update` runs).
pub latest_version: Option<String>, pub latest_version: Option<String>,
/// The child `cortiq serve` process has been spawned. /// Per-model server status.
pub running: bool, pub servers: Vec<CmfServerStatus>,
/// `/healthz` on the local server answered 200.
pub healthy: bool,
/// OpenAI base URL the local server is registered under.
pub base_url: String,
/// Path of the `.cmf` being served.
pub model: String,
/// Last fatal error, if any.
pub last_error: Option<String>,
/// Recent lifecycle log lines (bounded). /// Recent lifecycle log lines (bounded).
pub log: Vec<String>, pub log: Vec<String>,
} }
/// Holds the managed child process and its live status. /// A spawned server: its pool id and the child process handle.
struct Managed {
id: String,
child: Child,
}
/// Holds the managed child processes and their live status.
pub struct CmfRuntime { pub struct CmfRuntime {
status: Mutex<CmfStatus>, status: Mutex<CmfStatus>,
child: Mutex<Option<Child>>, children: Mutex<Vec<Managed>>,
} }
impl CmfRuntime { impl CmfRuntime {
pub fn new() -> Arc<Self> { pub fn new() -> Arc<Self> {
Arc::new(Self { Arc::new(Self {
status: Mutex::new(CmfStatus::default()), status: Mutex::new(CmfStatus::default()),
child: Mutex::new(None), children: Mutex::new(Vec::new()),
}) })
} }
@@ -59,32 +76,115 @@ impl CmfRuntime {
let mut s = self.status.lock().unwrap(); let mut s = self.status.lock().unwrap();
s.log.push(msg); s.log.push(msg);
let n = s.log.len(); let n = s.log.len();
if n > 50 { if n > 80 {
s.log.drain(0..n - 50); s.log.drain(0..n - 80);
} }
} }
/// A global (install/update) failure — not tied to a single server.
fn fail(&self, msg: impl Into<String>) { fn fail(&self, msg: impl Into<String>) {
let msg = msg.into(); let msg = msg.into();
tracing::warn!(target: "cmf", "{msg}"); tracing::warn!(target: "cmf", "{msg}");
self.status.lock().unwrap().log.push(format!("✗ {msg}"));
}
/// Record a fatal error against a specific server (and the log).
fn fail_server(&self, id: &str, msg: impl Into<String>) {
let msg = msg.into();
tracing::warn!(target: "cmf", "[{id}] {msg}");
let mut s = self.status.lock().unwrap(); let mut s = self.status.lock().unwrap();
s.last_error = Some(msg.clone()); if let Some(sv) = s.servers.iter_mut().find(|x| x.id == id) {
s.log.push(format!("✗ {msg}")); sv.last_error = Some(msg.clone());
}
s.log.push(format!("✗ [{id}] {msg}"));
} }
fn set_child(&self, child: Child) { /// Seed the per-server status slots from the effective server list.
*self.child.lock().unwrap() = Some(child); fn init_servers(&self, servers: &[CmfServer], host: &str) {
self.status.lock().unwrap().running = true; let mut s = self.status.lock().unwrap();
s.enabled = true;
s.servers = servers
.iter()
.map(|sv| CmfServerStatus {
id: sv.id.clone(),
model: sv.model.clone(),
port: sv.port,
base_url: format!("http://{host}:{}/v1", sv.port),
running: false,
healthy: false,
last_error: None,
})
.collect();
} }
/// Stop the managed server (used on shutdown / reconfigure). fn set_running(&self, id: &str, running: bool) {
if let Some(sv) = self
.status
.lock()
.unwrap()
.servers
.iter_mut()
.find(|x| x.id == id)
{
sv.running = running;
}
}
fn set_healthy(&self, id: &str, healthy: bool) {
if let Some(sv) = self
.status
.lock()
.unwrap()
.servers
.iter_mut()
.find(|x| x.id == id)
{
sv.healthy = healthy;
}
}
fn add_child(&self, id: String, child: Child) {
self.children.lock().unwrap().push(Managed {
id: id.clone(),
child,
});
self.set_running(&id, true);
}
fn is_running(&self, id: &str) -> bool {
self.status
.lock()
.unwrap()
.servers
.iter()
.any(|s| s.id == id && s.running)
}
/// Stop all managed servers (used on shutdown / reconfigure).
pub async fn stop(&self) { pub async fn stop(&self) {
if let Some(mut child) = self.child.lock().unwrap().take() { let mut kids = std::mem::take(&mut *self.children.lock().unwrap());
let _ = child.start_kill(); for m in kids.iter_mut() {
let _ = m.child.start_kill();
} }
let mut s = self.status.lock().unwrap(); let mut s = self.status.lock().unwrap();
s.running = false; for sv in s.servers.iter_mut() {
s.healthy = false; sv.running = false;
sv.healthy = false;
}
}
/// Stop and forget a single managed server (used when a model is deleted).
pub async fn stop_one(&self, id: &str) {
let removed = {
let mut kids = self.children.lock().unwrap();
kids.iter()
.position(|m| m.id == id)
.map(|pos| kids.remove(pos))
};
if let Some(mut m) = removed {
let _ = m.child.start_kill();
}
self.status.lock().unwrap().servers.retain(|sv| sv.id != id);
} }
} }
@@ -92,7 +192,7 @@ impl Default for CmfRuntime {
fn default() -> Self { fn default() -> Self {
Self { Self {
status: Mutex::new(CmfStatus::default()), status: Mutex::new(CmfStatus::default()),
child: Mutex::new(None), children: Mutex::new(Vec::new()),
} }
} }
} }
@@ -113,14 +213,15 @@ async fn latest_crates_version() -> Option<String> {
.await .await
.ok()?; .ok()?;
let v: serde_json::Value = serde_json::from_str(&body).ok()?; let v: serde_json::Value = serde_json::from_str(&body).ok()?;
v["crate"]["newest_version"] v["crate"]["newest_version"].as_str().map(|s| s.to_string())
.as_str()
.map(|s| s.to_string())
} }
/// `<bin> --version` → the trailing version token (e.g. "cortiq 0.1.2" → "0.1.2"). /// `<bin> --version` → the trailing version token (e.g. "cortiq 0.1.2" → "0.1.2").
pub(crate) fn installed_version(bin: &str) -> Option<String> { pub(crate) fn installed_version(bin: &str) -> Option<String> {
let out = std::process::Command::new(bin).arg("--version").output().ok()?; let out = std::process::Command::new(bin)
.arg("--version")
.output()
.ok()?;
if !out.status.success() { if !out.status.success() {
return None; return None;
} }
@@ -130,10 +231,14 @@ pub(crate) fn installed_version(bin: &str) -> Option<String> {
fn parse_ver(s: &str) -> (u32, u32, u32) { fn parse_ver(s: &str) -> (u32, u32, u32) {
let mut it = s.split('.').map(|x| x.trim().parse::<u32>().unwrap_or(0)); let mut it = s.split('.').map(|x| x.trim().parse::<u32>().unwrap_or(0));
(it.next().unwrap_or(0), it.next().unwrap_or(0), it.next().unwrap_or(0)) (
it.next().unwrap_or(0),
it.next().unwrap_or(0),
it.next().unwrap_or(0),
)
} }
fn version_lt(a: &str, b: &str) -> bool { pub(crate) fn version_lt(a: &str, b: &str) -> bool {
parse_ver(a) < parse_ver(b) parse_ver(a) < parse_ver(b)
} }
@@ -161,30 +266,33 @@ pub async fn install_now(rt: Arc<CmfRuntime>, bin: String) {
Ok(()) => { Ok(()) => {
let v = installed_version(&bin); let v = installed_version(&bin);
rt.status.lock().unwrap().installed_version = v.clone(); rt.status.lock().unwrap().installed_version = v.clone();
rt.log(format!("installed cortiq-cli {}", v.as_deref().unwrap_or("?"))); rt.log(format!(
"installed cortiq-cli {}",
v.as_deref().unwrap_or("?")
));
} }
Err(e) => rt.fail(format!("cargo install cortiq-cli failed: {e}")), Err(e) => rt.fail(format!("cargo install cortiq-cli failed: {e}")),
} }
} }
/// Spawn `cortiq serve <model> --host <host> --port <port>`; the child is killed /// Spawn `cortiq serve <model> --host <host> --port <port>` for one server; the
/// when its handle is dropped (i.e. when the gateway shuts down). /// child is killed when its handle is dropped (i.e. when the gateway shuts down).
fn spawn_serve(cfg: &CmfCfg) -> Result<Child, String> { fn spawn_serve(server: &CmfServer, cfg: &CmfCfg) -> Result<Child, String> {
let mut cmd = Command::new(&cfg.cortiq_bin); let mut cmd = Command::new(&cfg.cortiq_bin);
cmd.arg("serve") cmd.arg("serve")
.arg(&cfg.local_model) .arg(&server.model)
.arg("--host") .arg("--host")
.arg(&cfg.local_host) .arg(&cfg.local_host)
.arg("--port") .arg("--port")
.arg(cfg.local_port.to_string()); .arg(server.port.to_string());
// Parallelise decode matvecs across cores (CMF_THREADS). 0 = leave the // Parallelise decode matvecs across cores (CMF_THREADS). 0/1 = leave the
// runtime's own default; N>1 gave ~2.8x on a 15B CPU model. // runtime's own default; N>1 gave ~2.8x on a 15B CPU model.
if cfg.threads > 1 { if server.threads > 1 {
cmd.env("CMF_THREADS", cfg.threads.to_string()); cmd.env("CMF_THREADS", server.threads.to_string());
} }
// Offload large matvecs to the GPU (Metal on macOS). Memory-bandwidth-bound // Offload large matvecs to the GPU (Metal on macOS). Memory-bandwidth-bound
// on unified memory, so mostly a prefill win; opt-in. // on unified memory, so mostly a prefill win; opt-in.
if cfg.gpu { if server.gpu {
cmd.env("CMF_GPU", "1"); cmd.env("CMF_GPU", "1");
} }
cmd.stdout(Stdio::null()) cmd.stdout(Stdio::null())
@@ -214,24 +322,17 @@ async fn wait_healthy(host: &str, port: u16, tries: u32) -> bool {
false false
} }
/// Orchestrate the whole lifecycle: ensure installed → maybe update → spawn → /// Orchestrate the whole lifecycle: ensure installed → maybe update → spawn each
/// wait healthy. Safe to call in a background task; it never panics. /// configured local model → wait healthy. Safe to call in a background task; it
/// never panics.
pub async fn manage(rt: Arc<CmfRuntime>, cfg: CmfCfg) { pub async fn manage(rt: Arc<CmfRuntime>, cfg: CmfCfg) {
if !cfg.manage_server || cfg.local_model.trim().is_empty() { let servers = cfg.effective_servers();
return; if servers.is_empty() {
}
{
let mut s = rt.status.lock().unwrap();
s.enabled = true;
s.base_url = format!("http://{}:{}/v1", cfg.local_host, cfg.local_port);
s.model = cfg.local_model.clone();
}
if !std::path::Path::new(&cfg.local_model).exists() {
rt.fail(format!("local_model not found: {}", cfg.local_model));
return; return;
} }
rt.init_servers(&servers, &cfg.local_host);
// 1. Ensure the `cortiq` binary is available. // 1. Ensure the `cortiq` binary is available (once for all servers).
let mut ver = installed_version(&cfg.cortiq_bin); let mut ver = installed_version(&cfg.cortiq_bin);
if ver.is_none() { if ver.is_none() {
if cfg.auto_install { if cfg.auto_install {
@@ -245,7 +346,9 @@ pub async fn manage(rt: Arc<CmfRuntime>, cfg: CmfCfg) {
rt.log("installed cortiq-cli"); rt.log("installed cortiq-cli");
} }
} else { } else {
rt.fail("cortiq binary not found (set [cmf].auto_install = true, or install cortiq-cli)"); rt.fail(
"cortiq binary not found (set [cmf].auto_install = true, or install cortiq-cli)",
);
return; return;
} }
} }
@@ -255,8 +358,15 @@ pub async fn manage(rt: Arc<CmfRuntime>, cfg: CmfCfg) {
if cfg.auto_update { if cfg.auto_update {
if let Some(latest) = latest_crates_version().await { if let Some(latest) = latest_crates_version().await {
rt.status.lock().unwrap().latest_version = Some(latest.clone()); rt.status.lock().unwrap().latest_version = Some(latest.clone());
if ver.as_deref().map(|c| version_lt(c, &latest)).unwrap_or(false) { if ver
rt.log(format!("updating cortiq-cli {} → {latest}…", ver.as_deref().unwrap_or("?"))); .as_deref()
.map(|c| version_lt(c, &latest))
.unwrap_or(false)
{
rt.log(format!(
"updating cortiq-cli {} → {latest}…",
ver.as_deref().unwrap_or("?")
));
if let Err(e) = cargo_install(true).await { if let Err(e) = cargo_install(true).await {
rt.log(format!("update failed (keeping current): {e}")); rt.log(format!("update failed (keeping current): {e}"));
} else { } else {
@@ -268,27 +378,38 @@ pub async fn manage(rt: Arc<CmfRuntime>, cfg: CmfCfg) {
} }
} }
// 3. Spawn the local server. // 3. Spawn each server (fast).
rt.log(format!( for server in &servers {
"starting: cortiq serve {} --host {} --port {}", if !std::path::Path::new(&server.model).exists() {
cfg.local_model, cfg.local_host, cfg.local_port rt.fail_server(
)); &server.id,
match spawn_serve(&cfg) { format!("model file not found: {}", server.model),
Ok(child) => rt.set_child(child), );
Err(e) => { continue;
rt.fail(format!("failed to spawn cortiq serve: {e}")); }
return; rt.log(format!(
"starting: cortiq serve {} --host {} --port {}",
server.model, cfg.local_host, server.port
));
match spawn_serve(server, &cfg) {
Ok(child) => rt.add_child(server.id.clone(), child),
Err(e) => rt.fail_server(&server.id, format!("failed to spawn: {e}")),
} }
} }
// 4. Wait until it is serving (model load can take a while for big models). // 4. Wait until each spawned server is serving (model load can be slow).
if wait_healthy(&cfg.local_host, cfg.local_port, 240).await { for server in &servers {
rt.status.lock().unwrap().healthy = true; if !rt.is_running(&server.id) {
rt.log(format!( continue; // failed to spawn / file missing
"local CMF server ready — registered as model '{}' at http://{}:{}/v1", }
cfg.model_id, cfg.local_host, cfg.local_port if wait_healthy(&cfg.local_host, server.port, 240).await {
)); rt.set_healthy(&server.id, true);
} else { rt.log(format!(
rt.fail("local CMF server did not become healthy in time"); "local CMF server ready — '{}' at http://{}:{}/v1",
server.id, cfg.local_host, server.port
));
} else {
rt.fail_server(&server.id, "did not become healthy in time");
}
} }
} }
+69 -5
View File
@@ -10,10 +10,18 @@ use std::path::PathBuf;
/// `~/.cortiq-gateway` — so defaults work regardless of the process CWD /// `~/.cortiq-gateway` — so defaults work regardless of the process CWD
/// (important once the gateway is installed rather than run from its repo). /// (important once the gateway is installed rather than run from its repo).
pub fn data_dir() -> PathBuf { pub fn data_dir() -> PathBuf {
for (var, sub) in [("CORTIQ_GATEWAY_HOME", ""), ("XDG_DATA_HOME", "cortiq-gateway"), ("HOME", ".cortiq-gateway")] { for (var, sub) in [
("CORTIQ_GATEWAY_HOME", ""),
("XDG_DATA_HOME", "cortiq-gateway"),
("HOME", ".cortiq-gateway"),
] {
if let Ok(v) = std::env::var(var) { if let Ok(v) = std::env::var(var) {
if !v.is_empty() { if !v.is_empty() {
return if sub.is_empty() { PathBuf::from(v) } else { PathBuf::from(v).join(sub) }; return if sub.is_empty() {
PathBuf::from(v)
} else {
PathBuf::from(v).join(sub)
};
} }
} }
} }
@@ -59,6 +67,24 @@ pub struct Config {
pub cmf: CmfCfg, pub cmf: CmfCfg,
} }
/// One managed local model: the gateway runs a dedicated `cortiq serve` for it
/// on its own port and registers it in the pool under `id`.
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
pub struct CmfServer {
/// Pool id it is registered under (e.g. "cmf-qwen3-5-2b").
pub id: String,
/// Path to the `.cmf` to serve.
pub model: String,
/// TCP port the dedicated `cortiq serve` binds to (must be unique).
pub port: u16,
/// Worker threads (CMF_THREADS). 0 = runtime default.
#[serde(default = "default_cmf_threads")]
pub threads: usize,
/// Offload matvecs to the GPU (CMF_GPU=1).
#[serde(default)]
pub gpu: bool,
}
/// CMF-format model factory: import a HuggingFace model, convert it to a /// CMF-format model factory: import a HuggingFace model, convert it to a
/// local `.cmf` (quantized), and serve it via cortiq-server. Points the /// local `.cmf` (quantized), and serve it via cortiq-server. Points the
/// gateway at the Python converter + runtime on the host. /// gateway at the Python converter + runtime on the host.
@@ -121,6 +147,35 @@ pub struct CmfCfg {
/// (helps prefill of long prompts). Off by default. /// (helps prefill of long prompts). Off by default.
#[serde(default)] #[serde(default)]
pub gpu: bool, pub gpu: bool,
/// Managed local models. When non-empty (and `manage_server` is on), the
/// gateway spawns and supervises one `cortiq serve` per entry — each on its
/// own port — and registers each in the pool. Empty = fall back to the single
/// legacy `local_model` / `local_port` / `model_id` below (back-compat).
#[serde(default)]
pub servers: Vec<CmfServer>,
}
impl CmfCfg {
/// The managed local models actually in effect: the `servers` list when set,
/// otherwise the single legacy `local_model` (as a one-element list). Empty
/// when `manage_server` is off or nothing is configured.
pub fn effective_servers(&self) -> Vec<CmfServer> {
if !self.manage_server {
return Vec::new();
}
if !self.servers.is_empty() {
return self.servers.clone();
}
if !self.local_model.trim().is_empty() {
return vec![CmfServer {
id: self.model_id.clone(),
model: self.local_model.clone(),
port: self.local_port,
threads: self.threads,
gpu: self.gpu,
}];
}
Vec::new()
}
} }
impl Default for CmfCfg { impl Default for CmfCfg {
fn default() -> Self { fn default() -> Self {
@@ -141,6 +196,7 @@ impl Default for CmfCfg {
local_only: false, local_only: false,
threads: default_cmf_threads(), threads: default_cmf_threads(),
gpu: false, gpu: false,
servers: Vec::new(),
} }
} }
} }
@@ -597,7 +653,8 @@ impl Config {
// A managed local CMF model counts as an available model even though it // A managed local CMF model counts as an available model even though it
// is registered dynamically rather than via a [[models]] entry — this // is registered dynamically rather than via a [[models]] entry — this
// is what lets the gateway run on local models alone (no cloud, no router). // is what lets the gateway run on local models alone (no cloud, no router).
let has_local = self.cmf.manage_server && !self.cmf.local_model.trim().is_empty(); let servers = self.cmf.effective_servers();
let has_local = !servers.is_empty();
if self.models.is_empty() && !has_local { if self.models.is_empty() && !has_local {
// Fresh install: start with an EMPTY model pool so the admin console // Fresh install: start with an EMPTY model pool so the admin console
// comes up — the user adds models in /admin → Models (or points [cmf] // comes up — the user adds models in /admin → Models (or points [cmf]
@@ -616,8 +673,15 @@ impl Config {
// every target in [routing] must exist in the model pool // every target in [routing] must exist in the model pool
let mut ids: std::collections::HashSet<&str> = let mut ids: std::collections::HashSet<&str> =
self.models.iter().map(|m| m.id.as_str()).collect(); self.models.iter().map(|m| m.id.as_str()).collect();
if has_local { // managed local servers are known ids too; also enforce unique ports/ids
ids.insert(self.cmf.model_id.as_str()); let mut ports = std::collections::HashSet::new();
for s in &servers {
if !ids.insert(s.id.as_str()) {
anyhow::bail!("config: duplicate managed model id '{}'", s.id);
}
if !ports.insert(s.port) {
anyhow::bail!("config: two managed models share port {}", s.port);
}
} }
if !self.routing.default.is_empty() && !ids.contains(self.routing.default.as_str()) { if !self.routing.default.is_empty() && !ids.contains(self.routing.default.as_str()) {
anyhow::bail!( anyhow::bail!(
+155 -29
View File
@@ -31,13 +31,19 @@ fn cleanup_output(path: &str) {
} }
fn now() -> u64 { fn now() -> u64 {
SystemTime::now().duration_since(UNIX_EPOCH).unwrap_or_default().as_secs() SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap_or_default()
.as_secs()
} }
/// Job id from a hash of (repo, nanos) — unique without a uuid dependency. /// Job id from a hash of (repo, nanos) — unique without a uuid dependency.
fn gen_id(repo: &str) -> String { fn gen_id(repo: &str) -> String {
use std::hash::{Hash, Hasher}; use std::hash::{Hash, Hasher};
let nanos = SystemTime::now().duration_since(UNIX_EPOCH).unwrap_or_default().as_nanos(); let nanos = SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap_or_default()
.as_nanos();
let mut h = std::collections::hash_map::DefaultHasher::new(); let mut h = std::collections::hash_map::DefaultHasher::new();
repo.hash(&mut h); repo.hash(&mut h);
nanos.hash(&mut h); nanos.hash(&mut h);
@@ -65,6 +71,20 @@ pub struct ImportParams {
pub shard_max_gb: Option<f32>, pub shard_max_gb: Option<f32>,
#[serde(default)] #[serde(default)]
pub skip_mtp: bool, pub skip_mtp: bool,
/// O(1) Nyström attention hint (native converter, cortiq ≥ 0.2.0):
/// `all` | `deepN` | `i,j,k`. Weights pass through unchanged — the runtime
/// reads the hint at load. Empty/None/"off" = exact attention.
#[serde(default)]
pub o1: Option<String>,
/// Landmark budget for the o1 hint (validated default 32).
#[serde(default)]
pub o1_m: Option<usize>,
/// Exact-window width for the o1 hint (validated default 128).
#[serde(default)]
pub o1_window: Option<usize>,
/// Permanent exact sink keys for the o1 hint (validated default 4).
#[serde(default)]
pub o1_sink: Option<usize>,
} }
fn default_quant() -> String { fn default_quant() -> String {
"Q8_2F".into() "Q8_2F".into()
@@ -82,8 +102,8 @@ pub struct Job {
pub started: u64, pub started: u64,
pub finished: Option<u64>, pub finished: Option<u64>,
pub size_bytes: Option<u64>, pub size_bytes: Option<u64>,
pub progress: Option<f32>, // 0..1, parsed from converter @PROGRESS pub progress: Option<f32>, // 0..1, parsed from converter @PROGRESS
pub phase: Option<String>, // current phase label pub phase: Option<String>, // current phase label
} }
#[derive(Default)] #[derive(Default)]
@@ -178,16 +198,24 @@ impl JobStore {
} }
pub fn list(&self) -> Vec<Job> { pub fn list(&self) -> Vec<Job> {
let mut v: Vec<Job> = self.jobs.lock().unwrap().values().cloned().collect(); let mut v: Vec<Job> = self.jobs.lock().unwrap().values().cloned().collect();
v.sort_by(|a, b| b.started.cmp(&a.started)); v.sort_by_key(|x| std::cmp::Reverse(x.started));
v.truncate(20); v.truncate(20);
v v
} }
} }
/// Proxy HuggingFace model search (avoids browser CORS + adds our token). /// Proxy HuggingFace model search (avoids browser CORS + adds our token).
pub async fn hf_search(query: &str, limit: usize, token: Option<&str>) -> Result<serde_json::Value, String> { pub async fn hf_search(
query: &str,
limit: usize,
token: Option<&str>,
) -> Result<serde_json::Value, String> {
let q: String = url_escape(query); let q: String = url_escape(query);
let sort = if query.trim().is_empty() { "trendingScore" } else { "downloads" }; let sort = if query.trim().is_empty() {
"trendingScore"
} else {
"downloads"
};
let url = format!( let url = format!(
"https://huggingface.co/api/models?search={q}&sort={sort}&direction=-1&limit={limit}&full=false" "https://huggingface.co/api/models?search={q}&sort={sort}&direction=-1&limit={limit}&full=false"
); );
@@ -218,7 +246,13 @@ fn url_escape(s: &str) -> String {
fn sanitize(s: &str) -> String { fn sanitize(s: &str) -> String {
s.chars() s.chars()
.map(|c| if c.is_ascii_alphanumeric() || c == '-' || c == '_' { c } else { '-' }) .map(|c| {
if c.is_ascii_alphanumeric() || c == '-' || c == '_' {
c
} else {
'-'
}
})
.collect() .collect()
} }
@@ -244,7 +278,38 @@ pub fn start_import(store: Arc<JobStore>, cfg: &CmfCfg, p: ImportParams) -> Resu
// doesn't support yet (linear-attention folding, v-bit shaping). // doesn't support yet (linear-attention folding, v-bit shaping).
let use_python = p.linear_core.is_some() || p.vbit_shape.is_some() || p.mean_bits.is_some(); let use_python = p.linear_core.is_some() || p.vbit_shape.is_some() || p.mean_bits.is_some();
if use_python && !std::path::Path::new(&cfg.converter).exists() { if use_python && !std::path::Path::new(&cfg.converter).exists() {
return Err(format!("advanced options need the Python converter, not found: {}", cfg.converter)); return Err(format!(
"advanced options need the Python converter, not found: {}",
cfg.converter
));
}
// O(1) attention hint — native-converter feature (cortiq ≥ 0.2.0). Fail fast
// with a clear message instead of a mid-job "unexpected argument" error.
let o1 =
p.o1.as_deref()
.map(str::trim)
.filter(|s| !s.is_empty() && *s != "off")
.map(str::to_string);
if o1.is_some() {
if use_python {
return Err(
"O(1) attention is supported by the native converter only — remove the \
linear-core / v-bit shape / mean-bits options to use it"
.into(),
);
}
if p.repo.to_ascii_lowercase().contains("gguf") {
return Err("O(1) attention is not supported for GGUF imports yet".into());
}
let ver = crate::cmf_runtime::installed_version(&cfg.cortiq_bin);
if let Some(v) = &ver {
if crate::cmf_runtime::version_lt(v, "0.2.0") {
return Err(format!(
"O(1) attention needs cortiq ≥ 0.2.0 (installed: {v}) — update the runtime \
in Settings → Local models"
));
}
}
} }
let _ = std::fs::create_dir_all(&cfg.models_dir); let _ = std::fs::create_dir_all(&cfg.models_dir);
let base = if p.name.trim().is_empty() { let base = if p.name.trim().is_empty() {
@@ -288,18 +353,43 @@ pub fn start_import(store: Arc<JobStore>, cfg: &CmfCfg, p: ImportParams) -> Resu
let (program, args, workdir): (String, Vec<String>, std::path::PathBuf) = if use_python { let (program, args, workdir): (String, Vec<String>, std::path::PathBuf) = if use_python {
let conv = std::path::Path::new(&cfg.converter); let conv = std::path::Path::new(&cfg.converter);
let mut a = vec![ let mut a = vec![
std::fs::canonicalize(conv).map(|p| p.to_string_lossy().to_string()).unwrap_or_else(|_| cfg.converter.clone()), std::fs::canonicalize(conv)
"--model".into(), p.repo.clone(), .map(|p| p.to_string_lossy().to_string())
"--quant".into(), p.quant.clone(), .unwrap_or_else(|_| cfg.converter.clone()),
"--output".into(), output_abs.clone(), "--model".into(),
p.repo.clone(),
"--quant".into(),
p.quant.clone(),
"--output".into(),
output_abs.clone(),
]; ];
if let Some(lc) = &p.linear_core { a.push("--linear-core".into()); a.push(lc.clone()); } if let Some(lc) = &p.linear_core {
if let Some(n) = p.nphase { a.push("--nphase".into()); a.push(n.to_string()); } a.push("--linear-core".into());
if let Some(vs) = &p.vbit_shape { a.push("--vbit-shape".into()); a.push(vs.clone()); } a.push(lc.clone());
if let Some(mb) = p.mean_bits { a.push("--mean-bits".into()); a.push(mb.to_string()); } }
if let Some(g) = p.shard_max_gb { a.push("--shard-max-gb".into()); a.push(g.to_string()); } if let Some(n) = p.nphase {
if p.skip_mtp { a.push("--skip-mtp".into()); } a.push("--nphase".into());
let wd = conv.parent().map(|d| d.to_path_buf()).unwrap_or_else(|| std::path::PathBuf::from(".")); a.push(n.to_string());
}
if let Some(vs) = &p.vbit_shape {
a.push("--vbit-shape".into());
a.push(vs.clone());
}
if let Some(mb) = p.mean_bits {
a.push("--mean-bits".into());
a.push(mb.to_string());
}
if let Some(g) = p.shard_max_gb {
a.push("--shard-max-gb".into());
a.push(g.to_string());
}
if p.skip_mtp {
a.push("--skip-mtp".into());
}
let wd = conv
.parent()
.map(|d| d.to_path_buf())
.unwrap_or_else(|| std::path::PathBuf::from("."));
(cfg.python_bin.clone(), a, wd) (cfg.python_bin.clone(), a, wd)
} else if p.repo.to_ascii_lowercase().contains("gguf") { } else if p.repo.to_ascii_lowercase().contains("gguf") {
// GGUF repo → native `cortiq import-gguf` (downloads + dequantizes any // GGUF repo → native `cortiq import-gguf` (downloads + dequantizes any
@@ -307,21 +397,57 @@ pub fn start_import(store: Arc<JobStore>, cfg: &CmfCfg, p: ImportParams) -> Resu
let mut a = vec![ let mut a = vec![
"import-gguf".into(), "import-gguf".into(),
p.repo.clone(), p.repo.clone(),
"--quant".into(), map_quant(&p.quant).into(), "--quant".into(),
"--output".into(), output_abs.clone(), map_quant(&p.quant).into(),
"--output".into(),
output_abs.clone(),
]; ];
if let Some(t) = &hf_token { a.push("--hf-token".into()); a.push(t.clone()); } if let Some(t) = &hf_token {
(cfg.cortiq_bin.clone(), a, std::env::current_dir().unwrap_or_else(|_| std::path::PathBuf::from("."))) a.push("--hf-token".into());
a.push(t.clone());
}
(
cfg.cortiq_bin.clone(),
a,
std::env::current_dir().unwrap_or_else(|_| std::path::PathBuf::from(".")),
)
} else { } else {
// safetensors → native `cortiq convert` (dense / MoE / GatedDeltaNet). // safetensors → native `cortiq convert` (dense / MoE / GatedDeltaNet).
let mut a = vec![ let mut a = vec![
"convert".into(), "convert".into(),
"--model".into(), p.repo.clone(), "--model".into(),
"--quant".into(), map_quant(&p.quant).into(), p.repo.clone(),
"--output".into(), output_abs.clone(), "--quant".into(),
map_quant(&p.quant).into(),
"--output".into(),
output_abs.clone(),
]; ];
if let Some(t) = &hf_token { a.push("--hf-token".into()); a.push(t.clone()); } if let Some(t) = &hf_token {
(cfg.cortiq_bin.clone(), a, std::env::current_dir().unwrap_or_else(|_| std::path::PathBuf::from("."))) a.push("--hf-token".into());
a.push(t.clone());
}
// O(1) attention hint (weights unchanged; the runtime reads it at load)
if let Some(spec) = &o1 {
a.push("--o1".into());
a.push(spec.clone());
if let Some(m) = p.o1_m {
a.push("--o1-m".into());
a.push(m.to_string());
}
if let Some(w) = p.o1_window {
a.push("--o1-window".into());
a.push(w.to_string());
}
if let Some(s) = p.o1_sink {
a.push("--o1-sink".into());
a.push(s.to_string());
}
}
(
cfg.cortiq_bin.clone(),
a,
std::env::current_dir().unwrap_or_else(|_| std::path::PathBuf::from(".")),
)
}; };
let ret_id = id.clone(); let ret_id = id.clone();
+44 -12
View File
@@ -208,15 +208,25 @@ impl Pipeline {
}; };
let (decision, route_source, candidates) = if live.cfg.cmf.local_only { let (decision, route_source, candidates) = if live.cfg.cmf.local_only {
(mk("local-only", "local"), "local".to_string(), live.local_candidates()) (
mk("local-only", "local"),
"local".to_string(),
live.local_candidates(),
)
} else { } else {
match &req.routing { match &req.routing {
RoutingDirective::Pinned { model_id } => { RoutingDirective::Pinned { model_id } => (
(mk("pinned", "pinned"), "pinned".to_string(), vec![model_id.clone()]) mk("pinned", "pinned"),
} "pinned".to_string(),
vec![model_id.clone()],
),
RoutingDirective::Auto { profile } => { RoutingDirective::Auto { profile } => {
if !live.cfg.router.enabled { if !live.cfg.router.enabled {
(mk("router-disabled", "local"), "local".to_string(), live.local_candidates()) (
mk("router-disabled", "local"),
"local".to_string(),
live.local_candidates(),
)
} else { } else {
let prof = profile.as_deref().unwrap_or(&live.cfg.route.profile); let prof = profile.as_deref().unwrap_or(&live.cfg.route.profile);
match live.router.route(&text, prof).await { match live.router.route(&text, prof).await {
@@ -229,7 +239,11 @@ impl Pipeline {
tracing::warn!( tracing::warn!(
"Router is unavailable. Gracefully degrading to the local/default model." "Router is unavailable. Gracefully degrading to the local/default model."
); );
(mk("degraded", "fallback"), "fallback".to_string(), live.local_candidates()) (
mk("degraded", "fallback"),
"fallback".to_string(),
live.local_candidates(),
)
} }
} }
} }
@@ -253,9 +267,21 @@ impl Pipeline {
} }
} }
// Local-first safety net: if routing produced no usable candidate (e.g. // An explicit pin must never silently answer from a different model: if
// an unknown pinned model while the router is off), fall back to the // the pinned id is unknown or was filtered out, fail loudly so the caller
// managed local model so a configured local backend still answers. // knows their choice was not honoured (instead of getting a stray reply
// from the managed local model via the safety net below).
if final_candidates.is_empty() {
if let RoutingDirective::Pinned { model_id } = &req.routing {
return Err(GatewayError::UpstreamUnavailable(format!(
"pinned model '{model_id}' is not available — it is not in the pool (deleted?) or lacks a required capability such as tools"
)));
}
}
// Local-first safety net: for AUTO/degraded routes only, if routing
// produced no usable candidate, fall back to the managed local model so a
// configured local backend still answers.
if final_candidates.is_empty() && live.cfg.cmf.manage_server { if final_candidates.is_empty() && live.cfg.cmf.manage_server {
for id in live.local_candidates() { for id in live.local_candidates() {
if let Some(prov) = live.registry.get(&id) { if let Some(prov) = live.registry.get(&id) {
@@ -289,7 +315,10 @@ impl Pipeline {
&& !sh.local_model_id.is_empty() && !sh.local_model_id.is_empty()
&& matches!(req.routing, RoutingDirective::Auto { .. }) && matches!(req.routing, RoutingDirective::Auto { .. })
&& state.promotion.serves_local(&decision.task_label) && state.promotion.serves_local(&decision.task_label)
&& !matches!(decision.complexity_tier.as_str(), "high" | "hard" | "complex") && !matches!(
decision.complexity_tier.as_str(),
"high" | "hard" | "complex"
)
&& !final_candidates.iter().any(|m| m == &sh.local_model_id) && !final_candidates.iter().any(|m| m == &sh.local_model_id)
{ {
final_candidates.insert(0, sh.local_model_id.clone()); final_candidates.insert(0, sh.local_model_id.clone());
@@ -713,7 +742,7 @@ fn parse_judge(text: &str) -> (u8, bool) {
let score = text let score = text
.split("score") .split("score")
.nth(1) .nth(1)
.and_then(|t| t.chars().skip_while(|c| !c.is_ascii_digit()).next()) .and_then(|t| t.chars().find(|c| c.is_ascii_digit()))
.and_then(|c| c.to_digit(10)) .and_then(|c| c.to_digit(10))
.map(|d| d.min(4) as u8); .map(|d| d.min(4) as u8);
let hard_fail = text let hard_fail = text
@@ -747,7 +776,10 @@ async fn shadow_capture(
}; };
// 1. Local answer (free, CPU). // 1. Local answer (free, CPU).
let a_local = match local.chat(shadow_req(&cfg.local_model_id, clip(&prompt, 4000), 512)).await { let a_local = match local
.chat(shadow_req(&cfg.local_model_id, clip(&prompt, 4000), 512))
.await
{
Ok(r) => r Ok(r) => r
.choices .choices
.into_iter() .into_iter()
+96 -30
View File
@@ -18,7 +18,10 @@ use std::sync::Mutex;
use std::time::{SystemTime, UNIX_EPOCH}; use std::time::{SystemTime, UNIX_EPOCH};
fn now_secs() -> u64 { fn now_secs() -> u64 {
SystemTime::now().duration_since(UNIX_EPOCH).unwrap_or_default().as_secs() SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap_or_default()
.as_secs()
} }
/// One judgment of a local answer (JSONL line, replayed on startup). /// One judgment of a local answer (JSONL line, replayed on startup).
@@ -62,14 +65,14 @@ pub enum PromotionState {
#[derive(Clone, Debug)] #[derive(Clone, Debug)]
pub struct PromotionCfg { pub struct PromotionCfg {
pub enabled: bool, pub enabled: bool,
pub window: usize, // rolling judgments per label (W) pub window: usize, // rolling judgments per label (W)
pub n_min: usize, // min judgments before promotion is even considered pub n_min: usize, // min judgments before promotion is even considered
pub promote_lb: f64, // Wilson-95 LB of pass must clear this pub promote_lb: f64, // Wilson-95 LB of pass must clear this
pub demote_pass: f64, // point pass-rate below this → demote a served label pub demote_pass: f64, // point pass-rate below this → demote a served label
pub hardfail_ub_max: f64,// Wilson-95 UPPER bound of hard_fail must stay under this pub hardfail_ub_max: f64, // Wilson-95 UPPER bound of hard_fail must stay under this
pub soak: usize, // extra Canary judgments before LocalServed pub soak: usize, // extra Canary judgments before LocalServed
pub z: f64, // 1.96 = 95% pub z: f64, // 1.96 = 95%
pub file: Option<String>,// judge.jsonl path (append + replay) pub file: Option<String>, // judge.jsonl path (append + replay)
} }
impl Default for PromotionCfg { impl Default for PromotionCfg {
@@ -92,7 +95,7 @@ struct LabelState {
passes: VecDeque<bool>, // rolling window of pass/fail passes: VecDeque<bool>, // rolling window of pass/fail
hard_fails: VecDeque<bool>, // rolling window of hard_fail flags hard_fails: VecDeque<bool>, // rolling window of hard_fail flags
state: PromotionState, state: PromotionState,
soaked: usize, // judgments accumulated while in Canary soaked: usize, // judgments accumulated while in Canary
} }
impl LabelState { impl LabelState {
@@ -124,20 +127,29 @@ pub fn wilson_bound(s: usize, n: usize, z: f64, upper: bool) -> f64 {
let denom = 1.0 + z2 / n; let denom = 1.0 + z2 / n;
let centre = p + z2 / (2.0 * n); let centre = p + z2 / (2.0 * n);
let margin = z * ((p * (1.0 - p) / n) + z2 / (4.0 * n * n)).sqrt(); let margin = z * ((p * (1.0 - p) / n) + z2 / (4.0 * n * n)).sqrt();
let v = if upper { centre + margin } else { centre - margin }; let v = if upper {
centre + margin
} else {
centre - margin
};
(v / denom).clamp(0.0, 1.0) (v / denom).clamp(0.0, 1.0)
} }
impl Promotion { impl Promotion {
pub fn new(cfg: PromotionCfg) -> std::sync::Arc<Self> { pub fn new(cfg: PromotionCfg) -> std::sync::Arc<Self> {
let p = std::sync::Arc::new(Self { cfg, labels: Mutex::new(HashMap::new()) }); let p = std::sync::Arc::new(Self {
cfg,
labels: Mutex::new(HashMap::new()),
});
p.replay(); p.replay();
p p
} }
fn replay(&self) { fn replay(&self) {
let Some(path) = &self.cfg.file else { return }; let Some(path) = &self.cfg.file else { return };
let Ok(content) = std::fs::read_to_string(path) else { return }; let Ok(content) = std::fs::read_to_string(path) else {
return;
};
for line in content.lines() { for line in content.lines() {
if let Ok(rec) = serde_json::from_str::<JudgeRecord>(line) { if let Ok(rec) = serde_json::from_str::<JudgeRecord>(line) {
self.apply(&rec); self.apply(&rec);
@@ -151,8 +163,10 @@ impl Promotion {
let st = self.apply(&rec); let st = self.apply(&rec);
if let Some(path) = &self.cfg.file { if let Some(path) = &self.cfg.file {
if let Ok(line) = serde_json::to_string(&rec) { if let Ok(line) = serde_json::to_string(&rec) {
if let Ok(mut f) = if let Ok(mut f) = std::fs::OpenOptions::new()
std::fs::OpenOptions::new().create(true).append(true).open(path) .create(true)
.append(true)
.open(path)
{ {
let _ = writeln!(f, "{line}"); let _ = writeln!(f, "{line}");
} }
@@ -163,7 +177,9 @@ impl Promotion {
fn apply(&self, rec: &JudgeRecord) -> PromotionState { fn apply(&self, rec: &JudgeRecord) -> PromotionState {
let mut map = self.labels.lock().unwrap(); let mut map = self.labels.lock().unwrap();
let ls = map.entry(rec.task_label.clone()).or_insert_with(LabelState::new); let ls = map
.entry(rec.task_label.clone())
.or_insert_with(LabelState::new);
push_cap(&mut ls.passes, rec.pass, self.cfg.window); push_cap(&mut ls.passes, rec.pass, self.cfg.window);
push_cap(&mut ls.hard_fails, rec.hard_fail, self.cfg.window); push_cap(&mut ls.hard_fails, rec.hard_fail, self.cfg.window);
self.advance(ls); self.advance(ls);
@@ -188,7 +204,12 @@ impl Promotion {
ls.state = match ls.state { ls.state = match ls.state {
PromotionState::CloudOnly | PromotionState::ShadowJudged => { PromotionState::CloudOnly | PromotionState::ShadowJudged => {
if bar { ls.soaked = 0; PromotionState::Canary } else { PromotionState::ShadowJudged } if bar {
ls.soaked = 0;
PromotionState::Canary
} else {
PromotionState::ShadowJudged
}
} }
PromotionState::Canary => { PromotionState::Canary => {
if regressed { if regressed {
@@ -203,7 +224,11 @@ impl Promotion {
} }
} }
PromotionState::LocalServed => { PromotionState::LocalServed => {
if regressed { PromotionState::Demoted } else { PromotionState::LocalServed } if regressed {
PromotionState::Demoted
} else {
PromotionState::LocalServed
}
} }
// Demoted goes back to SHADOW to re-earn promotion (skill stays in .cmf). // Demoted goes back to SHADOW to re-earn promotion (skill stays in .cmf).
PromotionState::Demoted => PromotionState::ShadowJudged, PromotionState::Demoted => PromotionState::ShadowJudged,
@@ -223,7 +248,10 @@ impl Promotion {
/// Should a request of this label be SERVED by the local model? /// Should a request of this label be SERVED by the local model?
/// (Canary/LocalServed; caller still applies the per-request veto.) /// (Canary/LocalServed; caller still applies the per-request veto.)
pub fn serves_local(&self, label: &str) -> bool { pub fn serves_local(&self, label: &str) -> bool {
matches!(self.state(label), PromotionState::Canary | PromotionState::LocalServed) matches!(
self.state(label),
PromotionState::Canary | PromotionState::LocalServed
)
} }
/// Snapshot for /admin: (state, n, pass-rate, Wilson-LB) per label. /// Snapshot for /admin: (state, n, pass-rate, Wilson-LB) per label.
@@ -235,10 +263,16 @@ impl Promotion {
let n = ls.passes.len(); let n = ls.passes.len();
let s = ls.passes.iter().filter(|&&p| p).count(); let s = ls.passes.iter().filter(|&&p| p).count();
let p = if n > 0 { s as f64 / n as f64 } else { 0.0 }; let p = if n > 0 { s as f64 / n as f64 } else { 0.0 };
(k.clone(), ls.state, n, p, wilson_bound(s, n, self.cfg.z, false)) (
k.clone(),
ls.state,
n,
p,
wilson_bound(s, n, self.cfg.z, false),
)
}) })
.collect(); .collect();
out.sort_by(|a, b| b.2.cmp(&a.2)); out.sort_by_key(|x| std::cmp::Reverse(x.2));
out out
} }
@@ -259,12 +293,24 @@ mod tests {
use super::*; use super::*;
fn cfg() -> PromotionCfg { fn cfg() -> PromotionCfg {
PromotionCfg { enabled: true, n_min: 200, soak: 0, ..Default::default() } PromotionCfg {
enabled: true,
n_min: 200,
soak: 0,
..Default::default()
}
} }
fn rec(label: &str, pass: bool, hard_fail: bool) -> JudgeRecord { fn rec(label: &str, pass: bool, hard_fail: bool) -> JudgeRecord {
JudgeRecord { JudgeRecord {
ts: 0, task_label: label.into(), pass, hard_fail, score: if pass { 4 } else { 0 }, ts: 0,
source: "checker".into(), recon_e: None, born_conf: None, judge_tokens: 0, task_label: label.into(),
pass,
hard_fail,
score: if pass { 4 } else { 0 },
source: "checker".into(),
recon_e: None,
born_conf: None,
judge_tokens: 0,
} }
} }
@@ -316,7 +362,10 @@ mod tests {
match row { match row {
Some((_, st, n, pr, lb)) => println!( Some((_, st, n, pr, lb)) => println!(
" {label:<12} {:<13} n={n:<4} pass={:.0}% LB={:.3} serves_local={}", " {label:<12} {:<13} n={n:<4} pass={:.0}% LB={:.3} serves_local={}",
format!("{st:?}"), pr * 100.0, lb, p.serves_local(label) format!("{st:?}"),
pr * 100.0,
lb,
p.serves_local(label)
), ),
None => println!(" {label:<12} (cloud-only, no judgments)"), None => println!(" {label:<12} (cloud-only, no judgments)"),
} }
@@ -324,19 +373,29 @@ mod tests {
println!("\n── self-warming loop, live ──"); println!("\n── self-warming loop, live ──");
// A mature skill (e.g. ru-tech): local answers pass the judge ~99%. // A mature skill (e.g. ru-tech): local answers pass the judge ~99%.
println!(" 1) SHADOW→CANARY: судим локаль 'ru_tech' (сильный скилл ~99% pass), 210 судейств:"); println!(
" 1) SHADOW→CANARY: судим локаль 'ru_tech' (сильный скилл ~99% pass), 210 судейств:"
);
for i in 0..210 { for i in 0..210 {
p.record(rec("ru_tech", i % 100 != 0, false)); p.record(rec("ru_tech", i % 100 != 0, false));
} }
show("ru_tech"); show("ru_tech");
assert_eq!(p.state("ru_tech"), PromotionState::Canary, "n≥200 & LB≥0.95 → CANARY"); assert_eq!(
p.state("ru_tech"),
PromotionState::Canary,
"n≥200 & LB≥0.95 → CANARY"
);
println!(" 2) CANARY soak (ещё 60 судейств держат планку) → LOCAL_SERVED:"); println!(" 2) CANARY soak (ещё 60 судейств держат планку) → LOCAL_SERVED:");
for i in 0..60 { for i in 0..60 {
p.record(rec("ru_tech", i % 100 != 0, false)); p.record(rec("ru_tech", i % 100 != 0, false));
} }
show("ru_tech"); show("ru_tech");
assert_eq!(p.state("ru_tech"), PromotionState::LocalServed, "soak → LOCAL_SERVED"); assert_eq!(
p.state("ru_tech"),
PromotionState::LocalServed,
"soak → LOCAL_SERVED"
);
assert!(p.serves_local("ru_tech"), "теперь обслуживается локально"); assert!(p.serves_local("ru_tech"), "теперь обслуживается локально");
// The code dataset: local fails the checker → never promotes. // The code dataset: local fails the checker → never promotes.
@@ -345,7 +404,11 @@ mod tests {
p.record(rec("code", false, false)); p.record(rec("code", false, false));
} }
show("code"); show("code");
assert_eq!(p.state("code"), PromotionState::ShadowJudged, "остаётся на облаке — безопасно"); assert_eq!(
p.state("code"),
PromotionState::ShadowJudged,
"остаётся на облаке — безопасно"
);
assert!(!p.serves_local("code")); assert!(!p.serves_local("code"));
// Drift on the promoted label → auto-demote back to cloud. // Drift on the promoted label → auto-demote back to cloud.
@@ -376,6 +439,9 @@ mod tests {
for _ in 0..300 { for _ in 0..300 {
p2.record(rec("t", false, false)); // quality collapses p2.record(rec("t", false, false)); // quality collapses
} }
assert!(matches!(p2.state("t"), PromotionState::Demoted | PromotionState::ShadowJudged)); assert!(matches!(
p2.state("t"),
PromotionState::Demoted | PromotionState::ShadowJudged
));
} }
} }
+13 -10
View File
@@ -21,24 +21,27 @@ impl Registry {
.as_ref() .as_ref()
.and_then(|name| secrets.resolve(name)); .and_then(|name| secrets.resolve(name));
let provider: Arc<dyn Provider> = match m.provider.as_str() { let provider: Arc<dyn Provider> = match m.provider.as_str() {
"openai" | "ollama" | "http" => Arc::new(OpenAiProvider::new(m, api_key)?), // openrouter + lmstudio are OpenAI-compatible (Bearer + /chat/completions)
"openai" | "ollama" | "http" | "openrouter" | "lmstudio" => {
Arc::new(OpenAiProvider::new(m, api_key)?)
}
"anthropic" => Arc::new(AnthropicProvider::new(m, api_key)?), "anthropic" => Arc::new(AnthropicProvider::new(m, api_key)?),
other => anyhow::bail!("unknown provider '{}' for model '{}'", other, m.id), other => anyhow::bail!("unknown provider '{}' for model '{}'", other, m.id),
}; };
by_id.insert(m.id.clone(), provider); by_id.insert(m.id.clone(), provider);
} }
// Managed local CMF server: register it as an OpenAI-compatible provider // Managed local CMF servers: register each as an OpenAI-compatible
// so it can be routed to like any other backend — including when the // provider so it can be routed to like any other backend — including when
// router is unavailable or the gateway is in local-only mode. // the router is unavailable or the gateway is in local-only mode.
if cfg.cmf.manage_server for s in cfg.cmf.effective_servers() {
&& !cfg.cmf.local_model.trim().is_empty() if by_id.contains_key(&s.id) {
&& !by_id.contains_key(&cfg.cmf.model_id) continue; // a static [[models]] entry with the same id wins
{ }
let m = ModelCfg { let m = ModelCfg {
id: cfg.cmf.model_id.clone(), id: s.id.clone(),
provider: "openai".into(), provider: "openai".into(),
base_url: format!("http://{}:{}/v1", cfg.cmf.local_host, cfg.cmf.local_port), base_url: format!("http://{}:{}/v1", cfg.cmf.local_host, s.port),
model: "cortiq".into(), model: "cortiq".into(),
cost_tier: "local".into(), cost_tier: "local".into(),
price_in: 0.0, price_in: 0.0,
+7 -3
View File
@@ -49,8 +49,8 @@ impl Live {
/// (local-only / disabled) or unavailable, so CMF models serve with no router. /// (local-only / disabled) or unavailable, so CMF models serve with no router.
pub fn local_candidates(&self) -> Vec<String> { pub fn local_candidates(&self) -> Vec<String> {
let mut v = Vec::new(); let mut v = Vec::new();
if self.cfg.cmf.manage_server && !self.cfg.cmf.local_model.trim().is_empty() { for s in self.cfg.cmf.effective_servers() {
v.push(self.cfg.cmf.model_id.clone()); v.push(s.id);
} }
let def = self.routing.default_model().to_string(); let def = self.routing.default_model().to_string();
if !def.is_empty() && !v.contains(&def) { if !def.is_empty() && !v.contains(&def) {
@@ -90,7 +90,11 @@ fn promotion_cfg(s: &crate::config::ShadowCfg) -> crate::promotion::PromotionCfg
n_min: s.n_min, n_min: s.n_min,
promote_lb: s.promote_lb, promote_lb: s.promote_lb,
soak: s.soak, soak: s.soak,
file: if s.file.is_empty() { None } else { Some(s.file.clone()) }, file: if s.file.is_empty() {
None
} else {
Some(s.file.clone())
},
..Default::default() ..Default::default()
} }
} }
+93 -41
View File
@@ -83,9 +83,23 @@ impl Totals {
self.cost_usd += r.cost_usd; self.cost_usd += r.cost_usd;
self.latency_ms_sum += r.latency_ms; self.latency_ms_sum += r.latency_ms;
} }
/// Accumulate another Totals into this one (for windowed aggregation).
fn merge(&mut self, o: &Totals) {
self.requests += o.requests;
self.ok += o.ok;
self.errors += o.errors;
self.failovers += o.failovers;
self.prompt_tokens += o.prompt_tokens;
self.completion_tokens += o.completion_tokens;
self.cost_usd += o.cost_usd;
self.latency_ms_sum += o.latency_ms_sum;
}
} }
/// Per-minute bucket for time-series charts. /// Per-minute bucket. Beyond the chart fields it carries the per-group breakdown
/// and failovers so the admin snapshot can be computed for ANY time window by
/// summing the in-window buckets (the internal fields are not serialized into
/// the `series` payload).
#[derive(Clone, Debug, Default, Serialize)] #[derive(Clone, Debug, Default, Serialize)]
pub struct Bucket { pub struct Bucket {
pub minute: u64, // unix seconds, rounded down to the minute pub minute: u64, // unix seconds, rounded down to the minute
@@ -95,15 +109,25 @@ pub struct Bucket {
pub completion_tokens: u64, pub completion_tokens: u64,
pub cost_usd: f64, pub cost_usd: f64,
pub latency_ms_sum: u64, pub latency_ms_sum: u64,
#[serde(skip)]
pub failovers: u64,
#[serde(skip)]
pub by_model: HashMap<String, Totals>,
#[serde(skip)]
pub by_account: HashMap<String, Totals>,
#[serde(skip)]
pub by_tier: HashMap<String, Totals>,
#[serde(skip)]
pub by_label: HashMap<String, Totals>,
} }
#[derive(Default)] #[derive(Default)]
struct Inner { struct Inner {
/// All-time totals + per-model (used by the Prometheus counters, which must
/// be monotonic — the windowed admin snapshot is derived from the buckets).
total: Totals, total: Totals,
by_model: HashMap<String, Totals>, by_model: HashMap<String, Totals>,
by_account: HashMap<String, Totals>, /// Per-minute buckets — the single source for the windowed admin snapshot.
by_tier: HashMap<String, Totals>,
by_label: HashMap<String, Totals>,
buckets: VecDeque<Bucket>, buckets: VecDeque<Bucket>,
recent: VecDeque<RequestRecord>, recent: VecDeque<RequestRecord>,
} }
@@ -166,31 +190,14 @@ impl Stats {
} }
fn apply_aggregates(inner: &mut Inner, rec: &RequestRecord, retention_secs: u64) { fn apply_aggregates(inner: &mut Inner, rec: &RequestRecord, retention_secs: u64) {
// all-time counters for Prometheus; the windowed snapshot is derived
// from the buckets below.
inner.total.apply(rec); inner.total.apply(rec);
inner inner
.by_model .by_model
.entry(rec.model_id.clone()) .entry(rec.model_id.clone())
.or_default() .or_default()
.apply(rec); .apply(rec);
inner
.by_account
.entry(if rec.account.is_empty() {
"anonymous".to_string()
} else {
rec.account.clone()
})
.or_default()
.apply(rec);
inner
.by_tier
.entry(rec.tier.clone())
.or_default()
.apply(rec);
inner
.by_label
.entry(rec.task_label.clone())
.or_default()
.apply(rec);
let minute = rec.ts - (rec.ts % 60); let minute = rec.ts - (rec.ts % 60);
match inner.buckets.back_mut() { match inner.buckets.back_mut() {
@@ -250,13 +257,31 @@ impl Stats {
.filter(|b| b.minute >= cutoff) .filter(|b| b.minute >= cutoff)
.collect(); .collect();
let breakdown_src = match groupby { // Windowed totals + breakdown = sum of the in-window buckets, so the KPIs
"account" => &inner.by_account, // and the breakdown recompute whenever the range changes.
"tier" => &inner.by_tier, let mut win = Totals::default();
"label" => &inner.by_label, let mut group: HashMap<String, Totals> = HashMap::new();
_ => &inner.by_model, for b in &series {
}; win.requests += b.requests;
let mut breakdown: Vec<serde_json::Value> = breakdown_src win.errors += b.errors;
win.ok += b.requests.saturating_sub(b.errors);
win.failovers += b.failovers;
win.prompt_tokens += b.prompt_tokens;
win.completion_tokens += b.completion_tokens;
win.cost_usd += b.cost_usd;
win.latency_ms_sum += b.latency_ms_sum;
let src = match groupby {
"account" => &b.by_account,
"tier" => &b.by_tier,
"label" => &b.by_label,
_ => &b.by_model,
};
for (k, t) in src {
group.entry(k.clone()).or_default().merge(t);
}
}
let mut breakdown: Vec<serde_json::Value> = group
.iter() .iter()
.map(|(k, t)| { .map(|(k, t)| {
serde_json::json!({ serde_json::json!({
@@ -280,17 +305,17 @@ impl Stats {
serde_json::json!({ serde_json::json!({
"totals": { "totals": {
"requests": inner.total.requests, "requests": win.requests,
"ok": inner.total.ok, "ok": win.ok,
"errors": inner.total.errors, "errors": win.errors,
"failovers": inner.total.failovers, "failovers": win.failovers,
"prompt_tokens": inner.total.prompt_tokens, "prompt_tokens": win.prompt_tokens,
"completion_tokens": inner.total.completion_tokens, "completion_tokens": win.completion_tokens,
"total_tokens": inner.total.prompt_tokens + inner.total.completion_tokens, "total_tokens": win.prompt_tokens + win.completion_tokens,
"cost_usd": inner.total.cost_usd, "cost_usd": win.cost_usd,
"avg_latency_ms": avg(inner.total.latency_ms_sum, inner.total.requests), "avg_latency_ms": avg(win.latency_ms_sum, win.requests),
"success_rate": if inner.total.requests > 0 { "success_rate": if win.requests > 0 {
inner.total.ok as f64 / inner.total.requests as f64 win.ok as f64 / win.requests as f64
} else { 0.0 }, } else { 0.0 },
}, },
"groupby": groupby, "groupby": groupby,
@@ -299,6 +324,18 @@ impl Stats {
}) })
} }
/// Reset all in-memory stats and truncate the JSONL log — the "clear logs"
/// admin action.
pub fn clear(&self) {
{
let mut inner = self.inner.lock().unwrap();
*inner = Inner::default();
}
if let Some(path) = &self.file {
let _ = std::fs::write(path, "");
}
}
/// Recent requests (newest first), with pagination. /// Recent requests (newest first), with pagination.
pub fn recent(&self, limit: usize, offset: usize) -> Vec<RequestRecord> { pub fn recent(&self, limit: usize, offset: usize) -> Vec<RequestRecord> {
let inner = self.inner.lock().unwrap(); let inner = self.inner.lock().unwrap();
@@ -358,10 +395,25 @@ fn fill_bucket(b: &mut Bucket, r: &RequestRecord) {
if r.outcome != "ok" { if r.outcome != "ok" {
b.errors += 1; b.errors += 1;
} }
if r.failover {
b.failovers += 1;
}
b.prompt_tokens += r.prompt_tokens as u64; b.prompt_tokens += r.prompt_tokens as u64;
b.completion_tokens += r.completion_tokens as u64; b.completion_tokens += r.completion_tokens as u64;
b.cost_usd += r.cost_usd; b.cost_usd += r.cost_usd;
b.latency_ms_sum += r.latency_ms; b.latency_ms_sum += r.latency_ms;
// per-group breakdown, so the snapshot can be windowed by summing buckets
b.by_model.entry(r.model_id.clone()).or_default().apply(r);
b.by_account
.entry(if r.account.is_empty() {
"anonymous".to_string()
} else {
r.account.clone()
})
.or_default()
.apply(r);
b.by_tier.entry(r.tier.clone()).or_default().apply(r);
b.by_label.entry(r.task_label.clone()).or_default().apply(r);
} }
fn avg(sum: u64, n: u64) -> u64 { fn avg(sum: u64, n: u64) -> u64 {
+5 -1
View File
@@ -60,7 +60,8 @@ export const api = {
listModels: () => j("GET", "/models"), listModels: () => j("GET", "/models"),
createModel: (b) => j("POST", "/models", b), createModel: (b) => j("POST", "/models", b),
updateModel: (id, b) => j("PUT", "/models/" + encodeURIComponent(id), b), updateModel: (id, b) => j("PUT", "/models/" + encodeURIComponent(id), b),
deleteModel: (id) => j("DELETE", "/models/" + encodeURIComponent(id)), deleteModel: (id, deleteFile = true) =>
j("DELETE", "/models/" + encodeURIComponent(id) + "?delete_file=" + (deleteFile ? "true" : "false")),
probeModel: (id) => j("POST", "/models/" + encodeURIComponent(id) + "/probe"), probeModel: (id) => j("POST", "/models/" + encodeURIComponent(id) + "/probe"),
// routing // routing
@@ -87,6 +88,7 @@ export const api = {
// stats // stats
stats: (range, groupby) => j("GET", `/stats?range=${range}&groupby=${groupby}`), stats: (range, groupby) => j("GET", `/stats?range=${range}&groupby=${groupby}`),
clearStats: () => j("DELETE", "/stats"),
requests: (limit = 50, offset = 0) => j("GET", `/requests?limit=${limit}&offset=${offset}`), requests: (limit = 50, offset = 0) => j("GET", `/requests?limit=${limit}&offset=${offset}`),
// playground // playground
@@ -99,6 +101,8 @@ export const api = {
cmfStatus: () => j("GET", "/cmf"), cmfStatus: () => j("GET", "/cmf"),
cmfInstall: () => j("POST", "/cmf/install"), cmfInstall: () => j("POST", "/cmf/install"),
cmfPortCheck: (port) => j("GET", "/cmf/port?port=" + encodeURIComponent(port)), cmfPortCheck: (port) => j("GET", "/cmf/port?port=" + encodeURIComponent(port)),
cmfFiles: () => j("GET", "/cmf/files"),
providerModels: (b) => j("POST", "/provider/models", b),
// CMF model factory (HuggingFace → local .cmf) // CMF model factory (HuggingFace → local .cmf)
hfSearch: (q, limit = 24) => hfSearch: (q, limit = 24) =>
+46
View File
@@ -59,6 +59,9 @@ export default {
// dashboard // dashboard
"dash.title": "Dashboard", "dash.title": "Dashboard",
"dash.subtitle": "Live traffic, cost and routing analytics for your model pool.", "dash.subtitle": "Live traffic, cost and routing analytics for your model pool.",
"dash.clear": "Clear logs",
"dash.clearConfirm": "Clear all recorded stats and logs? This cannot be undone.",
"dash.cleared": "Logs cleared",
"dash.range": "Range", "dash.range": "Range",
"dash.range.1h": "1h", "dash.range.1h": "1h",
"dash.range.24h": "24h", "dash.range.24h": "24h",
@@ -129,6 +132,22 @@ export default {
"models.form.secretMissing": "not set", "models.form.secretMissing": "not set",
"models.form.secretPlaceholder": "set or replace key (write-only)", "models.form.secretPlaceholder": "set or replace key (write-only)",
"models.deleteConfirm": "Delete model \"{id}\"?", "models.deleteConfirm": "Delete model \"{id}\"?",
"models.deleteFile": "also delete the model file from disk",
"models.deletedWithFile": "Deleted (file removed from disk)",
"models.form.idOrModel": "Enter an id or a model name",
"models.form.apiKey": "API Key",
"models.form.apiKeyPlaceholder": "paste your API key",
"models.form.apiKeyNotNeeded": "not required (local server)",
"models.form.apiKeyHint": "Stored securely on the gateway; it is not shown again.",
"models.form.keyStored": "A key is saved — leave blank to keep it, or paste a new one.",
"models.form.keyFromEnv": "Using a key from an environment variable.",
"models.form.testKey": "Test & load models",
"models.form.testing": "checking…",
"models.form.foundModels": "{n} models loaded",
"models.form.modelPlaceholder": "type to search…",
"models.form.modelHint": "Click “Test & load models” to pull the list from your key, then pick one.",
"models.managed": "managed",
"models.managedHint": "Provided by the managed CMF server (Settings → Local models). Configure it there.",
// routing // routing
"routing.title": "Routing", "routing.title": "Routing",
@@ -242,9 +261,15 @@ export default {
"settings.cmf.threads": "Worker threads", "settings.cmf.threads": "Worker threads",
"settings.cmf.gpu": "Use GPU (Metal / wgpu)", "settings.cmf.gpu": "Use GPU (Metal / wgpu)",
"settings.cmf.speedNote": "Threads parallelise the decode matvecs (~3x faster; 0 = runtime default). GPU offload mainly helps prefill of long prompts — decode is memory-bandwidth-bound, so on Apple Silicon it is roughly neutral.", "settings.cmf.speedNote": "Threads parallelise the decode matvecs (~3x faster; 0 = runtime default). GPU offload mainly helps prefill of long prompts — decode is memory-bandwidth-bound, so on Apple Silicon it is roughly neutral.",
"settings.cmf.pick": "select a model",
"settings.cmf.models": "Local models",
"settings.cmf.modelsHelp": "Each row is one model the gateway runs (its own cortiq serve on its own port) and registers in the pool. Add as many as you have .cmf files and free ports.",
"settings.cmf.rowHint": "id · .cmf file · port · threads · GPU. Changes apply on restart; deleting a row stops that server.",
"settings.cmf.addServer": "+ Add local model",
"settings.cmf.checkPort": "Check port", "settings.cmf.checkPort": "Check port",
"settings.cmf.portFree": "port is free", "settings.cmf.portFree": "port is free",
"settings.cmf.portBusy": "port is in use", "settings.cmf.portBusy": "port is in use",
"settings.cmf.portOwn": "in use by this gateway's current server — freed on restart, replacing is fine",
"settings.cmf.useSuggested": "use {port}", "settings.cmf.useSuggested": "use {port}",
"settings.cmf.serving": "serving {model} on :{port}", "settings.cmf.serving": "serving {model} on :{port}",
"settings.cmf.intro": "Run models locally with the cortiq engine — works even offline or when the router is down.", "settings.cmf.intro": "Run models locally with the cortiq engine — works even offline or when the router is down.",
@@ -297,6 +322,10 @@ export default {
"settings.route.maxChars": "Max chars", "settings.route.maxChars": "Max chars",
"settings.route.cacheTtl": "Cache TTL", "settings.route.cacheTtl": "Cache TTL",
"settings.route.profile": "Default profile", "settings.route.profile": "Default profile",
"settings.route.strategyHelp": "What text is sent to the router for classification: last_user = only the last user message; last_user_plus_system = last user message + system prompt; concat_all = the whole conversation.",
"settings.route.maxCharsHelp": "Max characters of that text sent to the router (longer is truncated). More = more accurate classification, but slower/pricier router calls.",
"settings.route.cacheTtlHelp": "How long a routing decision is cached for an identical request before the router is asked again (e.g. 60s, 5m).",
"settings.route.profileHelp": "Default routing profile when a request doesn't specify one: cost-saver (cheapest), balanced, or quality-first (best model). Biases which model the router picks.",
"settings.breaker": "Circuit breaker", "settings.breaker": "Circuit breaker",
"settings.breaker.threshold": "Error threshold", "settings.breaker.threshold": "Error threshold",
"settings.breaker.cooldown": "Cooldown", "settings.breaker.cooldown": "Cooldown",
@@ -374,4 +403,21 @@ export default {
"import.adv.shard": "Shard max (GB)", "import.adv.shard": "Shard max (GB)",
"import.adv.noShard": "no sharding", "import.adv.noShard": "no sharding",
"import.adv.skipMtp": "skip MTP head", "import.adv.skipMtp": "skip MTP head",
"import.adv.o1": "O(1) attention",
"import.adv.o1Hint": "Constant-memory streaming attention: the KV cache is released after prefill, so long contexts don't grow memory. Weights stay byte-identical — only a load-time hint is recorded; you can switch back anytime (serve with --o1 off). Validated on Qwen3-0.6B: all layers ≈ ×1.18 perplexity.",
"import.adv.o1Off": "off — exact attention (default)",
"import.adv.o1All": "all — convert every softmax layer",
"import.adv.o1Deep": "deep N — only the N deepest layers",
"import.adv.o1Custom": "custom — explicit layer list",
"import.adv.o1DeepN": "N deep layers",
"import.adv.o1Layers": "Layers (comma-separated)",
"import.adv.o1mHint": "landmark budget (default 32)",
"import.adv.o1mLabel": "Landmarks (--o1-m)",
"import.adv.o1mHelp": "How many landmark tokens summarize the whole older history when attention is approximated. More = closer to exact attention but slower and more memory; fewer = faster but rougher. 32 is the validated sweet spot — change only if you know why.",
"import.adv.o1windowLabel": "Exact window (--o1-window)",
"import.adv.o1windowHelp": "The last N tokens always get exact (non-approximated) attention. This protects quality on the text the model just wrote/read. Bigger = better local quality, more memory. Default 128.",
"import.adv.o1sinkLabel": "Sink keys (--o1-sink)",
"import.adv.o1sinkHelp": "The first N tokens of the context are kept exact forever ('attention sinks'). Models lean on these anchors — keeping them exact stabilizes long generations. Default 4; rarely needs changing.",
"import.adv.o1windowHint": "exact-window width (default 128)",
"import.adv.o1sinkHint": "permanent exact sink keys (default 4)",
}; };
+46
View File
@@ -58,6 +58,9 @@ export default {
// dashboard // dashboard
"dash.title": "Дашборд", "dash.title": "Дашборд",
"dash.subtitle": "Трафик в реальном времени, расходы и аналитика роутинга по пулу моделей.", "dash.subtitle": "Трафик в реальном времени, расходы и аналитика роутинга по пулу моделей.",
"dash.clear": "Очистить логи",
"dash.clearConfirm": "Очистить всю статистику и логи? Отменить нельзя.",
"dash.cleared": "Логи очищены",
"dash.range": "Период", "dash.range": "Период",
"dash.range.1h": "1ч", "dash.range.1h": "1ч",
"dash.range.24h": "24ч", "dash.range.24h": "24ч",
@@ -128,6 +131,22 @@ export default {
"models.form.secretMissing": "не задан", "models.form.secretMissing": "не задан",
"models.form.secretPlaceholder": "задать или заменить ключ (только запись)", "models.form.secretPlaceholder": "задать или заменить ключ (только запись)",
"models.deleteConfirm": "Удалить модель \"{id}\"?", "models.deleteConfirm": "Удалить модель \"{id}\"?",
"models.deleteFile": "также удалить файл модели с диска",
"models.deletedWithFile": "Удалено (файл стёрт с диска)",
"models.form.idOrModel": "Укажите id или имя модели",
"models.form.apiKey": "API-ключ",
"models.form.apiKeyPlaceholder": "вставьте API-ключ",
"models.form.apiKeyNotNeeded": "не нужен (локальный сервер)",
"models.form.apiKeyHint": "Хранится безопасно на шлюзе; больше не показывается.",
"models.form.keyStored": "Ключ сохранён — оставьте пустым, чтобы не менять, или вставьте новый.",
"models.form.keyFromEnv": "Используется ключ из переменной окружения.",
"models.form.testKey": "Проверить и загрузить модели",
"models.form.testing": "проверка…",
"models.form.foundModels": "загружено моделей: {n}",
"models.form.modelPlaceholder": "введите для поиска…",
"models.form.modelHint": "Нажмите «Проверить и загрузить модели», чтобы подтянуть список по ключу, затем выберите.",
"models.managed": "управляемая",
"models.managedHint": "Предоставляется CMF-сервером (Настройки → Локальные модели). Настраивается там.",
// routing // routing
"routing.title": "Роутинг", "routing.title": "Роутинг",
@@ -241,9 +260,15 @@ export default {
"settings.cmf.threads": "Рабочие потоки", "settings.cmf.threads": "Рабочие потоки",
"settings.cmf.gpu": "Использовать GPU (Metal / wgpu)", "settings.cmf.gpu": "Использовать GPU (Metal / wgpu)",
"settings.cmf.speedNote": "Потоки распараллеливают декод-матвеки (~×3; 0 = по умолчанию рантайма). GPU-оффлоад помогает в основном на prefill длинных промптов — декод упирается в пропускную способность памяти, поэтому на Apple Silicon почти нейтрален.", "settings.cmf.speedNote": "Потоки распараллеливают декод-матвеки (~×3; 0 = по умолчанию рантайма). GPU-оффлоад помогает в основном на prefill длинных промптов — декод упирается в пропускную способность памяти, поэтому на Apple Silicon почти нейтрален.",
"settings.cmf.pick": "выберите модель",
"settings.cmf.models": "Локальные модели",
"settings.cmf.modelsHelp": "Каждая строка — отдельная модель: шлюз поднимает свой cortiq serve на своём порту и регистрирует её в пуле. Добавляйте столько, сколько есть .cmf-файлов и свободных портов.",
"settings.cmf.rowHint": "id · файл .cmf · порт · потоки · GPU. Изменения применяются после перезапуска; удаление строки останавливает сервер.",
"settings.cmf.addServer": "+ Добавить модель",
"settings.cmf.checkPort": "Проверить порт", "settings.cmf.checkPort": "Проверить порт",
"settings.cmf.portFree": "порт свободен", "settings.cmf.portFree": "порт свободен",
"settings.cmf.portBusy": "порт занят", "settings.cmf.portBusy": "порт занят",
"settings.cmf.portOwn": "занят текущим сервером шлюза — освободится после перезапуска, замена допустима",
"settings.cmf.useSuggested": "занять {port}", "settings.cmf.useSuggested": "занять {port}",
"settings.cmf.serving": "работает: {model} на :{port}", "settings.cmf.serving": "работает: {model} на :{port}",
"settings.cmf.intro": "Запускайте модели локально движком cortiq — работает даже офлайн или когда роутер недоступен.", "settings.cmf.intro": "Запускайте модели локально движком cortiq — работает даже офлайн или когда роутер недоступен.",
@@ -296,6 +321,10 @@ export default {
"settings.route.maxChars": "Макс. символов", "settings.route.maxChars": "Макс. символов",
"settings.route.cacheTtl": "TTL кэша", "settings.route.cacheTtl": "TTL кэша",
"settings.route.profile": "Профиль по умолчанию", "settings.route.profile": "Профиль по умолчанию",
"settings.route.strategyHelp": "Какой текст отправляется роутеру для классификации: last_user — только последнее сообщение пользователя; last_user_plus_system — последнее сообщение + системный промпт; concat_all — вся переписка.",
"settings.route.maxCharsHelp": "Максимум символов этого текста, отправляемого роутеру (длиннее — обрезается). Больше = точнее классификация, но медленнее/дороже вызовы роутера.",
"settings.route.cacheTtlHelp": "Сколько кэшируется решение роутинга для одинакового запроса, прежде чем роутер спрашивается снова (напр. 60s, 5m).",
"settings.route.profileHelp": "Профиль роутинга по умолчанию, если запрос его не задаёт: cost-saver (дешевле), balanced (баланс) или quality-first (качество). Влияет на выбор модели роутером.",
"settings.breaker": "Автоматический выключатель", "settings.breaker": "Автоматический выключатель",
"settings.breaker.threshold": "Порог ошибок", "settings.breaker.threshold": "Порог ошибок",
"settings.breaker.cooldown": "Время восстановления", "settings.breaker.cooldown": "Время восстановления",
@@ -373,4 +402,21 @@ export default {
"import.adv.shard": "Шард макс (ГБ)", "import.adv.shard": "Шард макс (ГБ)",
"import.adv.noShard": "без шардинга", "import.adv.noShard": "без шардинга",
"import.adv.skipMtp": "пропустить MTP-голову", "import.adv.skipMtp": "пропустить MTP-голову",
"import.adv.o1": "O(1)-внимание",
"import.adv.o1Hint": "Потоковое внимание с постоянной памятью: KV-кэш освобождается после prefill, длинный контекст не раздувает память. Веса не меняются — записывается лишь подсказка на загрузку; вернуться можно всегда (запуск с --o1 off). Проверено на Qwen3-0.6B: все слои ≈ ×1.18 перплексии.",
"import.adv.o1Off": "выкл — точное внимание (по умолчанию)",
"import.adv.o1All": "all — конвертировать все softmax-слои",
"import.adv.o1Deep": "deep N — только N самых глубоких слоёв",
"import.adv.o1Custom": "вручную — явный список слоёв",
"import.adv.o1DeepN": "N глубоких слоёв",
"import.adv.o1Layers": "Слои (через запятую)",
"import.adv.o1mHint": "число опорных точек (по умолчанию 32)",
"import.adv.o1mLabel": "Опорные точки (--o1-m)",
"import.adv.o1mHelp": "Сколько опорных токенов представляют всю старую историю при аппроксимации внимания. Больше — ближе к точному вниманию, но медленнее и больше памяти; меньше — быстрее, но грубее. 32 — проверенное значение; меняйте, только если понимаете зачем.",
"import.adv.o1windowLabel": "Точное окно (--o1-window)",
"import.adv.o1windowHelp": "Последние N токенов всегда обрабатываются точным (не аппроксимированным) вниманием — это защищает качество на только что написанном/прочитанном тексте. Больше — лучше локальное качество, больше памяти. По умолчанию 128.",
"import.adv.o1sinkLabel": "Sink-ключи (--o1-sink)",
"import.adv.o1sinkHelp": "Первые N токенов контекста навсегда остаются точными («якоря внимания»). Модели опираются на эти якоря — их точность стабилизирует длинную генерацию. По умолчанию 4; менять почти никогда не нужно.",
"import.adv.o1windowHint": "ширина точного окна (по умолчанию 128)",
"import.adv.o1sinkHint": "постоянные точные sink-ключи (по умолчанию 4)",
}; };
+3 -1
View File
@@ -25,7 +25,9 @@ export function clear(node) {
export function mount(node, ...children) { export function mount(node, ...children) {
clear(node); clear(node);
for (const c of children.flat()) if (c != null && c !== false) node.appendChild(c); for (const c of children.flat())
if (c != null && c !== false)
node.appendChild(typeof c === "object" ? c : document.createTextNode(String(c)));
return node; return node;
} }
+19 -5
View File
@@ -1,5 +1,5 @@
// Dashboard — KPIs, time-series charts, breakdown, recent requests, health. // Dashboard — KPIs, time-series charts, breakdown, recent requests, health.
import { h, mount, money, num, ms, pct, timeAgo } from "../ui.js"; import { h, mount, money, num, ms, pct, timeAgo, toast, confirmDialog } from "../ui.js";
import { t } from "../i18n.js"; import { t } from "../i18n.js";
import { api } from "../api.js"; import { api } from "../api.js";
import { lineChart, barList } from "../charts.js"; import { lineChart, barList } from "../charts.js";
@@ -53,10 +53,24 @@ export async function renderDashboard() {
"div", "div",
{ class: "flex" }, { class: "flex" },
h("div", { class: "grow" }, h("h2", {}, t("dash.title")), h("p", {}, t("dash.subtitle"))), h("div", { class: "grow" }, h("h2", {}, t("dash.title")), h("p", {}, t("dash.subtitle"))),
seg(RANGES, range, (r) => { h("div", { class: "flex", style: "gap:8px;align-items:center" },
range = r; seg(RANGES, range, (r) => {
load(); range = r;
}, (r) => t("dash.range." + r)) load();
}, (r) => t("dash.range." + r)),
h("button", {
class: "btn sm danger",
onClick: async () => {
if (!(await confirmDialog(t("dash.clearConfirm")))) return;
try {
await api.clearStats();
toast(t("dash.cleared"), "good");
await load();
} catch (e) {
toast(String(e.message || e), "bad");
}
},
}, t("dash.clear")))
) )
), ),
+50 -2
View File
@@ -15,8 +15,14 @@ const QUANTS = [
["F32", "import.q.f32"], ["F32", "import.q.f32"],
]; ];
function field(label, control, hint) { // A small "?" badge with a hover/focus tooltip (same style as Settings).
return h("label", { class: "field" }, h("span", {}, label), control, function help(text) {
return text ? h("span", { class: "help", title: text, tabindex: "0", role: "img", "aria-label": text }, "?") : null;
}
function field(label, control, hint, helpText) {
return h("label", { class: "field" },
h("span", {}, label, helpText ? " " : null, help(helpText)),
control,
hint ? h("div", { class: "hint" }, hint) : null); hint ? h("div", { class: "hint" }, hint) : null);
} }
function opt(v, label, sel) { function opt(v, label, sel) {
@@ -125,6 +131,32 @@ export function renderImport() {
const meanBitsIn = h("input", { type: "number", step: "0.1", placeholder: "5.0" }); const meanBitsIn = h("input", { type: "number", step: "0.1", placeholder: "5.0" });
const shardIn = h("input", { type: "number", step: "0.5", placeholder: t("import.adv.noShard") }); const shardIn = h("input", { type: "number", step: "0.5", placeholder: t("import.adv.noShard") });
const skipMtp = h("input", { type: "checkbox" }); const skipMtp = h("input", { type: "checkbox" });
// O(1) attention (cortiq ≥ 0.2.0): constant-memory streaming attention hint,
// weights byte-identical — the runtime applies it at load.
const o1Sel = h("select", {},
opt("", t("import.adv.o1Off")),
opt("all", t("import.adv.o1All")),
opt("deep", t("import.adv.o1Deep")),
opt("custom", t("import.adv.o1Custom")));
const o1DeepIn = h("input", { type: "number", min: "1", placeholder: "12" });
const o1CustomIn = h("input", { placeholder: "0,4,8" });
const o1M = h("input", { type: "number", min: "1", placeholder: "32" });
const o1Window = h("input", { type: "number", min: "0", placeholder: "128" });
const o1Sink = h("input", { type: "number", min: "0", placeholder: "4" });
const o1SpecRow = h("div", { class: "row", style: "display:none" },
field(t("import.adv.o1DeepN"), o1DeepIn),
field(t("import.adv.o1Layers"), o1CustomIn));
const o1Knobs = h("div", { class: "row", style: "display:none" },
field(t("import.adv.o1mLabel"), o1M, t("import.adv.o1mHint"), t("import.adv.o1mHelp")),
field(t("import.adv.o1windowLabel"), o1Window, t("import.adv.o1windowHint"), t("import.adv.o1windowHelp")),
field(t("import.adv.o1sinkLabel"), o1Sink, t("import.adv.o1sinkHint"), t("import.adv.o1sinkHelp")));
o1Sel.addEventListener("change", () => {
const on = !!o1Sel.value;
o1Knobs.style.display = on ? "" : "none";
o1SpecRow.style.display = o1Sel.value === "deep" || o1Sel.value === "custom" ? "" : "none";
o1DeepIn.parentElement.style.display = o1Sel.value === "deep" ? "" : "none";
o1CustomIn.parentElement.style.display = o1Sel.value === "custom" ? "" : "none";
});
const advBody = h( const advBody = h(
"div", "div",
{ class: "adv-body", style: "display:none" }, { class: "adv-body", style: "display:none" },
@@ -137,6 +169,9 @@ export function renderImport() {
h("div", { class: "row" }, h("div", { class: "row" },
field(t("import.adv.shard"), shardIn), field(t("import.adv.shard"), shardIn),
field("MTP", h("label", { class: "check" }, skipMtp, t("import.adv.skipMtp")))), field("MTP", h("label", { class: "check" }, skipMtp, t("import.adv.skipMtp")))),
field(t("import.adv.o1"), o1Sel, t("import.adv.o1Hint")),
o1SpecRow,
o1Knobs,
); );
const advToggle = h("button", { class: "btn ghost sm" }, "⚙ " + t("import.adv.title")); const advToggle = h("button", { class: "btn ghost sm" }, "⚙ " + t("import.adv.title"));
advToggle.addEventListener("click", () => { advToggle.addEventListener("click", () => {
@@ -156,6 +191,19 @@ export function renderImport() {
if (vbitSel.value) params.vbit_shape = vbitSel.value; if (vbitSel.value) params.vbit_shape = vbitSel.value;
if (meanBitsIn.value) params.mean_bits = parseFloat(meanBitsIn.value); if (meanBitsIn.value) params.mean_bits = parseFloat(meanBitsIn.value);
if (shardIn.value) params.shard_max_gb = parseFloat(shardIn.value); if (shardIn.value) params.shard_max_gb = parseFloat(shardIn.value);
if (o1Sel.value) {
// spec: all | deepN | explicit layer list
params.o1 =
o1Sel.value === "deep" ? "deep" + (parseInt(o1DeepIn.value) || 12)
: o1Sel.value === "custom" ? o1CustomIn.value.trim()
: "all";
if (!params.o1) delete params.o1;
else {
if (o1M.value) params.o1_m = parseInt(o1M.value);
if (o1Window.value) params.o1_window = parseInt(o1Window.value);
if (o1Sink.value) params.o1_sink = parseInt(o1Sink.value);
}
}
goBtn.disabled = true; goBtn.disabled = true;
try { try {
await api.startImport(params); await api.startImport(params);
+150 -41
View File
@@ -1,5 +1,5 @@
// Models — pool table with add/edit modal, probe, and per-model secret entry. // Models — pool table with add/edit modal, probe, and per-model secret entry.
import { h, mount, modal, toast, confirmDialog, money } from "../ui.js"; import { h, mount, modal, toast, money } from "../ui.js";
import { t } from "../i18n.js"; import { t } from "../i18n.js";
import { api } from "../api.js"; import { api } from "../api.js";
import { appState } from "../app.js"; import { appState } from "../app.js";
@@ -16,52 +16,142 @@ function keyBadge(src) {
return h("span", { class: "badge " + (map[src] ?? "") }, src); return h("span", { class: "badge " + (map[src] ?? "") }, src);
} }
// Verified per-provider defaults (base URL, key env, whether a key is needed).
const PROVIDER_DEFAULTS = {
openai: { base: "https://api.openai.com/v1", keyEnv: "OPENAI_API_KEY", needsKey: true, tier: "mid", caps: ["tools", "vision"] },
anthropic: { base: "https://api.anthropic.com", keyEnv: "ANTHROPIC_API_KEY", needsKey: true, tier: "expensive", caps: ["tools", "vision"] },
openrouter: { base: "https://openrouter.ai/api/v1", keyEnv: "OPENROUTER_API_KEY", needsKey: true, tier: "mid", caps: ["tools", "vision"] },
lmstudio: { base: "http://localhost:1234/v1", keyEnv: "", needsKey: false, tier: "local", caps: ["tools", "vision"] },
ollama: { base: "http://localhost:11434/v1", keyEnv: "", needsKey: false, tier: "local", caps: ["tools"] },
http: { base: "", keyEnv: "", needsKey: false, tier: "cheap", caps: ["tools"] },
};
// Derive a clean model id from a model name (last path segment, slugified).
// e.g. "qwen/qwen3.7-max" → "qwen3-7-max".
function slugId(name) {
const base = (name || "").split("/").pop() || "";
return base.toLowerCase().replace(/[^a-z0-9]+/g, "-").replace(/^-+|-+$/g, "");
}
function modelForm(meta, existing) { function modelForm(meta, existing) {
const m = existing || { provider: "openai", kind: "chat", cost_tier: "cheap", caps: [], price_in: 0, price_out: 0 }; const m = existing || { provider: "openai", kind: "chat", cost_tier: "mid", caps: ["tools"], price_in: 0, price_out: 0 };
const idIn = h("input", { value: m.id || "", placeholder: "local-qwen", disabled: existing ? true : null }); const idIn = h("input", { value: m.id || "", placeholder: "my-model", disabled: existing ? true : null });
const providerSel = h("select", {}, ...(meta.providers || ["openai"]).map((p) => opt(p, p, p === m.provider))); const providerSel = h("select", {}, ...(meta.providers || ["openai"]).map((p) => opt(p, p, p === m.provider)));
const baseUrlIn = h("input", { value: m.base_url || "", placeholder: "http://localhost:8000/v1" }); const baseUrlIn = h("input", { value: m.base_url || "", placeholder: "https://api.openai.com/v1" });
const modelIn = h("input", { value: m.model || "", placeholder: "qwen2.5-7b-instruct" });
// API key — typed directly, stored securely as a secret under a provider env name.
const keyIn = h("input", { type: "password", placeholder: t("models.form.apiKeyPlaceholder"), autocomplete: "new-password" });
const testBtn = h("button", { class: "btn small", type: "button" }, t("models.form.testKey"));
const keyStatus = h("span", { class: "small", style: "margin-left:8px" });
// Model — below the key (some providers list models from the key); searchable datalist.
const dlId = "model-opts-" + (m.id || "new");
const dl = h("datalist", { id: dlId });
if (m.model) dl.appendChild(h("option", { value: m.model }));
const modelIn = h("input", { value: m.model || "", placeholder: t("models.form.modelPlaceholder"), list: dlId, autocomplete: "off" });
const kindSel = h("select", {}, ...(meta.kinds || ["chat"]).map((k) => opt(k, k, k === m.kind))); const kindSel = h("select", {}, ...(meta.kinds || ["chat"]).map((k) => opt(k, k, k === m.kind)));
const tierSel = h("select", {}, ...(meta.cost_tiers || ["cheap"]).map((c) => opt(c, c, c === m.cost_tier))); const tierSel = h("select", {}, ...(meta.cost_tiers || ["cheap"]).map((c) => opt(c, c, c === m.cost_tier)));
const priceInIn = h("input", { type: "number", step: "0.01", value: m.price_in ?? 0 }); const priceInIn = h("input", { type: "number", step: "0.01", value: m.price_in ?? 0 });
const priceOutIn = h("input", { type: "number", step: "0.01", value: m.price_out ?? 0 }); const priceOutIn = h("input", { type: "number", step: "0.01", value: m.price_out ?? 0 });
const keyEnvIn = h("input", { value: m.api_key_env || "", placeholder: "OPENAI_API_KEY" });
const secretIn = h("input", { type: "password", placeholder: t("models.form.secretPlaceholder") });
const capInputs = (meta.caps || ["tools", "vision"]).map((c) => { const capInputs = (meta.caps || ["tools", "vision"]).map((c) => {
const cb = h("input", { type: "checkbox", checked: (m.caps || []).includes(c) ? true : null }); const cb = h("input", { type: "checkbox", checked: (m.caps || []).includes(c) ? true : null });
return { c, cb }; return { c, cb };
}); });
const envFor = () =>
(existing && existing.api_key_env) ||
(PROVIDER_DEFAULTS[providerSel.value] || {}).keyEnv ||
providerSel.value.toUpperCase().replace(/[^A-Z0-9]/g, "_") + "_API_KEY";
// Fill in a provider's verified defaults when it is selected.
function applyDefaults() {
const d = PROVIDER_DEFAULTS[providerSel.value];
if (!d) return;
baseUrlIn.value = d.base;
tierSel.value = d.tier;
capInputs.forEach(({ c, cb }) => { cb.checked = d.caps.includes(c); });
keyIn.placeholder = d.needsKey ? t("models.form.apiKeyPlaceholder") : t("models.form.apiKeyNotNeeded");
mount(keyStatus, "");
}
providerSel.addEventListener("change", applyDefaults);
if (!existing) applyDefaults();
// Auto-fill the id from the model name (until the user edits id by hand).
if (!existing) {
modelIn.addEventListener("input", () => {
if (!idIn.value || idIn._auto) {
idIn.value = slugId(modelIn.value);
idIn._auto = true;
}
});
idIn.addEventListener("input", () => { idIn._auto = false; });
}
async function testKey() {
if (!baseUrlIn.value.trim()) { mount(keyStatus, h("span", { style: "color:var(--bad)" }, t("models.form.baseUrl") + "?")); return; }
testBtn.disabled = true;
mount(keyStatus, t("models.form.testing"));
try {
const r = await api.providerModels({
provider: providerSel.value,
base_url: baseUrlIn.value.trim(),
api_key: keyIn.value.trim() || null,
api_key_env: envFor(),
});
if (r.ok) {
dl.replaceChildren(...(r.models || []).map((mm) => h("option", { value: mm })));
mount(keyStatus, h("span", { style: "color:var(--good)" }, "✓ " + t("models.form.foundModels", { n: r.count })));
modelIn.focus();
} else {
mount(keyStatus, h("span", { style: "color:var(--bad)" }, "✗ " + (r.error || t("common.error"))));
}
} catch (e) {
mount(keyStatus, h("span", { style: "color:var(--bad)" }, "✗ " + String(e.message || e)));
}
testBtn.disabled = false;
}
testBtn.addEventListener("click", testKey);
const keyHint =
existing && existing.key_source === "store" ? t("models.form.keyStored")
: existing && existing.key_source === "env" ? t("models.form.keyFromEnv")
: t("models.form.apiKeyHint");
const node = h( const node = h(
"div", "div",
{}, {},
h("div", { class: "row" }, field(t("models.form.id"), idIn, existing ? null : t("models.form.idHint")), field(t("models.form.provider"), providerSel)), h("div", { class: "row" }, field(t("models.form.id"), idIn, existing ? null : t("models.form.idHint")), field(t("models.form.provider"), providerSel)),
field(t("models.form.baseUrl"), baseUrlIn), field(t("models.form.baseUrl"), baseUrlIn),
field(t("models.form.model"), modelIn), field(t("models.form.apiKey"), h("div", { class: "flex wrap", style: "gap:8px;align-items:center" }, keyIn, testBtn, keyStatus), keyHint),
field(t("models.form.model"), modelIn, t("models.form.modelHint")),
dl,
h("div", { class: "row" }, field(t("models.form.kind"), kindSel), field(t("models.form.costTier"), tierSel)), h("div", { class: "row" }, field(t("models.form.kind"), kindSel), field(t("models.form.costTier"), tierSel)),
h("div", { class: "row" }, field(t("models.form.priceIn"), priceInIn), field(t("models.form.priceOut"), priceOutIn)), h("div", { class: "row" }, field(t("models.form.priceIn"), priceInIn), field(t("models.form.priceOut"), priceOutIn)),
field(t("models.form.caps"), h("div", { class: "flex wrap" }, ...capInputs.map(({ c, cb }) => h("label", { class: "check" }, cb, c)))), field(t("models.form.caps"), h("div", { class: "flex wrap" }, ...capInputs.map(({ c, cb }) => h("label", { class: "check" }, cb, c))))
field(t("models.form.apiKeyEnv"), keyEnvIn, t("models.form.apiKeyEnvHint")),
field(t("models.form.secret"), secretIn)
); );
const getValue = () => ({ const getValue = () => {
model: { const typedKey = keyIn.value.trim();
id: existing ? m.id : idIn.value.trim(), const hasKey = !!typedKey || !!(existing && existing.api_key_env);
provider: providerSel.value, const keyEnv = envFor();
base_url: baseUrlIn.value.trim(), return {
model: modelIn.value.trim(), model: {
kind: kindSel.value, id: existing ? m.id : idIn.value.trim(),
cost_tier: tierSel.value, provider: providerSel.value,
price_in: parseFloat(priceInIn.value) || 0, base_url: baseUrlIn.value.trim(),
price_out: parseFloat(priceOutIn.value) || 0, model: modelIn.value.trim(),
api_key_env: keyEnvIn.value.trim() || null, kind: kindSel.value,
caps: capInputs.filter(({ cb }) => cb.checked).map(({ c }) => c), cost_tier: tierSel.value,
}, price_in: parseFloat(priceInIn.value) || 0,
secret: secretIn.value, price_out: parseFloat(priceOutIn.value) || 0,
keyEnv: keyEnvIn.value.trim(), api_key_env: hasKey ? keyEnv : null,
}); caps: capInputs.filter(({ cb }) => cb.checked).map(({ c }) => c),
},
secret: typedKey,
keyEnv,
};
};
return { node, getValue }; return { node, getValue };
} }
@@ -70,12 +160,14 @@ async function openModal(meta, existing, reload) {
const { node, getValue } = modelForm(meta, existing); const { node, getValue } = modelForm(meta, existing);
modal(existing ? t("models.form.editTitle") : t("models.form.addTitle"), node, async () => { modal(existing ? t("models.form.editTitle") : t("models.form.addTitle"), node, async () => {
const { model, secret, keyEnv } = getValue(); const { model, secret, keyEnv } = getValue();
// Forgot the id? Derive it from the model name so saving never errors on a
// blank id.
if (!model.id) model.id = slugId(model.model);
if (!model.id) { if (!model.id) {
toast(t("models.form.id") + "?", "bad"); toast(t("models.form.idOrModel"), "bad");
return false; return false;
} }
// strip null api_key_env so serde keeps it optional if (!model.api_key_env) delete model.api_key_env; // keep serde field optional
if (!model.api_key_env) delete model.api_key_env;
try { try {
if (existing) await api.updateModel(existing.id, model); if (existing) await api.updateModel(existing.id, model);
else await api.createModel(model); else await api.createModel(model);
@@ -168,7 +260,8 @@ function row(m, meta, reload) {
return h( return h(
"tr", "tr",
{}, {},
h("td", { class: "mono" }, m.id), h("td", { class: "mono" }, m.id,
m.managed ? h("span", { class: "badge", style: "margin-left:6px", title: t("models.managedHint") }, t("models.managed")) : null),
h("td", {}, m.provider), h("td", {}, m.provider),
h("td", { class: "mono" }, m.model), h("td", { class: "mono" }, m.model),
h("td", {}, h("span", { class: "badge " + (m.cost_tier || "") }, m.cost_tier || "—")), h("td", {}, h("span", { class: "badge " + (m.cost_tier || "") }, m.cost_tier || "—")),
@@ -182,20 +275,36 @@ function row(m, meta, reload) {
"div", "div",
{ class: "flex", style: "justify-content:flex-end" }, { class: "flex", style: "justify-content:flex-end" },
probeBtn, probeBtn,
h("button", { class: "btn sm ghost", onClick: () => openModal(meta, m, reload) }, t("common.edit")), m.managed ? null : h("button", { class: "btn sm ghost", onClick: () => openModal(meta, m, reload) }, t("common.edit")),
h( h(
"button", "button",
{ {
class: "btn sm danger", class: "btn sm danger",
onClick: async () => { onClick: () => {
if (!(await confirmDialog(t("models.deleteConfirm", { id: m.id })))) return; // local models may have a .cmf on disk — let the user decide its fate
try { const isLocal = m.managed || /127\.0\.0\.1|localhost/.test(m.base_url || "");
await api.deleteModel(m.id); const fileCb = h("input", { type: "checkbox", checked: isLocal ? true : null });
toast(t("toast.deleted"), "good"); const body = h(
reload(); "div",
} catch (e) { {},
toast(String(e.message || e), "bad"); h("p", {}, t("models.deleteConfirm", { id: m.id })),
} isLocal
? h("label", { class: "check" }, fileCb, t("models.deleteFile"))
: null
);
modal(t("common.delete") + ": " + m.id, body, async () => {
try {
const r = await api.deleteModel(m.id, isLocal && fileCb.checked);
toast(
r.file_removed ? t("models.deletedWithFile") : t("toast.deleted"),
"good"
);
await reload();
} catch (e) {
toast(String(e.message || e), "bad");
return false;
}
}, t("common.delete"));
}, },
}, },
t("common.delete") t("common.delete")
+91 -66
View File
@@ -207,11 +207,67 @@ export async function renderSettings() {
const cmfManage = check(t("settings.cmf.manage"), s.cmf?.manage_server, t("settings.cmf.manageHelp")); const cmfManage = check(t("settings.cmf.manage"), s.cmf?.manage_server, t("settings.cmf.manageHelp"));
const cmfAutoInstall = check(t("settings.cmf.autoInstall"), s.cmf?.auto_install, t("settings.cmf.autoInstallHelp")); const cmfAutoInstall = check(t("settings.cmf.autoInstall"), s.cmf?.auto_install, t("settings.cmf.autoInstallHelp"));
const cmfAutoUpdate = check(t("settings.cmf.autoUpdate"), s.cmf?.auto_update, t("settings.cmf.autoUpdateHelp")); const cmfAutoUpdate = check(t("settings.cmf.autoUpdate"), s.cmf?.auto_update, t("settings.cmf.autoUpdateHelp"));
const cmfModel = h("input", { value: s.cmf?.local_model || "", placeholder: "models/my-model.cmf" }); // ── managed local models: one supervised `cortiq serve` per row ──
const cmfPort = h("input", { type: "number", value: s.cmf?.local_port ?? 8081 }); const serverRows = [];
const cmfThreads = h("input", { type: "number", min: 0, value: s.cmf?.threads ?? 8 }); const serversWrap = h("div", {});
const cmfGpu = check(t("settings.cmf.gpu"), s.cmf?.gpu, t("settings.cmf.gpuHelp")); const legacyServers = s.cmf?.local_model
const cmfModelIdVal = s.cmf?.model_id || "cmf-local"; // auto — no manual "model id" field ? [{ id: s.cmf.model_id || "cmf-local", model: s.cmf.local_model, port: s.cmf.local_port ?? 8081, threads: s.cmf.threads ?? 8, gpu: !!s.cmf.gpu }]
: [];
const initServers = (s.cmf?.servers && s.cmf.servers.length) ? s.cmf.servers : legacyServers;
// available .cmf files under models_dir → offer as a dropdown, not a typed path
const cmfFiles = ((await api.cmfFiles().catch(() => ({ files: [] }))).files) || [];
const fileLabel = (f) => f.name + (f.size ? ` (${(f.size / 1073741824).toFixed(f.size >= 1073741824 ? 1 : 2)} GB)` : f.is_dir ? " (dir)" : "");
function autoId(path) {
const base = (path || "").split("/").pop().replace(/\.cmf$/i, "");
return base ? "cmf-" + base.replace(/[^a-zA-Z0-9_-]/g, "-") : "";
}
function makeServerRow(sv) {
sv = sv || { id: "", model: "", port: 8090, threads: 8, gpu: false };
const idIn = h("input", { value: sv.id || "", placeholder: "cmf-id", style: "width:130px" });
// pick from existing .cmf files rather than typing a path
const modelSel = h("select", { style: "flex:2;min-width:150px" });
const seen = new Set();
modelSel.appendChild(h("option", { value: "" }, "— " + t("settings.cmf.pick") + " —"));
cmfFiles.forEach((f) => { modelSel.appendChild(h("option", { value: f.path }, fileLabel(f))); seen.add(f.path); });
if (sv.model && !seen.has(sv.model)) modelSel.appendChild(h("option", { value: sv.model }, sv.model));
modelSel.value = sv.model || "";
const portIn = h("input", { type: "number", value: sv.port ?? 8090, style: "width:82px" });
const threadsIn = h("input", { type: "number", min: 0, value: sv.threads ?? 8, style: "width:60px", title: t("settings.cmf.threadsHelp") });
const gpuIn = h("input", { type: "checkbox", checked: sv.gpu ? true : null, title: t("settings.cmf.gpuHelp") });
const checkBtn = h("button", { class: "btn small", type: "button" }, t("settings.cmf.checkPort"));
const removeBtn = h("button", { class: "btn small danger", type: "button", title: t("common.delete") }, "✕");
const st = h("span", { class: "small" });
modelSel.addEventListener("change", () => { if (!idIn.value || idIn._auto) { idIn.value = autoId(modelSel.value); idIn._auto = true; } });
idIn.addEventListener("input", () => { idIn._auto = false; });
checkBtn.addEventListener("click", async () => {
const p = parseInt(portIn.value, 10); if (!p) return;
checkBtn.disabled = true; mount(st, t("status.checking"));
try {
const r = await api.cmfPortCheck(p);
if (r.available) mount(st, h("span", { style: "color:var(--good)" }, "✓ " + t("settings.cmf.portFree")));
else if (r.managed) {
// held by this gateway's own managed server — replacing the model on
// this port is fine, it frees up on restart
mount(st, h("span", { style: "color:var(--accent)" }, "ℹ " + t("settings.cmf.portOwn")));
} else {
const kids = [h("span", { style: "color:var(--bad)" }, "✗ " + t("settings.cmf.portBusy"))];
if (r.suggested) { const b = h("button", { class: "btn small", type: "button" }, t("settings.cmf.useSuggested", { port: r.suggested })); b.addEventListener("click", () => { portIn.value = r.suggested; mount(st, ""); }); kids.push(b); }
mount(st, h("span", {}, ...kids));
}
} catch (e) { mount(st, String(e.message || e)); }
checkBtn.disabled = false;
});
const row = h("div", { class: "flex wrap", style: "gap:6px;align-items:center;margin:6px 0" },
idIn, modelSel, portIn, checkBtn, threadsIn, h("label", { class: "flex", style: "gap:4px" }, gpuIn, "GPU"), removeBtn, st);
const entry = { node: row, read: () => ({ id: idIn.value.trim() || autoId(modelSel.value), model: modelSel.value.trim(), port: parseInt(portIn.value) || 8090, threads: parseInt(threadsIn.value) || 0, gpu: gpuIn.checked }) };
removeBtn.addEventListener("click", () => { const i = serverRows.indexOf(entry); if (i >= 0) serverRows.splice(i, 1); row.remove(); });
serverRows.push(entry);
serversWrap.appendChild(row);
return entry;
}
initServers.forEach((sv) => makeServerRow(sv));
const addServerBtn = h("button", { class: "btn small", type: "button" }, t("settings.cmf.addServer"));
addServerBtn.addEventListener("click", () => makeServerRow());
// "Install cortiq" — the CLI that RUNS the .cmf format. Must be present before // "Install cortiq" — the CLI that RUNS the .cmf format. Must be present before
// a local model server can start; this is step 1 for a new user. // a local model server can start; this is step 1 for a new user.
const cmfStatusEl = h("span", { class: "muted" }, "…"); const cmfStatusEl = h("span", { class: "muted" }, "…");
@@ -228,14 +284,17 @@ export async function renderSettings() {
? h("span", {}, h("span", { class: "badge ok" }, "✓"), " cortiq " + (st.version || "")) ? h("span", {}, h("span", { class: "badge ok" }, "✓"), " cortiq " + (st.version || ""))
: h("span", {}, h("span", { class: "badge bad" }, "×"), " " + t("settings.cmf.notInstalled")) : h("span", {}, h("span", { class: "badge bad" }, "×"), " " + t("settings.cmf.notInstalled"))
); );
// reflect the managed server state under the "run locally" toggle // per-server serving indicators
if (st.manage_server && st.running) { const servers = st.servers || [];
mount(cmfServingEl, h("span", { style: "color:var(--good)" }, if (!servers.length) {
"▶ " + t("settings.cmf.serving", { model: st.model_id || "", port: st.local_port }) + (st.healthy ? "" : " …")));
} else if (st.last_error) {
mount(cmfServingEl, h("span", { style: "color:var(--bad)" }, "✗ " + st.last_error));
} else {
mount(cmfServingEl, ""); mount(cmfServingEl, "");
} else {
mount(cmfServingEl, h("div", {}, ...servers.map((sv) => {
if (sv.last_error) return h("div", { style: "color:var(--bad)" }, "✗ " + sv.id + ": " + sv.last_error);
if (sv.running) return h("div", { style: "color:var(--good)" },
"▶ " + t("settings.cmf.serving", { model: sv.id, port: sv.port }) + (sv.healthy ? "" : " …"));
return h("div", { class: "muted" }, "• " + sv.id + " :" + sv.port);
})));
} }
} catch (_) { } catch (_) {
mount(cmfStatusEl, "—"); mount(cmfStatusEl, "—");
@@ -251,7 +310,6 @@ export async function renderSettings() {
await new Promise((r) => setTimeout(r, 3000)); await new Promise((r) => setTimeout(r, 3000));
const st = await api.cmfStatus(); const st = await api.cmfStatus();
if (st.installed) { toast(t("settings.cmf.installedOk"), "good"); break; } if (st.installed) { toast(t("settings.cmf.installedOk"), "good"); break; }
if (st.last_error) { toast(st.last_error, "bad"); break; }
} }
} catch (e) { } catch (e) {
toast(String(e.message || e), "bad"); toast(String(e.message || e), "bad");
@@ -261,36 +319,6 @@ export async function renderSettings() {
}); });
refreshCmfStatus(); refreshCmfStatus();
// Port availability check — let the user confirm the chosen port is free
// before enabling the managed server (the default 8081 often clashes).
const cmfPortStatus = h("span", { class: "small", style: "margin-left:8px" });
const cmfPortCheckBtn = h("button", { class: "btn small", type: "button" }, t("settings.cmf.checkPort"));
async function checkPort() {
const p = parseInt(cmfPort.value, 10);
if (!p) { mount(cmfPortStatus, ""); return; }
cmfPortCheckBtn.disabled = true;
mount(cmfPortStatus, t("status.checking"));
try {
const r = await api.cmfPortCheck(p);
if (r.available) {
mount(cmfPortStatus, h("span", { style: "color:var(--good)" }, "✓ " + t("settings.cmf.portFree")));
} else {
const kids = [h("span", { style: "color:var(--bad)" }, "✗ " + t("settings.cmf.portBusy"))];
if (r.suggested) {
const useBtn = h("button", { class: "btn small", type: "button", style: "margin-left:8px" },
t("settings.cmf.useSuggested", { port: r.suggested }));
useBtn.addEventListener("click", () => { cmfPort.value = r.suggested; checkPort(); });
kids.push(useBtn);
}
mount(cmfPortStatus, h("span", {}, ...kids));
}
} catch (e) {
mount(cmfPortStatus, String(e.message || e));
}
cmfPortCheckBtn.disabled = false;
}
cmfPortCheckBtn.addEventListener("click", checkPort);
const saveBtn = h("button", { class: "btn primary" }, t("common.save")); const saveBtn = h("button", { class: "btn primary" }, t("common.save"));
saveBtn.addEventListener("click", async () => { saveBtn.addEventListener("click", async () => {
saveBtn.disabled = true; saveBtn.disabled = true;
@@ -335,11 +363,9 @@ export async function renderSettings() {
manage_server: cmfManage.cb.checked, manage_server: cmfManage.cb.checked,
auto_install: cmfAutoInstall.cb.checked, auto_install: cmfAutoInstall.cb.checked,
auto_update: cmfAutoUpdate.cb.checked, auto_update: cmfAutoUpdate.cb.checked,
local_model: cmfModel.value.trim(), // the managed-models list is the source of truth; clear the legacy single
local_port: parseInt(cmfPort.value) || 8081, servers: serverRows.map((r) => r.read()).filter((sv) => sv.model && sv.id),
model_id: cmfModelIdVal, local_model: "",
threads: parseInt(cmfThreads.value) || 0,
gpu: cmfGpu.cb.checked,
}, },
}; };
// serde: drop null optionals // serde: drop null optionals
@@ -420,9 +446,11 @@ export async function renderSettings() {
"div", "div",
{ class: "card" }, { class: "card" },
h("div", { class: "card-head" }, h("h3", {}, t("settings.route"))), h("div", { class: "card-head" }, h("h3", {}, t("settings.route"))),
field(t("settings.route.strategy"), strategySel), field(t("settings.route.strategy"), strategySel, null, t("settings.route.strategyHelp")),
h("div", { class: "row" }, field(t("settings.route.maxChars"), maxChars), field(t("settings.route.cacheTtl"), cacheTtl)), h("div", { class: "row" },
field(t("settings.route.profile"), profileSel) field(t("settings.route.maxChars"), maxChars, null, t("settings.route.maxCharsHelp")),
field(t("settings.route.cacheTtl"), cacheTtl, null, t("settings.route.cacheTtlHelp"))),
field(t("settings.route.profile"), profileSel, null, t("settings.route.profileHelp"))
), ),
h( h(
"div", "div",
@@ -477,24 +505,21 @@ export async function renderSettings() {
), ),
h("div", { class: "row" }, cmfAutoInstall.node, cmfAutoUpdate.node), h("div", { class: "row" }, cmfAutoInstall.node, cmfAutoUpdate.node),
h("div", { class: "divider" }), h("div", { class: "divider" }),
// Step 2 — add a model. Models are imported in the Models/Import tab; here // Step 2 — managed local models: one supervised cortiq serve per row, each
// you just point the local server at the .cmf file it produced. // on its own port. Add as many as you have .cmf files + free ports.
h("div", { class: "field" }, h("span", {}, "2. " + t("settings.cmf.addModel"), " ", help(t("settings.cmf.addModelHelp"))), h("div", { class: "field" },
h("span", {}, "2. " + t("settings.cmf.models"), " ", help(t("settings.cmf.modelsHelp"))),
h("div", { class: "hint" }, t("settings.cmf.rowHint")),
h("div", { class: "hint" }, t("settings.cmf.importNote"), h("div", { class: "hint" }, t("settings.cmf.importNote"),
" ", h("a", { href: "#/import" }, t("settings.cmf.importLink") + " ↗"))), " ", h("a", { href: "#/import" }, t("settings.cmf.importLink") + " ↗"))),
field(t("settings.cmf.model"), cmfModel, null, t("settings.cmf.modelHelp")), serversWrap,
h("div", { class: "divider" }), addServerBtn,
// Step 3 — run it locally + performance.
cmfManage.node,
cmfServingEl, cmfServingEl,
h("div", { class: "row" }, h("div", { class: "hint" }, t("settings.cmf.speedNote")),
field(t("settings.cmf.port"), h("div", { class: "divider" }),
h("div", { class: "flex wrap" }, cmfPort, cmfPortCheckBtn, cmfPortStatus), // Step 3 — how the gateway uses them.
null, t("settings.cmf.portHelp")), cmfManage.node,
field(t("settings.cmf.threads"), cmfThreads, null, t("settings.cmf.threadsHelp"))), cmfLocalOnly.node
cmfGpu.node,
cmfLocalOnly.node,
h("div", { class: "hint" }, t("settings.cmf.speedNote"))
) )
), ),
h( h(