Compare commits

...
7 Commits
Author SHA1 Message Date
infosave2007 d93ac537c8 docs(import): human-readable labels + hover help for the O(1) knobs
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
Landmarks / Exact window / Sink keys now carry a '?' tooltip explaining in
plain language what each knob does, its trade-off, and the validated default
(32 / 128 / 4), in EN + RU.
2026-07-14 11:19:00 +03:00
infosave2007 4220bc27c2 feat(models): ask whether to delete the model file from disk
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
- The delete button now opens a dialog; for local models it includes an "also
  delete the model file from disk" checkbox (checked by default), so a pool
  entry can be removed while keeping the .cmf for later.
- DELETE /admin/api/models/:id accepts ?delete_file=true|false (default true).
  The shared-file guard still applies: a file referenced by another model is
  never removed.
- cmf_file_for_model now treats any loopback base_url as local (managed servers
  can sit on any port, not just the legacy local_port).
2026-07-14 11:09:44 +03:00
infosave2007 6fdbe0e159 fix(settings): replacing a managed local model no longer breaks save
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
- put_settings now heals [routing] after the managed-servers list changes: a
  removed server whose port is taken over by a new one is treated as a
  replacement (old id → new id in default + every tier); truly vanished ids are
  stripped; an emptied default falls back to the first managed server / pool
  model. Previously the stale id failed validation and the save errored.
- /admin/api/cmf/port now returns managed:true when the port is held by this
  gateway's own managed server; the Settings port check shows that as info
  ("freed on restart, replacing is fine") instead of a red "port busy".
2026-07-14 10:30:39 +03:00
infosave2007 243d8f4c7d feat(import): O(1) attention options in Advanced (cortiq >= 0.2.0)
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
- New Advanced controls on the import page: O(1) attention spec (off / all /
  deep N / explicit layer list) plus the --o1-m / --o1-window / --o1-sink knobs
  with validated defaults (32 / 128 / 4) and explanatory hints.
- Backend passes --o1* to the native `cortiq convert`; fails fast with clear
  messages when combined with Python-only options, used on GGUF imports, or
  when the installed cortiq predates 0.2.0 (the hint needs the new runtime).
- The hint is recorded in the .cmf header (weights byte-identical); the runtime
  applies it at load. Verified end-to-end: converted a model with --o1 all and
  the runtime logged "o1 nystrom attention: 2/2 layer(s)".
2026-07-14 10:16:54 +03:00
infosave2007 4f72711a1e feat(dashboard): windowed stats per range + clear-logs button
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
- Stats snapshot now computes the KPI totals AND the breakdown from the
  in-window per-minute buckets (previously they used all-time accumulators, so
  switching 1h/24h/7d changed only the chart, not the numbers). Buckets now
  carry failovers + per-group breakdown; Prometheus keeps all-time counters.
- New DELETE /admin/api/stats clears all in-memory stats and truncates the
  JSONL log; the dashboard gets a "Clear logs" button (with confirm).
2026-07-09 17:07:05 +03:00
infosave2007 7855472499 docs(settings): add help tooltips to routing-input fields (strategy, max chars, cache TTL, profile)
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
2026-07-09 16:49:16 +03:00
infosave2007 a6ed974ece feat(models): auto-derive id from model name; actionable probe auth errors
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
- The model id is now auto-filled from the model name as you type, and if left
  blank it's derived on save (slug of the last path segment, e.g.
  "qwen/qwen3.7-max" → "qwen3-7-max") — saving no longer errors on a blank id.
- probe/test auth failures (401 / "User not found" / "No auth") now append a
  hint to check the API key (for OpenRouter: use an inference key, not a
  provisioning/management key).
2026-07-09 16:30:48 +03:00
13 changed files with 453 additions and 77 deletions
Generated
+1 -1
View File
@@ -285,7 +285,7 @@ checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570"
[[package]]
name = "cortiq-gateway"
version = "0.2.11"
version = "0.2.18"
dependencies = [
"anyhow",
"arc-swap",
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "cortiq-gateway"
version = "0.2.11"
version = "0.2.18"
edition = "2021"
rust-version = "1.75"
license = "Apache-2.0"
+112 -10
View File
@@ -165,7 +165,7 @@ pub fn api_routes(admin_token: String) -> Router<SharedState> {
"/admin/api/secrets",
get(list_secrets).put(set_secret).delete(clear_secret),
)
.route("/admin/api/stats", get(get_stats))
.route("/admin/api/stats", get(get_stats).delete(clear_stats))
.route("/admin/api/shadow", get(get_shadow))
.route("/admin/api/hf/search", get(hf_search))
.route("/admin/api/import", get(list_imports).post(start_import))
@@ -606,7 +606,10 @@ fn cmf_file_for_model(
cmf: &crate::config::CmfCfg,
m: &crate::config::ModelCfg,
) -> Option<std::path::PathBuf> {
// Any loopback URL counts as local — managed servers may sit on any port.
let local = m.base_url == cmf.cortiq_server_url
|| m.base_url.contains("//127.0.0.1:")
|| m.base_url.contains("//localhost:")
|| m.base_url
.contains(&format!("{}:{}", cmf.local_host, cmf.local_port));
let stem = m.model.trim();
@@ -646,12 +649,24 @@ fn cmf_path_referenced(cfg: &crate::config::Config, path: &std::path::Path) -> b
.any(|m| cmf_file_for_model(&cfg.cmf, m).as_deref() == Some(path))
}
async fn delete_model(State(state): State<SharedState>, Path(id): Path<String>) -> ApiResult {
#[derive(Deserialize)]
struct DeleteModelQuery {
/// Also remove the local `.cmf` from disk. Defaults to true (the historical
/// behaviour); the UI asks the user and passes it explicitly.
delete_file: Option<bool>,
}
async fn delete_model(
State(state): State<SharedState>,
Path(id): Path<String>,
Query(q): Query<DeleteModelQuery>,
) -> ApiResult {
let mut cfg = current_cfg(&state);
let delete_file = q.delete_file.unwrap_or(true);
// Managed local models live in [cmf], not [[models]]. Deleting one stops its
// server, removes it from the managed config, and deletes its file — unless
// another model still points at the same file.
// server, removes it from the managed config, and (when asked) deletes its
// file — unless another model still points at the same file.
let managed = cfg.cmf.effective_servers().into_iter().find(|s| s.id == id);
if let Some(server) = managed {
if !cfg.models.iter().any(|x| x.id == id) {
@@ -666,10 +681,10 @@ async fn delete_model(State(state): State<SharedState>, Path(id): Path<String>)
let shared = cmf_path_referenced(&cfg, &path);
state.cmf.stop_one(&id).await;
state.reload(cfg)?;
let file_removed = if shared {
false
} else {
let file_removed = if delete_file && !shared {
remove_cmf_path(&path)
} else {
false
};
return ok(json!({ "ok": true, "file_removed": file_removed }));
}
@@ -685,8 +700,8 @@ async fn delete_model(State(state): State<SharedState>, Path(id): Path<String>)
let file = victim
.as_ref()
.and_then(|m| cmf_file_for_model(&cfg.cmf, m));
// Only remove the file if no remaining model still references it.
let file = file.filter(|p| !cmf_path_referenced(&cfg, p));
// Only remove the file if asked to AND no remaining model still references it.
let file = file.filter(|p| delete_file && !cmf_path_referenced(&cfg, p));
state.reload(cfg)?;
let file_removed = file.map(|p| remove_cmf_path(&p)).unwrap_or(false);
ok(json!({ "ok": true, "file_removed": file_removed }))
@@ -749,7 +764,21 @@ async fn probe_model(State(state): State<SharedState>, Path(id): Path<String>) -
let latency_ms = started.elapsed().as_millis() as u64;
match res {
Ok(_) => ok(json!({ "ok": true, "latency_ms": latency_ms })),
Err(e) => ok(json!({ "ok": false, "latency_ms": latency_ms, "error": e.to_string() })),
Err(e) => {
let msg = e.to_string();
// Make auth failures actionable — a raw "User not found" (OpenRouter's
// 401) usually means the API key is wrong or the wrong type.
let hint = if msg.contains("401")
|| msg.contains("Unauthorized")
|| msg.contains("User not found")
|| msg.contains("No auth")
{
" — check the API key (for OpenRouter use an inference key, not a provisioning/management key)"
} else {
""
};
ok(json!({ "ok": false, "latency_ms": latency_ms, "error": format!("{msg}{hint}") }))
}
}
}
@@ -891,12 +920,75 @@ async fn put_settings(State(state): State<SharedState>, Json(b): Json<SettingsBo
{
needs_restart = true;
}
let old_servers = cfg.cmf.effective_servers();
cfg.cmf = v;
// Replacing or renaming a managed model must not brick the save: routing
// may still reference the old id. Map a vanished id to its replacement
// (the new server on the same port), strip ids that are truly gone, and
// repair the default so validate() passes.
heal_routing_after_cmf_change(&mut cfg, &old_servers);
}
state.reload(cfg)?;
ok(json!({ "ok": true, "needs_restart": needs_restart }))
}
/// After the managed-servers list changed, rewrite [routing] so it only
/// references ids that still exist. A removed server whose port is now held by
/// a NEW server is treated as a replacement (old id → new id); other vanished
/// ids are stripped. An emptied default falls back to the first managed server,
/// then the first pool model.
fn heal_routing_after_cmf_change(cfg: &mut Config, old_servers: &[crate::config::CmfServer]) {
let servers = cfg.cmf.effective_servers();
let known: std::collections::HashSet<String> = cfg
.models
.iter()
.map(|m| m.id.clone())
.chain(servers.iter().map(|s| s.id.clone()))
.collect();
let new_by_port: HashMap<u16, String> =
servers.iter().map(|s| (s.port, s.id.clone())).collect();
let mut rename: HashMap<String, String> = HashMap::new();
for old in old_servers {
if !known.contains(&old.id) {
if let Some(new_id) = new_by_port.get(&old.port) {
rename.insert(old.id.clone(), new_id.clone());
}
}
}
let fix = |id: &str| -> Option<String> {
if known.contains(id) {
Some(id.to_string())
} else {
rename.get(id).cloned()
}
};
if !cfg.routing.default.is_empty() {
cfg.routing.default = fix(&cfg.routing.default).unwrap_or_default();
}
if cfg.routing.default.is_empty() {
cfg.routing.default = servers
.first()
.map(|s| s.id.clone())
.or_else(|| cfg.models.first().map(|m| m.id.clone()))
.unwrap_or_default();
}
for tt in cfg.routing.tiers.values_mut() {
let list = match tt {
TierTargets::List(v) => v.clone(),
TierTargets::One(s) => vec![s.clone()],
};
let mut healed: Vec<String> = Vec::new();
for id in &list {
if let Some(new_id) = fix(id) {
if !healed.contains(&new_id) {
healed.push(new_id);
}
}
}
*tt = TierTargets::List(healed);
}
}
// ───────────────────────────── cmf runtime ─────────────────────────────────
/// Status of the `cortiq` CLI (the CMF format runtime): is it installed, which
/// version, and whether a managed server is running. Drives the "Install cortiq"
@@ -1014,6 +1106,9 @@ async fn cmf_port_check(State(state): State<SharedState>, Query(q): Query<PortQu
ok(json!({
"port": port,
"available": available,
// true = held by this gateway's own managed server (a replaced/edited
// row frees it on restart) — the UI shows this as info, not an error
"managed": running_here,
"detail": detail,
"suggested": suggested,
}))
@@ -1183,6 +1278,13 @@ async fn get_stats(State(state): State<SharedState>, Query(q): Query<StatsQuery>
ok(snap)
}
/// Clear all recorded stats/logs (in-memory aggregates, series, recent requests)
/// and truncate the JSONL log — the dashboard "Clear logs" button.
async fn clear_stats(State(state): State<SharedState>) -> ApiResult {
state.stats.clear();
ok(json!({ "ok": true }))
}
/// Self-warming shadow loop: per-task-type promotion state, Wilson-LB of the
/// judged pass-rate, and whether the local model is serving it yet.
async fn get_shadow(State(state): State<SharedState>) -> ApiResult {
+1 -1
View File
@@ -238,7 +238,7 @@ fn parse_ver(s: &str) -> (u32, u32, u32) {
)
}
fn version_lt(a: &str, b: &str) -> bool {
pub(crate) fn version_lt(a: &str, b: &str) -> bool {
parse_ver(a) < parse_ver(b)
}
+59
View File
@@ -71,6 +71,20 @@ pub struct ImportParams {
pub shard_max_gb: Option<f32>,
#[serde(default)]
pub skip_mtp: bool,
/// O(1) Nyström attention hint (native converter, cortiq ≥ 0.2.0):
/// `all` | `deepN` | `i,j,k`. Weights pass through unchanged — the runtime
/// reads the hint at load. Empty/None/"off" = exact attention.
#[serde(default)]
pub o1: Option<String>,
/// Landmark budget for the o1 hint (validated default 32).
#[serde(default)]
pub o1_m: Option<usize>,
/// Exact-window width for the o1 hint (validated default 128).
#[serde(default)]
pub o1_window: Option<usize>,
/// Permanent exact sink keys for the o1 hint (validated default 4).
#[serde(default)]
pub o1_sink: Option<usize>,
}
fn default_quant() -> String {
"Q8_2F".into()
@@ -269,6 +283,34 @@ pub fn start_import(store: Arc<JobStore>, cfg: &CmfCfg, p: ImportParams) -> Resu
cfg.converter
));
}
// O(1) attention hint — native-converter feature (cortiq ≥ 0.2.0). Fail fast
// with a clear message instead of a mid-job "unexpected argument" error.
let o1 =
p.o1.as_deref()
.map(str::trim)
.filter(|s| !s.is_empty() && *s != "off")
.map(str::to_string);
if o1.is_some() {
if use_python {
return Err(
"O(1) attention is supported by the native converter only — remove the \
linear-core / v-bit shape / mean-bits options to use it"
.into(),
);
}
if p.repo.to_ascii_lowercase().contains("gguf") {
return Err("O(1) attention is not supported for GGUF imports yet".into());
}
let ver = crate::cmf_runtime::installed_version(&cfg.cortiq_bin);
if let Some(v) = &ver {
if crate::cmf_runtime::version_lt(v, "0.2.0") {
return Err(format!(
"O(1) attention needs cortiq ≥ 0.2.0 (installed: {v}) — update the runtime \
in Settings → Local models"
));
}
}
}
let _ = std::fs::create_dir_all(&cfg.models_dir);
let base = if p.name.trim().is_empty() {
sanitize(p.repo.rsplit('/').next().unwrap_or(&p.repo))
@@ -384,6 +426,23 @@ pub fn start_import(store: Arc<JobStore>, cfg: &CmfCfg, p: ImportParams) -> Resu
a.push("--hf-token".into());
a.push(t.clone());
}
// O(1) attention hint (weights unchanged; the runtime reads it at load)
if let Some(spec) = &o1 {
a.push("--o1".into());
a.push(spec.clone());
if let Some(m) = p.o1_m {
a.push("--o1-m".into());
a.push(m.to_string());
}
if let Some(w) = p.o1_window {
a.push("--o1-window".into());
a.push(w.to_string());
}
if let Some(s) = p.o1_sink {
a.push("--o1-sink".into());
a.push(s.to_string());
}
}
(
cfg.cortiq_bin.clone(),
a,
+93 -41
View File
@@ -83,9 +83,23 @@ impl Totals {
self.cost_usd += r.cost_usd;
self.latency_ms_sum += r.latency_ms;
}
/// Accumulate another Totals into this one (for windowed aggregation).
fn merge(&mut self, o: &Totals) {
self.requests += o.requests;
self.ok += o.ok;
self.errors += o.errors;
self.failovers += o.failovers;
self.prompt_tokens += o.prompt_tokens;
self.completion_tokens += o.completion_tokens;
self.cost_usd += o.cost_usd;
self.latency_ms_sum += o.latency_ms_sum;
}
}
/// Per-minute bucket for time-series charts.
/// Per-minute bucket. Beyond the chart fields it carries the per-group breakdown
/// and failovers so the admin snapshot can be computed for ANY time window by
/// summing the in-window buckets (the internal fields are not serialized into
/// the `series` payload).
#[derive(Clone, Debug, Default, Serialize)]
pub struct Bucket {
pub minute: u64, // unix seconds, rounded down to the minute
@@ -95,15 +109,25 @@ pub struct Bucket {
pub completion_tokens: u64,
pub cost_usd: f64,
pub latency_ms_sum: u64,
#[serde(skip)]
pub failovers: u64,
#[serde(skip)]
pub by_model: HashMap<String, Totals>,
#[serde(skip)]
pub by_account: HashMap<String, Totals>,
#[serde(skip)]
pub by_tier: HashMap<String, Totals>,
#[serde(skip)]
pub by_label: HashMap<String, Totals>,
}
#[derive(Default)]
struct Inner {
/// All-time totals + per-model (used by the Prometheus counters, which must
/// be monotonic — the windowed admin snapshot is derived from the buckets).
total: Totals,
by_model: HashMap<String, Totals>,
by_account: HashMap<String, Totals>,
by_tier: HashMap<String, Totals>,
by_label: HashMap<String, Totals>,
/// Per-minute buckets — the single source for the windowed admin snapshot.
buckets: VecDeque<Bucket>,
recent: VecDeque<RequestRecord>,
}
@@ -166,31 +190,14 @@ impl Stats {
}
fn apply_aggregates(inner: &mut Inner, rec: &RequestRecord, retention_secs: u64) {
// all-time counters for Prometheus; the windowed snapshot is derived
// from the buckets below.
inner.total.apply(rec);
inner
.by_model
.entry(rec.model_id.clone())
.or_default()
.apply(rec);
inner
.by_account
.entry(if rec.account.is_empty() {
"anonymous".to_string()
} else {
rec.account.clone()
})
.or_default()
.apply(rec);
inner
.by_tier
.entry(rec.tier.clone())
.or_default()
.apply(rec);
inner
.by_label
.entry(rec.task_label.clone())
.or_default()
.apply(rec);
let minute = rec.ts - (rec.ts % 60);
match inner.buckets.back_mut() {
@@ -250,13 +257,31 @@ impl Stats {
.filter(|b| b.minute >= cutoff)
.collect();
let breakdown_src = match groupby {
"account" => &inner.by_account,
"tier" => &inner.by_tier,
"label" => &inner.by_label,
_ => &inner.by_model,
};
let mut breakdown: Vec<serde_json::Value> = breakdown_src
// Windowed totals + breakdown = sum of the in-window buckets, so the KPIs
// and the breakdown recompute whenever the range changes.
let mut win = Totals::default();
let mut group: HashMap<String, Totals> = HashMap::new();
for b in &series {
win.requests += b.requests;
win.errors += b.errors;
win.ok += b.requests.saturating_sub(b.errors);
win.failovers += b.failovers;
win.prompt_tokens += b.prompt_tokens;
win.completion_tokens += b.completion_tokens;
win.cost_usd += b.cost_usd;
win.latency_ms_sum += b.latency_ms_sum;
let src = match groupby {
"account" => &b.by_account,
"tier" => &b.by_tier,
"label" => &b.by_label,
_ => &b.by_model,
};
for (k, t) in src {
group.entry(k.clone()).or_default().merge(t);
}
}
let mut breakdown: Vec<serde_json::Value> = group
.iter()
.map(|(k, t)| {
serde_json::json!({
@@ -280,17 +305,17 @@ impl Stats {
serde_json::json!({
"totals": {
"requests": inner.total.requests,
"ok": inner.total.ok,
"errors": inner.total.errors,
"failovers": inner.total.failovers,
"prompt_tokens": inner.total.prompt_tokens,
"completion_tokens": inner.total.completion_tokens,
"total_tokens": inner.total.prompt_tokens + inner.total.completion_tokens,
"cost_usd": inner.total.cost_usd,
"avg_latency_ms": avg(inner.total.latency_ms_sum, inner.total.requests),
"success_rate": if inner.total.requests > 0 {
inner.total.ok as f64 / inner.total.requests as f64
"requests": win.requests,
"ok": win.ok,
"errors": win.errors,
"failovers": win.failovers,
"prompt_tokens": win.prompt_tokens,
"completion_tokens": win.completion_tokens,
"total_tokens": win.prompt_tokens + win.completion_tokens,
"cost_usd": win.cost_usd,
"avg_latency_ms": avg(win.latency_ms_sum, win.requests),
"success_rate": if win.requests > 0 {
win.ok as f64 / win.requests as f64
} else { 0.0 },
},
"groupby": groupby,
@@ -299,6 +324,18 @@ impl Stats {
})
}
/// Reset all in-memory stats and truncate the JSONL log — the "clear logs"
/// admin action.
pub fn clear(&self) {
{
let mut inner = self.inner.lock().unwrap();
*inner = Inner::default();
}
if let Some(path) = &self.file {
let _ = std::fs::write(path, "");
}
}
/// Recent requests (newest first), with pagination.
pub fn recent(&self, limit: usize, offset: usize) -> Vec<RequestRecord> {
let inner = self.inner.lock().unwrap();
@@ -358,10 +395,25 @@ fn fill_bucket(b: &mut Bucket, r: &RequestRecord) {
if r.outcome != "ok" {
b.errors += 1;
}
if r.failover {
b.failovers += 1;
}
b.prompt_tokens += r.prompt_tokens as u64;
b.completion_tokens += r.completion_tokens as u64;
b.cost_usd += r.cost_usd;
b.latency_ms_sum += r.latency_ms;
// per-group breakdown, so the snapshot can be windowed by summing buckets
b.by_model.entry(r.model_id.clone()).or_default().apply(r);
b.by_account
.entry(if r.account.is_empty() {
"anonymous".to_string()
} else {
r.account.clone()
})
.or_default()
.apply(r);
b.by_tier.entry(r.tier.clone()).or_default().apply(r);
b.by_label.entry(r.task_label.clone()).or_default().apply(r);
}
fn avg(sum: u64, n: u64) -> u64 {
+3 -1
View File
@@ -60,7 +60,8 @@ export const api = {
listModels: () => j("GET", "/models"),
createModel: (b) => j("POST", "/models", b),
updateModel: (id, b) => j("PUT", "/models/" + encodeURIComponent(id), b),
deleteModel: (id) => j("DELETE", "/models/" + encodeURIComponent(id)),
deleteModel: (id, deleteFile = true) =>
j("DELETE", "/models/" + encodeURIComponent(id) + "?delete_file=" + (deleteFile ? "true" : "false")),
probeModel: (id) => j("POST", "/models/" + encodeURIComponent(id) + "/probe"),
// routing
@@ -87,6 +88,7 @@ export const api = {
// stats
stats: (range, groupby) => j("GET", `/stats?range=${range}&groupby=${groupby}`),
clearStats: () => j("DELETE", "/stats"),
requests: (limit = 50, offset = 0) => j("GET", `/requests?limit=${limit}&offset=${offset}`),
// playground
+28
View File
@@ -59,6 +59,9 @@ export default {
// dashboard
"dash.title": "Dashboard",
"dash.subtitle": "Live traffic, cost and routing analytics for your model pool.",
"dash.clear": "Clear logs",
"dash.clearConfirm": "Clear all recorded stats and logs? This cannot be undone.",
"dash.cleared": "Logs cleared",
"dash.range": "Range",
"dash.range.1h": "1h",
"dash.range.24h": "24h",
@@ -129,6 +132,9 @@ export default {
"models.form.secretMissing": "not set",
"models.form.secretPlaceholder": "set or replace key (write-only)",
"models.deleteConfirm": "Delete model \"{id}\"?",
"models.deleteFile": "also delete the model file from disk",
"models.deletedWithFile": "Deleted (file removed from disk)",
"models.form.idOrModel": "Enter an id or a model name",
"models.form.apiKey": "API Key",
"models.form.apiKeyPlaceholder": "paste your API key",
"models.form.apiKeyNotNeeded": "not required (local server)",
@@ -263,6 +269,7 @@ export default {
"settings.cmf.checkPort": "Check port",
"settings.cmf.portFree": "port is free",
"settings.cmf.portBusy": "port is in use",
"settings.cmf.portOwn": "in use by this gateway's current server — freed on restart, replacing is fine",
"settings.cmf.useSuggested": "use {port}",
"settings.cmf.serving": "serving {model} on :{port}",
"settings.cmf.intro": "Run models locally with the cortiq engine — works even offline or when the router is down.",
@@ -315,6 +322,10 @@ export default {
"settings.route.maxChars": "Max chars",
"settings.route.cacheTtl": "Cache TTL",
"settings.route.profile": "Default profile",
"settings.route.strategyHelp": "What text is sent to the router for classification: last_user = only the last user message; last_user_plus_system = last user message + system prompt; concat_all = the whole conversation.",
"settings.route.maxCharsHelp": "Max characters of that text sent to the router (longer is truncated). More = more accurate classification, but slower/pricier router calls.",
"settings.route.cacheTtlHelp": "How long a routing decision is cached for an identical request before the router is asked again (e.g. 60s, 5m).",
"settings.route.profileHelp": "Default routing profile when a request doesn't specify one: cost-saver (cheapest), balanced, or quality-first (best model). Biases which model the router picks.",
"settings.breaker": "Circuit breaker",
"settings.breaker.threshold": "Error threshold",
"settings.breaker.cooldown": "Cooldown",
@@ -392,4 +403,21 @@ export default {
"import.adv.shard": "Shard max (GB)",
"import.adv.noShard": "no sharding",
"import.adv.skipMtp": "skip MTP head",
"import.adv.o1": "O(1) attention",
"import.adv.o1Hint": "Constant-memory streaming attention: the KV cache is released after prefill, so long contexts don't grow memory. Weights stay byte-identical — only a load-time hint is recorded; you can switch back anytime (serve with --o1 off). Validated on Qwen3-0.6B: all layers ≈ ×1.18 perplexity.",
"import.adv.o1Off": "off — exact attention (default)",
"import.adv.o1All": "all — convert every softmax layer",
"import.adv.o1Deep": "deep N — only the N deepest layers",
"import.adv.o1Custom": "custom — explicit layer list",
"import.adv.o1DeepN": "N deep layers",
"import.adv.o1Layers": "Layers (comma-separated)",
"import.adv.o1mHint": "landmark budget (default 32)",
"import.adv.o1mLabel": "Landmarks (--o1-m)",
"import.adv.o1mHelp": "How many landmark tokens summarize the whole older history when attention is approximated. More = closer to exact attention but slower and more memory; fewer = faster but rougher. 32 is the validated sweet spot — change only if you know why.",
"import.adv.o1windowLabel": "Exact window (--o1-window)",
"import.adv.o1windowHelp": "The last N tokens always get exact (non-approximated) attention. This protects quality on the text the model just wrote/read. Bigger = better local quality, more memory. Default 128.",
"import.adv.o1sinkLabel": "Sink keys (--o1-sink)",
"import.adv.o1sinkHelp": "The first N tokens of the context are kept exact forever ('attention sinks'). Models lean on these anchors — keeping them exact stabilizes long generations. Default 4; rarely needs changing.",
"import.adv.o1windowHint": "exact-window width (default 128)",
"import.adv.o1sinkHint": "permanent exact sink keys (default 4)",
};
+28
View File
@@ -58,6 +58,9 @@ export default {
// dashboard
"dash.title": "Дашборд",
"dash.subtitle": "Трафик в реальном времени, расходы и аналитика роутинга по пулу моделей.",
"dash.clear": "Очистить логи",
"dash.clearConfirm": "Очистить всю статистику и логи? Отменить нельзя.",
"dash.cleared": "Логи очищены",
"dash.range": "Период",
"dash.range.1h": "1ч",
"dash.range.24h": "24ч",
@@ -128,6 +131,9 @@ export default {
"models.form.secretMissing": "не задан",
"models.form.secretPlaceholder": "задать или заменить ключ (только запись)",
"models.deleteConfirm": "Удалить модель \"{id}\"?",
"models.deleteFile": "также удалить файл модели с диска",
"models.deletedWithFile": "Удалено (файл стёрт с диска)",
"models.form.idOrModel": "Укажите id или имя модели",
"models.form.apiKey": "API-ключ",
"models.form.apiKeyPlaceholder": "вставьте API-ключ",
"models.form.apiKeyNotNeeded": "не нужен (локальный сервер)",
@@ -262,6 +268,7 @@ export default {
"settings.cmf.checkPort": "Проверить порт",
"settings.cmf.portFree": "порт свободен",
"settings.cmf.portBusy": "порт занят",
"settings.cmf.portOwn": "занят текущим сервером шлюза — освободится после перезапуска, замена допустима",
"settings.cmf.useSuggested": "занять {port}",
"settings.cmf.serving": "работает: {model} на :{port}",
"settings.cmf.intro": "Запускайте модели локально движком cortiq — работает даже офлайн или когда роутер недоступен.",
@@ -314,6 +321,10 @@ export default {
"settings.route.maxChars": "Макс. символов",
"settings.route.cacheTtl": "TTL кэша",
"settings.route.profile": "Профиль по умолчанию",
"settings.route.strategyHelp": "Какой текст отправляется роутеру для классификации: last_user — только последнее сообщение пользователя; last_user_plus_system — последнее сообщение + системный промпт; concat_all — вся переписка.",
"settings.route.maxCharsHelp": "Максимум символов этого текста, отправляемого роутеру (длиннее — обрезается). Больше = точнее классификация, но медленнее/дороже вызовы роутера.",
"settings.route.cacheTtlHelp": "Сколько кэшируется решение роутинга для одинакового запроса, прежде чем роутер спрашивается снова (напр. 60s, 5m).",
"settings.route.profileHelp": "Профиль роутинга по умолчанию, если запрос его не задаёт: cost-saver (дешевле), balanced (баланс) или quality-first (качество). Влияет на выбор модели роутером.",
"settings.breaker": "Автоматический выключатель",
"settings.breaker.threshold": "Порог ошибок",
"settings.breaker.cooldown": "Время восстановления",
@@ -391,4 +402,21 @@ export default {
"import.adv.shard": "Шард макс (ГБ)",
"import.adv.noShard": "без шардинга",
"import.adv.skipMtp": "пропустить MTP-голову",
"import.adv.o1": "O(1)-внимание",
"import.adv.o1Hint": "Потоковое внимание с постоянной памятью: KV-кэш освобождается после prefill, длинный контекст не раздувает память. Веса не меняются — записывается лишь подсказка на загрузку; вернуться можно всегда (запуск с --o1 off). Проверено на Qwen3-0.6B: все слои ≈ ×1.18 перплексии.",
"import.adv.o1Off": "выкл — точное внимание (по умолчанию)",
"import.adv.o1All": "all — конвертировать все softmax-слои",
"import.adv.o1Deep": "deep N — только N самых глубоких слоёв",
"import.adv.o1Custom": "вручную — явный список слоёв",
"import.adv.o1DeepN": "N глубоких слоёв",
"import.adv.o1Layers": "Слои (через запятую)",
"import.adv.o1mHint": "число опорных точек (по умолчанию 32)",
"import.adv.o1mLabel": "Опорные точки (--o1-m)",
"import.adv.o1mHelp": "Сколько опорных токенов представляют всю старую историю при аппроксимации внимания. Больше — ближе к точному вниманию, но медленнее и больше памяти; меньше — быстрее, но грубее. 32 — проверенное значение; меняйте, только если понимаете зачем.",
"import.adv.o1windowLabel": "Точное окно (--o1-window)",
"import.adv.o1windowHelp": "Последние N токенов всегда обрабатываются точным (не аппроксимированным) вниманием — это защищает качество на только что написанном/прочитанном тексте. Больше — лучше локальное качество, больше памяти. По умолчанию 128.",
"import.adv.o1sinkLabel": "Sink-ключи (--o1-sink)",
"import.adv.o1sinkHelp": "Первые N токенов контекста навсегда остаются точными («якоря внимания»). Модели опираются на эти якоря — их точность стабилизирует длинную генерацию. По умолчанию 4; менять почти никогда не нужно.",
"import.adv.o1windowHint": "ширина точного окна (по умолчанию 128)",
"import.adv.o1sinkHint": "постоянные точные sink-ключи (по умолчанию 4)",
};
+19 -5
View File
@@ -1,5 +1,5 @@
// Dashboard — KPIs, time-series charts, breakdown, recent requests, health.
import { h, mount, money, num, ms, pct, timeAgo } from "../ui.js";
import { h, mount, money, num, ms, pct, timeAgo, toast, confirmDialog } from "../ui.js";
import { t } from "../i18n.js";
import { api } from "../api.js";
import { lineChart, barList } from "../charts.js";
@@ -53,10 +53,24 @@ export async function renderDashboard() {
"div",
{ class: "flex" },
h("div", { class: "grow" }, h("h2", {}, t("dash.title")), h("p", {}, t("dash.subtitle"))),
seg(RANGES, range, (r) => {
range = r;
load();
}, (r) => t("dash.range." + r))
h("div", { class: "flex", style: "gap:8px;align-items:center" },
seg(RANGES, range, (r) => {
range = r;
load();
}, (r) => t("dash.range." + r)),
h("button", {
class: "btn sm danger",
onClick: async () => {
if (!(await confirmDialog(t("dash.clearConfirm")))) return;
try {
await api.clearStats();
toast(t("dash.cleared"), "good");
await load();
} catch (e) {
toast(String(e.message || e), "bad");
}
},
}, t("dash.clear")))
)
),
+50 -2
View File
@@ -15,8 +15,14 @@ const QUANTS = [
["F32", "import.q.f32"],
];
function field(label, control, hint) {
return h("label", { class: "field" }, h("span", {}, label), control,
// A small "?" badge with a hover/focus tooltip (same style as Settings).
function help(text) {
return text ? h("span", { class: "help", title: text, tabindex: "0", role: "img", "aria-label": text }, "?") : null;
}
function field(label, control, hint, helpText) {
return h("label", { class: "field" },
h("span", {}, label, helpText ? " " : null, help(helpText)),
control,
hint ? h("div", { class: "hint" }, hint) : null);
}
function opt(v, label, sel) {
@@ -125,6 +131,32 @@ export function renderImport() {
const meanBitsIn = h("input", { type: "number", step: "0.1", placeholder: "5.0" });
const shardIn = h("input", { type: "number", step: "0.5", placeholder: t("import.adv.noShard") });
const skipMtp = h("input", { type: "checkbox" });
// O(1) attention (cortiq ≥ 0.2.0): constant-memory streaming attention hint,
// weights byte-identical — the runtime applies it at load.
const o1Sel = h("select", {},
opt("", t("import.adv.o1Off")),
opt("all", t("import.adv.o1All")),
opt("deep", t("import.adv.o1Deep")),
opt("custom", t("import.adv.o1Custom")));
const o1DeepIn = h("input", { type: "number", min: "1", placeholder: "12" });
const o1CustomIn = h("input", { placeholder: "0,4,8" });
const o1M = h("input", { type: "number", min: "1", placeholder: "32" });
const o1Window = h("input", { type: "number", min: "0", placeholder: "128" });
const o1Sink = h("input", { type: "number", min: "0", placeholder: "4" });
const o1SpecRow = h("div", { class: "row", style: "display:none" },
field(t("import.adv.o1DeepN"), o1DeepIn),
field(t("import.adv.o1Layers"), o1CustomIn));
const o1Knobs = h("div", { class: "row", style: "display:none" },
field(t("import.adv.o1mLabel"), o1M, t("import.adv.o1mHint"), t("import.adv.o1mHelp")),
field(t("import.adv.o1windowLabel"), o1Window, t("import.adv.o1windowHint"), t("import.adv.o1windowHelp")),
field(t("import.adv.o1sinkLabel"), o1Sink, t("import.adv.o1sinkHint"), t("import.adv.o1sinkHelp")));
o1Sel.addEventListener("change", () => {
const on = !!o1Sel.value;
o1Knobs.style.display = on ? "" : "none";
o1SpecRow.style.display = o1Sel.value === "deep" || o1Sel.value === "custom" ? "" : "none";
o1DeepIn.parentElement.style.display = o1Sel.value === "deep" ? "" : "none";
o1CustomIn.parentElement.style.display = o1Sel.value === "custom" ? "" : "none";
});
const advBody = h(
"div",
{ class: "adv-body", style: "display:none" },
@@ -137,6 +169,9 @@ export function renderImport() {
h("div", { class: "row" },
field(t("import.adv.shard"), shardIn),
field("MTP", h("label", { class: "check" }, skipMtp, t("import.adv.skipMtp")))),
field(t("import.adv.o1"), o1Sel, t("import.adv.o1Hint")),
o1SpecRow,
o1Knobs,
);
const advToggle = h("button", { class: "btn ghost sm" }, "⚙ " + t("import.adv.title"));
advToggle.addEventListener("click", () => {
@@ -156,6 +191,19 @@ export function renderImport() {
if (vbitSel.value) params.vbit_shape = vbitSel.value;
if (meanBitsIn.value) params.mean_bits = parseFloat(meanBitsIn.value);
if (shardIn.value) params.shard_max_gb = parseFloat(shardIn.value);
if (o1Sel.value) {
// spec: all | deepN | explicit layer list
params.o1 =
o1Sel.value === "deep" ? "deep" + (parseInt(o1DeepIn.value) || 12)
: o1Sel.value === "custom" ? o1CustomIn.value.trim()
: "all";
if (!params.o1) delete params.o1;
else {
if (o1M.value) params.o1_m = parseInt(o1M.value);
if (o1Window.value) params.o1_window = parseInt(o1Window.value);
if (o1Sink.value) params.o1_sink = parseInt(o1Sink.value);
}
}
goBtn.disabled = true;
try {
await api.startImport(params);
+48 -11
View File
@@ -1,5 +1,5 @@
// Models — pool table with add/edit modal, probe, and per-model secret entry.
import { h, mount, modal, toast, confirmDialog, money } from "../ui.js";
import { h, mount, modal, toast, money } from "../ui.js";
import { t } from "../i18n.js";
import { api } from "../api.js";
import { appState } from "../app.js";
@@ -26,6 +26,13 @@ const PROVIDER_DEFAULTS = {
http: { base: "", keyEnv: "", needsKey: false, tier: "cheap", caps: ["tools"] },
};
// Derive a clean model id from a model name (last path segment, slugified).
// e.g. "qwen/qwen3.7-max" → "qwen3-7-max".
function slugId(name) {
const base = (name || "").split("/").pop() || "";
return base.toLowerCase().replace(/[^a-z0-9]+/g, "-").replace(/^-+|-+$/g, "");
}
function modelForm(meta, existing) {
const m = existing || { provider: "openai", kind: "chat", cost_tier: "mid", caps: ["tools"], price_in: 0, price_out: 0 };
const idIn = h("input", { value: m.id || "", placeholder: "my-model", disabled: existing ? true : null });
@@ -70,6 +77,17 @@ function modelForm(meta, existing) {
providerSel.addEventListener("change", applyDefaults);
if (!existing) applyDefaults();
// Auto-fill the id from the model name (until the user edits id by hand).
if (!existing) {
modelIn.addEventListener("input", () => {
if (!idIn.value || idIn._auto) {
idIn.value = slugId(modelIn.value);
idIn._auto = true;
}
});
idIn.addEventListener("input", () => { idIn._auto = false; });
}
async function testKey() {
if (!baseUrlIn.value.trim()) { mount(keyStatus, h("span", { style: "color:var(--bad)" }, t("models.form.baseUrl") + "?")); return; }
testBtn.disabled = true;
@@ -142,8 +160,11 @@ async function openModal(meta, existing, reload) {
const { node, getValue } = modelForm(meta, existing);
modal(existing ? t("models.form.editTitle") : t("models.form.addTitle"), node, async () => {
const { model, secret, keyEnv } = getValue();
// Forgot the id? Derive it from the model name so saving never errors on a
// blank id.
if (!model.id) model.id = slugId(model.model);
if (!model.id) {
toast(t("models.form.id") + "?", "bad");
toast(t("models.form.idOrModel"), "bad");
return false;
}
if (!model.api_key_env) delete model.api_key_env; // keep serde field optional
@@ -259,15 +280,31 @@ function row(m, meta, reload) {
"button",
{
class: "btn sm danger",
onClick: async () => {
if (!(await confirmDialog(t("models.deleteConfirm", { id: m.id })))) return;
try {
await api.deleteModel(m.id);
toast(t("toast.deleted"), "good");
await reload();
} catch (e) {
toast(String(e.message || e), "bad");
}
onClick: () => {
// local models may have a .cmf on disk — let the user decide its fate
const isLocal = m.managed || /127\.0\.0\.1|localhost/.test(m.base_url || "");
const fileCb = h("input", { type: "checkbox", checked: isLocal ? true : null });
const body = h(
"div",
{},
h("p", {}, t("models.deleteConfirm", { id: m.id })),
isLocal
? h("label", { class: "check" }, fileCb, t("models.deleteFile"))
: null
);
modal(t("common.delete") + ": " + m.id, body, async () => {
try {
const r = await api.deleteModel(m.id, isLocal && fileCb.checked);
toast(
r.file_removed ? t("models.deletedWithFile") : t("toast.deleted"),
"good"
);
await reload();
} catch (e) {
toast(String(e.message || e), "bad");
return false;
}
}, t("common.delete"));
},
},
t("common.delete")
+10 -4
View File
@@ -245,7 +245,11 @@ export async function renderSettings() {
try {
const r = await api.cmfPortCheck(p);
if (r.available) mount(st, h("span", { style: "color:var(--good)" }, "✓ " + t("settings.cmf.portFree")));
else {
else if (r.managed) {
// held by this gateway's own managed server — replacing the model on
// this port is fine, it frees up on restart
mount(st, h("span", { style: "color:var(--accent)" }, "ℹ " + t("settings.cmf.portOwn")));
} else {
const kids = [h("span", { style: "color:var(--bad)" }, "✗ " + t("settings.cmf.portBusy"))];
if (r.suggested) { const b = h("button", { class: "btn small", type: "button" }, t("settings.cmf.useSuggested", { port: r.suggested })); b.addEventListener("click", () => { portIn.value = r.suggested; mount(st, ""); }); kids.push(b); }
mount(st, h("span", {}, ...kids));
@@ -442,9 +446,11 @@ export async function renderSettings() {
"div",
{ class: "card" },
h("div", { class: "card-head" }, h("h3", {}, t("settings.route"))),
field(t("settings.route.strategy"), strategySel),
h("div", { class: "row" }, field(t("settings.route.maxChars"), maxChars), field(t("settings.route.cacheTtl"), cacheTtl)),
field(t("settings.route.profile"), profileSel)
field(t("settings.route.strategy"), strategySel, null, t("settings.route.strategyHelp")),
h("div", { class: "row" },
field(t("settings.route.maxChars"), maxChars, null, t("settings.route.maxCharsHelp")),
field(t("settings.route.cacheTtl"), cacheTtl, null, t("settings.route.cacheTtlHelp"))),
field(t("settings.route.profile"), profileSel, null, t("settings.route.profileHelp"))
),
h(
"div",