Compare commits

...
13 Commits
Author SHA1 Message Date
infosave2007 b867a6276e chore(release): v0.2.29 — fix duplicate managed model registration conflict
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
2026-07-21 23:39:47 +03:00
infosave2007 c18ddb5475 chore(release): v0.2.28 — auto-detect ready CMF quant & update section title
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
2026-07-21 23:35:14 +03:00
infosave2007 ef9b3738fd chore(release): v0.2.27 — auto-spawn local CMF server on model registration
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
2026-07-21 23:15:33 +03:00
infosave2007 924ec403bd chore(release): v0.2.26 — robust multi-GB streaming downloads & ready CMF quants
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
2026-07-21 23:08:02 +03:00
infosave2007 0c91f09113 chore(release): v0.2.25 — direct stream download for ready CMF repos
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
2026-07-21 22:51:58 +03:00
infosave2007 45deb6645b chore(release): v0.2.24 — new quantizations (q4t, q1t, q1s, q1p, q1) and featured CMF models
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
2026-07-21 22:45:43 +03:00
infosave2007 243d8f4c7d feat(import): O(1) attention options in Advanced (cortiq >= 0.2.0)
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
- New Advanced controls on the import page: O(1) attention spec (off / all /
  deep N / explicit layer list) plus the --o1-m / --o1-window / --o1-sink knobs
  with validated defaults (32 / 128 / 4) and explanatory hints.
- Backend passes --o1* to the native `cortiq convert`; fails fast with clear
  messages when combined with Python-only options, used on GGUF imports, or
  when the installed cortiq predates 0.2.0 (the hint needs the new runtime).
- The hint is recorded in the .cmf header (weights byte-identical); the runtime
  applies it at load. Verified end-to-end: converted a model with --o1 all and
  the runtime logged "o1 nystrom attention: 2/2 layer(s)".
2026-07-14 10:16:54 +03:00
infosave2007 4f72711a1e feat(dashboard): windowed stats per range + clear-logs button
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
- Stats snapshot now computes the KPI totals AND the breakdown from the
  in-window per-minute buckets (previously they used all-time accumulators, so
  switching 1h/24h/7d changed only the chart, not the numbers). Buckets now
  carry failovers + per-group breakdown; Prometheus keeps all-time counters.
- New DELETE /admin/api/stats clears all in-memory stats and truncates the
  JSONL log; the dashboard gets a "Clear logs" button (with confirm).
2026-07-09 17:07:05 +03:00
infosave2007 7855472499 docs(settings): add help tooltips to routing-input fields (strategy, max chars, cache TTL, profile)
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
2026-07-09 16:49:16 +03:00
infosave2007 a6ed974ece feat(models): auto-derive id from model name; actionable probe auth errors
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
- The model id is now auto-filled from the model name as you type, and if left
  blank it's derived on save (slug of the last path segment, e.g.
  "qwen/qwen3.7-max" → "qwen3-7-max") — saving no longer errors on a blank id.
- probe/test auth failures (401 / "User not found" / "No auth") now append a
  hint to check the API key (for OpenRouter: use an inference key, not a
  provisioning/management key).
2026-07-09 16:30:48 +03:00
infosave2007 4c11b97197 fix(models): Add button, probe of managed models, delete list refresh
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
- ui.js: mount() now wraps non-node children in text nodes (like h()), so
  mount(el, "") no longer throws in appendChild — this is what broke the
  "Add model" form (applyDefaults cleared a status via mount(x,"")).
- probe_model: resolve managed local models too (they are synthesized into the
  registry, not [[models]]) — probing them no longer returns "model not found".
- models.js: await reload() after delete so the row is removed immediately.
2026-07-09 16:13:07 +03:00
infosave2007 87f00ee502 feat(models): friendlier add-model form — OpenRouter/LM Studio, direct key, test & load
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
- Providers: add openrouter (https://openrouter.ai/api/v1) and lmstudio
  (http://localhost:1234/v1), both OpenAI-compatible. Selecting a provider now
  auto-fills its verified base URL, cost tier and default capabilities.
- API key is typed directly in the form and stored securely as a secret under the
  provider's env name — no manual env var to manage.
- New POST /admin/api/provider/models validates the key and lists the provider's
  models; a "Test & load models" button fills a searchable dropdown. The model
  field now sits below the key (some providers list models from the key).
- Fuller capability set: tools, vision, audio, reasoning, json, caching.
2026-07-09 15:45:21 +03:00
infosave2007 4fa1b8ea22 feat(settings): pick a local model from a dropdown of existing .cmf files
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
- New GET /admin/api/cmf/files lists the .cmf files (and tensor-dir .cmf) under
  models_dir, with sizes.
- Settings → Local models: the per-row model field is now a dropdown of those
  files instead of a hand-typed path, and auto-fills the id from the filename.
2026-07-09 15:25:30 +03:00
17 changed files with 964 additions and 119 deletions
+28
View File
@@ -3,8 +3,36 @@
All notable changes to this project are documented here. All notable changes to this project are documented here.
The format is based on [Keep a Changelog](https://keepachangelog.com/). The format is based on [Keep a Changelog](https://keepachangelog.com/).
## [0.2.29] - 2026-07-21
### Fixed
- **Duplicate Local Model Registration**: Fixed `duplicate managed model id` conflict in `register_import` by cleanly routing managed CMF models directly into `[[cmf.servers]]` with automatic free port allocation without pushing duplicate entries to `[[models]]`.
### Fixed
- **Quantization Badge & Title for Pre-Converted Models**: Automatically detect quantization (`Q1`, `Q1T`) for ready-to-run `.cmf` models (e.g. `infosave/Bonsai-27Bcmf`) from repo name and downloaded filename, resolving default `Q8_2F` badge fallback. Updated section title to "Загрузки / Конвертации" ("Downloads & Conversions").
### Fixed
- **Automatic CMF Local Server Lifecycle**: Registering an imported `.cmf` model automatically adds it to `[[cmf.servers]]` and reloads the runtime (`cmf_runtime::manage`), automatically spawning `cortiq serve` on port 8090 so chat completion requests route seamlessly without `upstream unavailable` errors.
### Fixed
- **Multi-GB Ready CMF Download Stream**: Replaced HTTP stream decoder with a streaming process downloader (`curl`) for pre-converted `.cmf` models to support multi-gigabyte Hugging Face LFS repositories without 30s/60s socket timeouts.
- **Ready CMF Quantization & UI Labels**: Set correct quantization badges (`Q1`, `Q1T`) for featured ready-to-run `.cmf` models and removed misleading "converting" log messages for direct downloads.
### Fixed
- **Direct Ready CMF Download**: Importing pre-converted `.cmf` model repositories (e.g. `infosave/Bonsai-8B_2bit_cmf`) directly streams the ready `.cmf` file from Hugging Face instead of attempting a safetensors conversion without `config.json`.
### Added
- **New Quantization Schemes**: Added support for `Q4_TILED` (q4t), `Q1T` (1.58-bit ternary), `Q1S` (1-bit sparse), `Q1P` (1-bit PTQ error diffusion), and `Q1` (pure binary) in converter backend and web admin UI.
- **Featured Ready-to-Run .cmf Models**: Added 1-tap download cards for pre-converted `.cmf` models (`infosave/Bonsai-1.7Bcmf`, `infosave/Bonsai-8B_2bit_cmf`, `infosave/Bonsai-27Bcmf`).
## [Unreleased] ## [Unreleased]
### Added ### Added
- **Docker Compose** quickstart (`docker-compose.yml` + `config/gateway.compose.toml`): - **Docker Compose** quickstart (`docker-compose.yml` + `config/gateway.compose.toml`):
gateway + a local Ollama model in one command. GHCR badge in the READMEs. gateway + a local Ollama model in one command. GHCR badge in the READMEs.
Generated
+1 -1
View File
@@ -285,7 +285,7 @@ checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570"
[[package]] [[package]]
name = "cortiq-gateway" name = "cortiq-gateway"
version = "0.2.8" version = "0.2.29"
dependencies = [ dependencies = [
"anyhow", "anyhow",
"arc-swap", "arc-swap",
+1 -1
View File
@@ -1,6 +1,6 @@
[package] [package]
name = "cortiq-gateway" name = "cortiq-gateway"
version = "0.2.8" version = "0.2.29"
edition = "2021" edition = "2021"
rust-version = "1.75" rust-version = "1.75"
license = "Apache-2.0" license = "Apache-2.0"
+188 -12
View File
@@ -165,7 +165,7 @@ pub fn api_routes(admin_token: String) -> Router<SharedState> {
"/admin/api/secrets", "/admin/api/secrets",
get(list_secrets).put(set_secret).delete(clear_secret), get(list_secrets).put(set_secret).delete(clear_secret),
) )
.route("/admin/api/stats", get(get_stats)) .route("/admin/api/stats", get(get_stats).delete(clear_stats))
.route("/admin/api/shadow", get(get_shadow)) .route("/admin/api/shadow", get(get_shadow))
.route("/admin/api/hf/search", get(hf_search)) .route("/admin/api/hf/search", get(hf_search))
.route("/admin/api/import", get(list_imports).post(start_import)) .route("/admin/api/import", get(list_imports).post(start_import))
@@ -178,6 +178,8 @@ pub fn api_routes(admin_token: String) -> Router<SharedState> {
.route("/admin/api/cmf", get(cmf_status)) .route("/admin/api/cmf", get(cmf_status))
.route("/admin/api/cmf/install", post(cmf_install)) .route("/admin/api/cmf/install", post(cmf_install))
.route("/admin/api/cmf/port", get(cmf_port_check)) .route("/admin/api/cmf/port", get(cmf_port_check))
.route("/admin/api/cmf/files", get(cmf_files))
.route("/admin/api/provider/models", post(provider_models))
.route("/admin/api/requests", get(get_requests)) .route("/admin/api/requests", get(get_requests))
.route("/admin/api/test", post(run_test)) .route("/admin/api/test", post(run_test))
.route("/admin/api/test/stream", post(run_test_stream)) .route("/admin/api/test/stream", post(run_test_stream))
@@ -390,18 +392,105 @@ async fn router_probe(State(state): State<SharedState>) -> ApiResult {
ok(json!({ "ok": true, "status": "ok", "latency_ms": latency_ms, "usage": usage })) ok(json!({ "ok": true, "status": "ok", "latency_ms": latency_ms, "usage": usage }))
} }
#[derive(Deserialize)]
struct ProviderModelsBody {
provider: String,
base_url: String,
/// Key typed into the form (takes priority)…
#[serde(default)]
api_key: Option<String>,
/// …or the name of a stored secret to reuse if no key was typed.
#[serde(default)]
api_key_env: Option<String>,
}
/// Concise error line from a provider's JSON error body.
fn provider_err(body: &str, status: u16) -> String {
if let Ok(v) = serde_json::from_str::<Value>(body) {
if let Some(m) = v["error"]["message"]
.as_str()
.or_else(|| v["error"].as_str())
.or_else(|| v["message"].as_str())
{
return format!("{status}: {m}");
}
}
format!("HTTP {status}")
}
/// Validate a provider API key and list its models by calling the provider's
/// `/models` endpoint — powers the "Test key & load models" button. The key is
/// used transiently (from the form) or resolved from a stored secret; it is not
/// persisted here.
async fn provider_models(
State(state): State<SharedState>,
Json(b): Json<ProviderModelsBody>,
) -> ApiResult {
let base = b.base_url.trim_end_matches('/').to_string();
if base.is_empty() {
return Err(ApiError::bad("base URL is required"));
}
let key = b.api_key.filter(|k| !k.trim().is_empty()).or_else(|| {
b.api_key_env
.as_deref()
.and_then(|e| state.secrets.resolve(e))
});
let key = key.as_deref();
let client = reqwest::Client::builder()
.timeout(std::time::Duration::from_secs(15))
.build()
.map_err(|e| ApiError::bad(e.to_string()))?;
// Anthropic uses x-api-key + anthropic-version and /v1/models; every other
// (OpenAI-compatible) provider uses a Bearer token and /models.
let resp = if b.provider == "anthropic" {
let mut r = client
.get(format!("{base}/v1/models"))
.header("anthropic-version", "2023-06-01");
if let Some(k) = key {
r = r.header("x-api-key", k);
}
r.send().await
} else {
let mut r = client.get(format!("{base}/models"));
if let Some(k) = key {
r = r.header("authorization", format!("Bearer {k}"));
}
r.send().await
};
let resp = resp.map_err(|e| ApiError::bad(format!("request failed: {e}")))?;
let status = resp.status().as_u16();
let body = resp.text().await.unwrap_or_default();
if !(200..300).contains(&status) {
return ok(json!({ "ok": false, "status": status, "error": provider_err(&body, status) }));
}
// OpenAI + Anthropic both return { "data": [ { "id": ... } ] }.
let v: Value = serde_json::from_str(&body).unwrap_or(json!({}));
let mut models: Vec<String> = v["data"]
.as_array()
.map(|arr| {
arr.iter()
.filter_map(|m| m["id"].as_str().map(|s| s.to_string()))
.collect()
})
.unwrap_or_default();
models.sort();
ok(json!({ "ok": true, "count": models.len(), "models": models }))
}
async fn meta() -> ApiResult { async fn meta() -> ApiResult {
ok(json!({ ok(json!({
"version": env!("CARGO_PKG_VERSION"), "version": env!("CARGO_PKG_VERSION"),
"providers": ["openai", "anthropic", "ollama", "http"], "providers": ["openai", "anthropic", "openrouter", "lmstudio", "ollama", "http"],
"provider_impl": { "openai": true, "ollama": true, "http": true, "anthropic": true }, "provider_impl": { "openai": true, "anthropic": true, "openrouter": true, "lmstudio": true, "ollama": true, "http": true },
"cost_tiers": ["cheap", "mid", "expensive"], "cost_tiers": ["cheap", "mid", "expensive", "local"],
"kinds": ["chat", "embedding"], "kinds": ["chat", "embedding"],
"profiles": ["cost-saver", "balanced", "quality-first"], "profiles": ["cost-saver", "balanced", "quality-first"],
"text_strategies": ["last_user", "last_user_plus_system", "concat_all"], "text_strategies": ["last_user", "last_user_plus_system", "concat_all"],
"policy_modes": ["fixed_table", "cost_aware"], "policy_modes": ["fixed_table", "cost_aware"],
"tiers": ["low", "medium", "high"], "tiers": ["low", "medium", "high"],
"caps": ["tools", "vision"], "caps": ["tools", "vision", "audio", "reasoning", "json", "caching"],
"protocols_impl": { "protocols_impl": {
"openai_chat": true, "openai_completions": true, "openai_embeddings": true, "openai_chat": true, "openai_completions": true, "openai_embeddings": true,
"openai_models": true, "anthropic_messages": true, "mcp": true, "native_passthrough": true "openai_models": true, "anthropic_messages": true, "mcp": true, "native_passthrough": true
@@ -605,15 +694,25 @@ async fn delete_model(State(state): State<SharedState>, Path(id): Path<String>)
async fn probe_model(State(state): State<SharedState>, Path(id): Path<String>) -> ApiResult { async fn probe_model(State(state): State<SharedState>, Path(id): Path<String>) -> ApiResult {
let live = state.live(); let live = state.live();
let mcfg = live // Provider string: from the static pool, or "openai" for a managed local
// model (which is synthesized into the registry, not [[models]]).
let provider_str = live
.cfg .cfg
.models .models
.iter() .iter()
.find(|m| m.id == id) .find(|m| m.id == id)
.ok_or_else(|| ApiError::not_found(format!("model '{id}' not found")))? .map(|m| m.provider.clone())
.clone(); .or_else(|| {
live.cfg
.cmf
.effective_servers()
.iter()
.find(|s| s.id == id)
.map(|_| "openai".to_string())
})
.ok_or_else(|| ApiError::not_found(format!("model '{id}' not found")))?;
if mcfg.provider == "anthropic" { if provider_str == "anthropic" {
return ok(json!({ return ok(json!({
"ok": false, "ok": false,
"error": "anthropic provider not yet implemented (planned v0.2)", "error": "anthropic provider not yet implemented (planned v0.2)",
@@ -650,7 +749,21 @@ async fn probe_model(State(state): State<SharedState>, Path(id): Path<String>) -
let latency_ms = started.elapsed().as_millis() as u64; let latency_ms = started.elapsed().as_millis() as u64;
match res { match res {
Ok(_) => ok(json!({ "ok": true, "latency_ms": latency_ms })), Ok(_) => ok(json!({ "ok": true, "latency_ms": latency_ms })),
Err(e) => ok(json!({ "ok": false, "latency_ms": latency_ms, "error": e.to_string() })), Err(e) => {
let msg = e.to_string();
// Make auth failures actionable — a raw "User not found" (OpenRouter's
// 401) usually means the API key is wrong or the wrong type.
let hint = if msg.contains("401")
|| msg.contains("Unauthorized")
|| msg.contains("User not found")
|| msg.contains("No auth")
{
" — check the API key (for OpenRouter use an inference key, not a provisioning/management key)"
} else {
""
};
ok(json!({ "ok": false, "latency_ms": latency_ms, "error": format!("{msg}{hint}") }))
}
} }
} }
@@ -843,6 +956,36 @@ async fn cmf_install(State(state): State<SharedState>) -> ApiResult {
ok(json!({ "started": true })) ok(json!({ "started": true }))
} }
/// List the `.cmf` model files available under `models_dir` so the UI can offer
/// them as a dropdown instead of a hand-typed path. Includes tensor-dir `.cmf`.
async fn cmf_files(State(state): State<SharedState>) -> ApiResult {
let dir = state.live().cfg.cmf.models_dir.clone();
let mut entries: Vec<(String, String, Option<u64>, bool)> = Vec::new();
if let Ok(rd) = std::fs::read_dir(&dir) {
for e in rd.flatten() {
let name = e.file_name().to_string_lossy().to_string();
if !name.ends_with(".cmf") {
continue;
}
let p = e.path();
let is_dir = p.is_dir();
let size = if is_dir {
None
} else {
std::fs::metadata(&p).ok().map(|m| m.len())
};
let path = format!("{}/{}", dir.trim_end_matches('/'), name);
entries.push((name, path, size, is_dir));
}
}
entries.sort_by(|a, b| a.0.cmp(&b.0));
let files: Vec<Value> = entries
.into_iter()
.map(|(name, path, size, is_dir)| json!({ "name": name, "path": path, "size": size, "is_dir": is_dir }))
.collect();
ok(json!({ "dir": dir, "files": files }))
}
#[derive(Deserialize)] #[derive(Deserialize)]
struct PortQuery { struct PortQuery {
port: u16, port: u16,
@@ -1054,6 +1197,13 @@ async fn get_stats(State(state): State<SharedState>, Query(q): Query<StatsQuery>
ok(snap) ok(snap)
} }
/// Clear all recorded stats/logs (in-memory aggregates, series, recent requests)
/// and truncate the JSONL log — the dashboard "Clear logs" button.
async fn clear_stats(State(state): State<SharedState>) -> ApiResult {
state.stats.clear();
ok(json!({ "ok": true }))
}
/// Self-warming shadow loop: per-task-type promotion state, Wilson-LB of the /// Self-warming shadow loop: per-task-type promotion state, Wilson-LB of the
/// judged pass-rate, and whether the local model is serving it yet. /// judged pass-rate, and whether the local model is serving it yet.
async fn get_shadow(State(state): State<SharedState>) -> ApiResult { async fn get_shadow(State(state): State<SharedState>) -> ApiResult {
@@ -1154,9 +1304,34 @@ async fn register_import(State(state): State<SharedState>, Path(job): Path<Strin
.unwrap_or_else(|| "cmf-local".into()); .unwrap_or_else(|| "cmf-local".into());
let id = format!("cmf-{base}"); let id = format!("cmf-{base}");
let mut cfg = current_cfg(&state); let mut cfg = current_cfg(&state);
if cfg.models.iter().any(|m| m.id == id) {
return Err(ApiError::bad(format!("model '{id}' already registered"))); let is_in_models = cfg.models.iter().any(|m| m.id == id);
let is_in_servers = cfg
.cmf
.servers
.iter()
.any(|s| s.id == id || s.model == j.output);
if is_in_models || is_in_servers {
return Err(ApiError::bad(format!("model '{id}' is already registered")));
} }
if cfg.cmf.manage_server {
let host = if cfg.cmf.local_host.trim().is_empty() {
"127.0.0.1"
} else {
&cfg.cmf.local_host
};
let base_port = cfg.cmf.servers.iter().map(|s| s.port).max().unwrap_or(8089);
let port = next_free_port(host, base_port).unwrap_or_else(|| base_port + 1);
cfg.cmf.servers.push(crate::config::CmfServer {
id: id.clone(),
model: j.output.clone(),
port,
threads: cfg.cmf.threads,
gpu: cfg.cmf.gpu,
});
} else {
cfg.models.push(crate::config::ModelCfg { cfg.models.push(crate::config::ModelCfg {
id: id.clone(), id: id.clone(),
provider: "openai".into(), provider: "openai".into(),
@@ -1169,6 +1344,7 @@ async fn register_import(State(state): State<SharedState>, Path(job): Path<Strin
api_key_env: None, api_key_env: None,
caps: Vec::new(), caps: Vec::new(),
}); });
}
state.reload(cfg)?; state.reload(cfg)?;
ok(json!({ "ok": true, "model_id": id })) ok(json!({ "ok": true, "model_id": id }))
} }
+1 -1
View File
@@ -238,7 +238,7 @@ fn parse_ver(s: &str) -> (u32, u32, u32) {
) )
} }
fn version_lt(a: &str, b: &str) -> bool { pub(crate) fn version_lt(a: &str, b: &str) -> bool {
parse_ver(a) < parse_ver(b) parse_ver(a) < parse_ver(b)
} }
+283 -3
View File
@@ -71,6 +71,20 @@ pub struct ImportParams {
pub shard_max_gb: Option<f32>, pub shard_max_gb: Option<f32>,
#[serde(default)] #[serde(default)]
pub skip_mtp: bool, pub skip_mtp: bool,
/// O(1) Nyström attention hint (native converter, cortiq ≥ 0.2.0):
/// `all` | `deepN` | `i,j,k`. Weights pass through unchanged — the runtime
/// reads the hint at load. Empty/None/"off" = exact attention.
#[serde(default)]
pub o1: Option<String>,
/// Landmark budget for the o1 hint (validated default 32).
#[serde(default)]
pub o1_m: Option<usize>,
/// Exact-window width for the o1 hint (validated default 128).
#[serde(default)]
pub o1_window: Option<usize>,
/// Permanent exact sink keys for the o1 hint (validated default 4).
#[serde(default)]
pub o1_sink: Option<usize>,
} }
fn default_quant() -> String { fn default_quant() -> String {
"Q8_2F".into() "Q8_2F".into()
@@ -146,6 +160,14 @@ impl JobStore {
} }
} }
} }
fn update_quant(&self, id: &str, quant: String) {
if let Some(j) = self.jobs.lock().unwrap().get_mut(id) {
j.quant = quant.clone();
if !j.log.is_empty() && j.log[0].starts_with("→ downloading ready .cmf model") {
j.log[0] = format!("→ downloading ready .cmf model {} ({})", j.repo, quant);
}
}
}
fn set_state(&self, id: &str, state: &str) { fn set_state(&self, id: &str, state: &str) {
if let Some(j) = self.jobs.lock().unwrap().get_mut(id) { if let Some(j) = self.jobs.lock().unwrap().get_mut(id) {
j.state = state.into(); j.state = state.into();
@@ -246,14 +268,166 @@ fn sanitize(s: &str) -> String {
/// Map the UI quant label to the native `cortiq` quant flag. /// Map the UI quant label to the native `cortiq` quant flag.
fn map_quant(q: &str) -> &'static str { fn map_quant(q: &str) -> &'static str {
match q.to_ascii_uppercase().as_str() { match q.to_ascii_uppercase().as_str() {
s if s.starts_with("Q4T") || s == "Q4_TILED" => "q4t",
s if s.starts_with("Q4") => "q4", s if s.starts_with("Q4") => "q4",
"Q8_2F" => "q8_2f", "Q8_2F" | "Q82F" => "q8_2f",
"F16" | "FP16" => "f16", "F16" | "FP16" => "f16",
"VBIT" => "vbit", "VBIT" => "vbit",
"Q1" => "q1",
"Q1P" | "Q1_PTQ" => "q1p",
"Q1S" | "Q1_MASK" => "q1s",
"Q1T" | "Q1_TERNARY" => "q1t",
_ => "q8", _ => "q8",
} }
} }
fn is_cmf_repo(repo: &str) -> bool {
let lower = repo.to_ascii_lowercase();
lower.contains("cmf") || lower.ends_with(".cmf")
}
async fn download_cmf_repo(
repo: String,
output_abs: String,
hf_token: Option<String>,
store: Arc<JobStore>,
id: String,
cancel: Arc<Notify>,
) {
store.set_progress(&id, 0.0, "listing files".into());
let tree_url = format!("https://huggingface.co/api/models/{repo}/tree/main?recursive=true");
let client = reqwest::Client::new();
let mut req = client.get(&tree_url).header("User-Agent", "cortiq-gateway");
if let Some(t) = &hf_token {
req = req.bearer_auth(t);
}
let mut cmf_rel_path = None;
if let Ok(resp) = req.send().await {
if resp.status().is_success() {
if let Ok(json) = resp.json::<serde_json::Value>().await {
if let Some(arr) = json.as_array() {
for item in arr {
if let Some(path) = item.get("path").and_then(|p| p.as_str()) {
if path.to_ascii_lowercase().ends_with(".cmf") {
cmf_rel_path = Some(path.to_string());
break;
}
}
}
}
}
}
}
let cmf_path = match cmf_rel_path {
Some(p) => p,
None => {
let repo_name = repo.split('/').last().unwrap_or(&repo);
format!("{repo_name}.cmf")
}
};
let path_lower = cmf_path.to_ascii_lowercase();
let detected_quant =
if path_lower.contains("-q1t.") || repo.to_ascii_lowercase().contains("2bit") {
"Q1T".to_string()
} else if path_lower.contains("-q1.") || repo.to_ascii_lowercase().contains("cmf") {
"Q1".to_string()
} else if path_lower.contains("-q4.") {
"Q4".to_string()
} else if path_lower.contains("-q8.") {
"Q8".to_string()
} else {
"READY_CMF".to_string()
};
store.update_quant(&id, detected_quant);
store.push_line(&id, format!("→ downloading {cmf_path} from HuggingFace..."));
store.set_progress(&id, 0.05, "downloading".into());
let download_url = format!("https://huggingface.co/{repo}/resolve/main/{cmf_path}");
let mut cmd = tokio::process::Command::new("curl");
cmd.arg("-f").arg("-L").arg("-#").arg("-o").arg(&output_abs);
if let Some(t) = &hf_token {
cmd.arg("-H").arg(format!("Authorization: Bearer {t}"));
}
cmd.arg(&download_url);
cmd.stderr(std::process::Stdio::piped());
cmd.stdout(std::process::Stdio::null());
let mut child = match cmd.spawn() {
Ok(c) => c,
Err(e) => {
store.push_line(&id, format!("✗ failed to start curl download: {e}"));
store.finish(&id, false);
store.drop_cancel(&id);
return;
}
};
let stderr = child.stderr.take();
let mut t_err = None;
if let Some(err_stream) = stderr {
let store_clone = store.clone();
let id_clone = id.clone();
t_err = Some(tokio::spawn(async move {
use tokio::io::AsyncReadExt;
let mut reader = err_stream;
let mut buf = [0u8; 256];
while let Ok(n) = reader.read(&mut buf).await {
if n == 0 {
break;
}
let text = String::from_utf8_lossy(&buf[..n]);
for word in text.split_whitespace() {
if let Some(num_str) = word.strip_suffix('%') {
if let Ok(pct) = num_str.parse::<f32>() {
let frac = (pct / 100.0).clamp(0.0, 1.0);
store_clone.set_progress(&id_clone, frac, "downloading".into());
}
}
}
}
}));
}
tokio::select! {
status_res = child.wait() => {
if let Some(t) = t_err { t.abort(); }
match status_res {
Ok(status) if status.success() => {
store.push_line(&id, "✓ done".into());
store.finish(&id, true);
}
Ok(status) => {
cleanup_output(&output_abs);
store.push_line(&id, format!("✗ download failed: exit code {:?}", status.code()));
store.finish(&id, false);
}
Err(e) => {
cleanup_output(&output_abs);
store.push_line(&id, format!("✗ download process error: {e}"));
store.finish(&id, false);
}
}
}
_ = cancel.notified() => {
let _ = child.start_kill();
let _ = child.wait().await;
if let Some(t) = t_err { t.abort(); }
cleanup_output(&output_abs);
store.push_line(&id, "✗ cancelled by user — partial output removed".into());
store.set_state(&id, "cancelled");
}
}
store.drop_cancel(&id);
}
/// into the job log. Fails fast if the converter script is missing. /// into the job log. Fails fast if the converter script is missing.
pub fn start_import(store: Arc<JobStore>, cfg: &CmfCfg, p: ImportParams) -> Result<String, String> { pub fn start_import(store: Arc<JobStore>, cfg: &CmfCfg, p: ImportParams) -> Result<String, String> {
if p.repo.trim().is_empty() { if p.repo.trim().is_empty() {
@@ -269,6 +443,34 @@ pub fn start_import(store: Arc<JobStore>, cfg: &CmfCfg, p: ImportParams) -> Resu
cfg.converter cfg.converter
)); ));
} }
// O(1) attention hint — native-converter feature (cortiq ≥ 0.2.0). Fail fast
// with a clear message instead of a mid-job "unexpected argument" error.
let o1 =
p.o1.as_deref()
.map(str::trim)
.filter(|s| !s.is_empty() && *s != "off")
.map(str::to_string);
if o1.is_some() {
if use_python {
return Err(
"O(1) attention is supported by the native converter only — remove the \
linear-core / v-bit shape / mean-bits options to use it"
.into(),
);
}
if p.repo.to_ascii_lowercase().contains("gguf") {
return Err("O(1) attention is not supported for GGUF imports yet".into());
}
let ver = crate::cmf_runtime::installed_version(&cfg.cortiq_bin);
if let Some(v) = &ver {
if crate::cmf_runtime::version_lt(v, "0.2.0") {
return Err(format!(
"O(1) attention needs cortiq ≥ 0.2.0 (installed: {v}) — update the runtime \
in Settings → Local models"
));
}
}
}
let _ = std::fs::create_dir_all(&cfg.models_dir); let _ = std::fs::create_dir_all(&cfg.models_dir);
let base = if p.name.trim().is_empty() { let base = if p.name.trim().is_empty() {
sanitize(p.repo.rsplit('/').next().unwrap_or(&p.repo)) sanitize(p.repo.rsplit('/').next().unwrap_or(&p.repo))
@@ -284,13 +486,37 @@ pub fn start_import(store: Arc<JobStore>, cfg: &CmfCfg, p: ImportParams) -> Resu
.unwrap_or_else(|_| output.clone()); .unwrap_or_else(|_| output.clone());
let id = gen_id(&p.repo); let id = gen_id(&p.repo);
let is_cmf = is_cmf_repo(&p.repo);
let quant_display = if is_cmf {
let r = p.repo.to_lowercase();
if r.contains("2bit") || r.contains("q1t") {
"Q1T".to_string()
} else if r.contains("1.7bcmf") || r.contains("27bcmf") || r.contains("cmf") {
"Q1".to_string()
} else if p.quant.is_empty() || p.quant == "auto" || p.quant == "Q8_2F" {
"Q1".to_string()
} else {
p.quant.clone()
}
} else if p.quant.is_empty() || p.quant == "auto" {
"q8".to_string()
} else {
p.quant.clone()
};
store.insert(Job { store.insert(Job {
id: id.clone(), id: id.clone(),
repo: p.repo.clone(), repo: p.repo.clone(),
quant: p.quant.clone(), quant: quant_display.clone(),
output: output_abs.clone(), output: output_abs.clone(),
state: "running".into(), state: "running".into(),
log: vec![format!("→ converting {} to {} ({})", p.repo, base, p.quant)], log: if is_cmf {
vec![format!(
"→ downloading ready .cmf model {} ({})",
p.repo, quant_display
)]
} else {
vec![format!("→ converting {} to {} ({})", p.repo, base, p.quant)]
},
started: now(), started: now(),
finished: None, finished: None,
size_bytes: None, size_bytes: None,
@@ -306,6 +532,14 @@ pub fn start_import(store: Arc<JobStore>, cfg: &CmfCfg, p: ImportParams) -> Resu
std::env::var(&cfg.hf_token_env).ok() std::env::var(&cfg.hf_token_env).ok()
}; };
if is_cmf && !use_python {
let ret_id = id.clone();
tokio::spawn(async move {
download_cmf_repo(p.repo, output_abs, hf_token, store, id, cancel).await;
});
return Ok(ret_id);
}
// Program + args: `cortiq convert` by default, or the Python converter for // Program + args: `cortiq convert` by default, or the Python converter for
// advanced options. Both stream `@PROGRESS <frac>` markers into the log. // advanced options. Both stream `@PROGRESS <frac>` markers into the log.
let (program, args, workdir): (String, Vec<String>, std::path::PathBuf) = if use_python { let (program, args, workdir): (String, Vec<String>, std::path::PathBuf) = if use_python {
@@ -384,6 +618,23 @@ pub fn start_import(store: Arc<JobStore>, cfg: &CmfCfg, p: ImportParams) -> Resu
a.push("--hf-token".into()); a.push("--hf-token".into());
a.push(t.clone()); a.push(t.clone());
} }
// O(1) attention hint (weights unchanged; the runtime reads it at load)
if let Some(spec) = &o1 {
a.push("--o1".into());
a.push(spec.clone());
if let Some(m) = p.o1_m {
a.push("--o1-m".into());
a.push(m.to_string());
}
if let Some(w) = p.o1_window {
a.push("--o1-window".into());
a.push(w.to_string());
}
if let Some(s) = p.o1_sink {
a.push("--o1-sink".into());
a.push(s.to_string());
}
}
( (
cfg.cortiq_bin.clone(), cfg.cortiq_bin.clone(),
a, a,
@@ -471,3 +722,32 @@ pub fn start_import(store: Arc<JobStore>, cfg: &CmfCfg, p: ImportParams) -> Resu
Ok(ret_id) Ok(ret_id)
} }
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_map_quant_variants() {
assert_eq!(map_quant("Q8_2F"), "q8_2f");
assert_eq!(map_quant("q82f"), "q8_2f");
assert_eq!(map_quant("Q8_ROW"), "q8");
assert_eq!(map_quant("Q4_BLOCK"), "q4");
assert_eq!(map_quant("Q4_TILED"), "q4t");
assert_eq!(map_quant("q4t"), "q4t");
assert_eq!(map_quant("vbit"), "vbit");
assert_eq!(map_quant("q1"), "q1");
assert_eq!(map_quant("Q1P"), "q1p");
assert_eq!(map_quant("q1s"), "q1s");
assert_eq!(map_quant("Q1T"), "q1t");
assert_eq!(map_quant("F16"), "f16");
}
#[test]
fn test_is_cmf_repo_check() {
assert!(is_cmf_repo("infosave/Bonsai-8B_2bit_cmf"));
assert!(is_cmf_repo("infosave/Bonsai-1.7Bcmf"));
assert!(is_cmf_repo("user/model.cmf"));
assert!(!is_cmf_repo("Qwen/Qwen2.5-0.5B-Instruct"));
}
}
+4 -1
View File
@@ -21,7 +21,10 @@ impl Registry {
.as_ref() .as_ref()
.and_then(|name| secrets.resolve(name)); .and_then(|name| secrets.resolve(name));
let provider: Arc<dyn Provider> = match m.provider.as_str() { let provider: Arc<dyn Provider> = match m.provider.as_str() {
"openai" | "ollama" | "http" => Arc::new(OpenAiProvider::new(m, api_key)?), // openrouter + lmstudio are OpenAI-compatible (Bearer + /chat/completions)
"openai" | "ollama" | "http" | "openrouter" | "lmstudio" => {
Arc::new(OpenAiProvider::new(m, api_key)?)
}
"anthropic" => Arc::new(AnthropicProvider::new(m, api_key)?), "anthropic" => Arc::new(AnthropicProvider::new(m, api_key)?),
other => anyhow::bail!("unknown provider '{}' for model '{}'", other, m.id), other => anyhow::bail!("unknown provider '{}' for model '{}'", other, m.id),
}; };
+8
View File
@@ -145,6 +145,14 @@ impl AppState {
let live = Live::build(new_cfg.clone(), &self.secrets, self.router_status.clone())?; let live = Live::build(new_cfg.clone(), &self.secrets, self.router_status.clone())?;
new_cfg.save(&self.config_path)?; new_cfg.save(&self.config_path)?;
self.live.store(Arc::new(live)); self.live.store(Arc::new(live));
let cmf_rt = self.cmf.clone();
let cmf_cfg = new_cfg.cmf.clone();
tokio::spawn(async move {
cmf_rt.stop().await;
crate::cmf_runtime::manage(cmf_rt, cmf_cfg).await;
});
Ok(()) Ok(())
} }
+92 -40
View File
@@ -83,9 +83,23 @@ impl Totals {
self.cost_usd += r.cost_usd; self.cost_usd += r.cost_usd;
self.latency_ms_sum += r.latency_ms; self.latency_ms_sum += r.latency_ms;
} }
/// Accumulate another Totals into this one (for windowed aggregation).
fn merge(&mut self, o: &Totals) {
self.requests += o.requests;
self.ok += o.ok;
self.errors += o.errors;
self.failovers += o.failovers;
self.prompt_tokens += o.prompt_tokens;
self.completion_tokens += o.completion_tokens;
self.cost_usd += o.cost_usd;
self.latency_ms_sum += o.latency_ms_sum;
}
} }
/// Per-minute bucket for time-series charts. /// Per-minute bucket. Beyond the chart fields it carries the per-group breakdown
/// and failovers so the admin snapshot can be computed for ANY time window by
/// summing the in-window buckets (the internal fields are not serialized into
/// the `series` payload).
#[derive(Clone, Debug, Default, Serialize)] #[derive(Clone, Debug, Default, Serialize)]
pub struct Bucket { pub struct Bucket {
pub minute: u64, // unix seconds, rounded down to the minute pub minute: u64, // unix seconds, rounded down to the minute
@@ -95,15 +109,25 @@ pub struct Bucket {
pub completion_tokens: u64, pub completion_tokens: u64,
pub cost_usd: f64, pub cost_usd: f64,
pub latency_ms_sum: u64, pub latency_ms_sum: u64,
#[serde(skip)]
pub failovers: u64,
#[serde(skip)]
pub by_model: HashMap<String, Totals>,
#[serde(skip)]
pub by_account: HashMap<String, Totals>,
#[serde(skip)]
pub by_tier: HashMap<String, Totals>,
#[serde(skip)]
pub by_label: HashMap<String, Totals>,
} }
#[derive(Default)] #[derive(Default)]
struct Inner { struct Inner {
/// All-time totals + per-model (used by the Prometheus counters, which must
/// be monotonic — the windowed admin snapshot is derived from the buckets).
total: Totals, total: Totals,
by_model: HashMap<String, Totals>, by_model: HashMap<String, Totals>,
by_account: HashMap<String, Totals>, /// Per-minute buckets — the single source for the windowed admin snapshot.
by_tier: HashMap<String, Totals>,
by_label: HashMap<String, Totals>,
buckets: VecDeque<Bucket>, buckets: VecDeque<Bucket>,
recent: VecDeque<RequestRecord>, recent: VecDeque<RequestRecord>,
} }
@@ -166,31 +190,14 @@ impl Stats {
} }
fn apply_aggregates(inner: &mut Inner, rec: &RequestRecord, retention_secs: u64) { fn apply_aggregates(inner: &mut Inner, rec: &RequestRecord, retention_secs: u64) {
// all-time counters for Prometheus; the windowed snapshot is derived
// from the buckets below.
inner.total.apply(rec); inner.total.apply(rec);
inner inner
.by_model .by_model
.entry(rec.model_id.clone()) .entry(rec.model_id.clone())
.or_default() .or_default()
.apply(rec); .apply(rec);
inner
.by_account
.entry(if rec.account.is_empty() {
"anonymous".to_string()
} else {
rec.account.clone()
})
.or_default()
.apply(rec);
inner
.by_tier
.entry(rec.tier.clone())
.or_default()
.apply(rec);
inner
.by_label
.entry(rec.task_label.clone())
.or_default()
.apply(rec);
let minute = rec.ts - (rec.ts % 60); let minute = rec.ts - (rec.ts % 60);
match inner.buckets.back_mut() { match inner.buckets.back_mut() {
@@ -250,13 +257,31 @@ impl Stats {
.filter(|b| b.minute >= cutoff) .filter(|b| b.minute >= cutoff)
.collect(); .collect();
let breakdown_src = match groupby { // Windowed totals + breakdown = sum of the in-window buckets, so the KPIs
"account" => &inner.by_account, // and the breakdown recompute whenever the range changes.
"tier" => &inner.by_tier, let mut win = Totals::default();
"label" => &inner.by_label, let mut group: HashMap<String, Totals> = HashMap::new();
_ => &inner.by_model, for b in &series {
win.requests += b.requests;
win.errors += b.errors;
win.ok += b.requests.saturating_sub(b.errors);
win.failovers += b.failovers;
win.prompt_tokens += b.prompt_tokens;
win.completion_tokens += b.completion_tokens;
win.cost_usd += b.cost_usd;
win.latency_ms_sum += b.latency_ms_sum;
let src = match groupby {
"account" => &b.by_account,
"tier" => &b.by_tier,
"label" => &b.by_label,
_ => &b.by_model,
}; };
let mut breakdown: Vec<serde_json::Value> = breakdown_src for (k, t) in src {
group.entry(k.clone()).or_default().merge(t);
}
}
let mut breakdown: Vec<serde_json::Value> = group
.iter() .iter()
.map(|(k, t)| { .map(|(k, t)| {
serde_json::json!({ serde_json::json!({
@@ -280,17 +305,17 @@ impl Stats {
serde_json::json!({ serde_json::json!({
"totals": { "totals": {
"requests": inner.total.requests, "requests": win.requests,
"ok": inner.total.ok, "ok": win.ok,
"errors": inner.total.errors, "errors": win.errors,
"failovers": inner.total.failovers, "failovers": win.failovers,
"prompt_tokens": inner.total.prompt_tokens, "prompt_tokens": win.prompt_tokens,
"completion_tokens": inner.total.completion_tokens, "completion_tokens": win.completion_tokens,
"total_tokens": inner.total.prompt_tokens + inner.total.completion_tokens, "total_tokens": win.prompt_tokens + win.completion_tokens,
"cost_usd": inner.total.cost_usd, "cost_usd": win.cost_usd,
"avg_latency_ms": avg(inner.total.latency_ms_sum, inner.total.requests), "avg_latency_ms": avg(win.latency_ms_sum, win.requests),
"success_rate": if inner.total.requests > 0 { "success_rate": if win.requests > 0 {
inner.total.ok as f64 / inner.total.requests as f64 win.ok as f64 / win.requests as f64
} else { 0.0 }, } else { 0.0 },
}, },
"groupby": groupby, "groupby": groupby,
@@ -299,6 +324,18 @@ impl Stats {
}) })
} }
/// Reset all in-memory stats and truncate the JSONL log — the "clear logs"
/// admin action.
pub fn clear(&self) {
{
let mut inner = self.inner.lock().unwrap();
*inner = Inner::default();
}
if let Some(path) = &self.file {
let _ = std::fs::write(path, "");
}
}
/// Recent requests (newest first), with pagination. /// Recent requests (newest first), with pagination.
pub fn recent(&self, limit: usize, offset: usize) -> Vec<RequestRecord> { pub fn recent(&self, limit: usize, offset: usize) -> Vec<RequestRecord> {
let inner = self.inner.lock().unwrap(); let inner = self.inner.lock().unwrap();
@@ -358,10 +395,25 @@ fn fill_bucket(b: &mut Bucket, r: &RequestRecord) {
if r.outcome != "ok" { if r.outcome != "ok" {
b.errors += 1; b.errors += 1;
} }
if r.failover {
b.failovers += 1;
}
b.prompt_tokens += r.prompt_tokens as u64; b.prompt_tokens += r.prompt_tokens as u64;
b.completion_tokens += r.completion_tokens as u64; b.completion_tokens += r.completion_tokens as u64;
b.cost_usd += r.cost_usd; b.cost_usd += r.cost_usd;
b.latency_ms_sum += r.latency_ms; b.latency_ms_sum += r.latency_ms;
// per-group breakdown, so the snapshot can be windowed by summing buckets
b.by_model.entry(r.model_id.clone()).or_default().apply(r);
b.by_account
.entry(if r.account.is_empty() {
"anonymous".to_string()
} else {
r.account.clone()
})
.or_default()
.apply(r);
b.by_tier.entry(r.tier.clone()).or_default().apply(r);
b.by_label.entry(r.task_label.clone()).or_default().apply(r);
} }
fn avg(sum: u64, n: u64) -> u64 { fn avg(sum: u64, n: u64) -> u64 {
+3
View File
@@ -87,6 +87,7 @@ export const api = {
// stats // stats
stats: (range, groupby) => j("GET", `/stats?range=${range}&groupby=${groupby}`), stats: (range, groupby) => j("GET", `/stats?range=${range}&groupby=${groupby}`),
clearStats: () => j("DELETE", "/stats"),
requests: (limit = 50, offset = 0) => j("GET", `/requests?limit=${limit}&offset=${offset}`), requests: (limit = 50, offset = 0) => j("GET", `/requests?limit=${limit}&offset=${offset}`),
// playground // playground
@@ -99,6 +100,8 @@ export const api = {
cmfStatus: () => j("GET", "/cmf"), cmfStatus: () => j("GET", "/cmf"),
cmfInstall: () => j("POST", "/cmf/install"), cmfInstall: () => j("POST", "/cmf/install"),
cmfPortCheck: (port) => j("GET", "/cmf/port?port=" + encodeURIComponent(port)), cmfPortCheck: (port) => j("GET", "/cmf/port?port=" + encodeURIComponent(port)),
cmfFiles: () => j("GET", "/cmf/files"),
providerModels: (b) => j("POST", "/provider/models", b),
// CMF model factory (HuggingFace → local .cmf) // CMF model factory (HuggingFace → local .cmf)
hfSearch: (q, limit = 24) => hfSearch: (q, limit = 24) =>
+42 -2
View File
@@ -59,6 +59,9 @@ export default {
// dashboard // dashboard
"dash.title": "Dashboard", "dash.title": "Dashboard",
"dash.subtitle": "Live traffic, cost and routing analytics for your model pool.", "dash.subtitle": "Live traffic, cost and routing analytics for your model pool.",
"dash.clear": "Clear logs",
"dash.clearConfirm": "Clear all recorded stats and logs? This cannot be undone.",
"dash.cleared": "Logs cleared",
"dash.range": "Range", "dash.range": "Range",
"dash.range.1h": "1h", "dash.range.1h": "1h",
"dash.range.24h": "24h", "dash.range.24h": "24h",
@@ -129,6 +132,18 @@ export default {
"models.form.secretMissing": "not set", "models.form.secretMissing": "not set",
"models.form.secretPlaceholder": "set or replace key (write-only)", "models.form.secretPlaceholder": "set or replace key (write-only)",
"models.deleteConfirm": "Delete model \"{id}\"?", "models.deleteConfirm": "Delete model \"{id}\"?",
"models.form.idOrModel": "Enter an id or a model name",
"models.form.apiKey": "API Key",
"models.form.apiKeyPlaceholder": "paste your API key",
"models.form.apiKeyNotNeeded": "not required (local server)",
"models.form.apiKeyHint": "Stored securely on the gateway; it is not shown again.",
"models.form.keyStored": "A key is saved — leave blank to keep it, or paste a new one.",
"models.form.keyFromEnv": "Using a key from an environment variable.",
"models.form.testKey": "Test & load models",
"models.form.testing": "checking…",
"models.form.foundModels": "{n} models loaded",
"models.form.modelPlaceholder": "type to search…",
"models.form.modelHint": "Click “Test & load models” to pull the list from your key, then pick one.",
"models.managed": "managed", "models.managed": "managed",
"models.managedHint": "Provided by the managed CMF server (Settings → Local models). Configure it there.", "models.managedHint": "Provided by the managed CMF server (Settings → Local models). Configure it there.",
@@ -244,9 +259,10 @@ export default {
"settings.cmf.threads": "Worker threads", "settings.cmf.threads": "Worker threads",
"settings.cmf.gpu": "Use GPU (Metal / wgpu)", "settings.cmf.gpu": "Use GPU (Metal / wgpu)",
"settings.cmf.speedNote": "Threads parallelise the decode matvecs (~3x faster; 0 = runtime default). GPU offload mainly helps prefill of long prompts — decode is memory-bandwidth-bound, so on Apple Silicon it is roughly neutral.", "settings.cmf.speedNote": "Threads parallelise the decode matvecs (~3x faster; 0 = runtime default). GPU offload mainly helps prefill of long prompts — decode is memory-bandwidth-bound, so on Apple Silicon it is roughly neutral.",
"settings.cmf.pick": "select a model",
"settings.cmf.models": "Local models", "settings.cmf.models": "Local models",
"settings.cmf.modelsHelp": "Each row is one model the gateway runs (its own cortiq serve on its own port) and registers in the pool. Add as many as you have .cmf files and free ports.", "settings.cmf.modelsHelp": "Each row is one model the gateway runs (its own cortiq serve on its own port) and registers in the pool. Add as many as you have .cmf files and free ports.",
"settings.cmf.rowHint": "id · path to .cmf · port · threads · GPU. Changes apply on restart; deleting a row stops that server.", "settings.cmf.rowHint": "id · .cmf file · port · threads · GPU. Changes apply on restart; deleting a row stops that server.",
"settings.cmf.addServer": "+ Add local model", "settings.cmf.addServer": "+ Add local model",
"settings.cmf.checkPort": "Check port", "settings.cmf.checkPort": "Check port",
"settings.cmf.portFree": "port is free", "settings.cmf.portFree": "port is free",
@@ -303,6 +319,10 @@ export default {
"settings.route.maxChars": "Max chars", "settings.route.maxChars": "Max chars",
"settings.route.cacheTtl": "Cache TTL", "settings.route.cacheTtl": "Cache TTL",
"settings.route.profile": "Default profile", "settings.route.profile": "Default profile",
"settings.route.strategyHelp": "What text is sent to the router for classification: last_user = only the last user message; last_user_plus_system = last user message + system prompt; concat_all = the whole conversation.",
"settings.route.maxCharsHelp": "Max characters of that text sent to the router (longer is truncated). More = more accurate classification, but slower/pricier router calls.",
"settings.route.cacheTtlHelp": "How long a routing decision is cached for an identical request before the router is asked again (e.g. 60s, 5m).",
"settings.route.profileHelp": "Default routing profile when a request doesn't specify one: cost-saver (cheapest), balanced, or quality-first (best model). Biases which model the router picks.",
"settings.breaker": "Circuit breaker", "settings.breaker": "Circuit breaker",
"settings.breaker.threshold": "Error threshold", "settings.breaker.threshold": "Error threshold",
"settings.breaker.cooldown": "Cooldown", "settings.breaker.cooldown": "Cooldown",
@@ -354,7 +374,7 @@ export default {
"import.failed": "Conversion failed — see the log.", "import.failed": "Conversion failed — see the log.",
"import.register": "Register as local model", "import.register": "Register as local model",
"import.registered": "Registered as '{id}'. Run cortiq-server to serve it.", "import.registered": "Registered as '{id}'. Run cortiq-server to serve it.",
"import.jobs": "Conversions", "import.jobs": "Downloads & Conversions",
"import.cancel": "Cancel", "import.cancel": "Cancel",
"import.cancelling": "Cancelling — cleaning up partial output…", "import.cancelling": "Cancelling — cleaning up partial output…",
"import.delete": "Delete", "import.delete": "Delete",
@@ -364,12 +384,21 @@ export default {
"import.state.done": "done", "import.state.done": "done",
"import.state.error": "error", "import.state.error": "error",
"import.state.cancelled": "cancelled", "import.state.cancelled": "cancelled",
"import.featuredTitle": "Featured Ready-to-Run .cmf Models",
"import.readyCmfBadge": "Ready .cmf",
"import.directDownload": "Download",
"import.q.q8_2f": "8-bit two-field (𝒲×θ) — recommended: best quality/size.", "import.q.q8_2f": "8-bit two-field (𝒲×θ) — recommended: best quality/size.",
"import.q.q8_row": "8-bit per-row — simple, robust.", "import.q.q8_row": "8-bit per-row — simple, robust.",
"import.q.q4": "4-bit block — smallest, lower quality.", "import.q.q4": "4-bit block — smallest, lower quality.",
"import.q.q4_tiled": "4-bit tiled (Q4_TILED) — 1.66x faster on Metal GPU.",
"import.q.vbit": "Variable 3–8 bit — per-expert bit budgets.", "import.q.vbit": "Variable 3–8 bit — per-expert bit budgets.",
"import.q.q1t": "1.58-bit ternary (Q1T) — BitNet b1.58 with sparse outliers.",
"import.q.q1s": "1-bit sparse (Q1S) — 1-bit base with holographic outliers.",
"import.q.q1p": "1-bit PTQ (Q1P) — 1-bit with error diffusion.",
"import.q.q1": "1-bit (Q1) — pure binary format.",
"import.q.f16": "16-bit — no quantization.", "import.q.f16": "16-bit — no quantization.",
"import.q.f32": "32-bit — full precision (large file).", "import.q.f32": "32-bit — full precision (large file).",
"import.adv.title": "Advanced settings", "import.adv.title": "Advanced settings",
"import.adv.auto": "auto", "import.adv.auto": "auto",
"import.adv.linearCore": "Linear core", "import.adv.linearCore": "Linear core",
@@ -380,4 +409,15 @@ export default {
"import.adv.shard": "Shard max (GB)", "import.adv.shard": "Shard max (GB)",
"import.adv.noShard": "no sharding", "import.adv.noShard": "no sharding",
"import.adv.skipMtp": "skip MTP head", "import.adv.skipMtp": "skip MTP head",
"import.adv.o1": "O(1) attention",
"import.adv.o1Hint": "Constant-memory streaming attention: the KV cache is released after prefill, so long contexts don't grow memory. Weights stay byte-identical — only a load-time hint is recorded; you can switch back anytime (serve with --o1 off). Validated on Qwen3-0.6B: all layers ≈ ×1.18 perplexity.",
"import.adv.o1Off": "off — exact attention (default)",
"import.adv.o1All": "all — convert every softmax layer",
"import.adv.o1Deep": "deep N — only the N deepest layers",
"import.adv.o1Custom": "custom — explicit layer list",
"import.adv.o1DeepN": "N deep layers",
"import.adv.o1Layers": "Layers (comma-separated)",
"import.adv.o1mHint": "landmark budget (default 32)",
"import.adv.o1windowHint": "exact-window width (default 128)",
"import.adv.o1sinkHint": "permanent exact sink keys (default 4)",
}; };
+42 -2
View File
@@ -58,6 +58,9 @@ export default {
// dashboard // dashboard
"dash.title": "Дашборд", "dash.title": "Дашборд",
"dash.subtitle": "Трафик в реальном времени, расходы и аналитика роутинга по пулу моделей.", "dash.subtitle": "Трафик в реальном времени, расходы и аналитика роутинга по пулу моделей.",
"dash.clear": "Очистить логи",
"dash.clearConfirm": "Очистить всю статистику и логи? Отменить нельзя.",
"dash.cleared": "Логи очищены",
"dash.range": "Период", "dash.range": "Период",
"dash.range.1h": "1ч", "dash.range.1h": "1ч",
"dash.range.24h": "24ч", "dash.range.24h": "24ч",
@@ -128,6 +131,18 @@ export default {
"models.form.secretMissing": "не задан", "models.form.secretMissing": "не задан",
"models.form.secretPlaceholder": "задать или заменить ключ (только запись)", "models.form.secretPlaceholder": "задать или заменить ключ (только запись)",
"models.deleteConfirm": "Удалить модель \"{id}\"?", "models.deleteConfirm": "Удалить модель \"{id}\"?",
"models.form.idOrModel": "Укажите id или имя модели",
"models.form.apiKey": "API-ключ",
"models.form.apiKeyPlaceholder": "вставьте API-ключ",
"models.form.apiKeyNotNeeded": "не нужен (локальный сервер)",
"models.form.apiKeyHint": "Хранится безопасно на шлюзе; больше не показывается.",
"models.form.keyStored": "Ключ сохранён — оставьте пустым, чтобы не менять, или вставьте новый.",
"models.form.keyFromEnv": "Используется ключ из переменной окружения.",
"models.form.testKey": "Проверить и загрузить модели",
"models.form.testing": "проверка…",
"models.form.foundModels": "загружено моделей: {n}",
"models.form.modelPlaceholder": "введите для поиска…",
"models.form.modelHint": "Нажмите «Проверить и загрузить модели», чтобы подтянуть список по ключу, затем выберите.",
"models.managed": "управляемая", "models.managed": "управляемая",
"models.managedHint": "Предоставляется CMF-сервером (Настройки → Локальные модели). Настраивается там.", "models.managedHint": "Предоставляется CMF-сервером (Настройки → Локальные модели). Настраивается там.",
@@ -243,9 +258,10 @@ export default {
"settings.cmf.threads": "Рабочие потоки", "settings.cmf.threads": "Рабочие потоки",
"settings.cmf.gpu": "Использовать GPU (Metal / wgpu)", "settings.cmf.gpu": "Использовать GPU (Metal / wgpu)",
"settings.cmf.speedNote": "Потоки распараллеливают декод-матвеки (~×3; 0 = по умолчанию рантайма). GPU-оффлоад помогает в основном на prefill длинных промптов — декод упирается в пропускную способность памяти, поэтому на Apple Silicon почти нейтрален.", "settings.cmf.speedNote": "Потоки распараллеливают декод-матвеки (~×3; 0 = по умолчанию рантайма). GPU-оффлоад помогает в основном на prefill длинных промптов — декод упирается в пропускную способность памяти, поэтому на Apple Silicon почти нейтрален.",
"settings.cmf.pick": "выберите модель",
"settings.cmf.models": "Локальные модели", "settings.cmf.models": "Локальные модели",
"settings.cmf.modelsHelp": "Каждая строка — отдельная модель: шлюз поднимает свой cortiq serve на своём порту и регистрирует её в пуле. Добавляйте столько, сколько есть .cmf-файлов и свободных портов.", "settings.cmf.modelsHelp": "Каждая строка — отдельная модель: шлюз поднимает свой cortiq serve на своём порту и регистрирует её в пуле. Добавляйте столько, сколько есть .cmf-файлов и свободных портов.",
"settings.cmf.rowHint": "id · путь к .cmf · порт · потоки · GPU. Изменения применяются после перезапуска; удаление строки останавливает сервер.", "settings.cmf.rowHint": "id · файл .cmf · порт · потоки · GPU. Изменения применяются после перезапуска; удаление строки останавливает сервер.",
"settings.cmf.addServer": "+ Добавить модель", "settings.cmf.addServer": "+ Добавить модель",
"settings.cmf.checkPort": "Проверить порт", "settings.cmf.checkPort": "Проверить порт",
"settings.cmf.portFree": "порт свободен", "settings.cmf.portFree": "порт свободен",
@@ -302,6 +318,10 @@ export default {
"settings.route.maxChars": "Макс. символов", "settings.route.maxChars": "Макс. символов",
"settings.route.cacheTtl": "TTL кэша", "settings.route.cacheTtl": "TTL кэша",
"settings.route.profile": "Профиль по умолчанию", "settings.route.profile": "Профиль по умолчанию",
"settings.route.strategyHelp": "Какой текст отправляется роутеру для классификации: last_user — только последнее сообщение пользователя; last_user_plus_system — последнее сообщение + системный промпт; concat_all — вся переписка.",
"settings.route.maxCharsHelp": "Максимум символов этого текста, отправляемого роутеру (длиннее — обрезается). Больше = точнее классификация, но медленнее/дороже вызовы роутера.",
"settings.route.cacheTtlHelp": "Сколько кэшируется решение роутинга для одинакового запроса, прежде чем роутер спрашивается снова (напр. 60s, 5m).",
"settings.route.profileHelp": "Профиль роутинга по умолчанию, если запрос его не задаёт: cost-saver (дешевле), balanced (баланс) или quality-first (качество). Влияет на выбор модели роутером.",
"settings.breaker": "Автоматический выключатель", "settings.breaker": "Автоматический выключатель",
"settings.breaker.threshold": "Порог ошибок", "settings.breaker.threshold": "Порог ошибок",
"settings.breaker.cooldown": "Время восстановления", "settings.breaker.cooldown": "Время восстановления",
@@ -353,7 +373,7 @@ export default {
"import.failed": "Конвертация не удалась — смотрите лог.", "import.failed": "Конвертация не удалась — смотрите лог.",
"import.register": "Подключить как локальную модель", "import.register": "Подключить как локальную модель",
"import.registered": "Подключено как '{id}'. Запустите cortiq-server для обслуживания.", "import.registered": "Подключено как '{id}'. Запустите cortiq-server для обслуживания.",
"import.jobs": "Конвертации", "import.jobs": "Загрузки / Конвертации",
"import.cancel": "Отменить", "import.cancel": "Отменить",
"import.cancelling": "Отмена — удаляю недописанный файл…", "import.cancelling": "Отмена — удаляю недописанный файл…",
"import.delete": "Удалить", "import.delete": "Удалить",
@@ -363,12 +383,21 @@ export default {
"import.state.done": "готово", "import.state.done": "готово",
"import.state.error": "ошибка", "import.state.error": "ошибка",
"import.state.cancelled": "отменено", "import.state.cancelled": "отменено",
"import.featuredTitle": "Рекомендуемые готовые .cmf модели",
"import.readyCmfBadge": "Готовая .cmf",
"import.directDownload": "Скачать",
"import.q.q8_2f": "8-бит двухполевой (𝒲×θ) — рекомендуется: лучшее качество/размер.", "import.q.q8_2f": "8-бит двухполевой (𝒲×θ) — рекомендуется: лучшее качество/размер.",
"import.q.q8_row": "8-бит построчно — просто, надёжно.", "import.q.q8_row": "8-бит построчно — просто, надёжно.",
"import.q.q4": "4-бит блочно — компактнее, ниже качество.", "import.q.q4": "4-бит блочно — компактнее, ниже качество.",
"import.q.q4_tiled": "4-бит тайловый (Q4_TILED) — х1.66 быстрее на Metal GPU.",
"import.q.vbit": "Переменная 3–8 бит — per-expert бюджеты.", "import.q.vbit": "Переменная 3–8 бит — per-expert бюджеты.",
"import.q.q1t": "1.58-бит тернарный (Q1T) — BitNet b1.58 с оутлайерами.",
"import.q.q1s": "1-бит спарс (Q1S) — 1-бит база с голографическими оутлайерами.",
"import.q.q1p": "1-бит PTQ (Q1P) — 1-бит с переносом ошибки (error diffusion).",
"import.q.q1": "1-бит (Q1) — чистый бинарный формат.",
"import.q.f16": "16-бит — без квантования.", "import.q.f16": "16-бит — без квантования.",
"import.q.f32": "32-бит — полная точность (большой файл).", "import.q.f32": "32-бит — полная точность (большой файл).",
"import.adv.title": "Расширенные настройки", "import.adv.title": "Расширенные настройки",
"import.adv.auto": "авто", "import.adv.auto": "авто",
"import.adv.linearCore": "Линейное ядро", "import.adv.linearCore": "Линейное ядро",
@@ -379,4 +408,15 @@ export default {
"import.adv.shard": "Шард макс (ГБ)", "import.adv.shard": "Шард макс (ГБ)",
"import.adv.noShard": "без шардинга", "import.adv.noShard": "без шардинга",
"import.adv.skipMtp": "пропустить MTP-голову", "import.adv.skipMtp": "пропустить MTP-голову",
"import.adv.o1": "O(1)-внимание",
"import.adv.o1Hint": "Потоковое внимание с постоянной памятью: KV-кэш освобождается после prefill, длинный контекст не раздувает память. Веса не меняются — записывается лишь подсказка на загрузку; вернуться можно всегда (запуск с --o1 off). Проверено на Qwen3-0.6B: все слои ≈ ×1.18 перплексии.",
"import.adv.o1Off": "выкл — точное внимание (по умолчанию)",
"import.adv.o1All": "all — конвертировать все softmax-слои",
"import.adv.o1Deep": "deep N — только N самых глубоких слоёв",
"import.adv.o1Custom": "вручную — явный список слоёв",
"import.adv.o1DeepN": "N глубоких слоёв",
"import.adv.o1Layers": "Слои (через запятую)",
"import.adv.o1mHint": "число опорных точек (по умолчанию 32)",
"import.adv.o1windowHint": "ширина точного окна (по умолчанию 128)",
"import.adv.o1sinkHint": "постоянные точные sink-ключи (по умолчанию 4)",
}; };
+3 -1
View File
@@ -25,7 +25,9 @@ export function clear(node) {
export function mount(node, ...children) { export function mount(node, ...children) {
clear(node); clear(node);
for (const c of children.flat()) if (c != null && c !== false) node.appendChild(c); for (const c of children.flat())
if (c != null && c !== false)
node.appendChild(typeof c === "object" ? c : document.createTextNode(String(c)));
return node; return node;
} }
+16 -2
View File
@@ -1,5 +1,5 @@
// Dashboard — KPIs, time-series charts, breakdown, recent requests, health. // Dashboard — KPIs, time-series charts, breakdown, recent requests, health.
import { h, mount, money, num, ms, pct, timeAgo } from "../ui.js"; import { h, mount, money, num, ms, pct, timeAgo, toast, confirmDialog } from "../ui.js";
import { t } from "../i18n.js"; import { t } from "../i18n.js";
import { api } from "../api.js"; import { api } from "../api.js";
import { lineChart, barList } from "../charts.js"; import { lineChart, barList } from "../charts.js";
@@ -53,10 +53,24 @@ export async function renderDashboard() {
"div", "div",
{ class: "flex" }, { class: "flex" },
h("div", { class: "grow" }, h("h2", {}, t("dash.title")), h("p", {}, t("dash.subtitle"))), h("div", { class: "grow" }, h("h2", {}, t("dash.title")), h("p", {}, t("dash.subtitle"))),
h("div", { class: "flex", style: "gap:8px;align-items:center" },
seg(RANGES, range, (r) => { seg(RANGES, range, (r) => {
range = r; range = r;
load(); load();
}, (r) => t("dash.range." + r)) }, (r) => t("dash.range." + r)),
h("button", {
class: "btn sm danger",
onClick: async () => {
if (!(await confirmDialog(t("dash.clearConfirm")))) return;
try {
await api.clearStats();
toast(t("dash.cleared"), "good");
await load();
} catch (e) {
toast(String(e.message || e), "bad");
}
},
}, t("dash.clear")))
) )
), ),
+96
View File
@@ -10,11 +10,23 @@ const QUANTS = [
["Q8_2F", "import.q.q8_2f"], ["Q8_2F", "import.q.q8_2f"],
["Q8_ROW", "import.q.q8_row"], ["Q8_ROW", "import.q.q8_row"],
["Q4_BLOCK", "import.q.q4"], ["Q4_BLOCK", "import.q.q4"],
["Q4_TILED", "import.q.q4_tiled"],
["VBIT", "import.q.vbit"], ["VBIT", "import.q.vbit"],
["Q1T", "import.q.q1t"],
["Q1S", "import.q.q1s"],
["Q1P", "import.q.q1p"],
["Q1", "import.q.q1"],
["F16", "import.q.f16"], ["F16", "import.q.f16"],
["F32", "import.q.f32"], ["F32", "import.q.f32"],
]; ];
const FEATURED_REPOS = [
{ id: "infosave/Bonsai-1.7Bcmf", label: "Bonsai 1.7B CMF", badge: "1.7B · Q1", quant: "Q1" },
{ id: "infosave/Bonsai-8B_2bit_cmf", label: "Bonsai 8B 2-bit CMF", badge: "8B · Q1T", quant: "Q1T" },
{ id: "infosave/Bonsai-27Bcmf", label: "Bonsai 27B CMF", badge: "27B · Q1", quant: "Q1" },
];
function field(label, control, hint) { function field(label, control, hint) {
return h("label", { class: "field" }, h("span", {}, label), control, return h("label", { class: "field" }, h("span", {}, label), control,
hint ? h("div", { class: "hint" }, hint) : null); hint ? h("div", { class: "hint" }, hint) : null);
@@ -125,6 +137,32 @@ export function renderImport() {
const meanBitsIn = h("input", { type: "number", step: "0.1", placeholder: "5.0" }); const meanBitsIn = h("input", { type: "number", step: "0.1", placeholder: "5.0" });
const shardIn = h("input", { type: "number", step: "0.5", placeholder: t("import.adv.noShard") }); const shardIn = h("input", { type: "number", step: "0.5", placeholder: t("import.adv.noShard") });
const skipMtp = h("input", { type: "checkbox" }); const skipMtp = h("input", { type: "checkbox" });
// O(1) attention (cortiq ≥ 0.2.0): constant-memory streaming attention hint,
// weights byte-identical — the runtime applies it at load.
const o1Sel = h("select", {},
opt("", t("import.adv.o1Off")),
opt("all", t("import.adv.o1All")),
opt("deep", t("import.adv.o1Deep")),
opt("custom", t("import.adv.o1Custom")));
const o1DeepIn = h("input", { type: "number", min: "1", placeholder: "12" });
const o1CustomIn = h("input", { placeholder: "0,4,8" });
const o1M = h("input", { type: "number", min: "1", placeholder: "32" });
const o1Window = h("input", { type: "number", min: "0", placeholder: "128" });
const o1Sink = h("input", { type: "number", min: "0", placeholder: "4" });
const o1SpecRow = h("div", { class: "row", style: "display:none" },
field(t("import.adv.o1DeepN"), o1DeepIn),
field(t("import.adv.o1Layers"), o1CustomIn));
const o1Knobs = h("div", { class: "row", style: "display:none" },
field("--o1-m", o1M, t("import.adv.o1mHint")),
field("--o1-window", o1Window, t("import.adv.o1windowHint")),
field("--o1-sink", o1Sink, t("import.adv.o1sinkHint")));
o1Sel.addEventListener("change", () => {
const on = !!o1Sel.value;
o1Knobs.style.display = on ? "" : "none";
o1SpecRow.style.display = o1Sel.value === "deep" || o1Sel.value === "custom" ? "" : "none";
o1DeepIn.parentElement.style.display = o1Sel.value === "deep" ? "" : "none";
o1CustomIn.parentElement.style.display = o1Sel.value === "custom" ? "" : "none";
});
const advBody = h( const advBody = h(
"div", "div",
{ class: "adv-body", style: "display:none" }, { class: "adv-body", style: "display:none" },
@@ -137,6 +175,9 @@ export function renderImport() {
h("div", { class: "row" }, h("div", { class: "row" },
field(t("import.adv.shard"), shardIn), field(t("import.adv.shard"), shardIn),
field("MTP", h("label", { class: "check" }, skipMtp, t("import.adv.skipMtp")))), field("MTP", h("label", { class: "check" }, skipMtp, t("import.adv.skipMtp")))),
field(t("import.adv.o1"), o1Sel, t("import.adv.o1Hint")),
o1SpecRow,
o1Knobs,
); );
const advToggle = h("button", { class: "btn ghost sm" }, "⚙ " + t("import.adv.title")); const advToggle = h("button", { class: "btn ghost sm" }, "⚙ " + t("import.adv.title"));
advToggle.addEventListener("click", () => { advToggle.addEventListener("click", () => {
@@ -156,6 +197,19 @@ export function renderImport() {
if (vbitSel.value) params.vbit_shape = vbitSel.value; if (vbitSel.value) params.vbit_shape = vbitSel.value;
if (meanBitsIn.value) params.mean_bits = parseFloat(meanBitsIn.value); if (meanBitsIn.value) params.mean_bits = parseFloat(meanBitsIn.value);
if (shardIn.value) params.shard_max_gb = parseFloat(shardIn.value); if (shardIn.value) params.shard_max_gb = parseFloat(shardIn.value);
if (o1Sel.value) {
// spec: all | deepN | explicit layer list
params.o1 =
o1Sel.value === "deep" ? "deep" + (parseInt(o1DeepIn.value) || 12)
: o1Sel.value === "custom" ? o1CustomIn.value.trim()
: "all";
if (!params.o1) delete params.o1;
else {
if (o1M.value) params.o1_m = parseInt(o1M.value);
if (o1Window.value) params.o1_window = parseInt(o1Window.value);
if (o1Sink.value) params.o1_sink = parseInt(o1Sink.value);
}
}
goBtn.disabled = true; goBtn.disabled = true;
try { try {
await api.startImport(params); await api.startImport(params);
@@ -316,6 +370,46 @@ export function renderImport() {
); );
} }
function featuredCard(item) {
const card = h(
"div",
{ class: "hf-card featured-card" },
h("div", { class: "row", style: "align-items:center; justify-content:space-between;" },
h("div", {},
h("div", { class: "hf-id" }, item.label),
h("div", { class: "hf-meta" },
h("span", { class: "badge" }, item.badge),
h("span", { class: "muted" }, item.id)
)
),
h("button", { class: "btn primary sm" }, "⬇ " + t("import.directDownload"))
)
);
card.querySelector("button").addEventListener("click", async (e) => {
e.stopPropagation();
try {
await api.startImport({
repo: item.id,
quant: item.quant || "READY_CMF",
name: item.id.split("/").pop().toLowerCase(),
});
toast(t("import.started"), "ok");
refreshJobs();
jobsHost.scrollIntoView({ behavior: "smooth", block: "nearest" });
} catch (err) {
toast(err.message, "err");
}
});
return card;
}
const featuredHost = h(
"div",
{ class: "card featured-section" },
h("div", { class: "card-title" }, "⚡ " + t("import.featuredTitle")),
h("div", { class: "hf-grid" }, ...FEATURED_REPOS.map(featuredCard))
);
const root = h( const root = h(
"div", "div",
{}, {},
@@ -323,6 +417,7 @@ export function renderImport() {
h("h2", {}, t("nav.import")), h("h2", {}, t("nav.import")),
h("div", { class: "muted" }, t("import.subtitle"))), h("div", { class: "muted" }, t("import.subtitle"))),
jobsHost, // progress panel on top, above the search jobsHost, // progress panel on top, above the search
featuredHost, // featured ready-to-run models
h("div", { class: "card" }, searchIn, results), h("div", { class: "card" }, searchIn, results),
formHost formHost
); );
@@ -330,3 +425,4 @@ export function renderImport() {
refreshJobs(); // resume: show any running/recent conversions immediately refreshJobs(); // resume: show any running/recent conversions immediately
return root; return root;
} }
+111 -19
View File
@@ -16,37 +16,126 @@ function keyBadge(src) {
return h("span", { class: "badge " + (map[src] ?? "") }, src); return h("span", { class: "badge " + (map[src] ?? "") }, src);
} }
// Verified per-provider defaults (base URL, key env, whether a key is needed).
const PROVIDER_DEFAULTS = {
openai: { base: "https://api.openai.com/v1", keyEnv: "OPENAI_API_KEY", needsKey: true, tier: "mid", caps: ["tools", "vision"] },
anthropic: { base: "https://api.anthropic.com", keyEnv: "ANTHROPIC_API_KEY", needsKey: true, tier: "expensive", caps: ["tools", "vision"] },
openrouter: { base: "https://openrouter.ai/api/v1", keyEnv: "OPENROUTER_API_KEY", needsKey: true, tier: "mid", caps: ["tools", "vision"] },
lmstudio: { base: "http://localhost:1234/v1", keyEnv: "", needsKey: false, tier: "local", caps: ["tools", "vision"] },
ollama: { base: "http://localhost:11434/v1", keyEnv: "", needsKey: false, tier: "local", caps: ["tools"] },
http: { base: "", keyEnv: "", needsKey: false, tier: "cheap", caps: ["tools"] },
};
// Derive a clean model id from a model name (last path segment, slugified).
// e.g. "qwen/qwen3.7-max" → "qwen3-7-max".
function slugId(name) {
const base = (name || "").split("/").pop() || "";
return base.toLowerCase().replace(/[^a-z0-9]+/g, "-").replace(/^-+|-+$/g, "");
}
function modelForm(meta, existing) { function modelForm(meta, existing) {
const m = existing || { provider: "openai", kind: "chat", cost_tier: "cheap", caps: [], price_in: 0, price_out: 0 }; const m = existing || { provider: "openai", kind: "chat", cost_tier: "mid", caps: ["tools"], price_in: 0, price_out: 0 };
const idIn = h("input", { value: m.id || "", placeholder: "local-qwen", disabled: existing ? true : null }); const idIn = h("input", { value: m.id || "", placeholder: "my-model", disabled: existing ? true : null });
const providerSel = h("select", {}, ...(meta.providers || ["openai"]).map((p) => opt(p, p, p === m.provider))); const providerSel = h("select", {}, ...(meta.providers || ["openai"]).map((p) => opt(p, p, p === m.provider)));
const baseUrlIn = h("input", { value: m.base_url || "", placeholder: "http://localhost:8000/v1" }); const baseUrlIn = h("input", { value: m.base_url || "", placeholder: "https://api.openai.com/v1" });
const modelIn = h("input", { value: m.model || "", placeholder: "qwen2.5-7b-instruct" });
// API key — typed directly, stored securely as a secret under a provider env name.
const keyIn = h("input", { type: "password", placeholder: t("models.form.apiKeyPlaceholder"), autocomplete: "new-password" });
const testBtn = h("button", { class: "btn small", type: "button" }, t("models.form.testKey"));
const keyStatus = h("span", { class: "small", style: "margin-left:8px" });
// Model — below the key (some providers list models from the key); searchable datalist.
const dlId = "model-opts-" + (m.id || "new");
const dl = h("datalist", { id: dlId });
if (m.model) dl.appendChild(h("option", { value: m.model }));
const modelIn = h("input", { value: m.model || "", placeholder: t("models.form.modelPlaceholder"), list: dlId, autocomplete: "off" });
const kindSel = h("select", {}, ...(meta.kinds || ["chat"]).map((k) => opt(k, k, k === m.kind))); const kindSel = h("select", {}, ...(meta.kinds || ["chat"]).map((k) => opt(k, k, k === m.kind)));
const tierSel = h("select", {}, ...(meta.cost_tiers || ["cheap"]).map((c) => opt(c, c, c === m.cost_tier))); const tierSel = h("select", {}, ...(meta.cost_tiers || ["cheap"]).map((c) => opt(c, c, c === m.cost_tier)));
const priceInIn = h("input", { type: "number", step: "0.01", value: m.price_in ?? 0 }); const priceInIn = h("input", { type: "number", step: "0.01", value: m.price_in ?? 0 });
const priceOutIn = h("input", { type: "number", step: "0.01", value: m.price_out ?? 0 }); const priceOutIn = h("input", { type: "number", step: "0.01", value: m.price_out ?? 0 });
const keyEnvIn = h("input", { value: m.api_key_env || "", placeholder: "OPENAI_API_KEY" });
const secretIn = h("input", { type: "password", placeholder: t("models.form.secretPlaceholder") });
const capInputs = (meta.caps || ["tools", "vision"]).map((c) => { const capInputs = (meta.caps || ["tools", "vision"]).map((c) => {
const cb = h("input", { type: "checkbox", checked: (m.caps || []).includes(c) ? true : null }); const cb = h("input", { type: "checkbox", checked: (m.caps || []).includes(c) ? true : null });
return { c, cb }; return { c, cb };
}); });
const envFor = () =>
(existing && existing.api_key_env) ||
(PROVIDER_DEFAULTS[providerSel.value] || {}).keyEnv ||
providerSel.value.toUpperCase().replace(/[^A-Z0-9]/g, "_") + "_API_KEY";
// Fill in a provider's verified defaults when it is selected.
function applyDefaults() {
const d = PROVIDER_DEFAULTS[providerSel.value];
if (!d) return;
baseUrlIn.value = d.base;
tierSel.value = d.tier;
capInputs.forEach(({ c, cb }) => { cb.checked = d.caps.includes(c); });
keyIn.placeholder = d.needsKey ? t("models.form.apiKeyPlaceholder") : t("models.form.apiKeyNotNeeded");
mount(keyStatus, "");
}
providerSel.addEventListener("change", applyDefaults);
if (!existing) applyDefaults();
// Auto-fill the id from the model name (until the user edits id by hand).
if (!existing) {
modelIn.addEventListener("input", () => {
if (!idIn.value || idIn._auto) {
idIn.value = slugId(modelIn.value);
idIn._auto = true;
}
});
idIn.addEventListener("input", () => { idIn._auto = false; });
}
async function testKey() {
if (!baseUrlIn.value.trim()) { mount(keyStatus, h("span", { style: "color:var(--bad)" }, t("models.form.baseUrl") + "?")); return; }
testBtn.disabled = true;
mount(keyStatus, t("models.form.testing"));
try {
const r = await api.providerModels({
provider: providerSel.value,
base_url: baseUrlIn.value.trim(),
api_key: keyIn.value.trim() || null,
api_key_env: envFor(),
});
if (r.ok) {
dl.replaceChildren(...(r.models || []).map((mm) => h("option", { value: mm })));
mount(keyStatus, h("span", { style: "color:var(--good)" }, "✓ " + t("models.form.foundModels", { n: r.count })));
modelIn.focus();
} else {
mount(keyStatus, h("span", { style: "color:var(--bad)" }, "✗ " + (r.error || t("common.error"))));
}
} catch (e) {
mount(keyStatus, h("span", { style: "color:var(--bad)" }, "✗ " + String(e.message || e)));
}
testBtn.disabled = false;
}
testBtn.addEventListener("click", testKey);
const keyHint =
existing && existing.key_source === "store" ? t("models.form.keyStored")
: existing && existing.key_source === "env" ? t("models.form.keyFromEnv")
: t("models.form.apiKeyHint");
const node = h( const node = h(
"div", "div",
{}, {},
h("div", { class: "row" }, field(t("models.form.id"), idIn, existing ? null : t("models.form.idHint")), field(t("models.form.provider"), providerSel)), h("div", { class: "row" }, field(t("models.form.id"), idIn, existing ? null : t("models.form.idHint")), field(t("models.form.provider"), providerSel)),
field(t("models.form.baseUrl"), baseUrlIn), field(t("models.form.baseUrl"), baseUrlIn),
field(t("models.form.model"), modelIn), field(t("models.form.apiKey"), h("div", { class: "flex wrap", style: "gap:8px;align-items:center" }, keyIn, testBtn, keyStatus), keyHint),
field(t("models.form.model"), modelIn, t("models.form.modelHint")),
dl,
h("div", { class: "row" }, field(t("models.form.kind"), kindSel), field(t("models.form.costTier"), tierSel)), h("div", { class: "row" }, field(t("models.form.kind"), kindSel), field(t("models.form.costTier"), tierSel)),
h("div", { class: "row" }, field(t("models.form.priceIn"), priceInIn), field(t("models.form.priceOut"), priceOutIn)), h("div", { class: "row" }, field(t("models.form.priceIn"), priceInIn), field(t("models.form.priceOut"), priceOutIn)),
field(t("models.form.caps"), h("div", { class: "flex wrap" }, ...capInputs.map(({ c, cb }) => h("label", { class: "check" }, cb, c)))), field(t("models.form.caps"), h("div", { class: "flex wrap" }, ...capInputs.map(({ c, cb }) => h("label", { class: "check" }, cb, c))))
field(t("models.form.apiKeyEnv"), keyEnvIn, t("models.form.apiKeyEnvHint")),
field(t("models.form.secret"), secretIn)
); );
const getValue = () => ({ const getValue = () => {
const typedKey = keyIn.value.trim();
const hasKey = !!typedKey || !!(existing && existing.api_key_env);
const keyEnv = envFor();
return {
model: { model: {
id: existing ? m.id : idIn.value.trim(), id: existing ? m.id : idIn.value.trim(),
provider: providerSel.value, provider: providerSel.value,
@@ -56,12 +145,13 @@ function modelForm(meta, existing) {
cost_tier: tierSel.value, cost_tier: tierSel.value,
price_in: parseFloat(priceInIn.value) || 0, price_in: parseFloat(priceInIn.value) || 0,
price_out: parseFloat(priceOutIn.value) || 0, price_out: parseFloat(priceOutIn.value) || 0,
api_key_env: keyEnvIn.value.trim() || null, api_key_env: hasKey ? keyEnv : null,
caps: capInputs.filter(({ cb }) => cb.checked).map(({ c }) => c), caps: capInputs.filter(({ cb }) => cb.checked).map(({ c }) => c),
}, },
secret: secretIn.value, secret: typedKey,
keyEnv: keyEnvIn.value.trim(), keyEnv,
}); };
};
return { node, getValue }; return { node, getValue };
} }
@@ -70,12 +160,14 @@ async function openModal(meta, existing, reload) {
const { node, getValue } = modelForm(meta, existing); const { node, getValue } = modelForm(meta, existing);
modal(existing ? t("models.form.editTitle") : t("models.form.addTitle"), node, async () => { modal(existing ? t("models.form.editTitle") : t("models.form.addTitle"), node, async () => {
const { model, secret, keyEnv } = getValue(); const { model, secret, keyEnv } = getValue();
// Forgot the id? Derive it from the model name so saving never errors on a
// blank id.
if (!model.id) model.id = slugId(model.model);
if (!model.id) { if (!model.id) {
toast(t("models.form.id") + "?", "bad"); toast(t("models.form.idOrModel"), "bad");
return false; return false;
} }
// strip null api_key_env so serde keeps it optional if (!model.api_key_env) delete model.api_key_env; // keep serde field optional
if (!model.api_key_env) delete model.api_key_env;
try { try {
if (existing) await api.updateModel(existing.id, model); if (existing) await api.updateModel(existing.id, model);
else await api.createModel(model); else await api.createModel(model);
@@ -193,7 +285,7 @@ function row(m, meta, reload) {
try { try {
await api.deleteModel(m.id); await api.deleteModel(m.id);
toast(t("toast.deleted"), "good"); toast(t("toast.deleted"), "good");
reload(); await reload();
} catch (e) { } catch (e) {
toast(String(e.message || e), "bad"); toast(String(e.message || e), "bad");
} }
+18 -7
View File
@@ -214,6 +214,9 @@ export async function renderSettings() {
? [{ id: s.cmf.model_id || "cmf-local", model: s.cmf.local_model, port: s.cmf.local_port ?? 8081, threads: s.cmf.threads ?? 8, gpu: !!s.cmf.gpu }] ? [{ id: s.cmf.model_id || "cmf-local", model: s.cmf.local_model, port: s.cmf.local_port ?? 8081, threads: s.cmf.threads ?? 8, gpu: !!s.cmf.gpu }]
: []; : [];
const initServers = (s.cmf?.servers && s.cmf.servers.length) ? s.cmf.servers : legacyServers; const initServers = (s.cmf?.servers && s.cmf.servers.length) ? s.cmf.servers : legacyServers;
// available .cmf files under models_dir → offer as a dropdown, not a typed path
const cmfFiles = ((await api.cmfFiles().catch(() => ({ files: [] }))).files) || [];
const fileLabel = (f) => f.name + (f.size ? ` (${(f.size / 1073741824).toFixed(f.size >= 1073741824 ? 1 : 2)} GB)` : f.is_dir ? " (dir)" : "");
function autoId(path) { function autoId(path) {
const base = (path || "").split("/").pop().replace(/\.cmf$/i, ""); const base = (path || "").split("/").pop().replace(/\.cmf$/i, "");
return base ? "cmf-" + base.replace(/[^a-zA-Z0-9_-]/g, "-") : ""; return base ? "cmf-" + base.replace(/[^a-zA-Z0-9_-]/g, "-") : "";
@@ -221,14 +224,20 @@ export async function renderSettings() {
function makeServerRow(sv) { function makeServerRow(sv) {
sv = sv || { id: "", model: "", port: 8090, threads: 8, gpu: false }; sv = sv || { id: "", model: "", port: 8090, threads: 8, gpu: false };
const idIn = h("input", { value: sv.id || "", placeholder: "cmf-id", style: "width:130px" }); const idIn = h("input", { value: sv.id || "", placeholder: "cmf-id", style: "width:130px" });
const modelIn = h("input", { value: sv.model || "", placeholder: "models/model.cmf", style: "flex:2;min-width:150px" }); // pick from existing .cmf files rather than typing a path
const modelSel = h("select", { style: "flex:2;min-width:150px" });
const seen = new Set();
modelSel.appendChild(h("option", { value: "" }, "— " + t("settings.cmf.pick") + " —"));
cmfFiles.forEach((f) => { modelSel.appendChild(h("option", { value: f.path }, fileLabel(f))); seen.add(f.path); });
if (sv.model && !seen.has(sv.model)) modelSel.appendChild(h("option", { value: sv.model }, sv.model));
modelSel.value = sv.model || "";
const portIn = h("input", { type: "number", value: sv.port ?? 8090, style: "width:82px" }); const portIn = h("input", { type: "number", value: sv.port ?? 8090, style: "width:82px" });
const threadsIn = h("input", { type: "number", min: 0, value: sv.threads ?? 8, style: "width:60px", title: t("settings.cmf.threadsHelp") }); const threadsIn = h("input", { type: "number", min: 0, value: sv.threads ?? 8, style: "width:60px", title: t("settings.cmf.threadsHelp") });
const gpuIn = h("input", { type: "checkbox", checked: sv.gpu ? true : null, title: t("settings.cmf.gpuHelp") }); const gpuIn = h("input", { type: "checkbox", checked: sv.gpu ? true : null, title: t("settings.cmf.gpuHelp") });
const checkBtn = h("button", { class: "btn small", type: "button" }, t("settings.cmf.checkPort")); const checkBtn = h("button", { class: "btn small", type: "button" }, t("settings.cmf.checkPort"));
const removeBtn = h("button", { class: "btn small danger", type: "button", title: t("common.delete") }, "✕"); const removeBtn = h("button", { class: "btn small danger", type: "button", title: t("common.delete") }, "✕");
const st = h("span", { class: "small" }); const st = h("span", { class: "small" });
modelIn.addEventListener("input", () => { if (!idIn.value || idIn._auto) { idIn.value = autoId(modelIn.value); idIn._auto = true; } }); modelSel.addEventListener("change", () => { if (!idIn.value || idIn._auto) { idIn.value = autoId(modelSel.value); idIn._auto = true; } });
idIn.addEventListener("input", () => { idIn._auto = false; }); idIn.addEventListener("input", () => { idIn._auto = false; });
checkBtn.addEventListener("click", async () => { checkBtn.addEventListener("click", async () => {
const p = parseInt(portIn.value, 10); if (!p) return; const p = parseInt(portIn.value, 10); if (!p) return;
@@ -245,8 +254,8 @@ export async function renderSettings() {
checkBtn.disabled = false; checkBtn.disabled = false;
}); });
const row = h("div", { class: "flex wrap", style: "gap:6px;align-items:center;margin:6px 0" }, const row = h("div", { class: "flex wrap", style: "gap:6px;align-items:center;margin:6px 0" },
idIn, modelIn, portIn, checkBtn, threadsIn, h("label", { class: "flex", style: "gap:4px" }, gpuIn, "GPU"), removeBtn, st); idIn, modelSel, portIn, checkBtn, threadsIn, h("label", { class: "flex", style: "gap:4px" }, gpuIn, "GPU"), removeBtn, st);
const entry = { node: row, read: () => ({ id: idIn.value.trim() || autoId(modelIn.value), model: modelIn.value.trim(), port: parseInt(portIn.value) || 8090, threads: parseInt(threadsIn.value) || 0, gpu: gpuIn.checked }) }; const entry = { node: row, read: () => ({ id: idIn.value.trim() || autoId(modelSel.value), model: modelSel.value.trim(), port: parseInt(portIn.value) || 8090, threads: parseInt(threadsIn.value) || 0, gpu: gpuIn.checked }) };
removeBtn.addEventListener("click", () => { const i = serverRows.indexOf(entry); if (i >= 0) serverRows.splice(i, 1); row.remove(); }); removeBtn.addEventListener("click", () => { const i = serverRows.indexOf(entry); if (i >= 0) serverRows.splice(i, 1); row.remove(); });
serverRows.push(entry); serverRows.push(entry);
serversWrap.appendChild(row); serversWrap.appendChild(row);
@@ -433,9 +442,11 @@ export async function renderSettings() {
"div", "div",
{ class: "card" }, { class: "card" },
h("div", { class: "card-head" }, h("h3", {}, t("settings.route"))), h("div", { class: "card-head" }, h("h3", {}, t("settings.route"))),
field(t("settings.route.strategy"), strategySel), field(t("settings.route.strategy"), strategySel, null, t("settings.route.strategyHelp")),
h("div", { class: "row" }, field(t("settings.route.maxChars"), maxChars), field(t("settings.route.cacheTtl"), cacheTtl)), h("div", { class: "row" },
field(t("settings.route.profile"), profileSel) field(t("settings.route.maxChars"), maxChars, null, t("settings.route.maxCharsHelp")),
field(t("settings.route.cacheTtl"), cacheTtl, null, t("settings.route.cacheTtlHelp"))),
field(t("settings.route.profile"), profileSel, null, t("settings.route.profileHelp"))
), ),
h( h(
"div", "div",