Compare commits

..
6 Commits
Author SHA1 Message Date
infosave2007 b867a6276e chore(release): v0.2.29 — fix duplicate managed model registration conflict
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
2026-07-21 23:39:47 +03:00
infosave2007 c18ddb5475 chore(release): v0.2.28 — auto-detect ready CMF quant & update section title
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
2026-07-21 23:35:14 +03:00
infosave2007 ef9b3738fd chore(release): v0.2.27 — auto-spawn local CMF server on model registration
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
2026-07-21 23:15:33 +03:00
infosave2007 924ec403bd chore(release): v0.2.26 — robust multi-GB streaming downloads & ready CMF quants
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
2026-07-21 23:08:02 +03:00
infosave2007 0c91f09113 chore(release): v0.2.25 — direct stream download for ready CMF repos
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
2026-07-21 22:51:58 +03:00
infosave2007 45deb6645b chore(release): v0.2.24 — new quantizations (q4t, q1t, q1s, q1p, q1) and featured CMF models
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
2026-07-21 22:45:43 +03:00
16 changed files with 420 additions and 403 deletions
+28
View File
@@ -3,8 +3,36 @@
All notable changes to this project are documented here. All notable changes to this project are documented here.
The format is based on [Keep a Changelog](https://keepachangelog.com/). The format is based on [Keep a Changelog](https://keepachangelog.com/).
## [0.2.29] - 2026-07-21
### Fixed
- **Duplicate Local Model Registration**: Fixed `duplicate managed model id` conflict in `register_import` by cleanly routing managed CMF models directly into `[[cmf.servers]]` with automatic free port allocation without pushing duplicate entries to `[[models]]`.
### Fixed
- **Quantization Badge & Title for Pre-Converted Models**: Automatically detect quantization (`Q1`, `Q1T`) for ready-to-run `.cmf` models (e.g. `infosave/Bonsai-27Bcmf`) from repo name and downloaded filename, resolving default `Q8_2F` badge fallback. Updated section title to "Загрузки / Конвертации" ("Downloads & Conversions").
### Fixed
- **Automatic CMF Local Server Lifecycle**: Registering an imported `.cmf` model automatically adds it to `[[cmf.servers]]` and reloads the runtime (`cmf_runtime::manage`), automatically spawning `cortiq serve` on port 8090 so chat completion requests route seamlessly without `upstream unavailable` errors.
### Fixed
- **Multi-GB Ready CMF Download Stream**: Replaced HTTP stream decoder with a streaming process downloader (`curl`) for pre-converted `.cmf` models to support multi-gigabyte Hugging Face LFS repositories without 30s/60s socket timeouts.
- **Ready CMF Quantization & UI Labels**: Set correct quantization badges (`Q1`, `Q1T`) for featured ready-to-run `.cmf` models and removed misleading "converting" log messages for direct downloads.
### Fixed
- **Direct Ready CMF Download**: Importing pre-converted `.cmf` model repositories (e.g. `infosave/Bonsai-8B_2bit_cmf`) directly streams the ready `.cmf` file from Hugging Face instead of attempting a safetensors conversion without `config.json`.
### Added
- **New Quantization Schemes**: Added support for `Q4_TILED` (q4t), `Q1T` (1.58-bit ternary), `Q1S` (1-bit sparse), `Q1P` (1-bit PTQ error diffusion), and `Q1` (pure binary) in converter backend and web admin UI.
- **Featured Ready-to-Run .cmf Models**: Added 1-tap download cards for pre-converted `.cmf` models (`infosave/Bonsai-1.7Bcmf`, `infosave/Bonsai-8B_2bit_cmf`, `infosave/Bonsai-27Bcmf`).
## [Unreleased] ## [Unreleased]
### Added ### Added
- **Docker Compose** quickstart (`docker-compose.yml` + `config/gateway.compose.toml`): - **Docker Compose** quickstart (`docker-compose.yml` + `config/gateway.compose.toml`):
gateway + a local Ollama model in one command. GHCR badge in the READMEs. gateway + a local Ollama model in one command. GHCR badge in the READMEs.
Generated
+1 -1
View File
@@ -285,7 +285,7 @@ checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570"
[[package]] [[package]]
name = "cortiq-gateway" name = "cortiq-gateway"
version = "0.2.22" version = "0.2.29"
dependencies = [ dependencies = [
"anyhow", "anyhow",
"arc-swap", "arc-swap",
+1 -1
View File
@@ -1,6 +1,6 @@
[package] [package]
name = "cortiq-gateway" name = "cortiq-gateway"
version = "0.2.22" version = "0.2.29"
edition = "2021" edition = "2021"
rust-version = "1.75" rust-version = "1.75"
license = "Apache-2.0" license = "Apache-2.0"
+60 -132
View File
@@ -606,10 +606,7 @@ fn cmf_file_for_model(
cmf: &crate::config::CmfCfg, cmf: &crate::config::CmfCfg,
m: &crate::config::ModelCfg, m: &crate::config::ModelCfg,
) -> Option<std::path::PathBuf> { ) -> Option<std::path::PathBuf> {
// Any loopback URL counts as local — managed servers may sit on any port.
let local = m.base_url == cmf.cortiq_server_url let local = m.base_url == cmf.cortiq_server_url
|| m.base_url.contains("//127.0.0.1:")
|| m.base_url.contains("//localhost:")
|| m.base_url || m.base_url
.contains(&format!("{}:{}", cmf.local_host, cmf.local_port)); .contains(&format!("{}:{}", cmf.local_host, cmf.local_port));
let stem = m.model.trim(); let stem = m.model.trim();
@@ -649,24 +646,12 @@ fn cmf_path_referenced(cfg: &crate::config::Config, path: &std::path::Path) -> b
.any(|m| cmf_file_for_model(&cfg.cmf, m).as_deref() == Some(path)) .any(|m| cmf_file_for_model(&cfg.cmf, m).as_deref() == Some(path))
} }
#[derive(Deserialize)] async fn delete_model(State(state): State<SharedState>, Path(id): Path<String>) -> ApiResult {
struct DeleteModelQuery {
/// Also remove the local `.cmf` from disk. Defaults to true (the historical
/// behaviour); the UI asks the user and passes it explicitly.
delete_file: Option<bool>,
}
async fn delete_model(
State(state): State<SharedState>,
Path(id): Path<String>,
Query(q): Query<DeleteModelQuery>,
) -> ApiResult {
let mut cfg = current_cfg(&state); let mut cfg = current_cfg(&state);
let delete_file = q.delete_file.unwrap_or(true);
// Managed local models live in [cmf], not [[models]]. Deleting one stops its // Managed local models live in [cmf], not [[models]]. Deleting one stops its
// server, removes it from the managed config, and (when asked) deletes its // server, removes it from the managed config, and deletes its file — unless
// file — unless another model still points at the same file. // another model still points at the same file.
let managed = cfg.cmf.effective_servers().into_iter().find(|s| s.id == id); let managed = cfg.cmf.effective_servers().into_iter().find(|s| s.id == id);
if let Some(server) = managed { if let Some(server) = managed {
if !cfg.models.iter().any(|x| x.id == id) { if !cfg.models.iter().any(|x| x.id == id) {
@@ -681,10 +666,10 @@ async fn delete_model(
let shared = cmf_path_referenced(&cfg, &path); let shared = cmf_path_referenced(&cfg, &path);
state.cmf.stop_one(&id).await; state.cmf.stop_one(&id).await;
state.reload(cfg)?; state.reload(cfg)?;
let file_removed = if delete_file && !shared { let file_removed = if shared {
remove_cmf_path(&path)
} else {
false false
} else {
remove_cmf_path(&path)
}; };
return ok(json!({ "ok": true, "file_removed": file_removed })); return ok(json!({ "ok": true, "file_removed": file_removed }));
} }
@@ -700,8 +685,8 @@ async fn delete_model(
let file = victim let file = victim
.as_ref() .as_ref()
.and_then(|m| cmf_file_for_model(&cfg.cmf, m)); .and_then(|m| cmf_file_for_model(&cfg.cmf, m));
// Only remove the file if asked to AND no remaining model still references it. // Only remove the file if no remaining model still references it.
let file = file.filter(|p| delete_file && !cmf_path_referenced(&cfg, p)); let file = file.filter(|p| !cmf_path_referenced(&cfg, p));
state.reload(cfg)?; state.reload(cfg)?;
let file_removed = file.map(|p| remove_cmf_path(&p)).unwrap_or(false); let file_removed = file.map(|p| remove_cmf_path(&p)).unwrap_or(false);
ok(json!({ "ok": true, "file_removed": file_removed })) ok(json!({ "ok": true, "file_removed": file_removed }))
@@ -908,101 +893,22 @@ async fn put_settings(State(state): State<SharedState>, Json(b): Json<SettingsBo
needs_restart = true; needs_restart = true;
cfg.cache = v; cfg.cache = v;
} }
let mut cmf_apply: Option<(Vec<crate::config::CmfServer>, CmfCfg)> = None;
if let Some(v) = b.cmf { if let Some(v) = b.cmf {
let old_servers = cfg.cmf.effective_servers(); // manage_server / local_model / threads / gpu apply on restart (the server
cfg.cmf = v; // is spawned once at startup with those env vars); local_only / router
// A managed row supersedes a same-id STATIC pool entry that points at a // routing apply immediately.
// local server (the shape Import→register creates): drop the static one if v.manage_server != cfg.cmf.manage_server
// instead of failing validation with "duplicate managed model id". || v.local_model != cfg.cmf.local_model
let managed_ids: std::collections::HashSet<String> = cfg || v.threads != cfg.cmf.threads
.cmf || v.gpu != cfg.cmf.gpu
.effective_servers() || v.servers != cfg.cmf.servers
.into_iter() {
.map(|s| s.id) needs_restart = true;
.collect();
cfg.models.retain(|m| {
!(managed_ids.contains(&m.id)
&& (m.base_url.contains("//127.0.0.1:") || m.base_url.contains("//localhost:")))
});
// Replacing or renaming a managed model must not brick the save: routing
// may still reference the old id. Map a vanished id to its replacement
// (the new server on the same port), strip ids that are truly gone, and
// repair the default so validate() passes.
heal_routing_after_cmf_change(&mut cfg, &old_servers);
// Server-list changes apply LIVE after a successful reload: removed /
// redefined servers are stopped (RAM freed), new ones are spawned.
if cfg.cmf.effective_servers() != old_servers {
cmf_apply = Some((old_servers, cfg.cmf.clone()));
} }
cfg.cmf = v;
} }
state.reload(cfg)?; state.reload(cfg)?;
let applied_live = cmf_apply.is_some(); ok(json!({ "ok": true, "needs_restart": needs_restart }))
if let Some((old, new_cfg)) = cmf_apply {
tokio::spawn(crate::cmf_runtime::apply_server_changes(
state.cmf.clone(),
old,
new_cfg,
));
}
ok(json!({ "ok": true, "needs_restart": needs_restart, "applied_live": applied_live }))
}
/// After the managed-servers list changed, rewrite [routing] so it only
/// references ids that still exist. A removed server whose port is now held by
/// a NEW server is treated as a replacement (old id → new id); other vanished
/// ids are stripped. An emptied default falls back to the first managed server,
/// then the first pool model.
fn heal_routing_after_cmf_change(cfg: &mut Config, old_servers: &[crate::config::CmfServer]) {
let servers = cfg.cmf.effective_servers();
let known: std::collections::HashSet<String> = cfg
.models
.iter()
.map(|m| m.id.clone())
.chain(servers.iter().map(|s| s.id.clone()))
.collect();
let new_by_port: HashMap<u16, String> =
servers.iter().map(|s| (s.port, s.id.clone())).collect();
let mut rename: HashMap<String, String> = HashMap::new();
for old in old_servers {
if !known.contains(&old.id) {
if let Some(new_id) = new_by_port.get(&old.port) {
rename.insert(old.id.clone(), new_id.clone());
}
}
}
let fix = |id: &str| -> Option<String> {
if known.contains(id) {
Some(id.to_string())
} else {
rename.get(id).cloned()
}
};
if !cfg.routing.default.is_empty() {
cfg.routing.default = fix(&cfg.routing.default).unwrap_or_default();
}
if cfg.routing.default.is_empty() {
cfg.routing.default = servers
.first()
.map(|s| s.id.clone())
.or_else(|| cfg.models.first().map(|m| m.id.clone()))
.unwrap_or_default();
}
for tt in cfg.routing.tiers.values_mut() {
let list = match tt {
TierTargets::List(v) => v.clone(),
TierTargets::One(s) => vec![s.clone()],
};
let mut healed: Vec<String> = Vec::new();
for id in &list {
if let Some(new_id) = fix(id) {
if !healed.contains(&new_id) {
healed.push(new_id);
}
}
}
*tt = TierTargets::List(healed);
}
} }
// ───────────────────────────── cmf runtime ───────────────────────────────── // ───────────────────────────── cmf runtime ─────────────────────────────────
@@ -1025,7 +931,6 @@ async fn cmf_status(State(state): State<SharedState>) -> ApiResult {
"port": s.port, "port": s.port,
"threads": s.threads, "threads": s.threads,
"gpu": s.gpu, "gpu": s.gpu,
"o1": s.o1,
"running": live.map(|l| l.running).unwrap_or(false), "running": live.map(|l| l.running).unwrap_or(false),
"healthy": live.map(|l| l.healthy).unwrap_or(false), "healthy": live.map(|l| l.healthy).unwrap_or(false),
"last_error": live.and_then(|l| l.last_error.clone()), "last_error": live.and_then(|l| l.last_error.clone()),
@@ -1123,9 +1028,6 @@ async fn cmf_port_check(State(state): State<SharedState>, Query(q): Query<PortQu
ok(json!({ ok(json!({
"port": port, "port": port,
"available": available, "available": available,
// true = held by this gateway's own managed server (a replaced/edited
// row frees it on restart) — the UI shows this as info, not an error
"managed": running_here,
"detail": detail, "detail": detail,
"suggested": suggested, "suggested": suggested,
})) }))
@@ -1402,21 +1304,47 @@ async fn register_import(State(state): State<SharedState>, Path(job): Path<Strin
.unwrap_or_else(|| "cmf-local".into()); .unwrap_or_else(|| "cmf-local".into());
let id = format!("cmf-{base}"); let id = format!("cmf-{base}");
let mut cfg = current_cfg(&state); let mut cfg = current_cfg(&state);
if cfg.models.iter().any(|m| m.id == id) {
return Err(ApiError::bad(format!("model '{id}' already registered"))); let is_in_models = cfg.models.iter().any(|m| m.id == id);
let is_in_servers = cfg
.cmf
.servers
.iter()
.any(|s| s.id == id || s.model == j.output);
if is_in_models || is_in_servers {
return Err(ApiError::bad(format!("model '{id}' is already registered")));
}
if cfg.cmf.manage_server {
let host = if cfg.cmf.local_host.trim().is_empty() {
"127.0.0.1"
} else {
&cfg.cmf.local_host
};
let base_port = cfg.cmf.servers.iter().map(|s| s.port).max().unwrap_or(8089);
let port = next_free_port(host, base_port).unwrap_or_else(|| base_port + 1);
cfg.cmf.servers.push(crate::config::CmfServer {
id: id.clone(),
model: j.output.clone(),
port,
threads: cfg.cmf.threads,
gpu: cfg.cmf.gpu,
});
} else {
cfg.models.push(crate::config::ModelCfg {
id: id.clone(),
provider: "openai".into(),
base_url: cfg.cmf.cortiq_server_url.clone(),
model: base,
cost_tier: "local".into(),
price_in: 0.0,
price_out: 0.0,
kind: "chat".into(),
api_key_env: None,
caps: Vec::new(),
});
} }
cfg.models.push(crate::config::ModelCfg {
id: id.clone(),
provider: "openai".into(),
base_url: cfg.cmf.cortiq_server_url.clone(),
model: base,
cost_tier: "local".into(),
price_in: 0.0,
price_out: 0.0,
kind: "chat".into(),
api_key_env: None,
caps: Vec::new(),
});
state.reload(cfg)?; state.reload(cfg)?;
ok(json!({ "ok": true, "model_id": id })) ok(json!({ "ok": true, "model_id": id }))
} }
-86
View File
@@ -173,31 +173,6 @@ impl CmfRuntime {
} }
} }
/// Ensure a status slot exists for `server` (used when servers are added at
/// runtime); refreshes model/port if the slot already exists.
fn add_server_slot(&self, server: &CmfServer, host: &str) {
let mut s = self.status.lock().unwrap();
s.enabled = true;
if let Some(sv) = s.servers.iter_mut().find(|x| x.id == server.id) {
sv.model = server.model.clone();
sv.port = server.port;
sv.base_url = format!("http://{host}:{}/v1", server.port);
sv.running = false;
sv.healthy = false;
sv.last_error = None;
} else {
s.servers.push(CmfServerStatus {
id: server.id.clone(),
model: server.model.clone(),
port: server.port,
base_url: format!("http://{host}:{}/v1", server.port),
running: false,
healthy: false,
last_error: None,
});
}
}
/// Stop and forget a single managed server (used when a model is deleted). /// Stop and forget a single managed server (used when a model is deleted).
pub async fn stop_one(&self, id: &str) { pub async fn stop_one(&self, id: &str) {
let removed = { let removed = {
@@ -320,11 +295,6 @@ fn spawn_serve(server: &CmfServer, cfg: &CmfCfg) -> Result<Child, String> {
if server.gpu { if server.gpu {
cmd.env("CMF_GPU", "1"); cmd.env("CMF_GPU", "1");
} }
// Serve-time O(1) attention override: "off" forces exact attention (undoes a
// too-aggressive --o1 conversion hint without re-converting the file).
if !server.o1.trim().is_empty() {
cmd.env("CMF_O1", server.o1.trim());
}
cmd.stdout(Stdio::null()) cmd.stdout(Stdio::null())
.stderr(Stdio::null()) .stderr(Stdio::null())
.kill_on_drop(true) .kill_on_drop(true)
@@ -352,62 +322,6 @@ async fn wait_healthy(host: &str, port: u16, tries: u32) -> bool {
false false
} }
/// Apply a changed managed-servers list LIVE (no gateway restart): stop servers
/// that were removed or redefined — freeing their memory immediately — and
/// spawn servers that are new or redefined.
pub async fn apply_server_changes(rt: Arc<CmfRuntime>, old: Vec<CmfServer>, cfg: CmfCfg) {
let new = cfg.effective_servers();
// 1. Stop what's gone or changed (kills the child → RAM released).
for o in &old {
let unchanged = new.iter().any(|n| n == o);
if !unchanged {
rt.stop_one(&o.id).await;
rt.log(format!("stopped '{}' (removed or changed)", o.id));
}
}
if new.is_empty() {
return;
}
// 2. Spawn what's new or changed.
if installed_version(&cfg.cortiq_bin).is_none() {
rt.fail("cortiq binary not found — cannot start local model servers");
return;
}
for n in &new {
let unchanged = old.iter().any(|o| o == n);
if unchanged {
continue; // already running with the same definition
}
rt.add_server_slot(n, &cfg.local_host);
if !std::path::Path::new(&n.model).exists() {
rt.fail_server(&n.id, format!("model file not found: {}", n.model));
continue;
}
rt.log(format!(
"starting: cortiq serve {} --host {} --port {}",
n.model, cfg.local_host, n.port
));
match spawn_serve(n, &cfg) {
Ok(child) => rt.add_child(n.id.clone(), child),
Err(e) => {
rt.fail_server(&n.id, format!("failed to spawn: {e}"));
continue;
}
}
let (rtc, host, id, port) = (rt.clone(), cfg.local_host.clone(), n.id.clone(), n.port);
tokio::spawn(async move {
if wait_healthy(&host, port, 240).await {
rtc.set_healthy(&id, true);
rtc.log(format!(
"local CMF server ready — '{id}' at http://{host}:{port}/v1"
));
} else {
rtc.fail_server(&id, "did not become healthy in time");
}
});
}
}
/// Orchestrate the whole lifecycle: ensure installed → maybe update → spawn each /// Orchestrate the whole lifecycle: ensure installed → maybe update → spawn each
/// configured local model → wait healthy. Safe to call in a background task; it /// configured local model → wait healthy. Safe to call in a background task; it
/// never panics. /// never panics.
+1 -36
View File
@@ -83,26 +83,6 @@ pub struct CmfServer {
/// Offload matvecs to the GPU (CMF_GPU=1). /// Offload matvecs to the GPU (CMF_GPU=1).
#[serde(default)] #[serde(default)]
pub gpu: bool, pub gpu: bool,
/// Serve-time O(1) attention override (exported as CMF_O1). Empty = follow
/// the hint recorded in the .cmf; "off" = force exact attention (fixes a
/// too-aggressive conversion without re-converting); "all"/"deepN"/"i,j,k"
/// = force-enable on those layers.
#[serde(default)]
pub o1: String,
/// Default generation params for this model — applied by the gateway when
/// the incoming request does not set them (an explicit request value wins).
#[serde(default, skip_serializing_if = "Option::is_none")]
pub temperature: Option<f32>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub top_p: Option<f32>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub max_tokens: Option<u32>,
/// Reasoning switch for thinking models (Qwen3/3.5 …): "off" injects
/// enable_thinking=false into requests that don't set it themselves —
/// the model answers directly, skipping the <think> phase. Needs
/// cortiq ≥ 0.2.1. Empty = model/template default.
#[serde(default)]
pub thinking: String,
} }
/// CMF-format model factory: import a HuggingFace model, convert it to a /// CMF-format model factory: import a HuggingFace model, convert it to a
@@ -192,11 +172,6 @@ impl CmfCfg {
port: self.local_port, port: self.local_port,
threads: self.threads, threads: self.threads,
gpu: self.gpu, gpu: self.gpu,
o1: String::new(),
temperature: None,
top_p: None,
max_tokens: None,
thinking: String::new(),
}]; }];
} }
Vec::new() Vec::new()
@@ -700,19 +675,9 @@ impl Config {
self.models.iter().map(|m| m.id.as_str()).collect(); self.models.iter().map(|m| m.id.as_str()).collect();
// managed local servers are known ids too; also enforce unique ports/ids // managed local servers are known ids too; also enforce unique ports/ids
let mut ports = std::collections::HashSet::new(); let mut ports = std::collections::HashSet::new();
let mut managed_seen = std::collections::HashSet::new();
for s in &servers { for s in &servers {
if !managed_seen.insert(s.id.as_str()) {
anyhow::bail!(
"config: two managed local models share the id '{}' — give one of the rows a different id",
s.id
);
}
if !ids.insert(s.id.as_str()) { if !ids.insert(s.id.as_str()) {
anyhow::bail!( anyhow::bail!("config: duplicate managed model id '{}'", s.id);
"config: managed local model '{}' has the same id as a pool model — rename the row or delete the pool entry",
s.id
);
} }
if !ports.insert(s.port) { if !ports.insert(s.port) {
anyhow::bail!("config: two managed models share port {}", s.port); anyhow::bail!("config: two managed models share port {}", s.port);
+224 -3
View File
@@ -160,6 +160,14 @@ impl JobStore {
} }
} }
} }
fn update_quant(&self, id: &str, quant: String) {
if let Some(j) = self.jobs.lock().unwrap().get_mut(id) {
j.quant = quant.clone();
if !j.log.is_empty() && j.log[0].starts_with("→ downloading ready .cmf model") {
j.log[0] = format!("→ downloading ready .cmf model {} ({})", j.repo, quant);
}
}
}
fn set_state(&self, id: &str, state: &str) { fn set_state(&self, id: &str, state: &str) {
if let Some(j) = self.jobs.lock().unwrap().get_mut(id) { if let Some(j) = self.jobs.lock().unwrap().get_mut(id) {
j.state = state.into(); j.state = state.into();
@@ -260,14 +268,166 @@ fn sanitize(s: &str) -> String {
/// Map the UI quant label to the native `cortiq` quant flag. /// Map the UI quant label to the native `cortiq` quant flag.
fn map_quant(q: &str) -> &'static str { fn map_quant(q: &str) -> &'static str {
match q.to_ascii_uppercase().as_str() { match q.to_ascii_uppercase().as_str() {
s if s.starts_with("Q4T") || s == "Q4_TILED" => "q4t",
s if s.starts_with("Q4") => "q4", s if s.starts_with("Q4") => "q4",
"Q8_2F" => "q8_2f", "Q8_2F" | "Q82F" => "q8_2f",
"F16" | "FP16" => "f16", "F16" | "FP16" => "f16",
"VBIT" => "vbit", "VBIT" => "vbit",
"Q1" => "q1",
"Q1P" | "Q1_PTQ" => "q1p",
"Q1S" | "Q1_MASK" => "q1s",
"Q1T" | "Q1_TERNARY" => "q1t",
_ => "q8", _ => "q8",
} }
} }
fn is_cmf_repo(repo: &str) -> bool {
let lower = repo.to_ascii_lowercase();
lower.contains("cmf") || lower.ends_with(".cmf")
}
async fn download_cmf_repo(
repo: String,
output_abs: String,
hf_token: Option<String>,
store: Arc<JobStore>,
id: String,
cancel: Arc<Notify>,
) {
store.set_progress(&id, 0.0, "listing files".into());
let tree_url = format!("https://huggingface.co/api/models/{repo}/tree/main?recursive=true");
let client = reqwest::Client::new();
let mut req = client.get(&tree_url).header("User-Agent", "cortiq-gateway");
if let Some(t) = &hf_token {
req = req.bearer_auth(t);
}
let mut cmf_rel_path = None;
if let Ok(resp) = req.send().await {
if resp.status().is_success() {
if let Ok(json) = resp.json::<serde_json::Value>().await {
if let Some(arr) = json.as_array() {
for item in arr {
if let Some(path) = item.get("path").and_then(|p| p.as_str()) {
if path.to_ascii_lowercase().ends_with(".cmf") {
cmf_rel_path = Some(path.to_string());
break;
}
}
}
}
}
}
}
let cmf_path = match cmf_rel_path {
Some(p) => p,
None => {
let repo_name = repo.split('/').last().unwrap_or(&repo);
format!("{repo_name}.cmf")
}
};
let path_lower = cmf_path.to_ascii_lowercase();
let detected_quant =
if path_lower.contains("-q1t.") || repo.to_ascii_lowercase().contains("2bit") {
"Q1T".to_string()
} else if path_lower.contains("-q1.") || repo.to_ascii_lowercase().contains("cmf") {
"Q1".to_string()
} else if path_lower.contains("-q4.") {
"Q4".to_string()
} else if path_lower.contains("-q8.") {
"Q8".to_string()
} else {
"READY_CMF".to_string()
};
store.update_quant(&id, detected_quant);
store.push_line(&id, format!("→ downloading {cmf_path} from HuggingFace..."));
store.set_progress(&id, 0.05, "downloading".into());
let download_url = format!("https://huggingface.co/{repo}/resolve/main/{cmf_path}");
let mut cmd = tokio::process::Command::new("curl");
cmd.arg("-f").arg("-L").arg("-#").arg("-o").arg(&output_abs);
if let Some(t) = &hf_token {
cmd.arg("-H").arg(format!("Authorization: Bearer {t}"));
}
cmd.arg(&download_url);
cmd.stderr(std::process::Stdio::piped());
cmd.stdout(std::process::Stdio::null());
let mut child = match cmd.spawn() {
Ok(c) => c,
Err(e) => {
store.push_line(&id, format!("✗ failed to start curl download: {e}"));
store.finish(&id, false);
store.drop_cancel(&id);
return;
}
};
let stderr = child.stderr.take();
let mut t_err = None;
if let Some(err_stream) = stderr {
let store_clone = store.clone();
let id_clone = id.clone();
t_err = Some(tokio::spawn(async move {
use tokio::io::AsyncReadExt;
let mut reader = err_stream;
let mut buf = [0u8; 256];
while let Ok(n) = reader.read(&mut buf).await {
if n == 0 {
break;
}
let text = String::from_utf8_lossy(&buf[..n]);
for word in text.split_whitespace() {
if let Some(num_str) = word.strip_suffix('%') {
if let Ok(pct) = num_str.parse::<f32>() {
let frac = (pct / 100.0).clamp(0.0, 1.0);
store_clone.set_progress(&id_clone, frac, "downloading".into());
}
}
}
}
}));
}
tokio::select! {
status_res = child.wait() => {
if let Some(t) = t_err { t.abort(); }
match status_res {
Ok(status) if status.success() => {
store.push_line(&id, "✓ done".into());
store.finish(&id, true);
}
Ok(status) => {
cleanup_output(&output_abs);
store.push_line(&id, format!("✗ download failed: exit code {:?}", status.code()));
store.finish(&id, false);
}
Err(e) => {
cleanup_output(&output_abs);
store.push_line(&id, format!("✗ download process error: {e}"));
store.finish(&id, false);
}
}
}
_ = cancel.notified() => {
let _ = child.start_kill();
let _ = child.wait().await;
if let Some(t) = t_err { t.abort(); }
cleanup_output(&output_abs);
store.push_line(&id, "✗ cancelled by user — partial output removed".into());
store.set_state(&id, "cancelled");
}
}
store.drop_cancel(&id);
}
/// into the job log. Fails fast if the converter script is missing. /// into the job log. Fails fast if the converter script is missing.
pub fn start_import(store: Arc<JobStore>, cfg: &CmfCfg, p: ImportParams) -> Result<String, String> { pub fn start_import(store: Arc<JobStore>, cfg: &CmfCfg, p: ImportParams) -> Result<String, String> {
if p.repo.trim().is_empty() { if p.repo.trim().is_empty() {
@@ -326,13 +486,37 @@ pub fn start_import(store: Arc<JobStore>, cfg: &CmfCfg, p: ImportParams) -> Resu
.unwrap_or_else(|_| output.clone()); .unwrap_or_else(|_| output.clone());
let id = gen_id(&p.repo); let id = gen_id(&p.repo);
let is_cmf = is_cmf_repo(&p.repo);
let quant_display = if is_cmf {
let r = p.repo.to_lowercase();
if r.contains("2bit") || r.contains("q1t") {
"Q1T".to_string()
} else if r.contains("1.7bcmf") || r.contains("27bcmf") || r.contains("cmf") {
"Q1".to_string()
} else if p.quant.is_empty() || p.quant == "auto" || p.quant == "Q8_2F" {
"Q1".to_string()
} else {
p.quant.clone()
}
} else if p.quant.is_empty() || p.quant == "auto" {
"q8".to_string()
} else {
p.quant.clone()
};
store.insert(Job { store.insert(Job {
id: id.clone(), id: id.clone(),
repo: p.repo.clone(), repo: p.repo.clone(),
quant: p.quant.clone(), quant: quant_display.clone(),
output: output_abs.clone(), output: output_abs.clone(),
state: "running".into(), state: "running".into(),
log: vec![format!("→ converting {} to {} ({})", p.repo, base, p.quant)], log: if is_cmf {
vec![format!(
"→ downloading ready .cmf model {} ({})",
p.repo, quant_display
)]
} else {
vec![format!("→ converting {} to {} ({})", p.repo, base, p.quant)]
},
started: now(), started: now(),
finished: None, finished: None,
size_bytes: None, size_bytes: None,
@@ -348,6 +532,14 @@ pub fn start_import(store: Arc<JobStore>, cfg: &CmfCfg, p: ImportParams) -> Resu
std::env::var(&cfg.hf_token_env).ok() std::env::var(&cfg.hf_token_env).ok()
}; };
if is_cmf && !use_python {
let ret_id = id.clone();
tokio::spawn(async move {
download_cmf_repo(p.repo, output_abs, hf_token, store, id, cancel).await;
});
return Ok(ret_id);
}
// Program + args: `cortiq convert` by default, or the Python converter for // Program + args: `cortiq convert` by default, or the Python converter for
// advanced options. Both stream `@PROGRESS <frac>` markers into the log. // advanced options. Both stream `@PROGRESS <frac>` markers into the log.
let (program, args, workdir): (String, Vec<String>, std::path::PathBuf) = if use_python { let (program, args, workdir): (String, Vec<String>, std::path::PathBuf) = if use_python {
@@ -530,3 +722,32 @@ pub fn start_import(store: Arc<JobStore>, cfg: &CmfCfg, p: ImportParams) -> Resu
Ok(ret_id) Ok(ret_id)
} }
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_map_quant_variants() {
assert_eq!(map_quant("Q8_2F"), "q8_2f");
assert_eq!(map_quant("q82f"), "q8_2f");
assert_eq!(map_quant("Q8_ROW"), "q8");
assert_eq!(map_quant("Q4_BLOCK"), "q4");
assert_eq!(map_quant("Q4_TILED"), "q4t");
assert_eq!(map_quant("q4t"), "q4t");
assert_eq!(map_quant("vbit"), "vbit");
assert_eq!(map_quant("q1"), "q1");
assert_eq!(map_quant("Q1P"), "q1p");
assert_eq!(map_quant("q1s"), "q1s");
assert_eq!(map_quant("Q1T"), "q1t");
assert_eq!(map_quant("F16"), "f16");
}
#[test]
fn test_is_cmf_repo_check() {
assert!(is_cmf_repo("infosave/Bonsai-8B_2bit_cmf"));
assert!(is_cmf_repo("infosave/Bonsai-1.7Bcmf"));
assert!(is_cmf_repo("user/model.cmf"));
assert!(!is_cmf_repo("Qwen/Qwen2.5-0.5B-Instruct"));
}
}
+2 -35
View File
@@ -301,35 +301,6 @@ impl Pipeline {
Ok((decision, route_source, final_candidates)) Ok((decision, route_source, final_candidates))
} }
/// Fill in the model's default generation params (from its Local-models row)
/// for anything the request left unset — an explicit request value wins.
fn apply_gen_defaults(req: &mut ChatRequest, cfg: &crate::config::Config, model_id: &str) {
if let Some(s) = cfg
.cmf
.effective_servers()
.into_iter()
.find(|s| s.id == model_id)
{
if req.params.temperature.is_none() {
req.params.temperature = s.temperature;
}
if req.params.top_p.is_none() {
req.params.top_p = s.top_p;
}
if req.params.max_tokens.is_none() {
req.params.max_tokens = s.max_tokens;
}
// reasoning switch: "off" → enable_thinking=false (cortiq ≥ 0.2.1
// renders the chat template with an empty <think> block); a client
// that sets enable_thinking itself wins.
if s.thinking == "off" && !req.params.passthrough.contains_key("enable_thinking") {
req.params
.passthrough
.insert("enable_thinking".into(), serde_json::Value::Bool(false));
}
}
}
async fn run_inner(&self, req: ChatRequest, state: &SharedState) -> Result<ChatResponse> { async fn run_inner(&self, req: ChatRequest, state: &SharedState) -> Result<ChatResponse> {
let live = state.live(); let live = state.live();
let (decision, route_source, mut final_candidates) = self.resolve(&req, &live).await?; let (decision, route_source, mut final_candidates) = self.resolve(&req, &live).await?;
@@ -370,9 +341,7 @@ impl Pipeline {
route_source route_source
); );
let mut attempt_req = req.clone(); match provider.chat(req.clone()).await {
Self::apply_gen_defaults(&mut attempt_req, &live.cfg, &model_id);
match provider.chat(attempt_req).await {
Ok(mut resp) => { Ok(mut resp) => {
let cost = let cost =
provider.price(resp.usage.prompt_tokens, resp.usage.completion_tokens); provider.price(resp.usage.prompt_tokens, resp.usage.completion_tokens);
@@ -446,9 +415,7 @@ impl Pipeline {
model_id, model_id,
route_source route_source
); );
let mut attempt_req = req.clone(); match provider.chat_stream(req.clone()).await {
Self::apply_gen_defaults(&mut attempt_req, &live.cfg, &model_id);
match provider.chat_stream(attempt_req).await {
Ok(stream) => { Ok(stream) => {
let info = RouteInfo { let info = RouteInfo {
task_label: decision.task_label.clone(), task_label: decision.task_label.clone(),
-3
View File
@@ -45,9 +45,6 @@ impl OpenAiProvider {
if let Some(m) = req.params.max_tokens { if let Some(m) = req.params.max_tokens {
body["max_tokens"] = m.into(); body["max_tokens"] = m.into();
} }
if let Some(p) = req.params.top_p {
body["top_p"] = p.into();
}
if !req.tools.is_empty() { if !req.tools.is_empty() {
body["tools"] = serde_json::Value::Array(req.tools.clone()); body["tools"] = serde_json::Value::Array(req.tools.clone());
} }
+8
View File
@@ -145,6 +145,14 @@ impl AppState {
let live = Live::build(new_cfg.clone(), &self.secrets, self.router_status.clone())?; let live = Live::build(new_cfg.clone(), &self.secrets, self.router_status.clone())?;
new_cfg.save(&self.config_path)?; new_cfg.save(&self.config_path)?;
self.live.store(Arc::new(live)); self.live.store(Arc::new(live));
let cmf_rt = self.cmf.clone();
let cmf_cfg = new_cfg.cmf.clone();
tokio::spawn(async move {
cmf_rt.stop().await;
crate::cmf_runtime::manage(cmf_rt, cmf_cfg).await;
});
Ok(()) Ok(())
} }
+1 -2
View File
@@ -60,8 +60,7 @@ export const api = {
listModels: () => j("GET", "/models"), listModels: () => j("GET", "/models"),
createModel: (b) => j("POST", "/models", b), createModel: (b) => j("POST", "/models", b),
updateModel: (id, b) => j("PUT", "/models/" + encodeURIComponent(id), b), updateModel: (id, b) => j("PUT", "/models/" + encodeURIComponent(id), b),
deleteModel: (id, deleteFile = true) => deleteModel: (id) => j("DELETE", "/models/" + encodeURIComponent(id)),
j("DELETE", "/models/" + encodeURIComponent(id) + "?delete_file=" + (deleteFile ? "true" : "false")),
probeModel: (id) => j("POST", "/models/" + encodeURIComponent(id) + "/probe"), probeModel: (id) => j("POST", "/models/" + encodeURIComponent(id) + "/probe"),
// routing // routing
+11 -19
View File
@@ -132,8 +132,6 @@ export default {
"models.form.secretMissing": "not set", "models.form.secretMissing": "not set",
"models.form.secretPlaceholder": "set or replace key (write-only)", "models.form.secretPlaceholder": "set or replace key (write-only)",
"models.deleteConfirm": "Delete model \"{id}\"?", "models.deleteConfirm": "Delete model \"{id}\"?",
"models.deleteFile": "also delete the model file from disk",
"models.deletedWithFile": "Deleted (file removed from disk)",
"models.form.idOrModel": "Enter an id or a model name", "models.form.idOrModel": "Enter an id or a model name",
"models.form.apiKey": "API Key", "models.form.apiKey": "API Key",
"models.form.apiKeyPlaceholder": "paste your API key", "models.form.apiKeyPlaceholder": "paste your API key",
@@ -264,20 +262,11 @@ export default {
"settings.cmf.pick": "select a model", "settings.cmf.pick": "select a model",
"settings.cmf.models": "Local models", "settings.cmf.models": "Local models",
"settings.cmf.modelsHelp": "Each row is one model the gateway runs (its own cortiq serve on its own port) and registers in the pool. Add as many as you have .cmf files and free ports.", "settings.cmf.modelsHelp": "Each row is one model the gateway runs (its own cortiq serve on its own port) and registers in the pool. Add as many as you have .cmf files and free ports.",
"settings.cmf.rowHint": "id · .cmf file · port · threads · GPU · O(1) · T°/top-p/max tok. Applied on save: a removed row stops its server (memory freed), a new row starts one.", "settings.cmf.rowHint": "id · .cmf file · port · threads · GPU. Changes apply on restart; deleting a row stops that server.",
"settings.cmf.thinkAuto": "thoughts: auto",
"settings.cmf.thinkOff": "thoughts: off",
"settings.cmf.thinkHelp": "Reasoning models (Qwen3/3.5) write out a thinking phase before the answer. Off = answer directly without it (faster, shorter; needs cortiq >= 0.2.1). A request that sets enable_thinking itself always wins.",
"settings.cmf.tempHelp": "Default temperature for this model (0 = deterministic, 0.7 = balanced, >1 = creative). Used when the request doesn't set one; a request value always wins. Empty = provider default.",
"settings.cmf.topPHelp": "Default nucleus sampling (top-p) for this model: only tokens within this cumulative probability are sampled (0.9 typical). Empty = provider default.",
"settings.cmf.maxTokHelp": "Default max output tokens for this model when the request doesn't set one. Protects against runaway long generations. Empty = provider default.",
"settings.cmf.o1Auto": "O(1): auto",
"settings.cmf.o1Help": "O(1) attention at serve time: auto — follow the hint recorded in the .cmf; off — force exact attention (fixes incoherent output after a too-aggressive --o1 conversion, no re-convert needed); all/deepN — force-enable on those layers.",
"settings.cmf.addServer": "+ Add local model", "settings.cmf.addServer": "+ Add local model",
"settings.cmf.checkPort": "Check port", "settings.cmf.checkPort": "Check port",
"settings.cmf.portFree": "port is free", "settings.cmf.portFree": "port is free",
"settings.cmf.portBusy": "port is in use", "settings.cmf.portBusy": "port is in use",
"settings.cmf.portOwn": "in use by this gateway's current server — freed on restart, replacing is fine",
"settings.cmf.useSuggested": "use {port}", "settings.cmf.useSuggested": "use {port}",
"settings.cmf.serving": "serving {model} on :{port}", "settings.cmf.serving": "serving {model} on :{port}",
"settings.cmf.intro": "Run models locally with the cortiq engine — works even offline or when the router is down.", "settings.cmf.intro": "Run models locally with the cortiq engine — works even offline or when the router is down.",
@@ -385,7 +374,7 @@ export default {
"import.failed": "Conversion failed — see the log.", "import.failed": "Conversion failed — see the log.",
"import.register": "Register as local model", "import.register": "Register as local model",
"import.registered": "Registered as '{id}'. Run cortiq-server to serve it.", "import.registered": "Registered as '{id}'. Run cortiq-server to serve it.",
"import.jobs": "Conversions", "import.jobs": "Downloads & Conversions",
"import.cancel": "Cancel", "import.cancel": "Cancel",
"import.cancelling": "Cancelling — cleaning up partial output…", "import.cancelling": "Cancelling — cleaning up partial output…",
"import.delete": "Delete", "import.delete": "Delete",
@@ -395,12 +384,21 @@ export default {
"import.state.done": "done", "import.state.done": "done",
"import.state.error": "error", "import.state.error": "error",
"import.state.cancelled": "cancelled", "import.state.cancelled": "cancelled",
"import.featuredTitle": "Featured Ready-to-Run .cmf Models",
"import.readyCmfBadge": "Ready .cmf",
"import.directDownload": "Download",
"import.q.q8_2f": "8-bit two-field (𝒲×θ) — recommended: best quality/size.", "import.q.q8_2f": "8-bit two-field (𝒲×θ) — recommended: best quality/size.",
"import.q.q8_row": "8-bit per-row — simple, robust.", "import.q.q8_row": "8-bit per-row — simple, robust.",
"import.q.q4": "4-bit block — smallest, lower quality.", "import.q.q4": "4-bit block — smallest, lower quality.",
"import.q.q4_tiled": "4-bit tiled (Q4_TILED) — 1.66x faster on Metal GPU.",
"import.q.vbit": "Variable 3–8 bit — per-expert bit budgets.", "import.q.vbit": "Variable 3–8 bit — per-expert bit budgets.",
"import.q.q1t": "1.58-bit ternary (Q1T) — BitNet b1.58 with sparse outliers.",
"import.q.q1s": "1-bit sparse (Q1S) — 1-bit base with holographic outliers.",
"import.q.q1p": "1-bit PTQ (Q1P) — 1-bit with error diffusion.",
"import.q.q1": "1-bit (Q1) — pure binary format.",
"import.q.f16": "16-bit — no quantization.", "import.q.f16": "16-bit — no quantization.",
"import.q.f32": "32-bit — full precision (large file).", "import.q.f32": "32-bit — full precision (large file).",
"import.adv.title": "Advanced settings", "import.adv.title": "Advanced settings",
"import.adv.auto": "auto", "import.adv.auto": "auto",
"import.adv.linearCore": "Linear core", "import.adv.linearCore": "Linear core",
@@ -420,12 +418,6 @@ export default {
"import.adv.o1DeepN": "N deep layers", "import.adv.o1DeepN": "N deep layers",
"import.adv.o1Layers": "Layers (comma-separated)", "import.adv.o1Layers": "Layers (comma-separated)",
"import.adv.o1mHint": "landmark budget (default 32)", "import.adv.o1mHint": "landmark budget (default 32)",
"import.adv.o1mLabel": "Landmarks (--o1-m)",
"import.adv.o1mHelp": "How many landmark tokens summarize the whole older history when attention is approximated. More = closer to exact attention but slower and more memory; fewer = faster but rougher. 32 is the validated sweet spot — change only if you know why.",
"import.adv.o1windowLabel": "Exact window (--o1-window)",
"import.adv.o1windowHelp": "The last N tokens always get exact (non-approximated) attention. This protects quality on the text the model just wrote/read. Bigger = better local quality, more memory. Default 128.",
"import.adv.o1sinkLabel": "Sink keys (--o1-sink)",
"import.adv.o1sinkHelp": "The first N tokens of the context are kept exact forever ('attention sinks'). Models lean on these anchors — keeping them exact stabilizes long generations. Default 4; rarely needs changing.",
"import.adv.o1windowHint": "exact-window width (default 128)", "import.adv.o1windowHint": "exact-window width (default 128)",
"import.adv.o1sinkHint": "permanent exact sink keys (default 4)", "import.adv.o1sinkHint": "permanent exact sink keys (default 4)",
}; };
+11 -19
View File
@@ -131,8 +131,6 @@ export default {
"models.form.secretMissing": "не задан", "models.form.secretMissing": "не задан",
"models.form.secretPlaceholder": "задать или заменить ключ (только запись)", "models.form.secretPlaceholder": "задать или заменить ключ (только запись)",
"models.deleteConfirm": "Удалить модель \"{id}\"?", "models.deleteConfirm": "Удалить модель \"{id}\"?",
"models.deleteFile": "также удалить файл модели с диска",
"models.deletedWithFile": "Удалено (файл стёрт с диска)",
"models.form.idOrModel": "Укажите id или имя модели", "models.form.idOrModel": "Укажите id или имя модели",
"models.form.apiKey": "API-ключ", "models.form.apiKey": "API-ключ",
"models.form.apiKeyPlaceholder": "вставьте API-ключ", "models.form.apiKeyPlaceholder": "вставьте API-ключ",
@@ -263,20 +261,11 @@ export default {
"settings.cmf.pick": "выберите модель", "settings.cmf.pick": "выберите модель",
"settings.cmf.models": "Локальные модели", "settings.cmf.models": "Локальные модели",
"settings.cmf.modelsHelp": "Каждая строка — отдельная модель: шлюз поднимает свой cortiq serve на своём порту и регистрирует её в пуле. Добавляйте столько, сколько есть .cmf-файлов и свободных портов.", "settings.cmf.modelsHelp": "Каждая строка — отдельная модель: шлюз поднимает свой cortiq serve на своём порту и регистрирует её в пуле. Добавляйте столько, сколько есть .cmf-файлов и свободных портов.",
"settings.cmf.rowHint": "id · файл .cmf · порт · потоки · GPU · O(1) · T°/top-p/max tok. Применяется сразу при сохранении: удалённая строка останавливает сервер (память освобождается), новая — запускает.", "settings.cmf.rowHint": "id · файл .cmf · порт · потоки · GPU. Изменения применяются после перезапуска; удаление строки останавливает сервер.",
"settings.cmf.thinkAuto": "мысли: авто",
"settings.cmf.thinkOff": "мысли: выкл",
"settings.cmf.thinkHelp": "Рассуждающие модели (Qwen3/3.5) сначала пишут фазу размышлений. Выкл = отвечать сразу, без неё (быстрее и короче; нужен cortiq ≥ 0.2.1). Если запрос сам задал enable_thinking — он важнее.",
"settings.cmf.tempHelp": "Температура по умолчанию для этой модели (0 = детерминированно, 0.7 = баланс, >1 = креативно). Подставляется, если запрос её не задал; значение из запроса всегда важнее. Пусто = дефолт провайдера.",
"settings.cmf.topPHelp": "Nucleus-сэмплинг (top-p) по умолчанию: выбираются только токены в пределах этой суммарной вероятности (обычно 0.9). Пусто = дефолт провайдера.",
"settings.cmf.maxTokHelp": "Максимум выходных токенов по умолчанию, если запрос не задал свой. Защищает от бесконечных генераций. Пусто = дефолт провайдера.",
"settings.cmf.o1Auto": "O(1): авто",
"settings.cmf.o1Help": "O(1)-внимание при запуске: авто — использовать подсказку из .cmf; off — принудительно точное внимание (лечит бессвязный вывод после слишком агрессивной --o1 конвертации, без переконвертации); all/deepN — принудительно включить на этих слоях.",
"settings.cmf.addServer": "+ Добавить модель", "settings.cmf.addServer": "+ Добавить модель",
"settings.cmf.checkPort": "Проверить порт", "settings.cmf.checkPort": "Проверить порт",
"settings.cmf.portFree": "порт свободен", "settings.cmf.portFree": "порт свободен",
"settings.cmf.portBusy": "порт занят", "settings.cmf.portBusy": "порт занят",
"settings.cmf.portOwn": "занят текущим сервером шлюза — освободится после перезапуска, замена допустима",
"settings.cmf.useSuggested": "занять {port}", "settings.cmf.useSuggested": "занять {port}",
"settings.cmf.serving": "работает: {model} на :{port}", "settings.cmf.serving": "работает: {model} на :{port}",
"settings.cmf.intro": "Запускайте модели локально движком cortiq — работает даже офлайн или когда роутер недоступен.", "settings.cmf.intro": "Запускайте модели локально движком cortiq — работает даже офлайн или когда роутер недоступен.",
@@ -384,7 +373,7 @@ export default {
"import.failed": "Конвертация не удалась — смотрите лог.", "import.failed": "Конвертация не удалась — смотрите лог.",
"import.register": "Подключить как локальную модель", "import.register": "Подключить как локальную модель",
"import.registered": "Подключено как '{id}'. Запустите cortiq-server для обслуживания.", "import.registered": "Подключено как '{id}'. Запустите cortiq-server для обслуживания.",
"import.jobs": "Конвертации", "import.jobs": "Загрузки / Конвертации",
"import.cancel": "Отменить", "import.cancel": "Отменить",
"import.cancelling": "Отмена — удаляю недописанный файл…", "import.cancelling": "Отмена — удаляю недописанный файл…",
"import.delete": "Удалить", "import.delete": "Удалить",
@@ -394,12 +383,21 @@ export default {
"import.state.done": "готово", "import.state.done": "готово",
"import.state.error": "ошибка", "import.state.error": "ошибка",
"import.state.cancelled": "отменено", "import.state.cancelled": "отменено",
"import.featuredTitle": "Рекомендуемые готовые .cmf модели",
"import.readyCmfBadge": "Готовая .cmf",
"import.directDownload": "Скачать",
"import.q.q8_2f": "8-бит двухполевой (𝒲×θ) — рекомендуется: лучшее качество/размер.", "import.q.q8_2f": "8-бит двухполевой (𝒲×θ) — рекомендуется: лучшее качество/размер.",
"import.q.q8_row": "8-бит построчно — просто, надёжно.", "import.q.q8_row": "8-бит построчно — просто, надёжно.",
"import.q.q4": "4-бит блочно — компактнее, ниже качество.", "import.q.q4": "4-бит блочно — компактнее, ниже качество.",
"import.q.q4_tiled": "4-бит тайловый (Q4_TILED) — х1.66 быстрее на Metal GPU.",
"import.q.vbit": "Переменная 3–8 бит — per-expert бюджеты.", "import.q.vbit": "Переменная 3–8 бит — per-expert бюджеты.",
"import.q.q1t": "1.58-бит тернарный (Q1T) — BitNet b1.58 с оутлайерами.",
"import.q.q1s": "1-бит спарс (Q1S) — 1-бит база с голографическими оутлайерами.",
"import.q.q1p": "1-бит PTQ (Q1P) — 1-бит с переносом ошибки (error diffusion).",
"import.q.q1": "1-бит (Q1) — чистый бинарный формат.",
"import.q.f16": "16-бит — без квантования.", "import.q.f16": "16-бит — без квантования.",
"import.q.f32": "32-бит — полная точность (большой файл).", "import.q.f32": "32-бит — полная точность (большой файл).",
"import.adv.title": "Расширенные настройки", "import.adv.title": "Расширенные настройки",
"import.adv.auto": "авто", "import.adv.auto": "авто",
"import.adv.linearCore": "Линейное ядро", "import.adv.linearCore": "Линейное ядро",
@@ -419,12 +417,6 @@ export default {
"import.adv.o1DeepN": "N глубоких слоёв", "import.adv.o1DeepN": "N глубоких слоёв",
"import.adv.o1Layers": "Слои (через запятую)", "import.adv.o1Layers": "Слои (через запятую)",
"import.adv.o1mHint": "число опорных точек (по умолчанию 32)", "import.adv.o1mHint": "число опорных точек (по умолчанию 32)",
"import.adv.o1mLabel": "Опорные точки (--o1-m)",
"import.adv.o1mHelp": "Сколько опорных токенов представляют всю старую историю при аппроксимации внимания. Больше — ближе к точному вниманию, но медленнее и больше памяти; меньше — быстрее, но грубее. 32 — проверенное значение; меняйте, только если понимаете зачем.",
"import.adv.o1windowLabel": "Точное окно (--o1-window)",
"import.adv.o1windowHelp": "Последние N токенов всегда обрабатываются точным (не аппроксимированным) вниманием — это защищает качество на только что написанном/прочитанном тексте. Больше — лучше локальное качество, больше памяти. По умолчанию 128.",
"import.adv.o1sinkLabel": "Sink-ключи (--o1-sink)",
"import.adv.o1sinkHelp": "Первые N токенов контекста навсегда остаются точными («якоря внимания»). Модели опираются на эти якоря — их точность стабилизирует длинную генерацию. По умолчанию 4; менять почти никогда не нужно.",
"import.adv.o1windowHint": "ширина точного окна (по умолчанию 128)", "import.adv.o1windowHint": "ширина точного окна (по умолчанию 128)",
"import.adv.o1sinkHint": "постоянные точные sink-ключи (по умолчанию 4)", "import.adv.o1sinkHint": "постоянные точные sink-ключи (по умолчанию 4)",
}; };
+59 -11
View File
@@ -10,19 +10,25 @@ const QUANTS = [
["Q8_2F", "import.q.q8_2f"], ["Q8_2F", "import.q.q8_2f"],
["Q8_ROW", "import.q.q8_row"], ["Q8_ROW", "import.q.q8_row"],
["Q4_BLOCK", "import.q.q4"], ["Q4_BLOCK", "import.q.q4"],
["Q4_TILED", "import.q.q4_tiled"],
["VBIT", "import.q.vbit"], ["VBIT", "import.q.vbit"],
["Q1T", "import.q.q1t"],
["Q1S", "import.q.q1s"],
["Q1P", "import.q.q1p"],
["Q1", "import.q.q1"],
["F16", "import.q.f16"], ["F16", "import.q.f16"],
["F32", "import.q.f32"], ["F32", "import.q.f32"],
]; ];
// A small "?" badge with a hover/focus tooltip (same style as Settings). const FEATURED_REPOS = [
function help(text) { { id: "infosave/Bonsai-1.7Bcmf", label: "Bonsai 1.7B CMF", badge: "1.7B · Q1", quant: "Q1" },
return text ? h("span", { class: "help", title: text, tabindex: "0", role: "img", "aria-label": text }, "?") : null; { id: "infosave/Bonsai-8B_2bit_cmf", label: "Bonsai 8B 2-bit CMF", badge: "8B · Q1T", quant: "Q1T" },
} { id: "infosave/Bonsai-27Bcmf", label: "Bonsai 27B CMF", badge: "27B · Q1", quant: "Q1" },
function field(label, control, hint, helpText) { ];
return h("label", { class: "field" },
h("span", {}, label, helpText ? " " : null, help(helpText)),
control, function field(label, control, hint) {
return h("label", { class: "field" }, h("span", {}, label), control,
hint ? h("div", { class: "hint" }, hint) : null); hint ? h("div", { class: "hint" }, hint) : null);
} }
function opt(v, label, sel) { function opt(v, label, sel) {
@@ -147,9 +153,9 @@ export function renderImport() {
field(t("import.adv.o1DeepN"), o1DeepIn), field(t("import.adv.o1DeepN"), o1DeepIn),
field(t("import.adv.o1Layers"), o1CustomIn)); field(t("import.adv.o1Layers"), o1CustomIn));
const o1Knobs = h("div", { class: "row", style: "display:none" }, const o1Knobs = h("div", { class: "row", style: "display:none" },
field(t("import.adv.o1mLabel"), o1M, t("import.adv.o1mHint"), t("import.adv.o1mHelp")), field("--o1-m", o1M, t("import.adv.o1mHint")),
field(t("import.adv.o1windowLabel"), o1Window, t("import.adv.o1windowHint"), t("import.adv.o1windowHelp")), field("--o1-window", o1Window, t("import.adv.o1windowHint")),
field(t("import.adv.o1sinkLabel"), o1Sink, t("import.adv.o1sinkHint"), t("import.adv.o1sinkHelp"))); field("--o1-sink", o1Sink, t("import.adv.o1sinkHint")));
o1Sel.addEventListener("change", () => { o1Sel.addEventListener("change", () => {
const on = !!o1Sel.value; const on = !!o1Sel.value;
o1Knobs.style.display = on ? "" : "none"; o1Knobs.style.display = on ? "" : "none";
@@ -364,6 +370,46 @@ export function renderImport() {
); );
} }
function featuredCard(item) {
const card = h(
"div",
{ class: "hf-card featured-card" },
h("div", { class: "row", style: "align-items:center; justify-content:space-between;" },
h("div", {},
h("div", { class: "hf-id" }, item.label),
h("div", { class: "hf-meta" },
h("span", { class: "badge" }, item.badge),
h("span", { class: "muted" }, item.id)
)
),
h("button", { class: "btn primary sm" }, "⬇ " + t("import.directDownload"))
)
);
card.querySelector("button").addEventListener("click", async (e) => {
e.stopPropagation();
try {
await api.startImport({
repo: item.id,
quant: item.quant || "READY_CMF",
name: item.id.split("/").pop().toLowerCase(),
});
toast(t("import.started"), "ok");
refreshJobs();
jobsHost.scrollIntoView({ behavior: "smooth", block: "nearest" });
} catch (err) {
toast(err.message, "err");
}
});
return card;
}
const featuredHost = h(
"div",
{ class: "card featured-section" },
h("div", { class: "card-title" }, "⚡ " + t("import.featuredTitle")),
h("div", { class: "hf-grid" }, ...FEATURED_REPOS.map(featuredCard))
);
const root = h( const root = h(
"div", "div",
{}, {},
@@ -371,6 +417,7 @@ export function renderImport() {
h("h2", {}, t("nav.import")), h("h2", {}, t("nav.import")),
h("div", { class: "muted" }, t("import.subtitle"))), h("div", { class: "muted" }, t("import.subtitle"))),
jobsHost, // progress panel on top, above the search jobsHost, // progress panel on top, above the search
featuredHost, // featured ready-to-run models
h("div", { class: "card" }, searchIn, results), h("div", { class: "card" }, searchIn, results),
formHost formHost
); );
@@ -378,3 +425,4 @@ export function renderImport() {
refreshJobs(); // resume: show any running/recent conversions immediately refreshJobs(); // resume: show any running/recent conversions immediately
return root; return root;
} }
+10 -28
View File
@@ -1,5 +1,5 @@
// Models — pool table with add/edit modal, probe, and per-model secret entry. // Models — pool table with add/edit modal, probe, and per-model secret entry.
import { h, mount, modal, toast, money } from "../ui.js"; import { h, mount, modal, toast, confirmDialog, money } from "../ui.js";
import { t } from "../i18n.js"; import { t } from "../i18n.js";
import { api } from "../api.js"; import { api } from "../api.js";
import { appState } from "../app.js"; import { appState } from "../app.js";
@@ -280,33 +280,15 @@ function row(m, meta, reload) {
"button", "button",
{ {
class: "btn sm danger", class: "btn sm danger",
onClick: () => { onClick: async () => {
// local models may have a .cmf on disk — let the user decide its fate if (!(await confirmDialog(t("models.deleteConfirm", { id: m.id })))) return;
const isLocal = m.managed || /127\.0\.0\.1|localhost/.test(m.base_url || ""); try {
// unchecked by default: deleting a pool entry should not silently await api.deleteModel(m.id);
// cost the user a multi-GB re-download/conversion toast(t("toast.deleted"), "good");
const fileCb = h("input", { type: "checkbox" }); await reload();
const body = h( } catch (e) {
"div", toast(String(e.message || e), "bad");
{}, }
h("p", {}, t("models.deleteConfirm", { id: m.id })),
isLocal
? h("label", { class: "check" }, fileCb, t("models.deleteFile"))
: null
);
modal(t("common.delete") + ": " + m.id, body, async () => {
try {
const r = await api.deleteModel(m.id, isLocal && fileCb.checked);
toast(
r.file_removed ? t("models.deletedWithFile") : t("toast.deleted"),
"good"
);
await reload();
} catch (e) {
toast(String(e.message || e), "bad");
return false;
}
}, t("common.delete"));
}, },
}, },
t("common.delete") t("common.delete")
+3 -27
View File
@@ -234,20 +234,6 @@ export async function renderSettings() {
const portIn = h("input", { type: "number", value: sv.port ?? 8090, style: "width:82px" }); const portIn = h("input", { type: "number", value: sv.port ?? 8090, style: "width:82px" });
const threadsIn = h("input", { type: "number", min: 0, value: sv.threads ?? 8, style: "width:60px", title: t("settings.cmf.threadsHelp") }); const threadsIn = h("input", { type: "number", min: 0, value: sv.threads ?? 8, style: "width:60px", title: t("settings.cmf.threadsHelp") });
const gpuIn = h("input", { type: "checkbox", checked: sv.gpu ? true : null, title: t("settings.cmf.gpuHelp") }); const gpuIn = h("input", { type: "checkbox", checked: sv.gpu ? true : null, title: t("settings.cmf.gpuHelp") });
// serve-time O(1) override: off cures a too-aggressive --o1 conversion
const o1Opts = ["", "off", "all", "deep4", "deep8", "deep12"];
const o1In = h("select", { style: "width:110px", title: t("settings.cmf.o1Help") },
...o1Opts.map((v) => h("option", { value: v, selected: (sv.o1 || "") === v ? true : null },
v === "" ? t("settings.cmf.o1Auto") : v)));
if (sv.o1 && !o1Opts.includes(sv.o1)) o1In.appendChild(h("option", { value: sv.o1, selected: true }, sv.o1));
// reasoning switch for thinking models (off → enable_thinking=false)
const thinkIn = h("select", { style: "width:118px", title: t("settings.cmf.thinkHelp") },
h("option", { value: "", selected: !sv.thinking ? true : null }, t("settings.cmf.thinkAuto")),
h("option", { value: "off", selected: sv.thinking === "off" ? true : null }, t("settings.cmf.thinkOff")));
// per-model generation defaults (applied when the request doesn't set them)
const tempIn = h("input", { type: "number", step: "0.1", min: 0, max: 2, value: sv.temperature ?? "", placeholder: "T° 0.7", style: "width:74px", title: t("settings.cmf.tempHelp") });
const topPIn = h("input", { type: "number", step: "0.05", min: 0, max: 1, value: sv.top_p ?? "", placeholder: "top-p", style: "width:70px", title: t("settings.cmf.topPHelp") });
const maxTokIn = h("input", { type: "number", min: 1, value: sv.max_tokens ?? "", placeholder: "max tok", style: "width:82px", title: t("settings.cmf.maxTokHelp") });
const checkBtn = h("button", { class: "btn small", type: "button" }, t("settings.cmf.checkPort")); const checkBtn = h("button", { class: "btn small", type: "button" }, t("settings.cmf.checkPort"));
const removeBtn = h("button", { class: "btn small danger", type: "button", title: t("common.delete") }, "✕"); const removeBtn = h("button", { class: "btn small danger", type: "button", title: t("common.delete") }, "✕");
const st = h("span", { class: "small" }); const st = h("span", { class: "small" });
@@ -259,11 +245,7 @@ export async function renderSettings() {
try { try {
const r = await api.cmfPortCheck(p); const r = await api.cmfPortCheck(p);
if (r.available) mount(st, h("span", { style: "color:var(--good)" }, "✓ " + t("settings.cmf.portFree"))); if (r.available) mount(st, h("span", { style: "color:var(--good)" }, "✓ " + t("settings.cmf.portFree")));
else if (r.managed) { else {
// held by this gateway's own managed server — replacing the model on
// this port is fine, it frees up on restart
mount(st, h("span", { style: "color:var(--accent)" }, "ℹ " + t("settings.cmf.portOwn")));
} else {
const kids = [h("span", { style: "color:var(--bad)" }, "✗ " + t("settings.cmf.portBusy"))]; const kids = [h("span", { style: "color:var(--bad)" }, "✗ " + t("settings.cmf.portBusy"))];
if (r.suggested) { const b = h("button", { class: "btn small", type: "button" }, t("settings.cmf.useSuggested", { port: r.suggested })); b.addEventListener("click", () => { portIn.value = r.suggested; mount(st, ""); }); kids.push(b); } if (r.suggested) { const b = h("button", { class: "btn small", type: "button" }, t("settings.cmf.useSuggested", { port: r.suggested })); b.addEventListener("click", () => { portIn.value = r.suggested; mount(st, ""); }); kids.push(b); }
mount(st, h("span", {}, ...kids)); mount(st, h("span", {}, ...kids));
@@ -272,14 +254,8 @@ export async function renderSettings() {
checkBtn.disabled = false; checkBtn.disabled = false;
}); });
const row = h("div", { class: "flex wrap", style: "gap:6px;align-items:center;margin:6px 0" }, const row = h("div", { class: "flex wrap", style: "gap:6px;align-items:center;margin:6px 0" },
idIn, modelSel, portIn, checkBtn, threadsIn, h("label", { class: "flex", style: "gap:4px" }, gpuIn, "GPU"), o1In, thinkIn, tempIn, topPIn, maxTokIn, removeBtn, st); idIn, modelSel, portIn, checkBtn, threadsIn, h("label", { class: "flex", style: "gap:4px" }, gpuIn, "GPU"), removeBtn, st);
const entry = { node: row, read: () => { const entry = { node: row, read: () => ({ id: idIn.value.trim() || autoId(modelSel.value), model: modelSel.value.trim(), port: parseInt(portIn.value) || 8090, threads: parseInt(threadsIn.value) || 0, gpu: gpuIn.checked }) };
const v = { id: idIn.value.trim() || autoId(modelSel.value), model: modelSel.value.trim(), port: parseInt(portIn.value) || 8090, threads: parseInt(threadsIn.value) || 0, gpu: gpuIn.checked, o1: o1In.value, thinking: thinkIn.value };
if (tempIn.value !== "") v.temperature = parseFloat(tempIn.value);
if (topPIn.value !== "") v.top_p = parseFloat(topPIn.value);
if (maxTokIn.value !== "") v.max_tokens = parseInt(maxTokIn.value);
return v;
} };
removeBtn.addEventListener("click", () => { const i = serverRows.indexOf(entry); if (i >= 0) serverRows.splice(i, 1); row.remove(); }); removeBtn.addEventListener("click", () => { const i = serverRows.indexOf(entry); if (i >= 0) serverRows.splice(i, 1); row.remove(); });
serverRows.push(entry); serverRows.push(entry);
serversWrap.appendChild(row); serversWrap.appendChild(row);