Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
6fdbe0e159 |
@@ -3,50 +3,8 @@
|
|||||||
All notable changes to this project are documented here.
|
All notable changes to this project are documented here.
|
||||||
The format is based on [Keep a Changelog](https://keepachangelog.com/).
|
The format is based on [Keep a Changelog](https://keepachangelog.com/).
|
||||||
|
|
||||||
## [0.2.31] - 2026-07-22
|
|
||||||
|
|
||||||
### Added
|
|
||||||
- **Structured O(1) Nyström Attention & Reasoning Control**: Enhanced settings drawer with structured parameter sections for O(1) Nyström Streaming Attention (`o1_m`, `o1_window`, `o1_sink`), Generation (Temp, Top-P, MaxTokens, System Prompt), Reasoning ("Disable reasoning" toggle mapping to `think_budget = 0` for Qwen3/3.5 models), and Runtime (`skip_mtp` via `CMF_MTP=0`).
|
|
||||||
- **Persistent Admin Token Configuration**: Support for explicit `token` field in `config/gateway.toml` under `[admin]` block to prevent authorization invalidation across restarts.
|
|
||||||
|
|
||||||
### Fixed
|
|
||||||
- **JavaScript UI Helper Restoration**: Restored `opt(...)` helper in `web/js/ui.js` and `web/js/views/import.js` to ensure HF model import modal options render smoothly.
|
|
||||||
- **Engine Command Flag Accuracy**: Fixed `CMF_MTP=0` environment flag pass-through for `cortiq serve` processes.
|
|
||||||
|
|
||||||
## [0.2.30] - 2026-07-22
|
|
||||||
|
|
||||||
### Added
|
|
||||||
- **Per-Model Generation & Reasoning Parameters**: Full support in Web UI and Gateway core for per-model generation defaults: Temperature, Top-P, Max Tokens, Think Budget (Reasoning mode), O(1) Constant-Memory Streaming Attention (`o1`), MTP (Multi-Token Prediction toggle `skip_mtp`), and default System Prompt.
|
|
||||||
|
|
||||||
|
|
||||||
### Fixed
|
|
||||||
- **Duplicate Local Model Registration**: Fixed `duplicate managed model id` conflict in `register_import` by cleanly routing managed CMF models directly into `[[cmf.servers]]` with automatic free port allocation without pushing duplicate entries to `[[models]]`.
|
|
||||||
|
|
||||||
|
|
||||||
### Fixed
|
|
||||||
- **Quantization Badge & Title for Pre-Converted Models**: Automatically detect quantization (`Q1`, `Q1T`) for ready-to-run `.cmf` models (e.g. `infosave/Bonsai-27Bcmf`) from repo name and downloaded filename, resolving default `Q8_2F` badge fallback. Updated section title to "Загрузки / Конвертации" ("Downloads & Conversions").
|
|
||||||
|
|
||||||
|
|
||||||
### Fixed
|
|
||||||
- **Automatic CMF Local Server Lifecycle**: Registering an imported `.cmf` model automatically adds it to `[[cmf.servers]]` and reloads the runtime (`cmf_runtime::manage`), automatically spawning `cortiq serve` on port 8090 so chat completion requests route seamlessly without `upstream unavailable` errors.
|
|
||||||
|
|
||||||
|
|
||||||
### Fixed
|
|
||||||
- **Multi-GB Ready CMF Download Stream**: Replaced HTTP stream decoder with a streaming process downloader (`curl`) for pre-converted `.cmf` models to support multi-gigabyte Hugging Face LFS repositories without 30s/60s socket timeouts.
|
|
||||||
- **Ready CMF Quantization & UI Labels**: Set correct quantization badges (`Q1`, `Q1T`) for featured ready-to-run `.cmf` models and removed misleading "converting" log messages for direct downloads.
|
|
||||||
|
|
||||||
|
|
||||||
### Fixed
|
|
||||||
- **Direct Ready CMF Download**: Importing pre-converted `.cmf` model repositories (e.g. `infosave/Bonsai-8B_2bit_cmf`) directly streams the ready `.cmf` file from Hugging Face instead of attempting a safetensors conversion without `config.json`.
|
|
||||||
|
|
||||||
|
|
||||||
### Added
|
|
||||||
- **New Quantization Schemes**: Added support for `Q4_TILED` (q4t), `Q1T` (1.58-bit ternary), `Q1S` (1-bit sparse), `Q1P` (1-bit PTQ error diffusion), and `Q1` (pure binary) in converter backend and web admin UI.
|
|
||||||
- **Featured Ready-to-Run .cmf Models**: Added 1-tap download cards for pre-converted `.cmf` models (`infosave/Bonsai-1.7Bcmf`, `infosave/Bonsai-8B_2bit_cmf`, `infosave/Bonsai-27Bcmf`).
|
|
||||||
|
|
||||||
## [Unreleased]
|
## [Unreleased]
|
||||||
|
|
||||||
|
|
||||||
### Added
|
### Added
|
||||||
- **Docker Compose** quickstart (`docker-compose.yml` + `config/gateway.compose.toml`):
|
- **Docker Compose** quickstart (`docker-compose.yml` + `config/gateway.compose.toml`):
|
||||||
gateway + a local Ollama model in one command. GHCR badge in the READMEs.
|
gateway + a local Ollama model in one command. GHCR badge in the READMEs.
|
||||||
|
|||||||
Generated
+1
-1
@@ -285,7 +285,7 @@ checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570"
|
|||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "cortiq-gateway"
|
name = "cortiq-gateway"
|
||||||
version = "0.2.30"
|
version = "0.2.16"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"anyhow",
|
"anyhow",
|
||||||
"arc-swap",
|
"arc-swap",
|
||||||
|
|||||||
+1
-1
@@ -1,6 +1,6 @@
|
|||||||
[package]
|
[package]
|
||||||
name = "cortiq-gateway"
|
name = "cortiq-gateway"
|
||||||
version = "0.2.31"
|
version = "0.2.16"
|
||||||
edition = "2021"
|
edition = "2021"
|
||||||
rust-version = "1.75"
|
rust-version = "1.75"
|
||||||
license = "Apache-2.0"
|
license = "Apache-2.0"
|
||||||
|
|||||||
+80
-57
@@ -905,12 +905,75 @@ async fn put_settings(State(state): State<SharedState>, Json(b): Json<SettingsBo
|
|||||||
{
|
{
|
||||||
needs_restart = true;
|
needs_restart = true;
|
||||||
}
|
}
|
||||||
|
let old_servers = cfg.cmf.effective_servers();
|
||||||
cfg.cmf = v;
|
cfg.cmf = v;
|
||||||
|
// Replacing or renaming a managed model must not brick the save: routing
|
||||||
|
// may still reference the old id. Map a vanished id to its replacement
|
||||||
|
// (the new server on the same port), strip ids that are truly gone, and
|
||||||
|
// repair the default so validate() passes.
|
||||||
|
heal_routing_after_cmf_change(&mut cfg, &old_servers);
|
||||||
}
|
}
|
||||||
state.reload(cfg)?;
|
state.reload(cfg)?;
|
||||||
ok(json!({ "ok": true, "needs_restart": needs_restart }))
|
ok(json!({ "ok": true, "needs_restart": needs_restart }))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// After the managed-servers list changed, rewrite [routing] so it only
|
||||||
|
/// references ids that still exist. A removed server whose port is now held by
|
||||||
|
/// a NEW server is treated as a replacement (old id → new id); other vanished
|
||||||
|
/// ids are stripped. An emptied default falls back to the first managed server,
|
||||||
|
/// then the first pool model.
|
||||||
|
fn heal_routing_after_cmf_change(cfg: &mut Config, old_servers: &[crate::config::CmfServer]) {
|
||||||
|
let servers = cfg.cmf.effective_servers();
|
||||||
|
let known: std::collections::HashSet<String> = cfg
|
||||||
|
.models
|
||||||
|
.iter()
|
||||||
|
.map(|m| m.id.clone())
|
||||||
|
.chain(servers.iter().map(|s| s.id.clone()))
|
||||||
|
.collect();
|
||||||
|
let new_by_port: HashMap<u16, String> =
|
||||||
|
servers.iter().map(|s| (s.port, s.id.clone())).collect();
|
||||||
|
let mut rename: HashMap<String, String> = HashMap::new();
|
||||||
|
for old in old_servers {
|
||||||
|
if !known.contains(&old.id) {
|
||||||
|
if let Some(new_id) = new_by_port.get(&old.port) {
|
||||||
|
rename.insert(old.id.clone(), new_id.clone());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let fix = |id: &str| -> Option<String> {
|
||||||
|
if known.contains(id) {
|
||||||
|
Some(id.to_string())
|
||||||
|
} else {
|
||||||
|
rename.get(id).cloned()
|
||||||
|
}
|
||||||
|
};
|
||||||
|
if !cfg.routing.default.is_empty() {
|
||||||
|
cfg.routing.default = fix(&cfg.routing.default).unwrap_or_default();
|
||||||
|
}
|
||||||
|
if cfg.routing.default.is_empty() {
|
||||||
|
cfg.routing.default = servers
|
||||||
|
.first()
|
||||||
|
.map(|s| s.id.clone())
|
||||||
|
.or_else(|| cfg.models.first().map(|m| m.id.clone()))
|
||||||
|
.unwrap_or_default();
|
||||||
|
}
|
||||||
|
for tt in cfg.routing.tiers.values_mut() {
|
||||||
|
let list = match tt {
|
||||||
|
TierTargets::List(v) => v.clone(),
|
||||||
|
TierTargets::One(s) => vec![s.clone()],
|
||||||
|
};
|
||||||
|
let mut healed: Vec<String> = Vec::new();
|
||||||
|
for id in &list {
|
||||||
|
if let Some(new_id) = fix(id) {
|
||||||
|
if !healed.contains(&new_id) {
|
||||||
|
healed.push(new_id);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
*tt = TierTargets::List(healed);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// ───────────────────────────── cmf runtime ─────────────────────────────────
|
// ───────────────────────────── cmf runtime ─────────────────────────────────
|
||||||
/// Status of the `cortiq` CLI (the CMF format runtime): is it installed, which
|
/// Status of the `cortiq` CLI (the CMF format runtime): is it installed, which
|
||||||
/// version, and whether a managed server is running. Drives the "Install cortiq"
|
/// version, and whether a managed server is running. Drives the "Install cortiq"
|
||||||
@@ -1028,6 +1091,9 @@ async fn cmf_port_check(State(state): State<SharedState>, Query(q): Query<PortQu
|
|||||||
ok(json!({
|
ok(json!({
|
||||||
"port": port,
|
"port": port,
|
||||||
"available": available,
|
"available": available,
|
||||||
|
// true = held by this gateway's own managed server (a replaced/edited
|
||||||
|
// row frees it on restart) — the UI shows this as info, not an error
|
||||||
|
"managed": running_here,
|
||||||
"detail": detail,
|
"detail": detail,
|
||||||
"suggested": suggested,
|
"suggested": suggested,
|
||||||
}))
|
}))
|
||||||
@@ -1304,64 +1370,21 @@ async fn register_import(State(state): State<SharedState>, Path(job): Path<Strin
|
|||||||
.unwrap_or_else(|| "cmf-local".into());
|
.unwrap_or_else(|| "cmf-local".into());
|
||||||
let id = format!("cmf-{base}");
|
let id = format!("cmf-{base}");
|
||||||
let mut cfg = current_cfg(&state);
|
let mut cfg = current_cfg(&state);
|
||||||
|
if cfg.models.iter().any(|m| m.id == id) {
|
||||||
let is_in_models = cfg.models.iter().any(|m| m.id == id);
|
return Err(ApiError::bad(format!("model '{id}' already registered")));
|
||||||
let is_in_servers = cfg
|
|
||||||
.cmf
|
|
||||||
.servers
|
|
||||||
.iter()
|
|
||||||
.any(|s| s.id == id || s.model == j.output);
|
|
||||||
|
|
||||||
if is_in_models || is_in_servers {
|
|
||||||
return Err(ApiError::bad(format!("model '{id}' is already registered")));
|
|
||||||
}
|
|
||||||
|
|
||||||
if cfg.cmf.manage_server {
|
|
||||||
let host = if cfg.cmf.local_host.trim().is_empty() {
|
|
||||||
"127.0.0.1"
|
|
||||||
} else {
|
|
||||||
&cfg.cmf.local_host
|
|
||||||
};
|
|
||||||
let base_port = cfg.cmf.servers.iter().map(|s| s.port).max().unwrap_or(8089);
|
|
||||||
let port = next_free_port(host, base_port).unwrap_or_else(|| base_port + 1);
|
|
||||||
cfg.cmf.servers.push(crate::config::CmfServer {
|
|
||||||
id: id.clone(),
|
|
||||||
model: j.output.clone(),
|
|
||||||
port,
|
|
||||||
threads: cfg.cmf.threads,
|
|
||||||
gpu: cfg.cmf.gpu,
|
|
||||||
temperature: None,
|
|
||||||
top_p: None,
|
|
||||||
max_tokens: None,
|
|
||||||
think_budget: None,
|
|
||||||
system_prompt: None,
|
|
||||||
o1: None,
|
|
||||||
o1_m: None,
|
|
||||||
o1_window: None,
|
|
||||||
o1_sink: None,
|
|
||||||
skip_mtp: false,
|
|
||||||
});
|
|
||||||
} else {
|
|
||||||
cfg.models.push(crate::config::ModelCfg {
|
|
||||||
id: id.clone(),
|
|
||||||
provider: "openai".into(),
|
|
||||||
base_url: cfg.cmf.cortiq_server_url.clone(),
|
|
||||||
model: base,
|
|
||||||
cost_tier: "local".into(),
|
|
||||||
price_in: 0.0,
|
|
||||||
price_out: 0.0,
|
|
||||||
kind: "chat".into(),
|
|
||||||
api_key_env: None,
|
|
||||||
caps: Vec::new(),
|
|
||||||
temperature: None,
|
|
||||||
top_p: None,
|
|
||||||
max_tokens: None,
|
|
||||||
think_budget: None,
|
|
||||||
system_prompt: None,
|
|
||||||
o1: None,
|
|
||||||
skip_mtp: false,
|
|
||||||
});
|
|
||||||
}
|
}
|
||||||
|
cfg.models.push(crate::config::ModelCfg {
|
||||||
|
id: id.clone(),
|
||||||
|
provider: "openai".into(),
|
||||||
|
base_url: cfg.cmf.cortiq_server_url.clone(),
|
||||||
|
model: base,
|
||||||
|
cost_tier: "local".into(),
|
||||||
|
price_in: 0.0,
|
||||||
|
price_out: 0.0,
|
||||||
|
kind: "chat".into(),
|
||||||
|
api_key_env: None,
|
||||||
|
caps: Vec::new(),
|
||||||
|
});
|
||||||
state.reload(cfg)?;
|
state.reload(cfg)?;
|
||||||
ok(json!({ "ok": true, "model_id": id }))
|
ok(json!({ "ok": true, "model_id": id }))
|
||||||
}
|
}
|
||||||
|
|||||||
+2
-22
@@ -290,31 +290,11 @@ fn spawn_serve(server: &CmfServer, cfg: &CmfCfg) -> Result<Child, String> {
|
|||||||
if server.threads > 1 {
|
if server.threads > 1 {
|
||||||
cmd.env("CMF_THREADS", server.threads.to_string());
|
cmd.env("CMF_THREADS", server.threads.to_string());
|
||||||
}
|
}
|
||||||
|
// Offload large matvecs to the GPU (Metal on macOS). Memory-bandwidth-bound
|
||||||
|
// on unified memory, so mostly a prefill win; opt-in.
|
||||||
if server.gpu {
|
if server.gpu {
|
||||||
cmd.env("CMF_GPU", "1");
|
cmd.env("CMF_GPU", "1");
|
||||||
}
|
}
|
||||||
// O(1) Nyström attention: layers spec + sub-parameters.
|
|
||||||
if let Some(o1) = &server.o1 {
|
|
||||||
let s = o1.trim();
|
|
||||||
if !s.is_empty() {
|
|
||||||
// Pass even "off" so it explicitly overrides a file converter hint.
|
|
||||||
cmd.arg("--o1").arg(s);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if let Some(m) = server.o1_m {
|
|
||||||
cmd.arg("--o1-m").arg(m.to_string());
|
|
||||||
}
|
|
||||||
if let Some(w) = server.o1_window {
|
|
||||||
cmd.arg("--o1-window").arg(w.to_string());
|
|
||||||
}
|
|
||||||
if let Some(sink) = server.o1_sink {
|
|
||||||
cmd.arg("--o1-sink").arg(sink.to_string());
|
|
||||||
}
|
|
||||||
// MTP (Multi-Token Prediction) speculative decoding: the engine reads
|
|
||||||
// CMF_MTP env; "0" disables. There is no CLI flag for this.
|
|
||||||
if server.skip_mtp {
|
|
||||||
cmd.env("CMF_MTP", "0");
|
|
||||||
}
|
|
||||||
cmd.stdout(Stdio::null())
|
cmd.stdout(Stdio::null())
|
||||||
.stderr(Stdio::null())
|
.stderr(Stdio::null())
|
||||||
.kill_on_drop(true)
|
.kill_on_drop(true)
|
||||||
|
|||||||
+2
-66
@@ -83,36 +83,6 @@ pub struct CmfServer {
|
|||||||
/// Offload matvecs to the GPU (CMF_GPU=1).
|
/// Offload matvecs to the GPU (CMF_GPU=1).
|
||||||
#[serde(default)]
|
#[serde(default)]
|
||||||
pub gpu: bool,
|
pub gpu: bool,
|
||||||
/// Default generation temperature (0.0 .. 2.0).
|
|
||||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
|
||||||
pub temperature: Option<f32>,
|
|
||||||
/// Default nucleus sampling threshold (0.0 .. 1.0).
|
|
||||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
|
||||||
pub top_p: Option<f32>,
|
|
||||||
/// Default maximum completion tokens.
|
|
||||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
|
||||||
pub max_tokens: Option<u32>,
|
|
||||||
/// Reasoning / thinking token budget.
|
|
||||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
|
||||||
pub think_budget: Option<u32>,
|
|
||||||
/// Default system prompt prepended if request lacks one.
|
|
||||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
|
||||||
pub system_prompt: Option<String>,
|
|
||||||
/// O(1) Nyström attention layer spec (all | deepN | i,j,k | off).
|
|
||||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
|
||||||
pub o1: Option<String>,
|
|
||||||
/// O(1) landmark budget (default 32; ≥4).
|
|
||||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
|
||||||
pub o1_m: Option<u32>,
|
|
||||||
/// O(1) exact-window width (default 128).
|
|
||||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
|
||||||
pub o1_window: Option<u32>,
|
|
||||||
/// O(1) permanent exact sink keys (default 4, StreamingLLM discipline).
|
|
||||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
|
||||||
pub o1_sink: Option<u32>,
|
|
||||||
/// Disable Multi-Token Prediction speculative decoding (CMF_MTP=0).
|
|
||||||
#[serde(default)]
|
|
||||||
pub skip_mtp: bool,
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/// CMF-format model factory: import a HuggingFace model, convert it to a
|
/// CMF-format model factory: import a HuggingFace model, convert it to a
|
||||||
@@ -202,16 +172,6 @@ impl CmfCfg {
|
|||||||
port: self.local_port,
|
port: self.local_port,
|
||||||
threads: self.threads,
|
threads: self.threads,
|
||||||
gpu: self.gpu,
|
gpu: self.gpu,
|
||||||
temperature: None,
|
|
||||||
top_p: None,
|
|
||||||
max_tokens: None,
|
|
||||||
think_budget: None,
|
|
||||||
system_prompt: None,
|
|
||||||
o1: None,
|
|
||||||
o1_m: None,
|
|
||||||
o1_window: None,
|
|
||||||
o1_sink: None,
|
|
||||||
skip_mtp: false,
|
|
||||||
}];
|
}];
|
||||||
}
|
}
|
||||||
Vec::new()
|
Vec::new()
|
||||||
@@ -346,27 +306,6 @@ pub struct ModelCfg {
|
|||||||
pub api_key_env: Option<String>,
|
pub api_key_env: Option<String>,
|
||||||
#[serde(default, skip_serializing_if = "Vec::is_empty")]
|
#[serde(default, skip_serializing_if = "Vec::is_empty")]
|
||||||
pub caps: Vec<String>,
|
pub caps: Vec<String>,
|
||||||
/// Default generation temperature (0.0 .. 2.0).
|
|
||||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
|
||||||
pub temperature: Option<f32>,
|
|
||||||
/// Default nucleus sampling threshold (0.0 .. 1.0).
|
|
||||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
|
||||||
pub top_p: Option<f32>,
|
|
||||||
/// Default maximum completion tokens.
|
|
||||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
|
||||||
pub max_tokens: Option<u32>,
|
|
||||||
/// Reasoning / thinking token budget.
|
|
||||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
|
||||||
pub think_budget: Option<u32>,
|
|
||||||
/// Default system prompt prepended if request lacks one.
|
|
||||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
|
||||||
pub system_prompt: Option<String>,
|
|
||||||
/// O(1) constant-memory streaming attention hint (all, deep, custom, off).
|
|
||||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
|
||||||
pub o1: Option<String>,
|
|
||||||
/// Skip Multi-Token Prediction (MTP) heads.
|
|
||||||
#[serde(default)]
|
|
||||||
pub skip_mtp: bool,
|
|
||||||
}
|
}
|
||||||
fn default_kind() -> String {
|
fn default_kind() -> String {
|
||||||
"chat".into()
|
"chat".into()
|
||||||
@@ -522,12 +461,10 @@ pub struct AdminCfg {
|
|||||||
/// Whether the admin panel and admin API (`/admin`, `/admin/api/*`) are enabled.
|
/// Whether the admin panel and admin API (`/admin`, `/admin/api/*`) are enabled.
|
||||||
#[serde(default = "default_admin_enabled")]
|
#[serde(default = "default_admin_enabled")]
|
||||||
pub enabled: bool,
|
pub enabled: bool,
|
||||||
/// Name of the env variable holding the admin token.
|
/// Name of the env variable holding the admin token. If unset (or empty),
|
||||||
|
/// a token is generated at startup and printed to the log.
|
||||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||||
pub token_env: Option<String>,
|
pub token_env: Option<String>,
|
||||||
/// Persistent admin token configured directly in toml.
|
|
||||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
|
||||||
pub token: Option<String>,
|
|
||||||
/// Optional separate bind address for the admin surface (e.g. `127.0.0.1:9001`).
|
/// Optional separate bind address for the admin surface (e.g. `127.0.0.1:9001`).
|
||||||
/// If unset, the admin interface lives on the main `listen` address under `/admin`.
|
/// If unset, the admin interface lives on the main `listen` address under `/admin`.
|
||||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||||
@@ -538,7 +475,6 @@ impl Default for AdminCfg {
|
|||||||
Self {
|
Self {
|
||||||
enabled: default_admin_enabled(),
|
enabled: default_admin_enabled(),
|
||||||
token_env: None,
|
token_env: None,
|
||||||
token: None,
|
|
||||||
listen: None,
|
listen: None,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
+3
-224
@@ -160,14 +160,6 @@ impl JobStore {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
fn update_quant(&self, id: &str, quant: String) {
|
|
||||||
if let Some(j) = self.jobs.lock().unwrap().get_mut(id) {
|
|
||||||
j.quant = quant.clone();
|
|
||||||
if !j.log.is_empty() && j.log[0].starts_with("→ downloading ready .cmf model") {
|
|
||||||
j.log[0] = format!("→ downloading ready .cmf model {} ({})", j.repo, quant);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
fn set_state(&self, id: &str, state: &str) {
|
fn set_state(&self, id: &str, state: &str) {
|
||||||
if let Some(j) = self.jobs.lock().unwrap().get_mut(id) {
|
if let Some(j) = self.jobs.lock().unwrap().get_mut(id) {
|
||||||
j.state = state.into();
|
j.state = state.into();
|
||||||
@@ -268,166 +260,14 @@ fn sanitize(s: &str) -> String {
|
|||||||
/// Map the UI quant label to the native `cortiq` quant flag.
|
/// Map the UI quant label to the native `cortiq` quant flag.
|
||||||
fn map_quant(q: &str) -> &'static str {
|
fn map_quant(q: &str) -> &'static str {
|
||||||
match q.to_ascii_uppercase().as_str() {
|
match q.to_ascii_uppercase().as_str() {
|
||||||
s if s.starts_with("Q4T") || s == "Q4_TILED" => "q4t",
|
|
||||||
s if s.starts_with("Q4") => "q4",
|
s if s.starts_with("Q4") => "q4",
|
||||||
"Q8_2F" | "Q82F" => "q8_2f",
|
"Q8_2F" => "q8_2f",
|
||||||
"F16" | "FP16" => "f16",
|
"F16" | "FP16" => "f16",
|
||||||
"VBIT" => "vbit",
|
"VBIT" => "vbit",
|
||||||
"Q1" => "q1",
|
|
||||||
"Q1P" | "Q1_PTQ" => "q1p",
|
|
||||||
"Q1S" | "Q1_MASK" => "q1s",
|
|
||||||
"Q1T" | "Q1_TERNARY" => "q1t",
|
|
||||||
_ => "q8",
|
_ => "q8",
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
fn is_cmf_repo(repo: &str) -> bool {
|
|
||||||
let lower = repo.to_ascii_lowercase();
|
|
||||||
lower.contains("cmf") || lower.ends_with(".cmf")
|
|
||||||
}
|
|
||||||
|
|
||||||
async fn download_cmf_repo(
|
|
||||||
repo: String,
|
|
||||||
output_abs: String,
|
|
||||||
hf_token: Option<String>,
|
|
||||||
store: Arc<JobStore>,
|
|
||||||
id: String,
|
|
||||||
cancel: Arc<Notify>,
|
|
||||||
) {
|
|
||||||
store.set_progress(&id, 0.0, "listing files".into());
|
|
||||||
|
|
||||||
let tree_url = format!("https://huggingface.co/api/models/{repo}/tree/main?recursive=true");
|
|
||||||
let client = reqwest::Client::new();
|
|
||||||
let mut req = client.get(&tree_url).header("User-Agent", "cortiq-gateway");
|
|
||||||
if let Some(t) = &hf_token {
|
|
||||||
req = req.bearer_auth(t);
|
|
||||||
}
|
|
||||||
|
|
||||||
let mut cmf_rel_path = None;
|
|
||||||
if let Ok(resp) = req.send().await {
|
|
||||||
if resp.status().is_success() {
|
|
||||||
if let Ok(json) = resp.json::<serde_json::Value>().await {
|
|
||||||
if let Some(arr) = json.as_array() {
|
|
||||||
for item in arr {
|
|
||||||
if let Some(path) = item.get("path").and_then(|p| p.as_str()) {
|
|
||||||
if path.to_ascii_lowercase().ends_with(".cmf") {
|
|
||||||
cmf_rel_path = Some(path.to_string());
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
let cmf_path = match cmf_rel_path {
|
|
||||||
Some(p) => p,
|
|
||||||
None => {
|
|
||||||
let repo_name = repo.split('/').last().unwrap_or(&repo);
|
|
||||||
format!("{repo_name}.cmf")
|
|
||||||
}
|
|
||||||
};
|
|
||||||
|
|
||||||
let path_lower = cmf_path.to_ascii_lowercase();
|
|
||||||
let detected_quant =
|
|
||||||
if path_lower.contains("-q1t.") || repo.to_ascii_lowercase().contains("2bit") {
|
|
||||||
"Q1T".to_string()
|
|
||||||
} else if path_lower.contains("-q1.") || repo.to_ascii_lowercase().contains("cmf") {
|
|
||||||
"Q1".to_string()
|
|
||||||
} else if path_lower.contains("-q4.") {
|
|
||||||
"Q4".to_string()
|
|
||||||
} else if path_lower.contains("-q8.") {
|
|
||||||
"Q8".to_string()
|
|
||||||
} else {
|
|
||||||
"READY_CMF".to_string()
|
|
||||||
};
|
|
||||||
store.update_quant(&id, detected_quant);
|
|
||||||
|
|
||||||
store.push_line(&id, format!("→ downloading {cmf_path} from HuggingFace..."));
|
|
||||||
store.set_progress(&id, 0.05, "downloading".into());
|
|
||||||
|
|
||||||
let download_url = format!("https://huggingface.co/{repo}/resolve/main/{cmf_path}");
|
|
||||||
|
|
||||||
let mut cmd = tokio::process::Command::new("curl");
|
|
||||||
cmd.arg("-f").arg("-L").arg("-#").arg("-o").arg(&output_abs);
|
|
||||||
|
|
||||||
if let Some(t) = &hf_token {
|
|
||||||
cmd.arg("-H").arg(format!("Authorization: Bearer {t}"));
|
|
||||||
}
|
|
||||||
|
|
||||||
cmd.arg(&download_url);
|
|
||||||
cmd.stderr(std::process::Stdio::piped());
|
|
||||||
cmd.stdout(std::process::Stdio::null());
|
|
||||||
|
|
||||||
let mut child = match cmd.spawn() {
|
|
||||||
Ok(c) => c,
|
|
||||||
Err(e) => {
|
|
||||||
store.push_line(&id, format!("✗ failed to start curl download: {e}"));
|
|
||||||
store.finish(&id, false);
|
|
||||||
store.drop_cancel(&id);
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
};
|
|
||||||
|
|
||||||
let stderr = child.stderr.take();
|
|
||||||
let mut t_err = None;
|
|
||||||
if let Some(err_stream) = stderr {
|
|
||||||
let store_clone = store.clone();
|
|
||||||
let id_clone = id.clone();
|
|
||||||
t_err = Some(tokio::spawn(async move {
|
|
||||||
use tokio::io::AsyncReadExt;
|
|
||||||
let mut reader = err_stream;
|
|
||||||
let mut buf = [0u8; 256];
|
|
||||||
while let Ok(n) = reader.read(&mut buf).await {
|
|
||||||
if n == 0 {
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
let text = String::from_utf8_lossy(&buf[..n]);
|
|
||||||
for word in text.split_whitespace() {
|
|
||||||
if let Some(num_str) = word.strip_suffix('%') {
|
|
||||||
if let Ok(pct) = num_str.parse::<f32>() {
|
|
||||||
let frac = (pct / 100.0).clamp(0.0, 1.0);
|
|
||||||
store_clone.set_progress(&id_clone, frac, "downloading".into());
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}));
|
|
||||||
}
|
|
||||||
|
|
||||||
tokio::select! {
|
|
||||||
status_res = child.wait() => {
|
|
||||||
if let Some(t) = t_err { t.abort(); }
|
|
||||||
match status_res {
|
|
||||||
Ok(status) if status.success() => {
|
|
||||||
store.push_line(&id, "✓ done".into());
|
|
||||||
store.finish(&id, true);
|
|
||||||
}
|
|
||||||
Ok(status) => {
|
|
||||||
cleanup_output(&output_abs);
|
|
||||||
store.push_line(&id, format!("✗ download failed: exit code {:?}", status.code()));
|
|
||||||
store.finish(&id, false);
|
|
||||||
}
|
|
||||||
Err(e) => {
|
|
||||||
cleanup_output(&output_abs);
|
|
||||||
store.push_line(&id, format!("✗ download process error: {e}"));
|
|
||||||
store.finish(&id, false);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
_ = cancel.notified() => {
|
|
||||||
let _ = child.start_kill();
|
|
||||||
let _ = child.wait().await;
|
|
||||||
if let Some(t) = t_err { t.abort(); }
|
|
||||||
cleanup_output(&output_abs);
|
|
||||||
store.push_line(&id, "✗ cancelled by user — partial output removed".into());
|
|
||||||
store.set_state(&id, "cancelled");
|
|
||||||
}
|
|
||||||
}
|
|
||||||
store.drop_cancel(&id);
|
|
||||||
}
|
|
||||||
|
|
||||||
/// into the job log. Fails fast if the converter script is missing.
|
/// into the job log. Fails fast if the converter script is missing.
|
||||||
pub fn start_import(store: Arc<JobStore>, cfg: &CmfCfg, p: ImportParams) -> Result<String, String> {
|
pub fn start_import(store: Arc<JobStore>, cfg: &CmfCfg, p: ImportParams) -> Result<String, String> {
|
||||||
if p.repo.trim().is_empty() {
|
if p.repo.trim().is_empty() {
|
||||||
@@ -486,37 +326,13 @@ pub fn start_import(store: Arc<JobStore>, cfg: &CmfCfg, p: ImportParams) -> Resu
|
|||||||
.unwrap_or_else(|_| output.clone());
|
.unwrap_or_else(|_| output.clone());
|
||||||
|
|
||||||
let id = gen_id(&p.repo);
|
let id = gen_id(&p.repo);
|
||||||
let is_cmf = is_cmf_repo(&p.repo);
|
|
||||||
let quant_display = if is_cmf {
|
|
||||||
let r = p.repo.to_lowercase();
|
|
||||||
if r.contains("2bit") || r.contains("q1t") {
|
|
||||||
"Q1T".to_string()
|
|
||||||
} else if r.contains("1.7bcmf") || r.contains("27bcmf") || r.contains("cmf") {
|
|
||||||
"Q1".to_string()
|
|
||||||
} else if p.quant.is_empty() || p.quant == "auto" || p.quant == "Q8_2F" {
|
|
||||||
"Q1".to_string()
|
|
||||||
} else {
|
|
||||||
p.quant.clone()
|
|
||||||
}
|
|
||||||
} else if p.quant.is_empty() || p.quant == "auto" {
|
|
||||||
"q8".to_string()
|
|
||||||
} else {
|
|
||||||
p.quant.clone()
|
|
||||||
};
|
|
||||||
store.insert(Job {
|
store.insert(Job {
|
||||||
id: id.clone(),
|
id: id.clone(),
|
||||||
repo: p.repo.clone(),
|
repo: p.repo.clone(),
|
||||||
quant: quant_display.clone(),
|
quant: p.quant.clone(),
|
||||||
output: output_abs.clone(),
|
output: output_abs.clone(),
|
||||||
state: "running".into(),
|
state: "running".into(),
|
||||||
log: if is_cmf {
|
log: vec![format!("→ converting {} to {} ({})", p.repo, base, p.quant)],
|
||||||
vec![format!(
|
|
||||||
"→ downloading ready .cmf model {} ({})",
|
|
||||||
p.repo, quant_display
|
|
||||||
)]
|
|
||||||
} else {
|
|
||||||
vec![format!("→ converting {} to {} ({})", p.repo, base, p.quant)]
|
|
||||||
},
|
|
||||||
started: now(),
|
started: now(),
|
||||||
finished: None,
|
finished: None,
|
||||||
size_bytes: None,
|
size_bytes: None,
|
||||||
@@ -532,14 +348,6 @@ pub fn start_import(store: Arc<JobStore>, cfg: &CmfCfg, p: ImportParams) -> Resu
|
|||||||
std::env::var(&cfg.hf_token_env).ok()
|
std::env::var(&cfg.hf_token_env).ok()
|
||||||
};
|
};
|
||||||
|
|
||||||
if is_cmf && !use_python {
|
|
||||||
let ret_id = id.clone();
|
|
||||||
tokio::spawn(async move {
|
|
||||||
download_cmf_repo(p.repo, output_abs, hf_token, store, id, cancel).await;
|
|
||||||
});
|
|
||||||
return Ok(ret_id);
|
|
||||||
}
|
|
||||||
|
|
||||||
// Program + args: `cortiq convert` by default, or the Python converter for
|
// Program + args: `cortiq convert` by default, or the Python converter for
|
||||||
// advanced options. Both stream `@PROGRESS <frac>` markers into the log.
|
// advanced options. Both stream `@PROGRESS <frac>` markers into the log.
|
||||||
let (program, args, workdir): (String, Vec<String>, std::path::PathBuf) = if use_python {
|
let (program, args, workdir): (String, Vec<String>, std::path::PathBuf) = if use_python {
|
||||||
@@ -722,32 +530,3 @@ pub fn start_import(store: Arc<JobStore>, cfg: &CmfCfg, p: ImportParams) -> Resu
|
|||||||
|
|
||||||
Ok(ret_id)
|
Ok(ret_id)
|
||||||
}
|
}
|
||||||
|
|
||||||
#[cfg(test)]
|
|
||||||
mod tests {
|
|
||||||
use super::*;
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_map_quant_variants() {
|
|
||||||
assert_eq!(map_quant("Q8_2F"), "q8_2f");
|
|
||||||
assert_eq!(map_quant("q82f"), "q8_2f");
|
|
||||||
assert_eq!(map_quant("Q8_ROW"), "q8");
|
|
||||||
assert_eq!(map_quant("Q4_BLOCK"), "q4");
|
|
||||||
assert_eq!(map_quant("Q4_TILED"), "q4t");
|
|
||||||
assert_eq!(map_quant("q4t"), "q4t");
|
|
||||||
assert_eq!(map_quant("vbit"), "vbit");
|
|
||||||
assert_eq!(map_quant("q1"), "q1");
|
|
||||||
assert_eq!(map_quant("Q1P"), "q1p");
|
|
||||||
assert_eq!(map_quant("q1s"), "q1s");
|
|
||||||
assert_eq!(map_quant("Q1T"), "q1t");
|
|
||||||
assert_eq!(map_quant("F16"), "f16");
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_is_cmf_repo_check() {
|
|
||||||
assert!(is_cmf_repo("infosave/Bonsai-8B_2bit_cmf"));
|
|
||||||
assert!(is_cmf_repo("infosave/Bonsai-1.7Bcmf"));
|
|
||||||
assert!(is_cmf_repo("user/model.cmf"));
|
|
||||||
assert!(!is_cmf_repo("Qwen/Qwen2.5-0.5B-Instruct"));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -61,11 +61,6 @@ fn resolve_admin_token(cfg: &Config, args: &Args) -> (String, bool) {
|
|||||||
return (t.clone(), false);
|
return (t.clone(), false);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if let Some(t) = &cfg.admin.token {
|
|
||||||
if !t.is_empty() {
|
|
||||||
return (t.clone(), false);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
(admin::random_token(24), true)
|
(admin::random_token(24), true)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -29,7 +29,6 @@ pub struct GenParams {
|
|||||||
pub temperature: Option<f32>,
|
pub temperature: Option<f32>,
|
||||||
pub max_tokens: Option<u32>,
|
pub max_tokens: Option<u32>,
|
||||||
pub top_p: Option<f32>,
|
pub top_p: Option<f32>,
|
||||||
pub think_budget: Option<u32>,
|
|
||||||
pub stop: Vec<String>,
|
pub stop: Vec<String>,
|
||||||
/// Fields we do not interpret but must proxy through to the provider.
|
/// Fields we do not interpret but must proxy through to the provider.
|
||||||
pub passthrough: serde_json::Map<String, serde_json::Value>,
|
pub passthrough: serde_json::Map<String, serde_json::Value>,
|
||||||
|
|||||||
+1
-58
@@ -341,64 +341,7 @@ impl Pipeline {
|
|||||||
route_source
|
route_source
|
||||||
);
|
);
|
||||||
|
|
||||||
let mut req_call = req.clone();
|
match provider.chat(req.clone()).await {
|
||||||
if let Some(m_cfg) = live.cfg.models.iter().find(|m| m.id == model_id) {
|
|
||||||
if req_call.params.temperature.is_none() {
|
|
||||||
req_call.params.temperature = m_cfg.temperature;
|
|
||||||
}
|
|
||||||
if req_call.params.top_p.is_none() {
|
|
||||||
req_call.params.top_p = m_cfg.top_p;
|
|
||||||
}
|
|
||||||
if req_call.params.max_tokens.is_none() {
|
|
||||||
req_call.params.max_tokens = m_cfg.max_tokens;
|
|
||||||
}
|
|
||||||
if req_call.params.think_budget.is_none() {
|
|
||||||
req_call.params.think_budget = m_cfg.think_budget;
|
|
||||||
}
|
|
||||||
if let Some(sys) = &m_cfg.system_prompt {
|
|
||||||
if !sys.trim().is_empty()
|
|
||||||
&& !req_call.messages.iter().any(|m| m.role == "system")
|
|
||||||
{
|
|
||||||
req_call.messages.insert(
|
|
||||||
0,
|
|
||||||
Message {
|
|
||||||
role: "system".into(),
|
|
||||||
content: sys.clone(),
|
|
||||||
tool_calls: Vec::new(),
|
|
||||||
},
|
|
||||||
);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
} else if let Some(s_cfg) = live.cfg.cmf.servers.iter().find(|s| s.id == model_id) {
|
|
||||||
if req_call.params.temperature.is_none() {
|
|
||||||
req_call.params.temperature = s_cfg.temperature;
|
|
||||||
}
|
|
||||||
if req_call.params.top_p.is_none() {
|
|
||||||
req_call.params.top_p = s_cfg.top_p;
|
|
||||||
}
|
|
||||||
if req_call.params.max_tokens.is_none() {
|
|
||||||
req_call.params.max_tokens = s_cfg.max_tokens;
|
|
||||||
}
|
|
||||||
if req_call.params.think_budget.is_none() {
|
|
||||||
req_call.params.think_budget = s_cfg.think_budget;
|
|
||||||
}
|
|
||||||
if let Some(sys) = &s_cfg.system_prompt {
|
|
||||||
if !sys.trim().is_empty()
|
|
||||||
&& !req_call.messages.iter().any(|m| m.role == "system")
|
|
||||||
{
|
|
||||||
req_call.messages.insert(
|
|
||||||
0,
|
|
||||||
Message {
|
|
||||||
role: "system".into(),
|
|
||||||
content: sys.clone(),
|
|
||||||
tool_calls: Vec::new(),
|
|
||||||
},
|
|
||||||
);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
match provider.chat(req_call).await {
|
|
||||||
Ok(mut resp) => {
|
Ok(mut resp) => {
|
||||||
let cost =
|
let cost =
|
||||||
provider.price(resp.usage.prompt_tokens, resp.usage.completion_tokens);
|
provider.price(resp.usage.prompt_tokens, resp.usage.completion_tokens);
|
||||||
|
|||||||
@@ -101,7 +101,6 @@ async fn handler(
|
|||||||
temperature: req.temperature,
|
temperature: req.temperature,
|
||||||
max_tokens: req.max_tokens,
|
max_tokens: req.max_tokens,
|
||||||
top_p: req.top_p,
|
top_p: req.top_p,
|
||||||
think_budget: None,
|
|
||||||
stop: Vec::new(),
|
stop: Vec::new(),
|
||||||
passthrough: Default::default(),
|
passthrough: Default::default(),
|
||||||
},
|
},
|
||||||
|
|||||||
@@ -49,10 +49,6 @@ struct OpenAiChatRequest {
|
|||||||
#[serde(default)]
|
#[serde(default)]
|
||||||
top_p: Option<f32>,
|
top_p: Option<f32>,
|
||||||
#[serde(default)]
|
#[serde(default)]
|
||||||
think_budget: Option<u32>,
|
|
||||||
#[serde(default)]
|
|
||||||
reasoning_effort: Option<u32>,
|
|
||||||
#[serde(default)]
|
|
||||||
stream: bool,
|
stream: bool,
|
||||||
#[serde(default)]
|
#[serde(default)]
|
||||||
tools: Vec<serde_json::Value>,
|
tools: Vec<serde_json::Value>,
|
||||||
@@ -97,7 +93,6 @@ async fn handler(
|
|||||||
temperature: req.temperature,
|
temperature: req.temperature,
|
||||||
max_tokens: req.max_tokens,
|
max_tokens: req.max_tokens,
|
||||||
top_p: req.top_p,
|
top_p: req.top_p,
|
||||||
think_budget: req.think_budget.or(req.reasoning_effort),
|
|
||||||
stop: Vec::new(),
|
stop: Vec::new(),
|
||||||
passthrough: req.rest,
|
passthrough: req.rest,
|
||||||
},
|
},
|
||||||
|
|||||||
@@ -82,7 +82,6 @@ async fn handler(
|
|||||||
temperature: req.temperature,
|
temperature: req.temperature,
|
||||||
max_tokens: req.max_tokens,
|
max_tokens: req.max_tokens,
|
||||||
top_p: req.top_p,
|
top_p: req.top_p,
|
||||||
think_budget: None,
|
|
||||||
stop: Vec::new(),
|
stop: Vec::new(),
|
||||||
passthrough: Default::default(),
|
passthrough: Default::default(),
|
||||||
},
|
},
|
||||||
|
|||||||
@@ -42,16 +42,9 @@ impl OpenAiProvider {
|
|||||||
if let Some(t) = req.params.temperature {
|
if let Some(t) = req.params.temperature {
|
||||||
body["temperature"] = t.into();
|
body["temperature"] = t.into();
|
||||||
}
|
}
|
||||||
if let Some(p) = req.params.top_p {
|
|
||||||
body["top_p"] = p.into();
|
|
||||||
}
|
|
||||||
if let Some(m) = req.params.max_tokens {
|
if let Some(m) = req.params.max_tokens {
|
||||||
body["max_tokens"] = m.into();
|
body["max_tokens"] = m.into();
|
||||||
}
|
}
|
||||||
if let Some(tb) = req.params.think_budget {
|
|
||||||
body["think_budget"] = tb.into();
|
|
||||||
body["reasoning_effort"] = tb.into();
|
|
||||||
}
|
|
||||||
if !req.tools.is_empty() {
|
if !req.tools.is_empty() {
|
||||||
body["tools"] = serde_json::Value::Array(req.tools.clone());
|
body["tools"] = serde_json::Value::Array(req.tools.clone());
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -49,13 +49,6 @@ impl Registry {
|
|||||||
kind: "chat".into(),
|
kind: "chat".into(),
|
||||||
api_key_env: None,
|
api_key_env: None,
|
||||||
caps: Vec::new(),
|
caps: Vec::new(),
|
||||||
temperature: s.temperature,
|
|
||||||
top_p: s.top_p,
|
|
||||||
max_tokens: s.max_tokens,
|
|
||||||
think_budget: s.think_budget,
|
|
||||||
system_prompt: s.system_prompt.clone(),
|
|
||||||
o1: s.o1.clone(),
|
|
||||||
skip_mtp: s.skip_mtp,
|
|
||||||
};
|
};
|
||||||
by_id.insert(m.id.clone(), Arc::new(OpenAiProvider::new(&m, None)?));
|
by_id.insert(m.id.clone(), Arc::new(OpenAiProvider::new(&m, None)?));
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -145,14 +145,6 @@ impl AppState {
|
|||||||
let live = Live::build(new_cfg.clone(), &self.secrets, self.router_status.clone())?;
|
let live = Live::build(new_cfg.clone(), &self.secrets, self.router_status.clone())?;
|
||||||
new_cfg.save(&self.config_path)?;
|
new_cfg.save(&self.config_path)?;
|
||||||
self.live.store(Arc::new(live));
|
self.live.store(Arc::new(live));
|
||||||
|
|
||||||
let cmf_rt = self.cmf.clone();
|
|
||||||
let cmf_cfg = new_cfg.cmf.clone();
|
|
||||||
tokio::spawn(async move {
|
|
||||||
cmf_rt.stop().await;
|
|
||||||
crate::cmf_runtime::manage(cmf_rt, cmf_cfg).await;
|
|
||||||
});
|
|
||||||
|
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
Vendored
+2
-22
@@ -139,18 +139,6 @@ export default {
|
|||||||
"models.form.apiKeyHint": "Stored securely on the gateway; it is not shown again.",
|
"models.form.apiKeyHint": "Stored securely on the gateway; it is not shown again.",
|
||||||
"models.form.keyStored": "A key is saved — leave blank to keep it, or paste a new one.",
|
"models.form.keyStored": "A key is saved — leave blank to keep it, or paste a new one.",
|
||||||
"models.form.keyFromEnv": "Using a key from an environment variable.",
|
"models.form.keyFromEnv": "Using a key from an environment variable.",
|
||||||
"models.form.paramsTitle": "Generation & Thinking Parameters (O(1), MTP, Temperature)",
|
|
||||||
"models.form.temperature": "Temperature",
|
|
||||||
"models.form.temperatureHint": "Generation temperature (0.0 = deterministic, 1.0+ = creative).",
|
|
||||||
"models.form.topP": "Top-P (Nucleus Sampling)",
|
|
||||||
"models.form.topPHint": "Sampling threshold (0.0 .. 1.0).",
|
|
||||||
"models.form.maxTokens": "Max Tokens",
|
|
||||||
"models.form.maxTokensHint": "Maximum completion tokens to generate.",
|
|
||||||
"models.form.thinkBudget": "Thinking Mode / Think Budget",
|
|
||||||
"models.form.thinkBudgetHint": "Reasoning token budget (0 = disabled, N > 0 = max reasoning tokens).",
|
|
||||||
"models.form.systemPrompt": "Default System Prompt",
|
|
||||||
"models.form.systemPromptHint": "Prepend automatically if request lacks a system prompt.",
|
|
||||||
"models.form.systemPromptPlaceholder": "You are a helpful assistant...",
|
|
||||||
"models.form.testKey": "Test & load models",
|
"models.form.testKey": "Test & load models",
|
||||||
"models.form.testing": "checking…",
|
"models.form.testing": "checking…",
|
||||||
"models.form.foundModels": "{n} models loaded",
|
"models.form.foundModels": "{n} models loaded",
|
||||||
@@ -279,6 +267,7 @@ export default {
|
|||||||
"settings.cmf.checkPort": "Check port",
|
"settings.cmf.checkPort": "Check port",
|
||||||
"settings.cmf.portFree": "port is free",
|
"settings.cmf.portFree": "port is free",
|
||||||
"settings.cmf.portBusy": "port is in use",
|
"settings.cmf.portBusy": "port is in use",
|
||||||
|
"settings.cmf.portOwn": "in use by this gateway's current server — freed on restart, replacing is fine",
|
||||||
"settings.cmf.useSuggested": "use {port}",
|
"settings.cmf.useSuggested": "use {port}",
|
||||||
"settings.cmf.serving": "serving {model} on :{port}",
|
"settings.cmf.serving": "serving {model} on :{port}",
|
||||||
"settings.cmf.intro": "Run models locally with the cortiq engine — works even offline or when the router is down.",
|
"settings.cmf.intro": "Run models locally with the cortiq engine — works even offline or when the router is down.",
|
||||||
@@ -386,7 +375,7 @@ export default {
|
|||||||
"import.failed": "Conversion failed — see the log.",
|
"import.failed": "Conversion failed — see the log.",
|
||||||
"import.register": "Register as local model",
|
"import.register": "Register as local model",
|
||||||
"import.registered": "Registered as '{id}'. Run cortiq-server to serve it.",
|
"import.registered": "Registered as '{id}'. Run cortiq-server to serve it.",
|
||||||
"import.jobs": "Downloads & Conversions",
|
"import.jobs": "Conversions",
|
||||||
"import.cancel": "Cancel",
|
"import.cancel": "Cancel",
|
||||||
"import.cancelling": "Cancelling — cleaning up partial output…",
|
"import.cancelling": "Cancelling — cleaning up partial output…",
|
||||||
"import.delete": "Delete",
|
"import.delete": "Delete",
|
||||||
@@ -396,21 +385,12 @@ export default {
|
|||||||
"import.state.done": "done",
|
"import.state.done": "done",
|
||||||
"import.state.error": "error",
|
"import.state.error": "error",
|
||||||
"import.state.cancelled": "cancelled",
|
"import.state.cancelled": "cancelled",
|
||||||
"import.featuredTitle": "Featured Ready-to-Run .cmf Models",
|
|
||||||
"import.readyCmfBadge": "Ready .cmf",
|
|
||||||
"import.directDownload": "Download",
|
|
||||||
"import.q.q8_2f": "8-bit two-field (𝒲×θ) — recommended: best quality/size.",
|
"import.q.q8_2f": "8-bit two-field (𝒲×θ) — recommended: best quality/size.",
|
||||||
"import.q.q8_row": "8-bit per-row — simple, robust.",
|
"import.q.q8_row": "8-bit per-row — simple, robust.",
|
||||||
"import.q.q4": "4-bit block — smallest, lower quality.",
|
"import.q.q4": "4-bit block — smallest, lower quality.",
|
||||||
"import.q.q4_tiled": "4-bit tiled (Q4_TILED) — 1.66x faster on Metal GPU.",
|
|
||||||
"import.q.vbit": "Variable 3–8 bit — per-expert bit budgets.",
|
"import.q.vbit": "Variable 3–8 bit — per-expert bit budgets.",
|
||||||
"import.q.q1t": "1.58-bit ternary (Q1T) — BitNet b1.58 with sparse outliers.",
|
|
||||||
"import.q.q1s": "1-bit sparse (Q1S) — 1-bit base with holographic outliers.",
|
|
||||||
"import.q.q1p": "1-bit PTQ (Q1P) — 1-bit with error diffusion.",
|
|
||||||
"import.q.q1": "1-bit (Q1) — pure binary format.",
|
|
||||||
"import.q.f16": "16-bit — no quantization.",
|
"import.q.f16": "16-bit — no quantization.",
|
||||||
"import.q.f32": "32-bit — full precision (large file).",
|
"import.q.f32": "32-bit — full precision (large file).",
|
||||||
|
|
||||||
"import.adv.title": "Advanced settings",
|
"import.adv.title": "Advanced settings",
|
||||||
"import.adv.auto": "auto",
|
"import.adv.auto": "auto",
|
||||||
"import.adv.linearCore": "Linear core",
|
"import.adv.linearCore": "Linear core",
|
||||||
|
|||||||
Vendored
+2
-22
@@ -138,18 +138,6 @@ export default {
|
|||||||
"models.form.apiKeyHint": "Хранится безопасно на шлюзе; больше не показывается.",
|
"models.form.apiKeyHint": "Хранится безопасно на шлюзе; больше не показывается.",
|
||||||
"models.form.keyStored": "Ключ сохранён — оставьте пустым, чтобы не менять, или вставьте новый.",
|
"models.form.keyStored": "Ключ сохранён — оставьте пустым, чтобы не менять, или вставьте новый.",
|
||||||
"models.form.keyFromEnv": "Используется ключ из переменной окружения.",
|
"models.form.keyFromEnv": "Используется ключ из переменной окружения.",
|
||||||
"models.form.paramsTitle": "Параметры генерации, O(1) и режим размышления",
|
|
||||||
"models.form.temperature": "Температура",
|
|
||||||
"models.form.temperatureHint": "Температура генерации (0.0 — точные факты, 1.0+ — творчество).",
|
|
||||||
"models.form.topP": "Top-P (Nucleus Sampling)",
|
|
||||||
"models.form.topPHint": "Порог сэмплирования токенов (0.0 .. 1.0).",
|
|
||||||
"models.form.maxTokens": "Макс. токенов (Max Tokens)",
|
|
||||||
"models.form.maxTokensHint": "Максимальное количество генерируемых токенов.",
|
|
||||||
"models.form.thinkBudget": "Режим размышления / Think Budget",
|
|
||||||
"models.form.thinkBudgetHint": "Бюджет токенов мыслей (0 = выключен, N > 0 = лимит рассуждений).",
|
|
||||||
"models.form.systemPrompt": "Системный промпт по умолчанию",
|
|
||||||
"models.form.systemPromptHint": "Автоматически подставляется, если запрос не содержит системное сообщение.",
|
|
||||||
"models.form.systemPromptPlaceholder": "Вы — полезный ассистент...",
|
|
||||||
"models.form.testKey": "Проверить и загрузить модели",
|
"models.form.testKey": "Проверить и загрузить модели",
|
||||||
"models.form.testing": "проверка…",
|
"models.form.testing": "проверка…",
|
||||||
"models.form.foundModels": "загружено моделей: {n}",
|
"models.form.foundModels": "загружено моделей: {n}",
|
||||||
@@ -278,6 +266,7 @@ export default {
|
|||||||
"settings.cmf.checkPort": "Проверить порт",
|
"settings.cmf.checkPort": "Проверить порт",
|
||||||
"settings.cmf.portFree": "порт свободен",
|
"settings.cmf.portFree": "порт свободен",
|
||||||
"settings.cmf.portBusy": "порт занят",
|
"settings.cmf.portBusy": "порт занят",
|
||||||
|
"settings.cmf.portOwn": "занят текущим сервером шлюза — освободится после перезапуска, замена допустима",
|
||||||
"settings.cmf.useSuggested": "занять {port}",
|
"settings.cmf.useSuggested": "занять {port}",
|
||||||
"settings.cmf.serving": "работает: {model} на :{port}",
|
"settings.cmf.serving": "работает: {model} на :{port}",
|
||||||
"settings.cmf.intro": "Запускайте модели локально движком cortiq — работает даже офлайн или когда роутер недоступен.",
|
"settings.cmf.intro": "Запускайте модели локально движком cortiq — работает даже офлайн или когда роутер недоступен.",
|
||||||
@@ -385,7 +374,7 @@ export default {
|
|||||||
"import.failed": "Конвертация не удалась — смотрите лог.",
|
"import.failed": "Конвертация не удалась — смотрите лог.",
|
||||||
"import.register": "Подключить как локальную модель",
|
"import.register": "Подключить как локальную модель",
|
||||||
"import.registered": "Подключено как '{id}'. Запустите cortiq-server для обслуживания.",
|
"import.registered": "Подключено как '{id}'. Запустите cortiq-server для обслуживания.",
|
||||||
"import.jobs": "Загрузки / Конвертации",
|
"import.jobs": "Конвертации",
|
||||||
"import.cancel": "Отменить",
|
"import.cancel": "Отменить",
|
||||||
"import.cancelling": "Отмена — удаляю недописанный файл…",
|
"import.cancelling": "Отмена — удаляю недописанный файл…",
|
||||||
"import.delete": "Удалить",
|
"import.delete": "Удалить",
|
||||||
@@ -395,21 +384,12 @@ export default {
|
|||||||
"import.state.done": "готово",
|
"import.state.done": "готово",
|
||||||
"import.state.error": "ошибка",
|
"import.state.error": "ошибка",
|
||||||
"import.state.cancelled": "отменено",
|
"import.state.cancelled": "отменено",
|
||||||
"import.featuredTitle": "Рекомендуемые готовые .cmf модели",
|
|
||||||
"import.readyCmfBadge": "Готовая .cmf",
|
|
||||||
"import.directDownload": "Скачать",
|
|
||||||
"import.q.q8_2f": "8-бит двухполевой (𝒲×θ) — рекомендуется: лучшее качество/размер.",
|
"import.q.q8_2f": "8-бит двухполевой (𝒲×θ) — рекомендуется: лучшее качество/размер.",
|
||||||
"import.q.q8_row": "8-бит построчно — просто, надёжно.",
|
"import.q.q8_row": "8-бит построчно — просто, надёжно.",
|
||||||
"import.q.q4": "4-бит блочно — компактнее, ниже качество.",
|
"import.q.q4": "4-бит блочно — компактнее, ниже качество.",
|
||||||
"import.q.q4_tiled": "4-бит тайловый (Q4_TILED) — х1.66 быстрее на Metal GPU.",
|
|
||||||
"import.q.vbit": "Переменная 3–8 бит — per-expert бюджеты.",
|
"import.q.vbit": "Переменная 3–8 бит — per-expert бюджеты.",
|
||||||
"import.q.q1t": "1.58-бит тернарный (Q1T) — BitNet b1.58 с оутлайерами.",
|
|
||||||
"import.q.q1s": "1-бит спарс (Q1S) — 1-бит база с голографическими оутлайерами.",
|
|
||||||
"import.q.q1p": "1-бит PTQ (Q1P) — 1-бит с переносом ошибки (error diffusion).",
|
|
||||||
"import.q.q1": "1-бит (Q1) — чистый бинарный формат.",
|
|
||||||
"import.q.f16": "16-бит — без квантования.",
|
"import.q.f16": "16-бит — без квантования.",
|
||||||
"import.q.f32": "32-бит — полная точность (большой файл).",
|
"import.q.f32": "32-бит — полная точность (большой файл).",
|
||||||
|
|
||||||
"import.adv.title": "Расширенные настройки",
|
"import.adv.title": "Расширенные настройки",
|
||||||
"import.adv.auto": "авто",
|
"import.adv.auto": "авто",
|
||||||
"import.adv.linearCore": "Линейное ядро",
|
"import.adv.linearCore": "Линейное ядро",
|
||||||
|
|||||||
Vendored
-4
@@ -23,10 +23,6 @@ export function clear(node) {
|
|||||||
while (node && node.firstChild) node.removeChild(node.firstChild);
|
while (node && node.firstChild) node.removeChild(node.firstChild);
|
||||||
}
|
}
|
||||||
|
|
||||||
export function opt(v, label, sel) {
|
|
||||||
return h("option", { value: v, selected: sel ? true : null }, label || v);
|
|
||||||
}
|
|
||||||
|
|
||||||
export function mount(node, ...children) {
|
export function mount(node, ...children) {
|
||||||
clear(node);
|
clear(node);
|
||||||
for (const c of children.flat())
|
for (const c of children.flat())
|
||||||
|
|||||||
Vendored
+4
-55
@@ -2,7 +2,7 @@
|
|||||||
// live progress view. Search HF, pick quantization + advanced params, watch
|
// live progress view. Search HF, pick quantization + advanced params, watch
|
||||||
// the conversion (multiple in parallel, resumable across tab switches), then
|
// the conversion (multiple in parallel, resumable across tab switches), then
|
||||||
// register the result as a local model.
|
// register the result as a local model.
|
||||||
import { h, mount, toast, opt } from "../ui.js";
|
import { h, mount, toast } from "../ui.js";
|
||||||
import { t } from "../i18n.js";
|
import { t } from "../i18n.js";
|
||||||
import { api } from "../api.js";
|
import { api } from "../api.js";
|
||||||
|
|
||||||
@@ -10,27 +10,18 @@ const QUANTS = [
|
|||||||
["Q8_2F", "import.q.q8_2f"],
|
["Q8_2F", "import.q.q8_2f"],
|
||||||
["Q8_ROW", "import.q.q8_row"],
|
["Q8_ROW", "import.q.q8_row"],
|
||||||
["Q4_BLOCK", "import.q.q4"],
|
["Q4_BLOCK", "import.q.q4"],
|
||||||
["Q4_TILED", "import.q.q4_tiled"],
|
|
||||||
["VBIT", "import.q.vbit"],
|
["VBIT", "import.q.vbit"],
|
||||||
["Q1T", "import.q.q1t"],
|
|
||||||
["Q1S", "import.q.q1s"],
|
|
||||||
["Q1P", "import.q.q1p"],
|
|
||||||
["Q1", "import.q.q1"],
|
|
||||||
["F16", "import.q.f16"],
|
["F16", "import.q.f16"],
|
||||||
["F32", "import.q.f32"],
|
["F32", "import.q.f32"],
|
||||||
];
|
];
|
||||||
|
|
||||||
const FEATURED_REPOS = [
|
|
||||||
{ id: "infosave/Bonsai-1.7Bcmf", label: "Bonsai 1.7B CMF", badge: "1.7B · Q1", quant: "Q1" },
|
|
||||||
{ id: "infosave/Bonsai-8B_2bit_cmf", label: "Bonsai 8B 2-bit CMF", badge: "8B · Q1T", quant: "Q1T" },
|
|
||||||
{ id: "infosave/Bonsai-27Bcmf", label: "Bonsai 27B CMF", badge: "27B · Q1", quant: "Q1" },
|
|
||||||
];
|
|
||||||
|
|
||||||
|
|
||||||
function field(label, control, hint) {
|
function field(label, control, hint) {
|
||||||
return h("label", { class: "field" }, h("span", {}, label), control,
|
return h("label", { class: "field" }, h("span", {}, label), control,
|
||||||
hint ? h("div", { class: "hint" }, hint) : null);
|
hint ? h("div", { class: "hint" }, hint) : null);
|
||||||
}
|
}
|
||||||
|
function opt(v, label, sel) {
|
||||||
|
return h("option", { value: v, selected: sel ? true : null }, label || v);
|
||||||
|
}
|
||||||
function fmtNum(n) {
|
function fmtNum(n) {
|
||||||
if (n == null) return "—";
|
if (n == null) return "—";
|
||||||
if (n >= 1e6) return (n / 1e6).toFixed(1) + "M";
|
if (n >= 1e6) return (n / 1e6).toFixed(1) + "M";
|
||||||
@@ -367,46 +358,6 @@ export function renderImport() {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
function featuredCard(item) {
|
|
||||||
const card = h(
|
|
||||||
"div",
|
|
||||||
{ class: "hf-card featured-card" },
|
|
||||||
h("div", { class: "row", style: "align-items:center; justify-content:space-between;" },
|
|
||||||
h("div", {},
|
|
||||||
h("div", { class: "hf-id" }, item.label),
|
|
||||||
h("div", { class: "hf-meta" },
|
|
||||||
h("span", { class: "badge" }, item.badge),
|
|
||||||
h("span", { class: "muted" }, item.id)
|
|
||||||
)
|
|
||||||
),
|
|
||||||
h("button", { class: "btn primary sm" }, "⬇ " + t("import.directDownload"))
|
|
||||||
)
|
|
||||||
);
|
|
||||||
card.querySelector("button").addEventListener("click", async (e) => {
|
|
||||||
e.stopPropagation();
|
|
||||||
try {
|
|
||||||
await api.startImport({
|
|
||||||
repo: item.id,
|
|
||||||
quant: item.quant || "READY_CMF",
|
|
||||||
name: item.id.split("/").pop().toLowerCase(),
|
|
||||||
});
|
|
||||||
toast(t("import.started"), "ok");
|
|
||||||
refreshJobs();
|
|
||||||
jobsHost.scrollIntoView({ behavior: "smooth", block: "nearest" });
|
|
||||||
} catch (err) {
|
|
||||||
toast(err.message, "err");
|
|
||||||
}
|
|
||||||
});
|
|
||||||
return card;
|
|
||||||
}
|
|
||||||
|
|
||||||
const featuredHost = h(
|
|
||||||
"div",
|
|
||||||
{ class: "card featured-section" },
|
|
||||||
h("div", { class: "card-title" }, "⚡ " + t("import.featuredTitle")),
|
|
||||||
h("div", { class: "hf-grid" }, ...FEATURED_REPOS.map(featuredCard))
|
|
||||||
);
|
|
||||||
|
|
||||||
const root = h(
|
const root = h(
|
||||||
"div",
|
"div",
|
||||||
{},
|
{},
|
||||||
@@ -414,7 +365,6 @@ export function renderImport() {
|
|||||||
h("h2", {}, t("nav.import")),
|
h("h2", {}, t("nav.import")),
|
||||||
h("div", { class: "muted" }, t("import.subtitle"))),
|
h("div", { class: "muted" }, t("import.subtitle"))),
|
||||||
jobsHost, // progress panel on top, above the search
|
jobsHost, // progress panel on top, above the search
|
||||||
featuredHost, // featured ready-to-run models
|
|
||||||
h("div", { class: "card" }, searchIn, results),
|
h("div", { class: "card" }, searchIn, results),
|
||||||
formHost
|
formHost
|
||||||
);
|
);
|
||||||
@@ -422,4 +372,3 @@ export function renderImport() {
|
|||||||
refreshJobs(); // resume: show any running/recent conversions immediately
|
refreshJobs(); // resume: show any running/recent conversions immediately
|
||||||
return root;
|
return root;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
Vendored
+4
-39
@@ -7,6 +7,9 @@ import { appState } from "../app.js";
|
|||||||
function field(label, control, hint) {
|
function field(label, control, hint) {
|
||||||
return h("label", { class: "field" }, h("span", {}, label), control, hint ? h("div", { class: "hint" }, hint) : null);
|
return h("label", { class: "field" }, h("span", {}, label), control, hint ? h("div", { class: "hint" }, hint) : null);
|
||||||
}
|
}
|
||||||
|
function opt(v, label, sel) {
|
||||||
|
return h("option", { value: v, selected: sel ? true : null }, label || v);
|
||||||
|
}
|
||||||
|
|
||||||
function keyBadge(src) {
|
function keyBadge(src) {
|
||||||
const map = { store: "store", env: "env", missing: "bad", none: "" };
|
const map = { store: "store", env: "env", missing: "bad", none: "" };
|
||||||
@@ -115,36 +118,6 @@ function modelForm(meta, existing) {
|
|||||||
: existing && existing.key_source === "env" ? t("models.form.keyFromEnv")
|
: existing && existing.key_source === "env" ? t("models.form.keyFromEnv")
|
||||||
: t("models.form.apiKeyHint");
|
: t("models.form.apiKeyHint");
|
||||||
|
|
||||||
// ⚙ Generation & Thinking Parameters
|
|
||||||
const tempIn = h("input", { type: "number", step: "0.05", min: "0", max: "2.0", placeholder: "0.7", value: m.temperature ?? "" });
|
|
||||||
const topPIn = h("input", { type: "number", step: "0.05", min: "0", max: "1.0", placeholder: "0.9", value: m.top_p ?? "" });
|
|
||||||
const maxTokIn = h("input", { type: "number", min: "1", placeholder: "2048", value: m.max_tokens ?? "" });
|
|
||||||
const thinkIn = h("input", { type: "number", min: "0", placeholder: "1024", value: m.think_budget ?? "" });
|
|
||||||
const o1Sel = h("select", {},
|
|
||||||
h("option", { value: "" }, t("import.adv.auto")),
|
|
||||||
h("option", { value: "all" }, "all — O(1) Attention"),
|
|
||||||
h("option", { value: "deep" }, "deep — O(1) Deep"),
|
|
||||||
h("option", { value: "off" }, "off — Standard"));
|
|
||||||
if (m.o1) o1Sel.value = m.o1;
|
|
||||||
const skipMtpCb = h("input", { type: "checkbox", checked: m.skip_mtp ? true : null });
|
|
||||||
const sysPromptIn = h("textarea", { rows: 2, placeholder: t("models.form.systemPromptPlaceholder"), value: m.system_prompt || "" });
|
|
||||||
|
|
||||||
const advParams = h(
|
|
||||||
"details",
|
|
||||||
{ class: "adv-params", style: "margin-top:12px;padding:10px;border:1px solid var(--border);border-radius:6px;" },
|
|
||||||
h("summary", { style: "font-weight:600;cursor:pointer;" }, "⚙ " + t("models.form.paramsTitle")),
|
|
||||||
h("div", { class: "row", style: "margin-top:8px" },
|
|
||||||
field(t("models.form.temperature"), tempIn, t("models.form.temperatureHint")),
|
|
||||||
field(t("models.form.topP"), topPIn, t("models.form.topPHint"))),
|
|
||||||
h("div", { class: "row" },
|
|
||||||
field(t("models.form.maxTokens"), maxTokIn, t("models.form.maxTokensHint")),
|
|
||||||
field(t("models.form.thinkBudget"), thinkIn, t("models.form.thinkBudgetHint"))),
|
|
||||||
h("div", { class: "row" },
|
|
||||||
field("O(1) Attention", o1Sel, t("import.adv.o1Hint")),
|
|
||||||
field("MTP", h("label", { class: "check" }, skipMtpCb, t("import.adv.skipMtp")))),
|
|
||||||
field(t("models.form.systemPrompt"), sysPromptIn, t("models.form.systemPromptHint"))
|
|
||||||
);
|
|
||||||
|
|
||||||
const node = h(
|
const node = h(
|
||||||
"div",
|
"div",
|
||||||
{},
|
{},
|
||||||
@@ -155,8 +128,7 @@ function modelForm(meta, existing) {
|
|||||||
dl,
|
dl,
|
||||||
h("div", { class: "row" }, field(t("models.form.kind"), kindSel), field(t("models.form.costTier"), tierSel)),
|
h("div", { class: "row" }, field(t("models.form.kind"), kindSel), field(t("models.form.costTier"), tierSel)),
|
||||||
h("div", { class: "row" }, field(t("models.form.priceIn"), priceInIn), field(t("models.form.priceOut"), priceOutIn)),
|
h("div", { class: "row" }, field(t("models.form.priceIn"), priceInIn), field(t("models.form.priceOut"), priceOutIn)),
|
||||||
field(t("models.form.caps"), h("div", { class: "flex wrap" }, ...capInputs.map(({ c, cb }) => h("label", { class: "check" }, cb, c)))),
|
field(t("models.form.caps"), h("div", { class: "flex wrap" }, ...capInputs.map(({ c, cb }) => h("label", { class: "check" }, cb, c))))
|
||||||
advParams
|
|
||||||
);
|
);
|
||||||
|
|
||||||
const getValue = () => {
|
const getValue = () => {
|
||||||
@@ -175,13 +147,6 @@ function modelForm(meta, existing) {
|
|||||||
price_out: parseFloat(priceOutIn.value) || 0,
|
price_out: parseFloat(priceOutIn.value) || 0,
|
||||||
api_key_env: hasKey ? keyEnv : null,
|
api_key_env: hasKey ? keyEnv : null,
|
||||||
caps: capInputs.filter(({ cb }) => cb.checked).map(({ c }) => c),
|
caps: capInputs.filter(({ cb }) => cb.checked).map(({ c }) => c),
|
||||||
temperature: tempIn.value !== "" ? parseFloat(tempIn.value) : null,
|
|
||||||
top_p: topPIn.value !== "" ? parseFloat(topPIn.value) : null,
|
|
||||||
max_tokens: maxTokIn.value !== "" ? parseInt(maxTokIn.value) : null,
|
|
||||||
think_budget: thinkIn.value !== "" ? parseInt(thinkIn.value) : null,
|
|
||||||
o1: o1Sel.value || null,
|
|
||||||
skip_mtp: skipMtpCb.checked,
|
|
||||||
system_prompt: sysPromptIn.value.trim() || null,
|
|
||||||
},
|
},
|
||||||
secret: typedKey,
|
secret: typedKey,
|
||||||
keyEnv,
|
keyEnv,
|
||||||
|
|||||||
Vendored
+7
-97
@@ -245,7 +245,11 @@ export async function renderSettings() {
|
|||||||
try {
|
try {
|
||||||
const r = await api.cmfPortCheck(p);
|
const r = await api.cmfPortCheck(p);
|
||||||
if (r.available) mount(st, h("span", { style: "color:var(--good)" }, "✓ " + t("settings.cmf.portFree")));
|
if (r.available) mount(st, h("span", { style: "color:var(--good)" }, "✓ " + t("settings.cmf.portFree")));
|
||||||
else {
|
else if (r.managed) {
|
||||||
|
// held by this gateway's own managed server — replacing the model on
|
||||||
|
// this port is fine, it frees up on restart
|
||||||
|
mount(st, h("span", { style: "color:var(--accent)" }, "ℹ " + t("settings.cmf.portOwn")));
|
||||||
|
} else {
|
||||||
const kids = [h("span", { style: "color:var(--bad)" }, "✗ " + t("settings.cmf.portBusy"))];
|
const kids = [h("span", { style: "color:var(--bad)" }, "✗ " + t("settings.cmf.portBusy"))];
|
||||||
if (r.suggested) { const b = h("button", { class: "btn small", type: "button" }, t("settings.cmf.useSuggested", { port: r.suggested })); b.addEventListener("click", () => { portIn.value = r.suggested; mount(st, ""); }); kids.push(b); }
|
if (r.suggested) { const b = h("button", { class: "btn small", type: "button" }, t("settings.cmf.useSuggested", { port: r.suggested })); b.addEventListener("click", () => { portIn.value = r.suggested; mount(st, ""); }); kids.push(b); }
|
||||||
mount(st, h("span", {}, ...kids));
|
mount(st, h("span", {}, ...kids));
|
||||||
@@ -253,103 +257,9 @@ export async function renderSettings() {
|
|||||||
} catch (e) { mount(st, String(e.message || e)); }
|
} catch (e) { mount(st, String(e.message || e)); }
|
||||||
checkBtn.disabled = false;
|
checkBtn.disabled = false;
|
||||||
});
|
});
|
||||||
|
|
||||||
// ── O(1) Nyström Attention ──
|
|
||||||
// NOT a "reasoning mode". This is a memory-efficient streaming attention
|
|
||||||
// kernel (Nyström/landmark) that replaces standard KV-cache attention,
|
|
||||||
// giving O(1) memory per decode step instead of O(n).
|
|
||||||
const o1Sel = h("select", { style: "width:120px", title: "O(1) Nyström attention layer spec: which layers replace KV-cache with streaming Nyström attention" },
|
|
||||||
h("option", { value: "" }, "O(1) авто"),
|
|
||||||
h("option", { value: "all" }, "O(1) все слои"),
|
|
||||||
h("option", { value: "deep" }, "O(1) глубокие"),
|
|
||||||
h("option", { value: "off" }, "O(1) выкл."));
|
|
||||||
if (sv.o1) o1Sel.value = sv.o1;
|
|
||||||
const o1mIn = h("input", { type: "number", min: "4", placeholder: "32", value: sv.o1_m ?? "", style: "width:55px", title: "Landmark budget (≥4, default 32). More landmarks ≠ better: m=64 measured worse." });
|
|
||||||
const o1wIn = h("input", { type: "number", min: "1", placeholder: "128", value: sv.o1_window ?? "", style: "width:60px", title: "Exact-window width — main quality lever (default 128)" });
|
|
||||||
const o1sinkIn = h("input", { type: "number", min: "0", placeholder: "4", value: sv.o1_sink ?? "", style: "width:50px", title: "Permanent exact sink keys, StreamingLLM discipline (default 4)" });
|
|
||||||
|
|
||||||
// ── Generation parameters ──
|
|
||||||
const tempIn = h("input", { type: "number", step: "0.05", min: "0", max: "2.0", placeholder: "0.7", value: sv.temperature ?? "", style: "width:60px" });
|
|
||||||
const topPIn = h("input", { type: "number", step: "0.05", min: "0", max: "1.0", placeholder: "0.9", value: sv.top_p ?? "", style: "width:60px" });
|
|
||||||
const maxTokIn = h("input", { type: "number", min: "1", placeholder: "2048", value: sv.max_tokens ?? "", style: "width:75px" });
|
|
||||||
// think_budget = 0 means reasoning disabled (enable_thinking=false);
|
|
||||||
// null = model decides; N>0 = token budget for the <think> block.
|
|
||||||
const noThinkCb = h("input", { type: "checkbox", checked: (sv.think_budget === 0) ? true : null, title: "Отключить режим размышлений (think_budget=0, enable_thinking=false для Qwen3/3.5)" });
|
|
||||||
const thinkIn = h("input", { type: "number", min: "1", placeholder: "авто", value: (sv.think_budget && sv.think_budget > 0) ? sv.think_budget : "", style: "width:75px", title: "Бюджет токенов на размышления (think_budget). Пустое = модель решает сама." });
|
|
||||||
if (sv.think_budget === 0) thinkIn.disabled = true;
|
|
||||||
noThinkCb.addEventListener("change", () => {
|
|
||||||
thinkIn.disabled = noThinkCb.checked;
|
|
||||||
if (noThinkCb.checked) thinkIn.value = "";
|
|
||||||
});
|
|
||||||
const sysIn = h("input", { placeholder: "System prompt (optional)", value: sv.system_prompt || "", style: "flex:1;min-width:200px" });
|
|
||||||
|
|
||||||
// ── MTP (Multi-Token Prediction): speculative decoding, NOT reasoning ──
|
|
||||||
const skipMtpCb = h("input", { type: "checkbox", checked: sv.skip_mtp ? true : null, title: "Disable MTP speculative decoding (env CMF_MTP=0)" });
|
|
||||||
|
|
||||||
// ── Settings drawer ──
|
|
||||||
const drawerStyle = "display:none;width:100%;margin-top:6px;padding:10px 12px;background:var(--bg-subtle);border-radius:6px;border:1px solid var(--border)";
|
|
||||||
const groupStyle = "display:flex;flex-wrap:wrap;gap:8px;align-items:center;padding:4px 0";
|
|
||||||
const labelStyle = "font-size:11px;color:var(--text-muted);font-weight:600;text-transform:uppercase;letter-spacing:0.5px;width:100%;margin-bottom:2px";
|
|
||||||
|
|
||||||
const drawer = h("div", { style: drawerStyle },
|
|
||||||
// Group 1: O(1) Nyström Attention
|
|
||||||
h("div", { style: labelStyle }, "O(1) Nyström Attention"),
|
|
||||||
h("div", { style: groupStyle },
|
|
||||||
h("span", { class: "small muted" }, "Слои:"), o1Sel,
|
|
||||||
h("span", { class: "small muted" }, "m:"), o1mIn,
|
|
||||||
h("span", { class: "small muted" }, "Window:"), o1wIn,
|
|
||||||
h("span", { class: "small muted" }, "Sink:"), o1sinkIn
|
|
||||||
),
|
|
||||||
h("hr", { style: "border:none;border-top:1px solid var(--border);margin:6px 0" }),
|
|
||||||
// Group 2: Generation
|
|
||||||
h("div", { style: labelStyle }, "Генерация"),
|
|
||||||
h("div", { style: groupStyle },
|
|
||||||
h("span", { class: "small muted" }, "Temp:"), tempIn,
|
|
||||||
h("span", { class: "small muted" }, "Top-P:"), topPIn,
|
|
||||||
h("span", { class: "small muted" }, "Max токенов:"), maxTokIn
|
|
||||||
),
|
|
||||||
h("div", { style: groupStyle },
|
|
||||||
h("span", { class: "small muted" }, "System prompt:"), sysIn
|
|
||||||
),
|
|
||||||
h("hr", { style: "border:none;border-top:1px solid var(--border);margin:6px 0" }),
|
|
||||||
// Group 3: Reasoning (thinking)
|
|
||||||
h("div", { style: labelStyle }, "Режим размышлений"),
|
|
||||||
h("div", { style: groupStyle },
|
|
||||||
h("label", { class: "check small", title: "Для моделей Qwen3/3.5: enable_thinking=false — модель отвечает напрямую без блока <think>" }, noThinkCb, "Отключить размышления"),
|
|
||||||
h("span", { class: "small muted" }, "Think budget:"), thinkIn
|
|
||||||
),
|
|
||||||
h("hr", { style: "border:none;border-top:1px solid var(--border);margin:6px 0" }),
|
|
||||||
// Group 4: Runtime
|
|
||||||
h("div", { style: labelStyle }, "Runtime"),
|
|
||||||
h("div", { style: groupStyle },
|
|
||||||
h("label", { class: "check small", title: "MTP — Multi-Token Prediction (спекулятивное декодирование). Отключение замедляет генерацию, но стабилизирует output." }, skipMtpCb, "Отключить MTP")
|
|
||||||
)
|
|
||||||
);
|
|
||||||
|
|
||||||
const gearBtn = h("button", { class: "btn small", type: "button", title: "⚙ Настройки модели" }, "⚙");
|
|
||||||
gearBtn.addEventListener("click", () => {
|
|
||||||
drawer.style.display = drawer.style.display === "none" ? "block" : "none";
|
|
||||||
});
|
|
||||||
|
|
||||||
const row = h("div", { class: "flex wrap", style: "gap:6px;align-items:center;margin:6px 0" },
|
const row = h("div", { class: "flex wrap", style: "gap:6px;align-items:center;margin:6px 0" },
|
||||||
idIn, modelSel, portIn, checkBtn, threadsIn, h("label", { class: "flex", style: "gap:4px" }, gpuIn, "GPU"), gearBtn, removeBtn, st, drawer);
|
idIn, modelSel, portIn, checkBtn, threadsIn, h("label", { class: "flex", style: "gap:4px" }, gpuIn, "GPU"), removeBtn, st);
|
||||||
const entry = { node: row, read: () => ({
|
const entry = { node: row, read: () => ({ id: idIn.value.trim() || autoId(modelSel.value), model: modelSel.value.trim(), port: parseInt(portIn.value) || 8090, threads: parseInt(threadsIn.value) || 0, gpu: gpuIn.checked }) };
|
||||||
id: idIn.value.trim() || autoId(modelSel.value),
|
|
||||||
model: modelSel.value.trim(),
|
|
||||||
port: parseInt(portIn.value) || 8090,
|
|
||||||
threads: parseInt(threadsIn.value) || 0,
|
|
||||||
gpu: gpuIn.checked,
|
|
||||||
temperature: tempIn.value !== "" ? parseFloat(tempIn.value) : null,
|
|
||||||
top_p: topPIn.value !== "" ? parseFloat(topPIn.value) : null,
|
|
||||||
max_tokens: maxTokIn.value !== "" ? parseInt(maxTokIn.value) : null,
|
|
||||||
think_budget: noThinkCb.checked ? 0 : (thinkIn.value !== "" ? parseInt(thinkIn.value) : null),
|
|
||||||
o1: o1Sel.value || null,
|
|
||||||
o1_m: o1mIn.value !== "" ? parseInt(o1mIn.value) : null,
|
|
||||||
o1_window: o1wIn.value !== "" ? parseInt(o1wIn.value) : null,
|
|
||||||
o1_sink: o1sinkIn.value !== "" ? parseInt(o1sinkIn.value) : null,
|
|
||||||
skip_mtp: skipMtpCb.checked,
|
|
||||||
system_prompt: sysIn.value.trim() || null,
|
|
||||||
}) };
|
|
||||||
removeBtn.addEventListener("click", () => { const i = serverRows.indexOf(entry); if (i >= 0) serverRows.splice(i, 1); row.remove(); });
|
removeBtn.addEventListener("click", () => { const i = serverRows.indexOf(entry); if (i >= 0) serverRows.splice(i, 1); row.remove(); });
|
||||||
serverRows.push(entry);
|
serverRows.push(entry);
|
||||||
serversWrap.appendChild(row);
|
serversWrap.appendChild(row);
|
||||||
|
|||||||
Reference in New Issue
Block a user