Compare commits

...
6 Commits
Author SHA1 Message Date
infosave2007 8e369f99af feat(playground): top-p and a thoughts switch (auto/off/on)
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
Playground gains top-p and a reasoning selector matching the Local-models row:
auto follows the model/row default; off/on override per request via
enable_thinking (cortiq >= 0.2.1 for local models). /admin/api/test(+stream)
accept top_p and enable_thinking, proxied through GenParams passthrough.
2026-07-14 16:13:19 +03:00
infosave2007 4754e1a065 feat(cmf): per-model reasoning switch — 'thoughts: off' for thinking models
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
Each Local-models row gains a thoughts selector (auto/off). Off injects
enable_thinking=false (cortiq >= 0.2.1 renders the chat template with an empty
<think> block) into requests that don't set it themselves — the model answers
directly, faster and shorter. A client-supplied enable_thinking always wins.
2026-07-14 15:55:43 +03:00
infosave2007 3c4be5e966 feat(cmf): per-model generation defaults; safer delete-file default
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
- Each Local-models row gains default generation params — temperature, top-p,
  max tokens — applied by the gateway when the incoming request doesn't set
  them (an explicit request value always wins). Works for both the plain and
  streaming paths; top_p is now forwarded to OpenAI-compatible providers.
- The "also delete the model file from disk" checkbox in the delete dialog now
  defaults to UNCHECKED — deleting a pool entry no longer silently costs a
  multi-GB re-download/conversion.
2026-07-14 15:22:26 +03:00
infosave2007 46c11b9b88 feat(cmf): live-apply managed-server changes; auto-resolve import id collisions
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
- Saving the Local models list now applies IMMEDIATELY: removed or redefined
  rows stop their `cortiq serve` child (memory freed on the spot), new or
  changed rows are spawned and health-checked — no gateway restart needed.
- A managed row that shares its id with a same-id STATIC pool entry pointing at
  a local server (the shape Import→register creates) now supersedes it: the
  static entry is dropped automatically instead of failing the save with
  "duplicate managed model id".
- validate() now tells the two duplicate cases apart with actionable messages
  (two managed rows vs managed-vs-pool collision).
2026-07-14 14:34:30 +03:00
infosave2007 851cc7f1cb feat(cmf): per-server O(1) attention override at serve time
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
Each managed local model row gains an O(1) selector (auto / off / all / deepN),
exported to the child `cortiq serve` as CMF_O1. "off" forces exact attention,
curing incoherent output from a too-aggressive `--o1` conversion hint without
re-converting the file; "all"/"deepN" force-enable it. Auto (default) follows
the hint recorded in the .cmf. Surfaced in /admin/api/cmf per-server status.
2026-07-14 12:32:51 +03:00
infosave2007 d93ac537c8 docs(import): human-readable labels + hover help for the O(1) knobs
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
Landmarks / Exact window / Sink keys now carry a '?' tooltip explaining in
plain language what each knob does, its trade-off, and the validated default
(32 / 128 / 4), in EN + RU.
2026-07-14 11:19:00 +03:00
13 changed files with 307 additions and 37 deletions
Generated
+1 -1
View File
@@ -285,7 +285,7 @@ checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570"
[[package]]
name = "cortiq-gateway"
version = "0.2.17"
version = "0.2.23"
dependencies = [
"anyhow",
"arc-swap",
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "cortiq-gateway"
version = "0.2.17"
version = "0.2.23"
edition = "2021"
rust-version = "1.75"
license = "Apache-2.0"
+56 -22
View File
@@ -908,28 +908,44 @@ async fn put_settings(State(state): State<SharedState>, Json(b): Json<SettingsBo
needs_restart = true;
cfg.cache = v;
}
let mut cmf_apply: Option<(Vec<crate::config::CmfServer>, CmfCfg)> = None;
if let Some(v) = b.cmf {
// manage_server / local_model / threads / gpu apply on restart (the server
// is spawned once at startup with those env vars); local_only / router
// routing apply immediately.
if v.manage_server != cfg.cmf.manage_server
|| v.local_model != cfg.cmf.local_model
|| v.threads != cfg.cmf.threads
|| v.gpu != cfg.cmf.gpu
|| v.servers != cfg.cmf.servers
{
needs_restart = true;
}
let old_servers = cfg.cmf.effective_servers();
cfg.cmf = v;
// A managed row supersedes a same-id STATIC pool entry that points at a
// local server (the shape Import→register creates): drop the static one
// instead of failing validation with "duplicate managed model id".
let managed_ids: std::collections::HashSet<String> = cfg
.cmf
.effective_servers()
.into_iter()
.map(|s| s.id)
.collect();
cfg.models.retain(|m| {
!(managed_ids.contains(&m.id)
&& (m.base_url.contains("//127.0.0.1:") || m.base_url.contains("//localhost:")))
});
// Replacing or renaming a managed model must not brick the save: routing
// may still reference the old id. Map a vanished id to its replacement
// (the new server on the same port), strip ids that are truly gone, and
// repair the default so validate() passes.
heal_routing_after_cmf_change(&mut cfg, &old_servers);
// Server-list changes apply LIVE after a successful reload: removed /
// redefined servers are stopped (RAM freed), new ones are spawned.
if cfg.cmf.effective_servers() != old_servers {
cmf_apply = Some((old_servers, cfg.cmf.clone()));
}
}
state.reload(cfg)?;
ok(json!({ "ok": true, "needs_restart": needs_restart }))
let applied_live = cmf_apply.is_some();
if let Some((old, new_cfg)) = cmf_apply {
tokio::spawn(crate::cmf_runtime::apply_server_changes(
state.cmf.clone(),
old,
new_cfg,
));
}
ok(json!({ "ok": true, "needs_restart": needs_restart, "applied_live": applied_live }))
}
/// After the managed-servers list changed, rewrite [routing] so it only
@@ -1009,6 +1025,7 @@ async fn cmf_status(State(state): State<SharedState>) -> ApiResult {
"port": s.port,
"threads": s.threads,
"gpu": s.gpu,
"o1": s.o1,
"running": live.map(|l| l.running).unwrap_or(false),
"healthy": live.map(|l| l.healthy).unwrap_or(false),
"last_error": live.and_then(|l| l.last_error.clone()),
@@ -1430,24 +1447,44 @@ struct TestBody {
temperature: Option<f32>,
#[serde(default)]
max_tokens: Option<u32>,
#[serde(default)]
top_p: Option<f32>,
/// Reasoning switch (cortiq ≥ 0.2.1): false = answer without the thinking
/// phase; true = force it on. Absent = model/row default.
#[serde(default)]
enable_thinking: Option<bool>,
}
fn default_test_model() -> String {
"cortiq-auto".into()
}
/// GenParams for a playground request: native sampling fields + the reasoning
/// switch proxied via passthrough (providers forward unknown fields as-is).
fn test_gen_params(b: &TestBody) -> GenParams {
let mut params = GenParams {
temperature: b.temperature,
max_tokens: b.max_tokens,
top_p: b.top_p,
..Default::default()
};
if let Some(v) = b.enable_thinking {
params
.passthrough
.insert("enable_thinking".into(), serde_json::Value::Bool(v));
}
params
}
async fn run_test(State(state): State<SharedState>, Json(b): Json<TestBody>) -> ApiResult {
if b.messages.is_empty() {
return Err(ApiError::bad("messages must not be empty"));
}
let params = test_gen_params(&b);
let req = ChatRequest {
routing: parse_routing(&b.model),
messages: b.messages,
tools: vec![],
params: GenParams {
temperature: b.temperature,
max_tokens: b.max_tokens,
..Default::default()
},
params,
stream: false,
meta: RequestMeta {
account: "playground".into(),
@@ -1496,15 +1533,12 @@ async fn run_test_stream(State(state): State<SharedState>, Json(b): Json<TestBod
if b.messages.is_empty() {
return ApiError::bad("messages must not be empty").into_response();
}
let params = test_gen_params(&b);
let req = ChatRequest {
routing: parse_routing(&b.model),
messages: b.messages,
tools: vec![],
params: GenParams {
temperature: b.temperature,
max_tokens: b.max_tokens,
..Default::default()
},
params,
stream: true,
meta: RequestMeta {
account: "playground".into(),
+86
View File
@@ -173,6 +173,31 @@ impl CmfRuntime {
}
}
/// Ensure a status slot exists for `server` (used when servers are added at
/// runtime); refreshes model/port if the slot already exists.
fn add_server_slot(&self, server: &CmfServer, host: &str) {
let mut s = self.status.lock().unwrap();
s.enabled = true;
if let Some(sv) = s.servers.iter_mut().find(|x| x.id == server.id) {
sv.model = server.model.clone();
sv.port = server.port;
sv.base_url = format!("http://{host}:{}/v1", server.port);
sv.running = false;
sv.healthy = false;
sv.last_error = None;
} else {
s.servers.push(CmfServerStatus {
id: server.id.clone(),
model: server.model.clone(),
port: server.port,
base_url: format!("http://{host}:{}/v1", server.port),
running: false,
healthy: false,
last_error: None,
});
}
}
/// Stop and forget a single managed server (used when a model is deleted).
pub async fn stop_one(&self, id: &str) {
let removed = {
@@ -295,6 +320,11 @@ fn spawn_serve(server: &CmfServer, cfg: &CmfCfg) -> Result<Child, String> {
if server.gpu {
cmd.env("CMF_GPU", "1");
}
// Serve-time O(1) attention override: "off" forces exact attention (undoes a
// too-aggressive --o1 conversion hint without re-converting the file).
if !server.o1.trim().is_empty() {
cmd.env("CMF_O1", server.o1.trim());
}
cmd.stdout(Stdio::null())
.stderr(Stdio::null())
.kill_on_drop(true)
@@ -322,6 +352,62 @@ async fn wait_healthy(host: &str, port: u16, tries: u32) -> bool {
false
}
/// Apply a changed managed-servers list LIVE (no gateway restart): stop servers
/// that were removed or redefined — freeing their memory immediately — and
/// spawn servers that are new or redefined.
pub async fn apply_server_changes(rt: Arc<CmfRuntime>, old: Vec<CmfServer>, cfg: CmfCfg) {
let new = cfg.effective_servers();
// 1. Stop what's gone or changed (kills the child → RAM released).
for o in &old {
let unchanged = new.iter().any(|n| n == o);
if !unchanged {
rt.stop_one(&o.id).await;
rt.log(format!("stopped '{}' (removed or changed)", o.id));
}
}
if new.is_empty() {
return;
}
// 2. Spawn what's new or changed.
if installed_version(&cfg.cortiq_bin).is_none() {
rt.fail("cortiq binary not found — cannot start local model servers");
return;
}
for n in &new {
let unchanged = old.iter().any(|o| o == n);
if unchanged {
continue; // already running with the same definition
}
rt.add_server_slot(n, &cfg.local_host);
if !std::path::Path::new(&n.model).exists() {
rt.fail_server(&n.id, format!("model file not found: {}", n.model));
continue;
}
rt.log(format!(
"starting: cortiq serve {} --host {} --port {}",
n.model, cfg.local_host, n.port
));
match spawn_serve(n, &cfg) {
Ok(child) => rt.add_child(n.id.clone(), child),
Err(e) => {
rt.fail_server(&n.id, format!("failed to spawn: {e}"));
continue;
}
}
let (rtc, host, id, port) = (rt.clone(), cfg.local_host.clone(), n.id.clone(), n.port);
tokio::spawn(async move {
if wait_healthy(&host, port, 240).await {
rtc.set_healthy(&id, true);
rtc.log(format!(
"local CMF server ready — '{id}' at http://{host}:{port}/v1"
));
} else {
rtc.fail_server(&id, "did not become healthy in time");
}
});
}
}
/// Orchestrate the whole lifecycle: ensure installed → maybe update → spawn each
/// configured local model → wait healthy. Safe to call in a background task; it
/// never panics.
+36 -1
View File
@@ -83,6 +83,26 @@ pub struct CmfServer {
/// Offload matvecs to the GPU (CMF_GPU=1).
#[serde(default)]
pub gpu: bool,
/// Serve-time O(1) attention override (exported as CMF_O1). Empty = follow
/// the hint recorded in the .cmf; "off" = force exact attention (fixes a
/// too-aggressive conversion without re-converting); "all"/"deepN"/"i,j,k"
/// = force-enable on those layers.
#[serde(default)]
pub o1: String,
/// Default generation params for this model — applied by the gateway when
/// the incoming request does not set them (an explicit request value wins).
#[serde(default, skip_serializing_if = "Option::is_none")]
pub temperature: Option<f32>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub top_p: Option<f32>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub max_tokens: Option<u32>,
/// Reasoning switch for thinking models (Qwen3/3.5 …): "off" injects
/// enable_thinking=false into requests that don't set it themselves —
/// the model answers directly, skipping the <think> phase. Needs
/// cortiq ≥ 0.2.1. Empty = model/template default.
#[serde(default)]
pub thinking: String,
}
/// CMF-format model factory: import a HuggingFace model, convert it to a
@@ -172,6 +192,11 @@ impl CmfCfg {
port: self.local_port,
threads: self.threads,
gpu: self.gpu,
o1: String::new(),
temperature: None,
top_p: None,
max_tokens: None,
thinking: String::new(),
}];
}
Vec::new()
@@ -675,9 +700,19 @@ impl Config {
self.models.iter().map(|m| m.id.as_str()).collect();
// managed local servers are known ids too; also enforce unique ports/ids
let mut ports = std::collections::HashSet::new();
let mut managed_seen = std::collections::HashSet::new();
for s in &servers {
if !managed_seen.insert(s.id.as_str()) {
anyhow::bail!(
"config: two managed local models share the id '{}' — give one of the rows a different id",
s.id
);
}
if !ids.insert(s.id.as_str()) {
anyhow::bail!("config: duplicate managed model id '{}'", s.id);
anyhow::bail!(
"config: managed local model '{}' has the same id as a pool model — rename the row or delete the pool entry",
s.id
);
}
if !ports.insert(s.port) {
anyhow::bail!("config: two managed models share port {}", s.port);
+35 -2
View File
@@ -301,6 +301,35 @@ impl Pipeline {
Ok((decision, route_source, final_candidates))
}
/// Fill in the model's default generation params (from its Local-models row)
/// for anything the request left unset — an explicit request value wins.
fn apply_gen_defaults(req: &mut ChatRequest, cfg: &crate::config::Config, model_id: &str) {
if let Some(s) = cfg
.cmf
.effective_servers()
.into_iter()
.find(|s| s.id == model_id)
{
if req.params.temperature.is_none() {
req.params.temperature = s.temperature;
}
if req.params.top_p.is_none() {
req.params.top_p = s.top_p;
}
if req.params.max_tokens.is_none() {
req.params.max_tokens = s.max_tokens;
}
// reasoning switch: "off" → enable_thinking=false (cortiq ≥ 0.2.1
// renders the chat template with an empty <think> block); a client
// that sets enable_thinking itself wins.
if s.thinking == "off" && !req.params.passthrough.contains_key("enable_thinking") {
req.params
.passthrough
.insert("enable_thinking".into(), serde_json::Value::Bool(false));
}
}
}
async fn run_inner(&self, req: ChatRequest, state: &SharedState) -> Result<ChatResponse> {
let live = state.live();
let (decision, route_source, mut final_candidates) = self.resolve(&req, &live).await?;
@@ -341,7 +370,9 @@ impl Pipeline {
route_source
);
match provider.chat(req.clone()).await {
let mut attempt_req = req.clone();
Self::apply_gen_defaults(&mut attempt_req, &live.cfg, &model_id);
match provider.chat(attempt_req).await {
Ok(mut resp) => {
let cost =
provider.price(resp.usage.prompt_tokens, resp.usage.completion_tokens);
@@ -415,7 +446,9 @@ impl Pipeline {
model_id,
route_source
);
match provider.chat_stream(req.clone()).await {
let mut attempt_req = req.clone();
Self::apply_gen_defaults(&mut attempt_req, &live.cfg, &model_id);
match provider.chat_stream(attempt_req).await {
Ok(stream) => {
let info = RouteInfo {
task_label: decision.task_label.clone(),
+3
View File
@@ -45,6 +45,9 @@ impl OpenAiProvider {
if let Some(m) = req.params.max_tokens {
body["max_tokens"] = m.into();
}
if let Some(p) = req.params.top_p {
body["top_p"] = p.into();
}
if !req.tools.is_empty() {
body["tools"] = serde_json::Value::Array(req.tools.clone());
}
+21 -1
View File
@@ -223,6 +223,12 @@ export default {
"pg.model": "Model",
"pg.temp": "Temperature",
"pg.maxTokens": "Max tokens",
"pg.think": "Thoughts",
"pg.think.auto": "auto (model default)",
"pg.think.off": "off — answer directly",
"pg.think.on": "on — force reasoning",
"pg.thinkHelp": "Reasoning phase of thinking models. Auto = model/row default; off/on override for this request (needs cortiq >= 0.2.1 for local models).",
"pg.topPHelp": "Nucleus sampling: only tokens within this cumulative probability are sampled. Empty = default.",
"pg.send": "Send",
"pg.stream": "Stream",
"pg.sending": "Sending…",
@@ -264,7 +270,15 @@ export default {
"settings.cmf.pick": "select a model",
"settings.cmf.models": "Local models",
"settings.cmf.modelsHelp": "Each row is one model the gateway runs (its own cortiq serve on its own port) and registers in the pool. Add as many as you have .cmf files and free ports.",
"settings.cmf.rowHint": "id · .cmf file · port · threads · GPU. Changes apply on restart; deleting a row stops that server.",
"settings.cmf.rowHint": "id · .cmf file · port · threads · GPU · O(1) · T°/top-p/max tok. Applied on save: a removed row stops its server (memory freed), a new row starts one.",
"settings.cmf.thinkAuto": "thoughts: auto",
"settings.cmf.thinkOff": "thoughts: off",
"settings.cmf.thinkHelp": "Reasoning models (Qwen3/3.5) write out a thinking phase before the answer. Off = answer directly without it (faster, shorter; needs cortiq >= 0.2.1). A request that sets enable_thinking itself always wins.",
"settings.cmf.tempHelp": "Default temperature for this model (0 = deterministic, 0.7 = balanced, >1 = creative). Used when the request doesn't set one; a request value always wins. Empty = provider default.",
"settings.cmf.topPHelp": "Default nucleus sampling (top-p) for this model: only tokens within this cumulative probability are sampled (0.9 typical). Empty = provider default.",
"settings.cmf.maxTokHelp": "Default max output tokens for this model when the request doesn't set one. Protects against runaway long generations. Empty = provider default.",
"settings.cmf.o1Auto": "O(1): auto",
"settings.cmf.o1Help": "O(1) attention at serve time: auto — follow the hint recorded in the .cmf; off — force exact attention (fixes incoherent output after a too-aggressive --o1 conversion, no re-convert needed); all/deepN — force-enable on those layers.",
"settings.cmf.addServer": "+ Add local model",
"settings.cmf.checkPort": "Check port",
"settings.cmf.portFree": "port is free",
@@ -412,6 +426,12 @@ export default {
"import.adv.o1DeepN": "N deep layers",
"import.adv.o1Layers": "Layers (comma-separated)",
"import.adv.o1mHint": "landmark budget (default 32)",
"import.adv.o1mLabel": "Landmarks (--o1-m)",
"import.adv.o1mHelp": "How many landmark tokens summarize the whole older history when attention is approximated. More = closer to exact attention but slower and more memory; fewer = faster but rougher. 32 is the validated sweet spot — change only if you know why.",
"import.adv.o1windowLabel": "Exact window (--o1-window)",
"import.adv.o1windowHelp": "The last N tokens always get exact (non-approximated) attention. This protects quality on the text the model just wrote/read. Bigger = better local quality, more memory. Default 128.",
"import.adv.o1sinkLabel": "Sink keys (--o1-sink)",
"import.adv.o1sinkHelp": "The first N tokens of the context are kept exact forever ('attention sinks'). Models lean on these anchors — keeping them exact stabilizes long generations. Default 4; rarely needs changing.",
"import.adv.o1windowHint": "exact-window width (default 128)",
"import.adv.o1sinkHint": "permanent exact sink keys (default 4)",
};
+21 -1
View File
@@ -222,6 +222,12 @@ export default {
"pg.model": "Модель",
"pg.temp": "Температура",
"pg.maxTokens": "Макс. токены",
"pg.think": "Мысли",
"pg.think.auto": "авто (по настройке модели)",
"pg.think.off": "выкл — отвечать сразу",
"pg.think.on": "вкл — принудительно размышлять",
"pg.thinkHelp": "Фаза размышлений reasoning-моделей. Авто = настройка модели; выкл/вкл переопределяют для этого запроса (для локальных нужен cortiq ≥ 0.2.1).",
"pg.topPHelp": "Nucleus-сэмплинг: выбираются токены в пределах этой суммарной вероятности. Пусто = по умолчанию.",
"pg.send": "Отправить",
"pg.stream": "Стриминг",
"pg.sending": "Отправка…",
@@ -263,7 +269,15 @@ export default {
"settings.cmf.pick": "выберите модель",
"settings.cmf.models": "Локальные модели",
"settings.cmf.modelsHelp": "Каждая строка — отдельная модель: шлюз поднимает свой cortiq serve на своём порту и регистрирует её в пуле. Добавляйте столько, сколько есть .cmf-файлов и свободных портов.",
"settings.cmf.rowHint": "id · файл .cmf · порт · потоки · GPU. Изменения применяются после перезапуска; удаление строки останавливает сервер.",
"settings.cmf.rowHint": "id · файл .cmf · порт · потоки · GPU · O(1) · T°/top-p/max tok. Применяется сразу при сохранении: удалённая строка останавливает сервер (память освобождается), новая — запускает.",
"settings.cmf.thinkAuto": "мысли: авто",
"settings.cmf.thinkOff": "мысли: выкл",
"settings.cmf.thinkHelp": "Рассуждающие модели (Qwen3/3.5) сначала пишут фазу размышлений. Выкл = отвечать сразу, без неё (быстрее и короче; нужен cortiq ≥ 0.2.1). Если запрос сам задал enable_thinking — он важнее.",
"settings.cmf.tempHelp": "Температура по умолчанию для этой модели (0 = детерминированно, 0.7 = баланс, >1 = креативно). Подставляется, если запрос её не задал; значение из запроса всегда важнее. Пусто = дефолт провайдера.",
"settings.cmf.topPHelp": "Nucleus-сэмплинг (top-p) по умолчанию: выбираются только токены в пределах этой суммарной вероятности (обычно 0.9). Пусто = дефолт провайдера.",
"settings.cmf.maxTokHelp": "Максимум выходных токенов по умолчанию, если запрос не задал свой. Защищает от бесконечных генераций. Пусто = дефолт провайдера.",
"settings.cmf.o1Auto": "O(1): авто",
"settings.cmf.o1Help": "O(1)-внимание при запуске: авто — использовать подсказку из .cmf; off — принудительно точное внимание (лечит бессвязный вывод после слишком агрессивной --o1 конвертации, без переконвертации); all/deepN — принудительно включить на этих слоях.",
"settings.cmf.addServer": "+ Добавить модель",
"settings.cmf.checkPort": "Проверить порт",
"settings.cmf.portFree": "порт свободен",
@@ -411,6 +425,12 @@ export default {
"import.adv.o1DeepN": "N глубоких слоёв",
"import.adv.o1Layers": "Слои (через запятую)",
"import.adv.o1mHint": "число опорных точек (по умолчанию 32)",
"import.adv.o1mLabel": "Опорные точки (--o1-m)",
"import.adv.o1mHelp": "Сколько опорных токенов представляют всю старую историю при аппроксимации внимания. Больше — ближе к точному вниманию, но медленнее и больше памяти; меньше — быстрее, но грубее. 32 — проверенное значение; меняйте, только если понимаете зачем.",
"import.adv.o1windowLabel": "Точное окно (--o1-window)",
"import.adv.o1windowHelp": "Последние N токенов всегда обрабатываются точным (не аппроксимированным) вниманием — это защищает качество на только что написанном/прочитанном тексте. Больше — лучше локальное качество, больше памяти. По умолчанию 128.",
"import.adv.o1sinkLabel": "Sink-ключи (--o1-sink)",
"import.adv.o1sinkHelp": "Первые N токенов контекста навсегда остаются точными («якоря внимания»). Модели опираются на эти якоря — их точность стабилизирует длинную генерацию. По умолчанию 4; менять почти никогда не нужно.",
"import.adv.o1windowHint": "ширина точного окна (по умолчанию 128)",
"import.adv.o1sinkHint": "постоянные точные sink-ключи (по умолчанию 4)",
};
+11 -5
View File
@@ -15,8 +15,14 @@ const QUANTS = [
["F32", "import.q.f32"],
];
function field(label, control, hint) {
return h("label", { class: "field" }, h("span", {}, label), control,
// A small "?" badge with a hover/focus tooltip (same style as Settings).
function help(text) {
return text ? h("span", { class: "help", title: text, tabindex: "0", role: "img", "aria-label": text }, "?") : null;
}
function field(label, control, hint, helpText) {
return h("label", { class: "field" },
h("span", {}, label, helpText ? " " : null, help(helpText)),
control,
hint ? h("div", { class: "hint" }, hint) : null);
}
function opt(v, label, sel) {
@@ -141,9 +147,9 @@ export function renderImport() {
field(t("import.adv.o1DeepN"), o1DeepIn),
field(t("import.adv.o1Layers"), o1CustomIn));
const o1Knobs = h("div", { class: "row", style: "display:none" },
field("--o1-m", o1M, t("import.adv.o1mHint")),
field("--o1-window", o1Window, t("import.adv.o1windowHint")),
field("--o1-sink", o1Sink, t("import.adv.o1sinkHint")));
field(t("import.adv.o1mLabel"), o1M, t("import.adv.o1mHint"), t("import.adv.o1mHelp")),
field(t("import.adv.o1windowLabel"), o1Window, t("import.adv.o1windowHint"), t("import.adv.o1windowHelp")),
field(t("import.adv.o1sinkLabel"), o1Sink, t("import.adv.o1sinkHint"), t("import.adv.o1sinkHelp")));
o1Sel.addEventListener("change", () => {
const on = !!o1Sel.value;
o1Knobs.style.display = on ? "" : "none";
+3 -1
View File
@@ -283,7 +283,9 @@ function row(m, meta, reload) {
onClick: () => {
// local models may have a .cmf on disk — let the user decide its fate
const isLocal = m.managed || /127\.0\.0\.1|localhost/.test(m.base_url || "");
const fileCb = h("input", { type: "checkbox", checked: isLocal ? true : null });
// unchecked by default: deleting a pool entry should not silently
// cost the user a multi-GB re-download/conversion
const fileCb = h("input", { type: "checkbox" });
const body = h(
"div",
{},
+11
View File
@@ -16,6 +16,12 @@ export async function renderPlayground() {
const promptIn = h("textarea", { placeholder: t("pg.promptPlaceholder"), rows: 5 });
const tempIn = h("input", { type: "number", step: "0.1", value: 0.7 });
const maxTokIn = h("input", { type: "number", value: 512 });
const topPIn = h("input", { type: "number", step: "0.05", min: 0, max: 1, placeholder: "0.9", title: t("pg.topPHelp") });
// reasoning switch: auto = model/row default; off/on override per request
const thinkSel = h("select", { title: t("pg.thinkHelp") },
h("option", { value: "" }, t("pg.think.auto")),
h("option", { value: "off" }, t("pg.think.off")),
h("option", { value: "on" }, t("pg.think.on")));
const streamCb = h("input", { type: "checkbox", checked: true });
const profileSel = h("select", {}, ...(meta.profiles || []).map((p) => h("option", { value: p }, p)));
const modelSel = h("select", {}, ...ids.map((id) => h("option", { value: id }, id)));
@@ -69,6 +75,8 @@ export async function renderPlayground() {
temperature: parseFloat(tempIn.value),
max_tokens: parseInt(maxTokIn.value) || null,
};
if (topPIn.value !== "") body.top_p = parseFloat(topPIn.value);
if (thinkSel.value) body.enable_thinking = thinkSel.value === "on";
try {
if (streamCb.checked) {
const started = performance.now();
@@ -151,6 +159,9 @@ export async function renderPlayground() {
h("label", { class: "field" }, h("span", {}, t("pg.mode")), modeSeg),
modeExtra,
h("div", { class: "row" }, h("label", { class: "field" }, h("span", {}, t("pg.temp")), tempIn), h("label", { class: "field" }, h("span", {}, t("pg.maxTokens")), maxTokIn)),
h("div", { class: "row" },
h("label", { class: "field" }, h("span", {}, "top-p"), topPIn),
h("label", { class: "field" }, h("span", {}, t("pg.think")), thinkSel)),
h(
"div",
{ class: "flex", style: "justify-content:space-between" },
+22 -2
View File
@@ -234,6 +234,20 @@ export async function renderSettings() {
const portIn = h("input", { type: "number", value: sv.port ?? 8090, style: "width:82px" });
const threadsIn = h("input", { type: "number", min: 0, value: sv.threads ?? 8, style: "width:60px", title: t("settings.cmf.threadsHelp") });
const gpuIn = h("input", { type: "checkbox", checked: sv.gpu ? true : null, title: t("settings.cmf.gpuHelp") });
// serve-time O(1) override: off cures a too-aggressive --o1 conversion
const o1Opts = ["", "off", "all", "deep4", "deep8", "deep12"];
const o1In = h("select", { style: "width:110px", title: t("settings.cmf.o1Help") },
...o1Opts.map((v) => h("option", { value: v, selected: (sv.o1 || "") === v ? true : null },
v === "" ? t("settings.cmf.o1Auto") : v)));
if (sv.o1 && !o1Opts.includes(sv.o1)) o1In.appendChild(h("option", { value: sv.o1, selected: true }, sv.o1));
// reasoning switch for thinking models (off → enable_thinking=false)
const thinkIn = h("select", { style: "width:118px", title: t("settings.cmf.thinkHelp") },
h("option", { value: "", selected: !sv.thinking ? true : null }, t("settings.cmf.thinkAuto")),
h("option", { value: "off", selected: sv.thinking === "off" ? true : null }, t("settings.cmf.thinkOff")));
// per-model generation defaults (applied when the request doesn't set them)
const tempIn = h("input", { type: "number", step: "0.1", min: 0, max: 2, value: sv.temperature ?? "", placeholder: "T° 0.7", style: "width:74px", title: t("settings.cmf.tempHelp") });
const topPIn = h("input", { type: "number", step: "0.05", min: 0, max: 1, value: sv.top_p ?? "", placeholder: "top-p", style: "width:70px", title: t("settings.cmf.topPHelp") });
const maxTokIn = h("input", { type: "number", min: 1, value: sv.max_tokens ?? "", placeholder: "max tok", style: "width:82px", title: t("settings.cmf.maxTokHelp") });
const checkBtn = h("button", { class: "btn small", type: "button" }, t("settings.cmf.checkPort"));
const removeBtn = h("button", { class: "btn small danger", type: "button", title: t("common.delete") }, "✕");
const st = h("span", { class: "small" });
@@ -258,8 +272,14 @@ export async function renderSettings() {
checkBtn.disabled = false;
});
const row = h("div", { class: "flex wrap", style: "gap:6px;align-items:center;margin:6px 0" },
idIn, modelSel, portIn, checkBtn, threadsIn, h("label", { class: "flex", style: "gap:4px" }, gpuIn, "GPU"), removeBtn, st);
const entry = { node: row, read: () => ({ id: idIn.value.trim() || autoId(modelSel.value), model: modelSel.value.trim(), port: parseInt(portIn.value) || 8090, threads: parseInt(threadsIn.value) || 0, gpu: gpuIn.checked }) };
idIn, modelSel, portIn, checkBtn, threadsIn, h("label", { class: "flex", style: "gap:4px" }, gpuIn, "GPU"), o1In, thinkIn, tempIn, topPIn, maxTokIn, removeBtn, st);
const entry = { node: row, read: () => {
const v = { id: idIn.value.trim() || autoId(modelSel.value), model: modelSel.value.trim(), port: parseInt(portIn.value) || 8090, threads: parseInt(threadsIn.value) || 0, gpu: gpuIn.checked, o1: o1In.value, thinking: thinkIn.value };
if (tempIn.value !== "") v.temperature = parseFloat(tempIn.value);
if (topPIn.value !== "") v.top_p = parseFloat(topPIn.value);
if (maxTokIn.value !== "") v.max_tokens = parseInt(maxTokIn.value);
return v;
} };
removeBtn.addEventListener("click", () => { const i = serverRows.indexOf(entry); if (i >= 0) serverRows.splice(i, 1); row.remove(); });
serverRows.push(entry);
serversWrap.appendChild(row);