cmf: parallelise the managed local server (CMF_THREADS)

The gateway spawned `cortiq serve` with no thread hint, so the runtime ran the
decode loop single-threaded (~0.25 tok/s on a 15B CPU model). Export CMF_THREADS
from a new [cmf].threads config field (default 8) in spawn_serve.

Measured: 0.25 -> 0.81 tok/s (~3.2x) on qwopus-vbit7 (15B). Scaling is sublinear
(GatedDeltaNet state recurrence is serial per token, Amdahl-bound); a real jump
needs the GPU path.
This commit is contained in:
infosave2007
2026-07-08 16:18:31 +03:00
parent c19edee7bb
commit 16a3e209b6
2 changed files with 18 additions and 4 deletions
+9
View File
@@ -111,6 +111,11 @@ pub struct CmfCfg {
/// the router and any external providers ("use only local models").
#[serde(default)]
pub local_only: bool,
/// Worker threads for the managed `cortiq serve` (exported as `CMF_THREADS`).
/// 0 = let the runtime decide; N>1 parallelises decode matvecs across N cores
/// (measured ~2.8x faster than serial on a 15B model). Default 8.
#[serde(default = "default_cmf_threads")]
pub threads: usize,
}
impl Default for CmfCfg {
fn default() -> Self {
@@ -129,9 +134,13 @@ impl Default for CmfCfg {
local_port: default_cmf_local_port(),
model_id: default_cmf_model_id(),
local_only: false,
threads: default_cmf_threads(),
}
}
}
fn default_cmf_threads() -> usize {
8
}
fn default_cortiq_bin() -> String {
"cortiq".into()
}