cmf: parallelise the managed local server (CMF_THREADS)

The gateway spawned `cortiq serve` with no thread hint, so the runtime ran the
decode loop single-threaded (~0.25 tok/s on a 15B CPU model). Export CMF_THREADS
from a new [cmf].threads config field (default 8) in spawn_serve.

Measured: 0.25 -> 0.81 tok/s (~3.2x) on qwopus-vbit7 (15B). Scaling is sublinear
(GatedDeltaNet state recurrence is serial per token, Amdahl-bound); a real jump
needs the GPU path.
This commit is contained in:
infosave2007
2026-07-08 16:18:31 +03:00
parent 8dd77ca7ff
commit e6d1a630c9
2 changed files with 18 additions and 4 deletions
+9 -4
View File
@@ -155,14 +155,19 @@ async fn cargo_install(force: bool) -> Result<(), String> {
/// Spawn `cortiq serve <model> --host <host> --port <port>`; the child is killed
/// when its handle is dropped (i.e. when the gateway shuts down).
fn spawn_serve(cfg: &CmfCfg) -> Result<Child, String> {
Command::new(&cfg.cortiq_bin)
.arg("serve")
let mut cmd = Command::new(&cfg.cortiq_bin);
cmd.arg("serve")
.arg(&cfg.local_model)
.arg("--host")
.arg(&cfg.local_host)
.arg("--port")
.arg(cfg.local_port.to_string())
.stdout(Stdio::null())
.arg(cfg.local_port.to_string());
// Parallelise decode matvecs across cores (CMF_THREADS). 0 = leave the
// runtime's own default; N>1 gave ~2.8x on a 15B CPU model.
if cfg.threads > 1 {
cmd.env("CMF_THREADS", cfg.threads.to_string());
}
cmd.stdout(Stdio::null())
.stderr(Stdio::null())
.kill_on_drop(true)
.spawn()