- Each Local-models row gains default generation params — temperature, top-p,
max tokens — applied by the gateway when the incoming request doesn't set
them (an explicit request value always wins). Works for both the plain and
streaming paths; top_p is now forwarded to OpenAI-compatible providers.
- The "also delete the model file from disk" checkbox in the delete dialog now
defaults to UNCHECKED — deleting a pool entry no longer silently costs a
multi-GB re-download/conversion.
Each managed local model row gains an O(1) selector (auto / off / all / deepN),
exported to the child `cortiq serve` as CMF_O1. "off" forces exact attention,
curing incoherent output from a too-aggressive `--o1` conversion hint without
re-converting the file; "all"/"deepN" force-enable it. Auto (default) follows
the hint recorded in the .cmf. Surfaced in /admin/api/cmf per-server status.
- put_settings now heals [routing] after the managed-servers list changes: a
removed server whose port is taken over by a new one is treated as a
replacement (old id → new id in default + every tier); truly vanished ids are
stripped; an emptied default falls back to the first managed server / pool
model. Previously the stale id failed validation and the save errored.
- /admin/api/cmf/port now returns managed:true when the port is held by this
gateway's own managed server; the Settings port check shows that as info
("freed on restart, replacing is fine") instead of a red "port busy".
- New GET /admin/api/cmf/files lists the .cmf files (and tensor-dir .cmf) under
models_dir, with sizes.
- Settings → Local models: the per-row model field is now a dropdown of those
files instead of a hand-typed path, and auto-fills the id from the filename.
- The gateway can now run several local .cmf models at once: a [[cmf.servers]]
list, one supervised `cortiq serve` per entry on its own port, each registered
in the pool and independently routable/pinnable. The single legacy
local_model/local_port still works (back-compat via effective_servers()).
- Settings → Local models is now a rows editor (id · path · port + check · threads
· GPU) with add/remove and per-server serving status.
- Deleting a managed model stops just its server and removes its file — unless
another model still points at the same file (shared-file guard).
- Settings: "?" help tooltips on every option; router key auto-probes the
connection right after saving (no more hanging "unavailable")
- Local models card reworked into 3 clear steps (install cortiq → add model →
run locally); removed the hand-entered .cmf path / pool-id fields — models
are added via the Import tab and the managed server registers them
- Fresh install ships an EMPTY model pool; the local .cmf is auto-registered by
the managed CMF server, so it needs no static [[models]] entry
- New: port-availability check — GET /admin/api/cmf/port + "Check port" button
with next-free-port suggestion. bind()+connect() probe so Docker-proxied /
dual-stack ports (e.g. 8081) are correctly detected as occupied
- /admin/api/cmf now reports the serving model + port; card shows "▶ serving …"
- On-demand runtime install: POST /admin/api/cmf/install
Expose the two speed knobs in the /admin Settings panel instead of TOML-only:
- [cmf].gpu (new) -> exports CMF_GPU=1 to the managed cortiq serve (Metal/wgpu).
On Apple Silicon decode is memory-bandwidth bound so this is ~neutral for
generation (helps prefill); off by default.
- [cmf].threads (added earlier) now has a UI number input.
- config.rs: CmfCfg.gpu bool (serde default false).
- cmf_runtime.rs: spawn_serve sets CMF_GPU=1 when gpu is on.
- admin/mod.rs: put_settings flags needs_restart on threads/gpu change (the
server is spawned once at startup with those env vars).
- settings.js: threads input + gpu checkbox + save payload + speed hint.
- i18n en/ru: threads/gpu/speedNote keys (other locales fall back to en, as the
whole cmf section already does there).
Run models in the CMF (.cmf) format locally, via the native `cortiq`
CLI from crates.io — no Python and no external model server.
- cmf_runtime: install/update `cortiq-cli` from crates.io, spawn and
health-check `cortiq serve` as a managed local backend.
- import: model-import jobs shell out to the native `cortiq convert`
(HF repo -> .cmf, quant mapped Q8_2F->q8_2f / Q8_ROW->q8 / Q4->q4);
the Python converter is used only for advanced/exotic options.
- pipeline: local-first routing — serve from the local CMF model when
the router is disabled/unreachable, with a "local only" hard switch.
- registry/state: register the local CMF server as an OpenAI provider
and track local candidates.
- config: data-dir resolution + CmfCfg (auto-install/update, managed
server, host/port/model) and relaxed validation for the managed model.
- admin + settings UI (en/ru): "Local models (CMF)" settings card.
- cache.rs: cosine-similarity store (threshold, TTL, ring buffer), partitioned by
routing signature; tracks hit rate and model cost saved
- pipeline: embed routing text, look up before the model call (route_source=cache on
hit), insert successful responses; stats record cache hits
- config [cache] + admin settings card; dashboard shows hit-rate and savings
- embedding model resolved from the pool (or first kind="embedding")
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
infosave2007