feat(cmf): per-server O(1) attention override at serve time
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
Each managed local model row gains an O(1) selector (auto / off / all / deepN), exported to the child `cortiq serve` as CMF_O1. "off" forces exact attention, curing incoherent output from a too-aggressive `--o1` conversion hint without re-converting the file; "all"/"deepN" force-enable it. Auto (default) follows the hint recorded in the .cmf. Surfaced in /admin/api/cmf per-server status.
This commit is contained in:
@@ -83,6 +83,12 @@ pub struct CmfServer {
|
||||
/// Offload matvecs to the GPU (CMF_GPU=1).
|
||||
#[serde(default)]
|
||||
pub gpu: bool,
|
||||
/// Serve-time O(1) attention override (exported as CMF_O1). Empty = follow
|
||||
/// the hint recorded in the .cmf; "off" = force exact attention (fixes a
|
||||
/// too-aggressive conversion without re-converting); "all"/"deepN"/"i,j,k"
|
||||
/// = force-enable on those layers.
|
||||
#[serde(default)]
|
||||
pub o1: String,
|
||||
}
|
||||
|
||||
/// CMF-format model factory: import a HuggingFace model, convert it to a
|
||||
@@ -172,6 +178,7 @@ impl CmfCfg {
|
||||
port: self.local_port,
|
||||
threads: self.threads,
|
||||
gpu: self.gpu,
|
||||
o1: String::new(),
|
||||
}];
|
||||
}
|
||||
Vec::new()
|
||||
|
||||
Reference in New Issue
Block a user