feat(import): O(1) attention options in Advanced (cortiq >= 0.2.0)
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
- New Advanced controls on the import page: O(1) attention spec (off / all / deep N / explicit layer list) plus the --o1-m / --o1-window / --o1-sink knobs with validated defaults (32 / 128 / 4) and explanatory hints. - Backend passes --o1* to the native `cortiq convert`; fails fast with clear messages when combined with Python-only options, used on GGUF imports, or when the installed cortiq predates 0.2.0 (the hint needs the new runtime). - The hint is recorded in the .cmf header (weights byte-identical); the runtime applies it at load. Verified end-to-end: converted a model with --o1 all and the runtime logged "o1 nystrom attention: 2/2 layer(s)".
This commit is contained in:
Vendored
+11
@@ -400,4 +400,15 @@ export default {
|
||||
"import.adv.shard": "Shard max (GB)",
|
||||
"import.adv.noShard": "no sharding",
|
||||
"import.adv.skipMtp": "skip MTP head",
|
||||
"import.adv.o1": "O(1) attention",
|
||||
"import.adv.o1Hint": "Constant-memory streaming attention: the KV cache is released after prefill, so long contexts don't grow memory. Weights stay byte-identical — only a load-time hint is recorded; you can switch back anytime (serve with --o1 off). Validated on Qwen3-0.6B: all layers ≈ ×1.18 perplexity.",
|
||||
"import.adv.o1Off": "off — exact attention (default)",
|
||||
"import.adv.o1All": "all — convert every softmax layer",
|
||||
"import.adv.o1Deep": "deep N — only the N deepest layers",
|
||||
"import.adv.o1Custom": "custom — explicit layer list",
|
||||
"import.adv.o1DeepN": "N deep layers",
|
||||
"import.adv.o1Layers": "Layers (comma-separated)",
|
||||
"import.adv.o1mHint": "landmark budget (default 32)",
|
||||
"import.adv.o1windowHint": "exact-window width (default 128)",
|
||||
"import.adv.o1sinkHint": "permanent exact sink keys (default 4)",
|
||||
};
|
||||
|
||||
Reference in New Issue
Block a user