chore(release): v0.2.8 — multiple managed local models
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
Docker / publish (push) Canceled after 0s
Release / crates-io (push) Canceled after 0s
Release / release (macos-14, aarch64-apple-darwin) (push) Canceled after 0s
Release / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
Release / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
- The gateway can now run several local .cmf models at once: a [[cmf.servers]] list, one supervised `cortiq serve` per entry on its own port, each registered in the pool and independently routable/pinnable. The single legacy local_model/local_port still works (back-compat via effective_servers()). - Settings → Local models is now a rows editor (id · path · port + check · threads · GPU) with add/remove and per-server serving status. - Deleting a managed model stops just its server and removes its file — unless another model still points at the same file (shared-file guard).
This commit is contained in:
+9
-9
@@ -28,17 +28,17 @@ impl Registry {
|
||||
by_id.insert(m.id.clone(), provider);
|
||||
}
|
||||
|
||||
// Managed local CMF server: register it as an OpenAI-compatible provider
|
||||
// so it can be routed to like any other backend — including when the
|
||||
// router is unavailable or the gateway is in local-only mode.
|
||||
if cfg.cmf.manage_server
|
||||
&& !cfg.cmf.local_model.trim().is_empty()
|
||||
&& !by_id.contains_key(&cfg.cmf.model_id)
|
||||
{
|
||||
// Managed local CMF servers: register each as an OpenAI-compatible
|
||||
// provider so it can be routed to like any other backend — including when
|
||||
// the router is unavailable or the gateway is in local-only mode.
|
||||
for s in cfg.cmf.effective_servers() {
|
||||
if by_id.contains_key(&s.id) {
|
||||
continue; // a static [[models]] entry with the same id wins
|
||||
}
|
||||
let m = ModelCfg {
|
||||
id: cfg.cmf.model_id.clone(),
|
||||
id: s.id.clone(),
|
||||
provider: "openai".into(),
|
||||
base_url: format!("http://{}:{}/v1", cfg.cmf.local_host, cfg.cmf.local_port),
|
||||
base_url: format!("http://{}:{}/v1", cfg.cmf.local_host, s.port),
|
||||
model: "cortiq".into(),
|
||||
cost_tier: "local".into(),
|
||||
price_in: 0.0,
|
||||
|
||||
Reference in New Issue
Block a user