feat: semantic (embedding-based) response cache
- cache.rs: cosine-similarity store (threshold, TTL, ring buffer), partitioned by routing signature; tracks hit rate and model cost saved - pipeline: embed routing text, look up before the model call (route_source=cache on hit), insert successful responses; stats record cache hits - config [cache] + admin settings card; dashboard shows hit-rate and savings - embedding model resolved from the pool (or first kind="embedding") Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
7dbc39c6a6
commit
a67564bc24
@@ -19,6 +19,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/).
|
||||
SSE), so Claude-native clients can target the gateway.
|
||||
- **Embeddings**: `POST /v1/embeddings` inbound + provider `embed()`; embedding models
|
||||
are now resolvable from the pool.
|
||||
- **Semantic cache**: embedding-based cache that returns a stored answer for prompts
|
||||
semantically near a previous one (cosine threshold, TTL, ring buffer), skipping the
|
||||
model call. Partitioned by routing signature; hit-rate and cost saved are tracked and
|
||||
shown on the dashboard; configurable from Settings (`[cache]`).
|
||||
|
||||
## [0.1.0] - 2026-06-30
|
||||
|
||||
|
||||
Reference in New Issue
Block a user