feat(proxy): T-X12 采纳 enterprise-ai-gateway 双桶限流——TPM token 桶 + 语义缓存命中率透出
- auth.RateLimiter.allow_tokens:TPM 令牌桶(容量=tpm_cap,连续回流 tpm/60 每秒, 与 RPM 桶共用 per-key 锁、判定相互独立;tpm_cap<=0 不限) - ProxyConfig.limits.tpm_per_key(默认 60000,0=不限);chat_completions 在并发槽 之后做 TPM 预扣判定(est=字符/3 与计费同口径),拒绝时释放并发槽并回 429 (文案区分并发/RPM/TPM 三种原因) - semcache.stats 增 misses 与 hit_rate;/admin/stats 新增 semcache 段透出 (entries/hits_exact/hits_semantic/misses/hit_rate,与账本 h_g 互补) - 新增 tests/test_tpm_limiter.py 6 项(容量/回流/禁用/双桶独立/key 隔离/命中率)
This commit is contained in:
@@ -104,6 +104,7 @@ class SemanticCache:
|
||||
self._clock = now # 假时钟注入(None = time.time)
|
||||
self.hits_exact = 0
|
||||
self.hits_semantic = 0
|
||||
self.misses = 0
|
||||
self._rebuild()
|
||||
|
||||
# ---------- 时钟 ----------
|
||||
@@ -166,6 +167,7 @@ class SemanticCache:
|
||||
# L2
|
||||
g = grams(norm_text)
|
||||
if len(g) < 3:
|
||||
self.misses += 1
|
||||
return None
|
||||
w_q = _weight(g)
|
||||
candidates: Dict[str, int] = {}
|
||||
@@ -195,6 +197,7 @@ class SemanticCache:
|
||||
best_score = score
|
||||
best_key = key
|
||||
if best_key is None or best_score < self.sim_threshold:
|
||||
self.misses += 1
|
||||
return None
|
||||
cand = self._l1[best_key]
|
||||
cand.hits += 1
|
||||
@@ -244,8 +247,11 @@ class SemanticCache:
|
||||
pass # 持久化失败不影响内存缓存(可重建)
|
||||
|
||||
def stats(self) -> Dict[str, Any]:
|
||||
total = self.hits_exact + self.hits_semantic + self.misses
|
||||
return {"entries": len(self._l1), "hits_exact": self.hits_exact,
|
||||
"hits_semantic": self.hits_semantic}
|
||||
"hits_semantic": self.hits_semantic, "misses": self.misses,
|
||||
"hit_rate": round((self.hits_exact + self.hits_semantic) / total, 4)
|
||||
if total else 0.0}
|
||||
|
||||
|
||||
class SingleFlight:
|
||||
|
||||
Reference in New Issue
Block a user