feat(proxy): T-X12 采纳 enterprise-ai-gateway 双桶限流——TPM token 桶 + 语义缓存命中率透出

- auth.RateLimiter.allow_tokens:TPM 令牌桶(容量=tpm_cap,连续回流 tpm/60 每秒,
  与 RPM 桶共用 per-key 锁、判定相互独立;tpm_cap<=0 不限)
- ProxyConfig.limits.tpm_per_key(默认 60000,0=不限);chat_completions 在并发槽
  之后做 TPM 预扣判定(est=字符/3 与计费同口径),拒绝时释放并发槽并回 429
  (文案区分并发/RPM/TPM 三种原因)
- semcache.stats 增 misses 与 hit_rate;/admin/stats 新增 semcache 段透出
  (entries/hits_exact/hits_semantic/misses/hit_rate,与账本 h_g 互补)
- 新增 tests/test_tpm_limiter.py 6 项(容量/回流/禁用/双桶独立/key 隔离/命中率)
This commit is contained in:
tzt
2026-09-19 10:09:56 +08:00
parent f04a6c4c43
commit b2b7a64cb3
5 changed files with 117 additions and 1 deletions
+7 -1
View File
@@ -104,6 +104,7 @@ class SemanticCache:
self._clock = now # 假时钟注入(None = time.time
self.hits_exact = 0
self.hits_semantic = 0
self.misses = 0
self._rebuild()
# ---------- 时钟 ----------
@@ -166,6 +167,7 @@ class SemanticCache:
# L2
g = grams(norm_text)
if len(g) < 3:
self.misses += 1
return None
w_q = _weight(g)
candidates: Dict[str, int] = {}
@@ -195,6 +197,7 @@ class SemanticCache:
best_score = score
best_key = key
if best_key is None or best_score < self.sim_threshold:
self.misses += 1
return None
cand = self._l1[best_key]
cand.hits += 1
@@ -244,8 +247,11 @@ class SemanticCache:
pass # 持久化失败不影响内存缓存(可重建)
def stats(self) -> Dict[str, Any]:
total = self.hits_exact + self.hits_semantic + self.misses
return {"entries": len(self._l1), "hits_exact": self.hits_exact,
"hits_semantic": self.hits_semantic}
"hits_semantic": self.hits_semantic, "misses": self.misses,
"hit_rate": round((self.hits_exact + self.hits_semantic) / total, 4)
if total else 0.0}
class SingleFlight: