feat(proxy): T-X12 采纳 enterprise-ai-gateway 双桶限流——TPM token 桶 + 语义缓存命中率透出

- auth.RateLimiter.allow_tokens:TPM 令牌桶(容量=tpm_cap,连续回流 tpm/60 每秒,
  与 RPM 桶共用 per-key 锁、判定相互独立;tpm_cap<=0 不限)
- ProxyConfig.limits.tpm_per_key(默认 60000,0=不限);chat_completions 在并发槽
  之后做 TPM 预扣判定(est=字符/3 与计费同口径),拒绝时释放并发槽并回 429
  (文案区分并发/RPM/TPM 三种原因)
- semcache.stats 增 misses 与 hit_rate;/admin/stats 新增 semcache 段透出
  (entries/hits_exact/hits_semantic/misses/hit_rate,与账本 h_g 互补)
- 新增 tests/test_tpm_limiter.py 6 项(容量/回流/禁用/双桶独立/key 隔离/命中率)
This commit is contained in:
tzt
2026-09-19 10:09:56 +08:00
parent f04a6c4c43
commit b2b7a64cb3
5 changed files with 117 additions and 1 deletions
+2
View File
@@ -52,6 +52,7 @@ class ProxyConfig:
rpm_per_key: int = 10
day_req_cap: int = 200
concurrent_per_key: int = 2
tpm_per_key: int = 60000 # T-X12:每分钟 token 桶(0 = 不限)
max_body_chars: int = 60000
semcache_enabled: bool = True
sim_threshold: float = 0.92
@@ -159,6 +160,7 @@ def build_proxy_config(settings_dict: Dict[str, Any]) -> ProxyConfig:
rpm_per_key=int(limits.get("rpm_per_key", 10) or 10),
day_req_cap=int(limits.get("day_req_cap", 200) or 200),
concurrent_per_key=int(limits.get("concurrent_per_key", 2) or 2),
tpm_per_key=int(limits.get("tpm_per_key", 60000) or 0),
max_body_chars=int(limits.get("max_body_chars", 60000) or 60000),
semcache_enabled=bool(sem.get("enabled", True)),
sim_threshold=float(sem.get("sim_threshold", 0.92) or 0.92),