feat(proxy): T-X12 采纳 enterprise-ai-gateway 双桶限流——TPM token 桶 + 语义缓存命中率透出
- auth.RateLimiter.allow_tokens:TPM 令牌桶(容量=tpm_cap,连续回流 tpm/60 每秒, 与 RPM 桶共用 per-key 锁、判定相互独立;tpm_cap<=0 不限) - ProxyConfig.limits.tpm_per_key(默认 60000,0=不限);chat_completions 在并发槽 之后做 TPM 预扣判定(est=字符/3 与计费同口径),拒绝时释放并发槽并回 429 (文案区分并发/RPM/TPM 三种原因) - semcache.stats 增 misses 与 hit_rate;/admin/stats 新增 semcache 段透出 (entries/hits_exact/hits_semantic/misses/hit_rate,与账本 h_g 互补) - 新增 tests/test_tpm_limiter.py 6 项(容量/回流/禁用/双桶独立/key 隔离/命中率)
This commit is contained in:
@@ -52,6 +52,7 @@ class ProxyConfig:
|
||||
rpm_per_key: int = 10
|
||||
day_req_cap: int = 200
|
||||
concurrent_per_key: int = 2
|
||||
tpm_per_key: int = 60000 # T-X12:每分钟 token 桶(0 = 不限)
|
||||
max_body_chars: int = 60000
|
||||
semcache_enabled: bool = True
|
||||
sim_threshold: float = 0.92
|
||||
@@ -159,6 +160,7 @@ def build_proxy_config(settings_dict: Dict[str, Any]) -> ProxyConfig:
|
||||
rpm_per_key=int(limits.get("rpm_per_key", 10) or 10),
|
||||
day_req_cap=int(limits.get("day_req_cap", 200) or 200),
|
||||
concurrent_per_key=int(limits.get("concurrent_per_key", 2) or 2),
|
||||
tpm_per_key=int(limits.get("tpm_per_key", 60000) or 0),
|
||||
max_body_chars=int(limits.get("max_body_chars", 60000) or 60000),
|
||||
semcache_enabled=bool(sem.get("enabled", True)),
|
||||
sim_threshold=float(sem.get("sim_threshold", 0.92) or 0.92),
|
||||
|
||||
Reference in New Issue
Block a user