chore: T-P-1 工作区收敛——并行会话成果与历史未入库文件整理入库
- 入库历史遗漏源码/测试：router_system 9 模块（agent/executors/inference/knowledge/ memory/planner/skills/trace）、tests 11 个测试文件、config/knowledge 领域知识 - 入库根目录方案文档（v2/v3/可行性×2）、references 文献（arxiv 14-18/cnki_open/ 参考文献清单）、research 论文素材（routerarena/paper/中文文献 PDF） - 前端构建产物刷新（新 hash）；webapp 误写文档删除 - gitignore 增补：deepseek-harness、research/_refs、.mimosa/.zcode、网关日志/pid、 临时调试脚本、tests/e2e/node_modules、AI代理功能开发/prefix - 基线确认：318 passed
This commit is contained in:
tzt
+38
@@ -35,3 +35,41 @@ Thumbs.db
|
||||
config/model_pool.json
|
||||
agent_runs/
|
||||
agent_workspace/
|
||||
|
||||
# 智能体工作目录：仅保留标注 README，工作产物不入库
|
||||
AI代理功能开发/*
|
||||
!AI代理功能开发/README.md
|
||||
!AI代理功能开发/方案_校园AI代理层.md
|
||||
!AI代理功能开发/实施方案_代理层与缓存层.md
|
||||
# 代理层：课程前缀资料（内容属课程方）与压测原始产物不入库
|
||||
AI代理功能开发/prefix/
|
||||
|
||||
# 参考仓库（本地查阅用，不入库）
|
||||
deepseek-harness/
|
||||
research/_refs/
|
||||
|
||||
# 工具与网关运行时状态
|
||||
.mimosa/
|
||||
.zcode/
|
||||
_gateway.pid
|
||||
_gateway.out.log
|
||||
_gateway.err.log
|
||||
|
||||
# 临时调试产物（历次会话残留，保留磁盘文件但隐身）
|
||||
_check.txt
|
||||
_diag.txt
|
||||
_t.txt
|
||||
_test_out.txt
|
||||
_run_out.txt
|
||||
deepseek_reply.txt
|
||||
response_check.txt
|
||||
response_output.txt
|
||||
check_response.py
|
||||
check_spa.py
|
||||
e2e_test.py
|
||||
test_backend_models.py
|
||||
test_deepseek.py
|
||||
test_models.py
|
||||
|
||||
# e2e 工程 node_modules（源文件入库）
|
||||
tests/e2e/node_modules/
|
||||
|
||||
Binary file not shown.
@@ -4,8 +4,10 @@
|
||||
|
||||
## 当前权威规划（先读这个）
|
||||
|
||||
**《实现方案_v2_端云协同LLM协作系统.md》**（仓库根目录）是当前唯一权威实施方案：
|
||||
**《实现方案_v2_端云协同编程智能体系统.md》**（仓库根目录）是 v2 基础权威方案，
|
||||
v3（Web 应用化）、v4（模型池与工具智能体）为其增量权威方案（均已落地）：
|
||||
大模型（API）任务分析/决策/终审 ＋ 小模型（本地 llama.cpp）实现/自验证 ＋「交流文本」结构化共享工作区 ＋ 人工检验队列。
|
||||
**论文题目（已定）**：《基于端云协同的编程智能体系统设计与实现》。
|
||||
|
||||
- 实现前必读其第 3 节（不得推翻的设计决策 D1–D11）与第 8 节（工程规约）。
|
||||
- 任务按其第 7 节 T1–T14 顺序执行；每完成一个任务，在《任务拆解与执行计划.md》登记一行。
|
||||
@@ -39,7 +41,11 @@
|
||||
|
||||
| 文档 | 用途 |
|
||||
|---|---|
|
||||
| 实现方案_v2_端云协同LLM协作系统.md | **当前权威规划**（架构/协议/任务分解/实验设计） |
|
||||
| 实现方案_v2_端云协同编程智能体系统.md | v2 基础方案（交流文本协议/任务分解/实验设计） |
|
||||
| 实现方案_v3_Web应用化.md | v3 Web 应用化（Vue3 SPA + SSE 实时协作可视化） |
|
||||
| 实现方案_v4_模型池与工具智能体.md | v4 增补（多价位模型池 + 工具智能体） |
|
||||
| AI代理功能开发/方案_校园AI代理层.md | 校园 AI 代理层设计（缓存经济学/规模测算/合规红线） |
|
||||
| AI代理功能开发/实施方案_代理层与缓存层.md | 代理层执行方案（T-P0–T-P8：文件级/DDL/接口/验收） |
|
||||
| 任务拆解与执行计划.md | 任务状态登记表（v1 历史任务 + v2 新任务追加处） |
|
||||
| 实现方案_多专业小模型+路由模型.md | v1 方案（已被 v2 取代方向，作历史参考） |
|
||||
| 可行性调研与落地实现路线报告.md | v1 期可行性论证（历史参考） |
|
||||
|
||||
@@ -0,0 +1,33 @@
|
||||
# AI 代理功能开发 · 校园 AI 代理层
|
||||
|
||||
> **定位标注**：本目录开发「端 ↔ 云之间的 **AI 代理层**」——面向校园场景的 LLM API 代理网关
|
||||
> （如代理 DeepSeek key：学生持代理 key，代理持有上游主 key）。
|
||||
> **商业模式 = API 差价 + 缓存收益**：校园问题高重复 → 高缓存命中 → 高毛利；
|
||||
> 校园网提供网络基础设施（托管/带宽/内网可达）。
|
||||
> **方案全文见 [`方案_校园AI代理层.md`](方案_校园AI代理层.md)**（架构 / 缓存经济学 / 复用映射 / MVP 任务 / 合规红线）。
|
||||
|
||||
## 当前状态
|
||||
|
||||
- [x] 立项草案（2026-09-04）：方案文档就绪
|
||||
- [ ] P1 透传网关 → P2 计量计费 → P3 缓存栈 → P4 账号配额 → P5 看板（见方案第 4 节）
|
||||
|
||||
## 工作区约定（智能体工具调用）
|
||||
|
||||
本目录同时是智能体的工作区之一：
|
||||
|
||||
1. **接入**：设置页 → 工作区选择本目录；或 `POST /agent/workspaces`，
|
||||
`{"path": "AI代理功能开发"}`。网关默认工作区仍为 `agent_workspace/`，互不影响。
|
||||
2. **安全边界**（对齐 实现方案_v4 / D12）：工具调用关押在本目录内，路径越界拒绝；
|
||||
`run_command` 默认关闭（allow_shell 开关）。
|
||||
|
||||
## Git 与安全约定
|
||||
|
||||
- 本目录**仅 README 与方案文档入库**；开发/实验产物由 `.gitignore` 忽略。
|
||||
- **上游主 key 只存 env / `config/settings.json`（已 gitignore）**；
|
||||
任何代码、示例、测试不得出现真实凭据字面量。
|
||||
|
||||
## 相关文档与代码
|
||||
|
||||
- 复用资产：`gateway/model_pool.py`（多价位池）、`router_system/cache.py`（语义缓存）、
|
||||
`gateway/api.py`（SSE 基建）、`review.py`（sqlite 模式 → billing 参照）
|
||||
- 设计：`实现方案_v4_模型池与工具智能体.md`；进展：`毕业设计_进度记录.md`
|
||||
@@ -0,0 +1,302 @@
|
||||
# 实施方案：校园 AI 代理层与缓存层（T-P0…T-P8 逐任务执行版）
|
||||
|
||||
> 编写日期：2026-09-05 状态：**执行版**（设计依据见《方案_校园AI代理层.md》；两者冲突时以本文为准）
|
||||
> 预期读者：负责实现的 AI Agent。实现前必读：第 1 节锁定决策、第 9 节测试纪律。
|
||||
> 总工期估算：约 13 个工作日；里程碑 M1（计费代理）→ M2（缓存栈）→ M3（管理面+验收）。
|
||||
|
||||
---
|
||||
|
||||
## 0. 范围与非目标
|
||||
|
||||
**范围**：现有 FastAPI app 内新增代理面（`/proxy/v1/*`）＋学生 key 鉴权计费＋两级缓存（精确→n-gram 语义）
|
||||
＋管理端点与前端页＋压测报告。
|
||||
|
||||
**MVP 非目标**（做了算超纲，除非另有指示）：多上游 key 轮换/前缀亲和、embedding 向量检索
|
||||
（预留 embedder 接口，M3 可选）、空闲时段自动调度（仅实现 `is_offpeak(ts)` 判定）、流式中途
|
||||
failover、内容审核、多币种。
|
||||
|
||||
**不改动**：`router_system/`（只允许 import `router_system.cache`）；现有 `/chat`、`/agent`、
|
||||
`/review` 行为与全部既有测试。
|
||||
|
||||
---
|
||||
|
||||
## 1. 锁定决策（实现中不得擅改）
|
||||
|
||||
- **D-P1 货币**：账本一律**毫元整数**（1 元 = 1000 毫元），禁止浮点；展示层格式化。
|
||||
- **D-P2 桶来源**：请求头 `X-Campus-Bucket`（缺省 `default`）；`model→bucket` 可在配置映射。
|
||||
- **D-P3 上游条目**：复用 `config/model_pool.json`，条目新增可选字段
|
||||
`"provider": "deepseek|openai|anthropic"`（usage 归一化用）与 `"in_hit_price"`（可选，缺省 = in_miss×1/30）。
|
||||
- **D-P4 failover**：仅在上游**首 token 返回前**允许切换；流中失败 = `aborted`，按已收 usage 计费
|
||||
（无 usage 按字符估算），**不写缓存**。
|
||||
- **D-P5 缓存准入**：只缓存 `finish_reason=="stop"` 的**完整单轮**响应；错误/超时/断连/多轮一律不缓存。
|
||||
- **D-P6 依赖纪律**：M1/M2 **零新第三方依赖**（语义缓存用 stdlib 倒排 n-gram）；embedding + numpy
|
||||
属 M3 可选项，启用须登记 requirements.txt 并说明理由。
|
||||
- **D-P7 开关**：`proxy.enabled == false` 时不注册任何 `/proxy` 路由，行为与现状逐字节一致。
|
||||
- **D-P8 凭据**：上游主 key 只从 env / `config/settings.json`（已 gitignore）读取，代码/测试零字面量。
|
||||
- **D-P9 进程模型**：**uvicorn 单进程（workers=1）是正确性前提**——singleflight、令牌桶、内存 LRU
|
||||
均为进程内状态，多 worker 会静默失效（限流失效/重复打上游）。扩容路径只有性能预算触发的
|
||||
数据面手术（方案 §3.5），不存在"加 worker"这个选项。
|
||||
- **D-P10 DB 访问纪律**：sqlite3 是同步库，**所有 DB 调用必须经 `asyncio.to_thread`**（写路径可进一步
|
||||
收敛为单写线程+队列）；连接 `check_same_thread=False` + `threading.Lock` 串行化写。
|
||||
热路径读（key→学生上下文）走进程内缓存：哈希→(student_id,status,caps) 的 LRU，TTL 30s，写后失效。
|
||||
违反此条，T-P8 的 P99 ≤50ms 会被同步 DB 卡死。
|
||||
- **D-P11 计费两阶段（防并发超扣）**：预扣用**原子 UPDATE**：
|
||||
`UPDATE students SET balance_milli=balance_milli-? WHERE id=? AND balance_milli>=?`（rowcount=0 即 402），
|
||||
est = in_miss 价×字符估算 + out 价×min(max_tokens,4096)（宁可高估）；settle 按真实 usage 回补
|
||||
est−actual 差额；void 全额退。不用 holds 表，一条 SQL 解决竞态。
|
||||
|
||||
---
|
||||
|
||||
## 2. 文件清单
|
||||
|
||||
**新建**：
|
||||
```
|
||||
gateway/proxy/
|
||||
├── __init__.py # build_proxy_router(settings, pool) -> APIRouter；唯一组装点
|
||||
├── config.py # ProxyConfig：从 settings 读取 + 校验 + 默认值
|
||||
├── errors.py # ProxyAuthError(401)/QuotaError(429)/BalanceError(402)/UpstreamError(502)
|
||||
├── auth.py # key 签发/校验/注销 + 令牌桶 + 日限额
|
||||
├── ledger.py # DDL 初始化 + students/keys CRUD + hold/settle/void + 日志写入
|
||||
├── pricing.py # PriceTable + is_offpeak(ts) + compute(usage, model, ts) -> CostBreakdown
|
||||
├── normalizer.py # bucket 解析 + 规范化哈希 + PrefixShaper（整形后请求体）
|
||||
├── semcache.py # 精确 L1 + n-gram 倒排 L2 + TTL/版本失效 + singleflight
|
||||
├── upstream.py # httpx 流式派发 + usage 三家归一化 + 首 token 前 failover
|
||||
└── routes.py # /proxy/v1/chat/completions、/proxy/v1/models、/proxy/admin/*
|
||||
scripts/bench_proxy.py # E-P1/E-P2 数据管道：200 条校园模拟请求 → 命中率/毛利报告
|
||||
webapp/src/views/ProxyView.vue
|
||||
tests/test_proxy_{auth,ledger,pricing,normalizer,semcache,upstream,routes}.py
|
||||
```
|
||||
|
||||
**修改**：`gateway/api.py`（`app.include_router`，受 `proxy.enabled` 门控，≤5 行）；
|
||||
`webapp/src/App.vue`（NAV 加"代理"项）；`config/model_pool.json`（条目加 provider/in_hit_price）；
|
||||
`任务拆解与执行计划.md`、`AGENTS.md`（登记）。
|
||||
|
||||
---
|
||||
|
||||
## 3. 数据模型（SQLite DDL，库文件 `data/proxy.sqlite3`，WAL 模式）
|
||||
|
||||
```sql
|
||||
CREATE TABLE students(
|
||||
id INTEGER PRIMARY KEY, name TEXT NOT NULL, class TEXT DEFAULT '',
|
||||
status TEXT NOT NULL DEFAULT 'active', -- active|suspended
|
||||
balance_milli INTEGER NOT NULL DEFAULT 0,
|
||||
daily_cap_milli INTEGER NOT NULL DEFAULT 5000,
|
||||
spent_today_milli INTEGER NOT NULL DEFAULT 0, spent_date TEXT DEFAULT '');
|
||||
|
||||
CREATE TABLE proxy_keys(
|
||||
id INTEGER PRIMARY KEY, key_hash TEXT UNIQUE NOT NULL, key_prefix TEXT NOT NULL,
|
||||
student_id INTEGER NOT NULL REFERENCES students(id),
|
||||
created_ts INTEGER NOT NULL, revoked INTEGER NOT NULL DEFAULT 0,
|
||||
rpm_cap INTEGER NOT NULL DEFAULT 10, day_cap_req INTEGER NOT NULL DEFAULT 200,
|
||||
req_today INTEGER NOT NULL DEFAULT 0, req_date TEXT DEFAULT '');
|
||||
|
||||
CREATE TABLE usage_ledger(
|
||||
request_id TEXT PRIMARY KEY, ts INTEGER NOT NULL, key_id INTEGER NOT NULL,
|
||||
model TEXT NOT NULL, bucket TEXT NOT NULL DEFAULT 'default',
|
||||
in_miss_tok INTEGER NOT NULL DEFAULT 0, in_hit_tok INTEGER NOT NULL DEFAULT 0,
|
||||
out_tok INTEGER NOT NULL DEFAULT 0, gateway_cached INTEGER NOT NULL DEFAULT 0,
|
||||
upstream_cost_milli INTEGER NOT NULL DEFAULT 0, charged_milli INTEGER NOT NULL DEFAULT 0,
|
||||
margin_milli INTEGER NOT NULL DEFAULT 0, ttfb_ms INTEGER, total_ms INTEGER,
|
||||
status TEXT NOT NULL); -- ok|cached|error|aborted|insufficient
|
||||
|
||||
CREATE TABLE semcache(
|
||||
cache_key TEXT PRIMARY KEY, -- bucket + '|' + sha256(norm_q)
|
||||
bucket TEXT NOT NULL, q_norm TEXT NOT NULL, answer TEXT NOT NULL, model TEXT NOT NULL,
|
||||
created_ts INTEGER NOT NULL, ttl_ts INTEGER NOT NULL,
|
||||
doc_version INTEGER NOT NULL DEFAULT 1, hits INTEGER NOT NULL DEFAULT 0);
|
||||
CREATE INDEX idx_semcache_bucket ON semcache(bucket, ttl_ts);
|
||||
CREATE INDEX idx_ledger_ts ON usage_ledger(ts); -- stats 按时间范围聚合用
|
||||
```
|
||||
|
||||
前缀资料文件（`doc_prefix_file` 指向的 txt）不入库不入 git（内容属课程方），gitignore 加
|
||||
`AI代理功能开发/prefix/`。
|
||||
|
||||
---
|
||||
|
||||
## 4. 配置 schema（settings.json 的 `proxy` 段）
|
||||
|
||||
```jsonc
|
||||
"proxy": {
|
||||
"enabled": true,
|
||||
"admin_key": "",
|
||||
"buckets": {
|
||||
"default": {"system_template": "你是校园学习助手。", "doc_prefix_file": null,
|
||||
"doc_version": 1, "ttl_hours": 72},
|
||||
"course_python": {"system_template": "你是 Python 课程助教。",
|
||||
"doc_prefix_file": "AI代理功能开发/prefix/python24.txt",
|
||||
"doc_version": 1, "ttl_hours": 168}
|
||||
},
|
||||
"pricing": { // 上游成本价：元/1M tokens；示例为 Flash 档高峰价，以官网价目为准
|
||||
"deepseek-chat": {"in_miss": 3.0, "in_hit": 0.1, "out": 9.0},
|
||||
"peak_window": {"start": "08:30", "end": "23:59"}, // 窗口外按 off-peak 系数 0.5
|
||||
"offpeak_factor": 0.5,
|
||||
"sale_discount": {"in": 0.5, "out": 0.5} // 学生售价折扣
|
||||
},
|
||||
"limits": {"rpm_per_key": 10, "day_req_cap": 200, "concurrent_per_key": 2,
|
||||
"max_body_chars": 60000},
|
||||
"semcache": {"enabled": true, "sim_threshold": 0.92, "max_entries": 300000,
|
||||
"promote_frequency": 5}
|
||||
}
|
||||
```
|
||||
|
||||
峰值/空闲窗口与官方"高峰时段"定义对齐，上线前按官网核对一次。
|
||||
|
||||
---
|
||||
|
||||
## 5. API 契约
|
||||
|
||||
### 5.1 学生面（OpenAI 兼容）
|
||||
- `POST /proxy/v1/chat/completions`，`Authorization: Bearer sk-campus-…`，
|
||||
body = OpenAI chat 格式 + 可选头 `X-Campus-Bucket`。**计量不依赖客户端行为**：
|
||||
代理向上游始终注入 `stream_options: {"include_usage": true}`，客户端未要求 usage 时
|
||||
过滤该 chunk 不下发。响应与客户端请求同构：stream=true 收 SSE，stream=false 收 JSON；
|
||||
**缓存命中也按此回放**（SSE 合成，见 §6）。
|
||||
- `GET /proxy/v1/models` → 池内允许的模型名列表。
|
||||
|
||||
### 5.2 管理面（X-Admin-Key 鉴权，见本节末）
|
||||
| 端点 | 说明 |
|
||||
|---|---|
|
||||
| `POST /proxy/admin/students` | 建学生 {name, class, balance_yuan, caps} |
|
||||
| `POST /proxy/admin/students/{id}/topup` | {amount_yuan} 充值 |
|
||||
| `POST /proxy/admin/keys` | {student_id, rpm_cap?, day_cap_req?} → **明文 key 只返回一次** |
|
||||
| `POST /proxy/admin/keys/{id}/revoke` | 注销 |
|
||||
| `GET /proxy/admin/stats` | {requests, h_g, h_p, revenue, cost, margin, by_bucket} |
|
||||
| `GET /proxy/admin/ledger?student_id=&limit=&offset=` | 流水分页 |
|
||||
|
||||
**管理面鉴权**：请求头 `X-Admin-Key` 与 `settings.proxy.admin_key` 比对
|
||||
（`hmac.compare_digest`，防时序侧信道）；未配置 admin_key 时仅放行 loopback 来源。
|
||||
**stats 口径**：h_g = cached/requests；h_p = Σin_hit/(Σin_hit+Σin_miss)；revenue=Σcharged、
|
||||
cost=Σupstream_cost、margin=Σmargin；支持 `?since=<ts>` 走 idx_ledger_ts。
|
||||
归档：usage_ledger 超 180 天的行由 T-P8 顺带清理（年度 650 万行会拖慢聚合）。
|
||||
|
||||
### 5.3 错误码
|
||||
401 无效/注销 key；403 学生 suspended；402 余额或日上限不足；429 限流；
|
||||
502 上游失败（首 token 前 failover 均失败）；body 超 `max_body_chars` → 413。
|
||||
|
||||
---
|
||||
|
||||
## 6. 模块接口签名（实现按此，不改名）
|
||||
|
||||
```python
|
||||
# auth.py
|
||||
def issue_key(ledger, student_id, rpm_cap=None, day_cap_req=None) -> str # 明文仅此一次
|
||||
def authenticate(authorization: str, ledger, limits, now) -> AuthContext # raises ProxyAuth/Quota
|
||||
class RateLimiter: allow(key_id, rpm_cap) -> bool
|
||||
|
||||
# ledger.py
|
||||
class Ledger:
|
||||
def init_db(cls, path) -> Ledger
|
||||
def upsert_student / topup / set_status(...)
|
||||
def check_and_count(key_id, student_id, now) -> None # 日限额双检
|
||||
def try_hold(student_id, est_milli) -> bool # 原子预扣（D-P11），False=402
|
||||
def settle(request_id, actual_milli, ...) -> None # 结算 + 回补 est-actual 差额
|
||||
def void(request_id) -> None # 全额退（上游失败）
|
||||
def record(UsageRow) -> None # 幂等：request_id 主键
|
||||
|
||||
# pricing.py
|
||||
def is_offpeak(ts, window) -> bool
|
||||
def compute(usage: Usage, model: str, ts: int, cfg) -> CostBreakdown
|
||||
# CostBreakdown: upstream_cost_milli, charged_milli, margin_milli（全整数毫元）
|
||||
|
||||
# normalizer.py
|
||||
def resolve_bucket(body, headers, cfg) -> BucketCfg
|
||||
def canonical_hash(bucket, doc_version, body) -> str # 稳定序列化→sha256
|
||||
def shape(body, bucket_cfg) -> dict # 整形后上游请求体
|
||||
|
||||
# semcache.py
|
||||
class SemanticCache:
|
||||
def lookup(bucket, doc_version, norm_hash, norm_text, now) -> Hit|None
|
||||
def put(bucket, doc_version, norm_hash, norm_text, answer, model, now) -> None
|
||||
def stats() -> {entries, hits, h_g}
|
||||
|
||||
# upstream.py —— 模块级 httpx.AsyncClient 单例（keepalive；limits.max_connections=100），
|
||||
# 超时 connect=10s / read=120s（流式整段）/ write=10s / pool=30s；始终注入 include_usage（§5.1）。
|
||||
async def stream(body, entry, usage_sink) -> AsyncIterator[bytes]
|
||||
def normalize_usage(provider, usage_dict) -> Usage # 三家字段→统一 Usage
|
||||
```
|
||||
|
||||
**规范化规则（normalizer，决定缓存命中率的代码，测试最重）**：
|
||||
1. messages 序列化：role 与 content 交替拼接为 `role\u0001content\u0002`；
|
||||
2. 剔除易变字段：`temperature/frequency_penalty/seed/request_id/时间戳类内容行`；
|
||||
3. 系统模板与资料前缀**不参与** L1 哈希（桶+doc_version 已表达），只参与上游整形；
|
||||
4. 整形后消息顺序固定：`[canonical_system] → [doc_prefix] → [原 messages]`；
|
||||
5. 多轮判定：`len(messages) > 2`（system 之外 >1 条）→ `cacheable=False`。
|
||||
|
||||
**语义相似度精确定义（semcache L2，误命中=返回错答案，此节测试最重）**：
|
||||
- q_norm 取**字符 2-gram + 3-gram 集合**（中文天然适配，无需分词）；
|
||||
- 倒排索引驻内存（gram→cache_key 列表），**启动时由 semcache 表 q_norm 重建**；
|
||||
- 候选门限：共享 gram ≥3 才计分（避免全量比对）；分数 = 加权 Jaccard（3-gram 权 2、2-gram 权 1）；
|
||||
- ≥ sim_threshold(0.92) 判命中；**L2 语义命中累计 promote_frequency(5) 次后晋升为 L1 精确条目**
|
||||
（复用 v1 cache.py 的提升模式——考试周变体收敛后自动加速）。
|
||||
|
||||
**SSE 合成回放（缓存命中的流式客户端兼容）**：命中时按请求 stream 字段同构返回——
|
||||
stream=true 则用缓存内容合成 SSE（复用缓存 id/format，content 按 ~20 字符分块 +
|
||||
finish_reason=stop + `data: [DONE]`），stream=false 返回标准 JSON。
|
||||
**禁止**对 stream=true 客户端直返 JSON（OpenAI SDK 会解析失败）。
|
||||
|
||||
---
|
||||
|
||||
## 7. 请求处理时序（routes.py 主流程）
|
||||
|
||||
```
|
||||
auth(热路径缓存) → 限流/日额 → body 校验(413) → bucket/规范化 → L1 精确查 → L2 n-gram 查
|
||||
├命中: settle(cached, cost=0, charged=按售价) → SSE/JSON 同构回放（§6）
|
||||
└未命中: singleflight 登记 → try_hold(est 原子预扣，不足 402) → shape(body) → upstream.stream
|
||||
→ tee: 逐块转发客户端 + 累积（含被过滤的 usage chunk）
|
||||
→ 结束: usage 归一化 → compute → settle(actual，回补差额) → （stop 且单轮）semcache.put
|
||||
→ 异常: void 退预扣 / aborted 按已收 usage 记账；均不缓存；singleflight 广播同一结果
|
||||
```
|
||||
|
||||
singleflight：`dict[norm_hash → asyncio.Future]`，**上限 256 条**（超出旁路不合并，防考试周
|
||||
变体洪峰撑爆内存）；第二个到达者 await 同一 Future；等待方 60s 超时自行降级直连。
|
||||
|
||||
---
|
||||
|
||||
## 8. 测试计划
|
||||
|
||||
| 文件 | 必测用例 |
|
||||
|---|---|
|
||||
| test_proxy_auth | 签发→鉴权通过；错 key 401；注销 401；rpm 429；日请求上限 429 |
|
||||
| test_proxy_ledger | hold/settle/void 全链条（预扣-结算-回补一致）；request_id 幂等；日限额跨日重置（注入日期）；**并发 N 路同时预扣，余额不足者精确拒绝、不超扣** |
|
||||
| test_proxy_pricing | 峰谷边界（假时钟 ±1 分钟）；毫元取整不漂移（≥10 组黄金用例）；margin = charged − cost |
|
||||
| test_proxy_normalizer | 同语义输入同哈希；易变字段剔除；整形顺序固定；doc_version+1 后旧缓存不可见；多轮 cacheable=False |
|
||||
| test_proxy_semcache | 精确命中；n-gram 阈值上命中/下未中；TTL 过期（假时钟）；失败不缓存；entries 上限 LRU |
|
||||
| test_proxy_upstream | 三家 usage 形状归一；SSE 透传逐字节一致；首 token 前 failover；流中失败=aborted；**usage 注入且客户端未要求时被过滤** |
|
||||
| test_proxy_routes | 端到端：缓存命中账目(cost=0,charged>0,cached=1)；未命中全程；402/429/413；singleflight 两并发一次上游（计数 Mock）；**stream=true 命中回放为合法 SSE**；**L2 命中 5 次后 L1 直接命中** |
|
||||
|
||||
**隔离清单**：每测试独立 `tmp_path/proxy.sqlite3`；假时钟注入（禁 sleep）；MockTransport 假上游；
|
||||
不动 `config/settings.json` 真实文件（沿用 v4 事故后的备份/恢复模式）。
|
||||
|
||||
---
|
||||
|
||||
## 9. WBS：T-P0…T-P8（依赖顺序执行，每任务一 commit `feat(proxy): T-Pn 描述`）
|
||||
|
||||
| # | 任务 | 产出/步骤 | 验收（可执行命令级） | 估时 |
|
||||
|---|---|---|---|---|
|
||||
| T-P0 | 骨架 | gateway/proxy/ 六文件空实现 + DDL + 挂路由（enabled 门控） | `pytest tests -q` 全绿（318→新增骨架测试）；开关关闭时 `/proxy/*` 404 | 0.5d |
|
||||
| T-P1 | 鉴权+账本 | auth.py + ledger.py（原子预扣 try_hold/settle/void + 热路径缓存）+ 管理端点 | test_proxy_auth/ledger 全绿；并发预扣不超扣；curl 用签发 key 过 401 | 2d |
|
||||
| T-P2 | 上游客户端 | upstream.py：AsyncClient 单例/超时/流式派发 + usage 注入与过滤 + 三家归一化 + failover | test_proxy_upstream 全绿；curl 流式可见逐块输出 | 2d |
|
||||
| T-P3 | 计价+结算 | pricing.py + 账本结算接线 | test_proxy_pricing 全绿（≥10 黄金用例） | 1d |
|
||||
| T-P4 | 路由端到端 | routes.py 主时序（不含缓存分支） | OpenAI SDK 指 `/proxy/v1` 对话成功；账本三值一致；402/429/413 正确 | 1.5d |
|
||||
| T-P5 | 规范化+桶 | normalizer.py + 配置桶 | test_proxy_normalizer 全绿 | 1d |
|
||||
| T-P6 | 语义缓存 | semcache.py（含倒排索引重建/晋升/LRU）+ SSE 合成回放 + singleflight + 接线 + `scripts/warm_prefix.py`（桶前缀预热，max_tokens=1） | test_proxy_semcache 全绿；两并发同请求上游仅 1 次；stream=true 命中回放合法 SSE | 2.5d |
|
||||
| T-P7 | 管理面+前端 | stats/ledger 端点 + ProxyView.vue（三卡片） | 看板真实数据渲染；`npm run build` 产物更新 | 2d |
|
||||
| T-P8 | 压测+预算 | scripts/bench_proxy.py：200 条（重复≥50%）+200 并发；**`--live` 模式**（真实 key 50 条子集测真实 h_p，mock 模式 h_p 为可配置常数仅供联调）；顺带 ledger >180d 归档清理 | P99 附加延迟≤50ms；内存≤1GB；账目零不一致；报告入 `AI代理功能开发/bench/` | 1.5d |
|
||||
|
||||
依赖链：T-P0 → T-P1 → T-P2 → T-P3 → T-P4 → (T-P5 → T-P6) → T-P7 → T-P8。
|
||||
**总验收 = M1(T-P0..4) + M2(T-P5..6) + M3(T-P7..8)**：E-P1 报告显示 h_g+h_p ≥ 50% 且统一 5 折毛利为正
|
||||
（**h_p 须以 `--live` 实测为准**——mock 上游给不出真实的 prompt_cache_hit_tokens）。
|
||||
|
||||
---
|
||||
|
||||
## 10. 风险与回滚
|
||||
|
||||
| 风险 | 对策 |
|
||||
|---|---|
|
||||
| n-gram 语义误命中（相似≠可答） | 阈值 0.92 起步 + 桶隔离 + TTL；E-P3 抽检正确率，误答率高先调阈值再考虑 M3 embedding |
|
||||
| 单进程瓶颈 | 性能预算压测（T-P8）触发才议数据面手术（见方案 §3.5） |
|
||||
| 上游调价 | 价格表纯配置热改；model_pool 多供应商条目 |
|
||||
| 账目漂移 | 毫元整数（D-P1）+ request_id 幂等 + T-P8 账目零不一致验收 |
|
||||
| 测试污染真实配置 | 隔离清单 + settings 备份/恢复模式（v4 事故先例） |
|
||||
| 误改多 worker 部署 | D-P9 写入部署文档与 serve.py 启动参数校验（workers>1 时拒绝启动并提示） |
|
||||
| 同步 DB 卡事件循环 | D-P10 强制 to_thread + 代码评审清单项；T-P8 压测显式测 DB 路径 P99 |
|
||||
@@ -0,0 +1,247 @@
|
||||
# 方案：校园 AI 代理层（AI Proxy Gateway）
|
||||
|
||||
> 编写日期：2026-09-04 状态：立项草案（待实现）
|
||||
> 定位：在「端（学生 / 本地小模型）」与「云（DeepSeek 等 LLM API）」之间加一层 **AI 代理网关**——
|
||||
> "端云协同"的第三层。开发工作（代码/实验/数据）全部落在本目录 `AI代理功能开发/`。
|
||||
|
||||
---
|
||||
|
||||
## 0. 一页速览
|
||||
|
||||
- **商业模式 = API 差价 + 缓存收益**。代理持有上游主 key（如 DeepSeek），学生持代理 key；
|
||||
学生按明牌单价/包月付费，代理按实际混合成本向上游结算。
|
||||
- **关键洞察**：校园场景问题高度重复（同课程、同作业、同考点）→ 通过"网关语义缓存直答 +
|
||||
请求前缀整形"把上游**缓存命中价输入**的占比做到极高（上游缓存命中价通常为未命中价的 1/4～1/10，
|
||||
以官网最新价目为准），而计费按未命中口径 → 差值即毛利。
|
||||
- **校园网提供基础设施**：托管、带宽、内网可达零成本；本地小模型层可跑在校内机器 → 近零成本兜底层。
|
||||
- **北极星指标**：综合缓存命中率 `h = h_g + h_p`（网关语义缓存命中 + 上游前缀缓存命中）与
|
||||
毛利/千次请求。
|
||||
|
||||
## 1. 架构
|
||||
|
||||
```
|
||||
学生（校园网内；Web / SDK / 任意 OpenAI 兼容客户端）
|
||||
│ ① 代理 key（学生凭据，非上游 key）
|
||||
▼
|
||||
┌─ 校园 AI 代理网关（本目录开发）──────────────────────┐
|
||||
│ ② 鉴权 / 配额 / 限流（学生账户、代理 key 签发注销） │
|
||||
│ ③ 缓存栈 L0：语义缓存直答（命中 = 上游成本 0，全毛利） │
|
||||
│ L1：前缀整形（统一 system + 课程资料前置， │
|
||||
│ 用户问题永远在末尾 → 上游前缀缓存高命中） │
|
||||
│ ④ 计量计费账本（usage / prompt_cache_hit_tokens 分账） │
|
||||
│ ⑤ 多价位调度（复用 model_pool：local / budget / premium）│
|
||||
└──────────────────┬─────────────────────────┘
|
||||
▼ ⑥ 上游主 key（env / config/settings.json 注入，绝不入库）
|
||||
云端 API（DeepSeek 等；前缀缓存自动生效）
|
||||
```
|
||||
|
||||
## 2. 缓存经济学（"赚缓存钱"的原理）
|
||||
|
||||
上游对**前缀缓存命中**的输入 token 计价通常为未命中价的 1/4～1/10（比例随上游版本变动，
|
||||
以官网最新价目为准）。代理有三个牟利杠杆，按毛利从高到低：
|
||||
|
||||
1. **L0 语义缓存直答**：同义问题直接复用历史答案，上游成本为 0，毛利率 100%；
|
||||
2. **L1 前缀整形**：所有请求强制统一前缀结构（固定 system 模板 + 课程资料/RAG 内容前置 +
|
||||
用户问题置末），使上游自动前缀缓存命中率大幅上升，输入成本降至命中价；
|
||||
3. **计费口径差**：学生按明牌价计费，代理按实际混合成本结算。
|
||||
|
||||
毛利公式（输入部分，示意）：
|
||||
|
||||
```
|
||||
cost_in = (1 − h_g − h_p) · P_miss + h_p · P_hit （h_g：网关直答命中率，h_p：上游前缀命中率）
|
||||
收入 = P_sale · tokens
|
||||
```
|
||||
|
||||
校园场景 `h` 高的三个理由：课件/题库做共享前缀（同一门课几百人同前缀）、考试周问题重复率极高、
|
||||
班级级 system 模板天然统一。
|
||||
|
||||
## 2.5 经济性测算（2026-09-04，讨论"官方价 5 折"定价）
|
||||
|
||||
**价格锚点**（DeepSeek 2026 分时价，每 1M tokens，[官方价目](https://api-docs.deepseek.com/zh-cn/quick_start/pricing)）：
|
||||
输入未命中 空闲 ¥1.5 / 高峰 ¥3.0（Flash 档）；输入命中低至 ¥0.025–0.1（≈未命中价 **1/30**）；
|
||||
输出 空闲 ¥4.5 起 / 高峰 ¥9 起。
|
||||
|
||||
**命中率规划值**（校园集中域）：
|
||||
h_g 网关语义缓存 保守 15% / 中性 25% / 考试周 40–50%；
|
||||
h_p 上游前缀命中（token 加权，前缀整形后）保守 30% / 中性 50% / 乐观 65%。
|
||||
综合输入命中率：保守 40% / 中性 63% / 考试周 79%。
|
||||
独有优势：前缀缓存挂在主 key 账号下 → **全校请求共享同一缓存池**（学生各自持 key 做不到）。
|
||||
|
||||
**单请求模型**（3K 输入含 1.5K 共享前缀 + 0.8K 输出，高峰）：
|
||||
令 o = 输出成本/输入未命中成本 = 0.8；r = 命中/未命中 = 1/30。
|
||||
统一 5 折收入 = 0.5(1+o)；差异化（输入 5 折、输出 8 折）收入 = 0.5 + 0.8o。
|
||||
免上游率 h₀ = h_g + 本地分流率（model_pool local 档）。
|
||||
|
||||
| 场景 | h₀ | 统一 5 折毛利 | 输入 5 折/输出 8 折 |
|
||||
|---|---|---|---|
|
||||
| 保守 | 20% | **−34%** | −6%（近打平） |
|
||||
| 中性 | 40% | +12% | **+31%** |
|
||||
| 考试周 | 55% | +41% | +54% |
|
||||
|
||||
盈亏平衡 h₀：统一 5 折 ≈ 32%；差异化 ≈ **13%**。
|
||||
|
||||
**结论**：① 全线统一 5 折结构性危险——输出 token 固定亏 50%，缓存利润补不平；
|
||||
**推荐"输入 5 折 / 输出 8 折"差异化定价**。② 杠杆排序：差异化定价 > 本地分流 > h_g 语义缓存
|
||||
> 空闲时段调度（成本直接半价）> h_p 前缀整形。③ 规模：300 活跃用户问答场景月毛利仅数百元；
|
||||
**放大器 = 编程智能体闭环**（单任务 tokens 为问答 30–50 倍，本系统消费本网关，月毛利可上 2000–5000 元）。
|
||||
|
||||
### 2.6 规模化测算（2026-09-04：5000 活跃用户 + 学校采纳情景）
|
||||
|
||||
**用量假设（制度性流量：课程绑定 + 校赛指定，非自然增长）**：
|
||||
活跃 5000；DAU 平时 30%（1500）/ 高峰周 50%（2500）；人均日请求 12/18；40 教学周 ≈ **650 万请求/年**；
|
||||
画像：问答 3.5K in + 0.8K out；智能体任务 50K in + 10K out。
|
||||
|
||||
**商业运营账**（差异化定价，缓存按规模校准：问答段 h_g 30–40%、智能体段 5–15%）：
|
||||
|
||||
| 情景 | 构成 | 年收入 | 年净利 |
|
||||
|---|---|---|---|
|
||||
| S1 纯问答 | agent 占 0 | ≈¥5.7 万 | ≈¥1.7 万（30%） |
|
||||
| S2 +10% 智能体 | 校赛试点+编程课 | ≈¥12.8 万 | ≈¥2.9 万（23%） |
|
||||
| S3 +20% 智能体+竞赛按量 | 完整高位优势 | ≈¥20 万+竞赛经费 | ≈¥6 万+竞赛毛利 |
|
||||
|
||||
**学校采纳模式对比（关键结论：差价是副产品，平台采购才是规模答案）**：
|
||||
|
||||
| 模式 | 收入形式 | 首年收益 | 风险 |
|
||||
|---|---|---|---|
|
||||
| C1 纯转售 | 差价+缓存 | 净利 ¥3–6 万 | 转售合规 + 上游调价（2026 已涨 57–214%） |
|
||||
| **C2 学校采购（推荐）** | 建设立项 ¥10–20 万 + 年度服务费 ¥3–10 万 + 竞赛按量 | **¥15–35 万** | 项目制回款 |
|
||||
| C3 混合 | 学校平台 + 学生增值付费 | 介于两者 | 定价需校批 |
|
||||
|
||||
C2 附带收益：合规消解（学校主体采购上游商用授权）、不垫资不担调价（按流水抽成/年费）、
|
||||
软著+论文+奖项+校级平台经历。
|
||||
|
||||
**两条风险红线**：① 上游调价生死线 → model_pool 多供应商路由为生存设计；
|
||||
② 学校自建私有化为最大替代 → 护城河 = 软件层（缓存整形/计费/交流文本/智能体平台），即毕设系统本身。
|
||||
|
||||
### 2.7 增补（2026-09-04）：C 端客户端 + 商用批量采购对模型的修正
|
||||
|
||||
**两个新变量**：① llama.cpp 推理在 C 端——免费客户端（即本毕设端侧系统，捆绑 llama.cpp）
|
||||
分发给学生，本地推理用学生硬件，学校本地层硬件成本归零；客户端**限制使用学校代理**
|
||||
（学号登录换 key、不暴露 base_url、按 key 限流计费）。② 上游 key 走商用批量采购，
|
||||
采购价为个人牌价 d 折（具体折扣商务洽谈，用敏感性覆盖）。
|
||||
|
||||
**统一 5 折敏感性矩阵**（毛利占收入比；r=1/30、o=0.8）：
|
||||
|
||||
| 场景 | h₀ | d=1.0 | d=0.9 | d=0.8 | d=0.7 | d=0.6 |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 保守 | 20% | −34% | −16% | −7% | +6% | +19% |
|
||||
| 中性 | 40% | +12% | +21% | +30% | +39% | +47% |
|
||||
| 考试周 | 55% | +41% | +49% | +53% | +59% | +65% |
|
||||
|
||||
盈亏平衡采购折扣：保守 d<7.5 折 / 中性 d<8.5 折 / 考试周 d<9.6 折。
|
||||
**结论**：商用采购 ≤9 折时统一 5 折在中性场景 +21% 以上，可行；差异化定价（输入 5 折/输出 8 折）
|
||||
降级为上游调价时的保险杠杆。5000 人年账（中性 d=0.8）：商业净利约 3–5 万，学校采购（C2）仍为收益主体。
|
||||
|
||||
**C 端客户端的边界（诚实评估）**：llama.cpp 开源，技术上无法阻止学生自装直连——
|
||||
锁定的是统一体验/计费合规/学校背书（默认通道），不是防破解。端侧缓存越强打代理流量越少，
|
||||
对学校上游配额是省、对代理毛利中性偏负；客户端免费层能力边界（本地档位/上下文长度）
|
||||
是与学校对齐的定价杠杆。商用合同以学校主体签订 → 转售合规与备案红线基本消解。
|
||||
|
||||
## 3. 与现有系统的复用映射
|
||||
|
||||
| 代理层需要 | 现有资产 | 改造量 |
|
||||
|---|---|---|
|
||||
| 多价位上游池 | `gateway/model_pool.py`（local/budget/premium） | 复用 |
|
||||
| 语义缓存 | `router_system/cache.py`（L1 精确 + L2 n-gram） | 可选升 embedding |
|
||||
| token 计量分账 | `V2Stats.by_model` | 补 cache_hit 维度 |
|
||||
| 学生账户/配额 | `review.py` 的 sqlite 模式 | 新建 `billing.py` |
|
||||
| 流式网关 | v3 SSE 基建 | 透传上游 SSE |
|
||||
| 滥用兜底 | ReviewQueue 思路 + 限流 | 复用思想 |
|
||||
|
||||
## 3.5 技术选型与实现架构（实现 agent 按此执行，细化 §4）
|
||||
|
||||
**总原则**：不换语言、不加服务、不动 `router_system/`。被否选项：Go/Rust 独立服务（丢全部复用，
|
||||
校园负载用不上）、Nginx/OpenResty+Lua（写不了 AI 感知逻辑）、Envoy/Kong/Cloudflare AI Gateway
|
||||
（运维重/出内网）。**结论：Python 3.14 + FastAPI APIRouter 挂现有 app**——峰值 45K 请求/日
|
||||
≈ 1.6 req/s、瞬时 30–50 路流式，单进程 uvicorn 足够。
|
||||
|
||||
**请求链路**：鉴权（代理 key 哈希存储→令牌桶限流→余额熔断）→ 桶识别+规范化序列化 →
|
||||
L0 两级缓存（精确哈希→语义向量 cosine≥0.92，命中即成本 0 照常计费）→ Singleflight 合并 →
|
||||
前缀整形（canonical system+课程资料置顶、问题置末）→ model_pool 派发（local→budget→premium，
|
||||
首 token 前才可 failover）→ 流式 tee（转发+累积）→ 回写账本/缓存/指标。
|
||||
|
||||
**模块落点 `gateway/proxy/`（6 模块）**：
|
||||
`auth.py`（key 签发/注销、令牌桶、日/并发上限）、`ledger.py`（sqlite WAL：students/proxy_keys/
|
||||
usage_ledger/semcache 四表；request_id 幂等；余额预扣-结算）、`normalizer.py`（桶识别、规范化、
|
||||
前缀整形器）、`semcache.py`（两级缓存+singleflight+TTL/版本失效+int8 向量 LRU 30 万条≈300MB）、
|
||||
`upstream.py`（派发+usage 归一化+流式转发）、`pricing.py`（miss/hit/out 三价×峰谷系数；可注入时钟）。
|
||||
|
||||
**usage 归一化**：DeepSeek `prompt_cache_hit_tokens`；OpenAI 兼容 `prompt_tokens_details.cached_tokens`；
|
||||
Anthropic `cache_read_input_tokens`。流式须带 `stream_options.include_usage`；断连按已收 usage 计，
|
||||
否则估算且不缓存。
|
||||
|
||||
**缓存感知技术清单（12 条）**：① 规范化序列化（稳定键序、剔易变字段）② 前缀钉扎（资料置顶问题置末
|
||||
→ 上游 1/30 价命中）③ 桶作用域+版本失效（资料更新=版本+1）④ 两级缓存（复用 cache.py L1/L2 骨架，
|
||||
L2 升级 embedding）⑤ 向量 int8+LRU ⑥ Singleflight 合并 ⑦ 空闲时段队列（非交互任务 off-peak 半价）
|
||||
⑧ 前缀预热（max_tokens=1 廉价调用）⑨ 上游命中遥测回灌校准阈值 ⑩ L0 只缓存单轮（多轮靠上游前缀
|
||||
自然命中，防上下文污染）⑪ 失败不缓存 ⑫ model_pool 补 hit_price 字段按命中价选上游。
|
||||
|
||||
**接入现有项目五步**：① `gateway/proxy/` 包 + `include_router`，router_system 零改动（缓存算法
|
||||
直接 import）；② `config/settings.json` 加 `proxy` 段（桶/价格表/限流/allow_proxy），主 key 走 env；
|
||||
③ `webapp/src/views/ProxyView.vue` 三卡片（key 管理/用量查询/命中率-毛利看板）+ `npm run build`；
|
||||
④ `data/proxy.sqlite3`（gitignore 已含 data/）；⑤ 任务登记《任务拆解与执行计划.md》代理层节
|
||||
（T-P1 上游客户端+usage 归一化 / T-P2 鉴权账本 / T-P3 流式透传 / T-P4 整形器 / T-P5 语义缓存 /
|
||||
T-P6 计价 / T-P7 前端页 / T-P8 200 并发压测），AGENTS.md 文档地图补一行；测试资源隔离清单追加
|
||||
proxy.sqlite3。
|
||||
|
||||
**性能定案与逃生通道（2026-09-04 补充）**：代理为 I/O 密集型流式转发（重计算全部在 C 侧：
|
||||
llama.cpp 推理/embedding、numpy 检索、sqlite、hashlib），Python 代理开销 1–3ms/请求，
|
||||
占端到端延迟 <0.5%，峰值负载 <10% 单核——**确定用 Python**。硬性对策：向量检索必须
|
||||
numpy/hnswlib 且限定课程桶内；embedding 必须走 llama-server 端点（禁止 torch 在代理内推理）。
|
||||
**性能预算（T-P8 压测验收）**：代理附加 P99 ≤50ms、进程内存 ≤1GB、单核 ≥50 req/s；
|
||||
超标才启动数据面（透传+精确缓存）换 Go/Envoy 的局部手术——控制面/数据面分层 + 状态全外置
|
||||
sqlite 保证该手术成本可控。减轻硬件消耗的优先级：缓存命中率设计 ≫ 本地模型量化 ≫ 空闲调度
|
||||
≫ 代理语言（换 Go 仅省 ~150MB 内存）。
|
||||
|
||||
## 4. MVP 任务分解（供实现 agent 按序执行）
|
||||
|
||||
> **执行版已细化**：文件清单、DDL、接口签名、API 契约、逐任务验收见
|
||||
> **《实施方案_代理层与缓存层.md》**（T-P0…T-P8，约 12 工作日）。本节 P1–P5 为摘要，以执行版为准。
|
||||
|
||||
> 约束：主 key 只从环境变量 / `config/settings.json` 读取（该文件已 gitignore），
|
||||
> 任何代码、示例、测试中**不得出现真实凭据字面量**。
|
||||
|
||||
| # | 任务 | 验收 |
|
||||
|---|---|---|
|
||||
| P1 | 透传网关：`/proxy/v1/chat/completions`（OpenAI 兼容、流式透传） | curl 可用；流式与非流式均通 |
|
||||
| P2 | 计量计费：解析上游 usage（含 `prompt_cache_hit_tokens`）；sqlite 账本；欠额熔断 | 每请求成本/收入可查；欠额请求被拒 |
|
||||
| P3 | 缓存栈：前缀整形器 + L0 语义缓存接入；命中率埋点 | 埋点输出 h_g、h_p、假想直连成本对比 |
|
||||
| P4 | 账号配额：代理 key 签发/注销、额度、限流 | key 生命周期可管理；限流生效 |
|
||||
| P5 | 看板：命中率/毛利/用量视图（复用 MetricsView 模式） | 三卡片可见 |
|
||||
|
||||
**端到端验收基线**：构造 ≥200 条校园模拟请求（重复/同义变体占 ≥50%），`h_g + h_p ≥ 50%`，
|
||||
在合理定价表下毛利为正。
|
||||
|
||||
## 5. 合规与风险（实现前必读）
|
||||
|
||||
1. **转售授权**：向上游 API 的转售/多租户使用可能受其服务条款限制，真实收费运营前必须确认
|
||||
上游商用/分销政策；个人 key 转售存在封号风险。
|
||||
2. **备案要求**：面向不特定公众提供生成式 AI 服务在国内需完成备案；校内限定人群合规负担小——
|
||||
**从校内试点起步**。
|
||||
3. **主 key 安全**：仅存 env / `config/settings.json`；代码、文档、测试零字面量。
|
||||
4. **缓存正确性**：语义缓存必须带 TTL 与失效策略，并按课程/时间分桶隔离——资料更新后旧答案
|
||||
不能续用；作业场景"相似题 ≠ 可复用答案"。
|
||||
5. **竞争**：学生自有 key、免费额度是替代品；定价锚定"便利性 + 稳定供给"，不锚定成本。
|
||||
|
||||
## 6. 实验设计（论文/答辩素材）
|
||||
|
||||
- **E-P1 命中率-毛利曲线**：模拟 500 条校园请求（重复率 0% / 30% / 60% 三档），
|
||||
测 h_g、h_p、混合成本、毛利率三组数字。
|
||||
- **E-P2 前缀整形 A/B**：整形 vs 不整形的上游 `prompt_cache_hit_tokens` 对比——
|
||||
证明"缓存钱"是设计出来的，不是碰运气。
|
||||
- **E-P3 语义缓存质量**：L0 直答复用的正确性抽检（分桶/TTL 策略对照）。
|
||||
|
||||
**推理引擎选型（2026-09-04 定案）**：客户端（C 端）**llama.cpp，无悬念**——学生 Windows 本机
|
||||
只有它能单文件零依赖运行（vLLM 仅 Linux + CUDA + torch 数 GB）。校端**默认 llama.cpp**：
|
||||
本地层峰值并发 10–50 路，`--parallel` 足够；`--cache-reuse`/KV 量化与交流文本协议已耦合并经
|
||||
E1 验证；与客户端同运行时同打包，不破坏"干净环境 20 分钟"验收。**vLLM = 条件启用的部署选项**，
|
||||
不是替换——它说 OpenAI 兼容协议，启用即 model_pool 加一个端点条目（零代码）。触发条件
|
||||
（三条同时满足）：学校提供 Linux GPU 服务器（16GB+ 显存）+ 本地层持续并发 >50–100 路 +
|
||||
本地层 token 占比成为吞吐瓶颈。推理引擎是可替换端点而非架构承诺；D1（只捆绑 llama.cpp
|
||||
上游二进制）约束客户端分发件，不限制服务端池子的上游类型。
|
||||
|
||||
## 7. 与毕业设计主命题的关系
|
||||
|
||||
不改变已定命题《基于端云协同的编程智能体系统设计与实现》。本方向作为扩展章/答辩亮点：
|
||||
**代理层 = 端侧基础设施的规模化形态**（从单机端侧到校园级端侧），核心贡献是
|
||||
"缓存感知的 LLM 代理网关"——命中率和毛利曲线都是可量化、可复现的系统贡献。
|
||||
@@ -1,4 +1,6 @@
|
||||
# 端云协同 LLM 协作系统（v2）
|
||||
# 端云协同编程智能体系统（v2）
|
||||
|
||||
> **论文题目**：《基于端云协同的编程智能体系统设计与实现》（Design and Implementation of a Programming Agent System Based on Device-Cloud Collaboration）
|
||||
|
||||
大模型（API，Architect）做任务分析/决策/终审 ＋ 本地小模型（llama.cpp，Worker）做实现/自验证，
|
||||
两者通过**「交流文本」**（一份 schema 约束的结构化 JSON 共享工作区）交接，互不共享内部状态，
|
||||
|
||||
@@ -0,0 +1,95 @@
|
||||
"""一次性验证脚本：真实 DeepSeek 架构师 + 脚手架 Worker，走完整协作四步。
|
||||
|
||||
步骤映射：
|
||||
1) architect.brief 分析拆解 -> 交流文本（交接文档）
|
||||
2) worker 读交接文档构建实现（第 1 次故意输出不合格 -> 触发问题）
|
||||
3) worker 验证失败 -> issue 交接至文档 -> architect.decide 裁决（真实 API）
|
||||
4) worker 按裁决修复 -> 全步完成 -> architect.final_review 终审（真实 API）
|
||||
"""
|
||||
import asyncio
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
|
||||
from gateway.api import build_agent_chat # noqa: E402 复用 .env 里的 key 构建客户端
|
||||
from router_system.architect import build_architect # noqa: E402
|
||||
from router_system.pipeline import CollaborativePipeline # noqa: E402
|
||||
from router_system.worker import WorkerLoop # noqa: E402
|
||||
|
||||
QUERY = "用 Python 写一个函数 count_primes(n)，返回小于 n 的质数个数，附单元测试。"
|
||||
|
||||
GOOD_CODE = '''```python
|
||||
def count_primes(n):
|
||||
if n < 3:
|
||||
return 0
|
||||
is_prime = [True] * n
|
||||
is_prime[0] = is_prime[1] = False
|
||||
for i in range(2, int(n ** 0.5) + 1):
|
||||
if is_prime[i]:
|
||||
for j in range(i * i, n, i):
|
||||
is_prime[j] = False
|
||||
return sum(is_prime)
|
||||
|
||||
|
||||
def test_count_primes():
|
||||
assert count_primes(0) == 0
|
||||
assert count_primes(2) == 0
|
||||
assert count_primes(10) == 4
|
||||
assert count_primes(20) == 8
|
||||
```'''
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
cfg = {"base_url": "https://api.deepseek.com", "model": "deepseek-chat"}
|
||||
architect = build_architect({**cfg, "api_key": _env_key()})
|
||||
print(f"[architect] model={architect.model} key={'已配置' if architect.api_key else '缺失'}")
|
||||
|
||||
calls = {"n": 0}
|
||||
|
||||
async def scripted_generate(prompt: str) -> str:
|
||||
"""第 1 次故意交不合格产物（触发 issue->decide），第 2 次交合格实现。"""
|
||||
calls["n"] += 1
|
||||
if calls["n"] == 1:
|
||||
print("[worker] 第 1 次生成：故意输出不合格（无代码块）")
|
||||
return "这一步我没想清楚，先给个思路：应该用筛法，但代码还没写。"
|
||||
print("[worker] 第 2 次生成：交出完整实现（含测试）")
|
||||
return GOOD_CODE
|
||||
|
||||
worker = WorkerLoop(generate=scripted_generate, max_fix_attempts=1,
|
||||
model_used="scripted-worker")
|
||||
pipe = CollaborativePipeline(architect=architect, worker=worker,
|
||||
fast_path=False, rounds_cap=6, api_token_cap=30000)
|
||||
result = await pipe.run(QUERY, request_id="collabdemo01")
|
||||
|
||||
print("\n========== 路线 ==========")
|
||||
print(" -> ".join(result.route))
|
||||
ws = json.loads(Path(result.workspace_path).read_text(encoding="utf-8"))
|
||||
print("\n========== 交流文本关键内容 ==========")
|
||||
print("brief.goal:", ws["brief"]["goal"][:80])
|
||||
print("plan:", [(p["id"], p["task"][:36]) for p in ws["brief"]["plan"]])
|
||||
print("issues:", [(i["id"], i["step"], i["summary"][:40]) for i in ws.get("issues", [])])
|
||||
print("decisions:", [(d["ref"], d["reply"][:60]) for d in ws.get("decisions", [])])
|
||||
print("progress:", [(p["step"], p["status"]) for p in ws.get("progress", [])])
|
||||
print("final verdict:", ws["meta"].get("review_verdict", "(看 route)"))
|
||||
print("\n========== 结果 ==========")
|
||||
print("status:", result.status, "| rounds:", result.rounds_used,
|
||||
"| api_tokens:", result.api_input_tokens, "+", result.api_output_tokens,
|
||||
"| latency:", round(result.latency_ms), "ms")
|
||||
print("response 前 300 字：\n", result.response[:300])
|
||||
|
||||
|
||||
def _env_key() -> str:
|
||||
import os
|
||||
from dotenv import load_dotenv
|
||||
load_dotenv(Path(__file__).resolve().parent / ".env")
|
||||
key = os.environ.get("DEEPSEEK_API_KEY")
|
||||
if not key:
|
||||
print("缺少 DEEPSEEK_API_KEY，无法做真实协作验证")
|
||||
sys.exit(1)
|
||||
return key
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,97 @@
|
||||
# 代码领域知识规则（专家系统风格：关键词子串匹配 + 优先级）
|
||||
# subdomain：子领域细化标注（algorithm/debugging/web/database/devops/...）
|
||||
# 规则分类：template 规则 → 触发任务拆解；output 规则 → 前向链直接产出知识章节
|
||||
domain: code
|
||||
rules:
|
||||
# ---- 任务拆解类规则（template）----
|
||||
- id: code-sort
|
||||
priority: 90
|
||||
subdomain: algorithm
|
||||
patterns: ["排序", "快速排序", "排序算法", "二分查找", "sort", "quicksort", "binary search"]
|
||||
template: code-implement
|
||||
|
||||
- id: code-debug
|
||||
priority: 85
|
||||
subdomain: debugging
|
||||
patterns: ["报错", "错误", "调试", "bug", "debug", "typeerror", "异常", "报错信息"]
|
||||
template: code-debug
|
||||
|
||||
- id: code-algorithm
|
||||
priority: 82
|
||||
subdomain: algorithm
|
||||
patterns: ["算法", "数据结构", "递归", "动态规划", "回溯", "贪心", "algorithm", "recursion",
|
||||
"dp", "bfs", "dfs", "链表", "栈", "队列", "树", "图"]
|
||||
template: code-algorithm
|
||||
|
||||
- id: code-refactor
|
||||
priority: 80
|
||||
subdomain: refactoring
|
||||
patterns: ["重构", "优化代码", "改善", "refactor", "代码质量", "清理代码", "消除重复"]
|
||||
template: code-refactor
|
||||
|
||||
- id: code-database
|
||||
priority: 78
|
||||
subdomain: database
|
||||
patterns: ["数据库", "sql", "索引", "查询", "database"]
|
||||
template: code-implement
|
||||
|
||||
- id: code-explain
|
||||
priority: 76
|
||||
subdomain: reading
|
||||
patterns: ["解释这段代码", "这段代码什么意思", "代码讲解", "逐行解释", "explain this code",
|
||||
"理解这段代码", "分析这段代码"]
|
||||
template: code-explain
|
||||
|
||||
- id: code-test
|
||||
priority: 74
|
||||
subdomain: testing
|
||||
patterns: ["单元测试", "测试用例", "写测试", "unittest", "pytest", "单测", "测试代码"]
|
||||
template: code-test
|
||||
|
||||
- id: code-web
|
||||
priority: 72
|
||||
subdomain: web
|
||||
patterns: ["前端", "后端", "接口", "api", "flask", "django", "爬虫", "web", "restful"]
|
||||
template: code-implement
|
||||
|
||||
- id: code-implement-general
|
||||
priority: 50
|
||||
subdomain: implementation
|
||||
patterns: ["实现", "编写", "写一个", "函数", "代码", "编程", "用 python", "用 java",
|
||||
"用 javascript", "sql", "接口", "算法"]
|
||||
template: code-implement
|
||||
|
||||
# ---- 知识问答类规则（output：前向链直接产出答案章节）----
|
||||
- id: code-git-knowledge
|
||||
priority: 60
|
||||
subdomain: devops
|
||||
patterns: ["git", "版本控制", "提交", "分支", "merge", "rebase", "clone", "commit",
|
||||
"push", "pull", "回滚", "冲突"]
|
||||
output: |
|
||||
（git 知识）关于「{query}」：
|
||||
- git 核心概念：工作区 / 暂存区 / 本地仓库 / 远程仓库
|
||||
- 常用命令：git add → git commit → git push；git pull 同步远端
|
||||
- 回滚：git reset --hard <commit> 丢弃本地修改；git revert 生成反向提交
|
||||
- 冲突处理：先 git pull 拉取最新，手动解决冲突标记后 add+commit
|
||||
- 分支：git checkout -b <name> 新建并切换；git merge <branch> 合并
|
||||
|
||||
- id: code-docker-knowledge
|
||||
priority: 58
|
||||
subdomain: devops
|
||||
patterns: ["docker", "容器", "镜像", "k8s", "kubernetes", "部署", "compose"]
|
||||
output: |
|
||||
（部署知识）关于「{query}」：
|
||||
- 镜像 vs 容器：镜像=只读模板，容器=运行实例
|
||||
- 常用命令：docker build -t <name> . ; docker run -p 8000:8000 <name>
|
||||
- 多服务编排：docker-compose.yml 定义 services/ports/volumes
|
||||
- 生产要点：固定镜像版本、非 root 运行、健康检查、日志落盘
|
||||
|
||||
- id: code-python-knowledge
|
||||
priority: 55
|
||||
subdomain: environment
|
||||
patterns: ["python 语法", "python 技巧", "virtualenv", "pip", "venv", "环境", "安装依赖"]
|
||||
output: |
|
||||
（python 环境知识）关于「{query}」：
|
||||
- 虚拟环境：python -m venv .venv 创建；.venv/Scripts/activate 激活
|
||||
- 依赖管理：pip install -r requirements.txt；pip freeze > requirements.txt
|
||||
- 常用技巧：列表推导式、with 上下文管理、f-string 格式化
|
||||
@@ -0,0 +1,67 @@
|
||||
# 教育领域知识规则 + 事实表（学习/考试/语言/课程/职业）
|
||||
domain: education
|
||||
rules:
|
||||
- id: edu-study-method
|
||||
priority: 88
|
||||
subdomain: study
|
||||
patterns: ["学习方法", "怎么学", "高效学习", "记忆", "复习", "预习", "笔记", "专注",
|
||||
"study", "学习效率", "拖延"]
|
||||
template: edu-guide
|
||||
|
||||
- id: edu-exam
|
||||
priority: 86
|
||||
subdomain: exam
|
||||
patterns: ["考试", "备考", "复习计划", "刷题", "模拟考", "中考", "高考", "考研",
|
||||
"exam", "备考", "考前"]
|
||||
template: edu-guide
|
||||
|
||||
- id: edu-language
|
||||
priority: 84
|
||||
subdomain: language
|
||||
patterns: ["英语", "单词", "口语", "听力", "雅思", "托福", "四级", "六级", "日语",
|
||||
"英语学习", "language", "背单词"]
|
||||
template: edu-guide
|
||||
|
||||
- id: edu-course
|
||||
priority: 82
|
||||
subdomain: course
|
||||
patterns: ["选课", "课程", "专业选择", "报班", "网课", "自学", "教材", "course",
|
||||
"考研专业"]
|
||||
template: edu-guide
|
||||
|
||||
- id: edu-career
|
||||
priority: 80
|
||||
subdomain: career
|
||||
patterns: ["职业规划", "求职", "面试", "简历", "实习", "跳槽", "转行", "职场",
|
||||
"career", "job", "interview"]
|
||||
template: edu-guide
|
||||
|
||||
- id: edu-general
|
||||
priority: 50
|
||||
patterns: ["教育", "学习", "读书", "education", "learn", "培养"]
|
||||
template: edu-guide
|
||||
|
||||
facts:
|
||||
- id: edu-study-spaced
|
||||
keywords: ["学习方法", "记忆", "复习"]
|
||||
statement: "间隔重复优于集中突击：按 1 天/3 天/7 天节奏复习，用主动回忆（合上资料自测）代替重读"
|
||||
|
||||
- id: edu-focus-pomodoro
|
||||
keywords: ["专注", "拖延", "效率"]
|
||||
statement: "番茄工作法：25 分钟专注 + 5 分钟休息；学习前明确单一目标，关闭手机通知减少切换损耗"
|
||||
|
||||
- id: edu-exam-plan
|
||||
keywords: ["备考", "考试", "复习计划"]
|
||||
statement: "备考建议：先做真题定位薄弱点，再按薄弱点分配时间；考前一周以错题与框架回顾为主"
|
||||
|
||||
- id: edu-language-input
|
||||
keywords: ["英语", "口语", "听力"]
|
||||
statement: "语言学习坚持可理解输入：听力用慢速英语起步，口语先模仿跟读再自由表达，每日 30 分钟优于周末突击"
|
||||
|
||||
- id: edu-career-resume
|
||||
keywords: ["简历", "面试", "求职"]
|
||||
statement: "简历用 STAR 结构描述经历（情境-任务-行动-结果），量化成果；面试准备 2-3 个完整项目故事"
|
||||
|
||||
- id: edu-course-choice
|
||||
keywords: ["选课", "专业选择", "报班"]
|
||||
statement: "选课/选专业先看培养方案与就业方向，试听试学后再决定；网课优先选有答疑与作业反馈的"
|
||||
@@ -0,0 +1,73 @@
|
||||
# 金融领域知识规则 + 事实表（理财科普口径，非投资建议）
|
||||
domain: finance
|
||||
rules:
|
||||
# ---- 投资理财 ----
|
||||
- id: finance-investing
|
||||
priority: 90
|
||||
subdomain: investing
|
||||
patterns: ["理财", "投资", "基金", "股票", "债券", "收益", "定投", "指数", "回撤",
|
||||
"invest", "fund", "stock", "portfolio", "收益率"]
|
||||
template: finance-advice
|
||||
|
||||
- id: finance-saving
|
||||
priority: 78
|
||||
subdomain: saving
|
||||
patterns: ["存款", "储蓄", "定期", "活期", "利息", "存钱", "deposit", "interest"]
|
||||
template: finance-advice
|
||||
|
||||
- id: finance-loan
|
||||
priority: 84
|
||||
subdomain: loan
|
||||
patterns: ["贷款", "房贷", "车贷", "消费贷", "借呗", "花呗", "还款", "月供", "贷款利率",
|
||||
"loan", "mortgage", "lpr", "等额本息", "等额本金"]
|
||||
template: finance-advice
|
||||
|
||||
- id: finance-insurance
|
||||
priority: 82
|
||||
subdomain: insurance
|
||||
patterns: ["保险", "重疾险", "医疗险", "意外险", "寿险", "保费", "理赔", "退保",
|
||||
"insurance", "policy"]
|
||||
template: finance-advice
|
||||
|
||||
- id: finance-credit-card
|
||||
priority: 80
|
||||
subdomain: credit
|
||||
patterns: ["信用卡", "额度", "账单", "分期", "逾期", "征信", "信用", "credit card",
|
||||
"套现", "最低还款"]
|
||||
template: finance-advice
|
||||
|
||||
- id: finance-personal-budget
|
||||
priority: 76
|
||||
subdomain: budgeting
|
||||
patterns: ["记账", "预算", "开销", "省钱", "开支", "家庭财务", "财务规划", "budget"]
|
||||
template: finance-advice
|
||||
|
||||
- id: finance-general
|
||||
priority: 50
|
||||
patterns: ["金融", "钱", "财务", "finance", "money", "经济"]
|
||||
template: finance-advice
|
||||
|
||||
facts:
|
||||
- id: finance-risk-warning
|
||||
keywords: ["理财", "投资", "基金", "股票"]
|
||||
statement: "理财投资均存在风险，收益与风险成正比；过往业绩不代表未来表现，切勿用急用钱投资"
|
||||
|
||||
- id: finance-deposit-insurance
|
||||
keywords: ["存款", "储蓄", "银行"]
|
||||
statement: "我国实行存款保险制度，同一存款人在同一家银行的存款本息合计 50 万元以内受国家保障"
|
||||
|
||||
- id: finance-loan-lpr
|
||||
keywords: ["贷款", "房贷", "利率"]
|
||||
statement: "房贷利率通常与 LPR（贷款市场报价利率）挂钩，等额本息月供固定、等额本金逐月递减"
|
||||
|
||||
- id: finance-insurance-buy
|
||||
keywords: ["保险", "重疾险", "医疗险"]
|
||||
statement: "投保遵循先保障后理财原则，优先配置医疗险、重疾险等基础保障；如实告知健康状况，避免理赔纠纷"
|
||||
|
||||
- id: finance-credit-overdue
|
||||
keywords: ["信用卡", "逾期", "征信"]
|
||||
statement: "信用卡逾期会影响个人征信记录，产生违约金与利息；建议设置自动还款避免逾期"
|
||||
|
||||
- id: finance-emergency-fund
|
||||
keywords: ["预算", "记账", "省钱"]
|
||||
statement: "家庭财务建议预留 3-6 个月支出的应急金，再考虑投资；先记账 1-2 个月摸清开支结构再优化"
|
||||
@@ -0,0 +1,39 @@
|
||||
# 通用领域知识规则（subdomain：writing/translate/explain/compare）
|
||||
domain: general
|
||||
rules:
|
||||
# ---- 任务拆解类规则（template）----
|
||||
- id: general-explain
|
||||
priority: 35
|
||||
subdomain: explain
|
||||
patterns: ["总结", "介绍", "解释", "为什么", "优缺点", "是什么", "原理", "机制",
|
||||
"summarize", "explain", "what is", "推荐", "评价", "概念"]
|
||||
template: general-explain
|
||||
|
||||
- id: general-writing
|
||||
priority: 33
|
||||
subdomain: writing
|
||||
patterns: ["写一封", "写一篇", "作文", "邮件", "报告", "文案", "简历", "论文", "总结报告",
|
||||
"write an email", "essay"]
|
||||
template: general-writing
|
||||
|
||||
- id: general-compare
|
||||
priority: 31
|
||||
subdomain: compare
|
||||
patterns: ["区别", "对比", "比较", "哪个好", "差异", "versus", "vs", "compare", "difference"]
|
||||
template: general-explain
|
||||
|
||||
- id: general-translate
|
||||
priority: 30
|
||||
subdomain: translate
|
||||
patterns: ["翻译", "translate", "译成", "译成英文", "译成中文", "英文怎么说"]
|
||||
output: |
|
||||
（翻译提示）关于「{query}」：
|
||||
- 先确定源语言与目标语言、语气（正式/口语）与受众
|
||||
- 专有名词与术语保持一致性，必要时保留原文并注释
|
||||
- 翻译后回读检查语义完整性与自然度
|
||||
|
||||
- id: general-knowledge
|
||||
priority: 20
|
||||
subdomain: explain
|
||||
patterns: ["概念", "原理", "机制", "concept", "principle"]
|
||||
template: general-explain
|
||||
@@ -0,0 +1,120 @@
|
||||
# 法律领域知识规则 + 事实表（subdomain：labor/contract/ip/housing/family/...）
|
||||
domain: legal
|
||||
rules:
|
||||
# ---- 任务拆解类规则（template）----
|
||||
- id: legal-contract
|
||||
priority: 92
|
||||
subdomain: contract
|
||||
patterns: ["合同", "条款", "违约", "竞业", "contract", "clause", "赔偿", "违约金"]
|
||||
template: legal-advice
|
||||
|
||||
- id: legal-labor
|
||||
priority: 93
|
||||
subdomain: labor
|
||||
patterns: ["加班", "工资", "辞退", "裁员", "试用期", "社保", "公积金", "年假", "请假",
|
||||
"劳动法", "离职", "解除劳动合同", "加班费", "经济补偿"]
|
||||
template: legal-advice
|
||||
|
||||
- id: legal-ip
|
||||
priority: 86
|
||||
subdomain: ip
|
||||
patterns: ["专利", "版权", "商标", "知识产权", "盗版", "侵权", "patent", "copyright", "trademark"]
|
||||
template: legal-advice
|
||||
|
||||
- id: legal-housing
|
||||
priority: 84
|
||||
subdomain: housing
|
||||
patterns: ["租房", "买房", "购房", "房产", "物业", "租金", "押金", "房东", "房贷",
|
||||
"property", "rent", "lease"]
|
||||
template: legal-advice
|
||||
|
||||
- id: legal-marriage
|
||||
priority: 82
|
||||
subdomain: family
|
||||
patterns: ["离婚", "结婚", "婚姻", "继承", "遗产", "抚养权", "彩礼", "marriage", "divorce",
|
||||
"遗嘱", "赡养"]
|
||||
template: legal-advice
|
||||
|
||||
- id: legal-tax
|
||||
priority: 78
|
||||
subdomain: tax
|
||||
patterns: ["个税", "税务", "发票", "报税", "tax", "开票", "税率"]
|
||||
template: legal-advice
|
||||
|
||||
- id: legal-consumer
|
||||
priority: 76
|
||||
subdomain: consumer
|
||||
patterns: ["退款", "退货", "消费者", "七天无理由", "维权", "投诉", "商家", "网购",
|
||||
"consumer", "refund"]
|
||||
template: legal-advice
|
||||
|
||||
- id: legal-litigation
|
||||
priority: 74
|
||||
subdomain: litigation
|
||||
patterns: ["诉讼", "仲裁", "起诉", "判决", "上诉", "执行", "litigation", "arbitration"]
|
||||
template: legal-advice
|
||||
|
||||
- id: legal-compliance
|
||||
priority: 70
|
||||
subdomain: compliance
|
||||
patterns: ["合规", "法规", "法条", "监管", "compliance", "regulatory", "数据合规"]
|
||||
template: legal-advice
|
||||
|
||||
- id: legal-general
|
||||
priority: 50
|
||||
subdomain: general
|
||||
patterns: ["法律", "law", "legal", "民法典", "刑法"]
|
||||
template: legal-advice
|
||||
|
||||
facts:
|
||||
# ---- 劳动 ----
|
||||
- id: legal-noncompete
|
||||
keywords: ["竞业", "离职", "同业"]
|
||||
statement: "竞业限制期限不得超过二年，且用人单位应在限制期内按月给予经济补偿"
|
||||
|
||||
- id: legal-renew-compensation
|
||||
keywords: ["不续签", "经济补偿", "劳动合同"]
|
||||
statement: "劳动合同期满用人单位不续签的，通常应支付经济补偿（每满一年一个月工资）"
|
||||
|
||||
- id: legal-breach-penalty
|
||||
keywords: ["违约金", "违约"]
|
||||
statement: "违约金过高的，当事人可请求法院适当减少；以实际损失为基础，兼顾合同履行情况"
|
||||
|
||||
- id: legal-overtime-pay
|
||||
keywords: ["加班", "加班费", "加班工资"]
|
||||
statement: "工作日加班支付不低于工资 150% 的报酬；休息日加班不能补休的支付 200%；法定节假日加班支付 300%"
|
||||
|
||||
- id: legal-illegal-termination
|
||||
keywords: ["辞退", "开除", "解除劳动合同"]
|
||||
statement: "用人单位违法解除劳动合同的，应支付二倍经济补偿标准的赔偿金（2N）"
|
||||
|
||||
- id: legal-probation
|
||||
keywords: ["试用期", "试用"]
|
||||
statement: "试用期最长不得超过六个月，且同一用人单位与同一劳动者只能约定一次试用期"
|
||||
|
||||
# ---- 房产/租房 ----
|
||||
- id: legal-deposit-refund
|
||||
keywords: ["押金", "租房", "退租"]
|
||||
statement: "租赁期满房屋无损坏且费用结清的，押金应全额退还；扣除须有合同依据或实际损失"
|
||||
|
||||
- id: legal-rent-contract
|
||||
keywords: ["租房", "房东", "租金"]
|
||||
statement: "租房建议签订书面合同，明确租期、租金、押金、维修责任与违约责任"
|
||||
|
||||
# ---- 婚姻/继承 ----
|
||||
- id: legal-inheritance-order
|
||||
keywords: ["继承", "遗产", "遗嘱"]
|
||||
statement: "法定继承顺序：第一顺序为配偶、子女、父母；第二顺序为兄弟姐妹、祖父母、外祖父母"
|
||||
|
||||
- id: legal-common-property
|
||||
keywords: ["离婚", "财产", "婚后"]
|
||||
statement: "婚姻关系存续期间所得工资、投资收益等一般为夫妻共同财产；分割时协议优先，协商不成由法院判决"
|
||||
|
||||
# ---- 消费者 ----
|
||||
- id: legal-seven-day-return
|
||||
keywords: ["七天无理由", "退货", "退款"]
|
||||
statement: "网络购物一般适用七日无理由退货（定作、鲜活易腐等除外）；退货商品应保持完好"
|
||||
|
||||
- id: legal-patent-process
|
||||
keywords: ["专利", "申请"]
|
||||
statement: "专利申请流程：提交申请 → 受理 → 初步审查/实质审查（发明）→ 授权公告，费用含申请费与年费"
|
||||
@@ -0,0 +1,77 @@
|
||||
# 生活领域知识规则 + 事实表（生活百科：美食/旅行/家居/宠物/健身/天气）
|
||||
domain: life
|
||||
rules:
|
||||
- id: life-food
|
||||
priority: 88
|
||||
subdomain: food
|
||||
patterns: ["菜谱", "做饭", "烹饪", "美食", "炒", "炖", "烤箱", "烘焙", "食材", "做法",
|
||||
"recipe", "cook", "料理"]
|
||||
template: life-guide
|
||||
|
||||
- id: life-travel
|
||||
priority: 86
|
||||
subdomain: travel
|
||||
patterns: ["旅行", "旅游", "攻略", "行程", "机票", "酒店", "签证", "景点", "自驾",
|
||||
"travel", "trip", " itinerary", "路线"]
|
||||
template: life-guide
|
||||
|
||||
- id: life-home
|
||||
priority: 84
|
||||
subdomain: home
|
||||
patterns: ["装修", "收纳", "家居", "家电", "清洁", "打扫", "家具", "软装", "home",
|
||||
"整理", "断舍离"]
|
||||
template: life-guide
|
||||
|
||||
- id: life-pet
|
||||
priority: 82
|
||||
subdomain: pet
|
||||
patterns: ["宠物", "猫", "狗", "遛狗", "猫粮", "狗粮", "驱虫", "疫苗", "宠物医院",
|
||||
"pet", "cat", "dog", "铲屎"]
|
||||
template: life-guide
|
||||
|
||||
- id: life-fitness
|
||||
priority: 80
|
||||
subdomain: fitness
|
||||
patterns: ["健身", "锻炼", "跑步", "深蹲", "俯卧撑", "增肌", "减脂", "运动计划",
|
||||
"workout", "gym", "跑步机", "瑜伽"]
|
||||
template: life-guide
|
||||
|
||||
- id: life-weather
|
||||
priority: 78
|
||||
subdomain: weather
|
||||
patterns: ["天气", "气温", "下雨", "台风", "降温", "weather", "预报", "湿度"]
|
||||
output: |
|
||||
（生活常识）关于「{query}」：
|
||||
- 建议通过天气 App 查看实时预报（温度/降水概率/风力/空气质量）
|
||||
- 极端天气（台风/暴雨/高温）关注气象预警信号，做好防护
|
||||
- 出行穿衣参考体感温度（气温+湿度+风力综合）
|
||||
|
||||
- id: life-general
|
||||
priority: 50
|
||||
patterns: ["生活", "常识", "小妙招", "life", "日常"]
|
||||
template: life-guide
|
||||
|
||||
facts:
|
||||
- id: life-travel-checklist
|
||||
keywords: ["旅行", "旅游", "出行"]
|
||||
statement: "出行前准备：证件（身份证/护照）、行程确认、常用药、充电设备；境外游注意签证与保险"
|
||||
|
||||
- id: life-food-cooking
|
||||
keywords: ["做饭", "菜谱", "烹饪"]
|
||||
statement: "家常菜要点：热锅冷油防粘、盐最后放保留鲜味、肉类先腌后炒更嫩；生熟分开防交叉污染"
|
||||
|
||||
- id: life-pet-vaccine
|
||||
keywords: ["宠物", "疫苗", "猫", "狗"]
|
||||
statement: "幼猫幼犬按龄接种疫苗（如猫三联/犬联苗），之后每年加强；定期体内外驱虫并记录免疫本"
|
||||
|
||||
- id: life-fitness-basic
|
||||
keywords: ["健身", "锻炼", "增肌", "减脂"]
|
||||
statement: "健身三要素：渐进负荷、营养充足（蛋白质）、充分恢复（睡眠）；新手先学动作模式再上重量"
|
||||
|
||||
- id: life-home-cleaning
|
||||
keywords: ["清洁", "收纳", "打扫"]
|
||||
statement: "收纳先分类后收纳（断舍离-分类-定位）；厨房油污用温碱水；浴室防霉注意通风除湿"
|
||||
|
||||
- id: life-travel-weather
|
||||
keywords: ["旅行", "天气"]
|
||||
statement: "长途旅行前查看目的地未来一周天气，准备应对衣物与雨具；山区/海边注意温差与防晒"
|
||||
@@ -0,0 +1,68 @@
|
||||
# 数学领域知识规则（subdomain：algebra/calculus/geometry/probability/...）
|
||||
domain: math
|
||||
rules:
|
||||
# ---- 任务拆解类规则（template）----
|
||||
- id: math-equation
|
||||
priority: 90
|
||||
subdomain: algebra
|
||||
patterns: ["方程", "求解", "求根", "solve", "equation", "解方程", "一元二次", "一元一次",
|
||||
"等于多少", "x 等于", "求 x"]
|
||||
template: math-solve
|
||||
|
||||
- id: math-calculus
|
||||
priority: 88
|
||||
subdomain: calculus
|
||||
patterns: ["积分", "导数", "微积分", "求导", "定积分", "不定积分", "integral", "derivative", "∫", "lim", "极限"]
|
||||
template: math-solve
|
||||
|
||||
- id: math-algebra
|
||||
priority: 84
|
||||
subdomain: algebra
|
||||
patterns: ["代数", "矩阵", "向量", "多项式", "代数式", "因式分解", "algebra", "matrix",
|
||||
"polynomial", "行列式", "线性方程组"]
|
||||
template: math-solve
|
||||
|
||||
- id: math-geometry
|
||||
priority: 82
|
||||
subdomain: geometry
|
||||
patterns: ["几何", "三角形", "圆", "面积", "体积", "周长", "勾股", "相似", "全等",
|
||||
"geometry", "angle", "直角"]
|
||||
template: math-solve
|
||||
|
||||
- id: math-proof
|
||||
priority: 80
|
||||
subdomain: proof
|
||||
patterns: ["证明", "定理", "推导", "反证", "归纳", "proof", "theorem", "推论"]
|
||||
template: math-proof
|
||||
|
||||
- id: math-probability
|
||||
priority: 78
|
||||
subdomain: probability
|
||||
patterns: ["概率", "统计", "期望", "方差", "分布", "组合", "排列", "probability",
|
||||
"statistics", "排列组合"]
|
||||
template: math-solve
|
||||
|
||||
- id: math-number-theory
|
||||
priority: 76
|
||||
subdomain: number-theory
|
||||
patterns: ["素数", "质数", "整除", "最大公约数", "最小公倍数", "同余", "数论", "gcd", "mod"]
|
||||
template: math-solve
|
||||
|
||||
- id: math-trigonometry
|
||||
priority: 74
|
||||
subdomain: trigonometry
|
||||
patterns: ["三角", "正弦", "余弦", "正切", "sin", "cos", "tan", "弧度", "角度"]
|
||||
template: math-solve
|
||||
|
||||
- id: math-optimization
|
||||
priority: 72
|
||||
subdomain: optimization
|
||||
patterns: ["最优化", "极值", "最大值", "最小值", "最大化", "最小化", "最优解", "optimize",
|
||||
"最大值最小值", "约束优化"]
|
||||
template: math-optimize
|
||||
|
||||
- id: math-general
|
||||
priority: 50
|
||||
subdomain: general
|
||||
patterns: ["数学", "计算", "等于", "math", "求值", "求和"]
|
||||
template: math-solve
|
||||
@@ -0,0 +1,115 @@
|
||||
# 医学领域知识规则 + 事实表（科普口径，非诊断；subdomain：chronic/drug/firstaid/...）
|
||||
domain: medical
|
||||
rules:
|
||||
# ---- 任务拆解类规则（template）----
|
||||
- id: medical-hypertension
|
||||
priority: 92
|
||||
subdomain: chronic
|
||||
patterns: ["高血压", "hypertension", "血压", "高血脂", "血脂"]
|
||||
template: medical-advice
|
||||
|
||||
- id: medical-drug
|
||||
priority: 93
|
||||
subdomain: medication
|
||||
patterns: ["药物", "吃药", "剂量", "副作用", "退烧药", "降压药", "抗生素", "dosage",
|
||||
"prescription", "用药"]
|
||||
template: medical-advice
|
||||
|
||||
- id: medical-common
|
||||
priority: 88
|
||||
subdomain: common
|
||||
patterns: ["感冒", "发烧", "咳嗽", "流感", "嗓子疼", "头痛", "cold", "fever", "cough"]
|
||||
template: medical-advice
|
||||
|
||||
- id: medical-chronic
|
||||
priority: 86
|
||||
subdomain: chronic
|
||||
patterns: ["糖尿病", "血糖", "diabetes", "痛风", "尿酸", "甲状腺"]
|
||||
template: medical-advice
|
||||
|
||||
- id: medical-digestive
|
||||
priority: 84
|
||||
subdomain: digestive
|
||||
patterns: ["胃", "腹泻", "便秘", "消化不良", "肠胃", "拉肚子", "腹胀", "stomach"]
|
||||
template: medical-advice
|
||||
|
||||
- id: medical-nutrition
|
||||
priority: 82
|
||||
subdomain: nutrition
|
||||
patterns: ["减肥", "饮食", "营养", "卡路里", "热量", "膳食", "节食", "diet", "nutrition"]
|
||||
template: medical-advice
|
||||
|
||||
- id: medical-mental
|
||||
priority: 80
|
||||
subdomain: mental
|
||||
patterns: ["失眠", "焦虑", "抑郁", "压力", "睡眠", "情绪", "insomnia", "anxiety"]
|
||||
template: medical-advice
|
||||
|
||||
- id: medical-firstaid
|
||||
priority: 78
|
||||
subdomain: firstaid
|
||||
patterns: ["烫伤", "烧伤", "止血", "扭伤", "中暑", "触电", "溺水", "急救", "first aid",
|
||||
"划伤", "擦伤"]
|
||||
template: medical-firstaid
|
||||
|
||||
- id: medical-pediatrics
|
||||
priority: 76
|
||||
subdomain: pediatrics
|
||||
patterns: ["儿童", "婴儿", "宝宝", "小孩", "儿科", "幼儿", "child", "baby", "infant"]
|
||||
template: medical-advice
|
||||
|
||||
- id: medical-general
|
||||
priority: 50
|
||||
subdomain: general
|
||||
patterns: ["医疗", "症状", "诊断", "治疗", "医生", "患者", "手术", "疫苗", "感染",
|
||||
"medical", "symptom", "disease", "clinic"]
|
||||
template: medical-advice
|
||||
|
||||
facts:
|
||||
# ---- 慢病 ----
|
||||
- id: medical-hypertension-diet
|
||||
keywords: ["高血压", "饮食"]
|
||||
statement: "高血压患者应低盐低脂饮食、控制体重、规律运动、戒烟限酒，并在医生指导下用药"
|
||||
|
||||
- id: medical-diabetes-diet
|
||||
keywords: ["糖尿病", "饮食"]
|
||||
statement: "糖尿病患者应控制总热量、少食多餐、选择低 GI 主食、规律监测血糖并遵医嘱用药"
|
||||
|
||||
- id: medical-hypertension-care
|
||||
keywords: ["高血压", "血压", "监测"]
|
||||
statement: "高血压患者建议每日固定时间测量血压并记录；收缩压持续高于 180 或出现剧烈头痛、胸闷应立即就医"
|
||||
|
||||
# ---- 用药 ----
|
||||
- id: medical-drug-side-effect
|
||||
keywords: ["副作用", "药物"]
|
||||
statement: "用药前阅读说明书，注意禁忌与相互作用；出现严重不良反应应立即停药并就医"
|
||||
|
||||
- id: medical-fever-drug
|
||||
keywords: ["发烧", "退烧"]
|
||||
statement: "体温超过 38.5℃ 可在药师指导下使用退烧药；持续发热或出现严重症状应及时就医"
|
||||
|
||||
- id: medical-antibiotic
|
||||
keywords: ["抗生素", "消炎药"]
|
||||
statement: "抗生素为处方药，需医生诊断后使用；切勿自行服用或随意停药，以免耐药"
|
||||
|
||||
# ---- 常见病 ----
|
||||
- id: medical-cold-care
|
||||
keywords: ["感冒", "咳嗽", "流鼻涕"]
|
||||
statement: "普通感冒多为自限性，注意休息、多饮水；症状持续超过一周或加重应就医"
|
||||
|
||||
- id: medical-diarrhea-care
|
||||
keywords: ["腹泻", "拉肚子"]
|
||||
statement: "腹泻注意补充水分与电解质；出现血便、高热、持续呕吐或脱水迹象应立即就医"
|
||||
|
||||
# ---- 生活 ----
|
||||
- id: medical-nutrition-guide
|
||||
keywords: ["减肥", "饮食", "营养"]
|
||||
statement: "均衡膳食：蔬果、全谷物、优质蛋白搭配；减肥应控制总热量并配合运动，避免极端节食"
|
||||
|
||||
- id: medical-insomnia
|
||||
keywords: ["失眠", "睡眠"]
|
||||
statement: "改善睡眠：固定作息、睡前避免咖啡因与屏幕蓝光、白天适量运动；长期失眠应就医评估"
|
||||
|
||||
- id: medical-firstaid-burn
|
||||
keywords: ["烫伤", "烧伤"]
|
||||
statement: "烫伤急救：立即用流动冷水冲洗 15-20 分钟，切勿涂抹牙膏/酱油；水泡不要挑破，严重者尽快就医"
|
||||
@@ -0,0 +1,119 @@
|
||||
# 任务模板：Planner 拆解规则（子任务 DAG 定义）
|
||||
# steps 中每个节点：id（唯一）、kind（执行器动作类型）、domain、deps（依赖）、desc
|
||||
task_templates:
|
||||
# ---- 代码 ----
|
||||
code-implement:
|
||||
steps:
|
||||
- {id: analyze, kind: analyze, domain: code, desc: "需求与约束分析"}
|
||||
- {id: design, kind: design, domain: code, deps: [analyze], desc: "算法与数据结构设计"}
|
||||
- {id: implement, kind: implement, domain: code, deps: [design], desc: "实现代码"}
|
||||
- {id: verify, kind: verify, domain: code, deps: [implement], desc: "自测校验"}
|
||||
|
||||
code-debug:
|
||||
steps:
|
||||
- {id: analyze, kind: analyze, domain: code, desc: "错误现象与复现分析"}
|
||||
- {id: diagnose, kind: diagnose, domain: code, deps: [analyze], desc: "定位错误根因"}
|
||||
- {id: fix, kind: fix, domain: code, deps: [diagnose], desc: "给出修复方案"}
|
||||
- {id: verify, kind: verify, domain: code, deps: [fix], desc: "修复后验证"}
|
||||
|
||||
code-algorithm:
|
||||
steps:
|
||||
- {id: analyze, kind: analyze, domain: code, desc: "题意与输入输出分析"}
|
||||
- {id: design, kind: design, domain: code, deps: [analyze], desc: "算法选型（复杂度权衡）"}
|
||||
- {id: implement, kind: implement, domain: code, deps: [design], desc: "编码实现"}
|
||||
- {id: complexity, kind: complexity, domain: code, deps: [implement], desc: "时间/空间复杂度分析"}
|
||||
- {id: verify, kind: verify, domain: code, deps: [complexity], desc: "自测校验"}
|
||||
|
||||
code-refactor:
|
||||
steps:
|
||||
- {id: analyze, kind: analyze, domain: code, desc: "现状与问题分析"}
|
||||
- {id: refactor, kind: refactor, domain: code, deps: [analyze], desc: "重构方案（结构/命名/去重）"}
|
||||
- {id: verify, kind: verify, domain: code, deps: [refactor], desc: "行为等价验证"}
|
||||
|
||||
code-explain:
|
||||
steps:
|
||||
- {id: analyze, kind: analyze, domain: code, desc: "代码结构与流程梳理"}
|
||||
- {id: explain, kind: explain, domain: code, deps: [analyze], desc: "逐段讲解代码"}
|
||||
- {id: verify, kind: verify, domain: code, deps: [explain], desc: "理解正确性核对"}
|
||||
|
||||
code-test:
|
||||
steps:
|
||||
- {id: analyze, kind: analyze, domain: code, desc: "被测功能与边界分析"}
|
||||
- {id: design, kind: design, domain: code, deps: [analyze], desc: "测试用例设计"}
|
||||
- {id: testcase, kind: testcase, domain: code, deps: [design], desc: "编写测试用例"}
|
||||
- {id: verify, kind: verify, domain: code, deps: [testcase], desc: "覆盖率与结果核对"}
|
||||
|
||||
# ---- 数学 ----
|
||||
math-solve:
|
||||
steps:
|
||||
- {id: conditions, kind: analyze, domain: math, desc: "明确已知条件与目标"}
|
||||
- {id: solve, kind: solve, domain: math, deps: [conditions], desc: "选择方法并求解"}
|
||||
- {id: verify, kind: verify, domain: math, deps: [solve], desc: "检查边界与验证"}
|
||||
|
||||
math-proof:
|
||||
steps:
|
||||
- {id: conditions, kind: analyze, domain: math, desc: "明确命题与已知条件"}
|
||||
- {id: solve, kind: solve, domain: math, deps: [conditions], desc: "选择证明方法（直接/反证/归纳）"}
|
||||
- {id: verify, kind: verify, domain: math, deps: [solve], desc: "检查证明严密性"}
|
||||
|
||||
math-optimize:
|
||||
steps:
|
||||
- {id: conditions, kind: analyze, domain: math, desc: "目标函数与约束条件"}
|
||||
- {id: solve, kind: optimize, domain: math, deps: [conditions], desc: "求极值/最优解"}
|
||||
- {id: verify, kind: verify, domain: math, deps: [solve], desc: "验证最优性与边界"}
|
||||
|
||||
# ---- 法律 ----
|
||||
legal-advice:
|
||||
steps:
|
||||
- {id: facts, kind: analyze, domain: legal, desc: "梳理事实与法律问题"}
|
||||
- {id: retrieve, kind: retrieve, domain: legal, deps: [facts], desc: "检索适用法规"}
|
||||
- {id: conclude, kind: conclude, domain: legal, deps: [retrieve], desc: "给出法律意见"}
|
||||
- {id: disclaimer, kind: disclaimer, domain: legal, deps: [conclude], desc: "免责提示"}
|
||||
|
||||
# ---- 医疗 ----
|
||||
medical-advice:
|
||||
steps:
|
||||
- {id: symptoms, kind: analyze, domain: medical, desc: "梳理症状与背景"}
|
||||
- {id: advise, kind: advise, domain: medical, deps: [symptoms], desc: "给出一般建议"}
|
||||
- {id: warning, kind: disclaimer, domain: medical, deps: [advise], desc: "就医警示"}
|
||||
|
||||
medical-firstaid:
|
||||
steps:
|
||||
- {id: symptoms, kind: analyze, domain: medical, desc: "判断伤情与风险等级"}
|
||||
- {id: advise, kind: advise, domain: medical, deps: [symptoms], desc: "给出急救处理步骤"}
|
||||
- {id: warning, kind: disclaimer, domain: medical, deps: [advise], desc: "危险信号与就医提示"}
|
||||
|
||||
# ---- 通用 ----
|
||||
general-explain:
|
||||
steps:
|
||||
- {id: outline, kind: analyze, domain: general, desc: "梳理主题要点"}
|
||||
- {id: explain, kind: explain, domain: general, deps: [outline], desc: "展开解释"}
|
||||
- {id: conclude, kind: conclude, domain: general, deps: [explain], desc: "总结"}
|
||||
|
||||
general-writing:
|
||||
steps:
|
||||
- {id: outline, kind: analyze, domain: general, desc: "明确文体、受众与要点"}
|
||||
- {id: draft, kind: draft, domain: general, deps: [outline], desc: "起草正文"}
|
||||
- {id: polish, kind: polish, domain: general, deps: [draft], desc: "润色与检查"}
|
||||
|
||||
# ---- 金融 ----
|
||||
finance-advice:
|
||||
steps:
|
||||
- {id: facts, kind: analyze, domain: finance, desc: "梳理财务需求与背景"}
|
||||
- {id: retrieve, kind: retrieve, domain: finance, deps: [facts], desc: "检索金融知识条目"}
|
||||
- {id: conclude, kind: conclude, domain: finance, deps: [retrieve], desc: "给出建议与风险提示"}
|
||||
- {id: disclaimer, kind: disclaimer, domain: finance, deps: [conclude], desc: "风险免责提示"}
|
||||
|
||||
# ---- 生活 ----
|
||||
life-guide:
|
||||
steps:
|
||||
- {id: context, kind: analyze, domain: life, desc: "梳理需求与场景"}
|
||||
- {id: advise, kind: advise, domain: life, deps: [context], desc: "给出实用建议"}
|
||||
- {id: conclude, kind: conclude, domain: life, deps: [advise], desc: "总结要点"}
|
||||
|
||||
# ---- 教育 ----
|
||||
edu-guide:
|
||||
steps:
|
||||
- {id: context, kind: analyze, domain: education, desc: "明确目标与现状"}
|
||||
- {id: plan, kind: design, domain: education, deps: [context], desc: "制定学习/行动方案"}
|
||||
- {id: conclude, kind: conclude, domain: education, deps: [plan], desc: "总结与下一步"}
|
||||
Binary file not shown.
+1
-1
@@ -180,7 +180,7 @@ try:
|
||||
_STATIC_DIR = Path(__file__).resolve().parent / "static"
|
||||
|
||||
app = FastAPI(
|
||||
title="端云协同 LLM 协作系统",
|
||||
title="端云协同编程智能体系统",
|
||||
description="大模型(Architect) + 本地小模型(Worker) 通过交流文本协作；v1 保留为 legacy 路由",
|
||||
version="2.0.0",
|
||||
)
|
||||
|
||||
+1
-1
File diff suppressed because one or more lines are too long
+1
-1
@@ -1 +1 @@
|
||||
import{A as e,D as t,G as n,I as r,L as i,N as a,O as o,P as s,V as c,W as l,j as u,k as d,s as f,t as p}from"./index-kbuKhaUa.js";var m={class:`metrics-view`},h={key:0,class:`loading`},g={key:1,class:`error`},_={class:`card-grid`},v={class:`metric-card`},y={class:`kv-list`},b={class:`metric-card`},x={class:`kv-list`},S={key:0,class:`metric-card highlight`},C={class:`kv-list`},w={key:0},T={key:1},E={key:1,class:`metric-card`},D={class:`by-model`},O={class:`mono`},k={key:2,class:`metric-card review-card`},A={class:`review-stats`},j={class:`stat-item`},M={class:`stat-num`},N={class:`stat-item`},P={class:`stat-num`},F={key:0,class:`progress-wrap`},I={class:`review-rate`},L={class:`raw-json`},R=p(a({__name:`MetricsView`,setup(a){let p=c(null),R=c(!1),z=c(``),B=o(()=>p.value?.v2?.by_model||null);async function V(){R.value=!0,z.value=``;try{p.value=await f()}catch(e){z.value=e instanceof Error?e.message:`指标加载失败，请检查后端服务`}finally{R.value=!1}}return s(V),(a,o)=>(r(),u(`div`,m,[d(`header`,{class:`metrics-header`},[o[0]||=d(`h2`,null,`系统指标`,-1),d(`button`,{class:`refresh`,onClick:V},`🔄 刷新`)]),R.value?(r(),u(`div`,h,`加载中…`)):z.value?(r(),u(`div`,g,n(z.value),1)):p.value?(r(),u(t,{key:2},[d(`div`,_,[d(`div`,v,[o[1]||=d(`h3`,null,`路由器（v1）`,-1),d(`div`,y,[(r(!0),u(t,null,i(p.value.router,(e,i)=>(r(),u(t,{key:i},[d(`span`,null,n(i),1),d(`b`,null,n(e),1)],64))),128))])]),d(`div`,b,[o[2]||=d(`h3`,null,`缓存`,-1),d(`div`,x,[(r(!0),u(t,null,i(p.value.cache,(e,i)=>(r(),u(t,{key:i},[d(`span`,null,n(i),1),d(`b`,null,n(e),1)],64))),128))])]),p.value.v2?(r(),u(`div`,S,[o[3]||=d(`h3`,null,`协作管线（v2）`,-1),d(`div`,C,[(r(!0),u(t,null,i(p.value.v2,(i,a)=>(r(),u(t,{key:a},[a===`by_model`?e(``,!0):(r(),u(`span`,w,n(a),1)),a===`by_model`?e(``,!0):(r(),u(`b`,T,n(i),1))],64))),128))])])):e(``,!0),B.value&&Object.keys(B.value).length?(r(),u(`div`,E,[o[5]||=d(`h3`,null,`按模型分账（token / 成本）`,-1),d(`table`,D,[o[4]||=d(`thead`,null,[d(`tr`,null,[d(`th`,null,`模型`),d(`th`,null,`次数`),d(`th`,null,`入`),d(`th`,null,`出`),d(`th`,null,`成本 $`)])],-1),d(`tbody`,null,[(r(!0),u(t,null,i(B.value,(e,t)=>(r(),u(`tr`,{key:t},[d(`td`,O,n(t),1),d(`td`,null,n(e.requests),1),d(`td`,null,n(e.input_tokens),1),d(`td`,null,n(e.output_tokens),1),d(`td`,null,n(e.cost_est_usd),1)]))),128))])]),o[6]||=d(`p`,{class:`hint`},`单价来自模型池条目（$/1M tokens）；经典设置下的模型成本不计入。`,-1)])):e(``,!0),p.value.review?(r(),u(`div`,k,[o[9]||=d(`h3`,null,`人工检验`,-1),d(`div`,A,[d(`div`,j,[d(`span`,M,n(p.value.review.pending),1),o[7]||=d(`span`,{class:`stat-label`},`待审核`,-1)]),d(`div`,N,[d(`span`,P,n(p.value.review.total),1),o[8]||=d(`span`,{class:`stat-label`},`总提交`,-1)])]),p.value.review.total>0?(r(),u(`div`,F,[d(`div`,{class:`reviewed-bar`,style:l({width:`${(p.value.review.total-p.value.review.pending)/p.value.review.total*100}%`})},null,4)])):e(``,!0),d(`p`,I,` 通过率： `+n(((p.value.review.total-p.value.review.pending)/p.value.review.total*100).toFixed(1))+`% `,1)])):e(``,!0)]),d(`details`,L,[o[10]||=d(`summary`,null,`原始 JSON`,-1),d(`pre`,null,n(JSON.stringify(p.value,null,2)),1)])],64)):e(``,!0)]))}}),[[`__scopeId`,`data-v-8b237097`]]);export{R as default};
|
||||
import{A as e,D as t,G as n,I as r,L as i,N as a,O as o,P as s,V as c,W as l,j as u,k as d,s as f,t as p}from"./index-BvHonIym.js";var m={class:`metrics-view`},h={key:0,class:`loading`},g={key:1,class:`error`},_={class:`card-grid`},v={class:`metric-card`},y={class:`kv-list`},b={class:`metric-card`},x={class:`kv-list`},S={key:0,class:`metric-card highlight`},C={class:`kv-list`},w={key:0},T={key:1},E={key:1,class:`metric-card`},D={class:`by-model`},O={class:`mono`},k={key:2,class:`metric-card review-card`},A={class:`review-stats`},j={class:`stat-item`},M={class:`stat-num`},N={class:`stat-item`},P={class:`stat-num`},F={key:0,class:`progress-wrap`},I={class:`review-rate`},L={class:`raw-json`},R=p(a({__name:`MetricsView`,setup(a){let p=c(null),R=c(!1),z=c(``),B=o(()=>p.value?.v2?.by_model||null);async function V(){R.value=!0,z.value=``;try{p.value=await f()}catch(e){z.value=e instanceof Error?e.message:`指标加载失败，请检查后端服务`}finally{R.value=!1}}return s(V),(a,o)=>(r(),u(`div`,m,[d(`header`,{class:`metrics-header`},[o[0]||=d(`h2`,null,`系统指标`,-1),d(`button`,{class:`refresh`,onClick:V},`🔄 刷新`)]),R.value?(r(),u(`div`,h,`加载中…`)):z.value?(r(),u(`div`,g,n(z.value),1)):p.value?(r(),u(t,{key:2},[d(`div`,_,[d(`div`,v,[o[1]||=d(`h3`,null,`路由器（v1）`,-1),d(`div`,y,[(r(!0),u(t,null,i(p.value.router,(e,i)=>(r(),u(t,{key:i},[d(`span`,null,n(i),1),d(`b`,null,n(e),1)],64))),128))])]),d(`div`,b,[o[2]||=d(`h3`,null,`缓存`,-1),d(`div`,x,[(r(!0),u(t,null,i(p.value.cache,(e,i)=>(r(),u(t,{key:i},[d(`span`,null,n(i),1),d(`b`,null,n(e),1)],64))),128))])]),p.value.v2?(r(),u(`div`,S,[o[3]||=d(`h3`,null,`协作管线（v2）`,-1),d(`div`,C,[(r(!0),u(t,null,i(p.value.v2,(i,a)=>(r(),u(t,{key:a},[a===`by_model`?e(``,!0):(r(),u(`span`,w,n(a),1)),a===`by_model`?e(``,!0):(r(),u(`b`,T,n(i),1))],64))),128))])])):e(``,!0),B.value&&Object.keys(B.value).length?(r(),u(`div`,E,[o[5]||=d(`h3`,null,`按模型分账（token / 成本）`,-1),d(`table`,D,[o[4]||=d(`thead`,null,[d(`tr`,null,[d(`th`,null,`模型`),d(`th`,null,`次数`),d(`th`,null,`入`),d(`th`,null,`出`),d(`th`,null,`成本 $`)])],-1),d(`tbody`,null,[(r(!0),u(t,null,i(B.value,(e,t)=>(r(),u(`tr`,{key:t},[d(`td`,O,n(t),1),d(`td`,null,n(e.requests),1),d(`td`,null,n(e.input_tokens),1),d(`td`,null,n(e.output_tokens),1),d(`td`,null,n(e.cost_est_usd),1)]))),128))])]),o[6]||=d(`p`,{class:`hint`},`单价来自模型池条目（$/1M tokens）；经典设置下的模型成本不计入。`,-1)])):e(``,!0),p.value.review?(r(),u(`div`,k,[o[9]||=d(`h3`,null,`人工检验`,-1),d(`div`,A,[d(`div`,j,[d(`span`,M,n(p.value.review.pending),1),o[7]||=d(`span`,{class:`stat-label`},`待审核`,-1)]),d(`div`,N,[d(`span`,P,n(p.value.review.total),1),o[8]||=d(`span`,{class:`stat-label`},`总提交`,-1)])]),p.value.review.total>0?(r(),u(`div`,F,[d(`div`,{class:`reviewed-bar`,style:l({width:`${(p.value.review.total-p.value.review.pending)/p.value.review.total*100}%`})},null,4)])):e(``,!0),d(`p`,I,` 通过率： `+n(((p.value.review.total-p.value.review.pending)/p.value.review.total*100).toFixed(1))+`% `,1)])):e(``,!0)]),d(`details`,L,[o[10]||=d(`summary`,null,`原始 JSON`,-1),d(`pre`,null,n(JSON.stringify(p.value,null,2)),1)])],64)):e(``,!0)]))}}),[[`__scopeId`,`data-v-8b237097`]]);export{R as default};
|
||||
+1
-1
@@ -1 +1 @@
|
||||
import{A as e,D as t,E as n,G as r,I as i,L as a,M as o,N as s,O as c,P as l,U as u,V as d,f,j as p,k as m,t as h,v as g,z as _}from"./index-kbuKhaUa.js";var v={class:`review-view`},y={class:`review-header`},b={class:`controls`},x={key:0,class:`loading`},S={key:1,class:`error`},C={key:2,class:`queue-list`},w={key:0,class:`empty`},T={class:`card-header`},E={class:`card-id`},D={class:`tags`},O={class:`date`},k={class:`query-block`},A={class:`response-block`},j={key:0,class:`actions`},M=[`onUpdate:modelValue`],N={class:`btn-row`},P=[`onClick`],F=[`onClick`],I={key:1,class:`correction`},L=h(s({__name:`ReviewView`,setup(s){let h=d([]),L=d(!1),R=d(``),z=d(`pending`),B=d({}),V=c(()=>z.value===`all`?h.value:h.value.filter(e=>e.verdict===z.value));async function H(){L.value=!0,R.value=``;try{h.value=await f()}catch(e){R.value=e instanceof Error?e.message:String(e)}finally{L.value=!1}}async function U(e,t){try{await g(e,t,B.value[e]||void 0),await H()}catch(e){R.value=e instanceof Error?e.message:String(e)}}return l(H),(s,c)=>(i(),p(`div`,v,[m(`header`,y,[c[4]||=m(`h2`,null,`人工检验队列`,-1),m(`div`,b,[m(`button`,{class:u({active:z.value===`all`}),onClick:c[0]||=e=>z.value=`all`},`全部`,2),m(`button`,{class:u({active:z.value===`pending`}),onClick:c[1]||=e=>z.value=`pending`},`待审核`,2),m(`button`,{class:u({active:z.value===`approved`}),onClick:c[2]||=e=>z.value=`approved`},`已通过`,2),m(`button`,{class:u({active:z.value===`rejected`}),onClick:c[3]||=e=>z.value=`rejected`},`已拒绝`,2),m(`button`,{class:`refresh-btn`,onClick:H},`🔄 刷新`)])]),L.value?(i(),p(`div`,x,`加载中…`)):R.value?(i(),p(`div`,S,r(R.value),1)):(i(),p(`div`,C,[V.value.length?e(``,!0):(i(),p(`div`,w,`队列为空。`)),(i(!0),p(t,null,a(V.value,s=>(i(),p(`div`,{key:s.id,class:`review-card`},[m(`div`,T,[m(`span`,E,`#`+r(s.id),1),m(`span`,{class:u([`verdict-badge`,s.verdict])},r(s.verdict),3),m(`span`,D,[(i(!0),p(t,null,a(s.tags,e=>(i(),p(`span`,{key:e,class:`tag`},r(e),1))),128))]),m(`span`,O,r(s.created_at),1)]),m(`div`,k,[c[5]||=m(`strong`,null,`Query：`,-1),o(r(s.query),1)]),m(`div`,A,[c[6]||=m(`strong`,null,`Response：`,-1),m(`pre`,null,r(s.response),1)]),s.verdict===`pending`?(i(),p(`div`,j,[_(m(`textarea`,{"onUpdate:modelValue":e=>B.value[s.id]=e,placeholder:`修正意见（可选）`,rows:`2`},null,8,M),[[n,B.value[s.id]]]),m(`div`,N,[m(`button`,{class:`approve`,onClick:e=>U(s.id,`approved`)},`✅ 通过`,8,P),m(`button`,{class:`reject`,onClick:e=>U(s.id,`rejected`)},`❌ 拒绝`,8,F)])])):s.correction?(i(),p(`div`,I,[c[7]||=m(`strong`,null,`修正：`,-1),o(r(s.correction),1)])):e(``,!0)]))),128))]))]))}}),[[`__scopeId`,`data-v-d5b38f1c`]]);export{L as default};
|
||||
import{A as e,D as t,E as n,G as r,I as i,L as a,M as o,N as s,O as c,P as l,U as u,V as d,f,j as p,k as m,t as h,v as g,z as _}from"./index-BvHonIym.js";var v={class:`review-view`},y={class:`review-header`},b={class:`controls`},x={key:0,class:`loading`},S={key:1,class:`error`},C={key:2,class:`queue-list`},w={key:0,class:`empty`},T={class:`card-header`},E={class:`card-id`},D={class:`tags`},O={class:`date`},k={class:`query-block`},A={class:`response-block`},j={key:0,class:`actions`},M=[`onUpdate:modelValue`],N={class:`btn-row`},P=[`onClick`],F=[`onClick`],I={key:1,class:`correction`},L=h(s({__name:`ReviewView`,setup(s){let h=d([]),L=d(!1),R=d(``),z=d(`pending`),B=d({}),V=c(()=>z.value===`all`?h.value:h.value.filter(e=>e.verdict===z.value));async function H(){L.value=!0,R.value=``;try{h.value=await f()}catch(e){R.value=e instanceof Error?e.message:String(e)}finally{L.value=!1}}async function U(e,t){try{await g(e,t,B.value[e]||void 0),await H()}catch(e){R.value=e instanceof Error?e.message:String(e)}}return l(H),(s,c)=>(i(),p(`div`,v,[m(`header`,y,[c[4]||=m(`h2`,null,`人工检验队列`,-1),m(`div`,b,[m(`button`,{class:u({active:z.value===`all`}),onClick:c[0]||=e=>z.value=`all`},`全部`,2),m(`button`,{class:u({active:z.value===`pending`}),onClick:c[1]||=e=>z.value=`pending`},`待审核`,2),m(`button`,{class:u({active:z.value===`approved`}),onClick:c[2]||=e=>z.value=`approved`},`已通过`,2),m(`button`,{class:u({active:z.value===`rejected`}),onClick:c[3]||=e=>z.value=`rejected`},`已拒绝`,2),m(`button`,{class:`refresh-btn`,onClick:H},`🔄 刷新`)])]),L.value?(i(),p(`div`,x,`加载中…`)):R.value?(i(),p(`div`,S,r(R.value),1)):(i(),p(`div`,C,[V.value.length?e(``,!0):(i(),p(`div`,w,`队列为空。`)),(i(!0),p(t,null,a(V.value,s=>(i(),p(`div`,{key:s.id,class:`review-card`},[m(`div`,T,[m(`span`,E,`#`+r(s.id),1),m(`span`,{class:u([`verdict-badge`,s.verdict])},r(s.verdict),3),m(`span`,D,[(i(!0),p(t,null,a(s.tags,e=>(i(),p(`span`,{key:e,class:`tag`},r(e),1))),128))]),m(`span`,O,r(s.created_at),1)]),m(`div`,k,[c[5]||=m(`strong`,null,`Query：`,-1),o(r(s.query),1)]),m(`div`,A,[c[6]||=m(`strong`,null,`Response：`,-1),m(`pre`,null,r(s.response),1)]),s.verdict===`pending`?(i(),p(`div`,j,[_(m(`textarea`,{"onUpdate:modelValue":e=>B.value[s.id]=e,placeholder:`修正意见（可选）`,rows:`2`},null,8,M),[[n,B.value[s.id]]]),m(`div`,N,[m(`button`,{class:`approve`,onClick:e=>U(s.id,`approved`)},`✅ 通过`,8,P),m(`button`,{class:`reject`,onClick:e=>U(s.id,`rejected`)},`❌ 拒绝`,8,F)])])):s.correction?(i(),p(`div`,I,[c[7]||=m(`strong`,null,`修正：`,-1),o(r(s.correction),1)])):e(``,!0)]))),128))]))]))}}),[[`__scopeId`,`data-v-d5b38f1c`]]);export{L as default};
|
||||
+1
-1
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -4,8 +4,8 @@
|
||||
<meta charset="UTF-8" />
|
||||
<link rel="icon" type="image/svg+xml" href="/static/favicon.svg" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<title>端云协同 LLM 协作系统</title>
|
||||
<script type="module" crossorigin src="/static/assets/index-kbuKhaUa.js"></script>
|
||||
<title>端云协同编程智能体系统</title>
|
||||
<script type="module" crossorigin src="/static/assets/index-BvHonIym.js"></script>
|
||||
<link rel="stylesheet" crossorigin href="/static/assets/index-BYO22xUl.css">
|
||||
</head>
|
||||
<body>
|
||||
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
File diff suppressed because one or more lines are too long
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1 @@
|
||||
<script src="/_guard/html.js?js=easy_slider_html"></script>
|
||||
Binary file not shown.
@@ -0,0 +1,60 @@
|
||||
# 毕业论文参考文献清单（基于端云协同的编程智能体系统设计与实现）
|
||||
|
||||
> 用途：毕业设计《基于端云协同的编程智能体系统设计与实现》参考文献初稿（2026-09-04 定稿改题，原题《基于端云协同的 LLM 服务成本优化系统设计与实现》）。改题不改变文献主题，清单继续适用。
|
||||
> 说明：分三类 —— ① 已下载 PDF（外文 arXiv / 中文开放获取）；② 知网可查但需付费/登录下载（学校图书馆知网可下）；③ 使用提醒。
|
||||
> 定稿前**务必在知网逐条核对**卷期、页码、作者（带 ⚠ 的字段请补全）。
|
||||
|
||||
---
|
||||
|
||||
## 一、已下载 PDF（直接可用，仓库内）
|
||||
|
||||
### 1.1 外文（arXiv 开放获取）→ `references/arxiv_papers/`
|
||||
|
||||
> 前 13 篇为仓库原有；14–18 为本次新增。
|
||||
|
||||
| # | 标题 | arXiv | 会议/期刊 | 相关性 |
|
||||
|---|------|-------|----------|--------|
|
||||
| 14 | FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance | [2305.05176](https://arxiv.org/abs/2305.05176) | arXiv 2023 | ⭐级联降本鼻祖，直接支撑"成本优化" |
|
||||
| 15 | What is the Role of Small Models in the LLM Era: A Survey | [2409.06857](https://arxiv.org/abs/2409.06857) | arXiv 2024 | 小模型价值综述，支撑"端侧小模型"前提 |
|
||||
| 16 | Dynamic Model Routing and Cascading for Efficient LLM Inference: A Survey | [2603.04445](https://arxiv.org/abs/2603.04445) | arXiv 2026 | 路由/级联最新综述，相关工作核心 |
|
||||
| 17 | Collaborative Inference and Learning between Edge SLMs and Cloud LLMs: A Survey | [2507.16731](https://arxiv.org/abs/2507.16731) | ACM Computing Surveys（arXiv 版） | ⭐端侧小模型+云侧大模型协作，与命题几乎同名 |
|
||||
| 18 | A Survey on Collaborating Small and Large Language Models | [2510.13890](https://arxiv.org/abs/2510.13890) | arXiv 2025 | 大小模型协作综述（成本/端云/可信） |
|
||||
|
||||
### 1.2 中文（开放获取 PDF）→ `references/cnki_open/`
|
||||
|
||||
| 标题 | 作者 | 出处 | 文件 | 相关性 |
|
||||
|------|------|------|------|--------|
|
||||
| 大语言模型算法演进综述 | 中兴通讯技术编辑部 | 《中兴通讯技术》 | `中_大语言模型算法演进综述_中兴通讯技术.pdf` | 背景/综述 |
|
||||
| 大小模型端云协同进化技术进展 | 王永威、沈弢、张圣宇、吴帆、赵洲、蔡海滨、吕承飞、马利庄、杨承磊、吴飞（浙大/上交/华师大/山大等） | 《中国图象图形学报》2024, 29(6) ⚠页码待补 | PDF 被反爬拦截，仅存题录+摘要（见下） | ⭐端云大小模型协同，最对口 |
|
||||
|
||||
> 注：第 2 篇期刊官网 `create_pdf.aspx` 有反爬 JS 校验，脚本无法直接下 PDF；但其全文 HTML 与摘要可在
|
||||
> `http://txtx.cjig.cn/jig/article/html/240011`（JS 渲染，需浏览器查看）获取，DOI: `10.11834/jig.240011`。
|
||||
> 若需 PDF，请在知网或该刊官网用浏览器手动下载。
|
||||
|
||||
---
|
||||
|
||||
## 二、知网可查、需付费/登录下载的中文文献
|
||||
|
||||
> 这些在知网（https://www.cnki.net）用**题名**检索即可找到，PDF 需学校图书馆知网账号或付费下载。
|
||||
|
||||
| 标题 | 出处 | 知网定位码（已知字段） | 类型 |
|
||||
|------|------|------------------------|------|
|
||||
| 大语言模型时代的人工智能：技术内涵、行业应用与挑战（陈光、郭军，北京邮电大学人工智能学院） | 《北京邮电大学学报》2024年04期 ⚠页码 | `BJYD202404003` | 期刊 |
|
||||
| 面向小规模大语言模型推理优化的推理路径排序方法 | 《中文信息学报》2026年01期 ⚠页码 | `MESS202601002` | 期刊 |
|
||||
| Token级多模型并联协作推理 | 《计算机学报》2025年11期 ⚠页码/作者 | `JSJX202511003` | 期刊（顶刊） |
|
||||
| 面向边缘智能的大模型研究进展 | 期刊（被引1）⚠出处待核 | — | 期刊 |
|
||||
| 融合小型专家模型的大语言模型推理增强研究 | 硕士学位论文 ⚠作者/学校/年份 | CMFD 学位库 | 学位论文 |
|
||||
| 面向时变环境的大小模型协同决策方法研究 | 《首届全国大模型与决策智能大会论文集》(2024) ⚠ 链接已失效，请知网检索题名 | — | 会议 |
|
||||
|
||||
---
|
||||
|
||||
## 三、使用提醒（重要）
|
||||
|
||||
1. **下载到库后逐条核对**：上表中所有 ⚠ 字段（页码/作者/年份/会议名）都需要你在知网打开原文献后补齐，**不要直接照抄本清单**。
|
||||
2. **外文 14–18 可放心用**：均已下载为有效 PDF（已验证 `%PDF` 文件头），arXiv ID 真实。
|
||||
3. **中文文献口径**：本科毕设外审优先认**期刊 + 学位论文**；会议论文集一般可用但需先问导师。上面 6 篇里建议优先补齐《计算机学报》《中文信息学报》《北京邮电大学学报》和那篇学位论文。
|
||||
4. **知网使用途径**：用你学校图书馆 → 数据库 → 知网入口（校外需 VPN/机构账号）检索并下载 PDF，通常免费。
|
||||
|
||||
---
|
||||
|
||||
*整理时间：见 git 记录。整理人：DSH Agent。*
|
||||
@@ -103,4 +103,28 @@
|
||||
|
||||
---
|
||||
|
||||
## 五、本调研衍生的项目产出
|
||||
|
||||
依据本调研的第 1 条建议（用 RouterArena 标准化评测），项目已落地以下产出：
|
||||
|
||||
| 产出 | 路径 | 说明 |
|
||||
|------|------|------|
|
||||
| RouterArena 接入方案 | `research/routerarena/00_integration_plan.md` | 5 槽位候选模型 + 8→9 域映射 + 4 阶段验证 |
|
||||
| 接入实现（ESExpertRouter） | `research/routerarena/adapter.py` | BaseRouter 子类，~130 行 |
|
||||
| Vendored BaseRouter | `research/routerarena/base_router.py` | SPDX 标注，Apache-2.0 |
|
||||
| 本地端到端 runner | `research/routerarena/local_runner.py` | mock 子集上 90 条跑通 |
|
||||
| 实验结果 + 差距分析 | `research/routerarena/01_results_and_gap_analysis.md` | 含 27-router 排行榜对比表 |
|
||||
| 8 个单测（全过） | `tests/test_routerarena_adapter.py` | 接口/映射/公式/Schema 全覆盖 |
|
||||
| **论文主体** | `research/paper/main.md` | ~9 千字 10 节 + 2 附录 Markdown |
|
||||
| 论文目录索引 | `research/paper/README.md` | 投稿 venue / 转化命令 / 待补事项 |
|
||||
|
||||
**核心实证发现**（详见论文 §5）：
|
||||
|
||||
- 规则分类器在 9 域平衡 mock 子集上 **74.4 %** 准确率（vs 24 样例 100 %）
|
||||
- 68.9 % query 触发升级到 mistral-medium，**cost_per_1k = $0.51**（vs 目标 Hybrid Router $0.04，12× 差距）
|
||||
- 英文/混合 query 覆盖率低是主导成本杠杆
|
||||
- 最高 ROI 下一步：替换 L0 规则分类器为 L1 训练分类器（BERT 级）
|
||||
|
||||
---
|
||||
|
||||
*调研人：Codex（2026-08-12）*
|
||||
|
||||
Binary file not shown.
@@ -0,0 +1,76 @@
|
||||
# 论文目录（research/paper/）
|
||||
|
||||
本目录是项目"基于规则的多专家路由系统"科研成果的论文化整理。
|
||||
|
||||
## 📄 主文件
|
||||
|
||||
| 文件 | 内容 | 字数 |
|
||||
|------|------|------|
|
||||
| [`main.md`](./main.md) | **完整论文正文**（Markdown，便于 pandoc 转 PDF / 投稿系统转换） | ~37KB / ~9 千字 |
|
||||
|
||||
## 🎯 论文定位
|
||||
|
||||
- **题目**：Open Evaluation of a Rule-Based Multi-Expert Router on the RouterArena Benchmark: Methods, Empirical Findings, and Open Challenges
|
||||
- **类型**：经验型系统论文（empirical systems paper），不是新算法论文
|
||||
- **目标 venue**：NeurIPS / ICML Datasets & Benchmarks track、ACL/EMNLP Industry track、KDD 评测 track、arXiv 预印本
|
||||
- **核心卖点**：把"自建 24 样例 100%"的脆弱基线换成 RouterArena 协议化评测 + **诚实报 gap**
|
||||
|
||||
## 🧱 论文结构（10 节 + 2 附录）
|
||||
|
||||
1. **Abstract** — 一段式，含 4 个具体数字（74.4 %, 68.9 %, $0.51/1K, 126/126 tests）
|
||||
2. **Introduction** — 动机 + 3 条贡献
|
||||
3. **Related Work** — 27 篇 + 13 篇 references/ 论文分类
|
||||
4. **System Design** — L0 五段式内核（KB / Classifier / Planner / Executor / Judge）
|
||||
5. **RouterArena Adapter** — 接口 / 选型 / 域映射 / 合规 / 单测
|
||||
6. **Empirical Evaluation** — 90 条 mock 协议对齐 + 3 张表 + 3 条发现
|
||||
7. **Discussion** — 3 课 + L1/L2 路线图
|
||||
8. **Limitations** — 6 条已知 gap（不掩盖）
|
||||
9. **Conclusion and Future Work** — 5 优先级
|
||||
10. **References** — 36 篇 + Acknowledgements
|
||||
|
||||
+ Appendix A：复现命令
|
||||
+ Appendix B：8→9 域映射表
|
||||
+ Appendix C：3 个核心发现压缩版
|
||||
|
||||
## 🔗 论文引用到的本项目文件
|
||||
|
||||
- `research/routerarena/adapter.py` — ESExpertRouter 实现
|
||||
- `research/routerarena/base_router.py` — vendored BaseRouter
|
||||
- `research/routerarena/local_runner.py` — 端到端 runner
|
||||
- `research/routerarena/00_integration_plan.md` — 接入方案
|
||||
- `research/routerarena/01_results_and_gap_analysis.md` — 详细数据
|
||||
- `tests/test_routerarena_adapter.py` — 8 个单测
|
||||
- `research/2026_papers_survey.md` — 调研综述（Reference [27]）
|
||||
|
||||
## 🛠️ 转 PDF / 投稿
|
||||
|
||||
论文用 Markdown 写，便于：
|
||||
|
||||
```bash
|
||||
# 转 PDF（需要 pandoc + LaTeX）
|
||||
pandoc main.md -o main.pdf --from markdown+yaml_metadata_block \
|
||||
--template=ieee --variable geometry:margin=1in
|
||||
|
||||
# 转 LaTeX 投稿
|
||||
pandoc main.md -o main.tex --from markdown+citations \
|
||||
--biblatex --citeproc
|
||||
|
||||
# 转 docx 评审
|
||||
pandoc main.md -o main.docx
|
||||
```
|
||||
|
||||
## ⚠️ 投稿前要补的事
|
||||
|
||||
- [ ] 真实 RouterArena sub_10 / full 数据接入（解决网络限制后）
|
||||
- [ ] 真实 LLM 推理（API key + \$1-5 预算）
|
||||
- [ ] Conformal Cascade 阈值校准（论文 §6.1 Lesson 3 提到）
|
||||
- [ ] 补 L1 实验作为对比基线（提升论文说服力）
|
||||
- [ ] 通讯作者邮箱 + 完整作者列表
|
||||
- [ ] 准备 GitHub Pages / OpenReview 链接
|
||||
- [ ] 致谢 + 资金声明（如适用）
|
||||
|
||||
## 📌 不要做的事
|
||||
|
||||
- **不要在没有真实数据的情况下提交到 leaderboard**（合规约束 + 论文失实风险）
|
||||
- **不要在 RouterArena 标签上训练/微调任何模型**（评估专数据规则，README 明文禁止）
|
||||
- **不要省略 Limitations 一节**（8 条 limitation 是论文可信度核心）
|
||||
@@ -0,0 +1,376 @@
|
||||
# Open Evaluation of a Rule-Based Multi-Expert Router on the RouterArena Benchmark: Methods, Empirical Findings, and Open Challenges
|
||||
|
||||
> **Authors:** Multi-Expert Router Working Group¹ (Corresponding: CodeGod, code_god@local)
|
||||
>
|
||||
> **Affiliations:**
|
||||
> ¹ Multi-Expert Router Project (open-source research artifact, this work)
|
||||
>
|
||||
> **Keywords:** LLM routing, expert system, RouterArena, open evaluation, rule-based classifier, cost–accuracy trade-off, cascaded inference
|
||||
>
|
||||
> **arXiv (preprint, to be submitted):** TBD
|
||||
>
|
||||
> **Code & Data:** https://github.com/MiniMax/projectAIpopular (sub-directory `research/routerarena/`)
|
||||
|
||||
---
|
||||
|
||||
## Abstract
|
||||
|
||||
LLM routing—automatically selecting the most cost-effective model from a heterogeneous pool for each incoming query—has emerged as a critical infrastructure primitive for cost-efficient LLM serving. While recent literature has produced dozens of routing algorithms evaluated on synthetic or in-house benchmarks, open standardized evaluation remains scarce. In this work, we report on the first (to our knowledge) integration of a **rule-based multi-expert router**—built on a classical expert-system kernel (knowledge base + task planner + rule executor + quality judge + last-resort fallback)—with the public RouterArena benchmark (ICLR 2026). We (i) describe the system's architecture, (ii) document a vendored, fully reproducible adapter that implements RouterArena's `BaseRouter` interface, (iii) report empirical findings on a 90-query mock subset constructed to mirror RouterArena's protocol, and (iv) candidly report the gaps that prevented us from running the full 8 400-query evaluation. Our principal empirical finding is that a pure rule-based classifier reaches **74.4 % domain-classification accuracy** on a balanced 9-domain query mix (down from 100 % on a self-curated 24-sample Chinese benchmark), which in turn triggers a **68.9 % escalation rate** to the most expensive candidate model, yielding a cost of **\$0.51 per 1 000 queries**—a 12× penalty relative to the public Hybrid Router baseline. We discuss the implications for the project roadmap and argue that the next research step—**replacing the rule classifier with a trained small-language-model classifier (L1)**—should be the primary lever to close this gap, while the L0 design remains a useful interpretable baseline. All code, configuration, mock dataset, and 8 unit tests (all passing, 126/126 total project tests) are released.
|
||||
|
||||
---
|
||||
|
||||
## 1. Introduction
|
||||
|
||||
The economic case for LLM routing is now well established. Recent surveys report that the arena score of low-cost routers on standardized benchmarks can already exceed that of GPT-5, with a cost advantage of two orders of magnitude [Lu et al., 2025; Hu et al., 2025]. Yet, the lion's share of published routers in the literature are evaluated on private or in-house testbeds, making it hard to reason about cross-paper comparisons. The RouterArena benchmark [Lu et al., 2025] has been proposed precisely to address this gap: a public 8 400-query dataset spanning 9 domains and 44 categories, accompanied by a five-dimensional evaluation protocol (accuracy, cost, optimality, robustness, latency) and a leaderboard open to public submissions.
|
||||
|
||||
The authors of this paper have, over the past year, been building a multi-expert router based on a deliberately non-neural architecture: an **expert-system kernel** that uses a hand-written rule knowledge base, a symbolic task planner, deterministic rule executors, a rule-based quality judge, and a fallback last-resort executor. The system is entirely parameter-free in its default L0 mode: every output is a deterministic template expansion of knowledge-base facts. The system is publicly available at the project URL above, with 118 unit tests passing prior to this work and 126 after.
|
||||
|
||||
**Why RouterArena, and why now?** Two considerations converge. First, the survey of the project's own research notes [Multi-Expert Router Project, 2026] explicitly identifies the *standardized evaluation* gap as priority #1, citing RouterArena as the canonical fix. Second, in the L0 mode, the project has nothing to lose: there are no neural parameters, no training data, and the comparison is fully apples-to-apples against published baselines that *do* use neural routing. This paper is therefore best understood as a *honest empirical case study* rather than an algorithm paper: it documents the system, the integration effort, the empirical results, the gaps, and what the gaps imply for the next research step.
|
||||
|
||||
**Contributions.** This paper makes three contributions:
|
||||
|
||||
1. **System and adapter description** (§3, §4). We describe the L0 expert-system kernel and document a vendored, SPDX-attributed, fully unit-tested adapter (`research/routerarena/`) that implements RouterArena's `BaseRouter` interface and the prediction-file schema.
|
||||
|
||||
2. **Empirical evaluation on a 90-query mock subset** (§5). We report domain-classification accuracy per domain, escalation rate, predicted cost per 1 000 queries, and an Arena Score computed with the official formula; we contrast these with the 27-router public leaderboard (Hybrid Router, R2-Router, GPT-5, etc.) and explicitly disavow any direct comparability given the proxy data.
|
||||
|
||||
3. **Roadmap analysis and open challenges** (§6, §7). We surface three concrete lessons for the project: (a) the rule classifier's English-query coverage is the dominant cost lever, (b) escalation to `mistral-medium` is the cost bottleneck under low confidence, and (c) the L0 design is best understood as a *scientific instrument*—an interpretable baseline—rather than a deployment target. We map the next research step to replacing the rule classifier with a trained small model (L1), and discuss the integration with Conformal Cascade-style threshold calibration [Angelopoulos et al., 2024].
|
||||
|
||||
The remainder of the paper is organized as follows. Section 2 reviews related work. Section 3 describes the L0 system. Section 4 describes the RouterArena adapter. Section 5 reports results. Section 6 discusses findings. Section 7 reports limitations. Section 8 concludes.
|
||||
|
||||
---
|
||||
|
||||
## 2. Related Work
|
||||
|
||||
### 2.1 Surveys and benchmarking of LLM routing
|
||||
|
||||
Doing More with Less [Hu et al., 2025] systematically catalogues low- and high-resource routing strategies and reports a 64.3 % vs 52.2 % gap in favour of query-level routing over domain-level routing on related datasets. A 2026 survey, *Dynamic Model Routing and Cascading for Efficient LLM Inference* [Wang et al., 2026], generalizes the field into six paradigms (difficulty-aware, cascading, budget-constrained, etc.). The RouterArena paper [Lu et al., 2025] provides the open platform; its 27-router leaderboard is the principal external reference for this work.
|
||||
|
||||
### 2.2 Routing algorithms
|
||||
|
||||
A non-exhaustive list of relevant routing methods: *The Avengers* [Yue et al., 2025] is the closest prior to our work—its "multi-model + lightweight scoring" recipe is a direct theoretical inspiration; R2-Router [Jin et al., 2026] treats output-token budget as a decision variable; Meta-Router [Park et al., 2026] uses causal-inference correction; HyDRA [Chen et al., 2026] performs shortfall matching between queries and model profiles; FusionRoute [Li et al., 2026] performs token-level routing with logit correction; Router-R1 [Zhang et al., 2025] formulates routing as RL; MESS+ [Wang et al., 2025] provides cost-optimal routing with SLA guarantees; ICL-Router [Kumar et al., 2026] uses in-context learning; StR [He et al., 2025] matches the project's two-stage "classify → cascade" design; RADAR [Liu et al., 2025] adds difficulty awareness; CARROT [Somerville et al., 2025] formalizes cost-aware rate-optimality; OmniRouter [Tang et al., 2025] supports budget/performance control; SpareLLM [Gao et al., 2025] provides minimal-cost selection under equivalence.
|
||||
|
||||
### 2.3 LoRA expert routing
|
||||
|
||||
Four LoRA-routing works inform the project's planned second-stage multi-domain expert pool: CARE [Huang et al., 2026] allocates budget by uncertainty, VI-MoLE [Zhao et al., 2026] routes by information value, Hard-Routed Mixture of Reasoning LoRAs [Kim et al., 2026] addresses scale mismatch, and ReMix [Lee et al., 2026] enforces fair LoRA participation.
|
||||
|
||||
### 2.4 Cascading, calibration and small-model capacity
|
||||
|
||||
For cascading and quality fallbacks (corresponding to our Judge + fallback layers): *Cluster, Route, Escalate* [Schmidt et al., 2026] reports a 4.7 ms TPOT penalty for selective escalation; *Conformal Cascade* [Bates et al., 2026] provides distribution-free confidence calibration; *The Routing Plateau* [Martin et al., 2026] identifies accuracy ceiling causes. For the small-model capacity premise: *Small Language Models: A Systematic Review* [Dubey et al., 2026] reports 85–90 % accuracy in domain tasks at 10–25 % of LLM cost, and *Specialization Beats Scale* [Industry Report, 2026] documents 3B specialists outperforming GPT-4o and Claude Opus 4.6 in extraction tasks.
|
||||
|
||||
### 2.5 Position of this work
|
||||
|
||||
In contrast to the methods above, this work is **not a new routing algorithm**. We instantiate a deliberately classical expert-system kernel, integrate it with a public benchmark, and report the empirical evidence. Our work is closest in spirit to *The Avengers* (multi-model + lightweight scoring) and to StR (two-stage classify-then-cascade), but is distinguished by the use of a hand-written, parameter-free, symbolic knowledge base as the routing substrate and by a candid, gap-disclosing evaluation protocol.
|
||||
|
||||
---
|
||||
|
||||
## 3. System Design: the L0 Expert-System Kernel
|
||||
|
||||
The system, illustrated in the project README, implements a five-stage pipeline: cache → domain classifier → task planner → rule executor → quality judge → last-resort fallback. Below we describe only the components material to the RouterArena adapter; complete architectural details are in the project repository.
|
||||
|
||||
### 3.1 Knowledge base
|
||||
|
||||
The system ships with a hand-written YAML knowledge base covering **8 domains** (code, math, legal, medical, finance, life, education, general), with **67 rules**, **17 task templates**, and **45 facts**. The knowledge base is consumed at three points: (a) the rule-based classifier uses keyword/pattern rules to map a query to a domain with a confidence score, (b) the task planner uses the matched rules to assemble a DAG of sub-tasks, and (c) the rule executors expand templates with knowledge-base facts to produce the final answer.
|
||||
|
||||
### 3.2 Rule-based classifier
|
||||
|
||||
The classifier is a deterministic function of the query. It uses per-domain keyword sets (Chinese and English) and a confidence function in `[0, 1]` defined as the normalized sum of matched weighted keywords. The classifier emits a `domain` label, a `subdomain` (level-2 fine grain), a `subdomain2` (level-3), a `difficulty` label (`easy|medium|hard`), and a `confidence` value.
|
||||
|
||||
**Empirical accuracy on self-curated benchmark.** On a self-curated 24-sample Chinese benchmark (3 per domain), the classifier achieves 100 % domain accuracy. We will see in §5 that this number does not generalize to a balanced 9-domain query mix.
|
||||
|
||||
### 3.3 Task planner and executors
|
||||
|
||||
The planner expands the matched rules into a DAG of sub-tasks (`analyze → design → implement → verify` for code, `facts → retrieve → conclude → disclaimer` for legal, etc.). The executors fill the templates deterministically with knowledge-base facts. The result is a fully traceable response where every clause is a knowledge-base assertion.
|
||||
|
||||
### 3.4 Quality judge and fallback
|
||||
|
||||
The judge is a 5-dimension rule-based scorer (covering knowledge reference, structure, length, language, and safety). When `quality < 0.70`, the response is escalated; when the classifier emits `confidence < 0.60`, the system short-circuits to the fallback. The fallback in L0 is a mock template expansion; in L1/L2 it can be a local small model or an OpenAI-compatible API.
|
||||
|
||||
### 3.5 Caches and trace
|
||||
|
||||
A two-stage cache (L1 exact, L2 n-gram semantic) deduplicates repeated queries with a reported 40 % hit rate on the self-curated benchmark. Every request is recorded with a unique `request_id` and a full trace, exposed via a FastAPI gateway at `/traces/{request_id}`.
|
||||
|
||||
---
|
||||
|
||||
## 4. RouterArena Adapter
|
||||
|
||||
### 4.1 Protocol
|
||||
|
||||
RouterArena's `BaseRouter` interface is a thin abstract class. The contract is:
|
||||
|
||||
```python
|
||||
class BaseRouter(ABC):
|
||||
def __init__(self, router_name: str, config_path: str = None): ...
|
||||
def get_prediction(self, query: str) -> str: ...
|
||||
@abstractmethod
|
||||
def _get_prediction(self, query: str) -> str: ...
|
||||
```
|
||||
|
||||
`_get_prediction(query)` must return a model name that is present in `config["pipeline_params"]["models"]`. The base class validates this automatically. The full evaluation pipeline—generate predictions → LLM inference → scoring—requires three subsequent scripts (`generate_prediction_file.py`, `llm_inference/run.py`, `llm_evaluation/run.py`) and an API key for each target model.
|
||||
|
||||
### 4.2 Vendoring choice
|
||||
|
||||
GitHub clone of the RouterArena repository timed out in our environment; we therefore *vendored* a minimal `BaseRouter` (with SPDX attribution, Apache-2.0) into `research/routerarena/base_router.py` and adapted `_default_config_path` to look in a project-local directory first. This decision trades a small drift risk (RouterArena's interface may evolve) for full reproducibility and zero new runtime dependencies.
|
||||
|
||||
### 4.3 Domain-to-model mapping
|
||||
|
||||
The candidate model pool is intentionally small (5 models), reflecting the design principle of keeping the router's model pool interpretable:
|
||||
|
||||
| L0 domain | Selected candidate | Rationale |
|
||||
|-----------|-------------------|-----------|
|
||||
| `code` | `gpt-4o-mini` | Strongest small code model; public leaderboard benchmarked. |
|
||||
| `math`, `legal`, `medical`, `finance` | `claude-3-haiku-20240307` | Reasoning + structured output; well-calibrated on legal/medical Q&A. |
|
||||
| `life`, `general` | `gemini-2.0-flash-001` | Lowest cost, sufficient for utility-class queries. |
|
||||
| `education` | `deepseek-chat` | Strong Chinese-language understanding. |
|
||||
| `confidence < 0.60` or `quality < 0.70` | `mistral-medium` (escalation) | Most expensive candidate, used as the last-resort slot. |
|
||||
|
||||
The thresholds (`0.60` for confidence, `0.70` for quality) are inherited from the project's default `config.yaml`. They are *not* tuned for RouterArena.
|
||||
|
||||
### 4.4 Adapter implementation
|
||||
|
||||
The adapter is `ESExpertRouter` (≈ 130 lines). It wraps the project's `build_router()` and implements the synchronous `_get_prediction` by dispatching to the underlying `async` router. Diagnostic information (full routing trace) is exposed via a separate `diagnostics(query)` method and is *not* part of the RouterArena protocol, preserving the eval-only-data constraint [Lu et al., 2025, §"Evaluating Your Router"].
|
||||
|
||||
### 4.5 Local runner and unit tests
|
||||
|
||||
A local runner (`local_runner.py`, ≈ 360 lines) wraps the adapter in a self-contained evaluation harness: it accepts a mock or real dataset, generates a RouterArena-compliant prediction JSON, runs a mock inference (using the L0 router's own response as a proxy for the target LLM's output), and computes the official Arena Score. Eight unit tests (all passing; 126/126 total in the project) cover: BaseRouter loading, prediction-in-models invariant, domain-to-slot mapping evidence, escalation logic, Arena-Score formula correctness (validated against the published Hybrid Router numbers), clamping, mock-dataset protocol, and prediction-file schema.
|
||||
|
||||
### 4.6 Compliance with eval-only constraint
|
||||
|
||||
RouterArena's data is *evaluation-only*: submissions that train, fit, or tune any router component on RouterArena labels will be rejected. Our system is fully compliant: the L0 knowledge base is hand-written; the L1 training plan uses internal Chinese data; the L2 expert fine-tuning plan uses domain-specific non-RouterArena corpora. The adapter uses only the prompt field of RouterArena entries, never the answer or option fields.
|
||||
|
||||
---
|
||||
|
||||
## 5. Empirical Evaluation
|
||||
|
||||
### 5.1 Setup
|
||||
|
||||
We evaluate on a 90-query mock dataset constructed by the authors. Each of 9 domains (code, math, legal, medical, finance, life, education, general, creative) is represented by 10 queries, half in Chinese and half in English, with one easy, one medium, and one hard difficulty per side. The dataset's schema is a strict subset of RouterArena's: `{global index, prompt, prompt_formatted, domain, difficulty}`. The `domain` field is a *ground-truth label* present only in the mock dataset; it is used solely to score our classifier and is **never** available to the adapter.
|
||||
|
||||
The candidate model prices used in cost calculations are drawn from the public `model_cost.json` snapshot: `gpt-4o-mini` (0.15 / 0.60 per 1M input/output tokens), `claude-3-haiku` (0.25 / 1.25), `gemini-2.0-flash` (0.075 / 0.30), `deepseek-chat` (0.14 / 0.28), `mistral-medium` (2.7 / 8.1).
|
||||
|
||||
We do **not** run real LLM inference in this paper. The `generated_result` field is populated with the L0 router's own template-filled response as a stand-in; this permits a methodologically valid Arena-Score computation (the official formula is verified by unit test against the published Hybrid Router number) but the resulting accuracy is the *domain-classification* accuracy, not the *answer-correctness* accuracy used by the public leaderboard. This distinction is fundamental and we restate it in §7.
|
||||
|
||||
### 5.2 Domain-classification accuracy
|
||||
|
||||
Table 1 reports the per-domain accuracy of the rule-based classifier on the 90-query mock dataset.
|
||||
|
||||
**Table 1.** Per-domain domain-classification accuracy (mock 90-query, 9 domains). "Conf" is the average L0 confidence.
|
||||
|
||||
| Ground-truth domain | n | L0 accuracy | Mean conf | Most common L0 label |
|
||||
|---|---|---|---|---|
|
||||
| code | 10 | **100 %** | 0.77 | `code` |
|
||||
| medical | 10 | 80 % | 0.67 | `medical`, `general` |
|
||||
| finance | 10 | 80 % | 0.69 | `finance`, `general` |
|
||||
| education | 10 | 60 % | 0.62 | `education`, `general`, `code` |
|
||||
| math | 10 | 60 % | 0.58 | `math`, `general` |
|
||||
| legal | 10 | 60 % | 0.60 | `legal`, `general` |
|
||||
| life | 10 | 40 % | 0.54 | `general`, `life` |
|
||||
| creative | 10 | 10 % | 0.50 | `general` |
|
||||
| general | 10 | 0 % | 0.33 | `general` (but always escalated) |
|
||||
| **Total** | **90** | **74.4 %** | **0.59** | — |
|
||||
|
||||
**Observation 1 (the dominant cost lever).** The classifier achieves 100 % on `code` and ≤ 80 % on every other domain; on `general` it is *always* escalated because the mean confidence (0.33) is below the 0.60 threshold. The pattern correlates strongly with the keyword coverage of the rule base: code is keyword-rich in both Chinese and English, while `general` and `creative` are intrinsically catch-all.
|
||||
|
||||
**Observation 2 (the bilingual gap).** For every domain except `code`, the classifier performs materially better on Chinese queries than on English ones, because the rule base was authored in Chinese first. We view this not as a bug but as a *measurable* consequence of the source-language bias of the rule base.
|
||||
|
||||
### 5.3 Routing distribution and cost
|
||||
|
||||
**Table 2.** Routing distribution and projected cost (mock 90-query, costs computed with official prices).
|
||||
|
||||
| Selected candidate | Count | Share | Cost per 1K (USD) |
|
||||
|---|---|---|---|
|
||||
| `gpt-4o-mini` | 8 | 8.9 % | 0.04–0.15 |
|
||||
| `claude-3-haiku-20240307` | 16 | 17.8 % | 0.05 |
|
||||
| `gemini-2.0-flash-001` | 2 | 2.2 % | 0.02 |
|
||||
| `deepseek-chat` | 2 | 2.2 % | 0.03 |
|
||||
| `mistral-medium` (escalation) | 62 | **68.9 %** | 0.50 |
|
||||
| **Total** | 90 | 100 % | **0.51** |
|
||||
|
||||
The escalation rate of 68.9 % is the single most important number in this section. Because `mistral-medium` is roughly 6× the cost of the next-most-expensive candidate, the projected cost is dominated by the escalation path.
|
||||
|
||||
**Arena Score (mock).** Plugging the observed mock accuracy and cost into the official formula `S = (1+β)·Acc·C / (β·Acc + C)` with `β = 0.1`, `c_max = 200`, `c_min = 0.0044` yields `S_raw = 0.7223`, or `72.23` on the leaderboard's `×100` scale. The unit test `test_compute_arena_score_matches_formula` independently validates the formula against the published Hybrid Router data point (cost \$0.04/1K, accuracy 71.38 %, leaderboard 72.08) and reproduces it within 0.03 %.
|
||||
|
||||
**We do not place this number on the leaderboard.** The 90-query mock dataset is not the public 8 400-query benchmark, and the accuracy we measure is domain-classification accuracy, not answer-correctness. The number is reported solely to demonstrate that the adapter + formula + cost model produce internally consistent numbers and to provide a project-internal baseline.
|
||||
|
||||
### 5.4 Comparison with the public leaderboard
|
||||
|
||||
For context, Table 3 shows the 27-router public leaderboard (snapshot 2026-07). The columns are reproduced from the public leaderboard; readers should treat the numbers as "what the field looks like" rather than as a direct comparand with our mock.
|
||||
|
||||
**Table 3.** RouterArena public leaderboard (2026-07 snapshot, selected rows).
|
||||
|
||||
| Rank | Router | Arena Score | Accuracy | Cost/1K |
|
||||
|---|---|---|---|---|
|
||||
| 1 | Cross-Router | 75.75 | 78.14 | \$0.40 |
|
||||
| 2 | Sqwish Router | 75.27 | 76.40 | \$0.18 |
|
||||
| 3 | vLLM-SR | 74.86 | 77.18 | \$0.42 |
|
||||
| 4 | AgentForge Router | 74.13 | 74.72 | \$0.13 |
|
||||
| ... | ... | ... | ... | ... |
|
||||
| 9 | Hybrid Router | 72.08 | 71.38 | **\$0.04** |
|
||||
| 10 | R2-Router | 71.60 | 71.23 | \$0.06 |
|
||||
| 19 | GPT-5 | 64.32 | 73.96 | **\$10.02** |
|
||||
| 28 | RouterDC | 33.75 | 32.01 | \$0.07 |
|
||||
|
||||
**Observation 3 (cost competitiveness, conditional on accuracy).** Our L0 router is at the boundary of being competitive on the cost axis (Hybrid Router \$0.04/1K is a 12× advantage we do not have) but is *not* competitive on the accuracy axis in the leaderboard sense. The bottleneck is, again, the rule-based classifier and its low English coverage.
|
||||
|
||||
### 5.5 What changed between the 24-sample and 90-sample benchmark?
|
||||
|
||||
The 24-sample Chinese benchmark is dominated by the queries for which the rule base has explicit keywords; the 90-sample mock has approximately half English queries and includes the `creative` and `general` catch-alls. This 30-point accuracy drop is itself a finding: it bounds the *floor* of the rule-classifier's generalization and is consistent with the project's hypothesis that L0 should be replaced by L1 (trained classifier) before any production claim.
|
||||
|
||||
---
|
||||
|
||||
## 6. Discussion
|
||||
|
||||
### 6.1 Three lessons
|
||||
|
||||
**Lesson 1: English query coverage is the dominant cost lever.** 68.9 % of the 90 queries escalate to the most expensive candidate because the rule classifier cannot reliably identify the domain of English queries. The 12× cost penalty relative to Hybrid Router is, to first order, a 12× penalty of untranslated keywords.
|
||||
|
||||
**Lesson 2: The L0 design is a scientific instrument, not a deployment target.** The 0.51 USD/1K cost, the 74.4 % domain accuracy, and the 100 % interpretability of the routing trace together suggest that L0 is best understood as a *baseline* whose primary value is reproducibility and auditability. Any production-grade claim must wait for L1.
|
||||
|
||||
**Lesson 3: Threshold calibration is the next immediate research target.** The two thresholds (0.60 confidence, 0.70 quality) were inherited from the project defaults and are not calibrated to RouterArena's data distribution. The Conformal Cascade literature [Bates et al., 2026] provides a distribution-free calibration procedure that would let us convert the empirical 68.9 % escalation rate into a *bounded* theoretical escalation rate, removing one source of variance in cost projection.
|
||||
|
||||
### 6.2 Roadmap: L1 = trained small classifier
|
||||
|
||||
The single highest-leverage next step is to **replace the rule classifier with a trained small model**. The skeleton `scripts/train_classifier.py` is already in the project. Following the BERT-class-routing literature (ICL-Router [Kumar et al., 2026], MIRT-BERT [Yang et al., 2025]), we expect:
|
||||
|
||||
| Metric | L0 (current) | L1 (expected) | Hybrid Router (public) |
|
||||
|---|---|---|---|
|
||||
| Domain accuracy (mock 9-domain) | 74.4 % | 90–95 % | — |
|
||||
| Escalation rate | 68.9 % | 15–25 % | — |
|
||||
| Cost per 1K | \$0.51 | \$0.08–0.20 | \$0.04 |
|
||||
| Arena Score (×100, projected) | 72.23 | 74–78 | 72.08 |
|
||||
|
||||
L1 closes the cost gap and the accuracy gap simultaneously by attacking the single failure mode of L0.
|
||||
|
||||
### 6.3 Roadmap: L2 = local-expert fine-tuning
|
||||
|
||||
The longer-term roadmap is L2: train per-domain LoRA experts following CARE/VI-MoLE-style confidence-adaptive budgeting [Huang et al., 2026; Zhao et al., 2026]. This is the path that takes the system from "competitive on cost" to "competitive on cost *and* answer quality on hard queries". The key open question is whether the hard queries, after L1, are rare enough that the cost increase is acceptable; the RouterArena robustness evaluation (420 noise-injected queries) is the natural testbed.
|
||||
|
||||
### 6.4 What we would have done differently
|
||||
|
||||
In hindsight, the first action after building L0 should have been a *standardized* evaluation rather than the 24-sample internal benchmark. The lesson for the authors and for other small teams building LLM infrastructure is that internal benchmarks anchor the team to an overconfident baseline; an open benchmark provides a more honest trajectory.
|
||||
|
||||
---
|
||||
|
||||
## 7. Limitations
|
||||
|
||||
**Limitation 1: Mock dataset.** The empirical evaluation in §5 is on a 90-query mock dataset, not on the public 8 400-query RouterArena benchmark. We were unable to download the public dataset in our environment (network restrictions on the Hugging Face and GitHub endpoints). A full re-run on the public benchmark is the obvious next step and is *not* claimed to be a minor one: the public dataset's difficulty distribution, query length distribution, and per-domain prevalence are unknown to us and could materially change the conclusions.
|
||||
|
||||
**Limitation 2: Mock inference.** We do not run real LLM inference. The "accuracy" in this paper is the *domain-classification accuracy of the rule classifier*, not the *answer-correctness* of the selected model. The two are different quantities; conflating them would be a serious error and we explicitly disavow any such reading. A faithful re-run would require API keys for the five candidate models and a budget on the order of \$1–5 for the 8 400-query full benchmark.
|
||||
|
||||
**Limitation 3: Thresholds not calibrated.** The 0.60 / 0.70 thresholds are project defaults and have not been calibrated. A Conformal Cascade calibration is the natural fix (§6.1 Lesson 3) but is outside the scope of this paper.
|
||||
|
||||
**Limitation 4: Vendor drift in the vendored BaseRouter.** Because we vendored a minimal `BaseRouter` rather than depending on the full RouterArena repository, future RouterArena interface changes will require a manual sync. We mitigate this by (a) keeping the vendored file small and self-contained, (b) adding a unit test that re-validates the formula against the published leaderboard numbers, and (c) documenting the integration path in `00_integration_plan.md`.
|
||||
|
||||
**Limitation 5: Candidate pool is small and hand-picked.** The 5-model pool is not the union of all 27 leaderboard candidates. A larger pool would test a different and more interesting question (when to *not* use the L0 default) but is deferred to L1.
|
||||
|
||||
**Limitation 6: Single seed, no variance estimate.** All numbers in §5 are point estimates on a 90-query set. The standard error of an accuracy estimate at *p* = 0.74, *n* = 90 is ≈ 4.6 percentage points; readers should not over-interpret differences below this threshold.
|
||||
|
||||
---
|
||||
|
||||
## 8. Conclusion and Future Work
|
||||
|
||||
This paper has reported the first (to our knowledge) integration of a rule-based, parameter-free, expert-system-kernel multi-expert router with the public RouterArena benchmark. The contribution is methodological: an honest, gap-disclosing empirical study of an L0 design, a vendored and unit-tested adapter, and a clear roadmap. The principal finding is that the rule-based classifier's English-query coverage is the dominant cost lever, and the single highest-ROI next step is to replace it with a trained small classifier (L1). The project is open-source; the adapter, the 90-query mock dataset, the eight unit tests, and the 118 pre-existing project tests are all available for replication.
|
||||
|
||||
**Future work, in priority order:**
|
||||
|
||||
1. **Run on the public sub_10 / full benchmark.** The obvious next step. Requires network access to the Hugging Face dataset.
|
||||
2. **L1 trained classifier** (highest ROI). Replace the rule classifier with a BERT-class small model trained on internal Chinese data, projected to close the cost gap to within 2× of Hybrid Router.
|
||||
3. **Conformal Cascade threshold calibration.** Convert the empirical escalation rate to a bounded theoretical rate.
|
||||
4. **Robustness evaluation** on the 420-query noise-injected split.
|
||||
5. **PR submission** to the RouterArena leaderboard, with the 8 400-query full run as the qualifying submission.
|
||||
|
||||
We invite other small teams building LLM infrastructure to use this paper as a *negative case study*: the lesson is that an internal benchmark of 24 samples is not a substitute for an open benchmark of 8 400 samples, and the gap between them is large enough to invert the engineering priorities of a project.
|
||||
|
||||
---
|
||||
|
||||
## Acknowledgements
|
||||
|
||||
We thank the RouterArena team (Yifan Lu, Rixin Liu, Jiayi Yuan, Xingqi Cui, Shenrun Zhang, Hongyi Liu, Jiarong Xing) for building and maintaining the open benchmark on which this paper critically depends. We thank the maintainers of the 13 reference papers collected in `references/arxiv_papers/` for the survey input that shaped this work.
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
1. Lu, Y., Liu, R., Yuan, J., Cui, X., Zhang, S., Liu, H., & Xing, J. (2025). *RouterArena: An Open Platform for Comprehensive Comparison of LLM Routers.* ICLR 2026 / arXiv:2510.00202. https://arxiv.org/abs/2510.00202
|
||||
2. Wang, X., et al. (2026). *Dynamic Model Routing and Cascading for Efficient LLM Inference: A Survey.* arXiv:2603.04445.
|
||||
3. Hu, Q., et al. (2025). *Doing More with Less: Implementing Routing Strategies in LLM-Based Systems (Extended Survey).* arXiv:2502.00409. (Available in `references/arxiv_papers/13_Doing_More_With_Less_2502.00409.pdf`)
|
||||
4. Yue, Y., et al. (2025). *The Avengers: A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giants.* AAAI 2025 / arXiv:2505.19797.
|
||||
5. Jin, H., et al. (2026). *R2-Router: Output-Token-Budget-Aware Routing.* ICML 2026 / arXiv:2602.02823.
|
||||
6. Park, S., et al. (2026). *Meta-Router: Causal-Inference Routing with Biased Preference Data.* ICLR 2026 / arXiv:2509.25535.
|
||||
7. Chen, M., et al. (2026). *HyDRA: Shortfall Matching for Multi-Dimensional Capability Routing.* arXiv:2605.17106.
|
||||
8. Li, W., et al. (2026). *FusionRoute: Token-Level Multi-LLM Routing with Logit Correction.* ICML 2026.
|
||||
9. Zhang, T., et al. (2025). *Router-R1: Reinforcement Learning for Multi-Turn Routing.* NeurIPS 2025.
|
||||
10. Wang, Z., et al. (2025). *MESS+: SLA-Guaranteed Cost-Optimal Routing.* NeurIPS 2025.
|
||||
11. Kumar, A., et al. (2026). *ICL-Router: In-Context-Learning Model Representation for Cold-Start Routing.* AAAI 2026.
|
||||
12. He, J., et al. (2025). *Select-then-Route: A Two-Stage Cascading Router.* EMNLP 2025 Industry.
|
||||
13. Liu, B., et al. (2025). *RADAR: Reasoning-Aware Difficulty-Aware Routing.* NeurIPS 2025 Workshop.
|
||||
14. Somerville, R., et al. (2025). *CARROT: Cost-Aware Rate-Optimal Routing.* ICLR 2025 Workshop / arXiv:2502.03261.
|
||||
15. Tang, H., et al. (2025). *OmniRouter: Budget- and Performance-Controllable Multi-LLM Routing.* KDD 2025.
|
||||
16. Gao, P., et al. (2025). *SpareLLM: Equivalence-Constrained Minimal-Cost Model Selection.* SIGMOD 2025.
|
||||
17. Huang, L., et al. (2026). *CARE: Spend Experts Where You Are Unsure — Confidence-Adaptive LoRA Routing.* arXiv:2607.26052.
|
||||
18. Zhao, Q., et al. (2026). *VI-MoLE: Value-of-Information Routing for Shared Budget.* arXiv:2608.02528.
|
||||
19. Kim, J., et al. (2026). *Hard-Routed Mixtures of Reasoning LoRAs.* arXiv:2606.31413.
|
||||
20. Lee, S., et al. (2026). *ReMix: Reinforced Routing Mixture of LoRA.* arXiv 2026.
|
||||
21. Schmidt, A., et al. (2026). *Cluster, Route, Escalate: Selective Escalation with Sub-5 ms TPOT Penalty.* arXiv:2606.27457.
|
||||
22. Bates, S., et al. (2026). *Conformal Cascade: Distribution-Free Confidence-Based Deferral.* arXiv:2607.25018.
|
||||
23. Martin, D., et al. (2026). *The Routing Plateau: Causes and Remedies for Router Accuracy Ceilings.* arXiv:2606.07587.
|
||||
24. Dubey, A., et al. (2026). *Small Language Models: A Systematic Review.* arXiv 2026.
|
||||
25. Industry Report (2026). *Specialization Beats Scale: When 3B Models Outperform GPT-4o.* Industry analysis.
|
||||
26. Yang, X., et al. (2025). *IRT-Router: MIRT-BERT and NIRT-BERT Item-Response-Theory Routers.* arXiv:2506.01048.
|
||||
27. Multi-Expert Router Project (2026). *2026 Papers Survey for Multi-Expert Router Project.* Internal research notes, `research/2026_papers_survey.md`.
|
||||
28. R2R Authors (2025). *R2R: Efficiently Navigating Divergent Reasoning Paths with Small-Large Model Token Routing.* NeurIPS 2025 / arXiv:2505.21600. (Available in `references/arxiv_papers/02_R2R_Token_Routing_2505.21600.pdf`)
|
||||
29. BEST-Route Authors (2025). *BEST-Route: Adaptive LLM Routing with Test-Time Optimal Compute.* ICML 2025 / arXiv:2506.22716. (Available in `references/arxiv_papers/03_BEST_Route_2506.22716.pdf`)
|
||||
30. SATER Authors (2025). *SATER: A Self-Aware and Token-Efficient Approach to Routing and Cascading.* EMNLP 2025 / arXiv:2510.05164. (Available in `references/arxiv_papers/04_SATER_2510.05164.pdf`)
|
||||
31. Mixture-of-Parrots Authors (2025). *Mixture of Parrots: Experts Improve Memorization More than Reasoning.* ICLR 2025 / arXiv:2410.19034. (Available in `references/arxiv_papers/07_Mixture_of_Parrots_2410.19034.pdf`)
|
||||
32. DomainCodeBench Authors (2025). *DomainCodeBench: A Multi-domain Code Generation Benchmark.* arXiv:2412.18573. (Available in `references/arxiv_papers/08_DomainCodeBench_2412.18573.pdf`)
|
||||
33. Model-SAT Authors (2025). *Capability Instruction Tuning: A New Paradigm for Dynamic LLM Routing.* AAAI 2025 / arXiv:2502.17282. (Available in `references/arxiv_papers/09_Model_SAT_CIT_2502.17282.pdf`)
|
||||
34. RouterRetriever Authors (2025). *RouterRetriever: Routing over a Mixture of Expert Embedding Models.* AAAI 2025 / arXiv:2409.02685. (Available in `references/arxiv_papers/10_RouterRetriever_2409.02685.pdf`)
|
||||
35. MergeBench Authors (2025). *MergeBench: A Benchmark for Merging Domain-Specialized LLMs.* NeurIPS 2025 / arXiv:2505.10833. (Available in `references/arxiv_papers/12_MergeBench_2505.10833.pdf`)
|
||||
36. Angelopoulos, A. N., et al. (2024). *Conformal Prediction: A Gentle Introduction.* Foundations and Trends in Machine Learning. (General reference for §6.1 Lesson 3.)
|
||||
|
||||
---
|
||||
|
||||
## Appendix A. Reproducibility
|
||||
|
||||
The paper corresponds to a single `git commit` of the project. The full state is reproducible with:
|
||||
|
||||
```bash
|
||||
# 1. Environment
|
||||
C:\Python314\python.exe -m venv .venv
|
||||
.venv\Scripts\python.exe -m pip install -r requirements.txt
|
||||
|
||||
# 2. Run the mock RouterArena evaluation
|
||||
.venv\Scripts\python.exe -m research.routerarena.local_runner
|
||||
|
||||
# 3. Inspect the prediction file
|
||||
cat research/routerarena/output/es-expert.json
|
||||
|
||||
# 4. Run the unit tests
|
||||
.venv\Scripts\python.exe -m pytest tests/test_routerarena_adapter.py -v
|
||||
```
|
||||
|
||||
Expected: 8/8 tests pass, 90 predictions generated, mock Arena Score 72.23, mock accuracy 74.4 %, cost \$0.51/1K, escalation rate 68.9 %.
|
||||
|
||||
## Appendix B. Mapping Table (Evidence from `00_integration_plan.md` §3.2)
|
||||
|
||||
The 8 L0 domain labels and their corresponding RouterArena 9-domain classes. The mapping is hand-designed (not learned) and is the single largest source of adapter bias.
|
||||
|
||||
| L0 domain | RouterArena domain | Adapter → candidate |
|
||||
|---|---|---|
|
||||
| `code` | code | `gpt-4o-mini` |
|
||||
| `math` | math | `claude-3-haiku-20240307` |
|
||||
| `legal` | legal | `claude-3-haiku-20240307` |
|
||||
| `medical` | medical | `claude-3-haiku-20240307` |
|
||||
| `finance` | finance | `claude-3-haiku-20240307` |
|
||||
| `life` | life | `gemini-2.0-flash-001` |
|
||||
| `education` | education | `deepseek-chat` |
|
||||
| `general` | general | `gemini-2.0-flash-001` |
|
||||
| — (no L0 label) | creative | `gemini-2.0-flash-001` (fallback) |
|
||||
| any | (escalation) | `mistral-medium` (if conf<0.60 or qual<0.70) |
|
||||
|
||||
## Appendix C. The Three Discoveries, Compressed
|
||||
|
||||
1. **English coverage is the dominant cost lever.** 68.9 % of 90 queries escalate to the most expensive candidate, because the rule classifier cannot reliably identify the domain of English queries. The 12× cost penalty relative to Hybrid Router is, to first order, a 12× penalty of untranslated keywords.
|
||||
2. **L0 is a scientific instrument, not a deployment target.** The 0.51 USD/1K cost, the 74.4 % domain accuracy, and the 100 % interpretability of the routing trace together suggest that L0 is best understood as a *baseline* whose primary value is reproducibility and auditability. Any production-grade claim must wait for L1.
|
||||
3. **The single highest-ROI next step is L1 (trained classifier).** A BERT-class small classifier is expected to close the cost gap to within 2× of Hybrid Router and the accuracy gap to within 5 percentage points of the best 2026 published routers.
|
||||
|
||||
— *End of paper* —
|
||||
@@ -0,0 +1,94 @@
|
||||
# RouterArena 接入方案（方案 v0.1）
|
||||
|
||||
> 撰写日期：2026-08-19
|
||||
> 依据：RouterArena README + base_router.py + generate_prediction_file.py + llm_evaluation/run.py（GitHub @ `RouteWorks/RouterArena`，2026-08-19 拉取）
|
||||
> 状态：方案已固化，开始 Step 3 实现
|
||||
|
||||
---
|
||||
|
||||
## 1. 目标
|
||||
|
||||
把本系统的 L0 专家系统路由器接入 RouterArena 标准化评测，得到可与排行榜 27 个公开路由器直接对比的 Arena Score、Accuracy、Cost/1K、Latency、Robustness 5 维指标。
|
||||
|
||||
承接 `research/2026_papers_survey.md` 第三节第 1 条建议："本项目下一步应优先做三件事"中的第一件。
|
||||
|
||||
## 2. 关键证据（来自 RouterArena 源码）
|
||||
|
||||
| 事实 | 来源 | 对本方案的影响 |
|
||||
|------|------|--------------|
|
||||
| 评测仅看 prompt + global_index，**不接触 ground truth answer** | README L34-37, generate_prediction_file.py L84-115 | 路由阶段只做"选模型"，避免与训练阶段耦合 → 合规 |
|
||||
| 接口 `_get_prediction(self, query) -> str` 返回的必须是 `config["pipeline_params"]["models"]` 中的名字 | base_router.py L139-167 | 必须把我们 L0 的"选规则模板"重映射为"选目标 LLM" |
|
||||
| 候选模型必须出现在 `universal_model_names.py` 与 `model_cost/model_cost.json` | README L102-107 | 候选模型要选排行榜主流、有公开价格、API 可达 |
|
||||
| sub_10 子集 809 条，full 8400 条 | generate_prediction_file.py L88-92 | 先 sub_10 烟测，再考虑 full |
|
||||
| Arena 公式 β=0.1, c_max=200, c_min=0.0044 | run.py L65-87 | 极低成本的路由器（$0.001/1K 量级）天然占优 |
|
||||
| 评测完整流程 = 路由 + 推理（API 调用）+ 评分 | README L100-130 | 推理阶段需要真实 API key 与钱 |
|
||||
|
||||
## 3. 适配方案（设计决策）
|
||||
|
||||
### 3.1 候选模型池（5 个，论文调研中"低成本代表"）
|
||||
|
||||
| 槽位 | 模型 | 选它的理由 | 预期价格（$ / 1K queries，按公开定价） |
|
||||
|------|------|----------|--------------------------------------|
|
||||
| `code-strong` | `gpt-4o-mini` | 排行榜出现频次最高，代码能力强 | 0.15-0.6 / 1M tok → ~$0.04-0.15/1K |
|
||||
| `reasoning-strong` | `claude-3-haiku-20240307` | 通用推理/法律/医疗 | $0.25/1M input → ~$0.05/1K |
|
||||
| `generalist-fast` | `gemini-2.0-flash-001` | 通用快速，覆盖 life/education | $0.075/1M → ~$0.02/1K |
|
||||
| `multilingual-cn` | `deepseek-chat` | 中文/教育/通用 | $0.14/1M → ~$0.03/1K |
|
||||
| `reasoning-mid` | `mistral-medium` | 备用 / 难样本兜底 | $2.7/1M → ~$0.5/1K |
|
||||
|
||||
> 注：精确价格需按 `model_cost/model_cost.json` 与 `universal_model_names.py` 校验。预期区间与排行榜 Hybrid Router $0.04/1K、GPT-5 $10.02/1K 同量级。
|
||||
|
||||
### 3.2 L0 域 → 模型槽位 映射表（核心决策）
|
||||
|
||||
基于本系统 8 领域 × LLM 擅长：
|
||||
|
||||
| L0 域 | 子域样本 | 选 | 理由 |
|
||||
|-------|---------|----|----|
|
||||
| `code` | algorithm, debugging, database | `code-strong` (gpt-4o-mini) | 编程类基准 SOTA（LiveCodeBench/Codeforces） |
|
||||
| `math` | algebra, geometry, calculus | `reasoning-strong` (claude-3-haiku) | 数学推理强项 |
|
||||
| `legal` | labor, contract | `reasoning-strong` (claude-3-haiku) | 法律长文本/严谨性 |
|
||||
| `medical` | chronic, medication, firstaid | `reasoning-strong` (claude-3-haiku) | 医疗警示/准确性优先 |
|
||||
| `finance` | investing, credit, loan | `reasoning-strong` (claude-3-haiku) | 数字严谨/风险提示 |
|
||||
| `life` | travel, fitness, food | `generalist-fast` (gemini-2.0-flash) | 实用建议快速返回 |
|
||||
| `education` | exam, study, career | `multilingual-cn` (deepseek-chat) | 中文/教育/应试 |
|
||||
| `general` | explain, writing | `generalist-fast` (gemini-2.0-flash) | 通用兜底成本最低 |
|
||||
|
||||
**兜底规则**：当 confidence < 0.60 或 quality_score < 0.70（与 L0 Judge 阈值一致），升一级到 `reasoning-mid` (mistral-medium)。
|
||||
|
||||
### 3.3 实现三件套
|
||||
|
||||
| 文件 | 角色 | 依赖 |
|
||||
|------|------|------|
|
||||
| `research/routerarena/adapter.py` | `BaseRouter` 子类，调用本系统 L0 router | 仅零依赖 |
|
||||
| `research/routerarena/config/es-expert.json` | 候选模型池 + 类名配置 | 无 |
|
||||
| `scripts/routerarena_generate.py` | 复用 RouterArena 的 `generate_prediction_file.py` 入口 | 需要把 adapter 注册进 RouterArena 路由器模块 |
|
||||
|
||||
### 3.4 验证策略（分阶段）
|
||||
|
||||
1. **烟测（无 API 成本）**：跑 `sub_10` 生成预测文件，验证 809 条记录格式合规、模型名映射一致
|
||||
2. **Mock 推理 + 评分方法学验证**：把 `generated_result` 填充为 mock 但带 token_usage，验证 Arena 公式可跑通
|
||||
3. **真子集（消耗 API）**：sub_10 上跑 5-10 条做端到端烟测，验证 API key 路径
|
||||
4. **全量评测（消耗 API）**：full 8400 条，需预算 ~$1-5（按 5 模型轮询成本估算）
|
||||
|
||||
### 3.5 风险与缺口
|
||||
|
||||
| 风险 | 影响 | 缓解 |
|
||||
|------|------|------|
|
||||
| 没有 API key | 完整评测跑不通 | 先做烟测 + 方法学验证；提交方案文档说明 gap |
|
||||
| RouterArena 用 8 领域，本系统 8 领域但标签不一一对应 | 路由决策可比性下降 | 文档里写清映射表 + 不能直接比的子域 |
|
||||
| L0 路由是基于关键词规则的，可能对 RouterArena 长 prompt 命中率下降 | 准确率会低于 100% | 这是预期：自建 24 样例 100% 不代表 809 条也是，要看真实分布 |
|
||||
| 模型价格/能力随时间变化 | 排行榜基线会漂移 | 报告里标注"基于 2026-07 快照"，未来要重测 |
|
||||
|
||||
## 4. 不做的事（明确边界）
|
||||
|
||||
- **不在 RouterArena 标签上训练/微调**（合规要求）
|
||||
- **不修改本系统核心 router**（只新增 adapter 适配层）
|
||||
- **不在第一阶段做 full 评测**（成本/时间/资源限制，先 sub_10 验证方法学）
|
||||
- **不立即对接 RouterArena PR 提交**（先有方法学验证 + gap 文档）
|
||||
|
||||
## 5. 验收标准
|
||||
|
||||
- [ ] `research/routerarena/adapter.py` 实现完整，可独立 import
|
||||
- [ ] 在 sub_10 上生成 809 条预测，文件格式通过 `check_config_prediction_files.py` 校验
|
||||
- [ ] mock 推理跑通 Arena 公式，得到本系统的 Arena Score 数字
|
||||
- [ ] 写完 `01_results_sub10.md` 报告，含与排行榜 5 个基线（Hybrid Router / R2-Router / GPT-5 / MIRT-BERT / NotDiamond）的对比表
|
||||
- [ ] 不引入新依赖到 requirements.txt（仅 Python 标准库 + 已有 deps）
|
||||
@@ -0,0 +1,225 @@
|
||||
# RouterArena 接入结果 + 差距分析报告（v0.1）
|
||||
|
||||
> 实验日期：2026-08-19
|
||||
> 路由器：本系统 L0 专家系统内核 + ESExpertRouter 适配器
|
||||
> 数据集：本项目内 mock 子集（90 条 × 9 域），**RouterArena 真实 sub_10 因环境网络受限未能拉取**
|
||||
> 评测公式：与 `RouteWorks/RouterArena/llm_evaluation/run.py` L65-87 一字不差（已通过单元测试 `test_compute_arena_score_matches_formula` 验证 Hybrid Router $0.04/1K, 71.38% acc → raw 0.7208，×100 = leaderboard 公开值 72.08）
|
||||
|
||||
---
|
||||
|
||||
## 1. 实验设置
|
||||
|
||||
### 1.1 路由器
|
||||
|
||||
- 名称：`es-expert`
|
||||
- 实现：`research/routerarena/adapter.py::ESExpertRouter`
|
||||
- 底层：本系统 L0 Router（规则分类器 + 任务规划 + 规则执行器 + Judge + 兜底）
|
||||
- 候选模型池（5 槽，config `es-expert.json`）：
|
||||
- `gpt-4o-mini`：code 域
|
||||
- `claude-3-haiku-20240307`：math/legal/medical/finance
|
||||
- `gemini-2.0-flash-001`：life/general
|
||||
- `deepseek-chat`：education
|
||||
- `mistral-medium`：升级兜底（conf<0.60 或 quality<0.70）
|
||||
|
||||
### 1.2 决策规则（核心证据：`adapter.py::_decide`）
|
||||
|
||||
```python
|
||||
def _decide(self, domain, confidence, quality_score):
|
||||
if domain not in DOMAIN_TO_MODEL_SLOT:
|
||||
domain = "general"
|
||||
if confidence < LOW_CONFIDENCE_THRESHOLD or quality_score < JUDGE_FALLBACK_THRESHOLD:
|
||||
return ESCALATION_MODEL_SLOT # mistral-medium
|
||||
return DOMAIN_TO_MODEL_SLOT[domain]
|
||||
```
|
||||
|
||||
阈值与本系统 `config/config.yaml` 默认一致：`LOW_CONFIDENCE_THRESHOLD=0.60`，`JUDGE_FALLBACK_THRESHOLD=0.70`。
|
||||
|
||||
### 1.3 数据集（mock）
|
||||
|
||||
9 域 × 10 query = 90 条，覆盖中英文 + 3 个难度级：
|
||||
|
||||
| 域 | 中文 query | 英文 query | 期望映射到 |
|
||||
|----|-----------|-----------|----------|
|
||||
| code | 用 Python 写一个快速排序函数 | Implement quicksort in Python | gpt-4o-mini |
|
||||
| math | 求方程 x^2+3x+2=0 的根 | Solve x^2 - 5x + 6 = 0 | claude-3-haiku-20240307 |
|
||||
| legal | 劳动合同到期不续签是否要给补偿金 | Is a non-compete clause for 2 years enforceable? | claude-3-haiku-20240307 |
|
||||
| medical | 高血压患者日常饮食 | What foods should hypertensive patients avoid | claude-3-haiku-20240307 |
|
||||
| finance | 基金定投收益率怎么计算 | How to calculate ROI on a fund | claude-3-haiku-20240307 |
|
||||
| life | 日本旅行攻略 | Travel itinerary for Japan in 7 days | gemini-2.0-flash-001 |
|
||||
| education | 考研英语怎么备考 | How to prepare for graduate English exam | deepseek-chat |
|
||||
| general | 为什么天空是蓝色的 | Why is the sky blue | gemini-2.0-flash-001 |
|
||||
| creative | Write a haiku about autumn | 写一首关于秋天的俳句 | gemini-2.0-flash-001 |
|
||||
|
||||
> 说明：creative 域在 RouterArena 中存在但本系统未实现专用规则，按"unknown domain → general"兜底；与 ground truth creative 比较时统一映射为 general。
|
||||
|
||||
### 1.4 推理
|
||||
|
||||
本次为方法学验证，不调真实 LLM（无 API key）。`local_runner.mock_inference()` 用本系统 L0 Router 自己的 response 模拟目标 LLM 的 generated_answer；token 数按字符数估算。
|
||||
|
||||
**重要标注**：mock 推理不替代真实 LLM 推理；本次 Arena Score **仅供方法学对齐参考**，不可与 leaderboard 直接比较。
|
||||
|
||||
## 2. 路由层结果（不含真实推理）
|
||||
|
||||
### 2.1 路由决策分布
|
||||
|
||||
| 槽位 | 选中次数 | 占比 | 期望（理想情况） |
|
||||
|------|---------|------|----------------|
|
||||
| gpt-4o-mini | 8 | 8.9% | 11.1% (code 域) |
|
||||
| claude-3-haiku-20240307 | 16 | 17.8% | 44.4% (math+legal+medical+finance) |
|
||||
| gemini-2.0-flash-001 | 2 | 2.2% | 22.2% (life+general) |
|
||||
| deepseek-chat | 2 | 2.2% | 11.1% (education) |
|
||||
| **mistral-medium (升级)** | **62** | **68.9%** | 0% |
|
||||
| 总计 | 90 | 100% | — |
|
||||
|
||||
### 2.2 按 ground truth domain 分类准确率
|
||||
|
||||
| GT 域 | L0 分类正确率 | 平均 conf | 主要决策路径 |
|
||||
|-------|-------------|-----------|-------------|
|
||||
| code | **100%** (10/10) | 0.77 | gpt-4o-mini 7 + mistral 升级 3 |
|
||||
| medical | 80% (8/10) | 0.67 | mistral 5 + claude-haiku 5 |
|
||||
| finance | 80% (8/10) | 0.69 | mistral 5 + claude-haiku 5 |
|
||||
| education | 60% (6/10) | 0.62 | mistral 7 + deepseek 2 |
|
||||
| math | 60% (6/10) | 0.58 | mistral 7 + claude-haiku 3 |
|
||||
| legal | 60% (6/10) | 0.60 | mistral 7 + claude-haiku 3 |
|
||||
| life | 40% (4/10) | 0.54 | mistral 8 + gemini 2 |
|
||||
| creative | 10% (1/10) | 0.50 | mistral 10（creative→general，1 个落到 code） |
|
||||
| general | 0% (0/10) | 0.33 | mistral 10（conf<0.60 全部升级） |
|
||||
| **整体** | **74.4% (67/90)** | 0.59 | — |
|
||||
|
||||
### 2.3 关键现象（带证据）
|
||||
|
||||
1. **英文 query 大量触发升级**：68.9% 走 mistral-medium，根因是 L0 规则分类器对英文长 query 关键词覆盖差（看 `code=10/10` vs `life=4/10` 对比，差异来自"implement/sort"等英文代码词被识别，"travel itinerary"未被识别为 life 域）。
|
||||
2. **general 域全部升级**：conf 平均 0.33 远低于 0.60 阈值，0/10 走非升级路径。
|
||||
3. **quality_score 极化**：90 条 query 中 quality_score 只有 0 和 1 两个值（mock response 太短，规则 Judge 评分困难）。
|
||||
4. **中文 vs 英文**：中文 query 在 life/education/legal 等域识别率显著高于对应英文 query（证据见 `output/es-expert_diagnostics.json` 中每条 query 的 domain 字段）。
|
||||
|
||||
## 3. Arena Score（mock 推理下的方法学验证）
|
||||
|
||||
| 指标 | 我们的 mock 值 | 备注 |
|
||||
|------|--------------|------|
|
||||
| n_queries | 90 | mock 子集大小 |
|
||||
| mock accuracy | 0.7444 | 域分类匹配率（不替代真实回答正确率） |
|
||||
| total_cost (USD) | 0.0460 | mock 推理 token 估算 × 模型价格 |
|
||||
| cost_per_1k (USD) | 0.511 | 偏高，因 68% 走 mistral-medium（$2.7-8.1/1M tok） |
|
||||
| arena_score (raw) | 0.7223 | 用 `compute_arena_score` 算出 |
|
||||
| arena_score (×100) | 72.23 | 与 leaderboard 标度对齐 |
|
||||
| routing_latency (ms/query) | 21.24 | 仅 L0 路由，不含目标 LLM 推理 |
|
||||
|
||||
**该分数的解读边界**：
|
||||
- 我们的 `compute_arena_score` 与官方公式一字不差（已通过单测验证 `Hybrid Router 0.7208 × 100 = 72.08`）
|
||||
- 但 mock 推理 ≠ 真实 LLM 推理
|
||||
- mock accuracy（域分类）≠ 真实评测的"回答正确率"
|
||||
- **该 72.23 不可与 leaderboard 直接比较**
|
||||
|
||||
## 4. 与排行榜基线对比（**仅供框架对齐参考，不可直接比**）
|
||||
|
||||
| 路由器 | arena_score | accuracy | cost/1K (USD) | 数据 |
|
||||
|--------|-------------|---------|--------------|------|
|
||||
| Cross-Router | 75.75 | 78.14 | 0.40 | leaderboard 公开 |
|
||||
| **es-expert (mock, 本项目)** | **72.23** | **74.4 (域分类)** | **0.51** | **mock** |
|
||||
| Sqwish Router | 75.27 | 76.40 | 0.18 | leaderboard 公开 |
|
||||
| vLLM-SR | 74.86 | 77.18 | 0.42 | leaderboard 公开 |
|
||||
| AgentForge Router | 74.13 | 74.72 | 0.13 | leaderboard 公开 |
|
||||
| Nadir-Tumbler | 73.44 | 75.34 | 0.37 | leaderboard 公开 |
|
||||
| Weave Router | 72.82 | 76.32 | 0.94 | leaderboard 公开 |
|
||||
| Nadir Router | 72.29 | 75.01 | 0.68 | leaderboard 公开 |
|
||||
| OrcaRouter-Adaptive | 72.08 | 75.54 | 1.00 | leaderboard 公开 |
|
||||
| Hybrid Router | 72.08 | 71.38 | **0.04** | leaderboard 公开 |
|
||||
| R2-Router | 71.60 | 71.23 | 0.06 | leaderboard 公开 |
|
||||
| LLM Router | 71.26 | 72.05 | 0.20 | leaderboard 公开 |
|
||||
| chuzom-solo-v32 | 70.61 | 70.59 | 0.10 | leaderboard 公开 |
|
||||
| Azure-Model-Router | 70.42 | 72.94 | 0.73 | leaderboard 公开 |
|
||||
| Auto Router | 70.05 | 70.17 | 0.12 | leaderboard 公开 |
|
||||
| Lynkr | 67.65 | 68.41 | 0.29 | leaderboard 公开 |
|
||||
| BARouter | 67.09 | 68.80 | 0.63 | leaderboard 公开 |
|
||||
| MIRT-BERT | 66.89 | 66.88 | 0.15 | leaderboard 公开 |
|
||||
| NIRT-BERT | 66.12 | 66.34 | 0.21 | leaderboard 公开 |
|
||||
| GPT-5 | 64.32 | 73.96 | 10.02 | leaderboard 公开 |
|
||||
| CARROT | 63.87 | 67.21 | 2.06 | leaderboard 公开 |
|
||||
| Chayan | 63.83 | 64.89 | 0.56 | leaderboard 公开 |
|
||||
| RouterBench-MLP | 57.56 | 61.62 | 4.83 | leaderboard 公开 |
|
||||
| NotDiamond | 57.29 | 60.83 | 4.10 | leaderboard 公开 |
|
||||
| GraphRouter | 57.22 | 57.00 | 0.34 | leaderboard 公开 |
|
||||
| RouterBench-KNN | 55.48 | 58.69 | 4.27 | leaderboard 公开 |
|
||||
| RouteLLM | 48.07 | 47.04 | 0.27 | leaderboard 公开 |
|
||||
| RouterDC | 33.75 | 32.01 | 0.07 | leaderboard 公开 |
|
||||
|
||||
**注：**
|
||||
- 我们的 mock 数据 90 条是均匀 9 域分布，RouterArena 真实 sub_10 809 条与 full 8400 条的真实分布未知
|
||||
- 我们的"accuracy"是域分类正确率 74.4%，leaderboard 的"accuracy"是回答正确率（不同度量）
|
||||
- 即使在 mock 框架下，**我们的 cost_per_1k = $0.51** 远高于 Hybrid Router $0.04，因为 68% query 触发升级到 mistral-medium
|
||||
- 真正低成本优势要等 L1 分类器上线（论文调研 §3 第 2 条建议）
|
||||
|
||||
## 5. 关键发现（科研价值）
|
||||
|
||||
### 5.1 L0 规则分类器的真实瓶颈
|
||||
|
||||
| 瓶颈 | 证据 | 影响 |
|
||||
|------|------|------|
|
||||
| 英文 query 覆盖率低 | code=100% vs life=40% 对比；creative=10% | 升级率 68%，成本失控 |
|
||||
| general 域 conf 天然低 | 10/10 conf<0.60 | general 域永远走升级 |
|
||||
| quality_score 极化 | 仅 0/1 两值 | Judge 阈值 0.70 在 mock 推理下失效 |
|
||||
| 关键词依赖 | 中文 query 在多域识别率高于英文 | 国际化能力差 |
|
||||
|
||||
### 5.2 论文调研的关键预测被验证
|
||||
|
||||
`research/2026_papers_survey.md` 第三节第 2 条建议（"分类器升级为训练模型"）**已被本实验间接验证**：规则分类器在 90 条 9 域样本上掉到 74.4%，远低于论文调研中 BERT 级分类器 94-97% 的水平。
|
||||
|
||||
### 5.3 路由决策科学性验证
|
||||
|
||||
- 接口契约 100% 合规（8/8 单测通过）
|
||||
- 公式与官方一字不差（单测 `test_compute_arena_score_matches_formula` 验证）
|
||||
- 升级路径合理（conf<0.60 升级，符合 RouterArena 论文中的"selective escalation"思想，引用 Cluster, Route, Escalate NeurIPS 2025）
|
||||
|
||||
## 6. Gap 与下一步
|
||||
|
||||
### 6.1 真实数据接入（最高优先级）
|
||||
|
||||
**缺口**：本次仅在 mock 子集上跑通；未拉取真实 RouterArena sub_10 / full 数据。
|
||||
|
||||
**接入路径**（待执行）：
|
||||
1. 安装 `datasets` 库到 venv
|
||||
2. `python scripts/process_datasets/prep_datasets.py`（按 RouterArena README L82-86）
|
||||
3. 复制本项目 `research/routerarena/local_runner.py` 为 RouterArena 仓库内的 `router_inference/router/es_expert.py` 并继承 `BaseRouter`
|
||||
4. 配置 `router_inference/config/es-expert.json`（已就位）
|
||||
5. 跑 `python router_inference/generate_prediction_file.py es-expert sub_10`
|
||||
6. 配置目标 LLM 的 API key
|
||||
7. 跑 `python llm_inference/run.py es-expert` → `python llm_evaluation/run.py es-expert sub_10`
|
||||
|
||||
### 6.2 L1 分类器替换（论文调研 §3 第 2 条建议）
|
||||
|
||||
按 `scripts/train_classifier.py` 训练 BERT 级分类器替换规则分类器，预期 90 条 mock accuracy 提升到 90%+，升级率从 68% 降到 15-20%，成本从 $0.51/1K 降到 $0.10-0.20/1K（与 Hybrid Router $0.04 同量级）。
|
||||
|
||||
### 6.3 Conformal Cascade 校准 Judge 阈值（论文调研 §3 第 4 条建议）
|
||||
|
||||
当前 0.70 阈值是经验值；用 Conformal Cascade 思路（arXiv 2607.25018）做分布无关的阈值校准，可对升级率给出理论保证。
|
||||
|
||||
### 6.4 路由器 PR 提交（远期）
|
||||
|
||||
完成上述两步后，按 RouterArena README L130-160 的提交流程，提交到官方仓库，触发 `/evaluate` 命令上 leaderboard。
|
||||
|
||||
## 7. 验收
|
||||
|
||||
- [x] `research/routerarena/adapter.py` 实现完整
|
||||
- [x] `research/routerarena/base_router.py` vendored with SPDX
|
||||
- [x] `research/routerarena/local_runner.py` 跑通 mock 子集
|
||||
- [x] `research/routerarena/config/es-expert.json` 5 候选模型
|
||||
- [x] 8/8 单元测试通过（test_routerarena_adapter.py）
|
||||
- [x] 完整测试套 126/126 通过（118 旧 + 8 新）
|
||||
- [x] 预测文件 `output/es-expert.json` 90 条符合 RouterArena 协议
|
||||
- [x] 路由层方法学验证（mock accuracy 74.4%）
|
||||
- [x] Arena Score 公式与官方一字不差
|
||||
- [x] **不引入新依赖**到 requirements.txt
|
||||
- [ ] 真实 sub_10 跑通（gap，环境网络受限）
|
||||
- [ ] 提交 RouterArena PR（远期）
|
||||
|
||||
## 8. 数据文件
|
||||
|
||||
| 文件 | 内容 |
|
||||
|------|------|
|
||||
| `research/routerarena/output/es-expert.json` | RouterArena 协议预测文件，90 条 |
|
||||
| `research/routerarena/output/es-expert_diagnostics.json` | 完整路由诊断（domain/conf/quality/route/selected_slot） |
|
||||
| `research/routerarena/output/es-expert_summary.json` | 聚合指标 + 路由分布 |
|
||||
| `tests/test_routerarena_adapter.py` | 8 项单元测试（接口/映射/升级/公式/协议/Schema） |
|
||||
| `research/routerarena/00_integration_plan.md` | 接入方案设计文档 |
|
||||
| `research/routerarena/01_results_and_gap_analysis.md` | 本报告 |
|
||||
@@ -0,0 +1 @@
|
||||
"""research.routerarena package 入口。"""
|
||||
@@ -0,0 +1,126 @@
|
||||
"""本系统 L0 路由器 → RouterArena BaseRouter 适配器。
|
||||
|
||||
核心职责：
|
||||
- 接收 RouterArena 的 prompt（不含 ground truth）
|
||||
- 调用本系统 Router.route() 得到 (domain, confidence, quality_score)
|
||||
- 按 `DOMAIN_TO_MODEL_SLOT` 映射到 config.models 中的目标模型名
|
||||
- 兜底：confidence<0.60 或 quality<0.70 → 升到 reasoning-mid
|
||||
|
||||
合规说明（依据 RouterArena README L34-37 + base_router.py L139-167）：
|
||||
- 路由阶段不接触 ground truth answer
|
||||
- 不在 RouterArena 数据上训练/微调
|
||||
- 8 领域映射表是手工设计，不基于 RouterArena 标签学习
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict
|
||||
|
||||
# 把项目根加入 path，以便 import router_system
|
||||
_PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent
|
||||
if str(_PROJECT_ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(_PROJECT_ROOT))
|
||||
|
||||
from router_system.router import Router, build_router # noqa: E402
|
||||
|
||||
from .base_router import BaseRouter # noqa: E402
|
||||
|
||||
|
||||
# L0 域 → RouterArena 候选模型槽位 的映射
|
||||
# 依据 00_integration_plan.md §3.2：
|
||||
# - code → gpt-4o-mini（代码强项）
|
||||
# - math/legal/medical/finance → claude-3-haiku-20240307（推理严谨性）
|
||||
# - life/general → gemini-2.0-flash-001（实用+快速）
|
||||
# - education → deepseek-chat（中文/教育）
|
||||
DOMAIN_TO_MODEL_SLOT: Dict[str, str] = {
|
||||
"code": "gpt-4o-mini",
|
||||
"math": "claude-3-haiku-20240307",
|
||||
"legal": "claude-3-haiku-20240307",
|
||||
"medical": "claude-3-haiku-20240307",
|
||||
"finance": "claude-3-haiku-20240307",
|
||||
"life": "gemini-2.0-flash-001",
|
||||
"education": "deepseek-chat",
|
||||
"general": "gemini-2.0-flash-001",
|
||||
}
|
||||
|
||||
# 兜底：低置信度或低质 → 升到更稳的模型
|
||||
ESCALATION_MODEL_SLOT = "mistral-medium"
|
||||
|
||||
# 阈值：与本系统 config.yaml 默认对齐
|
||||
LOW_CONFIDENCE_THRESHOLD = 0.60
|
||||
JUDGE_FALLBACK_THRESHOLD = 0.70
|
||||
|
||||
|
||||
class ESExpertRouter(BaseRouter):
|
||||
"""把本系统 L0 Router 适配为 RouterArena BaseRouter。
|
||||
|
||||
重要：_get_prediction 必须只基于 query（无 ground truth），
|
||||
返回值必须在 config.models 中。
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
router_name: str,
|
||||
config_path: str = "",
|
||||
underlying_router: Router = None,
|
||||
):
|
||||
super().__init__(router_name, config_path=config_path or None)
|
||||
# 复用本系统 L0 Router（默认配置；如已 build 过可注入）
|
||||
self._router = underlying_router or build_router()
|
||||
|
||||
def _get_prediction(self, query: str) -> str:
|
||||
"""依据 L0 路由决策返回目标模型名。"""
|
||||
# 注意：这里我们用同步方式跑异步 router.route()
|
||||
# BaseRouter 的 _get_prediction 是同步签名；RouterArena 的 generate_prediction_file
|
||||
# 默认单线程顺序调用 8400 条，async 包装完全兼容
|
||||
import asyncio
|
||||
try:
|
||||
loop = asyncio.get_event_loop()
|
||||
if loop.is_running():
|
||||
# 已经在 async 上下文（不太可能，但兜底）
|
||||
return self._get_prediction_sync(query)
|
||||
return loop.run_until_complete(self._route_one(query))
|
||||
except RuntimeError:
|
||||
# 没有 event loop，临时建一个
|
||||
return asyncio.run(self._route_one(query))
|
||||
|
||||
async def _route_one(self, query: str) -> str:
|
||||
result = await self._router.route(query)
|
||||
return self._decide(result.domain, result.confidence, result.quality_score)
|
||||
|
||||
def _get_prediction_sync(self, query: str) -> str:
|
||||
"""event loop 已运行时的兜底（按当前 router 的同步视图返回默认值）。"""
|
||||
# 我们没有同步入口，但 generate_prediction_file 是顺序同步调用 _get_prediction，
|
||||
# 不会与 async 上下文冲突，所以此分支极少触发；保守返回 generalist-fast
|
||||
return DOMAIN_TO_MODEL_SLOT["general"]
|
||||
|
||||
def _decide(self, domain: str, confidence: float, quality_score: float) -> str:
|
||||
"""路由决策：按阈值升档。"""
|
||||
if domain not in DOMAIN_TO_MODEL_SLOT:
|
||||
domain = "general"
|
||||
# 兜底升级
|
||||
if confidence < LOW_CONFIDENCE_THRESHOLD or quality_score < JUDGE_FALLBACK_THRESHOLD:
|
||||
return ESCALATION_MODEL_SLOT
|
||||
return DOMAIN_TO_MODEL_SLOT[domain]
|
||||
|
||||
def diagnostics(self, query: str) -> Dict[str, Any]:
|
||||
"""返回完整路由诊断（用于科研报告，不影响 RouterArena 协议）。"""
|
||||
import asyncio
|
||||
return asyncio.run(self._diagnose_one(query))
|
||||
|
||||
async def _diagnose_one(self, query: str) -> Dict[str, Any]:
|
||||
result = await self._router.route(query)
|
||||
return {
|
||||
"query": query,
|
||||
"domain": result.domain,
|
||||
"subdomain": result.subdomain,
|
||||
"subdomain2": result.subdomain2,
|
||||
"difficulty": result.difficulty,
|
||||
"confidence": result.confidence,
|
||||
"quality_score": result.quality_score,
|
||||
"upgraded": result.upgraded,
|
||||
"model_used": result.model_used,
|
||||
"route": result.route,
|
||||
"selected_slot": self._decide(result.domain, result.confidence, result.quality_score),
|
||||
}
|
||||
@@ -0,0 +1,82 @@
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the RouterArena project
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
#
|
||||
# 本文件 vendored 自 RouteWorks/RouterArena（https://github.com/RouteWorks/RouterArena，
|
||||
# 2026-08-19 拉取，commit on main），仅保留本项目适配所需最小接口定义。
|
||||
# 原始完整代码与本项目无关；如 RouterArena 接口变化，需同步更新本文件。
|
||||
|
||||
"""RouterArena BaseRouter 最小 vendored 实现（适配本系统需要）。
|
||||
|
||||
仅保留以下能力：
|
||||
- 配置加载与模型列表提取
|
||||
- 模型名校验
|
||||
- 抽象方法 _get_prediction
|
||||
|
||||
去掉了原项目对 generate_prediction_file 路径的硬编码（用 ConfigResolver 解耦）。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
from abc import ABC, abstractmethod
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
|
||||
class BaseRouter(ABC):
|
||||
"""Abstract base class for router implementations.
|
||||
|
||||
子类必须实现 _get_prediction(query) -> str，返回 config.models 中存在的模型名。
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
router_name: str,
|
||||
config_path: Optional[str] = None,
|
||||
):
|
||||
self.router_name = router_name
|
||||
# 允许外部注入 config_path，避免 RouterArena 仓库的硬编码路径依赖
|
||||
if config_path is None:
|
||||
config_path = self._default_config_path()
|
||||
self.config_path = config_path
|
||||
self.config = self._load_config()
|
||||
self.models = self._extract_models()
|
||||
|
||||
def _default_config_path(self) -> str:
|
||||
# 默认查找约定：<project>/router_inference/config/<router_name>.json
|
||||
# 优先尝试项目内 research/routerarena/config/，再退到 RouterArena 约定路径
|
||||
here = os.path.dirname(os.path.abspath(__file__))
|
||||
candidate = os.path.join(here, "config", f"{self.router_name}.json")
|
||||
if os.path.exists(candidate):
|
||||
return candidate
|
||||
return candidate # 不存在时让 _load_config 抛 FileNotFoundError，给出明确路径
|
||||
|
||||
def _load_config(self) -> Dict[str, Any]:
|
||||
if not os.path.exists(self.config_path):
|
||||
raise FileNotFoundError(f"Config file not found: {self.config_path}")
|
||||
with open(self.config_path, "r", encoding="utf-8") as f:
|
||||
config = json.load(f)
|
||||
if "pipeline_params" not in config:
|
||||
raise ValueError(f"Invalid config: missing 'pipeline_params' in {self.config_path}")
|
||||
if "models" not in config["pipeline_params"]:
|
||||
raise ValueError(f"Invalid config: missing 'models' in pipeline_params")
|
||||
return config
|
||||
|
||||
def _extract_models(self) -> List[str]:
|
||||
return list(self.config["pipeline_params"]["models"])
|
||||
|
||||
def _validate_model(self, model_name: str) -> None:
|
||||
if model_name not in self.models:
|
||||
raise ValueError(
|
||||
f"Model '{model_name}' not in router config. "
|
||||
f"Available: {self.models}"
|
||||
)
|
||||
|
||||
@abstractmethod
|
||||
def _get_prediction(self, query: str) -> str:
|
||||
"""根据 query 返回 config.models 中存在的目标模型名。"""
|
||||
raise NotImplementedError
|
||||
|
||||
def get_prediction(self, query: str) -> str:
|
||||
model_name = self._get_prediction(query)
|
||||
self._validate_model(model_name)
|
||||
return model_name
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"pipeline_params": {
|
||||
"router_name": "es-expert",
|
||||
"router_cls_name": "ESExpertRouter",
|
||||
"models": [
|
||||
"gpt-4o-mini",
|
||||
"claude-3-haiku-20240307",
|
||||
"gemini-2.0-flash-001",
|
||||
"deepseek-chat",
|
||||
"mistral-medium"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,410 @@
|
||||
"""本地 Runner：不依赖 RouterArena 完整仓库，验证接入方法学。
|
||||
|
||||
职责：
|
||||
1. 加载数据集（mock 或真实 sub_10，真实数据需从 HF 拉取）
|
||||
2. 跑 adapter.get_prediction 拿到 (global_index, prompt, prediction)
|
||||
3. 写入 RouterArena 协议预测文件
|
||||
4. 提供 mock 推理 + Arena Score 计算，验证方法学
|
||||
5. 输出路由分布报告与排行榜基线对比占位
|
||||
|
||||
不依赖：API key、RouterArena 仓库
|
||||
仅依赖：Python 标准库 + 本项目 router_system
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
# 项目根加入 path
|
||||
_PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent
|
||||
if str(_PROJECT_ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(_PROJECT_ROOT))
|
||||
|
||||
|
||||
# --- 模型价格（USD / 1M tokens）---
|
||||
# 依据 RouterArena model_cost/model_cost.json 公开快照（2026-07）
|
||||
# 这里给的是 input+output 平均近似；精确值在官方文件
|
||||
MODEL_PRICING: Dict[str, Dict[str, float]] = {
|
||||
"gpt-4o-mini": {"input": 0.15, "output": 0.60},
|
||||
"claude-3-haiku-20240307": {"input": 0.25, "output": 1.25},
|
||||
"gemini-2.0-flash-001": {"input": 0.075, "output": 0.30},
|
||||
"deepseek-chat": {"input": 0.14, "output": 0.28},
|
||||
"mistral-medium": {"input": 2.7, "output": 8.1},
|
||||
}
|
||||
|
||||
# 排行榜公开基线（2026-07 快照，来源 RouterArena README L40-66）
|
||||
LEADERBOARD_BASELINES: Dict[str, Dict[str, float]] = {
|
||||
"Cross-Router": {"arena_score": 75.75, "accuracy": 78.14, "cost_per_1k": 0.40},
|
||||
"Hybrid Router": {"arena_score": 72.08, "accuracy": 71.38, "cost_per_1k": 0.04},
|
||||
"R2-Router": {"arena_score": 71.60, "accuracy": 71.23, "cost_per_1k": 0.06},
|
||||
"GPT-5": {"arena_score": 64.32, "accuracy": 73.96, "cost_per_1k": 10.02},
|
||||
"MIRT-BERT": {"arena_score": 66.89, "accuracy": 66.88, "cost_per_1k": 0.15},
|
||||
"NotDiamond": {"arena_score": 57.29, "accuracy": 60.83, "cost_per_1k": 4.10},
|
||||
"RouteLLM": {"arena_score": 48.07, "accuracy": 47.04, "cost_per_1k": 0.27},
|
||||
"RouterDC": {"arena_score": 33.75, "accuracy": 32.01, "cost_per_1k": 0.07},
|
||||
}
|
||||
|
||||
|
||||
# --- Arena Score 公式（依据 RouterArena llm_evaluation/run.py L65-87）---
|
||||
def compute_arena_score(
|
||||
cost_per_1k: float,
|
||||
accuracy: float,
|
||||
beta: float = 0.1,
|
||||
c_max: float = 200.0,
|
||||
c_min: float = 0.0044,
|
||||
) -> float:
|
||||
if cost_per_1k is None or cost_per_1k <= 0:
|
||||
raise ValueError("cost_per_1k must be positive")
|
||||
if accuracy is None or not (0 <= accuracy <= 1):
|
||||
raise ValueError("accuracy must be in [0, 1]")
|
||||
cost_clamped = max(c_min, min(cost_per_1k, c_max))
|
||||
C = (math.log2(c_max) - math.log2(cost_clamped)) / (math.log2(c_max) - math.log2(c_min))
|
||||
return ((1 + beta) * accuracy * C) / (beta * accuracy + C)
|
||||
|
||||
|
||||
# --- Mock 数据集生成 ---
|
||||
# 模仿 RouterArena sub_10 的 9 领域结构：9 domains × ~90 queries = 810 ≈ 809
|
||||
MOCK_DOMAIN_QUERIES: Dict[str, List[str]] = {
|
||||
"code": [
|
||||
"Implement quicksort in Python",
|
||||
"Write a function to reverse a linked list",
|
||||
"Debug this TypeError: undefined is not a function",
|
||||
"Optimize SQL query with index hints",
|
||||
"Implement binary search in Java",
|
||||
"用 Python 写一个快速排序函数",
|
||||
"解释这段 JavaScript 代码的 TypeError",
|
||||
"帮我优化 SQL 索引",
|
||||
"Implement merge sort",
|
||||
"Convert JSON to CSV in Python",
|
||||
],
|
||||
"math": [
|
||||
"Solve x^2 - 5x + 6 = 0",
|
||||
"Prove the Pythagorean theorem",
|
||||
"Calculate the integral of x^2 from 0 to 1",
|
||||
"求方程 x^2+3x+2=0 的根",
|
||||
"证明勾股定理",
|
||||
"计算 3x+5=20 中 x 的值",
|
||||
"Find eigenvalues of a 2x2 matrix",
|
||||
"Differentiate sin(x) * cos(x)",
|
||||
"求 ∫ x^2 dx",
|
||||
"Compute dot product of two vectors",
|
||||
],
|
||||
"legal": [
|
||||
"Is a non-compete clause for 2 years enforceable?",
|
||||
"How to calculate severance pay",
|
||||
"劳动合同到期不续签是否要给补偿金",
|
||||
"加班费怎么计算",
|
||||
"违约金上限 30% 合法吗",
|
||||
"What counts as wrongful termination",
|
||||
"Can I sue my employer for unpaid wages",
|
||||
"劳动合同里约定竞业限制是否有效",
|
||||
"Statute of limitations for breach of contract",
|
||||
"How does arbitration work in employment disputes",
|
||||
],
|
||||
"medical": [
|
||||
"What foods should hypertensive patients avoid",
|
||||
"First aid for burns",
|
||||
"高血压患者日常饮食",
|
||||
"感冒发烧 38.5 度需要吃退烧药吗",
|
||||
"Side effects of common blood pressure medications",
|
||||
"When to go to ER for chest pain",
|
||||
"烫伤后怎么处理",
|
||||
"感冒初期如何缓解症状",
|
||||
"How to treat a sprained ankle",
|
||||
"What are warning signs of diabetes",
|
||||
],
|
||||
"finance": [
|
||||
"How to calculate ROI on a fund",
|
||||
"What to do when credit card is overdue",
|
||||
"基金定投收益率怎么计算",
|
||||
"信用卡逾期怎么办",
|
||||
"房贷利率是 LPR 加多少",
|
||||
"Should I refinance my mortgage",
|
||||
"Best way to save for retirement",
|
||||
"理财产品和基金的区别",
|
||||
"How to read a stock balance sheet",
|
||||
"What is dollar-cost averaging",
|
||||
],
|
||||
"life": [
|
||||
"Travel itinerary for Japan in 7 days",
|
||||
"Beginner muscle building plan",
|
||||
"日本旅行攻略",
|
||||
"健身增肌计划",
|
||||
"家常菜推荐",
|
||||
"How to meal prep for a week",
|
||||
"Best hiking trails near San Francisco",
|
||||
"减脂餐怎么搭配",
|
||||
"How to start running for beginners",
|
||||
"室内绿植推荐",
|
||||
],
|
||||
"education": [
|
||||
"How to prepare for graduate English exam",
|
||||
"Effective study techniques",
|
||||
"考研英语怎么备考",
|
||||
"高效学习方法",
|
||||
"面试技巧有哪些",
|
||||
"How to write a research paper",
|
||||
"GRE quantitative prep strategy",
|
||||
"如何准备技术面试",
|
||||
"Best resources for learning Python",
|
||||
"时间管理方法",
|
||||
],
|
||||
"general": [
|
||||
"Why is the sky blue",
|
||||
"Explain the Transformer architecture",
|
||||
"为什么天空是蓝色的",
|
||||
"介绍 Transformer 架构",
|
||||
"Write a vacation request email",
|
||||
"What is quantum entanglement",
|
||||
"请写一封请假邮件",
|
||||
"Explain CRISPR in simple terms",
|
||||
"What is blockchain",
|
||||
"简单介绍下黑洞",
|
||||
],
|
||||
"creative": [
|
||||
"Write a haiku about autumn",
|
||||
"Suggest a name for a coffee shop",
|
||||
"Plot twist ideas for a mystery novel",
|
||||
"Write a short poem about the ocean",
|
||||
"Ideas for a 5-year-old's birthday party",
|
||||
"Story opening for a sci-fi short",
|
||||
"Suggest tagline for eco-friendly brand",
|
||||
"Lyrics for an upbeat summer song",
|
||||
"Title ideas for a romance novel",
|
||||
"Concept art description for a fantasy creature",
|
||||
],
|
||||
# 注：RouterArena 有 9 domains，creative 是第 9 类的代表（写作/创意）
|
||||
}
|
||||
|
||||
|
||||
def build_mock_dataset() -> List[Dict[str, Any]]:
|
||||
"""生成与 RouterArena sub_10 协议对齐的 mock 数据集。"""
|
||||
data: List[Dict[str, Any]] = []
|
||||
idx = 0
|
||||
for domain, queries in MOCK_DOMAIN_QUERIES.items():
|
||||
for i, q in enumerate(queries):
|
||||
data.append({
|
||||
"global index": f"mock_{idx:04d}",
|
||||
"prompt": q,
|
||||
"prompt_formatted": q,
|
||||
"domain": domain, # 仅用于本地诊断，不暴露给路由决策
|
||||
"difficulty": ["easy", "medium", "hard"][i % 3],
|
||||
})
|
||||
idx += 1
|
||||
return data
|
||||
|
||||
|
||||
# --- 加载器（支持 mock + 真实 sub_10 JSON）---
|
||||
def load_dataset(source: str, path: Optional[str] = None) -> List[Dict[str, Any]]:
|
||||
"""source ∈ {'mock', 'sub_10_file'}；path 为 None 时按 source 推断。"""
|
||||
if source == "mock":
|
||||
return build_mock_dataset()
|
||||
if source == "sub_10_file":
|
||||
if not path or not os.path.exists(path):
|
||||
raise FileNotFoundError(f"sub_10 dataset not found at {path}")
|
||||
with open(path, "r", encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
# RouterArena 协议字段：global index, prompt_formatted
|
||||
normalized = []
|
||||
for entry in data:
|
||||
normalized.append({
|
||||
"global index": entry.get("global index") or entry.get("global_index"),
|
||||
"prompt": entry.get("prompt_formatted") or entry.get("prompt"),
|
||||
"domain": None, # 真实数据无 ground truth
|
||||
"difficulty": None,
|
||||
})
|
||||
return normalized
|
||||
raise ValueError(f"Unknown source: {source}")
|
||||
|
||||
|
||||
# --- 模拟推理（不调 API）：用 L0 router 自身的 response 当 generated_answer ---
|
||||
def mock_inference(router, query: str) -> Dict[str, Any]:
|
||||
"""用本系统 L0 router 自身的 response 模拟目标 LLM 的输出。
|
||||
|
||||
注意：这只是验证"路由层 + 协议格式"正确，不替代真实 LLM 推理。
|
||||
真实评测需要 RouterArena 的 llm_inference/run.py + 目标模型 API key。
|
||||
"""
|
||||
import asyncio
|
||||
try:
|
||||
loop = asyncio.get_event_loop()
|
||||
if loop.is_running():
|
||||
# 极少见兜底
|
||||
return {"generated_answer": "[sync-fallback]", "success": True,
|
||||
"token_usage": {"output_tokens": 50}}
|
||||
result = loop.run_until_complete(router._router.route(query))
|
||||
except RuntimeError:
|
||||
result = asyncio.run(router._router.route(query))
|
||||
|
||||
answer = result.response or ""
|
||||
# 估算 token 数（中英文 1 token ≈ 1.5 字符）
|
||||
output_tokens = max(1, int(len(answer) / 1.5))
|
||||
return {
|
||||
"generated_answer": answer,
|
||||
"success": True,
|
||||
"model_used": result.model_used,
|
||||
"token_usage": {"output_tokens": output_tokens, "input_tokens": int(len(query) / 1.5)},
|
||||
}
|
||||
|
||||
|
||||
# --- 主流程 ---
|
||||
def estimate_cost(model_name: str, token_usage: Dict[str, int]) -> float:
|
||||
"""按 MODEL_PRICING 估算单条推理成本（USD）。"""
|
||||
p = MODEL_PRICING.get(model_name)
|
||||
if not p:
|
||||
return 0.0
|
||||
in_tok = token_usage.get("input_tokens", 0)
|
||||
out_tok = token_usage.get("output_tokens", 0)
|
||||
return (in_tok * p["input"] + out_tok * p["output"]) / 1_000_000
|
||||
|
||||
|
||||
def run_local(
|
||||
source: str = "mock",
|
||||
dataset_path: Optional[str] = None,
|
||||
router_name: str = "es-expert",
|
||||
config_path: Optional[str] = None,
|
||||
do_mock_inference: bool = True,
|
||||
output_dir: str = "research/routerarena/output",
|
||||
) -> Dict[str, Any]:
|
||||
"""跑本地端到端流程，输出预测文件 + 诊断报告。"""
|
||||
from .adapter import ESExpertRouter
|
||||
|
||||
if config_path is None:
|
||||
here = Path(__file__).resolve().parent
|
||||
config_path = str(here / "config" / f"{router_name}.json")
|
||||
|
||||
router = ESExpertRouter(router_name=router_name, config_path=config_path)
|
||||
dataset = load_dataset(source, dataset_path)
|
||||
print(f"[local_runner] router models = {router.models}")
|
||||
print(f"[local_runner] dataset size = {len(dataset)}")
|
||||
|
||||
# 1) 路由决策（按 RouterArena 协议）
|
||||
predictions: List[Dict[str, Any]] = []
|
||||
diagnostics: List[Dict[str, Any]] = []
|
||||
t0 = time.perf_counter()
|
||||
for entry in dataset:
|
||||
gi = entry["global index"]
|
||||
prompt = entry["prompt"]
|
||||
selected = router.get_prediction(prompt)
|
||||
# 同时记一份诊断（科研用，不影响协议）
|
||||
diag = router.diagnostics(prompt)
|
||||
diag["global_index"] = gi
|
||||
diag["ground_truth_domain"] = entry.get("domain") # 仅 mock 数据有
|
||||
diagnostics.append(diag)
|
||||
predictions.append({
|
||||
"global index": gi,
|
||||
"prompt": prompt,
|
||||
"prediction": selected,
|
||||
"generated_result": None,
|
||||
"cost": None,
|
||||
"accuracy": None,
|
||||
"for_optimality": False,
|
||||
})
|
||||
routing_latency_ms = (time.perf_counter() - t0) * 1000 / len(dataset)
|
||||
|
||||
# 2) 模拟推理（mock generated_result）
|
||||
if do_mock_inference:
|
||||
for pred, diag in zip(predictions, diagnostics):
|
||||
gen = mock_inference(router, pred["prompt"])
|
||||
pred["generated_result"] = gen
|
||||
cost = estimate_cost(pred["prediction"], gen["token_usage"])
|
||||
pred["cost"] = cost
|
||||
# mock 准确率：仅用于方法学验证（真实评测 RouterArena 用 ground truth）
|
||||
# 这里如果 router 选对了 ground truth domain → 1.0，否则按简单启发式
|
||||
if diag.get("ground_truth_domain") and diag.get("domain"):
|
||||
# 把 L0 的 8 域映射到 RouterArena 的 9 域（creative 算 general）
|
||||
gt = diag["ground_truth_domain"]
|
||||
pred_domain = diag["domain"]
|
||||
# RouterArena creative → 本系统 general 域
|
||||
gt_mapped = "general" if gt == "creative" else gt
|
||||
pred["accuracy"] = 1.0 if pred_domain == gt_mapped else 0.0
|
||||
else:
|
||||
pred["accuracy"] = None # 真实数据无 domain 标签，跳过
|
||||
|
||||
# 3) 写预测文件（RouterArena 协议）
|
||||
os.makedirs(output_dir, exist_ok=True)
|
||||
pred_path = os.path.join(output_dir, f"{router_name}.json")
|
||||
with open(pred_path, "w", encoding="utf-8") as f:
|
||||
json.dump(predictions, f, ensure_ascii=False, indent=2)
|
||||
diag_path = os.path.join(output_dir, f"{router_name}_diagnostics.json")
|
||||
with open(diag_path, "w", encoding="utf-8") as f:
|
||||
json.dump(diagnostics, f, ensure_ascii=False, indent=2)
|
||||
|
||||
# 4) 算指标
|
||||
n = len(predictions)
|
||||
acc_vals = [p["accuracy"] for p in predictions if p["accuracy"] is not None]
|
||||
cost_vals = [p["cost"] for p in predictions if p["cost"] is not None and p["cost"] > 0]
|
||||
avg_acc = sum(acc_vals) / len(acc_vals) if acc_vals else 0.0
|
||||
total_cost = sum(cost_vals) if cost_vals else 0.0
|
||||
cost_per_1k = (total_cost / n * 1000) if n > 0 else 0.0
|
||||
try:
|
||||
arena_score = compute_arena_score(cost_per_1k, avg_acc) if cost_per_1k > 0 else None
|
||||
except ValueError:
|
||||
arena_score = None
|
||||
|
||||
# 路由分布
|
||||
from collections import Counter
|
||||
routing_dist = Counter(p["prediction"] for p in predictions)
|
||||
domain_dist = Counter(d["domain"] for d in diagnostics)
|
||||
confidence_dist = {
|
||||
"min": min(d["confidence"] for d in diagnostics),
|
||||
"max": max(d["confidence"] for d in diagnostics),
|
||||
"mean": sum(d["confidence"] for d in diagnostics) / len(diagnostics),
|
||||
}
|
||||
|
||||
summary = {
|
||||
"router_name": router_name,
|
||||
"n_queries": n,
|
||||
"routing_latency_ms_per_query": routing_latency_ms,
|
||||
"domain_distribution": dict(domain_dist),
|
||||
"routing_distribution": dict(routing_dist),
|
||||
"confidence": confidence_dist,
|
||||
"mock_accuracy": avg_acc,
|
||||
"total_cost_usd": total_cost,
|
||||
"cost_per_1k_usd": cost_per_1k,
|
||||
"arena_score_mock": arena_score,
|
||||
"prediction_file": pred_path,
|
||||
"diagnostics_file": diag_path,
|
||||
}
|
||||
summary_path = os.path.join(output_dir, f"{router_name}_summary.json")
|
||||
with open(summary_path, "w", encoding="utf-8") as f:
|
||||
json.dump(summary, f, ensure_ascii=False, indent=2)
|
||||
return summary
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="RouterArena 本地接入烟测（无需 API）")
|
||||
parser.add_argument("--source", choices=["mock", "sub_10_file"], default="mock")
|
||||
parser.add_argument("--dataset-path", default=None)
|
||||
parser.add_argument("--router", default="es-expert")
|
||||
parser.add_argument("--config", default=None)
|
||||
parser.add_argument("--no-mock-inference", action="store_true",
|
||||
help="只跑路由不模拟推理（用于纯路由层验证）")
|
||||
parser.add_argument("--output-dir", default="research/routerarena/output")
|
||||
args = parser.parse_args()
|
||||
|
||||
summary = run_local(
|
||||
source=args.source,
|
||||
dataset_path=args.dataset_path,
|
||||
router_name=args.router,
|
||||
config_path=args.config,
|
||||
do_mock_inference=not args.no_mock_inference,
|
||||
output_dir=args.output_dir,
|
||||
)
|
||||
# 打印关键指标（确保终端 GBK 安全：写到文件再读）
|
||||
out_path = os.path.join(args.output_dir, f"{args.router}_summary.json")
|
||||
with open(out_path, "r", encoding="utf-8") as f:
|
||||
s = json.load(f)
|
||||
print("\n========== RouterArena Local Run Summary ==========")
|
||||
for k, v in s.items():
|
||||
print(f" {k}: {v}")
|
||||
print("====================================================")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"router_name": "es-expert",
|
||||
"n_queries": 90,
|
||||
"routing_latency_ms_per_query": 21.243365556519066,
|
||||
"domain_distribution": {
|
||||
"code": 12,
|
||||
"general": 40,
|
||||
"math": 6,
|
||||
"legal": 6,
|
||||
"medical": 8,
|
||||
"finance": 8,
|
||||
"life": 4,
|
||||
"education": 6
|
||||
},
|
||||
"routing_distribution": {
|
||||
"gpt-4o-mini": 8,
|
||||
"mistral-medium": 62,
|
||||
"claude-3-haiku-20240307": 16,
|
||||
"gemini-2.0-flash-001": 2,
|
||||
"deepseek-chat": 2
|
||||
},
|
||||
"confidence": {
|
||||
"min": 0.2592,
|
||||
"max": 0.99,
|
||||
"mean": 0.5881833333333334
|
||||
},
|
||||
"mock_accuracy": 0.7444444444444445,
|
||||
"total_cost_usd": 0.04596006,
|
||||
"cost_per_1k_usd": 0.5106673333333333,
|
||||
"arena_score_mock": 0.7223003144396156,
|
||||
"prediction_file": "research/routerarena/output\\es-expert.json",
|
||||
"diagnostics_file": "research/routerarena/output\\es-expert_diagnostics.json"
|
||||
}
|
||||
@@ -1,6 +1,6 @@
|
||||
# v2 实验目录（research/v2_experiments/）
|
||||
|
||||
端云协同 LLM 协作系统（《实现方案_v2》第 9 节）的论文数据来源。
|
||||
端云协同编程智能体系统（《实现方案_v2》第 9 节）的论文数据来源。
|
||||
|
||||
## E1 token 经济学（主实验）— 已有本地确定性结果 ✅
|
||||
|
||||
|
||||
File diff suppressed because one or more lines are too long
Binary file not shown.
File diff suppressed because one or more lines are too long
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,235 @@
|
||||
"""RouteAgent：Agent-Skill 路由器（T12：先行实现）。
|
||||
|
||||
核心思想（对齐用户架构决策）：
|
||||
- 用户只提供需求，不需要指定领域/模型/技能
|
||||
- Agent 自行分析需求（两级路由：自动组检测 → 领域/难度/三级子领域）
|
||||
- 规划 skill 调用计划（复用 Planner 任务模板 → 每个子任务映射到技能）
|
||||
- 按拓扑序执行技能调用，黑板协作，合并输出
|
||||
- 质量校验（judge skill）→ 不达标升级（fallback skill）
|
||||
|
||||
与 Router 的关系：Router.route() 是"编排管线"，RouteAgent.route() 是
|
||||
"技能调用式"同构实现——执行阶段通过 SkillRegistry 按技能名调用，
|
||||
推理链轨迹记录每次 skill 调用（可解释性）。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import uuid
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
from .classifier import RuleClassifier
|
||||
from .executors import NodeExecutor
|
||||
from .fallback import FallbackProvider
|
||||
from .inference import InferenceEngine
|
||||
from .judge import BaseJudge
|
||||
from .knowledge import KnowledgeBase
|
||||
from .memory import TaskGraph, WorkingMemory
|
||||
from .models import Classification, ExpertResponse, RouterResult, now_ms
|
||||
from .planner import Planner
|
||||
from .skills import SkillContext, SkillRegistry, build_skill_registry
|
||||
from .trace import TraceStore
|
||||
|
||||
# 子任务 kind → 技能名映射（retrieve 走知识库检索，其余走模板技能）
|
||||
_KIND_SKILL = {
|
||||
"analyze": "es.analyze", "design": "es.design", "implement": "es.implement",
|
||||
"solve": "es.solve", "diagnose": "es.diagnose", "fix": "es.fix",
|
||||
"retrieve": "kb.retrieve", "conclude": "es.conclude", "advise": "es.advise",
|
||||
"explain": "es.explain", "disclaimer": "es.disclaimer", "verify": "es.verify",
|
||||
"refactor": "es.refactor", "testcase": "es.testcase",
|
||||
"complexity": "es.complexity", "optimize": "es.optimize",
|
||||
"draft": "es.draft", "polish": "es.polish",
|
||||
}
|
||||
|
||||
|
||||
class RouteAgent:
|
||||
"""Agent-Skill 路由器：需求分析 → 技能规划 → 技能执行 → 校验升级。"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
classifier: RuleClassifier,
|
||||
planner: Planner,
|
||||
kb: KnowledgeBase,
|
||||
judge: BaseJudge,
|
||||
fallback: FallbackProvider,
|
||||
node_executor: Optional[NodeExecutor] = None,
|
||||
registry: Optional[SkillRegistry] = None,
|
||||
low_confidence_threshold: float = 0.60,
|
||||
judge_fallback_threshold: float = 0.70,
|
||||
):
|
||||
self.classifier = classifier
|
||||
self.planner = planner
|
||||
self.kb = kb
|
||||
self.judge = judge
|
||||
self.fallback = fallback
|
||||
self.node_executor = node_executor
|
||||
self.inference = InferenceEngine(kb)
|
||||
self.low_confidence_threshold = low_confidence_threshold
|
||||
self.judge_fallback_threshold = judge_fallback_threshold
|
||||
self.registry = registry or build_skill_registry(
|
||||
kb=kb, judge=judge, fallback=fallback,
|
||||
fallback_threshold=judge_fallback_threshold,
|
||||
)
|
||||
self.trace_store = TraceStore()
|
||||
|
||||
# ---------------------------------------------------------------
|
||||
async def route(self, query: str) -> RouterResult:
|
||||
start = now_ms()
|
||||
route: list = []
|
||||
request_id = uuid.uuid4().hex[:12]
|
||||
|
||||
# ---- Step 1: 需求分析（Agent 自行分析，无需用户指定） ----
|
||||
classification = self.classifier.classify(query)
|
||||
route.append(f"classify:{classification.domain}@{classification.confidence:.2f}/{classification.difficulty}")
|
||||
subdomain, subdomain2 = self._detect_subdomain(query, classification.domain)
|
||||
if subdomain:
|
||||
route.append(f"subdomain:{subdomain}")
|
||||
if subdomain2:
|
||||
route.append(f"subdomain2:{subdomain2}")
|
||||
|
||||
# ---- Step 2: 低置信 → fallback 技能（Agent 自主兜底） ----
|
||||
if classification.confidence < self.low_confidence_threshold:
|
||||
route.append("direct_fallback")
|
||||
resp = await self.registry.execute("fallback.call", SkillContext(
|
||||
query=query, domain=classification.domain,
|
||||
difficulty=classification.difficulty, memory=WorkingMemory(), kb=self.kb))
|
||||
latency = now_ms() - start
|
||||
result = RouterResult(
|
||||
query=query, response=resp, domain=classification.domain,
|
||||
difficulty=classification.difficulty,
|
||||
confidence=classification.confidence, upgraded=True,
|
||||
quality_score=0.0, model_used=self.fallback.name,
|
||||
route=route, latency_ms=latency, cost_est=0.0,
|
||||
subdomain=subdomain, subdomain2=subdomain2, request_id=request_id,
|
||||
)
|
||||
self._store_trace(result, route, request_id, query, latency)
|
||||
return result
|
||||
|
||||
# ---- Step 3: 技能规划（Planner 任务模板 → skill 调用计划） ----
|
||||
graph: TaskGraph = self.planner.plan(query, classification)
|
||||
route.extend(self.planner.explain_plan(graph))
|
||||
|
||||
# ---- Step 4: 黑板初始化 + 前向链 ----
|
||||
memory = WorkingMemory()
|
||||
self.inference.initialize(
|
||||
query, classification.domain, classification.difficulty,
|
||||
classification.confidence, memory)
|
||||
fired = self.inference.run(query, classification.domain, memory)
|
||||
if fired:
|
||||
route.append(f"rules:{','.join(fired[:5])}")
|
||||
|
||||
# ---- Step 5: 按拓扑序执行技能调用 ----
|
||||
order = graph.topo_order()
|
||||
last_model = f"rule:{classification.domain}"
|
||||
for node in order:
|
||||
model = await self._execute_skill(node, classification, memory, route)
|
||||
if model:
|
||||
last_model = model
|
||||
|
||||
# ---- Step 6: 合并 + 质量校验（judge 技能） ----
|
||||
response = memory.merge([n.id for n in order])
|
||||
node_ids = {n.id for n in order}
|
||||
extras = [memory.section(s) for s in memory.sections if s not in node_ids and memory.section(s)]
|
||||
if extras:
|
||||
response = (response + "\n\n" + "\n\n".join(extras)) if response.strip() else "\n\n".join(extras)
|
||||
if not response.strip():
|
||||
response = "（RouteAgent）未能生成有效回答。"
|
||||
route.append("merge:empty")
|
||||
|
||||
try:
|
||||
evaluation = await self.judge.evaluate(query, response, classification.domain)
|
||||
except Exception:
|
||||
evaluation = None
|
||||
route.append("judge_error")
|
||||
quality_score = evaluation.overall_score if evaluation else 0.0
|
||||
route.append(f"judge:{quality_score:.2f}")
|
||||
|
||||
upgraded = False
|
||||
if evaluation is not None and evaluation.needs_fallback:
|
||||
route.append("upgrade")
|
||||
response = await self.registry.execute("fallback.call", SkillContext(
|
||||
query=query, domain=classification.domain,
|
||||
difficulty=classification.difficulty, memory=memory, kb=self.kb))
|
||||
last_model = self.fallback.name
|
||||
upgraded = True
|
||||
|
||||
latency = now_ms() - start
|
||||
result = RouterResult(
|
||||
query=query, response=response, domain=classification.domain,
|
||||
difficulty=classification.difficulty,
|
||||
confidence=classification.confidence, upgraded=upgraded,
|
||||
quality_score=quality_score, model_used=last_model,
|
||||
route=route, latency_ms=latency, cost_est=0.0,
|
||||
subdomain=subdomain, subdomain2=subdomain2, request_id=request_id,
|
||||
)
|
||||
self._store_trace(result, route, request_id, query, latency)
|
||||
return result
|
||||
|
||||
# ---------------------------------------------------------------
|
||||
async def _execute_skill(self, node, classification: Classification,
|
||||
memory: WorkingMemory, route: list) -> Optional[str]:
|
||||
"""按节点 kind 调用技能；返回 model_used（失败 None）。"""
|
||||
for dep_id in node.deps:
|
||||
pass # 拓扑序已保证依赖先行；状态由节点自身管理
|
||||
node.status = "running"
|
||||
skill_name = _KIND_SKILL.get(node.kind, f"es.{node.kind}")
|
||||
try:
|
||||
if self.node_executor is not None and node.kind not in ("retrieve",):
|
||||
# L2 模式：NodeExecutor 后端（组内小模型）执行
|
||||
resp = await self.node_executor.execute(
|
||||
node, classification.domain, classification.difficulty, memory)
|
||||
text = resp.text
|
||||
model = resp.model_used
|
||||
else:
|
||||
ctx = SkillContext(
|
||||
query=node.query, domain=node.domain or classification.domain,
|
||||
difficulty=classification.difficulty, memory=memory, kb=self.kb,
|
||||
)
|
||||
text = await self.registry.execute(skill_name, ctx)
|
||||
model = skill_name
|
||||
node.output = text
|
||||
node.status = "done"
|
||||
memory.write_section(node.id, text)
|
||||
route.append(f"skill:{skill_name}@{node.id}")
|
||||
return model
|
||||
except Exception as e:
|
||||
node.status = "failed"
|
||||
node.error = str(e)
|
||||
route.append(f"skill:{skill_name}@{node.id}:error:{type(e).__name__}")
|
||||
return None
|
||||
|
||||
# ---------------------------------------------------------------
|
||||
def _detect_subdomain(self, query: str, domain: str) -> tuple:
|
||||
hits = self.kb.match(query, domain=domain)
|
||||
sub = sub2 = None
|
||||
for h in hits:
|
||||
if sub is None and h.subdomain:
|
||||
sub = h.subdomain
|
||||
if sub2 is None and h.subdomain2:
|
||||
sub2 = h.subdomain2
|
||||
if sub is not None and sub2 is not None:
|
||||
break
|
||||
return sub, sub2
|
||||
|
||||
def _store_trace(self, result: RouterResult, route: list, request_id: str,
|
||||
query: str, latency: float) -> None:
|
||||
self.trace_store.put(request_id, {
|
||||
"request_id": request_id,
|
||||
"query": query,
|
||||
"domain_group": None, # Agent 模式：无用户指定，完全自主
|
||||
"domain": result.domain,
|
||||
"difficulty": result.difficulty,
|
||||
"confidence": result.confidence,
|
||||
"subdomain": result.subdomain,
|
||||
"subdomain2": result.subdomain2,
|
||||
"route": list(route),
|
||||
"quality_score": result.quality_score,
|
||||
"upgraded": result.upgraded,
|
||||
"model_used": result.model_used,
|
||||
"latency_ms": round(latency, 2),
|
||||
"cache_hit": False,
|
||||
"cache_level": None,
|
||||
})
|
||||
|
||||
# ---------------------------------------------------------------
|
||||
def skills_catalog(self) -> list:
|
||||
"""暴露技能目录（Agent 能力清单）。"""
|
||||
return self.registry.catalog()
|
||||
@@ -0,0 +1,370 @@
|
||||
"""执行器体系（L0 默认专家 + NodeExecutor 后端抽象）。
|
||||
|
||||
设计对齐"专家系统风格"（《可行性调研与落地实现路线报告》第八章）：
|
||||
- 输出 = 结构化模板填充（回显查询、知识库事实、领域结构），不追求自然语言流畅度
|
||||
- 确定性：同输入 → 同输出（无采样随机）
|
||||
- 最小参数：零模型参数；L2 模式下同一节点可改由本地小模型执行（Router 按配置切换）
|
||||
|
||||
kind（子任务动作类型）与模板对应：
|
||||
analyze 需求/条件分析 | design 方案设计 | implement 代码实现 | solve 数学求解
|
||||
diagnose 错误定位 | fix 修复方案 | retrieve 知识检索 | conclude 结论
|
||||
advise 一般建议 | explain 展开解释 | disclaimer 免责/警示 | verify 自检
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
from .experts import Expert, extract_content_terms
|
||||
from .knowledge import KnowledgeBase
|
||||
from .memory import TaskNode, WorkingMemory
|
||||
from .models import ExpertResponse
|
||||
|
||||
# 各领域"分析"步骤的目标描述
|
||||
_GOALS = {
|
||||
"code": "输出可运行的代码实现",
|
||||
"math": "得到问题的解并给出推导",
|
||||
"legal": "给出法律结论与依据",
|
||||
"medical": "给出科普性建议",
|
||||
"finance": "给出理财/金融建议与风险提示",
|
||||
"life": "给出实用生活建议",
|
||||
"education": "给出学习/行动方案",
|
||||
"general": "给出结构化说明",
|
||||
}
|
||||
|
||||
# 各领域"约束/边界"提示
|
||||
_CONSTRAINTS = {
|
||||
"code": "边界条件（空输入、极端值）；复杂度目标",
|
||||
"math": "定义域、无解/多解情况、特殊值",
|
||||
"legal": "以现行有效法律为准，个案需咨询律师",
|
||||
"medical": "个体差异；非诊断，请遵医嘱",
|
||||
"finance": "市场有风险，投资需谨慎；不构成投资建议",
|
||||
"life": "结合个人实际情况，安全第一",
|
||||
"education": "结合个人基础与目标，循序渐进",
|
||||
"general": "围绕核心问题，避免无关展开",
|
||||
}
|
||||
|
||||
# 各领域"验证"清单
|
||||
_VERIFY_CHECKS = {
|
||||
"code": ["输入输出覆盖", "边界条件", "复杂度合理", "可运行性"],
|
||||
"math": ["中间步骤正确", "结果代入验证", "边界/特殊值", "单位与符号"],
|
||||
"legal": ["法条依据充分", "事实对应", "免责提示", "结论可执行"],
|
||||
"medical": ["建议有依据", "警示信号明确", "免责提示", "不构成诊断"],
|
||||
"finance": ["风险提示完整", "数据/规则准确", "免责提示", "建议可执行"],
|
||||
"life": ["建议实用", "安全提示", "贴合场景"],
|
||||
"education": ["方案可执行", "目标可衡量", "符合个人基础"],
|
||||
"general": ["要点覆盖", "逻辑连贯", "无事实错误"],
|
||||
}
|
||||
|
||||
|
||||
def _kw(query: str, n: int = 6) -> str:
|
||||
terms = extract_content_terms(query)
|
||||
return "、".join(terms[:n]) if terms else "该主题"
|
||||
|
||||
|
||||
class RuleExecutor(Expert):
|
||||
"""规则执行器：实现 Expert 接口；L0 模式的默认领域执行器。"""
|
||||
|
||||
name = "rule-executor"
|
||||
|
||||
def __init__(self, name: str = "rule-executor", domain: str = "general",
|
||||
kb: Optional[KnowledgeBase] = None):
|
||||
self.name = name
|
||||
self.domain = domain
|
||||
self.kb = kb
|
||||
|
||||
async def generate(self, query: str, difficulty: str,
|
||||
memory: Optional[WorkingMemory] = None,
|
||||
node: Optional[TaskNode] = None) -> ExpertResponse:
|
||||
"""按节点 kind 生成确定性输出。兼容 Expert 基类签名（后两参可选）。"""
|
||||
kind = node.kind if node is not None else "explain"
|
||||
domain = node.domain if node is not None else self.domain
|
||||
text = self._template(kind, domain, query, difficulty, memory)
|
||||
tokens = max(8, int(len(text) / 2.2))
|
||||
return ExpertResponse(
|
||||
text=text,
|
||||
model_used=f"rule:{domain}:{kind}",
|
||||
latency_ms=0.0,
|
||||
tokens=tokens,
|
||||
cost_est=0.0, # 零参数执行器无推理成本
|
||||
)
|
||||
|
||||
# ---------------------------------------------------------------
|
||||
def _template(self, kind: str, domain: str, query: str, difficulty: str,
|
||||
memory: Optional[WorkingMemory]) -> str:
|
||||
facts: Dict[str, Any] = memory.facts if memory else {}
|
||||
goal = _GOALS.get(domain, _GOALS["general"])
|
||||
constraints = _CONSTRAINTS.get(domain, _CONSTRAINTS["general"])
|
||||
kw = _kw(query)
|
||||
|
||||
if kind == "analyze":
|
||||
return (
|
||||
f"【{domain} 分析】\n"
|
||||
f"- 任务：{query}\n"
|
||||
f"- 关键要素：{kw}\n"
|
||||
f"- 目标：{goal}\n"
|
||||
f"- 约束/边界：{constraints}\n"
|
||||
f"- 难度评估：{difficulty}"
|
||||
)
|
||||
if kind == "design":
|
||||
return (
|
||||
f"【{domain} 方案设计】\n"
|
||||
f"针对「{query}」的设计思路：\n"
|
||||
f"1. 明确核心目标与验收标准\n"
|
||||
f"2. 选择合适的方法/数据结构（依据：{kw}）\n"
|
||||
f"3. 拆解实现步骤并标注复杂度\n"
|
||||
f"4. 预留边界处理与异常路径\n"
|
||||
f"5. 设计自测用例（正常/边界/异常）"
|
||||
)
|
||||
if kind == "implement":
|
||||
return (
|
||||
f"【{domain} 实现】\n"
|
||||
f"```python\n"
|
||||
f"def solve() -> None:\n"
|
||||
f" # 关键点：{kw}\n"
|
||||
f" # 1. 校验输入与边界条件\n"
|
||||
f" # 2. 核心逻辑（依据 design 步骤）\n"
|
||||
f" # 3. 输出结果\n"
|
||||
f" pass\n"
|
||||
f"```\n"
|
||||
f"要点：{kw}；复杂度与边界说明见 design/verify 步骤。"
|
||||
)
|
||||
if kind == "solve":
|
||||
return (
|
||||
f"【{domain} 求解】\n"
|
||||
f"题目：{query}\n"
|
||||
f"步骤：\n"
|
||||
f"1. 提取已知条件（{kw}）\n"
|
||||
f"2. 选择方法：代数变形/公式代入/逐步推导\n"
|
||||
f"3. 求解并化简中间结果\n"
|
||||
f"4. 检查特殊值与边界\n"
|
||||
f"结论：在标准假设下可得到闭合形式解；完整推导见正式解答。"
|
||||
)
|
||||
if kind == "diagnose":
|
||||
return (
|
||||
f"【{domain} 诊断】\n"
|
||||
f"错误现象：{query}\n"
|
||||
f"排查步骤：\n"
|
||||
f"1. 复现并定位出错行\n"
|
||||
f"2. 检查变量类型与取值（重点：{kw}）\n"
|
||||
f"3. 核对函数签名、作用域与返回值\n"
|
||||
f"4. 打印中间变量验证假设\n"
|
||||
f"5. 用最小样例隔离问题"
|
||||
)
|
||||
if kind == "fix":
|
||||
return (
|
||||
f"【{domain} 修复方案】\n"
|
||||
f"针对「{query}」：\n"
|
||||
f"1. 根因：见 diagnose 步骤\n"
|
||||
f"2. 修复：调整类型/增加空值判断/修正逻辑分支\n"
|
||||
f"```python\n"
|
||||
f"def fixed() -> None:\n"
|
||||
f" # 修复点：{kw}\n"
|
||||
f" pass\n"
|
||||
f"```\n"
|
||||
f"3. 回归：补充对应单测后重跑"
|
||||
)
|
||||
if kind == "retrieve":
|
||||
return self._retrieve(domain, query, memory)
|
||||
if kind == "conclude":
|
||||
return (
|
||||
f"【{domain} 结论】\n"
|
||||
f"综合「{query}」：\n"
|
||||
f"1. 事实梳理：{kw}\n"
|
||||
f"2. 适用规则/依据（见 retrieve 步骤）\n"
|
||||
f"3. 结论：在所述前提下，按上述规则处理\n"
|
||||
f"4. 注意事项：个案差异，必要时咨询专业人士"
|
||||
)
|
||||
if kind == "advise":
|
||||
return (
|
||||
f"【{domain} 建议】\n"
|
||||
f"关于「{query}」的一般性建议：\n"
|
||||
f"1. 基础注意事项（{kw}）\n"
|
||||
f"2. 可操作建议：分步执行并观察效果\n"
|
||||
f"3. 警示信号：出现下列情况应及时就医（见 warning 步骤）"
|
||||
)
|
||||
if kind == "explain":
|
||||
if domain == "code":
|
||||
return (
|
||||
f"【code 代码讲解】\n"
|
||||
f"代码/片段：{query}\n"
|
||||
f"讲解结构：\n"
|
||||
f"1. 整体目的：这段代码要解决什么问题（{kw}）\n"
|
||||
f"2. 执行流程：按行/按函数梳理数据流与调用链\n"
|
||||
f"3. 关键点：数据结构、边界处理、异常路径\n"
|
||||
f"4. 可改进点：命名/复杂度/可读性建议"
|
||||
)
|
||||
return (
|
||||
f"【{domain} 说明】\n"
|
||||
f"主题：{query}\n"
|
||||
f"1. 背景与定义\n"
|
||||
f"2. 核心要点：{kw}\n"
|
||||
f"3. 分类/维度/机制\n"
|
||||
f"4. 实际应用与注意事项\n"
|
||||
f"如需更深入分析，可补充上下文。"
|
||||
)
|
||||
if kind == "disclaimer":
|
||||
if domain == "legal":
|
||||
return (
|
||||
"⚠️ 提示：以上为一般性法律分析，不构成正式法律意见；"
|
||||
"个案请咨询执业律师。"
|
||||
)
|
||||
if domain == "medical":
|
||||
return (
|
||||
"⚠️ 提示：以上内容仅供健康科普，不能替代医生诊断；"
|
||||
"如有不适请及时就医。"
|
||||
)
|
||||
if domain == "finance":
|
||||
return (
|
||||
"⚠️ 提示：以上为一般性金融科普，不构成投资建议；"
|
||||
"投资有风险，决策前请结合自身情况并咨询专业人士。"
|
||||
)
|
||||
return ""
|
||||
if kind == "verify":
|
||||
checks = _VERIFY_CHECKS.get(domain, _VERIFY_CHECKS["general"])
|
||||
items = "\n".join(f"- {c}" for c in checks)
|
||||
return f"【{domain} 自检】\n{items}"
|
||||
if kind == "refactor":
|
||||
return (
|
||||
f"【code 重构方案】\n"
|
||||
f"针对「{query}」：\n"
|
||||
f"1. 现状问题：重复代码/长函数/命名不清/耦合（{kw}）\n"
|
||||
f"2. 重构手法：提取函数、消除魔法数字、引入类或模块、统一命名\n"
|
||||
f"3. 目标结构：单一职责、清晰分层、可测试性\n"
|
||||
f"4. 验证：重构前后行为等价（跑通全部测试）"
|
||||
)
|
||||
if kind == "testcase":
|
||||
return (
|
||||
f"【code 测试用例】\n"
|
||||
f"针对「{query}」设计测试：\n"
|
||||
f"```python\n"
|
||||
f"def test_xxx():\n"
|
||||
f" # 正常路径：{kw}\n"
|
||||
f" pass\n\n"
|
||||
f"def test_edge():\n"
|
||||
f" # 边界：空输入/极值/None\n"
|
||||
f" pass\n\n"
|
||||
f"def test_error():\n"
|
||||
f" # 异常路径：非法参数\n"
|
||||
f" pass\n"
|
||||
f"```\n"
|
||||
f"覆盖策略：正常 + 边界 + 异常三组，断言明确"
|
||||
)
|
||||
if kind == "complexity":
|
||||
return (
|
||||
f"【code 复杂度分析】\n"
|
||||
f"针对「{query}」：\n"
|
||||
f"1. 时间复杂度：核心循环/递归层数 → 平均与最坏情况（{kw}）\n"
|
||||
f"2. 空间复杂度：辅助数据结构占用\n"
|
||||
f"3. 优化建议：若可接受，给出降复杂度的替代思路"
|
||||
)
|
||||
if kind == "optimize":
|
||||
return (
|
||||
f"【math 最优化求解】\n"
|
||||
f"问题：{query}\n"
|
||||
f"步骤：\n"
|
||||
f"1. 建立目标函数与约束（{kw}）\n"
|
||||
f"2. 求导/配方/不等式法找候选极值点\n"
|
||||
f"3. 比较候选值并与边界比较\n"
|
||||
f"4. 结论：给出最大值/最小值及取到条件"
|
||||
)
|
||||
if kind == "draft":
|
||||
return (
|
||||
f"【写作初稿】\n"
|
||||
f"主题：{query}\n"
|
||||
f"结构：\n"
|
||||
f"1. 开头：点明主题与背景（{kw}）\n"
|
||||
f"2. 主体：分点展开，每点配一个例子或依据\n"
|
||||
f"3. 结尾：总结观点 + 行动建议\n"
|
||||
f"（初稿完成，待 polish 步骤润色）"
|
||||
)
|
||||
if kind == "polish":
|
||||
return (
|
||||
f"【写作润色】\n"
|
||||
f"基于初稿检查：\n"
|
||||
f"1. 语法与错别字\n"
|
||||
f"2. 逻辑衔接与段落过渡\n"
|
||||
f"3. 语气统一（正式/亲切）与受众匹配\n"
|
||||
f"4. 长度控制与重点突出（{kw}）"
|
||||
)
|
||||
# 未知 kind 兜底
|
||||
return f"（规则执行器）「{query}」：{kw}"
|
||||
|
||||
# ---------------------------------------------------------------
|
||||
def _retrieve(self, domain: str, query: str,
|
||||
memory: Optional[WorkingMemory]) -> str:
|
||||
"""知识检索：从知识库事实表取命中的条目；无命中则给出查阅建议。"""
|
||||
if self.kb is None:
|
||||
return (
|
||||
f"【{domain} 知识检索】\n"
|
||||
f"未配置知识库，建议查阅权威资料（{_kw(query)}）。"
|
||||
)
|
||||
facts = self.kb.facts(domain)
|
||||
hits = [f for f in facts if any(k in query for k in f.get("keywords", []))]
|
||||
if hits:
|
||||
lines = [f"- {f['statement']}" for f in hits]
|
||||
return f"【{domain} 知识检索】\n" + "\n".join(lines)
|
||||
return (
|
||||
f"【{domain} 知识检索】\n"
|
||||
f"未命中知识库条目；建议以现行有效法规/最新指南为准，"
|
||||
f"并结合个案情况分析（{_kw(query)}）。"
|
||||
)
|
||||
|
||||
|
||||
# ===============================================================
|
||||
# NodeExecutor：子任务执行后端抽象（T1：整体项目部分拆解·先行实现）
|
||||
#
|
||||
# Router._execute_node 不再内联 if-else 分支，而是依赖 NodeExecutor 接口：
|
||||
# - RuleNodeExecutor ：L0 规则执行器（零参数、确定性）
|
||||
# - ModelNodeExecutor：L2 专家池小模型（≤8B，按需加载）
|
||||
# - 未来可加：多路采样执行器、API 执行器、组内模型执行器……
|
||||
# 工厂按配置选择后端，新增后端无需改动 Router。
|
||||
# ===============================================================
|
||||
|
||||
|
||||
class NodeExecutor:
|
||||
"""子任务执行后端抽象接口。"""
|
||||
|
||||
name: str = "node-executor"
|
||||
|
||||
async def execute(self, node: TaskNode, domain: str, difficulty: str,
|
||||
memory: WorkingMemory) -> ExpertResponse:
|
||||
raise NotImplementedError
|
||||
|
||||
|
||||
class RuleNodeExecutor(NodeExecutor):
|
||||
"""L0：规则执行器后端（零参数、确定性、零成本）。"""
|
||||
|
||||
name = "rule"
|
||||
|
||||
def __init__(self, kb: Optional[KnowledgeBase] = None):
|
||||
self._rule = RuleExecutor("rule-executor", "general", kb=kb)
|
||||
|
||||
async def execute(self, node: TaskNode, domain: str, difficulty: str,
|
||||
memory: WorkingMemory) -> ExpertResponse:
|
||||
return await self._rule.generate(node.query, difficulty, memory, node)
|
||||
|
||||
|
||||
class ModelNodeExecutor(NodeExecutor):
|
||||
"""L2：专家池小模型后端（≤8B；组内模型按需加载，用完即卸载由推理服务管理）。"""
|
||||
|
||||
name = "model"
|
||||
|
||||
def __init__(self, experts: Dict[str, Expert]):
|
||||
self._experts = experts
|
||||
|
||||
async def execute(self, node: TaskNode, domain: str, difficulty: str,
|
||||
memory: WorkingMemory) -> ExpertResponse:
|
||||
expert = self._experts.get(node.domain) or self._experts.get("general")
|
||||
return await expert.generate(node.query, difficulty)
|
||||
|
||||
|
||||
def build_node_executor(backend: str, kb: Optional[KnowledgeBase] = None,
|
||||
experts: Optional[Dict[str, Expert]] = None) -> NodeExecutor:
|
||||
"""按配置选择子任务执行后端。"""
|
||||
if backend == "rule":
|
||||
return RuleNodeExecutor(kb=kb)
|
||||
if backend in ("hf", "api", "model"):
|
||||
if not experts:
|
||||
raise ValueError("ModelNodeExecutor 需要专家池（experts）")
|
||||
return ModelNodeExecutor(experts)
|
||||
raise ValueError(f"未知执行后端: {backend}（支持 rule | hf | api | model）")
|
||||
@@ -0,0 +1,93 @@
|
||||
"""前向链推理机：知识库规则驱动的工作记忆演化（专家系统推理核心，零依赖）。
|
||||
|
||||
流程（经典前向链 forward chaining）：
|
||||
1. 初始化黑板：写入领域/难度/置信度等事实
|
||||
2. 循环：在领域内匹配规则（未触发过的）→ 按优先级执行
|
||||
- 命中即记录轨迹 rule:<id>@<priority>
|
||||
- 规则带 output 模板 → 渲染后写入黑板章节（部分解）
|
||||
- 规则带 actions → 执行动作（写事实/写章节）
|
||||
3. 终止：无新规则可触发 / 达到步数上限（防死循环）
|
||||
|
||||
确定性保证：规则匹配基于子串包含，无随机性；同输入 → 同轨迹。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from .knowledge import KnowledgeBase, Rule
|
||||
from .memory import WorkingMemory
|
||||
|
||||
|
||||
def render_template(template: str, query: str, facts: Dict[str, Any]) -> str:
|
||||
"""渲染输出模板：替换 {query} 与 {facts.<key>} 占位符；缺失以 [未提供] 占位，不抛异常。"""
|
||||
out = template.replace("{query}", query)
|
||||
for key, value in facts.items():
|
||||
out = out.replace(f"{{facts.{key}}}", str(value))
|
||||
# 剩余占位符兜底
|
||||
while "{" in out and "}" in out:
|
||||
start = out.find("{")
|
||||
end = out.find("}", start)
|
||||
if end == -1:
|
||||
break
|
||||
out = out[:start] + "[未提供]" + out[end + 1:]
|
||||
return out
|
||||
|
||||
|
||||
class InferenceEngine:
|
||||
"""前向链推理机。"""
|
||||
|
||||
def __init__(self, kb: KnowledgeBase, max_steps: int = 20):
|
||||
self.kb = kb
|
||||
self.max_steps = max_steps
|
||||
|
||||
# ---------------------------------------------------------------
|
||||
def initialize(self, query: str, domain: str, difficulty: str,
|
||||
confidence: float, memory: WorkingMemory) -> None:
|
||||
"""把分类结果写入黑板（事实初始化）。"""
|
||||
memory.write_fact("query", query)
|
||||
memory.write_fact("domain", domain)
|
||||
memory.write_fact("difficulty", difficulty)
|
||||
memory.write_fact("confidence", round(confidence, 4))
|
||||
memory.add_trace(f"init:domain={domain},difficulty={difficulty},conf={confidence:.2f}")
|
||||
|
||||
# ---------------------------------------------------------------
|
||||
def run(self, query: str, domain: str, memory: WorkingMemory,
|
||||
max_steps: Optional[int] = None) -> List[str]:
|
||||
"""前向链主循环。返回触发规则 id 列表（按触发顺序）。"""
|
||||
steps = max_steps or self.max_steps
|
||||
fired: List[str] = []
|
||||
for _ in range(steps):
|
||||
rules = self.kb.match(query, domain=domain)
|
||||
# 选第一个"未触发过"的规则
|
||||
target: Optional[Rule] = None
|
||||
for r in rules:
|
||||
if r.id not in fired:
|
||||
target = r
|
||||
break
|
||||
if target is None:
|
||||
break # 无新规则可触发 → 终止
|
||||
fired.append(target.id)
|
||||
self._fire(target, query, memory)
|
||||
return fired
|
||||
|
||||
# ---------------------------------------------------------------
|
||||
def _fire(self, rule: Rule, query: str, memory: WorkingMemory) -> None:
|
||||
"""执行一条规则：记录轨迹 + 写事实 + 产出章节。"""
|
||||
memory.add_trace(f"rule:{rule.id}@{rule.priority}")
|
||||
# 规则动作
|
||||
for action in rule.actions:
|
||||
self._apply_action(action, rule, query, memory)
|
||||
# 规则输出模板 → 章节
|
||||
if rule.output:
|
||||
text = render_template(rule.output, query, memory.facts)
|
||||
memory.write_section(rule.id, text)
|
||||
|
||||
def _apply_action(self, action: str, rule: Rule, query: str,
|
||||
memory: WorkingMemory) -> None:
|
||||
"""动作格式：write_fact:key=value（value 支持 {query} 占位）。"""
|
||||
if action.startswith("write_fact:"):
|
||||
kv = action[len("write_fact:"):]
|
||||
key, _, value = kv.partition("=")
|
||||
value = value.replace("{query}", query)
|
||||
memory.write_fact(key.strip(), value.strip(), rule_id=rule.id)
|
||||
# 其他动作类型暂不实现（保留扩展位）
|
||||
@@ -0,0 +1,365 @@
|
||||
"""知识库：专家系统风格的规则与知识表示（零依赖，纯标准库）。
|
||||
|
||||
设计原则（对齐《可行性调研与落地实现路线报告》第八章"专家系统内核"）：
|
||||
- 领域知识显式化：写在规则文件里（config/knowledge/<domain>.yaml），不藏在模型参数中
|
||||
- 确定性：规则匹配 = 子串包含（大小写不敏感），同输入同输出
|
||||
- 可解释：每次命中都记录规则 id，形成推理轨迹
|
||||
- 最小参数：L0 模式零模型参数，规则即知识
|
||||
|
||||
规则文件格式（YAML；若 pyyaml 不可用，可提供同名 .json）：
|
||||
domain: code
|
||||
rules:
|
||||
- id: code-sort
|
||||
priority: 90 # 越大越先触发
|
||||
patterns: ["排序", "sort"] # 任一子串命中即触发
|
||||
template: code-implement # 可选：Planner 任务模板 id
|
||||
output: | # 可选：输出模板（{query} 等占位符）
|
||||
（规则输出）...
|
||||
facts: # 领域事实表（Judge 校验 / retrieve 执行器用）
|
||||
- id: legal-nc
|
||||
keywords: ["竞业"]
|
||||
statement: "竞业限制期限不得超过二年"
|
||||
|
||||
任务模板（config/knowledge/tasks.yaml）：
|
||||
task_templates:
|
||||
code-implement:
|
||||
steps:
|
||||
- {id: analyze, kind: analyze, domain: code}
|
||||
- {id: design, kind: design, domain: code, deps: [analyze]}
|
||||
|
||||
加载顺序：内置默认规则（代码内兜底）→ 文件规则按 id 合并覆盖。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
DEFAULT_RULES_DIR = Path(__file__).resolve().parent.parent / "config" / "knowledge"
|
||||
|
||||
|
||||
@dataclass
|
||||
class Rule:
|
||||
"""一条领域规则。"""
|
||||
id: str
|
||||
domain: str
|
||||
priority: int = 50
|
||||
patterns: List[str] = field(default_factory=list)
|
||||
template: Optional[str] = None # 引用的任务模板 id
|
||||
output: Optional[str] = None # 输出模板
|
||||
actions: List[str] = field(default_factory=list) # 保留字段：动作扩展
|
||||
subdomain: Optional[str] = None # 二级子领域（如 investing/labor/calculus）
|
||||
subdomain2: Optional[str] = None # 三级子领域（如 fund/overtime/sorting）
|
||||
|
||||
def matches(self, text: str) -> bool:
|
||||
"""任一 pattern 是 text 的子串即命中（大小写不敏感）。"""
|
||||
if not self.patterns:
|
||||
return False
|
||||
q = text.lower()
|
||||
return any(p.lower() in q for p in self.patterns)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------
|
||||
# 三级子领域映射（rule_id -> subdomain2）
|
||||
# 集中维护：新增规则时在此加一行即可完成三级细化标注
|
||||
# ---------------------------------------------------------------
|
||||
SUBDOMAIN2_MAP: Dict[str, str] = {
|
||||
# ---- code ----
|
||||
"code-sort": "sorting",
|
||||
"code-debug": "error-analysis",
|
||||
"code-algorithm": "algorithm-general",
|
||||
"code-refactor": "code-quality",
|
||||
"code-database": "sql",
|
||||
"code-explain": "code-reading",
|
||||
"code-test": "unit-test",
|
||||
"code-web": "web-dev",
|
||||
"code-implement-general": "implementation",
|
||||
"code-git-knowledge": "git",
|
||||
"code-docker-knowledge": "container",
|
||||
"code-python-knowledge": "python-env",
|
||||
# ---- math ----
|
||||
"math-equation": "equation",
|
||||
"math-calculus": "calculus",
|
||||
"math-algebra": "algebra",
|
||||
"math-geometry": "geometry",
|
||||
"math-proof": "proof",
|
||||
"math-probability": "probability",
|
||||
"math-number-theory": "number-theory",
|
||||
"math-trigonometry": "trigonometry",
|
||||
"math-optimization": "optimization",
|
||||
"math-general": "math-general",
|
||||
# ---- legal ----
|
||||
"legal-contract": "contract",
|
||||
"legal-labor": "labor",
|
||||
"legal-ip": "intellectual-property",
|
||||
"legal-housing": "housing",
|
||||
"legal-marriage": "family-law",
|
||||
"legal-tax": "tax",
|
||||
"legal-consumer": "consumer-rights",
|
||||
"legal-litigation": "litigation",
|
||||
"legal-compliance": "compliance",
|
||||
"legal-general": "legal-general",
|
||||
# ---- medical ----
|
||||
"medical-hypertension": "hypertension",
|
||||
"medical-drug": "medication",
|
||||
"medical-common": "common-illness",
|
||||
"medical-chronic": "chronic-disease",
|
||||
"medical-digestive": "digestive",
|
||||
"medical-nutrition": "nutrition",
|
||||
"medical-mental": "mental-health",
|
||||
"medical-firstaid": "first-aid",
|
||||
"medical-pediatrics": "pediatrics",
|
||||
"medical-general": "medical-general",
|
||||
# ---- finance ----
|
||||
"finance-investing": "investing",
|
||||
"finance-saving": "saving",
|
||||
"finance-loan": "loan",
|
||||
"finance-insurance": "insurance",
|
||||
"finance-credit-card": "credit",
|
||||
"finance-personal-budget": "budgeting",
|
||||
"finance-general": "finance-general",
|
||||
# ---- life ----
|
||||
"life-food": "cooking",
|
||||
"life-travel": "travel",
|
||||
"life-home": "home",
|
||||
"life-pet": "pet",
|
||||
"life-fitness": "fitness",
|
||||
"life-weather": "weather",
|
||||
"life-general": "life-general",
|
||||
# ---- education ----
|
||||
"edu-study-method": "study-method",
|
||||
"edu-exam": "exam",
|
||||
"edu-language": "language",
|
||||
"edu-course": "course",
|
||||
"edu-career": "career",
|
||||
"edu-general": "education-general",
|
||||
# ---- general ----
|
||||
"general-explain": "explain",
|
||||
"general-writing": "writing",
|
||||
"general-compare": "compare",
|
||||
"general-translate": "translate",
|
||||
"general-knowledge": "explain",
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------
|
||||
# 内置默认规则（兜底：即使规则文件缺失/损坏，系统仍可运行）
|
||||
# ---------------------------------------------------------------
|
||||
BUILTIN_RULES: List[Dict[str, Any]] = [
|
||||
# ---- code ----
|
||||
{"id": "code-sort", "domain": "code", "priority": 90,
|
||||
"patterns": ["排序", "快速排序", "排序算法", "sort", "quicksort"],
|
||||
"template": "code-implement"},
|
||||
{"id": "code-debug", "domain": "code", "priority": 85,
|
||||
"patterns": ["报错", "错误", "调试", "bug", "debug", "typeerror", "异常", "报 TypeError"],
|
||||
"template": "code-debug"},
|
||||
{"id": "code-implement-general", "domain": "code", "priority": 50,
|
||||
"patterns": ["实现", "编写", "写一个", "函数", "代码", "编程", "用 python", "用 java",
|
||||
"用 javascript", "sql", "接口", "算法"],
|
||||
"template": "code-implement"},
|
||||
# ---- math ----
|
||||
{"id": "math-equation", "domain": "math", "priority": 90,
|
||||
"patterns": ["方程", "求解", "求根", "solve", "equation", "解方程"],
|
||||
"template": "math-solve"},
|
||||
{"id": "math-calculus", "domain": "math", "priority": 85,
|
||||
"patterns": ["积分", "导数", "微积分", "求导", "integral", "derivative", "∫"],
|
||||
"template": "math-solve"},
|
||||
{"id": "math-general", "domain": "math", "priority": 50,
|
||||
"patterns": ["数学", "证明", "定理", "概率", "统计", "计算", "等于", "math", "不等式"],
|
||||
"template": "math-solve"},
|
||||
# ---- legal ----
|
||||
{"id": "legal-contract", "domain": "legal", "priority": 90,
|
||||
"patterns": ["合同", "条款", "违约", "离职", "竞业", "劳动", "contract", "clause", "赔偿"],
|
||||
"template": "legal-advice"},
|
||||
{"id": "legal-ip", "domain": "legal", "priority": 85,
|
||||
"patterns": ["专利", "版权", "商标", "知识产权", "patent", "copyright", "trademark"],
|
||||
"template": "legal-advice"},
|
||||
{"id": "legal-general", "domain": "legal", "priority": 50,
|
||||
"patterns": ["法律", "合规", "诉讼", "仲裁", "法条", "law", "legal", "法规"],
|
||||
"template": "legal-advice"},
|
||||
# ---- medical ----
|
||||
{"id": "medical-hypertension", "domain": "medical", "priority": 90,
|
||||
"patterns": ["高血压", "hypertension", "血压"],
|
||||
"template": "medical-advice"},
|
||||
{"id": "medical-drug", "domain": "medical", "priority": 85,
|
||||
"patterns": ["药物", "吃药", "剂量", "副作用", "退烧药", "降压药", "dosage", "prescription"],
|
||||
"template": "medical-advice"},
|
||||
{"id": "medical-general", "domain": "medical", "priority": 50,
|
||||
"patterns": ["医疗", "症状", "诊断", "治疗", "感冒", "发烧", "糖尿病", "医生", "患者",
|
||||
"体检", "疫苗", "medical", "symptom", "disease"],
|
||||
"template": "medical-advice"},
|
||||
# ---- general ----
|
||||
{"id": "general-explain", "domain": "general", "priority": 30,
|
||||
"patterns": ["总结", "介绍", "解释", "为什么", "优缺点", "是什么", "翻译", "邮件",
|
||||
"summarize", "explain", "what is", "写一封"],
|
||||
"template": "general-explain"},
|
||||
]
|
||||
|
||||
# 内置默认任务模板（兜底）
|
||||
BUILTIN_TASKS: Dict[str, Dict[str, Any]] = {
|
||||
"code-implement": {"steps": [
|
||||
{"id": "analyze", "kind": "analyze", "domain": "code", "desc": "需求与约束分析"},
|
||||
{"id": "design", "kind": "design", "domain": "code", "deps": ["analyze"], "desc": "算法与数据结构设计"},
|
||||
{"id": "implement", "kind": "implement", "domain": "code", "deps": ["design"], "desc": "实现代码"},
|
||||
{"id": "verify", "kind": "verify", "domain": "code", "deps": ["implement"], "desc": "自测校验"},
|
||||
]},
|
||||
"code-debug": {"steps": [
|
||||
{"id": "analyze", "kind": "analyze", "domain": "code", "desc": "错误现象与复现分析"},
|
||||
{"id": "diagnose", "kind": "diagnose", "domain": "code", "deps": ["analyze"], "desc": "定位错误根因"},
|
||||
{"id": "fix", "kind": "fix", "domain": "code", "deps": ["diagnose"], "desc": "给出修复方案"},
|
||||
{"id": "verify", "kind": "verify", "domain": "code", "deps": ["fix"], "desc": "修复后验证"},
|
||||
]},
|
||||
"math-solve": {"steps": [
|
||||
{"id": "conditions", "kind": "analyze", "domain": "math", "desc": "明确已知条件与目标"},
|
||||
{"id": "solve", "kind": "solve", "domain": "math", "deps": ["conditions"], "desc": "选择方法并求解"},
|
||||
{"id": "verify", "kind": "verify", "domain": "math", "deps": ["solve"], "desc": "检查边界与验证"},
|
||||
]},
|
||||
"legal-advice": {"steps": [
|
||||
{"id": "facts", "kind": "analyze", "domain": "legal", "desc": "梳理事实与法律问题"},
|
||||
{"id": "retrieve", "kind": "retrieve", "domain": "legal", "deps": ["facts"], "desc": "检索适用法规"},
|
||||
{"id": "conclude", "kind": "conclude", "domain": "legal", "deps": ["retrieve"], "desc": "给出法律意见"},
|
||||
{"id": "disclaimer", "kind": "disclaimer", "domain": "legal", "deps": ["conclude"], "desc": "免责提示"},
|
||||
]},
|
||||
"medical-advice": {"steps": [
|
||||
{"id": "symptoms", "kind": "analyze", "domain": "medical", "desc": "梳理症状与背景"},
|
||||
{"id": "advise", "kind": "advise", "domain": "medical", "deps": ["symptoms"], "desc": "给出一般建议"},
|
||||
{"id": "warning", "kind": "disclaimer", "domain": "medical", "deps": ["advise"], "desc": "就医警示"},
|
||||
]},
|
||||
"general-explain": {"steps": [
|
||||
{"id": "outline", "kind": "analyze", "domain": "general", "desc": "梳理主题要点"},
|
||||
{"id": "explain", "kind": "explain", "domain": "general", "deps": ["outline"], "desc": "展开解释"},
|
||||
{"id": "conclude", "kind": "conclude", "domain": "general", "deps": ["explain"], "desc": "总结"},
|
||||
]},
|
||||
}
|
||||
|
||||
# 内置默认事实表（兜底）
|
||||
BUILTIN_FACTS: Dict[str, List[Dict[str, Any]]] = {
|
||||
"legal": [
|
||||
{"id": "legal-noncompete", "keywords": ["竞业", "离职", "同业"],
|
||||
"statement": "竞业限制期限不得超过二年，且用人单位应在限制期内按月给予经济补偿"},
|
||||
{"id": "legal-renew-compensation", "keywords": ["不续签", "经济补偿", "劳动合同"],
|
||||
"statement": "劳动合同期满用人单位不续签的，通常应支付经济补偿（每满一年一个月工资）"},
|
||||
],
|
||||
"medical": [
|
||||
{"id": "medical-hypertension-diet", "keywords": ["高血压", "饮食"],
|
||||
"statement": "高血压患者应低盐低脂饮食、控制体重、规律运动、戒烟限酒，并在医生指导下用药"},
|
||||
{"id": "medical-fever-drug", "keywords": ["发烧", "退烧"],
|
||||
"statement": "体温超过 38.5℃ 可在药师指导下使用退烧药；持续发热或出现严重症状应及时就医"},
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def _try_load_yaml(path: Path) -> Optional[Dict[str, Any]]:
|
||||
try:
|
||||
import yaml # type: ignore
|
||||
except ImportError:
|
||||
return None
|
||||
try:
|
||||
with open(path, "r", encoding="utf-8") as f:
|
||||
data = yaml.safe_load(f)
|
||||
return data if isinstance(data, dict) else None
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def _try_load_json(path: Path) -> Optional[Dict[str, Any]]:
|
||||
json_path = path.with_suffix(".json")
|
||||
if not json_path.exists():
|
||||
return None
|
||||
try:
|
||||
with open(json_path, "r", encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
return data if isinstance(data, dict) else None
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
class KnowledgeBase:
|
||||
"""知识库：加载规则文件，提供规则匹配、任务模板、事实表查询。"""
|
||||
|
||||
def __init__(self, rules_dir: Optional[str | Path] = None):
|
||||
self.rules_dir = Path(rules_dir) if rules_dir else DEFAULT_RULES_DIR
|
||||
self._rules: Dict[str, Rule] = {}
|
||||
self._tasks: Dict[str, Dict[str, Any]] = {}
|
||||
self._facts: Dict[str, List[Dict[str, Any]]] = {}
|
||||
self.load()
|
||||
|
||||
# ---- 加载 ----
|
||||
def load(self) -> None:
|
||||
"""内置默认 + 规则文件合并（文件规则按 id 覆盖内置）。"""
|
||||
self._rules = {}
|
||||
self._tasks = dict(BUILTIN_TASKS)
|
||||
for item in BUILTIN_RULES:
|
||||
self._register_rule(item)
|
||||
self._facts = {d: [dict(f) for f in facts] for d, facts in BUILTIN_FACTS.items()}
|
||||
|
||||
if self.rules_dir.is_dir():
|
||||
for f in sorted(self.rules_dir.glob("*.yaml")):
|
||||
data = _try_load_yaml(f)
|
||||
if data is not None:
|
||||
self._load_file_data(f, data)
|
||||
for f in sorted(self.rules_dir.glob("*.json")):
|
||||
if f.name not in {p.name for p in self.rules_dir.glob("*.yaml")}:
|
||||
data = _try_load_json(f)
|
||||
if data is not None:
|
||||
self._load_file_data(f, data)
|
||||
|
||||
def _load_file_data(self, path: Path, data: Dict[str, Any]) -> None:
|
||||
name = path.stem
|
||||
if name == "tasks":
|
||||
for tid, tpl in (data.get("task_templates") or {}).items():
|
||||
if isinstance(tpl, dict) and isinstance(tpl.get("steps"), list):
|
||||
self._tasks[tid] = tpl
|
||||
return
|
||||
domain = data.get("domain", name)
|
||||
for item in data.get("rules") or []:
|
||||
if isinstance(item, dict) and item.get("id"):
|
||||
self._register_rule({**item, "domain": domain})
|
||||
for fact in data.get("facts") or []:
|
||||
if isinstance(fact, dict) and fact.get("id"):
|
||||
self._facts.setdefault(domain, []).append(fact)
|
||||
|
||||
def _register_rule(self, item: Dict[str, Any]) -> None:
|
||||
rule = Rule(
|
||||
id=str(item["id"]),
|
||||
domain=str(item.get("domain", "general")),
|
||||
priority=int(item.get("priority", 50)),
|
||||
patterns=[str(p) for p in item.get("patterns", [])],
|
||||
template=item.get("template"),
|
||||
output=item.get("output"),
|
||||
actions=[str(a) for a in item.get("actions", [])],
|
||||
subdomain=item.get("subdomain"),
|
||||
subdomain2=item.get("subdomain2") or SUBDOMAIN2_MAP.get(str(item["id"])),
|
||||
)
|
||||
self._rules[rule.id] = rule
|
||||
|
||||
# ---- 查询 ----
|
||||
def match(self, text: str, domain: Optional[str] = None) -> List[Rule]:
|
||||
"""返回命中的规则，按优先级降序。domain 为空则全领域匹配。"""
|
||||
hits = []
|
||||
for rule in self._rules.values():
|
||||
if domain is not None and rule.domain != domain:
|
||||
continue
|
||||
if rule.matches(text):
|
||||
hits.append(rule)
|
||||
hits.sort(key=lambda r: r.priority, reverse=True)
|
||||
return hits
|
||||
|
||||
def rule(self, rule_id: str) -> Optional[Rule]:
|
||||
return self._rules.get(rule_id)
|
||||
|
||||
def rules_count(self) -> int:
|
||||
return len(self._rules)
|
||||
|
||||
def task_template(self, tid: str) -> Optional[Dict[str, Any]]:
|
||||
return self._tasks.get(tid)
|
||||
|
||||
def task_ids(self) -> List[str]:
|
||||
return sorted(self._tasks.keys())
|
||||
|
||||
def facts(self, domain: str) -> List[Dict[str, Any]]:
|
||||
return self._facts.get(domain, [])
|
||||
|
||||
def domains(self) -> List[str]:
|
||||
return sorted({r.domain for r in self._rules.values()})
|
||||
@@ -0,0 +1,128 @@
|
||||
"""黑板（Blackboard）/ 工作记忆：专家系统风格的共享工作区（零依赖）。
|
||||
|
||||
- TaskNode：子任务节点（DAG 顶点），由 Planner 创建、Router 按拓扑序执行
|
||||
- TaskGraph：子任务 DAG，提供拓扑排序与状态查询
|
||||
- WorkingMemory：黑板，各知识源（执行器/规则）写入部分解，最后合并为最终答案
|
||||
|
||||
对齐《可行性调研与落地实现路线报告》第八章：
|
||||
"黑板协作：多知识源（领域专家/执行器）通过共享黑板协作，而不是一个模型全包"。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
|
||||
@dataclass
|
||||
class TaskNode:
|
||||
"""一个子任务节点。"""
|
||||
id: str
|
||||
kind: str # analyze | design | implement | solve | diagnose | fix
|
||||
# | retrieve | conclude | advise | explain | disclaimer | verify
|
||||
domain: str
|
||||
query: str # 子任务输入（通常为原始查询）
|
||||
status: str = "pending" # pending | running | done | failed | skipped
|
||||
output: Optional[str] = None
|
||||
rule_trace: List[str] = field(default_factory=list)
|
||||
deps: List[str] = field(default_factory=list)
|
||||
desc: str = ""
|
||||
error: Optional[str] = None
|
||||
|
||||
|
||||
class TaskGraph:
|
||||
"""子任务 DAG：节点 + 依赖边。"""
|
||||
|
||||
def __init__(self):
|
||||
self._nodes: Dict[str, TaskNode] = {}
|
||||
|
||||
def add_node(self, node: TaskNode) -> None:
|
||||
if node.id in self._nodes:
|
||||
raise ValueError(f"节点 id 重复: {node.id}")
|
||||
self._nodes[node.id] = node
|
||||
|
||||
def get(self, node_id: str) -> Optional[TaskNode]:
|
||||
return self._nodes.get(node_id)
|
||||
|
||||
def nodes(self) -> List[TaskNode]:
|
||||
return list(self._nodes.values())
|
||||
|
||||
def topo_order(self) -> List[TaskNode]:
|
||||
"""Kahn 拓扑排序：依赖在前。循环依赖时按插入序兜底（不崩溃）。"""
|
||||
indeg: Dict[str, int] = {}
|
||||
for n in self._nodes.values():
|
||||
indeg[n.id] = 0
|
||||
for n in self._nodes.values():
|
||||
for d in n.deps:
|
||||
if d in indeg:
|
||||
indeg[n.id] += 1
|
||||
ready = [n for n in self._nodes.values() if indeg[n.id] == 0]
|
||||
ready.sort(key=lambda n: list(self._nodes.keys()).index(n.id))
|
||||
order: List[TaskNode] = []
|
||||
while ready:
|
||||
n = ready.pop(0)
|
||||
order.append(n)
|
||||
for m in self._nodes.values():
|
||||
if n.id in m.deps:
|
||||
indeg[m.id] -= 1
|
||||
if indeg[m.id] == 0 and m not in order:
|
||||
ready.append(m)
|
||||
if len(order) < len(self._nodes):
|
||||
# 循环依赖兜底：剩余节点按插入序追加
|
||||
for n in self._nodes.values():
|
||||
if n not in order:
|
||||
order.append(n)
|
||||
return order
|
||||
|
||||
def all_done(self) -> bool:
|
||||
return all(n.status == "done" for n in self._nodes.values())
|
||||
|
||||
def failed(self) -> List[TaskNode]:
|
||||
return [n for n in self._nodes.values() if n.status == "failed"]
|
||||
|
||||
def __len__(self) -> int:
|
||||
return len(self._nodes)
|
||||
|
||||
|
||||
class WorkingMemory:
|
||||
"""黑板：facts（槽位事实）+ sections（章节部分解）+ trace（推理轨迹）。"""
|
||||
|
||||
def __init__(self):
|
||||
self.facts: Dict[str, Any] = {}
|
||||
self.sections: Dict[str, str] = {}
|
||||
self.trace: List[str] = []
|
||||
|
||||
# ---- 事实 ----
|
||||
def write_fact(self, key: str, value: Any, rule_id: Optional[str] = None) -> None:
|
||||
if key in self.facts:
|
||||
self.trace.append(f"overwrite:{key}@{rule_id or '?'}")
|
||||
self.facts[key] = value
|
||||
if rule_id:
|
||||
self.trace.append(f"fact:{key}={str(value)[:40]}@rule:{rule_id}")
|
||||
|
||||
def get_fact(self, key: str, default: Any = None) -> Any:
|
||||
return self.facts.get(key, default)
|
||||
|
||||
# ---- 章节 ----
|
||||
def write_section(self, sid: str, text: str) -> None:
|
||||
"""写入章节；同 id 覆盖（记录 trace）。"""
|
||||
if sid in self.sections:
|
||||
self.trace.append(f"overwrite_section:{sid}")
|
||||
self.sections[sid] = text
|
||||
|
||||
def section(self, sid: str) -> Optional[str]:
|
||||
return self.sections.get(sid)
|
||||
|
||||
def merge(self, order: Optional[List[str]] = None) -> str:
|
||||
"""按 order（章节顺序）合并为最终答案；order 为空则按写入顺序。"""
|
||||
if order:
|
||||
parts = [self.sections[s] for s in order if s in self.sections]
|
||||
if parts:
|
||||
return "\n\n".join(parts)
|
||||
return "\n\n".join(self.sections.values())
|
||||
|
||||
# ---- 轨迹 ----
|
||||
def add_trace(self, item: str) -> None:
|
||||
self.trace.append(item)
|
||||
|
||||
def explain(self) -> List[str]:
|
||||
return list(self.trace)
|
||||
@@ -0,0 +1,103 @@
|
||||
"""规则 Planner：把查询拆解为子任务 DAG（任务分解，专家系统风格，零参数）。
|
||||
|
||||
拆解逻辑（确定性规则）：
|
||||
1. 在分类领域内匹配知识规则
|
||||
2. 取最高优先级且带 template 的命中规则 → 对应任务模板
|
||||
3. 非 easy 难度且有模板 → 生成多节点 DAG（模板 steps 转 TaskNode，含依赖）
|
||||
4. easy 难度或无模板命中 → 单节点直接求解（不拆，最小开销）
|
||||
5. 拆解深度防护：节点不再递归拆解（当前为单层拆解，模板本身即最终粒度）
|
||||
|
||||
对齐架构目标："路由模型把任务拆解后分步骤交给各个小模型"，
|
||||
L0 模式下各子任务由规则执行器完成（零参数），L2 模式可交给本地小模型。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import List, Optional
|
||||
|
||||
from .knowledge import KnowledgeBase
|
||||
from .memory import TaskGraph, TaskNode
|
||||
from .models import Classification
|
||||
|
||||
# 单节点求解时按领域选择默认动作 kind
|
||||
_SINGLE_KIND = {
|
||||
"code": "implement",
|
||||
"math": "solve",
|
||||
"legal": "conclude",
|
||||
"medical": "advise",
|
||||
"general": "explain",
|
||||
"finance": "conclude",
|
||||
"life": "advise",
|
||||
"education": "design",
|
||||
}
|
||||
|
||||
# 强制拆解领域：即使 easy 也走完整任务模板
|
||||
# （legal 需要 retrieve+disclaimer，medical 需要 advise+warning，
|
||||
# finance 需要 retrieve+风险免责——均为领域硬要求）
|
||||
FORCE_SPLIT_DOMAINS = {"legal", "medical", "finance"}
|
||||
|
||||
# 强制拆解模板：命中即拆（debug 流程必须 analyze→diagnose→fix→verify）
|
||||
FORCE_SPLIT_TEMPLATES = {"code-debug"}
|
||||
|
||||
|
||||
class Planner:
|
||||
"""规则 Planner：查询 → 子任务 DAG。"""
|
||||
|
||||
def __init__(self, kb: KnowledgeBase, max_depth: int = 3):
|
||||
self.kb = kb
|
||||
self.max_depth = max_depth
|
||||
|
||||
# ---------------------------------------------------------------
|
||||
def plan(self, query: str, classification: Classification) -> TaskGraph:
|
||||
domain = classification.domain
|
||||
difficulty = classification.difficulty
|
||||
|
||||
# 1. 领域内匹配规则，取最高优先级带模板的规则
|
||||
template_id: Optional[str] = None
|
||||
hits = self.kb.match(query, domain=domain)
|
||||
for h in hits:
|
||||
if h.template:
|
||||
template_id = h.template
|
||||
break
|
||||
|
||||
graph = TaskGraph()
|
||||
|
||||
# 2. 非 easy / 强制拆解领域 / 强制拆解模板 → 多节点 DAG
|
||||
if template_id and (difficulty != "easy"
|
||||
or domain in FORCE_SPLIT_DOMAINS
|
||||
or template_id in FORCE_SPLIT_TEMPLATES):
|
||||
tpl = self.kb.task_template(template_id)
|
||||
if tpl and tpl.get("steps"):
|
||||
for step in tpl["steps"]:
|
||||
node = TaskNode(
|
||||
id=str(step["id"]),
|
||||
kind=str(step.get("kind", "solve")),
|
||||
domain=str(step.get("domain", domain)),
|
||||
query=query,
|
||||
deps=[str(d) for d in step.get("deps", [])],
|
||||
desc=str(step.get("desc", "")),
|
||||
)
|
||||
graph.add_node(node)
|
||||
return graph
|
||||
|
||||
# 3. easy / 无模板 → 单节点
|
||||
kind = _SINGLE_KIND.get(domain, "explain")
|
||||
graph.add_node(TaskNode(
|
||||
id="solve",
|
||||
kind=kind,
|
||||
domain=domain,
|
||||
query=query,
|
||||
desc=f"单节点求解（{domain}/{difficulty}）",
|
||||
))
|
||||
return graph
|
||||
|
||||
# ---------------------------------------------------------------
|
||||
def explain_plan(self, graph: TaskGraph) -> List[str]:
|
||||
"""把 DAG 渲染为可读的拆解轨迹（用于 route 与 --trace）。"""
|
||||
if len(graph) == 1:
|
||||
n = graph.nodes()[0]
|
||||
return [f"plan:single[{n.kind}]"]
|
||||
parts = []
|
||||
for n in graph.topo_order():
|
||||
dep = f"<{','.join(n.deps)}" if n.deps else ""
|
||||
parts.append(f"{n.id}:{n.kind}{dep}")
|
||||
return [f"plan:multi[{len(graph)}]({' -> '.join(parts)})"]
|
||||
@@ -0,0 +1,217 @@
|
||||
"""Skill 技能体系（T12：Agent-Skill 路由器·先行实现）。
|
||||
|
||||
把路由器独立为"技能注册表 + Agent 规划器"：
|
||||
- 用户只需提供需求，无需指定领域/模型
|
||||
- Agent 路由器自行分析需求 → 规划 skill 调用序列（可组合、可依赖）→ 执行
|
||||
|
||||
Skill 抽象：name（唯一标识）+ description（能力描述，供规划器选择）
|
||||
+ params（参数说明）+ execute(ctx)（执行，返回文本结果）。
|
||||
|
||||
内置技能（把专家系统内核能力封装为可调用单元）：
|
||||
- es.<kind> ：规则执行器模板生成（analyze/design/implement/solve/... 共 18 种）
|
||||
- kb.retrieve ：知识库事实检索（法律/医疗/金融事实条目）
|
||||
- kb.answer ：规则 output 知识问答（git/docker/天气/翻译等常识条目）
|
||||
- judge.evaluate：质量评分
|
||||
- fallback.call ：最后处理者（升级）
|
||||
- cache.get / cache.put：两阶段缓存
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from .executors import RuleExecutor
|
||||
from .knowledge import KnowledgeBase
|
||||
from .memory import TaskNode, WorkingMemory
|
||||
from .models import ExpertResponse
|
||||
|
||||
|
||||
@dataclass
|
||||
class SkillContext:
|
||||
"""一次 skill 调用的执行上下文（黑板 + 输入参数）。"""
|
||||
query: str
|
||||
domain: str
|
||||
difficulty: str
|
||||
memory: WorkingMemory
|
||||
kb: Optional[KnowledgeBase] = None
|
||||
args: Dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
|
||||
class Skill:
|
||||
"""技能抽象：可被 Agent 路由器调用的能力单元。"""
|
||||
|
||||
name: str = "skill"
|
||||
description: str = ""
|
||||
params: List[str] = field(default_factory=list)
|
||||
|
||||
async def execute(self, ctx: SkillContext) -> str:
|
||||
raise NotImplementedError
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f"<Skill {self.name}>"
|
||||
|
||||
|
||||
class SkillRegistry:
|
||||
"""技能注册表：注册 / 发现 / 执行。"""
|
||||
|
||||
def __init__(self):
|
||||
self._skills: Dict[str, Skill] = {}
|
||||
|
||||
def register(self, skill: Skill) -> None:
|
||||
if skill.name in self._skills:
|
||||
raise ValueError(f"技能重复注册: {skill.name}")
|
||||
self._skills[skill.name] = skill
|
||||
|
||||
def get(self, name: str) -> Optional[Skill]:
|
||||
return self._skills.get(name)
|
||||
|
||||
def has(self, name: str) -> bool:
|
||||
return name in self._skills
|
||||
|
||||
def list(self) -> List[str]:
|
||||
return sorted(self._skills.keys())
|
||||
|
||||
def catalog(self) -> List[Dict[str, Any]]:
|
||||
"""技能目录（供 Agent 规划器 / 用户发现使用）。"""
|
||||
return [
|
||||
{"name": s.name, "description": s.description, "params": list(s.params)}
|
||||
for s in sorted(self._skills.values(), key=lambda s: s.name)
|
||||
]
|
||||
|
||||
async def execute(self, name: str, ctx: SkillContext) -> str:
|
||||
skill = self._skills.get(name)
|
||||
if skill is None:
|
||||
raise KeyError(f"未注册技能: {name}（可用: {self.list()}）")
|
||||
return await skill.execute(ctx)
|
||||
|
||||
|
||||
# ===============================================================
|
||||
# 内置技能实现
|
||||
# ===============================================================
|
||||
|
||||
class TemplateSkill(Skill):
|
||||
"""es.<kind>：规则执行器模板生成（确定性、零参数）。"""
|
||||
|
||||
def __init__(self, kind: str, executor: RuleExecutor):
|
||||
self.kind = kind
|
||||
self._executor = executor
|
||||
self.name = f"es.{kind}"
|
||||
self.description = f"规则模板生成（{kind}）：结构化确定性输出"
|
||||
self.params = ["domain", "difficulty", "memory"]
|
||||
|
||||
async def execute(self, ctx: SkillContext) -> str:
|
||||
node = TaskNode(
|
||||
id="skill", kind=self.kind, domain=ctx.domain,
|
||||
query=ctx.query, desc=f"skill:{self.name}",
|
||||
)
|
||||
resp: ExpertResponse = await self._executor.generate(
|
||||
ctx.query, ctx.difficulty, ctx.memory, node)
|
||||
return resp.text
|
||||
|
||||
|
||||
class KBRetrieveSkill(Skill):
|
||||
"""kb.retrieve：知识库事实检索（法律/医疗/金融事实条目）。"""
|
||||
|
||||
name = "kb.retrieve"
|
||||
description = "从知识库事实表检索领域知识条目（法条/指南/理财常识等）"
|
||||
params = ["domain", "query"]
|
||||
|
||||
def __init__(self, kb: Optional[KnowledgeBase] = None):
|
||||
self._kb = kb
|
||||
|
||||
async def execute(self, ctx: SkillContext) -> str:
|
||||
kb = self._kb or ctx.kb
|
||||
if kb is None:
|
||||
return "（kb.retrieve）未配置知识库。"
|
||||
facts = kb.facts(ctx.domain)
|
||||
hits = [f for f in facts if any(k in ctx.query for k in f.get("keywords", []))]
|
||||
if hits:
|
||||
lines = [f"- {f['statement']}" for f in hits]
|
||||
return f"【{ctx.domain} 知识检索】\n" + "\n".join(lines)
|
||||
return (f"【{ctx.domain} 知识检索】\n未命中知识库条目；"
|
||||
f"建议以现行有效法规/最新指南为准。")
|
||||
|
||||
|
||||
class KBAnswerSkill(Skill):
|
||||
"""kb.answer：规则 output 知识问答（git/docker/天气/翻译等常识条目）。"""
|
||||
|
||||
name = "kb.answer"
|
||||
description = "知识库规则问答：命中 output 规则直接给出知识章节"
|
||||
params = ["domain", "query"]
|
||||
|
||||
def __init__(self, kb: Optional[KnowledgeBase] = None):
|
||||
self._kb = kb
|
||||
|
||||
async def execute(self, ctx: SkillContext) -> str:
|
||||
kb = self._kb or ctx.kb
|
||||
if kb is None:
|
||||
return "（kb.answer）未配置知识库。"
|
||||
hits = kb.match(ctx.query, domain=ctx.domain)
|
||||
for h in hits:
|
||||
if h.output:
|
||||
from .inference import render_template
|
||||
return render_template(h.output, ctx.query, {})
|
||||
return "（kb.answer）未命中知识规则。"
|
||||
|
||||
|
||||
class JudgeSkill(Skill):
|
||||
"""judge.evaluate：质量评分（5 维：覆盖/长度/格式/安全/知识引用）。"""
|
||||
|
||||
name = "judge.evaluate"
|
||||
description = "评估回答质量（0-1 分），低于阈值建议升级"
|
||||
params = ["query", "response", "domain"]
|
||||
|
||||
def __init__(self, judge, fallback_threshold: float = 0.70):
|
||||
self._judge = judge
|
||||
self._threshold = fallback_threshold
|
||||
|
||||
async def execute(self, ctx: SkillContext) -> str:
|
||||
response = ctx.args.get("response", "")
|
||||
domain = ctx.args.get("domain", ctx.domain)
|
||||
eval_result = await self._judge.evaluate(ctx.query, response, domain)
|
||||
return (f"【质量评分】{eval_result.overall_score:.2f} "
|
||||
f"{'(需升级)' if eval_result.needs_fallback else '(达标)'} "
|
||||
f"{'; '.join(eval_result.reasons)}")
|
||||
|
||||
|
||||
class FallbackSkill(Skill):
|
||||
"""fallback.call：最后处理者（升级/降级兜底）。"""
|
||||
|
||||
name = "fallback.call"
|
||||
description = "调用最后处理者（本地≤8B 模型 / 降级模板 / mock）"
|
||||
params = ["query"]
|
||||
|
||||
def __init__(self, fallback):
|
||||
self._fallback = fallback
|
||||
|
||||
async def execute(self, ctx: SkillContext) -> str:
|
||||
resp = await self._fallback.generate(ctx.query)
|
||||
return resp.text
|
||||
|
||||
|
||||
def build_skill_registry(kb: Optional[KnowledgeBase] = None,
|
||||
executor: Optional[RuleExecutor] = None,
|
||||
judge=None, fallback=None,
|
||||
fallback_threshold: float = 0.70) -> SkillRegistry:
|
||||
"""构建内置技能注册表。
|
||||
|
||||
模板技能自动注册 18 种 kind（analyze/design/implement/solve/diagnose/fix/
|
||||
retrieve/conclude/advise/explain/disclaimer/verify/refactor/testcase/
|
||||
complexity/optimize/draft/polish）；retrieve 由 kb.retrieve 接管（更专）。
|
||||
"""
|
||||
reg = SkillRegistry()
|
||||
exec_ = executor or RuleExecutor("rule-executor", "general", kb=kb)
|
||||
_EXCLUDED = {"retrieve"} # retrieve 用 kb.retrieve（知识库驱动）
|
||||
for kind in ("analyze", "design", "implement", "solve", "diagnose", "fix",
|
||||
"conclude", "advise", "explain", "disclaimer", "verify",
|
||||
"refactor", "testcase", "complexity", "optimize", "draft",
|
||||
"polish"):
|
||||
if kind not in _EXCLUDED:
|
||||
reg.register(TemplateSkill(kind, exec_))
|
||||
reg.register(KBRetrieveSkill(kb=kb))
|
||||
reg.register(KBAnswerSkill(kb=kb))
|
||||
if judge is not None:
|
||||
reg.register(JudgeSkill(judge, fallback_threshold=fallback_threshold))
|
||||
if fallback is not None:
|
||||
reg.register(FallbackSkill(fallback))
|
||||
return reg
|
||||
@@ -0,0 +1,49 @@
|
||||
"""推理链轨迹存储（T3：整体项目部分拆解·先行实现）。
|
||||
|
||||
内存环形缓冲（零依赖）：记录每次请求的完整推理链（两级路由决策、
|
||||
三级子领域、规则触发、任务拆解、节点执行、质量评分），支持按请求 ID 追溯。
|
||||
可解释性 = 专家系统 vs 黑盒 LLM 的差异化护城河。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import threading
|
||||
from collections import deque
|
||||
from typing import Any, Deque, Dict, Optional
|
||||
|
||||
|
||||
class TraceStore:
|
||||
"""请求推理链轨迹存储（线程安全，环形淘汰）。"""
|
||||
|
||||
def __init__(self, max_entries: int = 1000):
|
||||
self._max = max_entries
|
||||
self._entries: Dict[str, Dict[str, Any]] = {}
|
||||
self._order: Deque[str] = deque(maxlen=max_entries)
|
||||
self._lock = threading.Lock()
|
||||
|
||||
def put(self, request_id: str, trace: Dict[str, Any]) -> None:
|
||||
with self._lock:
|
||||
if request_id in self._entries:
|
||||
self._entries[request_id] = trace
|
||||
return
|
||||
if len(self._entries) >= self._max:
|
||||
# 环形淘汰最旧
|
||||
while self._order:
|
||||
oldest = self._order.popleft()
|
||||
if oldest in self._entries:
|
||||
del self._entries[oldest]
|
||||
break
|
||||
self._entries[request_id] = trace
|
||||
self._order.append(request_id)
|
||||
|
||||
def get(self, request_id: str) -> Optional[Dict[str, Any]]:
|
||||
with self._lock:
|
||||
return self._entries.get(request_id)
|
||||
|
||||
def size(self) -> int:
|
||||
with self._lock:
|
||||
return len(self._entries)
|
||||
|
||||
def clear(self) -> None:
|
||||
with self._lock:
|
||||
self._entries.clear()
|
||||
self._order.clear()
|
||||
@@ -1,4 +1,4 @@
|
||||
"""交流文本（Workspace）—— 端云协同 LLM 协作系统的核心协议（零依赖）。
|
||||
"""交流文本（Workspace）—— 端云协同编程智能体系统的核心协议（零依赖）。
|
||||
|
||||
大模型（Architect）与小模型（Worker）互不共享内部状态，只通过这份
|
||||
schema 约束的结构化 JSON 共享工作区交接（类比前后端通过 API 契约协作）。
|
||||
|
||||
@@ -0,0 +1,63 @@
|
||||
"""黑板 / DAG（WorkingMemory / TaskGraph）单元测试。"""
|
||||
from router_system.memory import TaskGraph, TaskNode, WorkingMemory
|
||||
|
||||
|
||||
def _graph():
|
||||
g = TaskGraph()
|
||||
g.add_node(TaskNode(id="a", kind="analyze", domain="code", query="q"))
|
||||
g.add_node(TaskNode(id="b", kind="design", domain="code", query="q", deps=["a"]))
|
||||
g.add_node(TaskNode(id="c", kind="implement", domain="code", query="q", deps=["b"]))
|
||||
return g
|
||||
|
||||
|
||||
def test_topo_order_respects_deps():
|
||||
g = _graph()
|
||||
order = [n.id for n in g.topo_order()]
|
||||
assert order.index("a") < order.index("b") < order.index("c")
|
||||
|
||||
|
||||
def test_cycle_fallback_no_crash():
|
||||
g = TaskGraph()
|
||||
g.add_node(TaskNode(id="x", kind="solve", domain="m", query="q", deps=["y"]))
|
||||
g.add_node(TaskNode(id="y", kind="solve", domain="m", query="q", deps=["x"]))
|
||||
order = [n.id for n in g.topo_order()]
|
||||
assert set(order) == {"x", "y"}
|
||||
|
||||
|
||||
def test_failed_and_all_done():
|
||||
g = _graph()
|
||||
assert g.all_done() is False
|
||||
for n in g.nodes():
|
||||
n.status = "done"
|
||||
assert g.all_done() is True
|
||||
g.get("a").status = "failed"
|
||||
assert [n.id for n in g.failed()] == ["a"]
|
||||
|
||||
|
||||
def test_merge_order():
|
||||
mem = WorkingMemory()
|
||||
mem.write_section("s1", "第一部分")
|
||||
mem.write_section("s2", "第二部分")
|
||||
assert mem.merge(["s1", "s2"]) == "第一部分\n\n第二部分"
|
||||
# 指定顺序可调换
|
||||
assert mem.merge(["s2", "s1"]) == "第二部分\n\n第一部分"
|
||||
# 无顺序 → 写入顺序
|
||||
assert mem.merge() == "第一部分\n\n第二部分"
|
||||
|
||||
|
||||
def test_overwrite_fact_records_trace():
|
||||
mem = WorkingMemory()
|
||||
mem.write_fact("k", "v1", rule_id="r1")
|
||||
mem.write_fact("k", "v2", rule_id="r2")
|
||||
assert mem.get_fact("k") == "v2"
|
||||
assert any(t.startswith("overwrite:k@r2") for t in mem.trace)
|
||||
|
||||
|
||||
def test_duplicate_node_id_rejected():
|
||||
g = TaskGraph()
|
||||
g.add_node(TaskNode(id="a", kind="solve", domain="m", query="q"))
|
||||
try:
|
||||
g.add_node(TaskNode(id="a", kind="solve", domain="m", query="q"))
|
||||
assert False, "重复节点 id 应报错"
|
||||
except ValueError:
|
||||
pass
|
||||
@@ -0,0 +1,90 @@
|
||||
"""两级路由（domain_group）测试：用户指定大领域 → 组内路由模型 → 组内执行。"""
|
||||
import pytest
|
||||
|
||||
from router_system.router import build_router
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def router():
|
||||
return build_router()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_explicit_group_tech(router):
|
||||
# 指定 tech 组：跳过 8 领域统一分类器，组内路由识别 code
|
||||
res = await router.route("用 Python 写一个快速排序函数", domain_group="tech")
|
||||
assert res.domain == "code"
|
||||
assert res.domain_group == "tech"
|
||||
assert any("group:tech@explicit" in s for s in res.route)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_explicit_group_professional(router):
|
||||
res = await router.route("加班费怎么计算", domain_group="professional")
|
||||
assert res.domain == "legal"
|
||||
assert res.domain_group == "professional"
|
||||
assert res.subdomain == "labor"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_explicit_group_restricts_domain(router):
|
||||
# 在 tech 组内问法律问题：组内分类器只认识 code/math → 低置信走最后处理者
|
||||
res = await router.route("劳动合同违约怎么办", domain_group="tech")
|
||||
assert res.upgraded is True
|
||||
assert "direct_fallback" in res.route
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_auto_group_detection(router):
|
||||
# 不指定 group：统一分类器识别 → 自动映射大领域组
|
||||
res = await router.route("基金定投的收益率怎么计算")
|
||||
assert res.domain == "finance"
|
||||
assert res.domain_group == "professional"
|
||||
assert any("group:professional@auto" in s for s in res.route)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_auto_group_tech(router):
|
||||
res = await router.route("求 ∫ x^2 dx 从 0 到 1 的定积分")
|
||||
assert res.domain == "math"
|
||||
assert res.domain_group == "tech"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_invalid_group_raises(router):
|
||||
with pytest.raises(ValueError):
|
||||
await router.route("任意查询", domain_group="不存在的组")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_group_router_is_smaller(router):
|
||||
# 组内分类器只认识组内领域：体积/匹配范围 ≈ 全量 1/4
|
||||
assert len(router._group_classifiers["tech"].domains) == 2
|
||||
assert len(router._group_classifiers["professional"].domains) == 3
|
||||
assert len(router.classifier.domains) == 8
|
||||
assert len(router.domain_groups) == 4
|
||||
|
||||
|
||||
def test_health_shows_groups(router):
|
||||
h = router.health()
|
||||
assert "domain_groups" in h
|
||||
assert "tech" in h["domain_groups"]
|
||||
assert h["domain_groups"]["tech"] == ["code", "math"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_group_with_subdomain2(router):
|
||||
res = await router.route("房贷利率是 LPR 加多少", domain_group="professional")
|
||||
assert res.domain == "finance"
|
||||
assert res.subdomain == "loan"
|
||||
assert res.subdomain2 == "loan"
|
||||
assert "⚠" in res.response # 金融风险免责
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_group_cache_roundtrip(router):
|
||||
q = "加班费怎么计算"
|
||||
r1 = await router.route(q, domain_group="professional")
|
||||
r2 = await router.route(q, domain_group="professional") # 缓存命中
|
||||
assert r2.cache_hit is True
|
||||
assert r2.domain_group == r1.domain_group == "professional"
|
||||
@@ -0,0 +1,167 @@
|
||||
"""新领域（finance/life/education）与子领域（subdomain）测试。"""
|
||||
import pytest
|
||||
|
||||
from router_system.classifier import RuleClassifier
|
||||
from router_system.knowledge import KnowledgeBase
|
||||
from router_system.models import Classification
|
||||
from router_system.planner import Planner
|
||||
from router_system.router import build_router
|
||||
|
||||
|
||||
# ---------------------------------------------------------------
|
||||
# 分类
|
||||
# ---------------------------------------------------------------
|
||||
def test_finance_classification():
|
||||
clf = RuleClassifier()
|
||||
for q in ["基金定投的收益率怎么计算", "房贷利率是多少", "信用卡逾期了怎么办"]:
|
||||
r = clf.classify(q)
|
||||
assert r.domain == "finance", f"{q} -> {r.domain}"
|
||||
|
||||
|
||||
def test_life_classification():
|
||||
clf = RuleClassifier()
|
||||
for q in ["日本旅行攻略", "健身增肌计划", "家常菜谱推荐", "宠物驱虫怎么做"]:
|
||||
r = clf.classify(q)
|
||||
assert r.domain == "life", f"{q} -> {r.domain}"
|
||||
|
||||
|
||||
def test_education_classification():
|
||||
clf = RuleClassifier()
|
||||
for q in ["考研英语怎么备考", "高效学习方法", "面试技巧有哪些"]:
|
||||
r = clf.classify(q)
|
||||
assert r.domain == "education", f"{q} -> {r.domain}"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------
|
||||
# 知识库加载
|
||||
# ---------------------------------------------------------------
|
||||
def test_new_domain_rules_loaded():
|
||||
kb = KnowledgeBase()
|
||||
for d in ("finance", "life", "education"):
|
||||
hits = kb.match("测试", domain=d) # 触发加载检查（domain 存在即可）
|
||||
assert kb.rules_count() >= 60, f"规则总数应 60+，当前 {kb.rules_count()}"
|
||||
# 新领域任务模板
|
||||
assert kb.task_template("finance-advice") is not None
|
||||
assert kb.task_template("life-guide") is not None
|
||||
assert kb.task_template("edu-guide") is not None
|
||||
# 新领域事实表
|
||||
assert len(kb.facts("finance")) >= 5
|
||||
assert len(kb.facts("life")) >= 5
|
||||
assert len(kb.facts("education")) >= 5
|
||||
|
||||
|
||||
# ---------------------------------------------------------------
|
||||
# 拆解
|
||||
# ---------------------------------------------------------------
|
||||
def _plan(query: str, domain: str, difficulty: str = "medium"):
|
||||
c = Classification(domain=domain, confidence=0.9, difficulty=difficulty)
|
||||
return Planner(KnowledgeBase()).plan(query, c)
|
||||
|
||||
|
||||
def test_finance_split_four():
|
||||
g = _plan("基金定投的收益率怎么计算", "finance", "medium")
|
||||
ids = [n.id for n in g.nodes()]
|
||||
assert ids == ["facts", "retrieve", "conclude", "disclaimer"]
|
||||
|
||||
|
||||
def test_life_split_three():
|
||||
g = _plan("日本旅行攻略", "life", "medium")
|
||||
kinds = [n.kind for n in g.nodes()]
|
||||
assert kinds == ["analyze", "advise", "conclude"]
|
||||
|
||||
|
||||
def test_edu_split_three():
|
||||
g = _plan("考研英语怎么备考", "education", "medium")
|
||||
kinds = [n.kind for n in g.nodes()]
|
||||
assert kinds == ["analyze", "design", "conclude"]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------
|
||||
# subdomain / subdomain2（三级子领域）端到端
|
||||
# ---------------------------------------------------------------
|
||||
@pytest.mark.asyncio
|
||||
async def test_subdomain_devops():
|
||||
r = build_router()
|
||||
res = await r.route("git 回滚代码怎么操作")
|
||||
assert res.subdomain == "devops"
|
||||
assert res.subdomain2 == "git"
|
||||
assert any("subdomain:devops" in s for s in res.route)
|
||||
assert any("subdomain2:git" in s for s in res.route)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_subdomain_labor():
|
||||
r = build_router()
|
||||
res = await r.route("加班费怎么计算")
|
||||
assert res.domain == "legal"
|
||||
assert res.subdomain == "labor"
|
||||
assert res.subdomain2 == "labor"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_subdomain_investing():
|
||||
r = build_router()
|
||||
res = await r.route("基金定投的收益率怎么计算")
|
||||
assert res.domain == "finance"
|
||||
assert res.subdomain == "investing"
|
||||
# finance 强制拆解 + 风险免责
|
||||
assert any("plan:multi[4]" in s for s in res.route)
|
||||
assert "⚠" in res.response
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_finance_facts_in_response():
|
||||
r = build_router()
|
||||
res = await r.route("信用卡逾期了怎么办")
|
||||
assert res.domain == "finance"
|
||||
assert "征信" in res.response or "逾期" in res.response
|
||||
|
||||
|
||||
# ---------------------------------------------------------------
|
||||
# 三级子领域（subdomain2）
|
||||
# ---------------------------------------------------------------
|
||||
@pytest.mark.asyncio
|
||||
async def test_subdomain2_sorting():
|
||||
r = build_router()
|
||||
res = await r.route("用 Python 写一个快速排序函数")
|
||||
assert res.domain == "code"
|
||||
assert res.subdomain == "algorithm"
|
||||
assert res.subdomain2 == "sorting"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_subdomain2_first_aid():
|
||||
r = build_router()
|
||||
res = await r.route("烫伤后怎么处理")
|
||||
assert res.domain == "medical"
|
||||
assert res.subdomain == "firstaid"
|
||||
assert res.subdomain2 == "first-aid"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_subdomain2_loan():
|
||||
r = build_router()
|
||||
res = await r.route("房贷利率是 LPR 加多少")
|
||||
assert res.domain == "finance"
|
||||
assert res.subdomain == "loan"
|
||||
assert res.subdomain2 == "loan"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_subdomain2_cache_roundtrip():
|
||||
r = build_router()
|
||||
q = "信用卡逾期了怎么办"
|
||||
r1 = await r.route(q)
|
||||
r2 = await r.route(q) # 缓存命中
|
||||
assert r2.cache_hit is True
|
||||
assert r2.subdomain == r1.subdomain
|
||||
assert r2.subdomain2 == r1.subdomain2
|
||||
assert r2.subdomain2 == "credit"
|
||||
|
||||
|
||||
def test_rule_subdomain2_from_map():
|
||||
kb = KnowledgeBase()
|
||||
r = kb.rule("code-sort")
|
||||
assert r.subdomain2 == "sorting"
|
||||
r2 = kb.rule("medical-firstaid")
|
||||
assert r2.subdomain2 == "first-aid"
|
||||
@@ -0,0 +1,68 @@
|
||||
"""前向链推理机（InferenceEngine）单元测试。"""
|
||||
from router_system.inference import InferenceEngine, render_template
|
||||
from router_system.knowledge import KnowledgeBase, Rule
|
||||
from router_system.memory import WorkingMemory
|
||||
|
||||
|
||||
def _engine():
|
||||
return InferenceEngine(KnowledgeBase())
|
||||
|
||||
|
||||
def test_forward_chain_fires_rules():
|
||||
eng = _engine()
|
||||
mem = WorkingMemory()
|
||||
eng.initialize("求解方程 x^2 - 5x + 6 = 0", "math", "medium", 0.9, mem)
|
||||
fired = eng.run("求解方程 x^2 - 5x + 6 = 0", "math", mem)
|
||||
assert fired, "数学查询应触发规则"
|
||||
# 优先级最高的 math-equation 最先触发
|
||||
assert fired[0] == "math-equation"
|
||||
# 轨迹包含规则记录
|
||||
assert any(t.startswith("rule:") for t in mem.trace)
|
||||
|
||||
|
||||
def test_no_match_terminates():
|
||||
eng = _engine()
|
||||
mem = WorkingMemory()
|
||||
eng.initialize("今天天气怎么样", "general", "medium", 0.5, mem)
|
||||
fired = eng.run("今天天气怎么样", "general", mem)
|
||||
assert fired == []
|
||||
|
||||
|
||||
def test_max_steps_limit():
|
||||
eng = _engine()
|
||||
mem = WorkingMemory()
|
||||
eng.initialize("求解方程 x^2 - 5x + 6 = 0", "math", "medium", 0.9, mem)
|
||||
fired = eng.run("求解方程 x^2 - 5x + 6 = 0", "math", mem, max_steps=1)
|
||||
assert len(fired) == 1
|
||||
|
||||
|
||||
def test_blackboard_facts_written():
|
||||
eng = _engine()
|
||||
mem = WorkingMemory()
|
||||
eng.initialize("求解方程 x^2 - 5x + 6 = 0", "math", "hard", 0.95, mem)
|
||||
assert mem.get_fact("domain") == "math"
|
||||
assert mem.get_fact("difficulty") == "hard"
|
||||
assert "init:" in mem.trace[0]
|
||||
|
||||
|
||||
def test_rule_output_renders_section():
|
||||
kb = KnowledgeBase()
|
||||
# 注入一条带 output 模板的规则
|
||||
kb._rules["test-rule"] = Rule(
|
||||
id="test-rule", domain="general", priority=10,
|
||||
patterns=["测试渲染"], output="领域={facts.domain} 查询={query}",
|
||||
)
|
||||
eng = InferenceEngine(kb)
|
||||
mem = WorkingMemory()
|
||||
eng.initialize("测试渲染一下", "general", "easy", 0.6, mem)
|
||||
fired = eng.run("测试渲染一下", "general", mem)
|
||||
assert "test-rule" in fired
|
||||
section = mem.section("test-rule")
|
||||
assert section is not None
|
||||
assert "领域=general" in section
|
||||
assert "查询=测试渲染一下" in section
|
||||
|
||||
|
||||
def test_render_missing_placeholder():
|
||||
out = render_template("你好 {query} {facts.缺失字段}", "世界", {})
|
||||
assert "你好 世界 [未提供]" == out
|
||||
@@ -0,0 +1,59 @@
|
||||
"""知识库（KnowledgeBase）单元测试。"""
|
||||
from router_system.knowledge import KnowledgeBase, Rule
|
||||
|
||||
|
||||
def test_builtin_rules_loaded():
|
||||
kb = KnowledgeBase()
|
||||
# 内置规则 + 文件规则合并后应至少有 10+ 条
|
||||
assert kb.rules_count() >= 10
|
||||
# 文件规则按 id 覆盖内置（code-sort 存在且可命中）
|
||||
r = kb.rule("code-sort")
|
||||
assert r is not None
|
||||
assert r.template == "code-implement"
|
||||
|
||||
|
||||
def test_match_priority_order():
|
||||
kb = KnowledgeBase()
|
||||
hits = kb.match("用 Python 写一个快速排序函数", domain="code")
|
||||
assert hits, "应命中 code 领域规则"
|
||||
# 优先级降序：排序规则(90) 应在通用实现规则(50) 之前
|
||||
assert hits[0].id == "code-sort"
|
||||
priorities = [h.priority for h in hits]
|
||||
assert priorities == sorted(priorities, reverse=True)
|
||||
|
||||
|
||||
def test_match_cross_domain():
|
||||
kb = KnowledgeBase()
|
||||
# 同一文本在不同领域过滤下命中不同规则
|
||||
code_hits = kb.match("帮我调试报错 TypeError", domain="code")
|
||||
assert code_hits and code_hits[0].id == "code-debug"
|
||||
# 领域过滤：不指定 domain 时全领域匹配
|
||||
all_hits = kb.match("帮我调试报错 TypeError")
|
||||
assert len(all_hits) >= len(code_hits)
|
||||
|
||||
|
||||
def test_task_template():
|
||||
kb = KnowledgeBase()
|
||||
tpl = kb.task_template("code-implement")
|
||||
assert tpl is not None
|
||||
steps = tpl["steps"]
|
||||
assert len(steps) == 4
|
||||
ids = [s["id"] for s in steps]
|
||||
assert ids == ["analyze", "design", "implement", "verify"]
|
||||
# 依赖关系存在
|
||||
assert steps[1]["deps"] == ["analyze"]
|
||||
|
||||
|
||||
def test_facts():
|
||||
kb = KnowledgeBase()
|
||||
legal_facts = kb.facts("legal")
|
||||
assert any("竞业" in f["statement"] for f in legal_facts)
|
||||
medical_facts = kb.facts("medical")
|
||||
assert any("高血压" in f["statement"] for f in medical_facts)
|
||||
|
||||
|
||||
def test_rule_matches_case_insensitive():
|
||||
r = Rule(id="t", domain="code", patterns=["Sort", "Python"])
|
||||
assert r.matches("用 python 写一个 sort 算法")
|
||||
assert r.matches("PYTHON 快速排序")
|
||||
assert not r.matches("java 写个函数")
|
||||
@@ -0,0 +1,130 @@
|
||||
"""知识库扩充验证测试（规则/模板/事实表/output 规则端到端）。"""
|
||||
import pytest
|
||||
|
||||
from router_system.classifier import RuleClassifier
|
||||
from router_system.knowledge import KnowledgeBase
|
||||
from router_system.models import Classification
|
||||
from router_system.planner import Planner
|
||||
|
||||
|
||||
def _kb():
|
||||
return KnowledgeBase()
|
||||
|
||||
|
||||
def _plan(query: str, domain: str, difficulty: str = "medium"):
|
||||
c = Classification(domain=domain, confidence=0.9, difficulty=difficulty)
|
||||
return Planner(_kb()).plan(query, c)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------
|
||||
# 规则 / 模板 / 事实表规模
|
||||
# ---------------------------------------------------------------
|
||||
def test_expanded_rules_count():
|
||||
kb = _kb()
|
||||
assert kb.rules_count() >= 40, f"知识库应扩充至 40+ 条规则，当前 {kb.rules_count()}"
|
||||
|
||||
|
||||
def test_new_task_templates_exist():
|
||||
kb = _kb()
|
||||
required = ["code-algorithm", "code-refactor", "code-explain", "code-test",
|
||||
"math-proof", "math-optimize", "medical-firstaid", "general-writing"]
|
||||
for tid in required:
|
||||
assert kb.task_template(tid) is not None, f"缺少任务模板 {tid}"
|
||||
|
||||
|
||||
def test_expanded_facts():
|
||||
kb = _kb()
|
||||
legal = kb.facts("legal")
|
||||
medical = kb.facts("medical")
|
||||
assert len(legal) >= 10
|
||||
assert len(medical) >= 10
|
||||
# 新事实条目存在
|
||||
legal_text = " ".join(f["statement"] for f in legal)
|
||||
assert "150%" in legal_text # 加班费
|
||||
assert "押金" in legal_text # 租房押金
|
||||
assert "无理由" in legal_text # 消费者退货
|
||||
assert "继承" in legal_text # 继承顺序
|
||||
medical_text = " ".join(f["statement"] for f in medical)
|
||||
assert "烫伤" in medical_text # 烫伤急救
|
||||
assert "抗生素" in medical_text # 抗生素
|
||||
assert "失眠" in medical_text # 失眠
|
||||
|
||||
|
||||
# ---------------------------------------------------------------
|
||||
# 新拆解行为
|
||||
# ---------------------------------------------------------------
|
||||
def test_algorithm_split_five():
|
||||
g = _plan("用动态规划实现背包问题", "code", "hard")
|
||||
ids = [n.id for n in g.nodes()]
|
||||
assert ids == ["analyze", "design", "implement", "complexity", "verify"]
|
||||
|
||||
|
||||
def test_refactor_split_three():
|
||||
g = _plan("这段代码重复太多，帮我重构", "code", "hard")
|
||||
ids = [n.id for n in g.nodes()]
|
||||
assert ids == ["analyze", "refactor", "verify"]
|
||||
|
||||
|
||||
def test_code_explain_split():
|
||||
g = _plan("帮我解释这段代码什么意思", "code", "medium")
|
||||
kinds = [n.kind for n in g.nodes()]
|
||||
assert kinds == ["analyze", "explain", "verify"]
|
||||
|
||||
|
||||
def test_testcase_split():
|
||||
g = _plan("给这个函数写单元测试", "code", "medium")
|
||||
kinds = [n.kind for n in g.nodes()]
|
||||
assert "testcase" in kinds
|
||||
|
||||
|
||||
def test_math_proof_and_optimize():
|
||||
g = _plan("证明勾股定理", "math", "hard")
|
||||
assert len(g) == 3
|
||||
g2 = _plan("求函数 f(x)=x^2 的最小值", "math", "medium")
|
||||
kinds = [n.kind for n in g2.nodes()]
|
||||
assert "optimize" in kinds
|
||||
|
||||
|
||||
def test_firstaid_split():
|
||||
g = _plan("烫伤后怎么处理", "medical", "medium")
|
||||
kinds = [n.kind for n in g.nodes()]
|
||||
assert kinds == ["analyze", "advise", "disclaimer"]
|
||||
|
||||
|
||||
def test_writing_split():
|
||||
g = _plan("写一封请假邮件", "general", "medium")
|
||||
kinds = [n.kind for n in g.nodes()]
|
||||
assert kinds == ["analyze", "draft", "polish"]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------
|
||||
# output 知识规则端到端
|
||||
# ---------------------------------------------------------------
|
||||
@pytest.mark.asyncio
|
||||
async def test_git_knowledge_in_response():
|
||||
from router_system.router import build_router
|
||||
r = build_router()
|
||||
res = await r.route("git 回滚代码怎么操作")
|
||||
assert "git 核心概念" in res.response
|
||||
assert "git revert" in res.response
|
||||
# 推理轨迹记录规则触发
|
||||
assert any("code-git-knowledge" in s for s in res.route)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_docker_knowledge_in_response():
|
||||
from router_system.router import build_router
|
||||
r = build_router()
|
||||
res = await r.route("docker 部署一个服务")
|
||||
assert "镜像 vs 容器" in res.response
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_medical_hypertension_full_flow():
|
||||
from router_system.router import build_router
|
||||
r = build_router()
|
||||
res = await r.route("高血压患者可以吃哪些降压药，副作用是什么")
|
||||
assert res.domain == "medical"
|
||||
assert any("plan:multi[3]" in s for s in res.route)
|
||||
assert "⚠" in res.response
|
||||
assert res.upgraded is False
|
||||
@@ -0,0 +1,68 @@
|
||||
"""NodeExecutor 接口抽象测试（T1：整体项目部分拆解·先行实现）。"""
|
||||
import pytest
|
||||
|
||||
from router_system.executors import (ModelNodeExecutor, RuleNodeExecutor,
|
||||
build_node_executor)
|
||||
from router_system.experts import build_expert_pool
|
||||
from router_system.knowledge import KnowledgeBase
|
||||
from router_system.memory import TaskNode, WorkingMemory
|
||||
from router_system.router import build_router
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_rule_node_executor_deterministic():
|
||||
ex = RuleNodeExecutor(kb=KnowledgeBase())
|
||||
node = TaskNode(id="t1", kind="solve", domain="math", query="求解方程 x^2-5x+6=0")
|
||||
mem = WorkingMemory()
|
||||
r1 = await ex.execute(node, "math", "medium", mem)
|
||||
r2 = await ex.execute(node, "math", "medium", mem)
|
||||
assert r1.text == r2.text # 确定性
|
||||
assert "求解" in r1.text
|
||||
assert r1.model_used.startswith("rule:")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_model_node_executor_uses_expert_pool():
|
||||
experts = build_expert_pool({}, ["code", "math", "legal", "medical",
|
||||
"finance", "life", "education", "general"])
|
||||
ex = ModelNodeExecutor(experts)
|
||||
node = TaskNode(id="t1", kind="implement", domain="code", query="写一个函数")
|
||||
mem = WorkingMemory()
|
||||
resp = await ex.execute(node, "code", "medium", mem)
|
||||
assert resp.text # mock 专家输出
|
||||
assert "mock" in resp.text
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_model_node_executor_fallback_to_general():
|
||||
# 未知领域 → general 专家兜底
|
||||
experts = {"general": build_expert_pool({}, ["general"])["general"]}
|
||||
ex = ModelNodeExecutor(experts)
|
||||
node = TaskNode(id="t1", kind="explain", domain="unknown", query="测试")
|
||||
resp = await ex.execute(node, "unknown", "easy", WorkingMemory())
|
||||
assert resp.text
|
||||
|
||||
|
||||
def test_factory_rule():
|
||||
ex = build_node_executor("rule", kb=KnowledgeBase())
|
||||
assert isinstance(ex, RuleNodeExecutor)
|
||||
|
||||
|
||||
def test_factory_model():
|
||||
experts = build_expert_pool({}, ["code"])
|
||||
ex = build_node_executor("api", experts=experts)
|
||||
assert isinstance(ex, ModelNodeExecutor)
|
||||
|
||||
|
||||
def test_factory_invalid():
|
||||
with pytest.raises(ValueError):
|
||||
build_node_executor("bad-backend")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_router_uses_node_executor():
|
||||
r = build_router()
|
||||
assert r.node_executor.name == "rule"
|
||||
res = await r.route("求解方程 x^2 - 5x + 6 = 0")
|
||||
assert res.domain == "math"
|
||||
assert res.response
|
||||
@@ -0,0 +1,63 @@
|
||||
"""Planner（任务拆解）单元测试。"""
|
||||
from router_system.classifier import RuleClassifier
|
||||
from router_system.knowledge import KnowledgeBase
|
||||
from router_system.planner import Planner
|
||||
|
||||
|
||||
def _plan(query: str):
|
||||
clf = RuleClassifier()
|
||||
c = clf.classify(query)
|
||||
return Planner(KnowledgeBase()).plan(query, c), c
|
||||
|
||||
|
||||
def test_simple_task_no_split():
|
||||
graph, c = _plan("2 + 2 等于多少")
|
||||
# easy 难度 → 单节点不拆
|
||||
assert c.domain == "math"
|
||||
assert c.difficulty == "easy"
|
||||
assert len(graph) == 1
|
||||
n = graph.nodes()[0]
|
||||
assert n.id == "solve"
|
||||
assert n.kind == "solve"
|
||||
|
||||
|
||||
def test_code_task_split_four():
|
||||
graph, c = _plan("用 Python 写一个快速排序函数，并解释时间复杂度")
|
||||
assert c.domain == "code"
|
||||
assert len(graph) == 4
|
||||
ids = [n.id for n in graph.nodes()]
|
||||
assert ids == ["analyze", "design", "implement", "verify"]
|
||||
|
||||
|
||||
def test_math_task_split_three():
|
||||
graph, c = _plan("求解方程 x^2 - 5x + 6 = 0")
|
||||
assert c.domain == "math"
|
||||
assert len(graph) == 3
|
||||
ids = [n.id for n in graph.nodes()]
|
||||
assert ids == ["conditions", "solve", "verify"]
|
||||
|
||||
|
||||
def test_deps_wired():
|
||||
graph, _ = _plan("用 Python 写一个快速排序函数，并解释时间复杂度")
|
||||
nodes = {n.id: n for n in graph.nodes()}
|
||||
assert nodes["design"].deps == ["analyze"]
|
||||
assert nodes["implement"].deps == ["design"]
|
||||
assert nodes["verify"].deps == ["implement"]
|
||||
|
||||
|
||||
def test_no_template_falls_back_to_single():
|
||||
graph, c = _plan("你好呀")
|
||||
# general 无模板命中（"你好"不在任何 pattern）→ 单节点
|
||||
assert c.domain == "general"
|
||||
assert len(graph) == 1
|
||||
n = graph.nodes()[0]
|
||||
assert n.kind == "explain"
|
||||
|
||||
|
||||
def test_explain_plan_trace():
|
||||
graph, _ = _plan("求解方程 x^2 - 5x + 6 = 0")
|
||||
trace = Planner(KnowledgeBase()).explain_plan(graph)
|
||||
assert trace and "plan:multi[3]" in trace[0]
|
||||
single_graph, _ = _plan("1 + 1 = ?")
|
||||
st = Planner(KnowledgeBase()).explain_plan(single_graph)
|
||||
assert "plan:single" in st[0]
|
||||
@@ -0,0 +1,112 @@
|
||||
"""专家系统内核端到端测试（L0 模式：零参数全链路）。"""
|
||||
import pytest
|
||||
|
||||
from router_system.knowledge import KnowledgeBase
|
||||
from router_system.router import Router, build_router
|
||||
from router_system.experts import build_expert_pool
|
||||
from router_system.judge import build_judge
|
||||
from router_system.fallback import build_fallback
|
||||
from router_system.cache import RouterCache
|
||||
from router_system.planner import Planner
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def es_router():
|
||||
return build_router()
|
||||
|
||||
|
||||
def _hybrid_router(expert_backend: str = "api"):
|
||||
"""构造 expert_backend=api 的路由（专家池为 mock，验证切换路径）。"""
|
||||
config = {"router": {"judge_fallback_threshold": 0.70},
|
||||
"execution": {"expert_backend": expert_backend}}
|
||||
kb = KnowledgeBase()
|
||||
classifier = build_router().classifier
|
||||
experts = build_expert_pool({}, ["code", "math", "legal", "medical", "general"])
|
||||
judge = build_judge({"type": "rule"}, 0.70, kb=kb)
|
||||
fallback = build_fallback({"type": "mock"})
|
||||
return Router(classifier, experts, judge, fallback, RouterCache(),
|
||||
None, config, kb=kb, planner=Planner(kb))
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_hybrid_backend_uses_expert_pool():
|
||||
r = _hybrid_router("api")
|
||||
res = await r.route("用 Python 写一个快速排序函数，并解释时间复杂度")
|
||||
# 子任务由专家池执行（mock 专家），而非规则执行器
|
||||
assert res.response
|
||||
assert any(s.startswith("analyze:") for s in res.route) or "plan:multi" in " ".join(res.route)
|
||||
# 专家池 mock 输出包含"mock"标识
|
||||
assert "mock" in res.response or "关键点" in res.response
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_dag_route_code(es_router):
|
||||
r = await es_router.route("用 Python 写一个快速排序函数，并解释时间复杂度")
|
||||
assert r.domain == "code"
|
||||
assert r.response
|
||||
# 拆解轨迹：plan:multi[4] 与 4 个节点执行
|
||||
assert any("plan:multi[4]" in step for step in r.route)
|
||||
node_steps = [s for s in r.route if s.startswith("analyze:") or s.startswith("design:")
|
||||
or s.startswith("implement:") or s.startswith("verify:")]
|
||||
assert len(node_steps) == 4
|
||||
# 合并响应包含各步骤产出
|
||||
assert "【code 分析】" in r.response
|
||||
assert "【code 自检】" in r.response
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_dag_route_math(es_router):
|
||||
r = await es_router.route("求解方程 x^2 - 5x + 6 = 0")
|
||||
assert r.domain == "math"
|
||||
assert any("plan:multi[3]" in step for step in r.route)
|
||||
assert "【math 求解】" in r.response
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_single_node_easy(es_router):
|
||||
# code/easy 高置信 → 单节点不拆
|
||||
r = await es_router.route("python 排序")
|
||||
assert r.upgraded is False
|
||||
assert any("plan:single" in step for step in r.route)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_deterministic_output(es_router):
|
||||
q = "用 Python 写一个快速排序函数，并解释时间复杂度"
|
||||
r1 = await es_router.route(q)
|
||||
r2 = await es_router.route(q)
|
||||
# 第二次缓存命中（响应一致）
|
||||
assert r1.response == r2.response
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_legal_retrieves_facts(es_router):
|
||||
r = await es_router.route("劳动合同到期不续签，公司需要支付经济补偿吗")
|
||||
assert r.domain == "legal"
|
||||
# retrieve 步骤命中知识库事实（竞业/经济补偿类）
|
||||
assert "知识检索" in r.response or "经济补偿" in r.response
|
||||
# 免责提示存在
|
||||
assert "⚠" in r.response
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_medical_disclaimer(es_router):
|
||||
r = await es_router.route("高血压患者日常饮食需要注意什么")
|
||||
assert r.domain == "medical"
|
||||
assert "⚠" in r.response
|
||||
assert any("warning:disclaimer" in s or s.startswith("warning:") for s in r.route)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_low_confidence_direct_fallback(es_router):
|
||||
r = await es_router.route("今天天气怎么样")
|
||||
assert r.upgraded is True
|
||||
assert "direct_fallback" in r.route
|
||||
|
||||
|
||||
def test_health_es(es_router):
|
||||
h = es_router.health()
|
||||
assert h["status"] == "ok"
|
||||
assert h["execution_mode"] == "rule"
|
||||
assert h["rules"] > 0
|
||||
assert "planner" in h
|
||||
@@ -0,0 +1,158 @@
|
||||
"""RouterArena 适配器最小测试。
|
||||
|
||||
覆盖：
|
||||
- BaseRouter 接口契约：返回模型名必须在 config.models
|
||||
- 路由决策：code→gpt-4o-mini、math→claude-3-haiku、低置信度→mistral-medium
|
||||
- 预测文件格式：包含 RouterArena 必填字段
|
||||
- Arena Score 公式与官方一致
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
# 让 tests/ 目录能找到项目根
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
|
||||
|
||||
from research.routerarena.adapter import ( # noqa: E402
|
||||
DOMAIN_TO_MODEL_SLOT,
|
||||
ESCALATION_MODEL_SLOT,
|
||||
ESExpertRouter,
|
||||
LOW_CONFIDENCE_THRESHOLD,
|
||||
)
|
||||
from research.routerarena.base_router import BaseRouter # noqa: E402
|
||||
from research.routerarena.local_runner import ( # noqa: E402
|
||||
MODEL_PRICING,
|
||||
build_mock_dataset,
|
||||
compute_arena_score,
|
||||
)
|
||||
|
||||
|
||||
CONFIG_PATH = str(
|
||||
Path(__file__).resolve().parent.parent
|
||||
/ "research"
|
||||
/ "routerarena"
|
||||
/ "config"
|
||||
/ "es-expert.json"
|
||||
)
|
||||
|
||||
|
||||
def test_base_router_loads_config():
|
||||
"""BaseRouter 应能加载 config 并提取 models 列表。"""
|
||||
r = ESExpertRouter(router_name="es-expert", config_path=CONFIG_PATH)
|
||||
assert "gpt-4o-mini" in r.models
|
||||
assert "claude-3-haiku-20240307" in r.models
|
||||
assert "mistral-medium" in r.models
|
||||
assert len(r.models) == 5
|
||||
|
||||
|
||||
def test_prediction_in_models():
|
||||
"""所有 get_prediction 返回值必须在 config.models 中。"""
|
||||
r = ESExpertRouter(router_name="es-expert", config_path=CONFIG_PATH)
|
||||
samples = [
|
||||
"用 Python 写一个快速排序函数",
|
||||
"Solve x^2 = 4",
|
||||
"Is a non-compete clause enforceable?",
|
||||
"高血压患者日常饮食",
|
||||
"How to calculate ROI on a fund",
|
||||
"Why is the sky blue",
|
||||
]
|
||||
for q in samples:
|
||||
m = r.get_prediction(q)
|
||||
assert m in r.models, f"{q} -> {m} not in {r.models}"
|
||||
|
||||
|
||||
def test_domain_to_slot_mapping_evidence():
|
||||
"""映射表与 00_integration_plan.md §3.2 严格一致。"""
|
||||
assert DOMAIN_TO_MODEL_SLOT["code"] == "gpt-4o-mini"
|
||||
assert DOMAIN_TO_MODEL_SLOT["math"] == "claude-3-haiku-20240307"
|
||||
assert DOMAIN_TO_MODEL_SLOT["legal"] == "claude-3-haiku-20240307"
|
||||
assert DOMAIN_TO_MODEL_SLOT["medical"] == "claude-3-haiku-20240307"
|
||||
assert DOMAIN_TO_MODEL_SLOT["finance"] == "claude-3-haiku-20240307"
|
||||
assert DOMAIN_TO_MODEL_SLOT["life"] == "gemini-2.0-flash-001"
|
||||
assert DOMAIN_TO_MODEL_SLOT["education"] == "deepseek-chat"
|
||||
assert DOMAIN_TO_MODEL_SLOT["general"] == "gemini-2.0-flash-001"
|
||||
|
||||
|
||||
def test_escalation_logic():
|
||||
"""_decide 在低置信度或低质量时升级到 ESCALATION_MODEL_SLOT。"""
|
||||
r = ESExpertRouter(router_name="es-expert", config_path=CONFIG_PATH)
|
||||
# 高置信度 + 高质量 → 主映射
|
||||
assert r._decide("code", 0.95, 0.97) == "gpt-4o-mini"
|
||||
# 低置信度 → 升级
|
||||
assert r._decide("code", 0.50, 0.97) == ESCALATION_MODEL_SLOT
|
||||
# 低质量 → 升级
|
||||
assert r._decide("code", 0.95, 0.50) == ESCALATION_MODEL_SLOT
|
||||
# 未知 domain → general
|
||||
assert r._decide("unknown_xyz", 0.95, 0.97) == "gemini-2.0-flash-001"
|
||||
|
||||
|
||||
def test_compute_arena_score_matches_formula():
|
||||
"""与 RouterArena run.py L65-87 公式逐项核对（0-1 标度）。"""
|
||||
# 已知：Hybrid Router $0.04/1K, 71.38% acc, leaderboard arena=72.08
|
||||
# 官方公式 raw 范围是 [0, 1]，leaderboard 显示 ×100 标度
|
||||
s = compute_arena_score(0.04, 0.7138, beta=0.1, c_max=200, c_min=0.0044)
|
||||
# raw 期望：~0.7208（×100 = leaderboard 的 72.08）
|
||||
assert 0.715 < s < 0.725, f"expected ~0.7208, got {s:.4f}"
|
||||
# 同时验证 leaderboard 标度对齐
|
||||
assert 71.5 < s * 100 < 72.5, f"×100 期望 ~72.08, got {s*100:.4f}"
|
||||
|
||||
|
||||
def test_compute_arena_score_clamping():
|
||||
"""成本超 c_max/c_min 应被 clamp。"""
|
||||
s1 = compute_arena_score(0.005, 0.7)
|
||||
s2 = compute_arena_score(0.01, 0.7)
|
||||
# 越便宜分越高（在范围内）
|
||||
assert s1 > s2 > 0
|
||||
# cost = c_max 时 C=0, S=0（公式定义）
|
||||
s_max = compute_arena_score(200, 0.7)
|
||||
assert s_max == 0.0
|
||||
# cost 被 clamp 到 c_min
|
||||
s_under = compute_arena_score(0.001, 0.7)
|
||||
s_at_min = compute_arena_score(0.0044, 0.7)
|
||||
assert abs(s_under - s_at_min) < 1e-9
|
||||
|
||||
|
||||
def test_mock_dataset_protocol():
|
||||
"""mock 数据集符合 RouterArena 协议字段。"""
|
||||
data = build_mock_dataset()
|
||||
assert len(data) >= 80 # 接近 sub_10 的 809
|
||||
for entry in data:
|
||||
assert "global index" in entry
|
||||
assert "prompt" in entry
|
||||
assert "prompt_formatted" in entry
|
||||
assert "domain" in entry # 仅本地诊断字段
|
||||
# prompt 非空
|
||||
assert len(entry["prompt"]) > 0
|
||||
|
||||
|
||||
def test_prediction_file_schema():
|
||||
"""预测文件应包含 RouterArena 必填字段。"""
|
||||
from research.routerarena.local_runner import run_local
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
summary = run_local(
|
||||
source="mock",
|
||||
router_name="es-expert",
|
||||
config_path=CONFIG_PATH,
|
||||
do_mock_inference=True,
|
||||
output_dir=tmp,
|
||||
)
|
||||
# summary 含关键字段
|
||||
assert summary["n_queries"] >= 80
|
||||
assert "routing_distribution" in summary
|
||||
assert "mock_accuracy" in summary
|
||||
assert "arena_score_mock" in summary
|
||||
# 预测文件存在 + 格式合规
|
||||
with open(summary["prediction_file"], "r", encoding="utf-8") as f:
|
||||
preds = json.load(f)
|
||||
for p in preds:
|
||||
assert "global index" in p
|
||||
assert "prompt" in p
|
||||
assert "prediction" in p
|
||||
assert "for_optimality" in p
|
||||
assert p["prediction"] in MODEL_PRICING or p["prediction"] is None
|
||||
@@ -0,0 +1,164 @@
|
||||
"""Skill 技能体系 + RouteAgent 测试（T12：Agent-Skill 路由器·先行实现）。"""
|
||||
import pytest
|
||||
|
||||
from router_system.agent import RouteAgent
|
||||
from router_system.classifier import RuleClassifier
|
||||
from router_system.fallback import build_fallback
|
||||
from router_system.judge import build_judge
|
||||
from router_system.knowledge import KnowledgeBase
|
||||
from router_system.planner import Planner
|
||||
from router_system.skills import (FallbackSkill, JudgeSkill, KBAnswerSkill,
|
||||
KBRetrieveSkill, Skill, SkillContext,
|
||||
SkillRegistry, TemplateSkill,
|
||||
build_skill_registry)
|
||||
|
||||
|
||||
def _registry():
|
||||
kb = KnowledgeBase()
|
||||
judge = build_judge({"type": "rule"}, 0.70, kb=kb)
|
||||
fallback = build_fallback({"type": "mock"})
|
||||
return build_skill_registry(kb=kb, judge=judge, fallback=fallback)
|
||||
|
||||
|
||||
def _agent():
|
||||
kb = KnowledgeBase()
|
||||
clf = RuleClassifier()
|
||||
return RouteAgent(
|
||||
classifier=clf, planner=Planner(kb), kb=kb,
|
||||
judge=build_judge({"type": "rule"}, 0.70, kb=kb),
|
||||
fallback=build_fallback({"type": "mock"}),
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------
|
||||
# Skill 注册表
|
||||
# ---------------------------------------------------------------
|
||||
def test_registry_registers_builtins():
|
||||
reg = _registry()
|
||||
names = reg.list()
|
||||
assert "es.analyze" in names
|
||||
assert "es.solve" in names
|
||||
assert "es.disclaimer" in names
|
||||
assert "kb.retrieve" in names
|
||||
assert "kb.answer" in names
|
||||
assert "judge.evaluate" in names
|
||||
assert "fallback.call" in names
|
||||
assert len(names) >= 20 # 17 模板 + kb×2 + judge + fallback
|
||||
|
||||
|
||||
def test_registry_duplicate_rejected():
|
||||
reg = SkillRegistry()
|
||||
|
||||
class S(Skill):
|
||||
name = "x"
|
||||
|
||||
reg.register(S())
|
||||
with pytest.raises(ValueError):
|
||||
reg.register(S())
|
||||
|
||||
|
||||
def test_registry_unknown_skill():
|
||||
reg = _registry()
|
||||
with pytest.raises(KeyError):
|
||||
import asyncio
|
||||
asyncio.run(reg.execute("no.such.skill", SkillContext(
|
||||
query="q", domain="general", difficulty="easy", memory=None)))
|
||||
|
||||
|
||||
def test_catalog_shape():
|
||||
reg = _registry()
|
||||
catalog = reg.catalog()
|
||||
assert catalog and all({"name", "description", "params"} <= set(c) for c in catalog)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------
|
||||
# 技能执行
|
||||
# ---------------------------------------------------------------
|
||||
@pytest.mark.asyncio
|
||||
async def test_template_skill_executes():
|
||||
reg = _registry()
|
||||
kb = KnowledgeBase()
|
||||
from router_system.memory import WorkingMemory
|
||||
text = await reg.execute("es.solve", SkillContext(
|
||||
query="求解方程 x^2-5x+6=0", domain="math", difficulty="medium",
|
||||
memory=WorkingMemory(), kb=kb))
|
||||
assert "求解" in text
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_kb_retrieve_skill():
|
||||
reg = _registry()
|
||||
from router_system.memory import WorkingMemory
|
||||
text = await reg.execute("kb.retrieve", SkillContext(
|
||||
query="加班费怎么计算", domain="legal", difficulty="easy",
|
||||
memory=WorkingMemory(), kb=KnowledgeBase()))
|
||||
assert "150%" in text # 命中加班费知识条目
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_kb_answer_skill():
|
||||
reg = _registry()
|
||||
from router_system.memory import WorkingMemory
|
||||
text = await reg.execute("kb.answer", SkillContext(
|
||||
query="git 回滚代码怎么操作", domain="code", difficulty="easy",
|
||||
memory=WorkingMemory(), kb=KnowledgeBase()))
|
||||
assert "git 核心概念" in text
|
||||
|
||||
|
||||
# ---------------------------------------------------------------
|
||||
# RouteAgent 端到端（无用户指定：完全自主分析 + 技能调用）
|
||||
# ---------------------------------------------------------------
|
||||
@pytest.mark.asyncio
|
||||
async def test_agent_code_flow_multi_skills():
|
||||
agent = _agent()
|
||||
res = await agent.route("用 Python 写一个快速排序函数，并解释时间复杂度")
|
||||
assert res.domain == "code"
|
||||
assert res.response
|
||||
# 多技能调用链：4 个 es.* 技能
|
||||
skill_steps = [s for s in res.route if s.startswith("skill:es.")]
|
||||
assert len(skill_steps) == 4
|
||||
assert any("es.analyze" in s for s in skill_steps)
|
||||
assert any("es.implement" in s for s in skill_steps)
|
||||
assert res.request_id
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_agent_legal_uses_retrieve_and_disclaimer():
|
||||
agent = _agent()
|
||||
res = await agent.route("加班费怎么计算")
|
||||
assert res.domain == "legal"
|
||||
# 技能组合：kb.retrieve（法条）+ es.disclaimer（免责）
|
||||
assert any("skill:kb.retrieve" in s for s in res.route)
|
||||
assert any("skill:es.disclaimer" in s for s in res.route)
|
||||
assert "150%" in res.response
|
||||
assert "⚠" in res.response
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_agent_low_confidence_fallback():
|
||||
agent = _agent()
|
||||
res = await agent.route("今天天气怎么样")
|
||||
assert res.upgraded is True
|
||||
assert "direct_fallback" in res.route
|
||||
assert "fallback" in res.model_used or "mock" in res.model_used
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_agent_no_user_domain_group():
|
||||
# Agent 模式：不要求 domain_group，完全自主分析
|
||||
agent = _agent()
|
||||
res = await agent.route("基金定投的收益率怎么计算")
|
||||
assert res.domain == "finance"
|
||||
assert res.subdomain == "investing"
|
||||
assert res.domain_group is None # 无用户指定概念
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_agent_trace_and_catalog():
|
||||
agent = _agent()
|
||||
res = await agent.route("求解方程 x^2 - 5x + 6 = 0")
|
||||
trace = agent.trace_store.get(res.request_id)
|
||||
assert trace is not None
|
||||
assert trace["domain"] == "math"
|
||||
catalog = agent.skills_catalog()
|
||||
assert any(s["name"] == "kb.retrieve" for s in catalog)
|
||||
+1
-1
@@ -4,7 +4,7 @@
|
||||
<meta charset="UTF-8" />
|
||||
<link rel="icon" type="image/svg+xml" href="/favicon.svg" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<title>端云协同 LLM 协作系统</title>
|
||||
<title>端云协同编程智能体系统</title>
|
||||
</head>
|
||||
<body>
|
||||
<div id="app"></div>
|
||||
|
||||
+1
-1
@@ -6,7 +6,7 @@
|
||||
<div class="brand-logo">🤖</div>
|
||||
<div class="brand-text">
|
||||
<b>端云协同</b>
|
||||
<span>LLM 协作系统</span>
|
||||
<span>编程智能体系统</span>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
@@ -1,2 +0,0 @@
|
||||
| T28 | 操作审批流：approval_policy(off/dangerous/all) + 挂起/裁决/超时 + /approve + 审批卡 | ✅ 完成 | T28 |
|
||||
| T29 | token 级流式：SSE 解析/tool_calls 碎片组装/回退 + DeltaThrottle + 打字机渲染 | ✅ 完成 | T29 |
|
||||
@@ -1,13 +0,0 @@
|
||||
|
||||
## 9. 增补：审批流与流式输出（D9/D10，T28/T29）
|
||||
|
||||
- **决策 D9（操作审批，对齐 dsh）**：`agent.approval_policy` = off | dangerous（默认：
|
||||
写/编辑/命令询问，只读自动放行）| all；工具执行前经 approval_hook 挂起等待，
|
||||
UI 两按钮（拒绝 / 允许一次），超时 `approval_timeout_s`（默认 120s）自动拒绝
|
||||
（fail-closed）；拒绝结果回喂模型可改道；approval_request/decided 事件对入审计。
|
||||
- **决策 D10（token 级流式）**：OpenAICompatChat 默认 stream=True（SSE 逐段解析，
|
||||
tool_calls 碎片按 index 组装、不在正文展示；include_usage 计量）；流式失败自动
|
||||
回退非流式一次（已有部分增量输出时如实抛出）；ToolLoop 经 on_delta 转发，
|
||||
DeltaThrottle ≥48 字符节流落 delta 事件；前端打字机式实时渲染 + 光标。
|
||||
- **配套修复**：Vue 响应式丢失 bug——闭包持有 push 前原始对象导致过程事件不渲染，
|
||||
改取响应式代理对象。
|
||||
+21
-1
@@ -61,7 +61,7 @@ P0 完成后的能力：干净的后端抽象 + 可量化的评测 + 可追溯
|
||||
|
||||
---
|
||||
|
||||
## 五、v2 任务登记（端云协同 LLM 协作系统，见《实现方案_v2_端云协同LLM协作系统.md》）
|
||||
## 五、v2 任务登记（端云协同编程智能体系统，见《实现方案_v2_端云协同编程智能体系统.md》）
|
||||
|
||||
> 每个任务一个 commit（`feat(v2): Tn 描述`），交付含封闭单测；v1 的 126 项测试保持全绿。
|
||||
|
||||
@@ -124,3 +124,23 @@ P0 完成后的能力：干净的后端抽象 + 可量化的评测 + 可追溯
|
||||
| T29 | token 级流式：SSE 解析/tool_calls 碎片组装/回退 + DeltaThrottle + 打字机渲染 | ✅ 完成 | T29 |
|
||||
| T30 | 安全加固（Mimosa 深度扫描驱动，D11）：路径 ID 白名单/artifacts 与工件名关押/下载 dest 关押+协议白名单/config 密钥打码/Host 信任围栏+回环绑定/危险命令独立拦截/CSPRNG 抽样 | ✅ 完成 | T30 |
|
||||
| T31 | dsh 功能对齐（D12）：LLM 重试退避/web_fetch 工具（SSRF 防护）/原子写入/慢工具线程卸载/search 目录修剪/重复调用提醒/会话重命名 | ✅ 完成 | T31 |
|
||||
|
||||
---
|
||||
|
||||
## 六、代理层任务登记（AI代理功能开发：校园 AI 代理层+缓存层）
|
||||
|
||||
> 执行版：《AI代理功能开发/实施方案_代理层与缓存层.md》（T-P0…T-P8，含 DDL/接口签名/验收命令，约 12 工作日）。
|
||||
> 设计与经济测算：《AI代理功能开发/方案_校园AI代理层.md》。约束：`router_system/` 零改动；
|
||||
> M1/M2 零新依赖；总验收 = 压测报告 h_g+h_p ≥ 50% 且统一 5 折毛利为正、账目零不一致。
|
||||
|
||||
| T | 内容 | 状态 | commit |
|
||||
|---|------|------|--------|
|
||||
| T-P0 | 骨架：gateway/proxy/ 包 + SQLite DDL + enabled 门控挂路由 | ⬜ 待办 | |
|
||||
| T-P1 | 鉴权+账本：key 签发/令牌桶/四表/request_id 幂等/日限额 | ⬜ 待办 | |
|
||||
| T-P2 | 上游客户端：流式派发+三家 usage 归一化+首 token 前 failover | ⬜ 待办 | |
|
||||
| T-P3 | 计价+结算：峰谷窗口/毫元整数/黄金用例 ≥10 组 | ⬜ 待办 | |
|
||||
| T-P4 | 路由端到端：/proxy/v1 非流式+流式+402/429/413 语义 | ⬜ 待办 | |
|
||||
| T-P5 | 规范化+桶：稳定哈希/整形顺序/doc_version 失效/多轮不缓存 | ⬜ 待办 | |
|
||||
| T-P6 | 语义缓存：L1 精确+L2 n-gram 倒排+singleflight+TTL+失败不缓存 | ⬜ 待办 | |
|
||||
| T-P7 | 管理面+前端：stats/ledger 端点+ProxyView 三卡片 | ⬜ 待办 | |
|
||||
| T-P8 | 压测+预算：200 并发流式；P99≤50ms/内存≤1GB/账目零不一致 | ⬜ 待办 | |
|
||||
|
||||
@@ -0,0 +1,378 @@
|
||||
# 可行性调研与落地实现路线报告
|
||||
|
||||
> 项目：多专业小模型 + 路由模型系统（MVP）
|
||||
> 报告日期：2026-08-12
|
||||
> 依据：项目内《可行性分析报告》《实现方案》/ 13 篇 arXiv 论文 / 2026 最新业界动态 / 当前代码库实测
|
||||
|
||||
---
|
||||
|
||||
## 一、TL;DR（先看结论）
|
||||
|
||||
1. **可行性：★★★★☆（强烈建议实施）**。技术、成本、生态三个维度证据充分：
|
||||
- 路由已成为独立学科与独立服务类别，低成本路由器在 RouterArena 排行榜上以 $0.04/1K 查询成本全面超越 GPT-5（$10.02/1K）；
|
||||
- 小模型 + 路由替代大模型已有多篇顶会实证（The Avengers/AAAI 2025、R2R/NeurIPS 2025、StR/EMNLP 2025）；
|
||||
- 本项目当前 MVP 已跑通全链路（20 项测试通过、demo 正常、分类 100%/15 条样例）。
|
||||
2. **核心问题答案：不需要全新架构（推倒重来），但需要"分层演进"**。
|
||||
- 现有架构骨架（缓存 → 分类 → 专家池 → Judge → 回退）与 2026 最新综述归纳的六种路由范式、与 Not Diamond / OrcaRouter / vLLM-SR 等生产路由产品完全同构；
|
||||
- 代码抽象（`Base*` 接口 + 配置驱动）已经为演进预留了接口，**换实现不换架构**；
|
||||
- 需要变化的是**每一层"内部实现"**（规则 → 训练模型、Mock → 真实模型、启发式 → LLM-as-Judge）和**部署形态**（单体进程 → 推理服务 + 网关），这两者属于"升级"而非"重写"。
|
||||
3. **什么情况下才需要全新架构**（见第四章）：多租户高并发（QPS≥100）、流式输出、分布式多机推理、模型热加载、多模态/长上下文。若这些成为硬需求，给出三套备选方案与迁移路径（第五章）。
|
||||
4. **推荐落地路线**（见第六章）：5 个阶段、约 14–20 周，从"完全本地真实链路"起步（不依赖任何 API key），到"本地小模型推理 + 训练分类器 + embedding 缓存 + RouterArena 评测 + 生产化"。
|
||||
|
||||
---
|
||||
|
||||
## 二、项目现状盘点（2026-08-12 实测）
|
||||
|
||||
### 2.1 已完成（对应实现方案第一阶段 MVP）
|
||||
|
||||
| 能力 | 状态 | 实测结果 |
|
||||
|------|------|---------|
|
||||
| 全链路路由（缓存→分类→专家→Judge→回退） | ✅ 可运行 | `scripts/demo.py` exit 0，平均延迟 ~14ms |
|
||||
| 5 领域意图分类（code/math/legal/medical/general） | ✅ 规则分类器 | 15 条评测样例准确率 100% |
|
||||
| 两阶段缓存（L1 精确 + L2 n-gram 语义） | ✅ 零依赖 | eval 缓存命中率 40% |
|
||||
| Judge 质量控制器 + 升级机制 | ✅ 启发式 | 升级率 20%（达标 ≤20%） |
|
||||
| FastAPI 网关（/chat /health /metrics） | ✅ | 20 项单元测试全部通过 |
|
||||
| 论文调研（13 篇 PDF + 2026 survey） | ✅ | `research/2026_papers_survey.md` |
|
||||
|
||||
### 2.2 当前短板（决定路线图的输入）
|
||||
|
||||
| 短板 | 现状 | 与目标的差距 |
|
||||
|------|------|-------------|
|
||||
| 分类器 | 关键词规则，仅 15 条样例验证 | 目标 ≥95% 且可扩展；规则在更大数据上会掉点 |
|
||||
| 专家池 | 全部 Mock 模板输出 | 无真实推理，成本/延迟/质量均未验证 |
|
||||
| Judge | 启发式打分（覆盖度/长度/格式） | 无法评估事实性与逻辑正确性 |
|
||||
| 缓存 | 字符 n-gram 相似度 | 语义质量弱，应升级为 embedding 检索 |
|
||||
| 回退层 | Mock | 升级为本地 7B 量化模型（DeepSeek-R1-Distill-Qwen-7B）兜底，完全本地 |
|
||||
| 评测 | 自建 15 条迷你基准 | 未接入 RouterArena 标准化评测 |
|
||||
| 路由决策 | 仅"领域+置信度" | 无预算感知（R2-Router）、无级联深度控制 |
|
||||
| 工程小问题 | eval.py 控制台 GBK 崩溃、test_cache.py 中文损坏 | 见第七章修复清单 |
|
||||
|
||||
---
|
||||
|
||||
## 三、可行性调研结论
|
||||
|
||||
### 3.1 技术可行性：证据充分（2025–2026 最新）
|
||||
|
||||
**（1）"小模型集合 + 路由"已被顶会反复验证**
|
||||
|
||||
- **The Avengers（AAAI 2025）**：轻量框架聚合小模型集体智能，路由+评分+投票在数学/代码/逻辑任务超越专有大模型——本项目最直接的理论支撑（论文已在仓库：`2505.19797v3.pdf`）。
|
||||
- **R2R（NeurIPS 2025）**：token 级路由，DeepSeek R1-1.5B + R1-32B 平均激活仅 5.6B 即超越 R1-14B，速度 2.8×。
|
||||
- **Select-then-Route / StR（EMNLP 2025 Industry）**：先按语义类目选模型子池再级联路由——**与本项目"分类器 + 专家池"结构完全一致**，直接验证了当前架构。
|
||||
- **Doing More with Less 综述（arXiv 2502.00409）**：query 级路由 64.3% vs 领域式 52.2%，支持"逐查询路由"而非"按领域分区"。
|
||||
|
||||
**（2）2026 年路由已从"方法"变成"独立服务类别"**
|
||||
|
||||
- 最新综述 [Dynamic Model Routing and Cascading（2603.04445）](https://huggingface.co/papers/2603.04445) 将路由研究归纳为六种范式（难度感知、级联、预算约束、多轮、多模态、token 级），本项目覆盖了其中四类（难度感知、级联、缓存、回退）。
|
||||
- [LLM Routers Have Become a Service Category of Their Own](https://techstrong.ai/articles/llm-routers-have-become-a-service-category-of-their-own/)（2026）：Not Diamond、Martian、OpenRouter 等已将路由产品化；[vLLM Semantic Router](https://vllm-sr.ai/blog/vllm-sr-fusion-api/) 将语义路由内置进主流推理框架——说明**路由不是实验性玩具，而是生产级基础设施**。
|
||||
- [OrcaRouter（2605.30736）](https://arxiv-org.ezproxy.obspm.fr/html/2605.30736v1)：生产导向的混合离线-在线学习路由器，验证了"规则 + 数据驱动 + 在线反馈"的渐进式升级路线（与本报告路线一致）。
|
||||
|
||||
**（3）必须正视的新风险（2026 年新增）**
|
||||
|
||||
- [When Routing Collapses（2602.03478）](https://huggingface.co/papers/2602.03478)：训练路由器时可能出现**退化收敛**——路由器坍缩到"总选同一个模型"，丧失区分度。影响：**本项目第一阶段用"可解释的规则/阈值路由"起步是正确选择**；当引入学习型路由器时（路线图阶段 4+），需监控路由选择熵，避免过早用数据训练路由器。
|
||||
- [Mixture of Parrots（ICLR 2025）](https://arxiv.org/abs/2410.19034)：任意数量的小专家有表征容量天花板，**必须保留大模型回退层**——本项目回退设计是必要的，不是可选项。
|
||||
|
||||
### 3.2 经济可行性：成本降低 80–96% 有实测支撑
|
||||
|
||||
| 方案 | 平均激活参数 | 相对成本 | 来源 |
|
||||
|------|-------------|---------|------|
|
||||
| 单一稠密大模型（70B） | 70B | 1.0× | 基线 |
|
||||
| MoE（激活 20B） | 20B | 0.29× | — |
|
||||
| **本项目路由系统（80% 小模型 + 20% 升级）** | **~3B** | **0.04–0.15×** | 可行性分析 §6.1 |
|
||||
| [RouteLLM 基准实测](https://klymentiev.com/blog/llm-router) | — | 0.15–0.70×（**成本降 30–85%**） | 2026 业界实测 |
|
||||
| [RouterArena 排行（2026-07）](https://huggingface.co/blog/JerryPotter/who-routes-the-routers) | — | Hybrid Router $0.04/1K vs GPT-5 $10.02/1K | 榜单实测 |
|
||||
|
||||
按实现方案 §4.2 估算（日均 10 万请求）：全量大模型 ¥50,000–80,000/月 vs 路由系统 ¥5,000–12,000/月，**节省 80–90%**。量越大优势越明显。
|
||||
|
||||
### 3.3 理论边界（必须承认的"墙"）
|
||||
|
||||
| 边界 | 说明 | 本项目应对 |
|
||||
|------|------|-----------|
|
||||
| 复杂推理 | 数学证明、逻辑链、多步规划仍需大模型 | 级联升级：仅 10–20% 请求升级（已实现，升级率 20%） |
|
||||
| 表征容量 | 专家宽度有下限（建议 ≥1B） | 配置已用 0.5B–7B 专家，符合下限 |
|
||||
| 能力密度 3.5 月翻倍 | 模型会快速过时 | 模型注册表 + 定期替换（实现方案 §4.2） |
|
||||
|
||||
### 3.4 综合评级（更新 2026 视角）
|
||||
|
||||
| 维度 | 评级 | 说明 |
|
||||
|------|------|------|
|
||||
| 技术可行性 | ★★★★★ | 顶会 + 商业产品双验证 |
|
||||
| 成本效益 | ★★★★★ | 30–85%（保守）~ 85–96%（乐观） |
|
||||
| 部署复杂度 | ★★★☆☆ | 真实模型接入是主要工程量 |
|
||||
| 推理天花板 | ★★★★☆ | 回退层兜底，可控 |
|
||||
| 生态成熟度 | ★★★★☆ | vLLM/RouterArena/评测框架已就绪 |
|
||||
| **综合** | **★★★★☆** | **强烈建议实施** |
|
||||
|
||||
---
|
||||
|
||||
## 四、核心问题：需不需要全新架构？
|
||||
|
||||
### 4.1 结论：不需要推倒重来，需要"分层演进"
|
||||
|
||||
**判断依据（为什么现有骨架是对的）：**
|
||||
|
||||
1. **结构与业界同构**：2026 综述（2603.04445）的六范式、StR 的两阶段结构、Not Diamond / OrcaRouter / vLLM-SR 的组件划分，与当前代码的 `Router → (Cache / Classifier / Experts / Judge / Fallback / Stats)` 完全对应。重写等于把已经验证正确的骨架再写一遍。
|
||||
2. **接口抽象已为演进预留**：`BaseClassifier` / `Expert` / `BaseJudge` / `FallbackProvider` 四个抽象接口 + `build_*` 工厂 + `config.yaml` 驱动。替换任何一层实现都不需要改 Router 主流程（20 项测试保证回归安全）。
|
||||
3. **成本不对称**：推倒重来 = 丢掉已验证的 20 项测试、评测基线（缓存 40%、升级率 20%、分类 100%）、以及 5 个领域的规则语料；而演进每步都可验证、可回滚。
|
||||
4. **现有代码量小且整洁**：核心 `router_system/` 仅 ~50KB，无历史包袱、无耦合陷阱，不值得"重构"（重构解决的是"坏味道"，这里没有）。
|
||||
|
||||
**需要变化的两件事（注意这不是"新架构"，是"换零件"）：**
|
||||
|
||||
| 层 | 现在 | 演进为 | 方式 |
|
||||
|----|------|--------|------|
|
||||
| 分类器 | 规则关键词 | 训练的小模型分类器（BERT / Qwen3-0.6B） | 换 `build_classifier` 实现，接口不变 |
|
||||
| 专家池 | Mock 模板 | 真实 LoRA 微调专家（vLLM 推理） | 换 `build_expert` 实现，接口不变 |
|
||||
| Judge | 启发式 | LLM-as-Judge / 规则+模型混合 | 换 `build_judge` 实现，接口不变 |
|
||||
| 缓存 L2 | n-gram 相似度 | embedding 向量检索（BGE / text-embedding） | 换 `RouterCache` 内部实现 |
|
||||
| 回退层 | Mock | 本地 7B 量化模型（DeepSeek-R1-Distill-Qwen-7B Q4，按需加载） | 新加 `build_fallback` 本地后端；`type: api` 保留为可选（默认关闭） |
|
||||
| 部署形态 | 单体进程 | 网关进程 + vLLM/llama.cpp 推理服务 + 向量库 | 新增部署层，不改核心代码 |
|
||||
|
||||
> **一句话：换的是"层内实现"和"部署拓扑"，不是"分层架构"本身。**
|
||||
|
||||
### 4.2 什么情况下才需要"全新架构"（触发条件）
|
||||
|
||||
出现以下任一**硬需求**时，单体演进方案不够用，需要系统形态升级（注意：即便如此，核心路由逻辑仍可复用，不是"从头写"）：
|
||||
|
||||
| # | 触发条件 | 具体信号 | 需要的架构形态 |
|
||||
|---|---------|---------|---------------|
|
||||
| 1 | 多租户 / 高并发生产服务 | 目标 QPS ≥ 100，需要 SLA、限流、租户隔离 | 无状态网关集群 + 独立推理服务 |
|
||||
| 2 | 流式输出（SSE/WebSocket） | 用户要求"打字机"效果，长文档生成 | 网关 → 推理服务全链路流式透传 |
|
||||
| 3 | 分布式多机推理 | 专家池超过单卡容量（>24GB），或需要并发多专家 | vLLM 推理集群 + 模型分发 |
|
||||
| 4 | 模型热加载 / 热替换 | 灰度发布、模型更新不停服 | 推理服务与路由解耦 + 版本路由 |
|
||||
| 5 | 多模态 / 长上下文 | 输入含图像/音频，或 128K+ 上下文 | 新增模态处理层、上下文工程 |
|
||||
| 6 | 大规模语义缓存共享 | 多实例缓存一致性 | 外部向量数据库（Redis/FAISS/Qdrant） |
|
||||
|
||||
> 判断方法：**先问"是不是必须"，再问"现在能不能不做"**。当前阶段（内部工具 / 限定领域 / 单机）以上 6 条都不是硬需求，因此**不触发**全新架构。建议在阶段 5 复查此表。
|
||||
|
||||
---
|
||||
|
||||
## 五、如果要用全新架构：三种方案详解
|
||||
|
||||
> 本节回答"怎么用"。三套方案按"改动量从小到大"排列，都基于同一原则：**保留 Router 核心逻辑（它是对的），重构外围与部署形态**。推荐方案 A。
|
||||
|
||||
### 方案 A：演进式分层架构（推荐 —— 当前形态的自然升级）
|
||||
|
||||
**形态**：单体代码库 + 进程拆分（网关与推理分离），仍是一个应用，但推理交给独立服务。
|
||||
|
||||
```
|
||||
┌─────────────┐ ┌─────────────────────────────┐
|
||||
│ 客户端/调用方 │────▶│ Router Gateway (FastAPI) │
|
||||
└─────────────┘ │ · 缓存(L1精确 + L2向量检索) │
|
||||
│ · 分类器(训练模型, 已替换规则) │
|
||||
│ · 路由决策 + 预算感知 │
|
||||
│ · Judge(规则+LLM混合) │
|
||||
│ · 升级/回退决策 │
|
||||
└──────┬──────────┬──────────┬──┘
|
||||
▼ ▼ ▼
|
||||
┌──────────┐ ┌──────────┐ ┌──────────────┐
|
||||
│ vLLM 服务 │ │ vLLM 服务 │ │ 本地 7B 兜底 │
|
||||
│ 专家池 │ │ Judge模型 │ │ (Q4量化, 按需加载)│
|
||||
│(多模型并发)│ │(1-3B) │ └──────────────┘
|
||||
└──────────┘ └──────────┘
|
||||
```
|
||||
|
||||
**落地步骤（每步可独立上线）：**
|
||||
1. 保持 `router_system/` 原样，新增 `serving/` 目录：`vllm_engine.py`（封装 OpenAI 兼容推理服务调用）。
|
||||
2. `experts.py` 的 `APIExpert` 指向本地 `http://localhost:8001/v1`（vLLM 启动多个 LoRA 专家的 OpenAI 兼容端点），**一行配置切换**：`experts.code.type: api`。
|
||||
3. 缓存 L2 升级：`RouterCache` 增加 embedding 后端抽象（`SemanticStore` 接口），默认 n-gram 实现保留，可选 `BGE` 向量实现（用 `sentence-transformers` 或调用本地 embedding 服务）。
|
||||
4. 分类器替换：训练后输出模型文件，`build_classifier` 增加 `type: trained`，加载本地 ONNX/transformers 模型。
|
||||
5. 网关增加流式透传（若需要）：`/chat/stream` 走 SSE。
|
||||
|
||||
**优点**：改动可控（每步 1–3 天）、逐步验证、可回滚；代码与测试资产全部保留。
|
||||
**缺点**：仍是"一个仓库一个应用"，多实例共享缓存需要外部存储（阶段 5 处理）。
|
||||
|
||||
---
|
||||
|
||||
### 方案 B：事件驱动微服务（生产 SaaS / 多租户时采用）
|
||||
|
||||
**形态**：路由决策与模型推理完全解耦为独立服务，通过消息队列异步编排。
|
||||
|
||||
```
|
||||
┌────────────┐ ┌─────────────┐
|
||||
Client ──▶ API Gateway ──▶ Kafka/RabbitMQ ──▶ Router Worker (决策)
|
||||
(限流/鉴权) │ ▲ │
|
||||
▼ │ ▼
|
||||
Redis Cache ◀───────────────┘ Expert Workers (vLLM池)
|
||||
(共享语义缓存) Judge Workers
|
||||
│
|
||||
▼
|
||||
Fallback (外部API)
|
||||
```
|
||||
|
||||
**关键设计决策：**
|
||||
| 决策点 | 选择 | 理由 |
|
||||
|--------|------|------|
|
||||
| 通信 | 同步（短任务）/ 异步队列（长任务）混合 | 80% 查询 <2s 走同步；升级/复杂任务走异步 |
|
||||
| 路由状态 | 无状态 Worker + Redis 缓存 | 水平扩展、故障恢复 |
|
||||
| 模型 | 每个专家一个 vLLM 实例（或单实例多 LoRA） | LoRA 切换成本低，GPU 利用率高 |
|
||||
| 观测 | OpenTelemetry 全链路 trace | 路由决策可审计（生产刚需） |
|
||||
| 评测 | RouterArena 离线流水线 + 在线采样评估 | 防路由退化（见 3.1 风险 3） |
|
||||
|
||||
**优点**：水平扩展、租户隔离、故障域隔离、可审计。
|
||||
**缺点**：工程量 4–8 周；引入 MQ/Redis 运维成本；对当前单机阶段是过度设计。
|
||||
|
||||
---
|
||||
|
||||
### 方案 C：拥抱推理框架生态（"不自己造轮子"路线）
|
||||
|
||||
**形态**：放弃自研路由决策的某些部分，用生态组件拼装：
|
||||
|
||||
| 组件 | 可选生态 | 说明 |
|
||||
|------|---------|------|
|
||||
| 推理 | vLLM / SGLang / llama.cpp / Ollama | 全支持 OpenAI 兼容 API |
|
||||
| 语义路由 | [vLLM Semantic Router](https://vllm-sr.ai/blog/vllm-sr-fusion-api/) | 把路由内置到推理框架 |
|
||||
| 路由器 | [RouteLLM](https://klymentiev.com/blog/llm-router)（开源） / Not Diamond（商业） | 数据驱动路由，含阈值校准 |
|
||||
| 评测 | [RouterArena](https://github.com/RouteWorks/RouterArena) | 标准 5 维评测 |
|
||||
| 缓存 | Redis + embedding（BGE / text-embedding-3） | 共享语义缓存 |
|
||||
|
||||
**适用场景**：团队规模小、不想维护推理细节；或希望直接使用成熟路由器的校准阈值。
|
||||
**注意**：vLLM-SR / RouteLLM 主要做"模型选择路由"，不解决本项目"领域专家 + Judge + 回退"的业务编排——**生态组件应作为"层内实现"嵌入方案 A，而不是替代 A**。
|
||||
|
||||
---
|
||||
|
||||
### 三方案对比
|
||||
|
||||
| 维度 | A 演进式（推荐） | B 微服务 | C 生态拼装 |
|
||||
|------|-----------------|---------|-----------|
|
||||
| 触发条件 | 现在即可（无硬需求） | QPS≥100 / 多租户 / 流式 | 团队小 / 想用现成 |
|
||||
| 改动量 | 每步 1–3 天 | 4–8 周 | 2–4 周 |
|
||||
| 保留现有代码 | 全部 | 路由核心逻辑 | 大部分（接口适配） |
|
||||
| 可扩展性 | 中（单机→双机） | 高 | 中高 |
|
||||
| 运维复杂度 | 低 | 高 | 中 |
|
||||
| 风险 | 低 | 中（分布式一致性问题） | 中（依赖第三方演进） |
|
||||
| 与路线图关系 | 是路线图的实现方式 | 路线图阶段 5 的选项 | 可嵌入 A |
|
||||
|
||||
**结论：先走 A；若业务爆发（触发条件 1/2），在 A 基础上平滑升级到 B 的部署形态；C 的选择性组件（vLLM、RouterArena）始终可用。**
|
||||
|
||||
---
|
||||
|
||||
## 六、推荐落地实现路线（5 个阶段，14–20 周）
|
||||
|
||||
> 每阶段都有独立验收标准，可单独交付。里程碑用 ✅ 标注。
|
||||
|
||||
### 阶段 1：真实链路打通（第 1–2 周）—— 完全本地化起步
|
||||
|
||||
**目标**：修复工程问题、下载本地模型、把 mock 换成真实本地推理，验证全链路真实成本/延迟/质量。**本阶段起即完全本地，不需要任何 API key。**
|
||||
|
||||
| 任务 | 产出 | 验收 |
|
||||
|------|------|------|
|
||||
| 修复工程问题（见 §7） | eval.py 可在 GBK 控制台运行；test_cache.py 中文修复 | 全部测试通过 |
|
||||
| 下载本地模型权重 | Qwen3-0.6B（分类）、BGE-small-zh（embedding）、Qwen2.5-Coder-7B / Qwen3-4B（专家）、DeepSeek-R1-Distill-Qwen-7B（回退，Q4 量化） | 离线可加载，无需联网 |
|
||||
| 回退层接入本地 7B | `build_fallback` 新本地后端（llama.cpp / vLLM 量化加载） | 低置信查询走本地 7B 返回真实回答 |
|
||||
| 专家池接入本地推理 | `experts.*.type: hf` 指向本地模型（或本地 OpenAI 兼容服务） | 5 领域各 2 条查询真实回答 |
|
||||
| 建立成本/延迟基线 | `scripts/eval.py` 记录真实 cost_est 与延迟 | 输出报告：升级率 / 平均成本 / 延迟 |
|
||||
|
||||
**验收标准（对齐实现方案 5.1，完全本地口径）**：端到端延迟 < 大模型 1.5×；升级率 ≤20%；全链路离线运行，无任何外部 API 依赖。
|
||||
|
||||
### 阶段 2：本地小模型推理（第 3–6 周）—— 真正降本的主体
|
||||
|
||||
**目标**：本地 vLLM/llama.cpp 跑 0.6B–7B 开源模型池（专家 + 回退），全链路离线，无任何 API 依赖。
|
||||
|
||||
| 任务 | 产出 | 验收 |
|
||||
|------|------|------|
|
||||
| 安装推理栈 | `pip install -r requirements-ml.txt` + vLLM（或 llama.cpp 量化） | RTX 4060 8GB 可跑 Qwen3-0.6B / Qwen2.5-Coder-7B 量化 |
|
||||
| 单实例多模型 | vLLM 启动多模型（`--served-model-name` 区分）或单实例多 LoRA | 5 领域专家并发可用 |
|
||||
| 配置切换 | `experts.*.type: hf` / 本地服务 `base_url: http://127.0.0.1:8001/v1` | 全链路本地推理跑通 |
|
||||
| 延迟/吞吐基线 | 压测脚本（10–50 并发） | P50 < 500ms，P99 < 2s（对齐实现方案 5.2） |
|
||||
| 分类器试跑真实模型 | `HuggingFaceClassifier` 加载 Qwen3-0.6B 或 BERT | 与规则分类器准确率对比（≥90% 则保留，否则回退规则） |
|
||||
|
||||
**验收**：全链路离线可跑（拔网线也能工作）；分类准确率 ≥90%；延迟 P50 < 500ms。
|
||||
|
||||
### 阶段 3：训练分类器 + 升级 Judge（第 5–10 周，与阶段 2 并行）
|
||||
|
||||
**目标**：分类器从规则升级为训练模型（94–97% 目标），Judge 从启发式升级为"规则+LLM 混合"。
|
||||
|
||||
| 任务 | 产出 | 验收 |
|
||||
|------|------|------|
|
||||
| 构建训练数据 | 每领域 500–2000 条（人工标注 + 合成 + 公开数据集） | `data/train.jsonl` / `data/dev.jsonl` 格式齐备 |
|
||||
| 训练分类器 | 跑通 `scripts/train_classifier.py`（BERT 或 Qwen3-0.6B LoRA） | 开发集准确率 ≥94%，推理 <50ms |
|
||||
| 分类器 A/B | 规则 vs 训练模型同评测集对比 | 训练模型胜出才切换（否则保留规则，成本更低） |
|
||||
| Judge 升级 | `RuleJudge` 保留快速过滤 + 增加本地 `LLMJudge`（本地 1–3B） | Judge 与人工评估一致性 ≥90%（实现方案 5.2） |
|
||||
| 阈值校准 | 参考 Conformal Cascade 校准 `judge_fallback_threshold` | 升级率 ≤20% 且有分布无关保证 |
|
||||
|
||||
**验收**：分类准确率 ≥95%（正式目标）；升级率 ≤20%；缓存命中率 ≥30%（对齐实现方案 5.2）。
|
||||
|
||||
### 阶段 4：语义缓存升级 + 标准化评测（第 9–14 周）
|
||||
|
||||
**目标**：L2 缓存从 n-gram 升级为 embedding 检索；接入 RouterArena 标准评测。
|
||||
|
||||
| 任务 | 产出 | 验收 |
|
||||
|------|------|------|
|
||||
| embedding 语义缓存 | `SemanticStore` 抽象 + BGE 实现（`BAAI/bge-small-zh-v1.5`） | 同义改写查询命中率提升 ≥15%（vs n-gram） |
|
||||
| 缓存评估 | 真实流量采样回放 | 缓存命中率 ≥30%（目标 40%+） |
|
||||
| RouterArena 接入 | 按 RouterArena 格式生成预测文件并评测 | 获得 Arena Score 与成本对比基线 |
|
||||
| 引入预算感知（可选进阶） | 参考 R2-Router：根据难度/置信度限制输出 token 预算 | 成本再降 20–30% 且质量 Δ≥0 |
|
||||
|
||||
**验收**：RouterArena 5 维指标全部可量化；成本/质量曲线优于"单一大模型"基线。
|
||||
|
||||
### 阶段 5：生产化（第 12–20 周，部分与阶段 4 并行）
|
||||
|
||||
**目标**：把系统从"能跑"变成"可运维、可监控、可替换"。
|
||||
|
||||
| 任务 | 产出 | 验收 |
|
||||
|------|------|------|
|
||||
| 监控 | Prometheus + Grafana：延迟/升级率/成本/路由分布/选择熵 | 面板上线，路由决策可审计 |
|
||||
| 模型注册表 + 热替换 | 配置文件驱动模型版本；新模型评测通过后灰度切换 | 不停服替换专家（对齐实现方案 5.3） |
|
||||
| 灰度/A-B | 网关按流量比例分流新旧路由器 | 可回滚 |
|
||||
| 复查 §4.2 触发条件 | 若 QPS≥100/多租户 → 按方案 B 升级部署形态 | 架构决策记录 |
|
||||
|
||||
**验收**：整体成本降低 ≥80%（对比"全部请求走云端大模型"基线）；系统可灰度发布；模型可热替换。
|
||||
|
||||
### 时间线总览
|
||||
|
||||
```
|
||||
第1-2周 第3-6周 第5-10周 第9-14周 第12-20周
|
||||
┌────────┐ ┌────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
|
||||
│阶段1 │→ │阶段2 │→ │阶段3 │→ │阶段4 │→ │阶段5 │
|
||||
│真实链路 │ │本地推理 │ │分类器+Judge│ │缓存+评测 │ │生产化 │
|
||||
└────────┘ └────────┘ └──────────┘ └──────────┘ └──────────┘
|
||||
修问题 本地推理 训练/微调 embedding 监控/热替换
|
||||
接本地模型 压测 A/B切换 RouterArena 复查触发条件
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 七、已知工程问题修复清单（建议阶段 1 一并处理）
|
||||
|
||||
| # | 问题 | 位置 | 修复 |
|
||||
|---|------|------|------|
|
||||
| 1 | `eval.py` 打印 `✅` 在 GBK 控制台抛 `UnicodeEncodeError`（已实测复现） | `scripts/eval.py:76` | 脚本开头 `sys.stdout.reconfigure(encoding="utf-8")`，或 emoji 改 ASCII |
|
||||
| 2 | `tests/test_cache.py` 中文被 `?` 替换（文件损坏，已确认字节级） | `tests/test_cache.py:16,17,20,27` | 重写为正确中文查询/注释 |
|
||||
| 3 | `scripts/serve.py --stop` 用 `os.kill(SIGTERM)`，Windows detached 进程可能杀不掉 | `scripts/serve.py:45` | 改用 `taskkill /PID <pid> /T /F` |
|
||||
| 4 | `conftest.py` 注释同样有 `?` 损坏 | `tests/conftest.py` | 同上修复 |
|
||||
|
||||
---
|
||||
|
||||
## 八、风险清单（2026 更新版）
|
||||
|
||||
| 风险 | 概率 | 影响 | 应对 | 阶段 |
|
||||
|------|------|------|------|------|
|
||||
| 路由误分类导致质量下降 | 中 | 高 | 置信度阈值 + 低置信走大模型（已实现）；阶段 3 训练分类器 | 1–3 |
|
||||
| 小模型推理天花板 | 高 | 中 | 级联升级兜底（已实现）；Judge 阈值校准 | 持续 |
|
||||
| **学习型路由器退化收敛（新）** | 低 | 高 | 规则/阈值起步；引入学习路由后监控选择熵（2602.03478） | 4–5 |
|
||||
| 多模型管理复杂度 | 中 | 中 | vLLM 多 LoRA 统一管理 + 模型注册表 | 2,5 |
|
||||
| 本地 GPU 资源不足 | 中 | 高 | 量化（4bit）+ 按需加载；回退降级为 4B 模型 | 2 |
|
||||
| 领域数据不足微调效果差 | 中 | 中 | 合成数据 + few-shot 先验证；规则分类器作为保底 | 3 |
|
||||
| 缓存击穿大模型负载飙升 | 低 | 高 | 限流 + 降级 + 预缓存热门查询 | 5 |
|
||||
| 能力密度变化导致选型过时 | 低 | 低 | 接口抽象化（已具备），模型替换不动路由层 | 持续 |
|
||||
|
||||
---
|
||||
|
||||
## 九、参考文献与资源（2026 更新）
|
||||
|
||||
**项目内已有**：
|
||||
- 《可行性分析报告_多专业小模型+路由模型路径.md》（15 篇论文）
|
||||
- 《实现方案_多专业小模型+路由模型.md》（三阶段 681 行）
|
||||
- `research/2026_papers_survey.md` + `references/` 13 篇 PDF
|
||||
|
||||
**2026 新增引用**：
|
||||
- [Dynamic Model Routing and Cascading: A Survey（2603.04445）](https://huggingface.co/papers/2603.04445) —— 路由六范式综述
|
||||
- [When Routing Collapses（2602.03478）](https://huggingface.co/papers/2602.03478) —— 路由器退化收敛风险
|
||||
- [OrcaRouter: Production-Oriented LLM Router（2605.30736）](https://arxiv-org.ezproxy.obspm.fr/html/2605.30736v1) —— 生产路由器设计
|
||||
- [RouteLLM 基准实测：成本降 30–85%](https://klymentiev.com/blog/llm-router)
|
||||
- [LLM Routers 已成为独立服务类别（Techstrong）](https://techstrong.ai/articles/llm-routers-have-become-a-service-category-of-their-own/)
|
||||
- [vLLM Semantic Router](https://vllm-sr.ai/blog/vllm-sr-fusion-api/)
|
||||
- [RouterArena 代码仓库](https://github.com/RouteWorks/RouterArena) 与 [博客](https://huggingface.co/blog/JerryPotter/who-routes-the-routers)
|
||||
|
||||
---
|
||||
|
||||
*报告完成。核心结论：可行性 ★★★★☆，架构沿用（分层演进），推荐方案 A 路线，5 阶段 14–20 周落地。*
|
||||
@@ -0,0 +1,444 @@
|
||||
# 实现方案 v2：端云协同编程智能体系统（Agent 交接执行版）
|
||||
|
||||
> 编写日期：2026-08-30
|
||||
> 状态：**当前权威规划**。方向上取代 v1《实现方案_多专业小模型+路由模型.md》与《任务拆解与执行计划.md》的路线；
|
||||
> v1 的 L0 专家系统内核**不删除**，降级为离线降级模式与可解释基线（见 5.4）。
|
||||
> 预期读者：负责实现的 AI Agent 或开发者。本文自包含。实现前必读：第 3 节（不得推翻的设计决策）、第 8 节（工程规约）。
|
||||
|
||||
---
|
||||
|
||||
## 0. 一页速览
|
||||
|
||||
**目标**：把仓库现有的 mock 原型升级为可安装、可演示、可量化的真实系统。
|
||||
|
||||
- **大模型（API，默认 DeepSeek）**：只做三件事——开局任务分析（brief）、中途疑难决策（decide）、最终审校（final review）。每次输入压缩到约 1K token 量级。
|
||||
- **小模型（本地 llama.cpp，捆绑分发）**：默认 Qwen3.5-4B（Q4_K_M），负责实现与自验证。
|
||||
- **协作媒介**：「交流文本」——一份 schema 约束的结构化 JSON 共享工作区（第 4 节）。两模型互不共享内部状态，只通过它交接（类比前后端通过 API 契约协作）。
|
||||
- **人工检验**：第三个协作者，复用同一协议，异步队列，不阻塞响应路径。
|
||||
|
||||
**一句话流程**：
|
||||
用户 query →（快路径）小模型直答＋自验证通过即返回 → 否则大模型写 brief → 小模型逐步实现（代码/文件类任务经受限工具集直接读写工作目录、运行命令与测试，类主流编码 agent 循环）＋自验证，失败自修 ≤2 次后写 issue → 大模型按 issue 决策（或兜底代做）→ 循环至满足验收标准 → 大模型终审 → 交付 / 人工检验队列。
|
||||
|
||||
**系统能力定位**：编程智能体（Coding Agent）——代码/文件类任务由 Worker 在独立工作目录中以受限工具集（read/write/edit/run）完成，产物即文件；问答类请求走快路径直答（不启用工具）。
|
||||
**论文题目（已定）**：《基于端云协同的编程智能体系统设计与实现》
|
||||
Design and Implementation of a Programming Agent System Based on Device-Cloud Collaboration
|
||||
关键词：端云协同；编程智能体；大小模型协同；工具调用；级联升级
|
||||
|
||||
**北极星指标**：在端到端质量不低于纯小模型直答的前提下，大模型 token 消耗相对"全量上下文"方案下降 ≥80%（可实测，见第 9 节 E1）。
|
||||
|
||||
**核心经济学原则**：贵的一方（API）少读少写，便宜的一方（本地）多读多干。
|
||||
|
||||
---
|
||||
|
||||
## 1. 背景与现状
|
||||
|
||||
### 1.1 为什么从 v1 转向（决策记录）
|
||||
|
||||
1. **领域分类路线确认放弃**。规则分类器在 9 域平衡集上仅 74.4%、68.9% 请求触发升级
|
||||
（实测见 `research/routerarena/01_results_and_gap_analysis.md`）；原计划的"微调分类器"依赖训练数据与训练环境，不符合当前约束。
|
||||
2. **v1 执行层是模板填充，系统实际不能回答问题**——必须接入真实模型。
|
||||
3. **用户新约束**：本地运行时的长上下文能力、参数量尽可能少、长上下文下低内存占用、
|
||||
模型切换间最大 token 节约。由此设计出"交流文本"协议（第 4 节）。
|
||||
|
||||
### 1.2 现有资产盘点
|
||||
|
||||
| 文件/模块 | 处置 | 说明 |
|
||||
|---|---|---|
|
||||
| `router_system/memory.py`（WorkingMemory/TaskGraph） | **改造** | 黑板思想升级为持久化 workspace；TaskGraph 的依赖处理思路可参考，但 plan 改为简单列表+deps |
|
||||
| `router_system/experts.py`（APIExpert） | **复用** | OpenAI 兼容客户端，llama-server 与 DeepSeek 都能接；需补超时、token 计量、messages 参数化 |
|
||||
| `router_system/fallback.py`（APIFallback） | **复用** | 大模型兜底直连可用 |
|
||||
| `router_system/judge.py`（RuleJudge 的 facts 核对） | **部分复用** | facts 对照逻辑迁入接地验证（5.1 T5）；LLMJudge 退役 |
|
||||
| `router_system/trace.py`（TraceStore） | **复用** | 协作轨迹落点，每回合写入 |
|
||||
| `router_system/cache.py` | **复用** | 快路径结果缓存 |
|
||||
| `router_system/classifier.py`、`planner.py`、`config.yaml` 的 `domain_groups` | **退役出主链路** | 保留在 legacy 路由（`POST /chat/legacy`）与既有测试中 |
|
||||
| `router_system/knowledge.py` + `config/knowledge/*.yaml` | **复用** | facts 表转作验证接地素材（不再用于分类） |
|
||||
| `gateway/api.py` | **扩展** | 新增 `/runs`、`/review` 端点；`/chat` 切到新管线 |
|
||||
| `scripts/eval.py`、`scripts/serve.py`、`scripts/demo.py` | **扩展/新增** | 新增 `scripts/setup_runtime.py`（下载运行时与模型）、`scripts/bench_tokens.py`（实验） |
|
||||
| `research/routerarena/*`、`research/2026_papers_survey.md` | **保留** | 评测方法学与文献引用来源 |
|
||||
| v1 测试套件（126 项） | **必须保持全绿** | legacy 路由不回归（见 D10） |
|
||||
|
||||
---
|
||||
|
||||
## 2. 架构总览
|
||||
|
||||
```
|
||||
用户 query
|
||||
│
|
||||
▼
|
||||
[快路径] Worker 小模型直答 + 自验证通过？ ──是──▶ 直接返回（约 40-50% 请求）
|
||||
│ 否（或自评不确定）
|
||||
▼
|
||||
[Architect·API·一次] 任务分析 → 写 交流文本.brief（goal/constraints/acceptance/plan/tags）
|
||||
│
|
||||
▼
|
||||
┌──────────── 协作循环（护栏内，见 D6）────────────┐
|
||||
│ [Worker·本地] 读 brief＋当前 step 相关上下文 │
|
||||
│ → 实现（写工件 artifacts/） │
|
||||
│ → 自验证（接地优先：跑代码/facts 对照/结构检查） │
|
||||
│ → 通过：写 progress；失败：自修 ≤2 次 │
|
||||
│ → 仍失败：写 issue 条目（增量、带锚点） │
|
||||
│ [Architect·API·按需] 只读 issues＋锚点片段渲染 │
|
||||
│ → 写 decisions / 修订 plan / 兜底代做该 step │
|
||||
└──────────────────────────────────────────────┘
|
||||
│ 所有 step done
|
||||
▼
|
||||
[Architect·API·一次] 终审（读 archive 摘要＋成品关键段）→ done / 打回修正
|
||||
│
|
||||
▼
|
||||
交付响应 + 入人工检验队列（抽样 / 强制规则：brief.tags 含 safety）
|
||||
```
|
||||
|
||||
组件与职责：
|
||||
|
||||
| 组件 | 运行位置 | 职责 |
|
||||
|---|---|---|
|
||||
| `CollaborativePipeline`（编排） | 本进程 | 快路径判定、循环控制、预算熔断、状态机推进 |
|
||||
| `ArchitectClient`（大模型） | DeepSeek API | brief / decide / final_review，全部 JSON Schema 约束输出 |
|
||||
| `WorkerLoop`（小模型） | llama-server 子进程 | 工具化实现（受限工具集操控工作目录）、自验证、自修、写 progress/issue |
|
||||
| `Workspace`（交流文本） | 磁盘 JSON | 唯一共享状态，schema 校验，锚点寻址，rollup 压缩 |
|
||||
| `LlamaServerManager`（运维） | 本进程 | 子进程启停、健康检查、崩溃重启、三档硬件模板 |
|
||||
| `ReviewQueue`（人工检验） | SQLite | 异步队列、审核端点、修正数据留存 |
|
||||
|
||||
---
|
||||
|
||||
## 3. 关键设计决策（实现时**不得擅自推翻**）
|
||||
|
||||
- **D1** llama.cpp 只捆绑上游官方 release 二进制（锁版本，放 `bin/`，gitignore），**禁止修改 llama.cpp 源码或自行编译**。
|
||||
- **D2** 交流文本是**单一 JSON 文档**（`runs/<request_id>/workspace.json`），brief 写入后不可变且恒定位于文档前部（prefix cache 友好）；所有写入必须通过 schema 校验。
|
||||
- **D3** 领域分类器退役出主链路；入口默认**快路径**（小模型先试，省 API 钱），brief 中的 `tags` 仅用于安全标记与验证接地，不做路由。
|
||||
- **D4** 验证分层优先级：**可执行验证（跑代码/跑测试）> facts 对照 > 结构检查 > 模型自由判断（最后手段）**。
|
||||
- **D5** 单 llama-server 实例单模型；默认 Qwen3.5-4B Q4_K_M；三档硬件模板（6.2）；验证器异构（换载 GLM 等）只作为**可选扩展**，MVP 不做。
|
||||
- **D6** 预算护栏硬熔断：`rounds_cap`（协作循环回合上限）、`api_token_cap`（API token 上限）任一触顶即退出循环；熔断行为 = Architect 兜底代做（有 key）或本地降级提示（无 key）。
|
||||
- **D7** Architect 的输入**永不包含工件全文**，只含 meta+issues+decisions+锚点片段，渲染目标 ≤1200 token（`Workspace.render_for_architect`）。
|
||||
- **D8** Windows 优先；核心包 `router_system/` 保持零第三方依赖（纯标准库）；运维层/网关可用 httpx、fastapi、pydantic（均已在 requirements.txt）。
|
||||
- **D9** 所有 LLM 结构化输出必须 JSON Schema 约束：llama-server 用 `response_format: json_schema`（或 GBNF），DeepSeek 用 `response_format: {"type": "json_object"}` + prompt 内嵌 schema；解析失败重试一次，仍失败走降级路径，禁止带病继续。
|
||||
- **D10** v1 的 126 项测试保持全绿；legacy 路由（`POST /chat/legacy`）行为不变，保证离线可跑、测试封闭。
|
||||
- **D11** 新模块的单元测试必须封闭可跑：API 调用用 `httpx.MockTransport` 注入假响应，**不依赖**真实 llama-server 或 API key。
|
||||
- **D12** Worker 工具调用必须受限：工具集固定为 list/read/write/edit/run 五种；作用域仅限 `runs/<id>/` 工作目录内；每 step 工具调用次数上限（默认 20）；run 类命令强制超时并回收子进程；所有调用（含参数摘要与退出码）入 trace。Windows 沙箱以"目录限制＋超时＋进程回收"为准，不承诺禁网（作为 limitation 写明）。
|
||||
|
||||
---
|
||||
|
||||
## 4. 交流文本协议规范
|
||||
|
||||
### 4.1 存储布局
|
||||
|
||||
```
|
||||
runs/<request_id>/
|
||||
├── workspace.json # 交流文本本体（唯一共享状态）
|
||||
└── artifacts/ # Worker 工作目录：工具调用的作用域（实现产物即文件）
|
||||
├── s1_main.py
|
||||
└── s2_tests.py
|
||||
```
|
||||
|
||||
锚点寻址：`a://s2_tests.py#L12-18`，目标即工作目录内文件（Worker 用工具写入、Architect 按锚点取片段，编排层每个 issue 最多附带 30 行）。
|
||||
|
||||
### 4.2 Workspace JSON Schema
|
||||
|
||||
实现为 `router_system/workspace.py` 内嵌常量 `WORKSPACE_SCHEMA`（draft-07），要点如下（字段长度上限同时是 rollup 依据）：
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"version": "1.0",
|
||||
"request_id": "hex12",
|
||||
"query": "用户原始需求（写入后不可变）",
|
||||
"meta": {
|
||||
"status": "draft|in_progress|reviewing|escalated|done|failed",
|
||||
"round": 0,
|
||||
"budget": {"api_input_tokens": 0, "api_output_tokens": 0,
|
||||
"api_token_cap": 8000, "rounds_cap": 6}
|
||||
},
|
||||
"brief": { // Architect 写一次后锁定
|
||||
"goal": "≤500字",
|
||||
"constraints": ["≤8条"],
|
||||
"tags": ["code|math|legal|medical|finance|life|education|general|safety"],
|
||||
"acceptance": [{"id": "a1", "check": "验收标准描述", "machine_checkable": true}],
|
||||
"plan": [{"id": "s1", "task": "≤300字", "deps": ["s0"], "done_criteria": "可观察判据"}] // ≤5 步
|
||||
},
|
||||
"progress": [ {"step": "s1", "status": "done|failed|blocked",
|
||||
"summary": "≤200字", "artifact": "a://s1_main.py"} ],
|
||||
"issues": [ {"id": "i1", "step": "s2", "anchor": "a://s2_main.py#L12-18",
|
||||
"observed": "≤300字", "expected": "≤300字",
|
||||
"tried": "已尝试的修复，≤300字", "ask": "请求架构师裁决的问题"} ],
|
||||
"decisions": [ {"ref": "i1", "reply": "≤600字",
|
||||
"patch_plan": [{"id": "s2", "task": "修订后的任务"}]} ],
|
||||
"archive": [ "s1: 快排实现20行，测试3/3通过（详情 artifacts/s1_main.py）" ] // 每条 ≤160字
|
||||
}
|
||||
```
|
||||
|
||||
### 4.3 状态机
|
||||
|
||||
```
|
||||
draft ──write brief──▶ in_progress ──所有 step done──▶ reviewing ──pass──▶ done
|
||||
│ ▲ └──fail──▶ in_progress（修正回合+1）
|
||||
issue 上报│ │decision 下发 / 兜底代做
|
||||
▼ │
|
||||
（回合数/budget 未触顶）
|
||||
│ 触顶 rounds_cap 或 api_token_cap
|
||||
▼
|
||||
escalated ──Architect 兜底代做──▶ reviewing
|
||||
任何终态 ──▶ 入 ReviewQueue（抽样率或 safety 强制）
|
||||
```
|
||||
|
||||
### 4.4 双方渲染规则（token 节约的核心实现）
|
||||
|
||||
- `render_for_architect(ws)`：meta＋query（截断 200 字）＋**全部 issues**＋最近 3 条 decisions＋最近回合的 progress 摘要＋每个 issue 的锚点片段（≤30 行）。**目标 ≤1200 token，超限先截断最旧 decisions。**
|
||||
- `render_for_worker(ws, step_id)`：brief 全文＋该 step 定义＋依赖 step 的 archive 摘要行＋该 step 现有工件全文（若存在）＋验收标准。**目标 ≤8K token**（小模型长上下文"中间迷失"的补偿：重要信息前置、总量受控）。
|
||||
|
||||
### 4.5 rollup 压缩规则
|
||||
|
||||
- step 标记 done 后，Worker 顺手生成 ≤60 字摘要进 `archive`，工件本体只在 artifacts/ 落盘。
|
||||
- workspace.json 主体超过 4K token 时，编排层强制压缩：将已 done 的 progress 条目折叠为 archive 行、截断已解决的 issues（保留 id 与结论行）。
|
||||
- 归档**只减不删**：压缩前的原文不保留（可控成本换取协议简单），如需审计走 TraceStore。
|
||||
|
||||
### 4.6 校验与容错
|
||||
|
||||
- 所有写入（brief/progress/issues/decisions）先过 schema；失败 → 将原始输出与错误一并回喂该模型重写一次 → 仍失败由编排层做保守修复（截断超长字段、丢弃多余条目）并记录 trace 告警。
|
||||
|
||||
---
|
||||
|
||||
## 5. 组件设计与文件规划
|
||||
|
||||
### 5.1 新增模块（每个含中文 docstring、工厂函数、独立单测）
|
||||
|
||||
| 文件 | 职责与关键接口 | 对应任务 |
|
||||
|---|---|---|
|
||||
| `runtime/__init__.py` | 运维层包 | T2 |
|
||||
| `runtime/hw_profile.py` | `detect() -> {"tier": "gpu12|gpu8|cpu", "ngl": int, "ctx": int, "kv_quant": str}`；探测 nvidia-smi/vulkaninfo/内存，失败回退 cpu 档 | T2 |
|
||||
| `runtime/llama_server.py` | `LlamaServerManager`: `start()/stop()/ensure_alive()/endpoint()`；子进程管理、/health 轮询、指数退避重启、日志落 `runs/llama_server.log` | T2 |
|
||||
| `router_system/workspace.py` | `Workspace`: `new/apply_brief/add_progress/add_issue/add_decision/rollup/render_for_architect/render_for_worker/validate/save`；内嵌 `WORKSPACE_SCHEMA` | T4 |
|
||||
| `router_system/architect.py` | `ArchitectClient`: `brief(query)/decide(ws)/final_review(ws) -> dict`；httpx 调 DeepSeek，JSON 约束输出，token 计量回写 meta.budget | T3 |
|
||||
| `router_system/worker.py` | `WorkerLoop`: `run_step(ws, step) -> StepOutcome`；受限工具循环（list/read/write/edit/run，D12 约束）实现→验证→自修≤2→issue；验证器按 D4 分层：code 域沙箱执行（run 工具跑测试，tempdir、超时）、facts 对照（复用 KnowledgeBase.facts）、结构检查 | T5 |
|
||||
| `router_system/pipeline.py` | `CollaborativePipeline`: `run(query) -> RouterResult`；快路径→循环→熔断→兜底→终审→入审核队列；对接 TraceStore 与 Stats | T6 |
|
||||
| `router_system/review.py` | `ReviewQueue`: SQLite（`data/review.sqlite3`）；`enqueue/get/list/submit(verdict, correction)` | T8 |
|
||||
| `scripts/setup_runtime.py` | 下载 llama-server release（GitHub，提供镜像说明）与默认 GGUF（hf-mirror.com，断点续传、文件大小校验）到 `bin/`、`models/`（均 gitignore） | T11 |
|
||||
| `scripts/bench_tokens.py` | E1 实验脚本（第 9 节） | T12 |
|
||||
|
||||
### 5.2 config/config.yaml v2（新增段，v1 段保留供 legacy 路由）
|
||||
|
||||
```yaml
|
||||
runtime:
|
||||
llama_server:
|
||||
binary: bin/llama-server.exe # 捆绑上游 release，不改源码（D1）
|
||||
model: models/qwen3.5-4b-q4_k_m.gguf
|
||||
port: 8901
|
||||
hw_profile: auto # auto | gpu12 | gpu8 | cpu
|
||||
extra_args: ["-fa", "-ctk", "q8_0", "-ctv", "q8_0", "--cache-reuse", "256"]
|
||||
tiers: # 三档硬件模板（保守默认，可手动覆盖）
|
||||
gpu12: {ngl: 99, ctx: 32768}
|
||||
gpu8: {ngl: 14, ctx: 16384}
|
||||
cpu: {ngl: 0, ctx: 8192}
|
||||
|
||||
architect: # 大模型（API）
|
||||
model: deepseek-chat
|
||||
base_url: https://api.deepseek.com/v1
|
||||
api_key_env: DEEPSEEK_API_KEY
|
||||
temperature: 0.2
|
||||
timeout_s: 60
|
||||
|
||||
worker: # 小模型（本地）
|
||||
backend: llama_server
|
||||
temperature: 0.3
|
||||
max_fix_attempts: 2
|
||||
per_step_timeout_s: 300
|
||||
|
||||
pipeline:
|
||||
fast_path: true
|
||||
rounds_cap: 6
|
||||
api_token_cap: 8000
|
||||
breach_policy: architect_do # architect_do | local_only
|
||||
|
||||
review:
|
||||
queue_db: data/review.sqlite3
|
||||
sample_rate: 0.10 # 随机抽样送审
|
||||
force_tags: [safety] # brief.tags 命中即强制送审
|
||||
```
|
||||
|
||||
### 5.3 网关 API 变更（gateway/api.py）
|
||||
|
||||
| 端点 | 变更 |
|
||||
|---|---|
|
||||
| `POST /chat` | 切到 `CollaborativePipeline`；响应增加 `request_id`、`rounds_used`、`api_tokens`、`fast_path` 字段 |
|
||||
| `POST /chat/legacy` | **新增**，原 v1 L0 行为原样保留（离线降级 + 测试封闭） |
|
||||
| `GET /runs/{request_id}/workspace` | **新增**，返回交流文本（脱敏可选） |
|
||||
| `GET /runs/{request_id}/artifacts/{name}` | **新增**，下载工件 |
|
||||
| `GET /review/queue`、`POST /review/{id}` | **新增**，人工检验（verdict: approve/edit/reject ＋ correction 文本） |
|
||||
| `GET /metrics` | 扩展：api tokens 累计、fast_path 命中率、回合数分布、熔断次数 |
|
||||
|
||||
### 5.4 退役/降级清单
|
||||
|
||||
- `classifier.py`、`planner.py`、`experts.py` 的 MockExpert/HFExpert、v1 `router.py`：整体归入 **legacy 路由**，由 `build_router()` 构造，仅 `POST /chat/legacy` 与既有测试使用；`domain_groups` 配置保留不动。
|
||||
- `judge.py` 的 `LLMJudge` 类：标记 deprecated，不再被 v2 引用。
|
||||
- 知识库 facts：`KnowledgeBase.facts(domain)` 保留，作为 Worker 接地验证素材；domain 来源改为 `brief.tags`。
|
||||
|
||||
---
|
||||
|
||||
## 6. 模型与运行时
|
||||
|
||||
### 6.1 选型结论（2026-08 时点；调研依据见 `research/2026_papers_survey.md` 与对话期检索记录）
|
||||
|
||||
| 角色 | 首选 | 备选 | 关键理由 |
|
||||
|---|---|---|---|
|
||||
| 本地 Worker（默认） | **Qwen3.5-4B Q4_K_M**（权重约 2.5GB，262K 上下文） | Qwen3.5-2B（低配）/ Qwen3.5-9B（高配） | IFEval 89.8（结构化输出可靠性=协议生命线）；官方端侧定位；GGUF 生态现成 |
|
||||
| Architect | deepseek-chat（API） | 任意 OpenAI 兼容 | 便宜、支持 json_object、有上下文缓存计费优惠 |
|
||||
|
||||
已知注意项：Qwen3.5 小模型**加 few-shot 示例反而掉分**（社区实测 0.8B 零样本 67% → 单示例 33%），所有 prompt 写清晰零样本指令，禁止堆示例。
|
||||
|
||||
### 6.2 llama-server 启动模板（三档硬件）
|
||||
|
||||
```
|
||||
# gpu12（≥12GB 显存）
|
||||
llama-server -m models/qwen3.5-4b-q4_k_m.gguf --port 8901 -ngl 99 -c 32768 -fa -ctk q8_0 -ctv q8_0 --cache-reuse 256
|
||||
# gpu8（≈8GB 显存）：-ngl 14 -c 16384
|
||||
# cpu（无独显，≥16GB 内存）：-ngl 0 -c 8192（提示用户速度受限）
|
||||
```
|
||||
|
||||
原则：`-c` 按需设定（不要拉满 262K，KV 预留会吃光内存）；KV 量化验证任务不用 q4（精度影响裁判）；`--cache-reuse` 配合"brief 恒定在前"的文档结构提高前缀命中率。
|
||||
|
||||
### 6.3 Architect API 约定
|
||||
|
||||
- 统一走 OpenAI 兼容 `/chat/completions`；`response_format: {"type": "json_object"}`，prompt 内嵌 schema 的紧凑描述。
|
||||
- 每次调用记录 usage 到 `meta.budget` 与 Stats；触达 `api_token_cap` 前预留 500 token 余量，不足则直接熔断。
|
||||
- DeepSeek 上下文缓存：brief 稳定前缀设计可命中 cache（价格更低），E1 实验单独统计 `prompt_cache_hit_tokens`。
|
||||
|
||||
### 6.4 模型/运行时下载（scripts/setup_runtime.py）
|
||||
|
||||
- GGUF：优先 `https://hf-mirror.com/...`（可用 env `HF_MIRROR` 覆盖），HTTP Range 断点续传，校验文件大小（±1MB）。
|
||||
- llama-server：GitHub releases 拉 Windows Vulkan 版 zip（免 CUDA 工具链，N/A 卡通用）；解压 `llama-server.exe` 到 `bin/`；网络失败时打印手动下载指引后优雅退出。
|
||||
- 完成后打印三档硬件检测结果与所选档位。
|
||||
|
||||
---
|
||||
|
||||
## 7. 任务分解（供 Agent 按序执行）
|
||||
|
||||
> 依赖链：T1 → T2 → (T3, T4) → T5 → T6 → (T7, T8, T9) → T10 → (T11, T12) → T13 → T14
|
||||
> 每个任务一个 commit（`feat(v2): Tn 描述`），交付必须含测试。完成即在《任务拆解与执行计划.md》追加登记。
|
||||
|
||||
### 阶段 A：真实链路
|
||||
|
||||
- **T1 环境与基线确认**（0.5 天）
|
||||
确认 126 测试全绿；手动起一次 llama-server 跑通一次补全；DeepSeek API 冒烟（无 key 则记录跳过）。
|
||||
验收：README 追加"开发环境备忘"；发现的环境问题记录到本文件 11 节。
|
||||
- **T2 运维层：hw_profile + llama_server 进程管理**（3 天）
|
||||
实现 5.1 两个模块；三档模板；健康检查/退避重启/优雅停止（Windows 进程语义注意）。
|
||||
验收：单测（MockTransport + 假二进制脚本）覆盖启停/崩溃重启/健康检查；真机手动验证三档参数生成正确。
|
||||
- **T3 ArchitectClient**（2 天）
|
||||
brief/decide/final_review 三方法；JSON 约束输出＋失败重试一次＋降级；token 计量。
|
||||
验收：MockTransport 单测（正常/坏 JSON/超时/熔断四路径）；无 key 时构造报错信息明确。
|
||||
- **T4 Workspace**（2 天）
|
||||
schema、校验、锚点寻址、双渲染函数（断言 token 预算）、rollup、持久化。
|
||||
验收：schema 单测（合法/非法样例 ≥10 组）；render_for_architect 输出对 3 组构造数据 ≤1200 token。
|
||||
|
||||
### 阶段 B：协作协议
|
||||
|
||||
- **T5 WorkerLoop + 工具循环 + 接地验证**（6 天）
|
||||
受限工具循环（D12：五工具、目录作用域、每 step ≤20 次调用、run 强制超时）→验证（D4 分层：run 工具跑测试 / facts 对照 / 结构检查）→自修≤2→issue。QA 快路径不启用工具（纯生成）。
|
||||
验收：单测覆盖"一次通过/自修成功/写 issue"三路径；工具循环单测（越界路径拒绝、超限熔断、超时回收）；端到端 demo：一个函数需求从 brief 到工作目录内产出可跑通的模块＋测试；facts 对照复用 knowledge YAML 现有数据出 3 个真实用例。
|
||||
- **T6 CollaborativePipeline 编排**（3 天）
|
||||
快路径、循环状态机（4.3）、双护栏熔断、Architect 兜底代做、终审、TraceStore 对接。
|
||||
验收：端到端集成测试（全程 Mock：假 llama-server = httpx MockTransport + 假 Architect 同理）跑通"一次通过""带 issue 修复""熔断兜底"三个剧本。
|
||||
- **T7 网关扩展**（1.5 天）：5.3 全部端点；`/chat` 切 v2，`/chat/legacy` 保留。
|
||||
验收：TestClient 测试新旧端点；legacy 测试全部原样通过。
|
||||
- **T8 人工检验队列**（2 天）：ReviewQueue + 端点；抽样与 force_tags 规则。
|
||||
验收：单测入队/出队/修正回写；端到端测试中 safety tag 的请求必入队。
|
||||
- **T9 token 计量与账单**（1 天）：每请求 API token 记账；/metrics 扩展。
|
||||
验收：E2E 测试断言 budget 回写正确；熔断在 cap=极小值的构造用例下触发。
|
||||
|
||||
### 阶段 C：整合
|
||||
|
||||
- **T10 rollup + prefix cache 调优**（2 天）：验证文档结构对 --cache-reuse 命中的影响；调 render 预算。
|
||||
验收：同一 10 步会话，第二轮起 prefill 时间显著下降（记录数字到实验目录）。
|
||||
- **T11 打包分发**（3 天）：setup_runtime.py + 一键启动（`scripts/serve.py` 扩展为自动拉起 llama-server）+ 首启引导。
|
||||
验收：**干净 Windows 环境 20 分钟内从 clone 到对话成功**（写入验收清单）；无 API key 可走纯本地降级模式。
|
||||
|
||||
### 阶段 D：实验与论文
|
||||
|
||||
- **T12 实验脚本与数据集**（2 天）：`scripts/bench_tokens.py` + `eval/` 数据集（第 9 节）。
|
||||
- **T13 跑数与报告**（4 天）：E1–E5 输出到 `research/v2_experiments/`（md + csv）。
|
||||
- **T14 文档收口**（1 天）：README v2 改写（架构图、快速开始、指标表）。
|
||||
|
||||
---
|
||||
|
||||
## 8. 工程规约（Agent 必读）
|
||||
|
||||
1. **环境事实**：Windows 11 + Git Bash；venv 位于 `.venv`（Python 3.14）；测试命令
|
||||
`.venv/Scripts/python.exe -m pytest tests -q`（2026-08-30 基线：126 passed）。
|
||||
2. **依赖纪律**：`router_system/` 新模块零第三方依赖（标准库 only）；`runtime/` 与网关可用
|
||||
httpx/fastapi/pydantic/uvicorn；确需新增依赖（如 psutil）→ 加到 requirements.txt 并在 PR 描述说明理由。
|
||||
3. **测试封闭性**（D11）：一切 LLM 交互用 `httpx.MockTransport` 注入；沙箱执行用临时目录＋超时。
|
||||
4. **Windows 兼容**：路径一律 `pathlib`；所有 CLI 出口 `sys.stdout.reconfigure(encoding="utf-8")`
|
||||
（先例见 `scripts/eval.py` 头部）；子进程用 `CREATE_NEW_PROCESS_GROUP`，终止用 terminate→kill 兜底。
|
||||
5. **代码风格**：中文 docstring；`build_xxx(cfg)` 工厂函数；与现有文件排版一致；注释只写代码无法自明的约束。
|
||||
6. **金额敏感**：任何 Architect 调用必须计量并受 `api_token_cap` 约束；禁止无熔断的重试循环。
|
||||
7. **禁止事项**：不改 llama.cpp 源码；不在主链路重新引入领域分类；Architect 输入包含工件全文；跳过 schema 校验；为演示造假数据。
|
||||
8. **文档同步**：每完成一个 T，在《任务拆解与执行计划.md》追加一行状态；架构级偏离（如有）必须先更新本文件再动代码。
|
||||
|
||||
---
|
||||
|
||||
## 9. 实验与评测设计（论文数据来源）
|
||||
|
||||
统一输出目录：`research/v2_experiments/`（每个实验一份 md 报告 + csv 原始数据）。
|
||||
|
||||
### E1 token 经济学（主实验）
|
||||
- 50 任务（代码 20 / 数学 10 / 通用 20），四臂：
|
||||
A1 全量上下文（每轮把完整历史发 Architect）｜A2 交流文本协议｜A3 A2+rollup｜A4 A3+prefix cache。
|
||||
- 指标：Architect input/output tokens、`prompt_cache_hit_tokens`、估算成本、回合数、最终质量分。
|
||||
- 预期结论：A4 vs A1 的 Architect token 下降 ≥80%。
|
||||
|
||||
### E2 端到端质量（三臂）
|
||||
- 100 任务：快路径 only / 完整协作管线 / 纯 Architect（大模型直答）。
|
||||
- 判分：`machine_checkable` 项用断言；其余用 LLM rubric 评分＋10% 人工抽检。
|
||||
- 数据集建议：HumanEval+ 子集、GSM8K 子集、C-Eval 子集、自建法律/医疗 QA 各 30 条（facts 库内可核对）。
|
||||
|
||||
### E3 协作健康度
|
||||
- 升级率、回合数分布、issue 率、自修成功率、熔断次数（直接汇总 E1/E2 运行数据）。
|
||||
|
||||
### E4 KV 量化内存-精度曲线
|
||||
- fp16 / q8_0 / q4_0 × 上下文 4K/16K/32K：进程内存（psutil RSS＋nvidia-smi 显存）× Worker 验证准确率（固定 50 题）。
|
||||
|
||||
### E5 验证器 P/R
|
||||
- 对 100 个产物注入 50 处模板化缺陷，测接地验证的拦截率/误杀率；对照组=纯小模型自由判断（预期显著更差，支撑 D4）。
|
||||
|
||||
---
|
||||
|
||||
## 10. 里程碑排期（12 周）
|
||||
|
||||
| 周 | 阶段 | 内容 | 硬验收 |
|
||||
|---|---|---|---|
|
||||
| 1 | A | T1 + T2 开工 | 真实 llama-server 冒烟 |
|
||||
| 2 | A | T2 完成 + T3 | 运维层单测全绿 |
|
||||
| 3–4 | B | T4 + T5 | Workspace/Worker 单测全绿 |
|
||||
| 5 | B | T6 | Mock 端到端三剧本通过 |
|
||||
| 6 | B/C | T7 + T8 + T9 | 新旧端点共存，测试全绿 |
|
||||
| 7 | C | T10 + T11 开工 | prefix cache 收益数据 |
|
||||
| 8 | C | T11 完成 | 干净环境 20 分钟跑通 |
|
||||
| 9–10 | D | T12 + T13 | E1 主实验出数 |
|
||||
| 11–12 | D | T13 收尾 + T14 + 论文 | 实验报告 + README v2 |
|
||||
|
||||
---
|
||||
|
||||
## 11. 风险清单
|
||||
|
||||
| 风险 | 缓解 |
|
||||
|---|---|
|
||||
| 小模型长上下文"中间迷失" | 协议本身即补偿（重要前置＋总量受控）；render_for_worker ≤8K 硬约束 |
|
||||
| llama.cpp 长上下文 prefill 慢（CPU 尤甚） | `--cache-reuse`＋brief 恒定前缀；`-c` 按需不拉满 |
|
||||
| 协作死循环/踢皮球 | rounds_cap＋api_token_cap 硬熔断（D6），熔断即兜底代做 |
|
||||
| 硬件多样性无法穷举测试 | 三档保守模板＋手动覆盖；超出范围提示"已按保守档运行" |
|
||||
| 国内模型下载失败 | hf-mirror 镜像＋断点续传＋手动下载指引 |
|
||||
| 用户无 API key | 本地降级模式：快路径＋本地兜底＋明确降级提示（v1 NoneFallback 思路） |
|
||||
| 打包吃掉全部时间（时间黑洞） | 只承诺"干净 Windows 20 分钟跑通"；签名/自动更新/多平台一律不做 |
|
||||
| schema 漂移导致协议断裂 | 所有写入过校验（4.6）；校验器有独立单测 |
|
||||
|
||||
---
|
||||
|
||||
## 12. 术语对照
|
||||
|
||||
| 术语 | 含义 |
|
||||
|---|---|
|
||||
| 交流文本 / Workspace | 两模型间的结构化 JSON 共享工作区（本协议核心） |
|
||||
| brief | Architect 开局写入的任务分析（目标/约束/验收/计划） |
|
||||
| issue / decision | Worker 上报的验证问题 / Architect 的裁决回复 |
|
||||
| 锚点（anchor） | `a://文件#L行区间`，指向工件片段的引用（替代全文复制） |
|
||||
| rollup | 已完成步骤压缩为 archive 摘要行的机制 |
|
||||
| 快路径 | 小模型直答＋自验证通过即返回，不经 Architect |
|
||||
| 熔断 | rounds_cap / api_token_cap 触顶后强制退出循环 |
|
||||
@@ -0,0 +1,212 @@
|
||||
# 实现方案 v3：端云协同编程智能体系统 Web 应用化
|
||||
|
||||
> 编写日期：2026（git 记录）
|
||||
> 状态：**Web 应用化权威设计**。在 v2（端云协同编程智能体系统）基础上，把系统做成一个**完整、可演示、可答辩的 Web 应用**。
|
||||
> 前置依赖：v2 已落地（见 `实现方案_v2_端云协同编程智能体系统.md` 与 `README.md`），229 项测试全绿。
|
||||
> 预期读者：负责实现的 AI Agent 或开发者。实现前必读：第 4 节（设计决策）、第 8 节（工程规约）。
|
||||
|
||||
---
|
||||
|
||||
## 0. 一页速览
|
||||
|
||||
- **后端**：保持 FastAPI + uvicorn **不重写**；仅把 `/chat` 从"同步等待"改为"异步任务 + SSE 事件流"，并**通过读 `workspace.json` 实现协作过程实时可视化**，**不触碰 `router_system` 核心逻辑**（保住 229 项测试）。
|
||||
- **前端**：新建 **Vue 3 + Vite + TypeScript** SPA，Vite 构建产物输出到 `gateway/static`，由 FastAPI 单端口托管。四个页面：**对话 / 协作过程 / 检验队列 / 指标**。
|
||||
- **实时性**：`POST /chat` 立即返回 `request_id`，后台 asyncio 任务跑协作管线，前端经 **SSE（Server-Sent Events）** 订阅 `/runs/{id}/stream`，把 Architect 与 Worker 一步步交接的「交流文本」过程实时画出来。
|
||||
- **工程约束**：单端口部署、sqlite + 文件系统存储、不上 Celery/Redis/微服务；`router_system` 保持零第三方依赖。
|
||||
|
||||
---
|
||||
|
||||
## 1. 背景与动机
|
||||
|
||||
### 1.1 为什么做 Web 应用化
|
||||
|
||||
1. **v2 前端是单文件原生 HTML**（`gateway/static/index.html`，21KB），功能齐全但：
|
||||
- 代码无工程化组织，不便于扩展、维护与答辩展示"完整 Web 应用"。
|
||||
- 协作过程是静态展示，无法**实时看到** Architect/Worker 多轮交接。
|
||||
2. **`/chat` 是同步阻塞**：协作管线多轮慢任务（本地小模型 + 大模型 API），前端只能干等，交互体验差，也无法演示"交流文本逐步推进"这一核心卖点。
|
||||
3. **课题定位**是"系统设计与实现"，一个工程化、可视化、可量化的 Web 应用与课题目标匹配，且能体现工作量。
|
||||
|
||||
### 1.2 现有资产盘点（Web 层）
|
||||
|
||||
| 模块 | 现状 | v3 处置 |
|
||||
|---|---|---|
|
||||
| `gateway/api.py` | FastAPI，8 组端点 | **保留** + 新增 SSE / 任务管理 |
|
||||
| `gateway/static/index.html` | 单文件前端 | **退役**，被 Vue SPA 构建产物取代 |
|
||||
| `gateway/settings.py` | 用户可调设置 | 保留 |
|
||||
| `router_system/*` | 纯标准库业务逻辑 | **一行不改**，只新增读接口 |
|
||||
| `scripts/serve.py` | 启动网关 | 扩展为同时托管构建产物 |
|
||||
|
||||
---
|
||||
|
||||
## 2. 总体架构
|
||||
|
||||
```
|
||||
浏览器 (Vue 3 SPA, 4 页面)
|
||||
│ REST + SSE
|
||||
▼
|
||||
gateway/ (FastAPI, 单进程 uvicorn)
|
||||
├── POST /chat → 提交后台任务，立即返回 request_id
|
||||
├── GET /runs/{id}/stream (SSE) → 推送协作过程增量 + 最终结果
|
||||
├── GET /runs/{id}/status → 任务状态 (pending/running/done/failed)
|
||||
├── GET /runs/{id}/workspace, /artifacts/{name}
|
||||
├── GET/POST /review/* → 人工检验队列
|
||||
├── GET /metrics → 指标（供 ECharts 看板）
|
||||
├── GET /config → 设置
|
||||
└── 静态托管 dist/（Vue 构建产物）
|
||||
│
|
||||
├── jobs.py ← 新增：asyncio 后台任务注册表 (request_id -> task+state)
|
||||
│
|
||||
└── router_system/ ← 保持零依赖，由后台任务调用
|
||||
pipeline.run(query) # 边跑边写 runs/{id}/workspace.json
|
||||
```
|
||||
|
||||
**数据流（一次对话）**：
|
||||
1. 前端 POST /chat → 后端建后台任务，返回 `{request_id, status:"running"}`。
|
||||
2. 后台任务调用 `pipeline.run(query)`；管线把「交流文本」增量写入 `runs/{id}/workspace.json`。
|
||||
3. 前端打开 `GET /runs/{id}/stream`（SSE）；后端生成器**监视该文件**（mtime/内容变化），按事件推送。
|
||||
4. 协作结束 → SSE 推送 `result`（最终答复 + token/cost/latency 指标）→ 前端渲染。
|
||||
|
||||
---
|
||||
|
||||
## 3. 技术栈选型
|
||||
|
||||
| 层 | 选型 | 理由 |
|
||||
|---|---|---|
|
||||
| 后端 | Python + FastAPI + uvicorn | `router_system` 为 Python 纯标准库，LLM/llama 集成均为 Python；**重写为巨大浪费** |
|
||||
| 异步/SSE | FastAPI `StreamingResponse` + 自实现 SSE 格式（或 `sse-starlette`） | 单个轻依赖，实现协作实时推送 |
|
||||
| 任务调度 | asyncio 后台任务（uvicorn 事件循环内） | 毕设量级足够；**不上 Celery/Redis** |
|
||||
| 存储 | sqlite（review 已用）+ 文件系统 `runs/` | 够用、易解释、免装数据库服务 |
|
||||
| 前端 | Vue 3 + Vite + TypeScript | 工程化 SPA，四页面，答辩加分 |
|
||||
| 状态管理 | Pinia（可选，轻量） | 跨页共享运行状态/指标 |
|
||||
| UI 组件 | Element Plus | 对话、表格、表单、队列管理开箱即用 |
|
||||
| 可视化 | ECharts | 交流文本时间轴、token 曲线、指标看板 |
|
||||
| 部署 | Vite `outDir=gateway/static` → FastAPI 静态托管，单端口 | 演示省心；可选 Docker 打包 |
|
||||
|
||||
---
|
||||
|
||||
## 4. 设计决策（不得推翻）
|
||||
|
||||
- **D1 后端不重写**：`gateway/api.py` 与 `router_system` 保持 Python；禁止改写成 Node/Go。
|
||||
- **D2 /chat 异步化**：`POST /chat` 提交后台 asyncio 任务并立即返回 `request_id`，不再同步等待管线跑完。
|
||||
- **D3 SSE 通过读文件实现，不改 pipeline 内核**：协作过程可视化靠 SSE 生成器监视 `runs/{id}/workspace.json`（管线已增量写该文件）。**不得修改 `router_system/pipeline.py` 以注入回调**，从而保住 229 项测试全绿。若未来要更精细事件，再作为独立优化（见 §9 展望），且需同步维护测试。
|
||||
- **D4 任务状态可查询**：后台任务必须有独立状态（pending/running/done/failed + 错误信息 + 时间戳），供 `GET /runs/{id}/status` 与前端刷新后恢复。
|
||||
- **D5 不上分布式中间件**：禁用 Celery / Redis / Kafka / 微服务；asyncio + sqlite + 文件系统即可满足。
|
||||
- **D6 单端口部署**：Vue 构建产物输出到 `gateway/static`，由 FastAPI 统一托管；不另开前端 dev 服务器作为生产入口（dev 模式可单独起 Vite，仅本地开发用）。
|
||||
- **D7 前端四页面划分**：对话 / 协作过程 / 检验队列 / 指标；用 Vue Router 管理路由。
|
||||
- **D8 兼容降级**：无 API key / 无本地模型时 `/chat` 仍走 mock/降级路径（沿用 v2），Web 应用不可因此崩溃。
|
||||
- **D9 参数化配置**：`/chat` 的领域、采样、熔断等沿用 v2 配置；Web 端不硬编码模型/金额参数。
|
||||
|
||||
---
|
||||
|
||||
## 5. 后端接口设计
|
||||
|
||||
### 5.1 变更端点
|
||||
|
||||
| 端点 | 变更 | 说明 |
|
||||
|---|---|---|
|
||||
| `POST /chat` | **改为异步** | 入参同 v2 `QueryRequest`；返回 `{request_id, status:"running"}`，后台跑管线 |
|
||||
| `GET /runs/{id}/stream` | **新增（SSE）** | `text/event-stream`；事件：`status`（状态变化）、`workspace`（交流文本增量/快照）、`result`（最终答复+指标）、`error` |
|
||||
| `GET /runs/{id}/status` | **新增** | 返回任务状态、已用轮次、token、错误等 |
|
||||
|
||||
### 5.2 不变端点（复用）
|
||||
|
||||
`POST /chat/legacy`、`GET /runs/{id}/workspace`、`GET /runs/{id}/artifacts/{name}`、`GET/POST /review/*`、`GET /metrics`、`GET/PUT /config`、`GET /health`。
|
||||
|
||||
### 5.3 SSE 事件协议（草稿）
|
||||
|
||||
```
|
||||
data: {"type":"status","value":"running","request_id":"..."}
|
||||
data: {"type":"workspace","version":N,"workspace":{...}}
|
||||
data: {"type":"result","response":"...","api_input_tokens":123,"cost_est":0.01,"latency_ms":4200,...}
|
||||
data: {"type":"error","detail":"..."}
|
||||
```
|
||||
|
||||
- `workspace` 事件按文件变化频率推送；前端据此推进「交流文本」状态机可视化。
|
||||
- `result` 为终态；此后连接由服务端关闭。
|
||||
|
||||
---
|
||||
|
||||
## 6. 前端页面设计
|
||||
|
||||
### 6.1 路由与视图
|
||||
|
||||
| 路由 | 页面 | 关键功能 |
|
||||
|---|---|---|
|
||||
| `/` | ChatView（对话） | 输入 query → POST /chat → 打开 SSE → 逐步可视化协作过程 + 展示最终答复/指标；历史运行列表 |
|
||||
| `/runs/:id` | CollaborationView（协作过程） | 只读查看一次运行的「交流文本」完整 JSON、工件下载、时间轴 |
|
||||
| `/review` | ReviewView（检验队列） | 待审列表、详情、verdict/correction 提交 |
|
||||
| `/metrics` | MetricsView（指标） | ECharts：token 经济学（A1 vs A2）、快路径命中率、护栏熔断、检验统计 |
|
||||
|
||||
### 6.2 核心组件
|
||||
|
||||
- `WorkspaceTimeline.vue`：把「交流文本」按状态机（brief→implement→verify→decide→final_review→done）渲染为流程时间轴。
|
||||
- `TokenEconomicsChart.vue`：A1 全量 vs A2 交流文本 的 token 对比图。
|
||||
- `ReviewQueueTable.vue`：人工检验队列表格 + 提交表单。
|
||||
- `SseClient.ts`：封装 EventSource + 断线重连 + 状态机事件分发。
|
||||
|
||||
---
|
||||
|
||||
## 7. 目录结构（新增）
|
||||
|
||||
```
|
||||
gateway/
|
||||
├── api.py # 保留；新增 /chat 异步 + /runs/{id}/stream + /runs/{id}/status
|
||||
├── jobs.py # 新增：后台任务注册表（request_id -> Task + 状态）
|
||||
└── static/ # Vite 构建产物（outDir 指向这里），FastAPI 静态托管
|
||||
webapp/ # 新增：Vue 3 + Vite + TS 前端源码
|
||||
├── index.html
|
||||
├── vite.config.ts # server.proxy /api -> localhost:8000；build.outDir -> ../gateway/static
|
||||
├── package.json
|
||||
└── src/
|
||||
├── main.ts, App.vue, router.ts
|
||||
├── api/ # rest.ts, sse.ts
|
||||
├── views/ # ChatView, CollaborationView, ReviewView, MetricsView
|
||||
├── components/ # WorkspaceTimeline, TokenEconomicsChart, ReviewQueueTable, ...
|
||||
└── stores/ # runStore.ts (Pinia, 可选)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. 工程规约
|
||||
|
||||
1. **`router_system/` 零第三方依赖不变**；`gateway/` 可用 fastapi/uvicorn/pydantic/httpx（沿用 v2）。
|
||||
2. **新增依赖需登记**：前端 Node 依赖记入 `webapp/package.json`；后端如引入 `sse-starlette` 需写入 `requirements.txt` 并说明理由。
|
||||
3. **测试**：`.venv/Scripts/python.exe -m pytest tests -q` 必须**保持 229 项全绿**；新增 SSE/异步的测试用 `httpx.MockTransport` + 临时文件，不依赖真实模型/API key（沿用 v2 D4）。
|
||||
4. **异步任务清理**：done/failed 任务定期回收（或按容量上限裁剪），防内存泄漏。
|
||||
5. **Windows 兼容**：`pathlib` 路径、UTF-8、子进程 terminate→kill 兜底（沿用 v2 第 6 条）。
|
||||
6. **金额敏感**：SSE/前端展示的 token/cost 均来自 v2 计量，不改动 `api_token_cap` 熔断逻辑。
|
||||
7. 每任务一个 commit，格式 `feat(v3): Tn 描述`。
|
||||
|
||||
---
|
||||
|
||||
## 9. 实施步骤（T1–T6）
|
||||
|
||||
- **T1 后端异步化**：`jobs.py` + `POST /chat` 改为后台任务 + `GET /runs/{id}/status`。
|
||||
- **T2 后端 SSE**：`GET /runs/{id}/stream`，生成器监视 `workspace.json`，推送 status/workspace/result。
|
||||
- **T3 前端脚手架**：Vite 创建 Vue3+TS 工程，配 proxy 与 `outDir=gateway/static`；接入路由与 4 个空页面。
|
||||
- **T4 对话页 + 协作可视化**：SseClient + WorkspaceTimeline，实现实时「交流文本」推进。
|
||||
- **T5 检验队列页 + 指标页**：ReviewQueueTable + ECharts 看板。
|
||||
- **T6 集成与部署**：`scripts/serve.py` 托管构建产物；`npm run build` 产物验证四页面；写测试覆盖 SSE 协议与任务状态机。
|
||||
|
||||
---
|
||||
|
||||
## 10. 验收标准
|
||||
|
||||
- [ ] `POST /chat` 立即返回 `request_id`（非阻塞）。
|
||||
- [ ] SSE `/runs/{id}/stream` 能实时推送 status/workspace/result，断线可重连。
|
||||
- [ ] 对话页能完整可视化一次协作的「交流文本」推进过程并展示最终答复与 token/cost。
|
||||
- [ ] 检验队列页能列表/提交审核；指标页 ECharts 正常渲染。
|
||||
- [ ] `pytest tests -q` 保持 229 项全绿（含新增 SSE/异步测试）。
|
||||
- [ ] 单端口（FastAPI）访问即可完成全部四页面操作与演示。
|
||||
|
||||
---
|
||||
|
||||
## 11. 风险与规避
|
||||
|
||||
| 风险 | 规避 |
|
||||
|---|---|
|
||||
| SSE 读到文件抖动/重复 | 用 version + 文件 mtime/大小变化去重，前端按 version 合并 |
|
||||
| 后台任务异常/进程重启丢状态 | `jobs.py` 记录状态与运行目录；重启后按 `runs/` 恢复"已完成"识别，进行中的标记为 failed/unknown |
|
||||
| 长连接占资源 | SSE 空闲超时自动关闭；前端自动重连 |
|
||||
| 引入 Node 工具链的复杂度 | 仅 dev 需要 npm；产物构建后为纯静态，演示/部署无需 Node |
|
||||
| 前端过度设计 | 固定 4 页面，不上复杂状态库/微前端，控制依赖数量 |
|
||||
+18
-3
@@ -1,7 +1,7 @@
|
||||
# 毕业设计 · 进度记录
|
||||
|
||||
> 本文件记录"端云协同 LLM 协作系统"作为**毕业设计**推进过程中的关键决策、产出与待办。
|
||||
> 对应仓库文档：`实现方案_v2_端云协同LLM协作系统.md`（系统权威方案）、`实现方案_v3_Web应用化.md`（Web 应用化设计）、`references/毕业论文_参考文献清单.md`（参考文献）。
|
||||
> 本文件记录"端云协同编程智能体系统"作为**毕业设计**推进过程中的关键决策、产出与待办。
|
||||
> 对应仓库文档：`实现方案_v2_端云协同编程智能体系统.md`（系统权威方案）、`实现方案_v3_Web应用化.md`（Web 应用化设计）、`references/毕业论文_参考文献清单.md`（参考文献）。
|
||||
|
||||
---
|
||||
|
||||
@@ -17,7 +17,8 @@
|
||||
|
||||
### 2.1 命题方向（已定）
|
||||
- 对象：本科 · 计算机科学与技术；侧重 **端云协同架构**。
|
||||
- **推荐命题**：《基于端云协同的 LLM 服务成本优化系统设计与实现》（副标题：——以异构大小模型非对称协作为核心）。
|
||||
- **推荐命题**（2026-08 版，已被下条取代）：《基于端云协同的 LLM 服务成本优化系统设计与实现》（副标题：——以异构大小模型非对称协作为核心）。
|
||||
- **✅ 定稿命题（2026-09-04）**：《基于端云协同的编程智能体系统设计与实现》（Design and Implementation of a Programming Agent System Based on Device-Cloud Collaboration；关键词：端云协同、编程智能体、大小模型协同、工具调用、级联升级）。改题理由：v4 落地后系统已是工具智能体形态（可操控工作目录文件），"编程智能体"更贴实现与演示重心；"成本优化"降为论文核心指标与实验章主题（北极星指标不变），不进标题。项目名/方案文档名/界面品牌已同步由"端云协同 LLM 协作系统"更名为"端云协同编程智能体系统"。
|
||||
- 核心卖点：① 非对称协作（贵的大模型少读少写 / 便宜的小模型多读多干）；②「交流文本」协议（原创交接层）；③ 可量化的北极星指标（token 下降目标 ≥80%，实测 ~61% 含缓存计费）。
|
||||
- 定位判断：这是**系统/工程类**毕设，非新算法类；命题应落在"面向成本约束的协作机制设计与实现"，避免宣称"提出了新模型/新方法"。
|
||||
- 提醒：北极星指标在论文中勿写死"80%"，表述为"缓存计费下下降约 61%，结合前缀缓存复用目标 ≥80%"。
|
||||
@@ -161,3 +162,17 @@
|
||||
深度协作质量待用户重配 DeepSeek key 后验证。
|
||||
- **测试**：281 passed（新增 4：history 注入/会话多轮/置忙拒绝/取消）。
|
||||
- **教训**：测试资源隔离清单再+1（settings.json、sessions 目录、pool.json、runs/）。
|
||||
|
||||
## 六、2026-09-04 推进：新方向立项——校园 AI 代理层
|
||||
|
||||
- **用户定调**：`AI代理功能开发/` 子目录定位为「端 ↔ 云之间的 AI 代理层」开发：
|
||||
面向校园场景的 LLM API 代理网关（如代理 DeepSeek key，学生持代理 key、代理持上游主 key）。
|
||||
商业模式 = **API 差价 + 缓存收益**；校园问题高重复 → 高缓存命中 → 高毛利；校园网提供基础设施。
|
||||
- **方案草案**：`AI代理功能开发/方案_校园AI代理层.md`（架构 / 缓存经济学 / 现有系统复用映射 /
|
||||
MVP 任务 P1–P5 / 合规红线 / 实验 E-P1–E-P3）。
|
||||
- **与主命题关系**：不改变已定命题《基于端云协同的编程智能体系统设计与实现》；
|
||||
作为扩展章与答辩亮点——代理层 = 端侧基础设施的规模化形态，核心贡献为"缓存感知的 LLM 代理网关"。
|
||||
- **合规提醒（实现前必读）**：真实收费运营前需确认上游转售政策与生成式 AI 服务备案要求，
|
||||
从校内试点起步；上游主 key 仅存 env / settings.json，代码与文档零凭据字面量。
|
||||
|
||||
*记录人：ZCode Agent（2026-09-04）。*
|
||||
|
||||
Reference in New Issue
Block a user