docs: add task-type accuracy benchmark (router vs keyword heuristic)

- bench/tasks.jsonl (37 natural-language prompts, 7 task types) + bench/accuracy.py
- measured on the live allaigate router: 100% (37/37) vs keyword heuristic 32%
- BENCHMARKS.md accuracy section; README (EN+RU) perf+accuracy tables and test matrix
- run.sh runs accuracy when CORTIQ_ROUTER_KEY is set

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
infosave2007
2026-07-01 07:45:16 +03:00
co-authored by Claude Opus 4.8
parent 6714c4f556
commit 9242806795
7 changed files with 188 additions and 8 deletions
+29
View File
@@ -27,6 +27,35 @@ path.
> Absolute numbers depend on hardware; the **ratios** between gateways are the point.
> Run `bench/run.sh` to get numbers for your environment.
## Accuracy: task-type routing
Speed is only half the story. Cortiq's other edge is **routing by task type** — sending
each prompt to the right model — via the allaigate semantic classifier. LiteLLM and
Portkey have **no semantic task router**: you either pin a model, write brittle rules,
or pay for an extra LLM classification call.
On a 37-prompt, 7-task-type set of **natural-language prompts** (no explicit trigger
words like "translate" / "summarize" — [`bench/tasks.jsonl`](bench/tasks.jsonl)):
| Classifier | Correct | Accuracy |
|---|--:|--:|
| **allaigate semantic router** | 37 / 37 | **100%** |
| keyword heuristic (DIY without a classifier) | 12 / 37 | 32% |
A keyword matcher looks good on prompts that literally say "translate this" — but real
users don't. On natural phrasing (`"'Guten Tag' — what does that mean in English?"`,
`"boil this down to a few key points"`, `"which companies are named in this report?"`)
surface matching collapses while the semantic router still routes correctly.
- **Methodology.** [`bench/accuracy.py`](bench/accuracy.py) sends each prompt to the live
allaigate router (`taxonomy = data-assistant`) and compares the predicted `task_label`
to the ground truth; a keyword heuristic is scored on the same set as a lower bound for
"route without a classifier."
- **Caveats.** Small, hand-labeled set; phrasing is deliberately keyword-light to test
semantic understanding vs surface matching. Expand `tasks.jsonl` and re-run. allaigate's
own published figure is **94% at separating close task types** (a harder eval than this one).
- **Reproduce.** `CORTIQ_ROUTER_KEY=cortiq_… python3 bench/accuracy.py`
## Why this matters
Agentic workloads make **many** gateway calls per task (plan → sub-tasks → tool calls