github/cortiq-gateway
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: add task-type accuracy benchmark (router vs keyword heuristic)

Browse Source 
- bench/tasks.jsonl (37 natural-language prompts, 7 task types) + bench/accuracy.py
- measured on the live allaigate router: 100% (37/37) vs keyword heuristic 32%
- BENCHMARKS.md accuracy section; README (EN+RU) perf+accuracy tables and test matrix
- run.sh runs accuracy when CORTIQ_ROUTER_KEY is set
This commit is contained in:
infosave2007
2026-07-01 07:45:16 +03:00
parent e773e18efd
commit 17c8186383
7 changed files with 188 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+3
View File
@@ -9,6 +9,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/).
- **Latency benchmark** vs LiteLLM and Portkey: a reproducible harness (`bench/`,
instant mock backend + Apache Bench) and `BENCHMARKS.md`. Measured gateway overhead:
Cortiq ~57k req/s / p99 1 ms vs Portkey ~5.8k / p99 9 ms vs LiteLLM ~1.2k / p99 59 ms.
- **Accuracy benchmark**: task-type routing on natural-language prompts (`bench/accuracy.py`,
`bench/tasks.jsonl`) — allaigate semantic router 100% (37/37) vs a keyword heuristic 32%
(competitors have no semantic task router).
## [0.2.1] - 2026-06-30