docs: add task-type accuracy benchmark (router vs keyword heuristic)

- bench/tasks.jsonl (37 natural-language prompts, 7 task types) + bench/accuracy.py
- measured on the live allaigate router: 100% (37/37) vs keyword heuristic 32%
- BENCHMARKS.md accuracy section; README (EN+RU) perf+accuracy tables and test matrix
- run.sh runs accuracy when CORTIQ_ROUTER_KEY is set

BENCHMARKS.md

@@ -27,6 +27,35 @@ path.
 > Absolute numbers depend on hardware; the **ratios** between gateways are the point.
 > Run `bench/run.sh` to get numbers for your environment.
 
+## Accuracy: task-type routing
+
+Speed is only half the story. Cortiq's other edge is **routing by task type** — sending
+each prompt to the right model — via the allaigate semantic classifier. LiteLLM and
+Portkey have **no semantic task router**: you either pin a model, write brittle rules,
+or pay for an extra LLM classification call.
+
+On a 37-prompt, 7-task-type set of **natural-language prompts** (no explicit trigger
+words like "translate" / "summarize" — [`bench/tasks.jsonl`](bench/tasks.jsonl)):
+
+| Classifier | Correct | Accuracy |
+|---|--:|--:|
+| **allaigate semantic router** | 37 / 37 | **100%** |
+| keyword heuristic (DIY without a classifier) | 12 / 37 | 32% |
+
+A keyword matcher looks good on prompts that literally say "translate this" — but real
+users don't. On natural phrasing (`"'Guten Tag' — what does that mean in English?"`,
+`"boil this down to a few key points"`, `"which companies are named in this report?"`)
+surface matching collapses while the semantic router still routes correctly.
+
+- **Methodology.** [`bench/accuracy.py`](bench/accuracy.py) sends each prompt to the live
+  allaigate router (`taxonomy = data-assistant`) and compares the predicted `task_label`
+  to the ground truth; a keyword heuristic is scored on the same set as a lower bound for
+  "route without a classifier."
+- **Caveats.** Small, hand-labeled set; phrasing is deliberately keyword-light to test
+  semantic understanding vs surface matching. Expand `tasks.jsonl` and re-run. allaigate's
+  own published figure is **94% at separating close task types** (a harder eval than this one).
+- **Reproduce.** `CORTIQ_ROUTER_KEY=cortiq_… python3 bench/accuracy.py`
+
 ## Why this matters
 
 Agentic workloads make **many** gateway calls per task (plan → sub-tasks → tool calls

CHANGELOG.md

@@ -9,6 +9,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/).
 - **Latency benchmark** vs LiteLLM and Portkey: a reproducible harness (`bench/`,
   instant mock backend + Apache Bench) and `BENCHMARKS.md`. Measured gateway overhead:
   Cortiq ~57k req/s / p99 1 ms vs Portkey ~5.8k / p99 9 ms vs LiteLLM ~1.2k / p99 59 ms.
+- **Accuracy benchmark**: task-type routing on natural-language prompts (`bench/accuracy.py`,
+  `bench/tasks.jsonl`) — allaigate semantic router 100% (37/37) vs a keyword heuristic 32%
+  (competitors have no semantic task router).
 
 ## [0.2.1] - 2026-06-30
 

README.md

@@ -220,9 +220,9 @@ Details — [docs/PROTOCOLS.md](https://github.com/infosave2007/cortiq-gateway/b
 
 ---
 
-## ⚡ Performance
+## ⚡ Performance & accuracy
 
-Gateway overhead proxying the same instant backend, `ab -k -r -c 20 -n 5000`:
+**Latency** — gateway overhead proxying the same instant backend, `ab -k -r -c 20 -n 5000`:
 
 | Gateway | req/s | p50 | p99 |
 |---|--:|--:|--:|
@@ -230,8 +230,16 @@ Gateway overhead proxying the same instant backend, `ab -k -r -c 20 -n 5000`:
 | Portkey (Node) | ~5,800 | 3 ms | 9 ms |
 | LiteLLM (Python, 4 workers) | ~1,200 | 9 ms | 59 ms |
 
+**Accuracy** — task-type routing on natural-language prompts (7 task types). LiteLLM/Portkey
+have no semantic task router; a keyword heuristic is the DIY stand-in:
+
+| Classifier | Accuracy |
+|---|--:|
+| **allaigate semantic router** | **100%** (37/37) |
+| keyword heuristic (no classifier) | 32% (12/37) |
+
 Methodology, caveats and a reproducible harness: **[BENCHMARKS.md](BENCHMARKS.md)**
-(`bash bench/run.sh`). Numbers vary by hardware; the ratios are the point.
+(`bash bench/run.sh`, `python3 bench/accuracy.py`). Numbers vary; the gaps are the point.
 
 ## Build & test
 
@@ -255,6 +263,7 @@ cargo test                 # config round-trip, routing validation
 | Routing — auto / pinned / failover / degrade | integration (mock **and** live allaigate router) | ✅ |
 | Hot config reload (no restart) | integration | ✅ |
 | Latency benchmark vs LiteLLM / Portkey | `bench/run.sh` (Apache Bench) — see [BENCHMARKS.md](BENCHMARKS.md) | ✅ |
+| Task-type routing accuracy | `bench/accuracy.py` — live router 100% vs keyword heuristic 32% | ✅ |
 
 ---
 

README.ru.md

@@ -169,9 +169,9 @@ print(resp.choices[0].message.content)
 
 ---
 
-## ⚡ Производительность
+## ⚡ Производительность и точность
 
-Overhead шлюза при проксировании одного и того же мгновенного бэкенда, `ab -k -r -c 20 -n 5000`:
+**Латентность** — overhead шлюза при проксировании одного мгновенного бэкенда, `ab -k -r -c 20 -n 5000`:
 
 | Шлюз | req/s | p50 | p99 |
 |---|--:|--:|--:|
@@ -179,8 +179,16 @@ Overhead шлюза при проксировании одного и того
 | Portkey (Node) | ~5 800 | 3 ms | 9 ms |
 | LiteLLM (Python, 4 воркера) | ~1 200 | 9 ms | 59 ms |
 
+**Точность** — определение типа задачи на естественных промптах (7 типов). У LiteLLM/Portkey
+семантического роутера нет; keyword-эвристика — это DIY-замена:
+
+| Классификатор | Точность |
+|---|--:|
+| **семантический роутер allaigate** | **100%** (37/37) |
+| keyword-эвристика (без классификатора) | 32% (12/37) |
+
 Методология, оговорки и воспроизводимый harness — **[BENCHMARKS.md](BENCHMARKS.md)**
-(`bash bench/run.sh`). Абсолютные числа зависят от железа; важны соотношения.
+(`bash bench/run.sh`, `python3 bench/accuracy.py`). Числа зависят от железа; важны разрывы.
 
 ## Сборка и тесты
 
@@ -204,6 +212,7 @@ cargo test                 # round-trip конфига, валидация ро
 | Роутинг — auto / pinned / failover / деградация | интеграционно (mock **и** живой allaigate-роутер) | ✅ |
 | Горячая перезагрузка конфига (без рестарта) | интеграционно | ✅ |
 | Бенчмарк latency vs LiteLLM / Portkey | `bench/run.sh` (Apache Bench) — см. [BENCHMARKS.md](BENCHMARKS.md) | ✅ |
+| Точность определения типа задачи | `bench/accuracy.py` — живой роутер 100% vs keyword-эвристика 32% | ✅ |
 
 ---
 

bench/accuracy.py

@@ -0,0 +1,86 @@
+#!/usr/bin/env python3
+"""Task-type classification accuracy: the allaigate router vs a naive keyword
+heuristic (what you get without a semantic classifier, e.g. with LiteLLM/Portkey).
+
+    CORTIQ_ROUTER_KEY=cortiq_... python3 bench/accuracy.py
+
+Env: CORTIQ_ROUTER_KEY (required), ROUTER_URL (default https://138.226.222.209),
+     TAXONOMY (default data-assistant).
+"""
+import json, os, ssl, sys, urllib.request
+
+ROUTER_URL = os.environ.get("ROUTER_URL", "https://138.226.222.209").rstrip("/")
+TAXONOMY = os.environ.get("TAXONOMY", "data-assistant")
+KEY = os.environ.get("CORTIQ_ROUTER_KEY", "")
+CTX = ssl._create_unverified_context()
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+DATA = os.path.join(HERE, "tasks.jsonl")
+
+
+def router_predict(text):
+    body = json.dumps({"input": {"text": text}, "taxonomy_id": TAXONOMY}).encode()
+    req = urllib.request.Request(
+        ROUTER_URL + "/v1/route",
+        data=body,
+        headers={"content-type": "application/json", "Authorization": "Bearer " + KEY},
+    )
+    with urllib.request.urlopen(req, context=CTX, timeout=30) as r:
+        d = json.load(r)
+    return d.get("decision", {}).get("task_label", "?")
+
+
+def keyword_predict(text):
+    """Naive DIY routing without a semantic classifier."""
+    t = text.lower()
+    def has(*words):
+        return any(w in t for w in words)
+    if has("translate", "say ", "in spanish", "in japanese", "in german", "in italian",
+           "to english", "to german", "from french", "render the phrase"):
+        return "translation"
+    if has("summarize", "summary", "tl;dr", "condense", "shorten", "bullet point"):
+        return "summarization"
+    if has("extract", "pull out", "list all", "find every", "find all"):
+        return "extraction"
+    if has("function", "python", "javascript", "rust", "sql", "query", "implement",
+           "bug", "linked list", "code"):
+        return "code"
+    if has("solve", "derivative", "integral", "compute", "sum of", "area of",
+           "limit of", "evaluate", "equation"):
+        return "math"
+    if t.strip().endswith("?") or t.split()[:1] and t.split()[0] in (
+        "who", "what", "when", "where", "which"):
+        return "qa"
+    return "chitchat"
+
+
+def main():
+    if not KEY:
+        print("set CORTIQ_ROUTER_KEY", file=sys.stderr)
+        sys.exit(2)
+    rows = [json.loads(l) for l in open(DATA) if l.strip()]
+    r_ok = k_ok = 0
+    misses = []
+    for row in rows:
+        gt = row["label"]
+        try:
+            rp = router_predict(row["text"])
+        except Exception as e:
+            rp = "ERR:" + str(e)[:30]
+        kp = keyword_predict(row["text"])
+        r_ok += rp == gt
+        k_ok += kp == gt
+        if rp != gt:
+            misses.append((gt, rp, row["text"][:45]))
+    n = len(rows)
+    print(f"dataset: {n} prompts, {len(set(r['label'] for r in rows))} task types")
+    print(f"allaigate router : {r_ok}/{n} = {100*r_ok/n:.1f}%")
+    print(f"keyword heuristic: {k_ok}/{n} = {100*k_ok/n:.1f}%")
+    if misses:
+        print("\nrouter misses (truth -> predicted | prompt):")
+        for gt, rp, txt in misses:
+            print(f"  {gt:>13} -> {rp:<13} | {txt}")
+
+
+if __name__ == "__main__":
+    main()

bench/run.sh

@@ -91,6 +91,13 @@ else
   echo "portkey           | (not installed — npm i -g @portkey-ai/gateway)"
 fi
 
+# --- task-type routing accuracy (optional; needs a live router key) ----------
+if [ -n "${CORTIQ_ROUTER_KEY:-}" ]; then
+  echo
+  echo "== task-type routing accuracy (allaigate router vs keyword heuristic) =="
+  python3 "$HERE/accuracy.py"
+fi
+
 echo
-echo "Note: numbers are gateway overhead over a zero-latency mock on one machine;"
-echo "subtract mock-direct to isolate proxy cost. Re-run on your own hardware."
+echo "Note: latency numbers are gateway overhead over a zero-latency mock on one"
+echo "machine; subtract mock-direct to isolate proxy cost. Re-run on your hardware."

bench/tasks.jsonl

@@ -0,0 +1,37 @@
+{"text": "Write a Python function to reverse a string", "label": "code"}
+{"text": "Reverse a linked list in place", "label": "code"}
+{"text": "Given an array of integers, return the indices of the two that add up to a target", "label": "code"}
+{"text": "My loop prints one extra line at the end — what's wrong with it?", "label": "code"}
+{"text": "Merge two sorted arrays into a single sorted array", "label": "code"}
+{"text": "Make this faster: it compares every element with every other element", "label": "code"}
+{"text": "Solve x^2 - 5x + 6 = 0", "label": "math"}
+{"text": "If a train covers 60 miles in 1.5 hours, how fast is it going?", "label": "math"}
+{"text": "A rectangle is twice as long as it is wide and its area is 50 — what are its sides?", "label": "math"}
+{"text": "What's 15 percent of 240?", "label": "math"}
+{"text": "Two dice are rolled — what are the odds the total is seven?", "label": "math"}
+{"text": "The sequence 2, 6, 18, 54 — what comes next?", "label": "math"}
+{"text": "Translate 'good morning' into Spanish", "label": "translation"}
+{"text": "'Guten Tag' — what does that mean in English?", "label": "translation"}
+{"text": "What's the Spanish word for butterfly?", "label": "translation"}
+{"text": "Put 'I would like a coffee, please' into French", "label": "translation"}
+{"text": "How do Italians say 'see you tomorrow'?", "label": "translation"}
+{"text": "Give me the gist of the text below in one line", "label": "summarization"}
+{"text": "What are the main takeaways from this report?", "label": "summarization"}
+{"text": "Boil this down to a few key points", "label": "summarization"}
+{"text": "In a sentence, what is this article about?", "label": "summarization"}
+{"text": "Cut this down to just the essentials", "label": "summarization"}
+{"text": "Ugh, Mondays. How was your weekend?", "label": "chitchat"}
+{"text": "I'm a little bored, tell me something interesting", "label": "chitchat"}
+{"text": "Do you think pineapple belongs on pizza?", "label": "chitchat"}
+{"text": "Feeling great today, the sun is finally out!", "label": "chitchat"}
+{"text": "Any fun plans for the holidays?", "label": "chitchat"}
+{"text": "Which one is the tallest mountain on Earth?", "label": "qa"}
+{"text": "At what temperature does water freeze in Fahrenheit?", "label": "qa"}
+{"text": "Who is the author of Pride and Prejudice?", "label": "qa"}
+{"text": "How many continents are there?", "label": "qa"}
+{"text": "What is the currency used in Japan called?", "label": "qa"}
+{"text": "From the paragraph below, grab all the phone numbers", "label": "extraction"}
+{"text": "Which companies are named in this report?", "label": "extraction"}
+{"text": "Give me just the dates and amounts from this receipt", "label": "extraction"}
+{"text": "Pick out the email addresses in this message", "label": "extraction"}
+{"text": "What are the product SKUs mentioned here?", "label": "extraction"}