diff --git a/BENCHMARKS.md b/BENCHMARKS.md index 8d0f119..133e496 100644 --- a/BENCHMARKS.md +++ b/BENCHMARKS.md @@ -27,6 +27,35 @@ path. > Absolute numbers depend on hardware; the **ratios** between gateways are the point. > Run `bench/run.sh` to get numbers for your environment. +## Accuracy: task-type routing + +Speed is only half the story. Cortiq's other edge is **routing by task type** — sending +each prompt to the right model — via the allaigate semantic classifier. LiteLLM and +Portkey have **no semantic task router**: you either pin a model, write brittle rules, +or pay for an extra LLM classification call. + +On a 37-prompt, 7-task-type set of **natural-language prompts** (no explicit trigger +words like "translate" / "summarize" — [`bench/tasks.jsonl`](bench/tasks.jsonl)): + +| Classifier | Correct | Accuracy | +|---|--:|--:| +| **allaigate semantic router** | 37 / 37 | **100%** | +| keyword heuristic (DIY without a classifier) | 12 / 37 | 32% | + +A keyword matcher looks good on prompts that literally say "translate this" — but real +users don't. On natural phrasing (`"'Guten Tag' — what does that mean in English?"`, +`"boil this down to a few key points"`, `"which companies are named in this report?"`) +surface matching collapses while the semantic router still routes correctly. + +- **Methodology.** [`bench/accuracy.py`](bench/accuracy.py) sends each prompt to the live + allaigate router (`taxonomy = data-assistant`) and compares the predicted `task_label` + to the ground truth; a keyword heuristic is scored on the same set as a lower bound for + "route without a classifier." +- **Caveats.** Small, hand-labeled set; phrasing is deliberately keyword-light to test + semantic understanding vs surface matching. Expand `tasks.jsonl` and re-run. allaigate's + own published figure is **94% at separating close task types** (a harder eval than this one). +- **Reproduce.** `CORTIQ_ROUTER_KEY=cortiq_… python3 bench/accuracy.py` + ## Why this matters Agentic workloads make **many** gateway calls per task (plan → sub-tasks → tool calls diff --git a/CHANGELOG.md b/CHANGELOG.md index e276d46..9d00b32 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/). - **Latency benchmark** vs LiteLLM and Portkey: a reproducible harness (`bench/`, instant mock backend + Apache Bench) and `BENCHMARKS.md`. Measured gateway overhead: Cortiq ~57k req/s / p99 1 ms vs Portkey ~5.8k / p99 9 ms vs LiteLLM ~1.2k / p99 59 ms. +- **Accuracy benchmark**: task-type routing on natural-language prompts (`bench/accuracy.py`, + `bench/tasks.jsonl`) — allaigate semantic router 100% (37/37) vs a keyword heuristic 32% + (competitors have no semantic task router). ## [0.2.1] - 2026-06-30 diff --git a/README.md b/README.md index 4521477..9e4fe71 100644 --- a/README.md +++ b/README.md @@ -220,9 +220,9 @@ Details — [docs/PROTOCOLS.md](https://github.com/infosave2007/cortiq-gateway/b --- -## ⚡ Performance +## ⚡ Performance & accuracy -Gateway overhead proxying the same instant backend, `ab -k -r -c 20 -n 5000`: +**Latency** — gateway overhead proxying the same instant backend, `ab -k -r -c 20 -n 5000`: | Gateway | req/s | p50 | p99 | |---|--:|--:|--:| @@ -230,8 +230,16 @@ Gateway overhead proxying the same instant backend, `ab -k -r -c 20 -n 5000`: | Portkey (Node) | ~5,800 | 3 ms | 9 ms | | LiteLLM (Python, 4 workers) | ~1,200 | 9 ms | 59 ms | +**Accuracy** — task-type routing on natural-language prompts (7 task types). LiteLLM/Portkey +have no semantic task router; a keyword heuristic is the DIY stand-in: + +| Classifier | Accuracy | +|---|--:| +| **allaigate semantic router** | **100%** (37/37) | +| keyword heuristic (no classifier) | 32% (12/37) | + Methodology, caveats and a reproducible harness: **[BENCHMARKS.md](BENCHMARKS.md)** -(`bash bench/run.sh`). Numbers vary by hardware; the ratios are the point. +(`bash bench/run.sh`, `python3 bench/accuracy.py`). Numbers vary; the gaps are the point. ## Build & test @@ -255,6 +263,7 @@ cargo test # config round-trip, routing validation | Routing — auto / pinned / failover / degrade | integration (mock **and** live allaigate router) | ✅ | | Hot config reload (no restart) | integration | ✅ | | Latency benchmark vs LiteLLM / Portkey | `bench/run.sh` (Apache Bench) — see [BENCHMARKS.md](BENCHMARKS.md) | ✅ | +| Task-type routing accuracy | `bench/accuracy.py` — live router 100% vs keyword heuristic 32% | ✅ | --- diff --git a/README.ru.md b/README.ru.md index d5fb95a..23ce0d8 100644 --- a/README.ru.md +++ b/README.ru.md @@ -169,9 +169,9 @@ print(resp.choices[0].message.content) --- -## ⚡ Производительность +## ⚡ Производительность и точность -Overhead шлюза при проксировании одного и того же мгновенного бэкенда, `ab -k -r -c 20 -n 5000`: +**Латентность** — overhead шлюза при проксировании одного мгновенного бэкенда, `ab -k -r -c 20 -n 5000`: | Шлюз | req/s | p50 | p99 | |---|--:|--:|--:| @@ -179,8 +179,16 @@ Overhead шлюза при проксировании одного и того | Portkey (Node) | ~5 800 | 3 ms | 9 ms | | LiteLLM (Python, 4 воркера) | ~1 200 | 9 ms | 59 ms | +**Точность** — определение типа задачи на естественных промптах (7 типов). У LiteLLM/Portkey +семантического роутера нет; keyword-эвристика — это DIY-замена: + +| Классификатор | Точность | +|---|--:| +| **семантический роутер allaigate** | **100%** (37/37) | +| keyword-эвристика (без классификатора) | 32% (12/37) | + Методология, оговорки и воспроизводимый harness — **[BENCHMARKS.md](BENCHMARKS.md)** -(`bash bench/run.sh`). Абсолютные числа зависят от железа; важны соотношения. +(`bash bench/run.sh`, `python3 bench/accuracy.py`). Числа зависят от железа; важны разрывы. ## Сборка и тесты @@ -204,6 +212,7 @@ cargo test # round-trip конфига, валидация ро | Роутинг — auto / pinned / failover / деградация | интеграционно (mock **и** живой allaigate-роутер) | ✅ | | Горячая перезагрузка конфига (без рестарта) | интеграционно | ✅ | | Бенчмарк latency vs LiteLLM / Portkey | `bench/run.sh` (Apache Bench) — см. [BENCHMARKS.md](BENCHMARKS.md) | ✅ | +| Точность определения типа задачи | `bench/accuracy.py` — живой роутер 100% vs keyword-эвристика 32% | ✅ | --- diff --git a/bench/accuracy.py b/bench/accuracy.py new file mode 100644 index 0000000..072c65e --- /dev/null +++ b/bench/accuracy.py @@ -0,0 +1,86 @@ +#!/usr/bin/env python3 +"""Task-type classification accuracy: the allaigate router vs a naive keyword +heuristic (what you get without a semantic classifier, e.g. with LiteLLM/Portkey). + + CORTIQ_ROUTER_KEY=cortiq_... python3 bench/accuracy.py + +Env: CORTIQ_ROUTER_KEY (required), ROUTER_URL (default https://138.226.222.209), + TAXONOMY (default data-assistant). +""" +import json, os, ssl, sys, urllib.request + +ROUTER_URL = os.environ.get("ROUTER_URL", "https://138.226.222.209").rstrip("/") +TAXONOMY = os.environ.get("TAXONOMY", "data-assistant") +KEY = os.environ.get("CORTIQ_ROUTER_KEY", "") +CTX = ssl._create_unverified_context() + +HERE = os.path.dirname(os.path.abspath(__file__)) +DATA = os.path.join(HERE, "tasks.jsonl") + + +def router_predict(text): + body = json.dumps({"input": {"text": text}, "taxonomy_id": TAXONOMY}).encode() + req = urllib.request.Request( + ROUTER_URL + "/v1/route", + data=body, + headers={"content-type": "application/json", "Authorization": "Bearer " + KEY}, + ) + with urllib.request.urlopen(req, context=CTX, timeout=30) as r: + d = json.load(r) + return d.get("decision", {}).get("task_label", "?") + + +def keyword_predict(text): + """Naive DIY routing without a semantic classifier.""" + t = text.lower() + def has(*words): + return any(w in t for w in words) + if has("translate", "say ", "in spanish", "in japanese", "in german", "in italian", + "to english", "to german", "from french", "render the phrase"): + return "translation" + if has("summarize", "summary", "tl;dr", "condense", "shorten", "bullet point"): + return "summarization" + if has("extract", "pull out", "list all", "find every", "find all"): + return "extraction" + if has("function", "python", "javascript", "rust", "sql", "query", "implement", + "bug", "linked list", "code"): + return "code" + if has("solve", "derivative", "integral", "compute", "sum of", "area of", + "limit of", "evaluate", "equation"): + return "math" + if t.strip().endswith("?") or t.split()[:1] and t.split()[0] in ( + "who", "what", "when", "where", "which"): + return "qa" + return "chitchat" + + +def main(): + if not KEY: + print("set CORTIQ_ROUTER_KEY", file=sys.stderr) + sys.exit(2) + rows = [json.loads(l) for l in open(DATA) if l.strip()] + r_ok = k_ok = 0 + misses = [] + for row in rows: + gt = row["label"] + try: + rp = router_predict(row["text"]) + except Exception as e: + rp = "ERR:" + str(e)[:30] + kp = keyword_predict(row["text"]) + r_ok += rp == gt + k_ok += kp == gt + if rp != gt: + misses.append((gt, rp, row["text"][:45])) + n = len(rows) + print(f"dataset: {n} prompts, {len(set(r['label'] for r in rows))} task types") + print(f"allaigate router : {r_ok}/{n} = {100*r_ok/n:.1f}%") + print(f"keyword heuristic: {k_ok}/{n} = {100*k_ok/n:.1f}%") + if misses: + print("\nrouter misses (truth -> predicted | prompt):") + for gt, rp, txt in misses: + print(f" {gt:>13} -> {rp:<13} | {txt}") + + +if __name__ == "__main__": + main() diff --git a/bench/run.sh b/bench/run.sh index 7ccee56..a68f033 100644 --- a/bench/run.sh +++ b/bench/run.sh @@ -91,6 +91,13 @@ else echo "portkey | (not installed — npm i -g @portkey-ai/gateway)" fi +# --- task-type routing accuracy (optional; needs a live router key) ---------- +if [ -n "${CORTIQ_ROUTER_KEY:-}" ]; then + echo + echo "== task-type routing accuracy (allaigate router vs keyword heuristic) ==" + python3 "$HERE/accuracy.py" +fi + echo -echo "Note: numbers are gateway overhead over a zero-latency mock on one machine;" -echo "subtract mock-direct to isolate proxy cost. Re-run on your own hardware." +echo "Note: latency numbers are gateway overhead over a zero-latency mock on one" +echo "machine; subtract mock-direct to isolate proxy cost. Re-run on your hardware." diff --git a/bench/tasks.jsonl b/bench/tasks.jsonl new file mode 100644 index 0000000..cf40030 --- /dev/null +++ b/bench/tasks.jsonl @@ -0,0 +1,37 @@ +{"text": "Write a Python function to reverse a string", "label": "code"} +{"text": "Reverse a linked list in place", "label": "code"} +{"text": "Given an array of integers, return the indices of the two that add up to a target", "label": "code"} +{"text": "My loop prints one extra line at the end — what's wrong with it?", "label": "code"} +{"text": "Merge two sorted arrays into a single sorted array", "label": "code"} +{"text": "Make this faster: it compares every element with every other element", "label": "code"} +{"text": "Solve x^2 - 5x + 6 = 0", "label": "math"} +{"text": "If a train covers 60 miles in 1.5 hours, how fast is it going?", "label": "math"} +{"text": "A rectangle is twice as long as it is wide and its area is 50 — what are its sides?", "label": "math"} +{"text": "What's 15 percent of 240?", "label": "math"} +{"text": "Two dice are rolled — what are the odds the total is seven?", "label": "math"} +{"text": "The sequence 2, 6, 18, 54 — what comes next?", "label": "math"} +{"text": "Translate 'good morning' into Spanish", "label": "translation"} +{"text": "'Guten Tag' — what does that mean in English?", "label": "translation"} +{"text": "What's the Spanish word for butterfly?", "label": "translation"} +{"text": "Put 'I would like a coffee, please' into French", "label": "translation"} +{"text": "How do Italians say 'see you tomorrow'?", "label": "translation"} +{"text": "Give me the gist of the text below in one line", "label": "summarization"} +{"text": "What are the main takeaways from this report?", "label": "summarization"} +{"text": "Boil this down to a few key points", "label": "summarization"} +{"text": "In a sentence, what is this article about?", "label": "summarization"} +{"text": "Cut this down to just the essentials", "label": "summarization"} +{"text": "Ugh, Mondays. How was your weekend?", "label": "chitchat"} +{"text": "I'm a little bored, tell me something interesting", "label": "chitchat"} +{"text": "Do you think pineapple belongs on pizza?", "label": "chitchat"} +{"text": "Feeling great today, the sun is finally out!", "label": "chitchat"} +{"text": "Any fun plans for the holidays?", "label": "chitchat"} +{"text": "Which one is the tallest mountain on Earth?", "label": "qa"} +{"text": "At what temperature does water freeze in Fahrenheit?", "label": "qa"} +{"text": "Who is the author of Pride and Prejudice?", "label": "qa"} +{"text": "How many continents are there?", "label": "qa"} +{"text": "What is the currency used in Japan called?", "label": "qa"} +{"text": "From the paragraph below, grab all the phone numbers", "label": "extraction"} +{"text": "Which companies are named in this report?", "label": "extraction"} +{"text": "Give me just the dates and amounts from this receipt", "label": "extraction"} +{"text": "Pick out the email addresses in this message", "label": "extraction"} +{"text": "What are the product SKUs mentioned here?", "label": "extraction"}