cortiq-gateway/BENCHMARKS.md

Gateway latency benchmark

How much latency does the gateway itself add? For an LLM proxy the model call dominates wall-clock time, but the gateway's own overhead is what differs between products — and it compounds across the many calls an agent makes. This benchmark isolates that overhead and compares Cortiq Gateway with LiteLLM and Portkey.

Reproduce on your own hardware: bash bench/run.sh (see Methodology).

TL;DR

Every gateway proxies the same instant mock backend in passthrough mode, under identical load (ab -k -r -c 20 -n 5000). Latency is in milliseconds.

Gateway	Throughput (req/s)	p50	p90	p99	mean
Cortiq Gateway (Rust)	~57,300	0 ms	1 ms	1 ms	0.35 ms
Portkey Gateway (Node)	~5,800	3 ms	5 ms	9 ms	3.5 ms
LiteLLM (Python, 4 workers)	~1,200	9 ms	37 ms	59 ms	16.6 ms
mock-direct (no gateway)	~29,000	1 ms	1 ms	3 ms	0.7 ms

On this machine Cortiq adds sub-millisecond p99 overhead and sustains ~10× the throughput of Portkey and ~47× of LiteLLM. The gap is architectural: a Rust / Tokio gateway uses every core in one process, with no interpreter or GC in the hot path.

Absolute numbers depend on hardware; the ratios between gateways are the point. Run bench/run.sh to get numbers for your environment.

Accuracy: task-type routing

Speed is only half the story. Cortiq's other edge is routing by task type — sending each prompt to the right model — via the allaigate semantic classifier. LiteLLM and Portkey have no semantic task router: you either pin a model, write brittle rules, or pay for an extra LLM classification call.

On a 37-prompt, 7-task-type set of natural-language prompts (no explicit trigger words like "translate" / "summarize" — bench/tasks.jsonl):

Classifier	Correct	Accuracy
allaigate semantic router	37 / 37	100%
keyword heuristic (DIY without a classifier)	12 / 37	32%

A keyword matcher looks good on prompts that literally say "translate this" — but real users don't. On natural phrasing ("'Guten Tag' — what does that mean in English?", "boil this down to a few key points", "which companies are named in this report?") surface matching collapses while the semantic router still routes correctly.

• Methodology. bench/accuracy.py sends each prompt to the live allaigate router (taxonomy = data-assistant) and compares the predicted task_label to the ground truth; a keyword heuristic is scored on the same set as a lower bound for "route without a classifier."
• Caveats. Small, hand-labeled set; phrasing is deliberately keyword-light to test semantic understanding vs surface matching. Expand tasks.jsonl and re-run. allaigate's own published figure is 94% at separating close task types (a harder eval than this one).
• Reproduce. CORTIQ_ROUTER_KEY=cortiq_… python3 bench/accuracy.py

Why this matters

Agentic workloads make many gateway calls per task (plan → sub-tasks → tool calls → synthesis). Overhead that looks small per call (tens of ms) multiplies across a run and caps the throughput a single instance can serve. Lower, more predictable tail latency (p99) is the developer-facing differentiator when the model time is equal.

Methodology

• Isolate gateway overhead. A tiny, dependency-free Node mock (bench/mock.js) returns a fixed OpenAI-compatible completion instantly. So measured latency is the gateway's own cost, not model time. Every gateway proxies this same mock.
• Apples-to-apples. Each gateway runs in passthrough / pinned mode (no routing or model-selection logic) so we compare raw proxy cost. Cortiq's intelligent routing (cortiq-auto) adds one classifier round-trip — that's a feature, measured separately, not folded into this overhead number.
• Load. Apache Bench, keep-alive, tolerant of length variance: ab -k -r -c 20 -n 5000 -p bench/body.json -T application/json.
• Fair to each runtime. Cortiq = one process (Tokio, multi-core). Portkey = one Node process (event loop). LiteLLM = 4 uvicorn workers (its recommended multi-process setup; a single Python worker is GIL-bound to one core and scores ~860 req/s).
• Same box, localhost, warm processes, results are representative of one run.

Caveats (read me)

• This measures gateway overhead only. Real requests are dominated by model latency, which is identical regardless of gateway — but overhead and tail latency are not.
• Localhost + mock backend removes network/model variance on purpose. Your absolute numbers will differ; reproduce with bench/run.sh.
• ab reports LiteLLM "Failed requests" as a non-zero count — these are response content-length variations, not HTTP errors (all responses are 200, verified).
• Tested versions: Cortiq Gateway (this repo, --release), LiteLLM proxy via uvicorn litellm.proxy.proxy_server:app --workers 4, Portkey @portkey-ai/gateway, Apache Bench. Hardware: Apple Silicon, macOS.

Reproduce

# prerequisites: node, ab (Apache Bench), a release build of the gateway
cargo build --release

# optional competitors (the harness auto-detects them):
pip install 'litellm[proxy]' uvicorn
npm install -g @portkey-ai/gateway

# run (tune load via env)
CONC=20 REQ=5000 bash bench/run.sh

The harness (bench/run.sh) starts the mock, then benchmarks the mock directly and each available gateway, printing a throughput / p50 / p90 / p99 table.