docs: add latency benchmark vs LiteLLM/Portkey

- bench/ reproducible harness: instant mock backend + Apache Bench (ab -k -r)
- measures gateway overhead in passthrough mode (apples-to-apples)
- BENCHMARKS.md: methodology, caveats, results (cortiq ~57k rps p99 1ms vs
  Portkey ~5.8k p99 9ms vs LiteLLM 4-workers ~1.2k p99 59ms)
- README (EN+RU): Performance section + highlight

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
infosave2007
2026-06-30 18:11:26 +03:00
co-authored by Claude Opus 4.8
parent e015555dff
commit a5966dd479
8 changed files with 279 additions and 1 deletions
+14
View File
@@ -35,6 +35,7 @@ One OpenAI-compatible endpoint → automatic model selection from your pool
## ✨ Highlights
- **Fast.** Rust/Tokio: **sub-millisecond p99** gateway overhead — ~10× the throughput of Portkey and ~47× of LiteLLM in a like-for-like proxy benchmark ([details](BENCHMARKS.md)).
- **Drop-in OpenAI API + streaming.** Point any OpenAI client at the gateway and send `model: "cortiq-auto"`; `stream: true` is fully supported (SSE).
- **Multi-protocol.** OpenAI Chat & Embeddings, **Anthropic Messages** (in/out, streaming), and an **MCP** server (`POST /mcp`) exposing routing as tools for agent orchestrators.
- **Intelligent routing.** `complexity.tier` → ordered model pool (`low → local`, `high → cloud`), with fallback and graceful degradation when the router is down.
@@ -219,6 +220,19 @@ Details — [docs/PROTOCOLS.md](https://github.com/infosave2007/cortiq-gateway/b
---
## ⚡ Performance
Gateway overhead proxying the same instant backend, `ab -k -r -c 20 -n 5000`:
| Gateway | req/s | p50 | p99 |
|---|--:|--:|--:|
| **Cortiq Gateway** (Rust) | **~57,300** | 0 ms | **1 ms** |
| Portkey (Node) | ~5,800 | 3 ms | 9 ms |
| LiteLLM (Python, 4 workers) | ~1,200 | 9 ms | 59 ms |
Methodology, caveats and a reproducible harness: **[BENCHMARKS.md](BENCHMARKS.md)**
(`bash bench/run.sh`). Numbers vary by hardware; the ratios are the point.
## Build & test
```bash