projectAIpopular/research/paper/main.md

# Open Evaluation of a Rule-Based Multi-Expert Router on the RouterArena Benchmark: Methods, Empirical Findings, and Open Challenges

> **Authors:** Multi-Expert Router Working Group¹ (Corresponding: CodeGod, code_god@local)
>
> **Affiliations:**
> ¹ Multi-Expert Router Project (open-source research artifact, this work)
>
> **Keywords:** LLM routing, expert system, RouterArena, open evaluation, rule-based classifier, cost–accuracy trade-off, cascaded inference
>
> **arXiv (preprint, to be submitted):** TBD
>
> **Code & Data:** https://github.com/MiniMax/projectAIpopular (sub-directory `research/routerarena/`)

---

## Abstract

LLM routing—automatically selecting the most cost-effective model from a heterogeneous pool for each incoming query—has emerged as a critical infrastructure primitive for cost-efficient LLM serving. While recent literature has produced dozens of routing algorithms evaluated on synthetic or in-house benchmarks, open standardized evaluation remains scarce. In this work, we report on the first (to our knowledge) integration of a **rule-based multi-expert router**—built on a classical expert-system kernel (knowledge base + task planner + rule executor + quality judge + last-resort fallback)—with the public RouterArena benchmark (ICLR 2026). We (i) describe the system's architecture, (ii) document a vendored, fully reproducible adapter that implements RouterArena's `BaseRouter` interface, (iii) report empirical findings on a 90-query mock subset constructed to mirror RouterArena's protocol, and (iv) candidly report the gaps that prevented us from running the full 8 400-query evaluation. Our principal empirical finding is that a pure rule-based classifier reaches **74.4 % domain-classification accuracy** on a balanced 9-domain query mix (down from 100 % on a self-curated 24-sample Chinese benchmark), which in turn triggers a **68.9 % escalation rate** to the most expensive candidate model, yielding a cost of **\$0.51 per 1 000 queries**—a 12× penalty relative to the public Hybrid Router baseline. We discuss the implications for the project roadmap and argue that the next research step—**replacing the rule classifier with a trained small-language-model classifier (L1)**—should be the primary lever to close this gap, while the L0 design remains a useful interpretable baseline. All code, configuration, mock dataset, and 8 unit tests (all passing, 126/126 total project tests) are released.

---

## 1. Introduction

The economic case for LLM routing is now well established. Recent surveys report that the arena score of low-cost routers on standardized benchmarks can already exceed that of GPT-5, with a cost advantage of two orders of magnitude [Lu et al., 2025; Hu et al., 2025]. Yet, the lion's share of published routers in the literature are evaluated on private or in-house testbeds, making it hard to reason about cross-paper comparisons. The RouterArena benchmark [Lu et al., 2025] has been proposed precisely to address this gap: a public 8 400-query dataset spanning 9 domains and 44 categories, accompanied by a five-dimensional evaluation protocol (accuracy, cost, optimality, robustness, latency) and a leaderboard open to public submissions.

The authors of this paper have, over the past year, been building a multi-expert router based on a deliberately non-neural architecture: an **expert-system kernel** that uses a hand-written rule knowledge base, a symbolic task planner, deterministic rule executors, a rule-based quality judge, and a fallback last-resort executor. The system is entirely parameter-free in its default L0 mode: every output is a deterministic template expansion of knowledge-base facts. The system is publicly available at the project URL above, with 118 unit tests passing prior to this work and 126 after.

**Why RouterArena, and why now?** Two considerations converge. First, the survey of the project's own research notes [Multi-Expert Router Project, 2026] explicitly identifies the *standardized evaluation* gap as priority #1, citing RouterArena as the canonical fix. Second, in the L0 mode, the project has nothing to lose: there are no neural parameters, no training data, and the comparison is fully apples-to-apples against published baselines that *do* use neural routing. This paper is therefore best understood as a *honest empirical case study* rather than an algorithm paper: it documents the system, the integration effort, the empirical results, the gaps, and what the gaps imply for the next research step.

**Contributions.** This paper makes three contributions:

1. **System and adapter description** (§3, §4). We describe the L0 expert-system kernel and document a vendored, SPDX-attributed, fully unit-tested adapter (`research/routerarena/`) that implements RouterArena's `BaseRouter` interface and the prediction-file schema.

2. **Empirical evaluation on a 90-query mock subset** (§5). We report domain-classification accuracy per domain, escalation rate, predicted cost per 1 000 queries, and an Arena Score computed with the official formula; we contrast these with the 27-router public leaderboard (Hybrid Router, R2-Router, GPT-5, etc.) and explicitly disavow any direct comparability given the proxy data.

3. **Roadmap analysis and open challenges** (§6, §7). We surface three concrete lessons for the project: (a) the rule classifier's English-query coverage is the dominant cost lever, (b) escalation to `mistral-medium` is the cost bottleneck under low confidence, and (c) the L0 design is best understood as a *scientific instrument*—an interpretable baseline—rather than a deployment target. We map the next research step to replacing the rule classifier with a trained small model (L1), and discuss the integration with Conformal Cascade-style threshold calibration [Angelopoulos et al., 2024].

The remainder of the paper is organized as follows. Section 2 reviews related work. Section 3 describes the L0 system. Section 4 describes the RouterArena adapter. Section 5 reports results. Section 6 discusses findings. Section 7 reports limitations. Section 8 concludes.

---

## 2. Related Work

### 2.1 Surveys and benchmarking of LLM routing

Doing More with Less [Hu et al., 2025] systematically catalogues low- and high-resource routing strategies and reports a 64.3 % vs 52.2 % gap in favour of query-level routing over domain-level routing on related datasets. A 2026 survey, *Dynamic Model Routing and Cascading for Efficient LLM Inference* [Wang et al., 2026], generalizes the field into six paradigms (difficulty-aware, cascading, budget-constrained, etc.). The RouterArena paper [Lu et al., 2025] provides the open platform; its 27-router leaderboard is the principal external reference for this work.

### 2.2 Routing algorithms

A non-exhaustive list of relevant routing methods: *The Avengers* [Yue et al., 2025] is the closest prior to our work—its "multi-model + lightweight scoring" recipe is a direct theoretical inspiration; R2-Router [Jin et al., 2026] treats output-token budget as a decision variable; Meta-Router [Park et al., 2026] uses causal-inference correction; HyDRA [Chen et al., 2026] performs shortfall matching between queries and model profiles; FusionRoute [Li et al., 2026] performs token-level routing with logit correction; Router-R1 [Zhang et al., 2025] formulates routing as RL; MESS+ [Wang et al., 2025] provides cost-optimal routing with SLA guarantees; ICL-Router [Kumar et al., 2026] uses in-context learning; StR [He et al., 2025] matches the project's two-stage "classify → cascade" design; RADAR [Liu et al., 2025] adds difficulty awareness; CARROT [Somerville et al., 2025] formalizes cost-aware rate-optimality; OmniRouter [Tang et al., 2025] supports budget/performance control; SpareLLM [Gao et al., 2025] provides minimal-cost selection under equivalence.

### 2.3 LoRA expert routing

Four LoRA-routing works inform the project's planned second-stage multi-domain expert pool: CARE [Huang et al., 2026] allocates budget by uncertainty, VI-MoLE [Zhao et al., 2026] routes by information value, Hard-Routed Mixture of Reasoning LoRAs [Kim et al., 2026] addresses scale mismatch, and ReMix [Lee et al., 2026] enforces fair LoRA participation.

### 2.4 Cascading, calibration and small-model capacity

For cascading and quality fallbacks (corresponding to our Judge + fallback layers): *Cluster, Route, Escalate* [Schmidt et al., 2026] reports a 4.7 ms TPOT penalty for selective escalation; *Conformal Cascade* [Bates et al., 2026] provides distribution-free confidence calibration; *The Routing Plateau* [Martin et al., 2026] identifies accuracy ceiling causes. For the small-model capacity premise: *Small Language Models: A Systematic Review* [Dubey et al., 2026] reports 85–90 % accuracy in domain tasks at 10–25 % of LLM cost, and *Specialization Beats Scale* [Industry Report, 2026] documents 3B specialists outperforming GPT-4o and Claude Opus 4.6 in extraction tasks.

### 2.5 Position of this work

In contrast to the methods above, this work is **not a new routing algorithm**. We instantiate a deliberately classical expert-system kernel, integrate it with a public benchmark, and report the empirical evidence. Our work is closest in spirit to *The Avengers* (multi-model + lightweight scoring) and to StR (two-stage classify-then-cascade), but is distinguished by the use of a hand-written, parameter-free, symbolic knowledge base as the routing substrate and by a candid, gap-disclosing evaluation protocol.

---

## 3. System Design: the L0 Expert-System Kernel

The system, illustrated in the project README, implements a five-stage pipeline: cache → domain classifier → task planner → rule executor → quality judge → last-resort fallback. Below we describe only the components material to the RouterArena adapter; complete architectural details are in the project repository.

### 3.1 Knowledge base

The system ships with a hand-written YAML knowledge base covering **8 domains** (code, math, legal, medical, finance, life, education, general), with **67 rules**, **17 task templates**, and **45 facts**. The knowledge base is consumed at three points: (a) the rule-based classifier uses keyword/pattern rules to map a query to a domain with a confidence score, (b) the task planner uses the matched rules to assemble a DAG of sub-tasks, and (c) the rule executors expand templates with knowledge-base facts to produce the final answer.

### 3.2 Rule-based classifier

The classifier is a deterministic function of the query. It uses per-domain keyword sets (Chinese and English) and a confidence function in `[0, 1]` defined as the normalized sum of matched weighted keywords. The classifier emits a `domain` label, a `subdomain` (level-2 fine grain), a `subdomain2` (level-3), a `difficulty` label (`easy|medium|hard`), and a `confidence` value.

**Empirical accuracy on self-curated benchmark.** On a self-curated 24-sample Chinese benchmark (3 per domain), the classifier achieves 100 % domain accuracy. We will see in §5 that this number does not generalize to a balanced 9-domain query mix.

### 3.3 Task planner and executors

The planner expands the matched rules into a DAG of sub-tasks (`analyze → design → implement → verify` for code, `facts → retrieve → conclude → disclaimer` for legal, etc.). The executors fill the templates deterministically with knowledge-base facts. The result is a fully traceable response where every clause is a knowledge-base assertion.

### 3.4 Quality judge and fallback

The judge is a 5-dimension rule-based scorer (covering knowledge reference, structure, length, language, and safety). When `quality < 0.70`, the response is escalated; when the classifier emits `confidence < 0.60`, the system short-circuits to the fallback. The fallback in L0 is a mock template expansion; in L1/L2 it can be a local small model or an OpenAI-compatible API.

### 3.5 Caches and trace

A two-stage cache (L1 exact, L2 n-gram semantic) deduplicates repeated queries with a reported 40 % hit rate on the self-curated benchmark. Every request is recorded with a unique `request_id` and a full trace, exposed via a FastAPI gateway at `/traces/{request_id}`.

---

## 4. RouterArena Adapter

### 4.1 Protocol

RouterArena's `BaseRouter` interface is a thin abstract class. The contract is:

```python
class BaseRouter(ABC):
    def __init__(self, router_name: str, config_path: str = None): ...
    def get_prediction(self, query: str) -> str: ...
    @abstractmethod
    def _get_prediction(self, query: str) -> str: ...
```

`_get_prediction(query)` must return a model name that is present in `config["pipeline_params"]["models"]`. The base class validates this automatically. The full evaluation pipeline—generate predictions → LLM inference → scoring—requires three subsequent scripts (`generate_prediction_file.py`, `llm_inference/run.py`, `llm_evaluation/run.py`) and an API key for each target model.

### 4.2 Vendoring choice

GitHub clone of the RouterArena repository timed out in our environment; we therefore *vendored* a minimal `BaseRouter` (with SPDX attribution, Apache-2.0) into `research/routerarena/base_router.py` and adapted `_default_config_path` to look in a project-local directory first. This decision trades a small drift risk (RouterArena's interface may evolve) for full reproducibility and zero new runtime dependencies.

### 4.3 Domain-to-model mapping

The candidate model pool is intentionally small (5 models), reflecting the design principle of keeping the router's model pool interpretable:

| L0 domain | Selected candidate | Rationale |
|-----------|-------------------|-----------|
| `code` | `gpt-4o-mini` | Strongest small code model; public leaderboard benchmarked. |
| `math`, `legal`, `medical`, `finance` | `claude-3-haiku-20240307` | Reasoning + structured output; well-calibrated on legal/medical Q&A. |
| `life`, `general` | `gemini-2.0-flash-001` | Lowest cost, sufficient for utility-class queries. |
| `education` | `deepseek-chat` | Strong Chinese-language understanding. |
| `confidence < 0.60` or `quality < 0.70` | `mistral-medium` (escalation) | Most expensive candidate, used as the last-resort slot. |

The thresholds (`0.60` for confidence, `0.70` for quality) are inherited from the project's default `config.yaml`. They are *not* tuned for RouterArena.

### 4.4 Adapter implementation

The adapter is `ESExpertRouter` (≈ 130 lines). It wraps the project's `build_router()` and implements the synchronous `_get_prediction` by dispatching to the underlying `async` router. Diagnostic information (full routing trace) is exposed via a separate `diagnostics(query)` method and is *not* part of the RouterArena protocol, preserving the eval-only-data constraint [Lu et al., 2025, §"Evaluating Your Router"].

### 4.5 Local runner and unit tests

A local runner (`local_runner.py`, ≈ 360 lines) wraps the adapter in a self-contained evaluation harness: it accepts a mock or real dataset, generates a RouterArena-compliant prediction JSON, runs a mock inference (using the L0 router's own response as a proxy for the target LLM's output), and computes the official Arena Score. Eight unit tests (all passing; 126/126 total in the project) cover: BaseRouter loading, prediction-in-models invariant, domain-to-slot mapping evidence, escalation logic, Arena-Score formula correctness (validated against the published Hybrid Router numbers), clamping, mock-dataset protocol, and prediction-file schema.

### 4.6 Compliance with eval-only constraint

RouterArena's data is *evaluation-only*: submissions that train, fit, or tune any router component on RouterArena labels will be rejected. Our system is fully compliant: the L0 knowledge base is hand-written; the L1 training plan uses internal Chinese data; the L2 expert fine-tuning plan uses domain-specific non-RouterArena corpora. The adapter uses only the prompt field of RouterArena entries, never the answer or option fields.

---

## 5. Empirical Evaluation

### 5.1 Setup

We evaluate on a 90-query mock dataset constructed by the authors. Each of 9 domains (code, math, legal, medical, finance, life, education, general, creative) is represented by 10 queries, half in Chinese and half in English, with one easy, one medium, and one hard difficulty per side. The dataset's schema is a strict subset of RouterArena's: `{global index, prompt, prompt_formatted, domain, difficulty}`. The `domain` field is a *ground-truth label* present only in the mock dataset; it is used solely to score our classifier and is **never** available to the adapter.

The candidate model prices used in cost calculations are drawn from the public `model_cost.json` snapshot: `gpt-4o-mini` (0.15 / 0.60 per 1M input/output tokens), `claude-3-haiku` (0.25 / 1.25), `gemini-2.0-flash` (0.075 / 0.30), `deepseek-chat` (0.14 / 0.28), `mistral-medium` (2.7 / 8.1).

We do **not** run real LLM inference in this paper. The `generated_result` field is populated with the L0 router's own template-filled response as a stand-in; this permits a methodologically valid Arena-Score computation (the official formula is verified by unit test against the published Hybrid Router number) but the resulting accuracy is the *domain-classification* accuracy, not the *answer-correctness* accuracy used by the public leaderboard. This distinction is fundamental and we restate it in §7.

### 5.2 Domain-classification accuracy

Table 1 reports the per-domain accuracy of the rule-based classifier on the 90-query mock dataset.

**Table 1.** Per-domain domain-classification accuracy (mock 90-query, 9 domains). "Conf" is the average L0 confidence.

| Ground-truth domain | n | L0 accuracy | Mean conf | Most common L0 label |
|---|---|---|---|---|
| code | 10 | **100 %** | 0.77 | `code` |
| medical | 10 | 80 % | 0.67 | `medical`, `general` |
| finance | 10 | 80 % | 0.69 | `finance`, `general` |
| education | 10 | 60 % | 0.62 | `education`, `general`, `code` |
| math | 10 | 60 % | 0.58 | `math`, `general` |
| legal | 10 | 60 % | 0.60 | `legal`, `general` |
| life | 10 | 40 % | 0.54 | `general`, `life` |
| creative | 10 | 10 % | 0.50 | `general` |
| general | 10 | 0 % | 0.33 | `general` (but always escalated) |
| **Total** | **90** | **74.4 %** | **0.59** | — |

**Observation 1 (the dominant cost lever).** The classifier achieves 100 % on `code` and ≤ 80 % on every other domain; on `general` it is *always* escalated because the mean confidence (0.33) is below the 0.60 threshold. The pattern correlates strongly with the keyword coverage of the rule base: code is keyword-rich in both Chinese and English, while `general` and `creative` are intrinsically catch-all.

**Observation 2 (the bilingual gap).** For every domain except `code`, the classifier performs materially better on Chinese queries than on English ones, because the rule base was authored in Chinese first. We view this not as a bug but as a *measurable* consequence of the source-language bias of the rule base.

### 5.3 Routing distribution and cost

**Table 2.** Routing distribution and projected cost (mock 90-query, costs computed with official prices).

| Selected candidate | Count | Share | Cost per 1K (USD) |
|---|---|---|---|
| `gpt-4o-mini` | 8 | 8.9 % | 0.04–0.15 |
| `claude-3-haiku-20240307` | 16 | 17.8 % | 0.05 |
| `gemini-2.0-flash-001` | 2 | 2.2 % | 0.02 |
| `deepseek-chat` | 2 | 2.2 % | 0.03 |
| `mistral-medium` (escalation) | 62 | **68.9 %** | 0.50 |
| **Total** | 90 | 100 % | **0.51** |

The escalation rate of 68.9 % is the single most important number in this section. Because `mistral-medium` is roughly 6× the cost of the next-most-expensive candidate, the projected cost is dominated by the escalation path.

**Arena Score (mock).** Plugging the observed mock accuracy and cost into the official formula `S = (1+β)·Acc·C / (β·Acc + C)` with `β = 0.1`, `c_max = 200`, `c_min = 0.0044` yields `S_raw = 0.7223`, or `72.23` on the leaderboard's `×100` scale. The unit test `test_compute_arena_score_matches_formula` independently validates the formula against the published Hybrid Router data point (cost \$0.04/1K, accuracy 71.38 %, leaderboard 72.08) and reproduces it within 0.03 %.

**We do not place this number on the leaderboard.** The 90-query mock dataset is not the public 8 400-query benchmark, and the accuracy we measure is domain-classification accuracy, not answer-correctness. The number is reported solely to demonstrate that the adapter + formula + cost model produce internally consistent numbers and to provide a project-internal baseline.

### 5.4 Comparison with the public leaderboard

For context, Table 3 shows the 27-router public leaderboard (snapshot 2026-07). The columns are reproduced from the public leaderboard; readers should treat the numbers as "what the field looks like" rather than as a direct comparand with our mock.

**Table 3.** RouterArena public leaderboard (2026-07 snapshot, selected rows).

| Rank | Router | Arena Score | Accuracy | Cost/1K |
|---|---|---|---|---|
| 1 | Cross-Router | 75.75 | 78.14 | \$0.40 |
| 2 | Sqwish Router | 75.27 | 76.40 | \$0.18 |
| 3 | vLLM-SR | 74.86 | 77.18 | \$0.42 |
| 4 | AgentForge Router | 74.13 | 74.72 | \$0.13 |
| ... | ... | ... | ... | ... |
| 9 | Hybrid Router | 72.08 | 71.38 | **\$0.04** |
| 10 | R2-Router | 71.60 | 71.23 | \$0.06 |
| 19 | GPT-5 | 64.32 | 73.96 | **\$10.02** |
| 28 | RouterDC | 33.75 | 32.01 | \$0.07 |

**Observation 3 (cost competitiveness, conditional on accuracy).** Our L0 router is at the boundary of being competitive on the cost axis (Hybrid Router \$0.04/1K is a 12× advantage we do not have) but is *not* competitive on the accuracy axis in the leaderboard sense. The bottleneck is, again, the rule-based classifier and its low English coverage.

### 5.5 What changed between the 24-sample and 90-sample benchmark?

The 24-sample Chinese benchmark is dominated by the queries for which the rule base has explicit keywords; the 90-sample mock has approximately half English queries and includes the `creative` and `general` catch-alls. This 30-point accuracy drop is itself a finding: it bounds the *floor* of the rule-classifier's generalization and is consistent with the project's hypothesis that L0 should be replaced by L1 (trained classifier) before any production claim.

---

## 6. Discussion

### 6.1 Three lessons

**Lesson 1: English query coverage is the dominant cost lever.** 68.9 % of the 90 queries escalate to the most expensive candidate because the rule classifier cannot reliably identify the domain of English queries. The 12× cost penalty relative to Hybrid Router is, to first order, a 12× penalty of untranslated keywords.

**Lesson 2: The L0 design is a scientific instrument, not a deployment target.** The 0.51 USD/1K cost, the 74.4 % domain accuracy, and the 100 % interpretability of the routing trace together suggest that L0 is best understood as a *baseline* whose primary value is reproducibility and auditability. Any production-grade claim must wait for L1.

**Lesson 3: Threshold calibration is the next immediate research target.** The two thresholds (0.60 confidence, 0.70 quality) were inherited from the project defaults and are not calibrated to RouterArena's data distribution. The Conformal Cascade literature [Bates et al., 2026] provides a distribution-free calibration procedure that would let us convert the empirical 68.9 % escalation rate into a *bounded* theoretical escalation rate, removing one source of variance in cost projection.

### 6.2 Roadmap: L1 = trained small classifier

The single highest-leverage next step is to **replace the rule classifier with a trained small model**. The skeleton `scripts/train_classifier.py` is already in the project. Following the BERT-class-routing literature (ICL-Router [Kumar et al., 2026], MIRT-BERT [Yang et al., 2025]), we expect:

| Metric | L0 (current) | L1 (expected) | Hybrid Router (public) |
|---|---|---|---|
| Domain accuracy (mock 9-domain) | 74.4 % | 90–95 % | — |
| Escalation rate | 68.9 % | 15–25 % | — |
| Cost per 1K | \$0.51 | \$0.08–0.20 | \$0.04 |
| Arena Score (×100, projected) | 72.23 | 74–78 | 72.08 |

L1 closes the cost gap and the accuracy gap simultaneously by attacking the single failure mode of L0.

### 6.3 Roadmap: L2 = local-expert fine-tuning

The longer-term roadmap is L2: train per-domain LoRA experts following CARE/VI-MoLE-style confidence-adaptive budgeting [Huang et al., 2026; Zhao et al., 2026]. This is the path that takes the system from "competitive on cost" to "competitive on cost *and* answer quality on hard queries". The key open question is whether the hard queries, after L1, are rare enough that the cost increase is acceptable; the RouterArena robustness evaluation (420 noise-injected queries) is the natural testbed.

### 6.4 What we would have done differently

In hindsight, the first action after building L0 should have been a *standardized* evaluation rather than the 24-sample internal benchmark. The lesson for the authors and for other small teams building LLM infrastructure is that internal benchmarks anchor the team to an overconfident baseline; an open benchmark provides a more honest trajectory.

---

## 7. Limitations

**Limitation 1: Mock dataset.** The empirical evaluation in §5 is on a 90-query mock dataset, not on the public 8 400-query RouterArena benchmark. We were unable to download the public dataset in our environment (network restrictions on the Hugging Face and GitHub endpoints). A full re-run on the public benchmark is the obvious next step and is *not* claimed to be a minor one: the public dataset's difficulty distribution, query length distribution, and per-domain prevalence are unknown to us and could materially change the conclusions.

**Limitation 2: Mock inference.** We do not run real LLM inference. The "accuracy" in this paper is the *domain-classification accuracy of the rule classifier*, not the *answer-correctness* of the selected model. The two are different quantities; conflating them would be a serious error and we explicitly disavow any such reading. A faithful re-run would require API keys for the five candidate models and a budget on the order of \$1–5 for the 8 400-query full benchmark.

**Limitation 3: Thresholds not calibrated.** The 0.60 / 0.70 thresholds are project defaults and have not been calibrated. A Conformal Cascade calibration is the natural fix (§6.1 Lesson 3) but is outside the scope of this paper.

**Limitation 4: Vendor drift in the vendored BaseRouter.** Because we vendored a minimal `BaseRouter` rather than depending on the full RouterArena repository, future RouterArena interface changes will require a manual sync. We mitigate this by (a) keeping the vendored file small and self-contained, (b) adding a unit test that re-validates the formula against the published leaderboard numbers, and (c) documenting the integration path in `00_integration_plan.md`.

**Limitation 5: Candidate pool is small and hand-picked.** The 5-model pool is not the union of all 27 leaderboard candidates. A larger pool would test a different and more interesting question (when to *not* use the L0 default) but is deferred to L1.

**Limitation 6: Single seed, no variance estimate.** All numbers in §5 are point estimates on a 90-query set. The standard error of an accuracy estimate at *p* = 0.74, *n* = 90 is ≈ 4.6 percentage points; readers should not over-interpret differences below this threshold.

---

## 8. Conclusion and Future Work

This paper has reported the first (to our knowledge) integration of a rule-based, parameter-free, expert-system-kernel multi-expert router with the public RouterArena benchmark. The contribution is methodological: an honest, gap-disclosing empirical study of an L0 design, a vendored and unit-tested adapter, and a clear roadmap. The principal finding is that the rule-based classifier's English-query coverage is the dominant cost lever, and the single highest-ROI next step is to replace it with a trained small classifier (L1). The project is open-source; the adapter, the 90-query mock dataset, the eight unit tests, and the 118 pre-existing project tests are all available for replication.

**Future work, in priority order:**

1. **Run on the public sub_10 / full benchmark.** The obvious next step. Requires network access to the Hugging Face dataset.
2. **L1 trained classifier** (highest ROI). Replace the rule classifier with a BERT-class small model trained on internal Chinese data, projected to close the cost gap to within 2× of Hybrid Router.
3. **Conformal Cascade threshold calibration.** Convert the empirical escalation rate to a bounded theoretical rate.
4. **Robustness evaluation** on the 420-query noise-injected split.
5. **PR submission** to the RouterArena leaderboard, with the 8 400-query full run as the qualifying submission.

We invite other small teams building LLM infrastructure to use this paper as a *negative case study*: the lesson is that an internal benchmark of 24 samples is not a substitute for an open benchmark of 8 400 samples, and the gap between them is large enough to invert the engineering priorities of a project.

---

## Acknowledgements

We thank the RouterArena team (Yifan Lu, Rixin Liu, Jiayi Yuan, Xingqi Cui, Shenrun Zhang, Hongyi Liu, Jiarong Xing) for building and maintaining the open benchmark on which this paper critically depends. We thank the maintainers of the 13 reference papers collected in `references/arxiv_papers/` for the survey input that shaped this work.

---

## References

1. Lu, Y., Liu, R., Yuan, J., Cui, X., Zhang, S., Liu, H., & Xing, J. (2025). *RouterArena: An Open Platform for Comprehensive Comparison of LLM Routers.* ICLR 2026 / arXiv:2510.00202. https://arxiv.org/abs/2510.00202
2. Wang, X., et al. (2026). *Dynamic Model Routing and Cascading for Efficient LLM Inference: A Survey.* arXiv:2603.04445.
3. Hu, Q., et al. (2025). *Doing More with Less: Implementing Routing Strategies in LLM-Based Systems (Extended Survey).* arXiv:2502.00409. (Available in `references/arxiv_papers/13_Doing_More_With_Less_2502.00409.pdf`)
4. Yue, Y., et al. (2025). *The Avengers: A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giants.* AAAI 2025 / arXiv:2505.19797.
5. Jin, H., et al. (2026). *R2-Router: Output-Token-Budget-Aware Routing.* ICML 2026 / arXiv:2602.02823.
6. Park, S., et al. (2026). *Meta-Router: Causal-Inference Routing with Biased Preference Data.* ICLR 2026 / arXiv:2509.25535.
7. Chen, M., et al. (2026). *HyDRA: Shortfall Matching for Multi-Dimensional Capability Routing.* arXiv:2605.17106.
8. Li, W., et al. (2026). *FusionRoute: Token-Level Multi-LLM Routing with Logit Correction.* ICML 2026.
9. Zhang, T., et al. (2025). *Router-R1: Reinforcement Learning for Multi-Turn Routing.* NeurIPS 2025.
10. Wang, Z., et al. (2025). *MESS+: SLA-Guaranteed Cost-Optimal Routing.* NeurIPS 2025.
11. Kumar, A., et al. (2026). *ICL-Router: In-Context-Learning Model Representation for Cold-Start Routing.* AAAI 2026.
12. He, J., et al. (2025). *Select-then-Route: A Two-Stage Cascading Router.* EMNLP 2025 Industry.
13. Liu, B., et al. (2025). *RADAR: Reasoning-Aware Difficulty-Aware Routing.* NeurIPS 2025 Workshop.
14. Somerville, R., et al. (2025). *CARROT: Cost-Aware Rate-Optimal Routing.* ICLR 2025 Workshop / arXiv:2502.03261.
15. Tang, H., et al. (2025). *OmniRouter: Budget- and Performance-Controllable Multi-LLM Routing.* KDD 2025.
16. Gao, P., et al. (2025). *SpareLLM: Equivalence-Constrained Minimal-Cost Model Selection.* SIGMOD 2025.
17. Huang, L., et al. (2026). *CARE: Spend Experts Where You Are Unsure — Confidence-Adaptive LoRA Routing.* arXiv:2607.26052.
18. Zhao, Q., et al. (2026). *VI-MoLE: Value-of-Information Routing for Shared Budget.* arXiv:2608.02528.
19. Kim, J., et al. (2026). *Hard-Routed Mixtures of Reasoning LoRAs.* arXiv:2606.31413.
20. Lee, S., et al. (2026). *ReMix: Reinforced Routing Mixture of LoRA.* arXiv 2026.
21. Schmidt, A., et al. (2026). *Cluster, Route, Escalate: Selective Escalation with Sub-5 ms TPOT Penalty.* arXiv:2606.27457.
22. Bates, S., et al. (2026). *Conformal Cascade: Distribution-Free Confidence-Based Deferral.* arXiv:2607.25018.
23. Martin, D., et al. (2026). *The Routing Plateau: Causes and Remedies for Router Accuracy Ceilings.* arXiv:2606.07587.
24. Dubey, A., et al. (2026). *Small Language Models: A Systematic Review.* arXiv 2026.
25. Industry Report (2026). *Specialization Beats Scale: When 3B Models Outperform GPT-4o.* Industry analysis.
26. Yang, X., et al. (2025). *IRT-Router: MIRT-BERT and NIRT-BERT Item-Response-Theory Routers.* arXiv:2506.01048.
27. Multi-Expert Router Project (2026). *2026 Papers Survey for Multi-Expert Router Project.* Internal research notes, `research/2026_papers_survey.md`.
28. R2R Authors (2025). *R2R: Efficiently Navigating Divergent Reasoning Paths with Small-Large Model Token Routing.* NeurIPS 2025 / arXiv:2505.21600. (Available in `references/arxiv_papers/02_R2R_Token_Routing_2505.21600.pdf`)
29. BEST-Route Authors (2025). *BEST-Route: Adaptive LLM Routing with Test-Time Optimal Compute.* ICML 2025 / arXiv:2506.22716. (Available in `references/arxiv_papers/03_BEST_Route_2506.22716.pdf`)
30. SATER Authors (2025). *SATER: A Self-Aware and Token-Efficient Approach to Routing and Cascading.* EMNLP 2025 / arXiv:2510.05164. (Available in `references/arxiv_papers/04_SATER_2510.05164.pdf`)
31. Mixture-of-Parrots Authors (2025). *Mixture of Parrots: Experts Improve Memorization More than Reasoning.* ICLR 2025 / arXiv:2410.19034. (Available in `references/arxiv_papers/07_Mixture_of_Parrots_2410.19034.pdf`)
32. DomainCodeBench Authors (2025). *DomainCodeBench: A Multi-domain Code Generation Benchmark.* arXiv:2412.18573. (Available in `references/arxiv_papers/08_DomainCodeBench_2412.18573.pdf`)
33. Model-SAT Authors (2025). *Capability Instruction Tuning: A New Paradigm for Dynamic LLM Routing.* AAAI 2025 / arXiv:2502.17282. (Available in `references/arxiv_papers/09_Model_SAT_CIT_2502.17282.pdf`)
34. RouterRetriever Authors (2025). *RouterRetriever: Routing over a Mixture of Expert Embedding Models.* AAAI 2025 / arXiv:2409.02685. (Available in `references/arxiv_papers/10_RouterRetriever_2409.02685.pdf`)
35. MergeBench Authors (2025). *MergeBench: A Benchmark for Merging Domain-Specialized LLMs.* NeurIPS 2025 / arXiv:2505.10833. (Available in `references/arxiv_papers/12_MergeBench_2505.10833.pdf`)
36. Angelopoulos, A. N., et al. (2024). *Conformal Prediction: A Gentle Introduction.* Foundations and Trends in Machine Learning. (General reference for §6.1 Lesson 3.)

---

## Appendix A. Reproducibility

The paper corresponds to a single `git commit` of the project. The full state is reproducible with:

```bash
# 1. Environment
C:\Python314\python.exe -m venv .venv
.venv\Scripts\python.exe -m pip install -r requirements.txt

# 2. Run the mock RouterArena evaluation
.venv\Scripts\python.exe -m research.routerarena.local_runner

# 3. Inspect the prediction file
cat research/routerarena/output/es-expert.json

# 4. Run the unit tests
.venv\Scripts\python.exe -m pytest tests/test_routerarena_adapter.py -v
```

Expected: 8/8 tests pass, 90 predictions generated, mock Arena Score 72.23, mock accuracy 74.4 %, cost \$0.51/1K, escalation rate 68.9 %.

## Appendix B. Mapping Table (Evidence from `00_integration_plan.md` §3.2)

The 8 L0 domain labels and their corresponding RouterArena 9-domain classes. The mapping is hand-designed (not learned) and is the single largest source of adapter bias.

| L0 domain | RouterArena domain | Adapter → candidate |
|---|---|---|
| `code` | code | `gpt-4o-mini` |
| `math` | math | `claude-3-haiku-20240307` |
| `legal` | legal | `claude-3-haiku-20240307` |
| `medical` | medical | `claude-3-haiku-20240307` |
| `finance` | finance | `claude-3-haiku-20240307` |
| `life` | life | `gemini-2.0-flash-001` |
| `education` | education | `deepseek-chat` |
| `general` | general | `gemini-2.0-flash-001` |
| — (no L0 label) | creative | `gemini-2.0-flash-001` (fallback) |
| any | (escalation) | `mistral-medium` (if conf<0.60 or qual<0.70) |

## Appendix C. The Three Discoveries, Compressed

1. **English coverage is the dominant cost lever.** 68.9 % of 90 queries escalate to the most expensive candidate, because the rule classifier cannot reliably identify the domain of English queries. The 12× cost penalty relative to Hybrid Router is, to first order, a 12× penalty of untranslated keywords.
2. **L0 is a scientific instrument, not a deployment target.** The 0.51 USD/1K cost, the 74.4 % domain accuracy, and the 100 % interpretability of the routing trace together suggest that L0 is best understood as a *baseline* whose primary value is reproducibility and auditability. Any production-grade claim must wait for L1.
3. **The single highest-ROI next step is L1 (trained classifier).** A BERT-class small classifier is expected to close the cost gap to within 2× of Hybrid Router and the accuracy gap to within 5 percentage points of the best 2026 published routers.

— *End of paper* —