feat(benchmark): hard tests that light up the high tier — judged on routing, not essays
Two "hard" probes (full suite only): a compiler-backend design and a security
protocol design, each dense enough for the router's complexity model to rate
them high. They are judged on the ROUTER's decision (judge: "route") — pass
means tier=high was reached and the request went to the big model; the answer
itself is not scored (open-ended essays aren't keyword-judgeable, and
reasoning models happily burn 3000+ thinking tokens on them — measured). With
max_tokens kept tiny the probes cost cents even on the priciest model, while
the cost column tells the whole story: $0.0148 for the high-tier request next
to $0.00005 for the easy ones.
Each run picks a random variant of the prompt: the router deliberately scores
familiar text as less novel, so a fixed wording would decay to medium over
repeated runs. In pinned-models mode route-judged tests are skipped.
Also: creative-writing budget 96→512 (deepseek was spending the whole cap
thinking and returning an empty poem), and when a model does burn its entire
budget inside <think> the table now says so ("budget spent on reasoning")
instead of the misleading "empty answer".
Server-side (cortiq-router prod, both replicas rolling-restarted): the high
band started at 0.66 while the complexity model tops out ~0.65 on the hardest
single prompts — high was unreachable. Band boundary moved to 0.50 based on
measured distribution (easy 0.25-0.36, medium 0.37-0.49, hard 0.50-0.65);
config backed up as config.prod.toml.bak-tiers.
This commit is contained in:
Vendored
+1
@@ -618,4 +618,5 @@ export default {
|
||||
"keys.h.st.notFound": "Auf der Gateway-Maschine nicht gefunden",
|
||||
"bm.degraded": "Router degradiert",
|
||||
"bm.routerDegraded": "Der Router hat nicht rechtzeitig geantwortet — die Anfrage nutzte die Fallback-Kette; die Klassifikation wird nicht gewertet.",
|
||||
"bm.cutByCap": "Budget ging fürs Denken drauf — keine Antwort mehr",
|
||||
};
|
||||
|
||||
Vendored
+1
@@ -611,4 +611,5 @@ export default {
|
||||
"keys.h.st.notFound": "Not found on the gateway machine",
|
||||
"bm.degraded": "router degraded",
|
||||
"bm.routerDegraded": "The router did not answer in time — the request used the fallback chain; classification is not judged.",
|
||||
"bm.cutByCap": "budget spent on reasoning — answer never came",
|
||||
};
|
||||
|
||||
Vendored
+1
@@ -618,4 +618,5 @@ export default {
|
||||
"keys.h.st.notFound": "No encontrado en la máquina de la pasarela",
|
||||
"bm.degraded": "router degradado",
|
||||
"bm.routerDegraded": "El router no respondió a tiempo — la petición usó la cadena de reserva; la clasificación no se cuenta.",
|
||||
"bm.cutByCap": "presupuesto gastado en razonar — sin respuesta final",
|
||||
};
|
||||
|
||||
Vendored
+1
@@ -618,4 +618,5 @@ export default {
|
||||
"keys.h.st.notFound": "Introuvable sur la machine de la passerelle",
|
||||
"bm.degraded": "routeur dégradé",
|
||||
"bm.routerDegraded": "Le routeur n'a pas répondu à temps — la requête a suivi la chaîne de secours ; la classification n'est pas comptée.",
|
||||
"bm.cutByCap": "budget épuisé en raisonnement — pas de réponse",
|
||||
};
|
||||
|
||||
Vendored
+1
@@ -610,4 +610,5 @@ export default {
|
||||
"keys.h.st.notFound": "Не найден на машине шлюза",
|
||||
"bm.degraded": "роутер деградировал",
|
||||
"bm.routerDegraded": "Роутер не успел ответить — запрос ушёл по резервной цепочке; классификация не засчитывается.",
|
||||
"bm.cutByCap": "бюджет ушёл на размышления — ответ не успел",
|
||||
};
|
||||
|
||||
Vendored
+1
@@ -618,4 +618,5 @@ export default {
|
||||
"keys.h.st.notFound": "Ağ geçidi makinesinde bulunamadı",
|
||||
"bm.degraded": "yönlendirici yedeğe düştü",
|
||||
"bm.routerDegraded": "Yönlendirici zamanında yanıt vermedi — istek yedek zincire gitti; sınıflandırma sayılmaz.",
|
||||
"bm.cutByCap": "bütçe düşünmeye gitti — yanıt gelemedi",
|
||||
};
|
||||
|
||||
Vendored
+1
@@ -618,4 +618,5 @@ export default {
|
||||
"keys.h.st.notFound": "网关所在机器上未找到",
|
||||
"bm.degraded": "路由器降级",
|
||||
"bm.routerDegraded": "路由器未及时响应 — 请求走了备用链路;分类不计入统计。",
|
||||
"bm.cutByCap": "预算全用于思考——未能给出答案",
|
||||
};
|
||||
|
||||
Reference in New Issue
Block a user