fix(benchmark): fair to reasoning models, fair to router degradation, real sub-cent costs
Three false signals in benchmark results, all seen on a live run:
- Reasoning models were scored ❌ because the per-test max_tokens cap cut
them mid-<think> before the actual answer. Every run now gets +256 tokens
of headroom; non-thinking models stop at EOS as before, thinking ones get
to finish (math-mult on nanbeige: false 0/1 → honest 1/1).
- When the router degraded (timeout → fallback chain) the table showed
"✗ degraded" and counted it against classification accuracy. There was no
classification to judge — such rows now show a muted "⚠ router degraded"
with an explanatory tooltip and are excluded from the accuracy KPI.
- Costs below $0.0001 (cheap flash models) collapsed to "$0.0000". money
formatting now grows precision until two significant digits survive
($0.000035 instead of $0.0000).
2 new strings in all 7 languages.
This commit is contained in:
Vendored
+2
@@ -616,4 +616,6 @@ export default {
|
||||
"keys.h.st.installed": "Auf der Gateway-Maschine installiert",
|
||||
"keys.h.st.configured": "Bereits für dieses Gateway konfiguriert",
|
||||
"keys.h.st.notFound": "Auf der Gateway-Maschine nicht gefunden",
|
||||
"bm.degraded": "Router degradiert",
|
||||
"bm.routerDegraded": "Der Router hat nicht rechtzeitig geantwortet — die Anfrage nutzte die Fallback-Kette; die Klassifikation wird nicht gewertet.",
|
||||
};
|
||||
|
||||
Vendored
+2
@@ -609,4 +609,6 @@ export default {
|
||||
"keys.h.st.installed": "Installed on the gateway machine",
|
||||
"keys.h.st.configured": "Already configured for this gateway",
|
||||
"keys.h.st.notFound": "Not found on the gateway machine",
|
||||
"bm.degraded": "router degraded",
|
||||
"bm.routerDegraded": "The router did not answer in time — the request used the fallback chain; classification is not judged.",
|
||||
};
|
||||
|
||||
Vendored
+2
@@ -616,4 +616,6 @@ export default {
|
||||
"keys.h.st.installed": "Instalado en la máquina de la pasarela",
|
||||
"keys.h.st.configured": "Ya configurado para esta pasarela",
|
||||
"keys.h.st.notFound": "No encontrado en la máquina de la pasarela",
|
||||
"bm.degraded": "router degradado",
|
||||
"bm.routerDegraded": "El router no respondió a tiempo — la petición usó la cadena de reserva; la clasificación no se cuenta.",
|
||||
};
|
||||
|
||||
Vendored
+2
@@ -616,4 +616,6 @@ export default {
|
||||
"keys.h.st.installed": "Installé sur la machine de la passerelle",
|
||||
"keys.h.st.configured": "Déjà configuré pour cette passerelle",
|
||||
"keys.h.st.notFound": "Introuvable sur la machine de la passerelle",
|
||||
"bm.degraded": "routeur dégradé",
|
||||
"bm.routerDegraded": "Le routeur n'a pas répondu à temps — la requête a suivi la chaîne de secours ; la classification n'est pas comptée.",
|
||||
};
|
||||
|
||||
Vendored
+2
@@ -608,4 +608,6 @@ export default {
|
||||
"keys.h.st.installed": "Установлен на машине шлюза",
|
||||
"keys.h.st.configured": "Уже настроен на этот шлюз",
|
||||
"keys.h.st.notFound": "Не найден на машине шлюза",
|
||||
"bm.degraded": "роутер деградировал",
|
||||
"bm.routerDegraded": "Роутер не успел ответить — запрос ушёл по резервной цепочке; классификация не засчитывается.",
|
||||
};
|
||||
|
||||
Vendored
+2
@@ -616,4 +616,6 @@ export default {
|
||||
"keys.h.st.installed": "Ağ geçidi makinesinde kurulu",
|
||||
"keys.h.st.configured": "Bu ağ geçidi için zaten yapılandırılmış",
|
||||
"keys.h.st.notFound": "Ağ geçidi makinesinde bulunamadı",
|
||||
"bm.degraded": "yönlendirici yedeğe düştü",
|
||||
"bm.routerDegraded": "Yönlendirici zamanında yanıt vermedi — istek yedek zincire gitti; sınıflandırma sayılmaz.",
|
||||
};
|
||||
|
||||
Vendored
+2
@@ -616,4 +616,6 @@ export default {
|
||||
"keys.h.st.installed": "已安装在网关所在机器上",
|
||||
"keys.h.st.configured": "已配置为使用此网关",
|
||||
"keys.h.st.notFound": "网关所在机器上未找到",
|
||||
"bm.degraded": "路由器降级",
|
||||
"bm.routerDegraded": "路由器未及时响应 — 请求走了备用链路;分类不计入统计。",
|
||||
};
|
||||
|
||||
Reference in New Issue
Block a user