English (US)Español (LA)Português (BR)

Evidence explorer

Each bar shows its denominator, its type and the field of the frozen run that it reads. No number here is a production measurement.

Intent accuracy

Simulation

The intent router against the keyword baseline on the same sealed held-out turns. The cases are team-written, not production traffic.

  • Keyword baseline69.2% on 308 turns · 95% range 59.1% to 79.2% · candidates.baseline.breakdown_overall.accuracy
  • LLM router, final version81.8% on 308 turns · 95% range 72.7% to 90.9% · candidates.router_v2.breakdown_overall.accuracy
  • Prompt v3, not served: it failed the safety gate97.4% on 308 turns · 95% range 93.8% to 100.0% · candidates.router_v3.breakdown_overall.accuracy

evidence/evaluation-runs/2024Q4-eval-v8/summary.json

By language

Simulation

Intent accuracy of the final LLM router for each variant. Each variant holds the same cases, written in that variant. The Portuguese cases are model-written and no native speaker reviewed them.

  • es-AR81.8% on 77 turns · candidates.router_v2.variant_losses.by_variant.es-AR.accuracy
  • es-CO81.8% on 77 turns · candidates.router_v2.variant_losses.by_variant.es-CO.accuracy
  • es-MX81.8% on 77 turns · candidates.router_v2.variant_losses.by_variant.es-MX.accuracy
  • pt-BR81.8% on 77 turns · candidates.router_v2.variant_losses.by_variant.pt-BR.accuracy

evidence/evaluation-runs/2024Q4-eval-v8/summary.json

Resolution ceiling

Simulation

The policy sets a ceiling on how many simulated cases may resolve. Both systems resolve every case under the ceiling, so the two systems do not differ. The set cannot separate the two systems on resolution.

  • Cases the policy lets resolve16 of 56 · ceiling.router_v2.resolvable
  • Resolved by the baseline16 of 56 · ceiling.baseline.resolved
  • Resolved by the LLM router16 of 56 · ceiling.router_v2.resolved

evidence/evaluation-runs/2024Q4-resolution-gap-v1/summary.json

Confidence cut-off

Simulation

Kind accuracy of the router with and without a confidence cut-off. The confidence of the model is saturated near the top. A cut-off near the top removes correct answers, not errors. The run holds three settings and no curve, so this page shows the three settings and no slider. The cut-offs stay off.

  • No cut-offs (our setting)99.0% on 198 turns · kind_accuracy.no_cutoffs.kind_accuracy
  • Cut-off rounded to the top54.0% on 198 turns · kind_accuracy.t_act_1_0.kind_accuracy
  • Cut-off before rounding77.8% on 198 turns · kind_accuracy.t_act_unrounded.kind_accuracy

evidence/evaluation-runs/2024Q4-cutoff-diagnosis-v1/summary.json

Attacks

Test suite

Each bar shows the attacks that production code stops. The other attacks pass only with the mock model, or are documented limits. No attack has an unsafe outcome.

  • Prompt injection5 of 8 · categories.A_prompt_injection.blocked_verified
  • Unauthorized access9 of 10 · categories.B_unauthorized_access.blocked_verified
  • Session6 of 6 · categories.C_session.blocked_verified
  • Tool failures6 of 6 · categories.D_tool_failures.blocked_verified
  • Multilingual ambiguity6 of 6 · categories.E_multilingual_ambiguity.blocked_verified
  • Decision disclosure6 of 6 · categories.F_decision_disclosure.blocked_verified
  • All attacks38 of 42 · totals.blocked_verified

Unsafe outcomes, all attacks: 0/42 · totals.unsafe_outcome_rate

evidence/adversarial/20261005T014816Z/summary.json

Latency

SimulationLive model calls on simulation cases

Time of one routed turn with the LLM router, in milliseconds. The calls are live. The cases are team-written.

  • Median (p50)1,474.20 ms 308 turns · candidates.router_v2.latency_ms.p50
  • Slow turn (p95)4,317.14 ms 308 turns · candidates.router_v2.latency_ms.p95

evidence/evaluation-runs/2024Q4-eval-v8/summary.json

Latency of a replay

SimulationReplay of recorded model answers, no live call

The resolution run replays recorded answers. It measures the code, not the model. Do not compare it with the live latency.

  • Median (p50)0.64 ms 56 turns · system.router_v2.latency_ms.p50
  • Slow turn (p95)1.09 ms 56 turns · system.router_v2.latency_ms.p95

evidence/evaluation-runs/2024Q4-resolution-v2/summary.json

Cost

SimulationLive model calls on simulation cases

Total price of the live calls of each router on the same turns. The prices come from the Fireworks model library, as the router models decision records.

  • LLM router, final versionUSD 0.04141 for 308 turns · candidates.router_v2.cost_usd.total

evidence/evaluation-runs/2024Q4-eval-v8/summary.json

The runs are listed in the evidence index. The types follow what is real.