English (US)Español (LA)Português (BR)

For banks

Concept

A bank asks three questions: can it reach production, is it safe, and can we trust the numbers? This page answers each question with evidence and limits.

Physical analogy

Think of a bank branch. The model is the clerk who listens. The policy is the rulebook. The code is the cashier who checks the receipt before any money moves.

Technical mechanics

Path to production

The prototype runs the same code as production, with the documented mocks. This list names only what must change. Full table.

  • Identity. Test session today; the bank identity provider and Azure Key Vault before production.
  • State. SQLite on one replica today; PostgreSQL for more than one instance before production.
  • Gold. A labelled mock in the local run today. Gold on Databricks before production; the request-time read is undecided.
  • Policy. Synthetic values today; the approved bank policy in the same format before production.
  • Handoff. One demo advisor view today. A queue to the bank CRM before production, routed by language and specialty.
  • LLM. GLM 5.3 Flash on Fireworks AI today. The same weights on Azure AI Foundry or Databricks before production, with quotas and a spend cap.
  • Capacity. No load test today. A load test as a release gate before production. See sizing.
  • Monitoring. Turn records today. Central traces and alerts for each country before production.
  • Cost. An estimate today. A projection, not a measurement, before production. See cost.

Verify it yourself

You can check the evidence and replay a frozen run offline. You need no model key, no bank data and no cloud credential. The full setup is in Run it locally.

git clone https://github.com/rdorta27/sentinel-engine.git
cd sentinel-engine/sentinel-ai-core
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev,eval]"
export SENTINEL_LLM_CHEAP_MODEL=accounts/fireworks/models/glm-5p3-flash
export SENTINEL_LLM_STRONG_MODEL=accounts/fireworks/models/glm-5p3-flash
python3 -m eval.run verify 2024Q4-train-v1

The two model names are public. They are not keys. The verify command makes no live call. It recomputes a frozen run from its recordings. It compares the result with the run summary.json. At the time of writing, 2024Q4-train-v1, 2024Q4-resolution-v1 and 2024Q4-resolution-v2 match. The team never edits a frozen run. A new measurement is a new run. Evidence index.

The attack suite writes a new run only when you ask for it: SENTINEL_WRITE_EVIDENCE=1 python3 -m pytest tests/adversarial -q.

Safety and control

What the model never receives

  • The customer id. The session binds the tools.
  • Personal identifiers. Code masks them by pattern before any model call.
  • The decision. The model labels the intent. The policy engine decides.

Limit: the masking finds only the patterns that it knows. It does not find other personal data. Decision 004.

Every decision has a reason

  • The policy engine returns a rule id with each decision.
  • A later "why?" gets the stored decision. The model does not make up a new answer.
  • The advisor sees the trace of each step.
Test suite

Attacks

0/42 unsafe outcomes in 42 attacks. The suite counts 3 injection attempts as safe only with the mock model. We also ran them on the real model: 0/3 unsafe. We say so.

evidence/adversarial/20261005T014816Z · totals; evidence/adversarial/20261005T204313Z · totals

Fairness limits

  • The data has no customer segment for the resolution cases, so we show no breakdown by segment.
  • The numbers by language and country are descriptive. Each group is small.

Results by language

Simulation. Intent accuracy of the LLM router on the sealed held-out cases. Each variant holds the same cases, written in that variant. The router scores the same in each variant. See the evidence explorer.

es-AR81.8%

on 77 cases

es-CO81.8%

on 77 cases

es-MX81.8%

on 77 cases

pt-BR81.8%

on 77 cases

A model wrote the Portuguese cases. No native speaker reviewed them. Decision 018.

Common questions

Is it in production?

No. It is a prototype. It runs the code that production would run, with documented mocks. No number on this site is a production measurement.

What is a mock?

A stand-in with the same contract as the production part. Production replaces the backend and keeps the code. The mocks page lists each one with its limit. The architecture diagram marks them.

Why is the resolution ceiling 16 of 56?

The policy lets only those simulated cases resolve. The other cases must end in a handoff or a question. Both the baseline and the LLM router resolve every case under the ceiling. The set cannot tell them apart on resolution.

Then what does the LLM router add?

It labels the intent better. It scores 81.8% against 69.2% for the keyword baseline on 308 held-out turns. That is a simulation on team-written cases.

Does the model decide?

No. The model labels the intent. Code masks, looks up, decides, acts and reads the case back.

What would a pilot require?

The path to production list names the work: identity, state, Gold, policy, handoff, LLM, capacity and monitoring. No date or price is part of this prototype.