Vaani: a voice agent that gets names, dates and numbers right

Vaani answers the phone for a lending company (NBFC). Callers speak Hindi, Hinglish, Marathi, Tamil or Indian English, and before Vaani can help it has to hear a name, date of birth, loan ID or city correctly through noisy speech recognition.

• Live demo: vaani.amantrivedi.com (invite only, synthetic data)
• Code: github.com/trivedi-aman/vaani
• In numbers: 2 helplines · 25 gated tools · 50 test scenarios · 982 matcher test items · 201 automated tests

What it does

Two lines share one agent:

• Customer helpline: application status, EMI schedule, foreclosure quote, pending documents, rejection reasons, complaints and service requests.
• Staff helpdesk: look up and diagnose stuck cases, add notes, escalate, retry a bureau pull, check queues.

Testers call from a phone simulator in the browser, by voice or typing, using one of 50 scenarios. A side panel shows every value as it is heard, normalised, matched and scored.

Design principles

• The model finds; code decides. Nova passes the caller's exact words to a tool. Plain code normalises, matches and scores them.
• Access is decided in code. Identity lives in the call session, never in model arguments.
• Honest about uncertainty. High confidence is accepted, medium is read back, low is asked again.
• A conversation, not a form. Verification is requested only when the request needs the account.

The problem: names

Letter edit distance is wrong for speech in both directions:

• Baadmer → Barmer is 2 letter edits but the same district (sound distance 0.60).
• Mohan → Rohan is 1 letter edit but a different person (sound distance 1.70).

The same customer can also arrive as Sultan Meena, सुल्तान मीणा or Meena Sultan. So Vaani compares sounds, not letters.

How the matcher works

Five deterministic steps in identify/, with no model involved:

1. One phonetic alphabet. Devanagari and Tamil become Roman, with long vowels doubled and retroflex consonants marked. Hindi schwa deletion (mohan, not mohana) and Tamil voicing (murugan) are applied.
2. Fold English spellings. For example ck/q → k, final -ey → e, ay → ai, and doubled consonants collapse. Pandey = Pande, Sanjay = संजय.
3. Split into sounds by greedy longest match over 37 consonants and 10 vowels (kh is one sound).
4. Price differences by articulation: place 0.45, manner 0.45, voicing 0.25, aspiration 0.15, vowel height/position 0.35, length 0.25. Twenty known confusions override the feature cost (e.g. r/ṛ 0.15, v/w 0.15, sh/s 0.30). Two-word names are also tried in swapped order, at a cost of 0.20.
5. Rank and set confidence from the best distance and its lead over the runner-up. High needs a lead of 1.0 or more, medium 0.4 or more.

Spoken values get the same treatment: dates (including relative dates), amounts (do lakh pachaas hazaar = 2,50,000), loan IDs (el en eleven = LN00011), pincodes and districts, in English, Hindi, Marathi and Tamil. Places are matched against all 745 districts and their aliases, and are cross-checked with the pincode.

Verification

A name only checks, it never verifies, because Anil Patel and Anil Patil are only 0.35 apart.

1. An exact identifier (DOB, mobile or loan ID) narrows the pool.
2. The name is checked by sound.
3. A one-time password proves identity.

Failure messages never say which detail was wrong. Three failed attempts lock the call, and at most three OTPs are sent per call.

Evaluation

The test set has 982 items. All three matchers get the same romanised input. Top-1 accuracy:

• Confidence is calibrated: every high, medium and low call was correct.
• Non-customers: of 200 invented callers, 194 were asked again, 5 were read back and 1 was wrongly matched with high confidence (Raman Patil → Raman Patel). That miss is why the OTP verifies identity, not the name.
• Caveats: the tests use text, not real recordings. The rule-made variants favour the matcher, and the hand-written set is not fully held out.

Architecture: one voice turn

1. The browser streams audio over WebRTC.
2. Sarvam transcribes it and detects the language.
3. Silero VAD ends the turn after 0.6 s of silence.
4. A gate selects the tools this caller may use, based on the session.
5. Nova returns a tool call with the caller's exact words, and Vaani speaks a short filler while it runs.
6. The tool re-checks access, runs the identification core and returns a value, a confidence and an action.
7. Nova writes the reply, which is guarded, cleaned for speech and gender-corrected.
8. Sarvam speaks it. Barge-in cancels a pending reply.

Tools by caller state:

Each turn is bounded at 4 tool calls per step, about 5 model calls per turn and 40 messages of history. Any failure produces a spoken apology and never crashes the call.

Deployment

git pull && docker compose up -d --build   # deploy / update
docker compose ps                          # vaani should be (healthy)
docker compose logs -f --tail 50 vaani

Sessions are held in memory, so a restart signs everyone out. Full guide: deploy/EC2.md.

Security

• App: tools read identity from the session and are gated twice. OTPs are salted hashes that expire in 5 minutes, with 3 tries. A reply guard blocks internal notes from reaching customers.
• Server: the app listens on localhost only, behind Caddy. Each call has its own hashed 256-bit token. Strict CSP and security headers are set, and there are no stack traces or API docs in production.
• Container: read-only, non-root, all capabilities dropped, with memory and process limits.
• Sign-in: invite only. Passwords use PBKDF2-SHA256 with 600k iterations. Timing is the same for unknown users and wrong passwords. Limits are 8 wrong attempts per 15 minutes per address, and sessions last 4 hours.
• AWS: MFA everywhere and no root keys. The server role is limited to two Bedrock actions on one model plus SSM. IMDSv2 with hop limit 1, an encrypted disk and automatic security updates.
• Logs: phone numbers, DOBs, OTPs, PINs and tokens are masked.
• Trusted on purpose (demo): the OTP is shown on the simulated phone, and all data is synthetic.

Cost controls

AWS Budgets sends emails at 50% and 80% of actual spend and at 100% of forecast. At 100% of actual spend, it attaches a deny-all-Bedrock policy to the server role. Billing data lags by hours, and Sarvam is billed outside AWS, so the in-app limits are the real brake.

Production path

• Scale matching: verification already scales because of the exact identifier. Staff search would index a coarse phonetic key, then rank candidates by the full distance.
• Learn from read-backs: confirmed pairs can tune confusion costs and thresholds. The rejected read-back rate is the live quality metric.
• Telephony: SIP instead of WebRTC. 8 kHz audio needs retuned thresholds, and SMS OTPs need DLT registration.
• Multi-server: Redis for sessions and limits, a database for records, stateless app servers behind a load balancer, and a TURN relay.
• More scripts: Bengali, Telugu, Kannada, Malayalam and Gujarati.
• DPDP Act: consent, purpose limits, retention and a tamper-evident audit trail.

Known gaps

• Not yet tested on recordings of real callers.
• Marathi number words need review by a native speaker.
• Sound-alike names can't be separated by sound alone; the identifier and OTP handle that.
• A single process keeps state in memory.
• Speech in and out is the largest share of the delay before each reply.
• There's no TURN relay yet, so UDP-blocked networks can only use typing.

Repository and tests