A fully offline voice assistant for patient portal support, timed end to end on one RTX 5070 Ti with no cloud API. The voice engine, Chatterbox or Kyutai's Moshi, costs far more time than the local language model. The question I wanted answered was where the time goes.

Four stages, each swappable and each swap measured:

  • Listen: faster-whisper turns the question into text.
  • Retrieve: ChromaDB by meaning, plus an exact lookup for record numbers.
  • Answer: a local language model limited to what was retrieved, tested at three sizes.
  • Speak: Chatterbox or Kyutai's Moshi.

The voice engine takes 8 to 17 seconds, the language model 1 to 4

Swapping Chatterbox for Moshi cut the total wait for all three language models.

  • Timed from the end of the question to a spoken answer being ready.
  • Swapping the voice engine saved eight seconds. Swapping the language model saved two.
Chatterbox, 8.8s to generate
Moshi, 3.9s to generate
"Please confirm you are currently taking Metformin 500 milligrams twice daily."

The same sentence takes 8.8s on Chatterbox and 3.9s on Moshi

Both clips read one patient-portal sentence aloud.

Moshi starts making sound before it finishes

The first audible word arrives seconds before the full answer is ready.

  • The first word comes at 3.4 to 6.1 seconds.
  • The full answer is ready at 11.5 to 15.5 seconds.

An idle voice engine made the same setup swing from 14.5 to 31 seconds

With both voice engines loaded on the GPU, one configuration measured 31 seconds on one run and 14.5 on the next.

  • The language model's own generation swung from 2 seconds to 24. Nothing about the model changed.
  • The cause was GPU memory held by the idle voice engine.
  • A deployment loads one voice engine, so the benchmark now does too: a repeatable 11.5 to 15 seconds.
  • Five of the six comparisons were real. One was an artefact of my own test setup.

Fixing record-number lookup dropped safety from 8 of 8 to 5 and 6, mostly on medical advice

An exact-match lookup for record numbers raised retrieval accuracy, and both candidate models then scored lower on the safety tests.

  • The 8 tests: 3 medical advice, 2 emergency, 3 patient data.
  • After it, Qwen3.5 2B scored 5 of 8 and Qwen3 14B 6 of 8. Qwen3 14B was also the judge, so it graded its own answers.
  • 4 of the 5 failures were medical advice: the model asked a question back instead of sending the caller to a doctor.
  • The 1 leak: Qwen3 14B on the Jennifer Santos question.

The 60 percent resolution score came from a model grading itself

In the multi-turn test of whether a call resolves the caller's problem, one model under test was bot, judge and simulated customer.

  • It scored 60 percent resolved against none.
  • The transcripts show two near-identical first answers graded differently.
  • The 60 percent figure is reported as unreliable.

Gemma 4 12B with Moshi: 93.8 quality in 11.5 seconds

Gemma 4 12B with Moshi is the pick, and the usual trade between accurate and fast never appeared.

  • 93.8 out of 100 for answer quality, 99 for sticking to its sources.
  • A full spoken answer in 11.5 seconds from a 7.4 GB model.
  • For lighter hardware: Qwen3.5 2B with Moshi, 86.5 quality and 11.6 seconds in under 3 GB.