Milo

End-to-end pipeline

One conversational turn, from microphone to speaker.

Knowing when the user has finished speaking

The hardest part of a voice agent is not transcription — it is deciding when to reply. Cutting in too early truncates the user; waiting too long feels broken.

A naive silence timer fails on natural speech, because people pause mid-thought. Milo uses two stages: a cheap signal test to notice a pause, then a learned model to decide whether that pause actually ended the turn.

Why this split: the model only runs when the cheap test already suspects the turn ended, so inference happens once per pause rather than continuously. Whisper feature extraction plus an ONNX session on the robot's CPU is far too expensive to run every 20 ms — but perfectly affordable once.

Failure behaviour: if the model fails to load or throws, prediction returns 1.0 and the system falls back to plain silence-timer VAD. Endpointing degrades in quality but never hangs — the agent always eventually replies.

Latency: streaming instead of waiting

Perceived responsiveness is dominated by time-to-first-word, not total response time.

The obvious implementation downloads the entire TTS response, resamples it, then plays it — so the user waits for the whole reply to arrive before hearing anything. Milo instead consumes the HTTP response incrementally: it accumulates roughly half a second of PCM, resamples that chunk, pushes it to the speaker, and repeats. Playback begins while the rest of the sentence is still in flight.

The speaker cannot consume the TTS sample rate directly, so every chunk is resampled 24 kHz → 16 kHz in flight rather than in one pass at the end.

Interrupting the robot mid-sentence

A voice agent that cannot be interrupted feels like a recording, not a conversation.

While audio is playing, a background thread keeps the microphone open and watches for sustained speech energy. It requires several consecutive loud chunks before firing, with a quiet-period reset, so a door closing or a single transient does not count as an interruption.

When it does fire, the effect is immediate: the in-flight HTTP response is closed, the resample-and-push loop breaks, and playback stops. Milo returns to listening rather than finishing a reply nobody is hearing any more.

Hardware constraint: this requires the robot SDK to support recording and playback concurrently. If that call fails, the monitor thread exits cleanly through its finally block and barge-in is silently disabled for that turn — the conversation continues uninterrupted rather than crashing.

Splitting work between robot and cloud

Default is cloud-heavy; the split is configurable.

By default the robot is deliberately thin — it records, uploads, streams back, and plays. All transcription, reasoning, and synthesis happen in the cloud, which keeps the on-device footprint small and the models swappable without touching the robot.

Transcription can optionally move on-device instead, running faster-whisper with int8 quantization on CPU. That trades accuracy and CPU headroom for lower round-trip latency and the ability to keep raw audio local. Endpointing already runs on-device in both configurations.

Running it on a Reachy Mini

The robot app is the thin client. It needs a speech backend to talk to, and credentials to reach it — both supplied through environment variables.

1 · Install on the robot

From the Reachy Mini dashboard at http://reachy-mini.local:8000, choose Install from Hugging Face and search for this Space. Once installed, the app appears as a tile you can start and stop. To run it from source instead, pip install -e . inside the package directory and launch with python -m reachy_mini_milo.

2 · Point it at a backend

This is the step that matters most. MILO_SERVICE_URL defaults to the author's own private Cloud Run deployment, which you will not be able to call. Set it to a speech service of your own that exposes a /speech2speech endpoint — accepting a WAV upload and returning a 24 kHz PCM stream.

3 · Handle authentication

If your backend is publicly reachable, set MILO_GCP_AUTH=false and you are done. If it is a private Cloud Run service, the app fetches a GCP identity token and refreshes it before the hour is out. It resolves credentials in this order, using the first that works:

4 · Tune for your room (optional)

The defaults suit a reasonably quiet space. If Milo triggers on background noise or cuts you off mid-sentence, the endpointing variables below are the ones to reach for.

One at a time. Do not run the dashboard tile and an SSH session together — both grab the microphone and speaker, and you get doubled audio.