A voice loop with captions

01 / What it was

The talking part of our April hackathon build, where you tap an everyday object and talk to it. You hold a button and speak. The words you said appear on screen as a caption, and then the answer comes back as a voice.

It was a small loop, but it was the first time we used speech instead of typing to talk to an AI, and it set a pattern we kept: always show what was heard.

02 / How it worked

• Push to talk: a mic button records while you hold it.
• Speech is turned into text on the phone, in the browser, by a small speech-recognition model that is cached after the first load.
• While the mic is on, the screen shows “listening”. As soon as the text is ready, a “you said” caption shows it, before the reply arrives.
• The reply is spoken in the character's own voice, and the audio starts playing before the whole file has arrived.
• A recording is capped at 60 seconds.

04 / Where it led

The same loop of speak, show what was heard, answer out loud came back in the holographic co-founder a month later.

Sources

1. Prototype source code, comment on the caption. hackathon prototype source, Apr 2026 “you said: …” toast that briefly echoes the Whisper transcript so the user has a clear signal their voice message landed.
2. Prototype source code, caption label. hackathon prototype source, Apr 2026 {isRecording && !heardText ? "listening" : "you said"}