Speech-to-text todos in under a second with Groq Whisper and an LLM

You press a button, say “call mom, pay rent Friday and gym tomorrow at 7”, and three todos appear. Getting that to feel instant turned out to be less about a faster model and more about where the waiting happens.

This is how we rebuilt voice capture in Crek, a free habit tracker and to-do list, around Groq's Whisper API. The same shape works for any app that turns speech into structured data: notes, calendar events, expenses, form filling.

The slow version: one call that did everything

The first version sent the recording to Gemini in a single request. Gemini accepts audio, so one call could transcribe the sentence and pull the todos out of it: a title, the details, a due date resolved against the user's timezone.

It worked, and it was simple. It also had three problems:

• The user waited for all of it. Listening, writing down the transcript and extracting the todos all happened before anything appeared on screen.
• The model typed the transcript back. We showed the transcript so people could spot a misheard word, which meant the model had to write the whole sentence out before it started on the todos. Output tokens are the slow part of any LLM reply.
• A misheard word meant starting over. If it heard “call Tom” instead of “call mom”, the only fix was to record the whole thing again, which was another full call.

So we split the job in two, with the user in the middle.

Step 1: transcribe with Groq Whisper

Speech-to-text is a solved problem with fast, cheap models. Groq's speech-to-text API runs OpenAI's Whisper on its own hardware, and it is quick: in our tests a 3 to 5 second clip came back as text in 0.3 to 0.9 seconds, including the upload.

Which Whisper model

Groq offers two:

We use turbo. It costs about a third as much, and the one thing large-v3 adds that matters is translation, which we don't need. The accuracy gap matters less than it looks, because in the new flow the user reads every transcript before anything is made from it.

Two details worth knowing before you pick either:

• Every request is billed as at least 10 seconds. A two-second todo costs the same as a ten-second one, about $0.0001 on turbo.
• Set the language if you know it. We pass language: "en". Detection is least reliable on exactly the short clips a voice input produces.

The request

It is a multipart upload with the file and a few fields. We call it with plain fetch rather than an SDK, because the app runs on Cloudflare Workers and a hand-written request has nothing Node-specific to trip over.

const body = new FormData();
body.append("file", new File([audio], "recording.wav", { type: "audio/wav" }));
body.append("model", "whisper-large-v3-turbo");
body.append("language", "en");
body.append("temperature", "0");

const response = await fetch("https://api.groq.com/openai/v1/audio/transcriptions", {
  method: "POST",
  headers: { Authorization: `Bearer ${process.env.GROQ_API_KEY}` },
  body,
});
const { text } = await response.json();

The browser never calls Groq itself. It posts the recording to our own route, and the server adds the key. A key shipped to the browser is a key anyone can read in the network tab.

On the recording side, browsers disagree about formats: Chrome records WebM, Safari MP4. We decode whatever the browser produced and re-encode it as 16 kHz mono WAV before uploading. That is one format from every browser, and 16 kHz mono is what Whisper converts everything to anyway. Groq accepts WebM and Ogg directly too, so sending the compressed original is a smaller upload if you don't need one format everywhere.

Whisper says “Thank you.” when nobody spoke

We sent three seconds of silence to see what came back, expecting an empty string. We got:

Thank you.

Room noise came back as ".". This is a known Whisper habit. It learned from audio paired with subtitles, and the quiet parts of that data are full of captions like “Thank you.” and “Thanks for watching!”. Hand it silence and it writes a caption.

The usual advice is to check each segment's no_speech_prob and drop the ones that are probably silence. On Groq that didn't help us: no_speech_prob was 0 for pure silence and for pure noise alike. So we reject transcripts that are only one of those phrases:

const SILENCE = new Set([
  "", "you", "bye", "thanks", "thank you",
  "thank you very much", "thanks for watching", "thank you for watching",
]);

function isSilence(transcript: string) {
  const words = transcript.toLowerCase().replace(/[^a-z' ]/g, "").trim();
  return SILENCE.has(words);
}

It matches the whole transcript, never a part of it: “thank you” inside a real sentence is speech. A silent recording now fails in under a second with “Didn't catch anything”, before an LLM ever sees it.

Step 2: show the words before anything is made of them

The transcript goes on screen as editable text under the word Heard. If Whisper wrote “Tom” where you said “mom”, you fix one word by typing it. You don't record the sentence again.

That screen also changes what accuracy means. With one call, a misheard word quietly became a wrong todo. With a confirm step, it's a typo you can see and fix before anything is created.

The obvious objection is that it adds a step, and the step adds a wait. It adds a step, but it doesn't have to add a wait.

Step 3: start the LLM call before the user confirms

The drafting takes Gemini about 2.6 seconds. Reading a sentence on the Heard screen takes a person a few seconds too. So the draft request goes out the moment the transcript arrives, not when the user presses Continue, and the two happen at the same time.

When Continue is pressed, there are two cases:

// When the transcript arrives: start drafting, don't wait for it.
let pending = requestDraft(transcript);
showHeard(transcript);

// When the user presses Continue:
if (pending.transcript !== edited) {
  pending.controller.abort();   // they changed it: this answer is for other words
  pending = requestDraft(edited);
}
const drafts = await pending.outcome;

If the text is untouched, Continue collects the answer that's already there. In our test run that took 4 ms from the key press to three drafts on screen. If the text was edited, the early request is dropped and a new one is sent with the corrected words, and the user waits the normal couple of seconds.

A few things make this safe rather than just fast:

• A late answer can't land on the wrong screen. Every exit (Escape, closing the overlay, recording again) aborts the request in flight. After waiting, the code also checks that the answer is still for the current request before showing it.
• A failed early answer isn't reused. If Gemini is busy, “Try again” goes back to the Heard screen with the words kept and sends a fresh request. It doesn't hand back the same failure.
• Aborting doesn't save money. Abort stops the browser waiting. The provider has already started work, and it still bills the call.

What it costs

The trade is two LLM calls instead of one whenever someone edits the transcript. The early one is wasted. With a text-only prompt that is a small cost for removing the whole wait from the common case. We log whether each transcript was edited, which tells us how often we pay it and whether the confirm step is worth its tap.

Step 4: send the LLM text, and don't make it type the transcript back

Gemini now gets the confirmed transcript as a plain text part after the prompt, and nothing else. Two small changes came with that:

• No audio tokens in. Text is a far smaller input than a recording.
• No transcript out. The old schema asked for the transcript alongside the todos, so the model spent output tokens retyping its own input. The new schema asks only for the todos, and the server puts the transcript back on each draft from the request.

The reply is still treated as untrusted input. Each draft goes through the same validator as a todo typed by hand before it can be saved, so whatever the model invents, or whatever a tampered client sends, gets the same checks.

Keeping older app versions working

Installed copies of our mobile app still post the recording straight to the draft endpoint and expect todos back. Rather than break them, the endpoint checks what it was sent. JSON means a confirmed transcript from the new flow. Multipart means a recording from an older build, so it transcribes it with Groq first and then drafts. Those builds keep working and get faster, just without the confirm step, and the audio branch comes out once nobody runs them.

What we measured, and what's left

On our dev server, with a sentence of a few seconds:

• Transcription: 0.3 to 0.9 s with whisper-large-v3-turbo on Groq.
• Drafting: about 2.6 s with Gemini on text, hidden behind the reading time.
• Continue to drafts: 4 ms when the transcript was kept as heard.

What we haven't done yet, written down so it doesn't get forgotten:

• No timeout on the Groq or Gemini calls. A hung upstream holds the request open.
• No retry when Gemini answers 503 “high demand”, which happened often enough in testing to notice.
• English only. The language is fixed to en. Supporting more means passing the user's language, or letting Whisper detect it and accepting the misses on short clips.

The part that surprised us most wasn't the model at all. Most of the wait was never removed; it was moved to where the user was already busy reading.