Local speech-to-text in the browser — Whisper without a server, reviewed

Speech recognition used to be a thing you rented. You sent audio to a cloud endpoint, paid per minute, and waited. The interesting shift of the last two years is that a capable model now fits inside a browser tab and never sends the audio anywhere. This is a review of how well that actually works — what is usable today, and where a server still earns its keep.

The short version: for short clips, dictation, and voice commands, local transcription in the browser is genuinely good. For an hour of multilingual meeting audio on a mid-range laptop, it is slow and sometimes painful. The gap is narrowing, but it is not closed.

What Whisper is, and why it travels well

Whisper is OpenAI’s open-weight speech model, released in 2022. It is a Transformer sequence-to-sequence model trained on a large, messy corpus of audio paired with transcripts, which is where the “weak supervision” in the paper’s title comes from. The training breadth is the reason it tolerates accents, background noise, and code-switching better than most of what came before it.

It ships in a ladder of sizes, and the size you pick is the single most important decision you will make:

• tiny — 39M parameters, ~1 GB VRAM, roughly 10x faster than large
• base — 74M parameters, ~1 GB VRAM, ~7x faster
• small — 244M parameters, ~2 GB VRAM, ~4x faster
• medium — 769M parameters, ~5 GB VRAM, ~2x faster
• large — 1550M parameters, ~10 GB VRAM, the accuracy baseline

Those figures are from OpenAI’s own model card. Each size below large has an English-only .en variant, and the documentation notes those tend to do better on English — the effect is most pronounced on tiny.en and base.en, where the multilingual head is mostly dead weight if you only ever transcribe English.

The relevant point for browser work is that the small end of this ladder is small. A quantized tiny or base model is a download on the order of tens of megabytes, which is a thing a web page can plausibly fetch and cache.

Path one: whisper.cpp compiled to WASM

The most direct route into the browser is whisper.cpp, Georgi Gerganov’s C/C++ reimplementation of the Whisper inference code. It has no Python and no framework dependency, which makes it a clean target for Emscripten. The project ships a whisper.wasm example that runs the whole model as WebAssembly inside the page.

The behaviour matches the privacy claim exactly: you load a GGML model file, pick an audio file or record from the microphone, and the audio is processed on your machine. Nothing is uploaded. There is a separate stream.wasm example that does near-real-time transcription from the microphone.

Two caveats are worth stating plainly, because the project states them itself. The first is that your browser must support WASM SIMD instructions — without them, it will not run. The second is the ceiling: the WASM example is documented as capable of running models “up to size small inclusive,” and beyond that the memory requirements and performance are unsatisfactory. So the WASM path is, in practice, a tiny/base/small path. That is fine for a lot of uses and a hard wall for others.

WASM here means CPU. It is portable and it works almost everywhere, but it does not touch the GPU, and that is the performance story in one sentence.

Path two: transformers.js with WebGPU

The second route is Hugging Face’s transformers.js, which in its v3 release added WebGPU as a backend. WebGPU is the browser standard for general-purpose GPU compute — the successor to WebGL — and it lets the model run on the machine’s graphics hardware instead of the CPU.

The API is almost aggressively simple. You build an automatic-speech-recognition pipeline pointed at a model such as onnx-community/whisper-tiny.en and pass { device: "webgpu" }. The same call shape handles quantization through a dtype option, so you can ask for a 4-bit (q4) or half-precision (fp16) build to shrink the download and the memory footprint.

On the speedup, I want to be careful. The v3 announcement is titled around WebGPU being “up to 100x faster than WASM,” and that headline number does the rounds, but the post itself gives no per-task benchmark for Whisper, and “up to” is carrying real weight in that sentence. The honest framing is qualitative: on a machine with a decent GPU and working WebGPU, the transformers.js Whisper pipeline is markedly faster than the CPU/WASM path, enough to make base and small comfortable where they previously dragged. I would not quote a multiplier I cannot reproduce.

The cost of this path is reach. WebGPU support was around 70% of global users as of late 2024 per caniuse, and on Firefox, Safari, and older Chromium it has at various points needed a feature flag. Any serious deployment needs a WASM fallback for the machines where WebGPU is missing or disabled, which means you end up shipping both paths anyway.

Path three: the lighter Moonshine models

The newer option is Moonshine, from Useful Sensors, built specifically for live transcription and voice commands on constrained hardware. It comes in two sizes — tiny at 27.1M parameters and base at 61.5M — both smaller than the Whisper models they compete with (whisper-tiny.en is 37.8M, base.en 72.6M).

The claim that earned it attention, from the paper and the model card, is a roughly 5x reduction in compute for a 10-second clip versus Whisper tiny.en, with no increase in word error rate on standard evaluation sets. The architectural reason is that Moonshine processes audio proportional to its length rather than padding everything to a fixed 30-second window the way Whisper does, so short utterances do not pay for silence they do not contain. That design choice is exactly why it suits voice commands and live captioning, where inputs are short and latency is the whole game.

The trade-off is scope. Moonshine’s flagship models are English-first and aimed at short-form audio. If you need broad multilingual coverage or robust handling of long recordings, this is not the tool, and Whisper remains the better-rounded choice.

Streaming versus batch

It is worth separating two things people lump together. Batch transcription takes a finished file and returns text; this is the comfortable case, and the only real variable is how long you wait. Streaming transcription consumes the microphone live and emits text as you speak, which is harder, because the model keeps revising its guess as more audio arrives and you have to decide when a word is final.

Both whisper.cpp (stream.wasm) and the transformers.js stack can do streaming, and Moonshine is built for it. But streaming on the CPU path with a larger model is where the local approach feels its limits most — the transcript lags the speaker, and the lag compounds. For live work, a small model on a GPU, or a purpose-built model like Moonshine, is the realistic combination.

The privacy upside, stated without theatre

The genuine advantage of all three paths is the one that does not require any benchmark: the audio never leaves the device. There is no upload, no third-party endpoint logging your voice, no retention policy to read. For medical notes, legal interviews, anything covered by a confidentiality obligation, or simply a person who would rather not narrate their day to a vendor, that is the entire argument, and it is a strong one. A local model that is merely good enough can beat a cloud model you are not allowed to use.

The verdict

Run the numbers honestly and the picture is clear. For short audio, dictation, voice commands, and captioning on a reasonably modern machine, local browser speech-to-text is ready. A quantized base Whisper model over WebGPU, or Moonshine for live work, gives you transcripts that are good and private, with a first-load model download as the main tax. I have been pleasantly surprised by how little ceremony the transformers.js pipeline now demands — it is two function calls and a model name.

Where the server still wins is the heavy end. Hour-long recordings, broad multilingual coverage, the accuracy of large, or guaranteed performance on hardware you do not control — these still favour a machine with a real GPU sitting somewhere you manage. The browser caps out around small on WASM and leans on WebGPU availability above that, and not every visitor brings a GPU to the page.

So the answer to “Whisper without a server?” is yes, with a clear boundary. Pick the small models, decide between WASM reach and WebGPU speed, and reach for Moonshine when the task is short and live. Ask for large-grade accuracy on long multilingual audio across arbitrary devices, and you are describing a server. The remarkable thing is how much now sits comfortably on the near side of that line.