Cheap speech is a systems problem

A technical paper on inference, caching, codecs, and the path to cheap speech.

The wrong unit

Per-character pricing hides idle hardware, scheduling jitter, and slow first audio. A system can look inexpensive on a rate card while wasting accelerator time or making users wait through silence.

We optimize complete spoken turns per accelerator-hour while holding perceptual quality and tail latency constant.

Three useful levers

Length-aware batching keeps short interactive turns from waiting behind audiobook paragraphs. Speculative acoustic decoding reduces sequential work while conservative acceptance protects audible quality. A codec chosen for decoder speed cuts the cost paid by every stream.

The gain appears when scheduling, model execution, transport, and playback are measured as one path.

What we measure

Every evaluation pairs listening scores with p50 and p95 time-to-first-audio, real-time factor, peak memory, failed-stream rate, and energy per synthesized minute.

The goal is simple: speech should become cheaper because the system wastes less, not because the voice becomes less human.