FELA · streaming CTC
Live On Device Captions / ASR
Setup
Does FELA do efficient audio too?
Pros
Yeah! Private and efficient in browser · ~43 MB one time download
Usage
Pick a clip and watch the model caption it live, one decode at a time!
How did FELA adapt to audio signals?
FELA's architecture is great at byte level modelling - due to its
high memory efficiency it can operate at much higher context lengths,
which is typically the barrier to byte level work. FELA ASR reads an 80 band log-mel
spectrogram, with a fixed size streaming state that does not grow with sequence input.
On its own the acoustic model has no notion of English
words, but typically ASR models are paired with small classical n-gram language models,
or even modern LLMs. FELA here was paired with a classical one.
Who could put this to use?
Anywhere a transcript needs to stay on device, private, and responsive: dictation, live captioning,
or any sensitive workflow where sending audio to a server is off the table. The
decode window below re runs on whatever audio has arrived so far, the same
loop a live microphone stream would use, so the behavior you see here is the
real streaming behavior, not a shortcut. Additionally, this model, due to its tiny memory and resource
footprint, would pair well with smart glasses, hearing aids, or other assistive technologies used in realtime that lack the ability to run more powerful language models in combination with ASR and CTC models
Why it's fast and light
The FNO mixer works in the frequency domain and the GLA layers carry a
fixed size recurrent state, so decoding a longer stream never grows memory
or restarts computation from scratch. Int8 quantization and WASM SIMD keep
the 35M parameter forward comfortably real time on a single CPU core. Hybridizing the smaller
35M parameter client side language model here with classical N-gram and lexicon beam search techniques dropped WER by ~12 percentage points - making for a powerhouse compact ASR model.
Full card on Hugging Face ↗