FELA · 1.5B
Solving the AI Compute Crisis
Setup
FELA vs The Transformer vs ENDLESS MOBY DICK
Pros
Traditional LLMs will never keep up with our efficient architectures.
Next Steps
FELA needs more training (read: we need more funding) to scale up.
A standard LLM has to keep tabs on every word it has ever seen. More words (tokens) means more memory needed - more toys for hyperscalers to buy and cause shortages for the rest of us.
We designed FELA - the Fourier Encoder with Linear Attention to replace the expensive parts of an LLM with smarter ones. FELA doesn't care, the first and last page reads use the same memory.
Yes - we are debuting a brand new, high efficiency, massively scalable AI architecture that can solve the AI compute crisis. Where a conventional LLM models language as a problem solved with graphics
computing - we model language sequence problems with the same math that powers an audio digital signal processor. Our language models have constant memory usage and subquadratic worst case compute complexity.
What? ~62 MiB of RAM? HOW?
More or less, it's the layers. We have 28 layers:
- 18 FNO layers hold a 512 tap buffer (~54.00 MiB)
- 6 Linear Attention layers hold a matrix valued state (~5 MiB)
- 4 Landmark Attention layers hold 64 summaries (~3 MiB)
None of those terms grow with the text context length!
Why the Transformer is bound to lose.
SmolLM2-135M is our "standard transformer" test model - its memory climbs at ~46 kB per token, which we measured. It balloons past FELA's RAM usage after just ~1400 tokens and keeps climbing until it runs out of memory and falls apart.
This tab gives it 128 MB, which it exhausts at ~2,900 tokens. Uncapped it would eat ~1.3 GiB of RAM - and if you've not heard there's a shortage! FELA's state never grows, so there is nothing to exhaust. We have a champion!
Demo Notes
This demo model is not live in browser like the others. It requires a non GPU, commodity server, because there is no browser sized FELA to download (yet!). SmolLM2 genuinely runs in your tab here. The two bars compare is the memory each model must carry per token of context. No matter where either model executes, one is flat and the other is... not! The right hand panel tries to show you the consequences of this tradeoff!
Full card on Hugging Face ↗