Groq achieves amazing inference speed by loading all the parameters it needs into Static RAM (SRAM) on LPUs. Each LPU can only hold a limited amount of SRAM and so models usually need lots (hundreds) of LPUs networked together.

This also means that LPUs don’t depend on access to expensive and scarce High Bandwidth Memory

Unlike GPUs with asynchronous execution and complex runtime schedulers, Groq's LPU is fully deterministic. The compiler—not runtime hardware logic—controls every aspect of execution. When you compile a model for Groq, the compiler generates a static, cycle-accurate execution plan that maps every operation and data movement to physical hardware resources ahead of time. This eliminates runtime decision-making entirely

When running inference:

1. Weights are pre-loaded into SRAM during initialization
2. Input tokens trigger the pre-compiled execution sequence
3. Data streams through the chip following the compiler's schedule
4. Each core performs its assigned operations (matrix multiplies, activations, etc.) in perfect synchronization
5. Results flow out deterministically with single-digit millisecond latency for large models

The trade-off is flexibility: models must be recompiled for specific hardware configurations, and the architecture is optimized for inference workloads with static compute graphs, not dynamic control flow or training.