Real-time billion-parameter LLM inference on FPGA.

Apex Compute demonstrates real-time, efficient billion-parameter LLM inference on FPGA hardware, significantly outperforming NVIDIA Jetson in both speed and energy efficiency. Running the Gemma 3 1B model via llama.cpp, our architecture achieves 15.6 tokens/sec at just 4.5 W, more than 2× faster and 3× more efficient than CUDA on Jetson Orin Nano.

Despite operating on slower memory, lower frequency, older process technology, and an FPGA—not an ASIC—Apex Compute consistently accelerates core transformer operations by at least an order of magnitude compared to CUDA. With a straightforward transition to a 12nm ASIC and LPDDR5 bandwidth, we expect more than 20× system speed-up and the possibility of a sub-1-watt chip capable of running billion-parameter models.

This capability lets robotics, drones, wearables, and smart-home devices run advanced AI models offline with minimal overhead, bringing AI power anywhere in the physical world.

Explore the implementation and sample Gemma 3 1B model on GitHub, or purchase a Kintex-7 board with two Apex Compute Unified Engines.

* Gemma 3 1B was tested with llama.cpp on NVIDIA Jetson with CUDA enabled. Because llama.cpp still performs some computation on the CPU, the CPU was limited to one thread. GPU-only performance is approximately 8 tokens/second, lower than the real-time demo above.

Apex Compute Architecture vs CUDA

Prototype hardware comparison

Potential 12nm ASIC speed-up

Our design is primarily memory-bandwidth-bound for weight matrix multiplication and compute-bound for SRAM-based operations. By transitioning to LPDDR5, we can realize a proportional performance increase directly in our architecture because there are no structural bottlenecks or external overheads in the hardware.

A standard LPDDR5 controller on a 12nm process node can achieve 6400 Mbps on a 128-bit interface, delivering approximately 104 GB/s of bandwidth—about 10× higher than our current FPGA-based system. With this improvement, the design enables roughly a 20× speed-up compared to NVIDIA Jetson platforms.

At this bandwidth, our compute engine only needs to operate at 800 MHz to fully utilize the 256-pair element-processing in the Apex Compute Unified Engine Unit. We already reach 333 MHz timing closure on a 16nm FPGA, so scaling to approximately 800 MHz on a 12nm ASIC is straightforward and expected. There is a high chance we can achieve a sub-1 W compute chip capable of running billion-parameter AI models efficiently.

More benchmark results

PyTorch is less efficient than optimized CUDA kernels for LLMs. We’ve included reference benchmarks below for comparison.

Apex Compute Architecture vs PyTorch CUDA