Taalas-Style On-Chip Weights on a $250 FPGA: a Language Model at 60k tok/s

I was so impressed by the chatjimmy.ai demo by Taalas, I wanted to see what I could squeeze inside the The FPGA's reconfigurable logic. Same thing as 'PL'. Where the whole model runs here. of a $250 Field-Programmable Gate Array: a chip full of reconfigurable logic you wire into a custom digital circuit, instead of running software on a fixed CPU.. By not using the (4 GB available) The off-chip DRAM (the board's main memory), ~20 GB/s, shared by both the CPU and the fabric. This shared controller is 'the bandwidth wall'., and keeping everything in UltraRAM: the big, wide on-chip SRAM (~18 Mb, 64 blocks). Holds the resident INT4 weight image. Crucially, it is true dual-ported, which enables the two-cohort 'split-brain'./Block RAM: small, flexible on-chip SRAM blocks (~5 Mb total). Holds activations, scratch, and the KV cache here., I set myself a target of 100,000 tok/s.

The headline, bit-exact and measured on silicon: 59,965 tokens per second on the fabric. The same model on this board’s own Arm cores manages 11 tok/s. My laptop’s RTX 3050 Ti manages 719.

Taalas managed to bake Llama 3.1 8B into their chip, and have a usable chat. I had ~3 MB of memory to work with, which means the most powerful model we can fit is a A synthetic corpus of about 2.1 million very simple children's stories, designed so that even tiny models can learn coherent English from it. The training data here.-class one: 3.16M parameters, ~1.5 MB at INT4. Initially, I wanted to shrink it further by Reduce each word to its dictionary root ('saying' to 'say', 'words' to 'word'). The Keviniser lemmatises the corpus down to content words, about 30% fewer characters for the same story. the training corpus and making it sound like Kevin Malone, ‘why use many word when few do trick’, but since this didn’t actually make the model any smaller, it just speeds it up by ~30%, I went back on myself and just allowed the hardware to flip-flop between both models.

First, talk to it.

The live demo

This widget is a real A persistent two-way connection between browser and server, unlike ordinary HTTP's request-then-hang-up. What lets the chat stream characters live. connection to the board. Your words go through a Cloudflare tunnel, to a serving box, to the Kria, into the fabric, and back. If the status dot is green, you are talking to a single FPGA in Wales. It’s a story generator, not an assistant. It doesn’t understand questions. Give it “once upon a time” and it finishes it.

kevin on kriaconnecting...

few word do trick. type something, kevin reply from inside the chip.

I hope you didn’t get your hopes up, there’s only so much you can do with ~1.5 MB of weights.

Why I even had this board

This is all running on my AMD KV260 dev board, featuring the K26 System-on-Module: the plug-in compute module that the KV260 board is built around.. This is one of the latest SoCs in the Zynq line (specifically the Zynq UltraScale+ ZU5CG/ZU5EV, catchy eh?), the successor to the OG 7000s. God, how I wanted a dev board for one of those 10 years ago.

I bought the KV260 for a different side project, a deterministic vision pipeline. The board is sold as a “vision AI starter kit,” but the AMD/Xilinx's application development stack for their boards. The KV260's 'vision AI' demos run on it, mostly on the Arm cores rather than the fabric. object detection runs on the quad core The quad-core Arm Cortex-A53 CPU on the KV260 (~1.33 GHz). The baseline the fabric is compared against, and the orchestrator when the CPU is in the loop.s, and the A53 is a weak core with no hardware matmul. The Vitis libraries turned out to be basically OpenCV on Linux rather than anything that pushes the fabric. If I wanted probabilistic AI running on a CPU, I sure as heck wouldn’t choose a quad core A53.

I did manage to get my deterministic depth engine running in Verilog on the fabric, but since it’s a robotics project, the hardware has a lot of downtime for safety reasons. So I put a fresh Linux image on a spare SD card and set up SSH so I can work on this from anywhere in what little free time I have (it very quickly spiralled out of control: another remote machine now has JTAG access to it, and the PSU is on a Zigbee relay so I can force reboots remotely).

The wall

One fact drives everything. Generating one token at a time is memory bound, not compute bound. To produce the next token you read every weight in the model once. The arithmetic is cheap, the reading is the cost.

On the KV260, the A53s and the fabric share one The off-chip DRAM (the board's main memory), ~20 GB/s, shared by both the CPU and the fabric. This shared controller is 'the bandwidth wall'. controller at roughly 20 GB/s. If the model lives in DDR, the fabric and the CPU drink through the same straw and the fabric buys you nothing. The round trip to DDR or CPU over The on-chip bus protocol family that connects the CPU to the fabric. kills you. The only escape is a model small enough that all of it fits in on-chip memory, where bandwidth is hundreds of GB/s. This is Taalas’ insight, and Groq’s, and Cerebras’: the memory wall is the enemy and on-chip weights are the escape. They spend hundreds of millions of dollars enlarging the on-chip budget. The KV260 gives you about 3 MB.

3 MB has to hold the weights, the activations, and the KV cache. That is not enough room for a smart model. It is barely enough room for a model that can string a sentence together. So the second lever, the one the big players mostly can’t pull: shrink the model until the problem disappears.

Making the model small enough (this is where Kevin comes in)

A 3.16M-parameter 4-bit integers (16 levels). The weights are stored as INT4, which is what makes the model small enough to fit on-chip. transformer fits in ~1.5 MB. That’s the budget met, but every byte still counts, and the last lever toward speed kept turning out to be “make the model dumber.” (or so I thought)

So the training corpus (TinyStories) is run through a tool that strips English to its content words. “Why waste time saying a lot of words when a few words do the trick” becomes “why waste time say lot word when few word do trick,” inspired by Kevin Malone’s communication philosophy from The Office, and yes, the model consequently talks like him. Across the full corpus the compression takes 371.7M words down to 260.5M, about 70%, measured.

I expected the compressed-corpus model to come out smaller. It came out exactly the same size, and in hindsight that’s obvious: the parameter count is fixed by the architecture, not the corpus. What the compression does is shrink the output distribution. The same story takes ~30% fewer characters to tell, so effective speed goes up even though the per-token rate is identical.

Honest version: the lemmatised corpus buys about 1.5x. The order-of-magnitude win is on-chip versus DDR. Kevin is the garnish on the chilli, not the meal.

Chasing 100k

My goal was 100,000 tok/s. I didn’t get there, but I gave it a bloody good go, eventually even compromising the usability by reducing the attention window to T=1 to remove as many round trips and race conditions as possible.

The record build is 16 parallel streams sharing one weight pass, entirely sequenced in the fabric, CPU touching nothing. It measures 59,965.5 tok/s at 200 MHz, 16 of 16 streams bit-exact against the integer reference, three runs of three. The confession: those sixteen streams remember nothing. Each decodes with an attention window of one token. A chat built on it emits one faithful character and falls down the stairs, “he he he he he.” It is fast and it is meaningless, and those are the same property taken one step too far.

So the deployed chat is a different, honest build. One stream, full trained context window, KV caching bit-exact to a full recompute. That one runs 19,242 tok/s of fabric by counted cycles, and ~21,300 tok/s measured live, a number that held flat from 1 to 2,000 concurrent connections in a load sweep with zero errors (peak observed 21,479). It remembers your last couple of turns, and it’s the one in the widget above. Neither number gets to borrow the record’s headline.

The path from 11 tok/s to 60k was a ladder, every rung measured on silicon:

the speed ladder, every rung named for what it removedtokens / second, log scale

MEASURED on silicon (3/3 runs, token stream bit-exact)SIM (RTL bit-exact vs reference; no bitstream)

A53 chat 11XPS15 torch CPU 356RTX 3050 Ti 719XPS15 ORT CPU 1,273

Act I: the CPU in the loop

Each rung deletes overhead, and the ladder asymptotes to the A53's own ~11 tok/s. Once the matmul is offloaded, the Arm core running the rest of the forward is the wall.

Act II: the sequencer (CPU out of the loop)

The architectural jump. One stream, the whole forward in fabric; from here the game is cycles and clock.

Act III: many streams (aggregate, T=1)

4 to 16 parallel streams share each weight pass, decoding with an attention window of 1. Real silicon, bit-exact, honestly degenerate text: these are aggregate tokens, not one conversation.

Act IV: Kevin remembers (faithful, N=1)

A different metric, not a regression: one stream with the full on-chip KV window, so every token attends to the whole conversation. These are tok/s that spell real messages.

And the ceiling is proven, not assumed. There’s a script that falsifies packing a third Multiply-ACcumulate: the one-multiply-one-add operation matrix multiplies are built from. The fundamental unit of compute here. into a DSP48E2: the FPGA's dedicated hardware multiplier blocks (1248 on this chip). Each can pack two INT4-by-INT8 multiply-accumulates. over 1.2M randomised products, and N=32 needs 2,048 DSPs the chip doesn’t have. The real limit of this architecture on this silicon is 62k to 78k. 100k was a finger-in-the-air guess before I even started, I just thought it would be a funny target number. Hitting 60k is close enough for me, and >20k usable beat the Taalas number, which was another goal I set myself.

Where it loses

The on-chip trick only wins while the model fits on-chip, and the crossover is around 6.3M parameters. Past that you spill to DDR and you’re back at the wall. Long context spills the KV cache the same way. This is a toy-model technique by construction, and the output is bad on purpose, which does not make it good. It’s a measurement instrument with a sense of humour.

For scale: Taalas etches weights into transistors on a taped-out ASIC per model. Cerebras keeps 44 GB of SRAM on a wafer. Groq keeps 230 MB per chip and gangs hundreds of them. Same one-sentence insight at budgets from nine figures down to the price of a nice dinner.

Go deeper

Everything above is the story. The engineering is in the sub-articles, and every number in them is bit-exact on silicon before it’s quoted:

• Inside the chip, the wide-word GEMV trick, the dual-port split-brain, the non-linear bricks, and the proof that 16 streams is a hard ceiling.
• Sampling without asking, how one unmeasured host loop ate 58% of every reply, and the Gumbel-max identity that collapsed 193 reads per token into one seed write.
• Serving an FPGA to strangers, the tunnel, the serving box, and why the dashboard deliberately lives off the board.
• War stories and the method, where iverilog lies, why silicon beats the timing report by 1.3 to 1.76x, and the optimisation I won and stopped anyway.

It’s all hand and LLM written Verilog, no High-Level Synthesis: tools that compile C/C++ into hardware so you don't write Verilog by hand. This project has none; every block is RTL written directly, by me and by Claude Code, never compiled from C.. Claude Code did a solid amount, it’s a side project on a side project after all, though pushing an FPGA to its limit is definitely not as comfortable for it as writing a CRUD app in TypeScript. I would probably put the balance at:

• Architecting: me 80%, Claude Code 20%
• Implementing: me 20%, Claude Code 80%

And despite saying it’s not that comfortable writing Hardware Description Language (Verilog, VHDL): code that describes circuits rather than instructions to execute. This project is Verilog throughout., it flipping did it. Outside of the stereo vision project, the last time I touched an FPGA was either a Lattice iCE40 about 10 years ago, or an even older Altera Cyclone IV on a DE0-Nano.

I haven’t investigated OSS synthesis and bitstream generators, but my workflow didn’t involve opening Xilinx's FPGA design suite: synthesis, place, route, and bitstream generation. once. It was all called via the CLI. On my 20-core i7 with 32 GB of RAM, a full run was around 30 minutes. Place and route slowed right down as I got closer to 100% memory usage, but it’s still amazing what can be done.

The receipt

59,965.5 tok/s at 200 MHz, measured, N=16, bit-exact, three runs of three. The deployed chat is the faithful build at 19,242 fabric tok/s by counted cycles, ~21,300 measured live under a 2,000-connection load test, live at chat.mikeayles.com and right here. Same thesis as the startup AMD just bought, on AMD’s own $250 board, reprogrammable in 25 seconds.

The code is now public: github.com/MichaelAyles/kev-gpt, the Keviniser, the model, the RTL, and the serving stack, including the bit-exact gates you can run on your laptop without a board.