NESformer: What It Means to Compile a 6502 CPU into Qwen3.5 Weights

Plenty of projects have tried to recreate an NES emulator with a neural network. NESformer's claim is different: it does not learn to imitate game frames. It compiles the NES 6502 CPU, 2 KB of RAM and a cartridge ROM into the weights and state of a Qwen3.5 checkpoint.

Drew Breunig's NESformer says the console runs inside a standard Qwen3_5ForCausalLM checkpoint. "Nobody trained these weights; a compiler computed every one," the README states. The model predicts the CPU's execution log one token at a time. A small host program draws the picture, reads the controller and fires the vblank interrupt.

What is inside the model, and what stays outside

The README splits the work clearly. The model contains the 6502 core's 151 official opcodes, addressing modes, flags, stack, BRK and NMI entry, 2 KB of work RAM, and 32 KB of PRG-ROM. The host contains the PPU renderer, palette, controllers, APU register writes and vblank timing.

That split matters. The deterministic CPU execution is in the model; graphics and I/O remain outside. NESformer is therefore better described as "a CPU inside the LLM generates an execution trace, and a host interprets that trace like a console" than as "an LLM plays a game."

Using one row of DeltaNet state as a memory cell

The central trick comes from Qwenputer. Each Gated DeltaNet head keeps a 128×128 state matrix updated by the delta rule at every token. Hold the gate open and make the key one-hot, and an update overwrites exactly one row. A one-hot query reads that row back. One row becomes one memory cell.

The NumPy example below is CodeBridge's educational model of that addressing scheme. It is not the Qwen3.5 or NESformer kernel.

import numpy as np

state = np.zeros((4, 3))              # 4 rows, 3 values per row
key = np.array([0.0, 1.0, 0.0, 0.0])  # select row 1
value = np.array([7.0, 8.0, 9.0])

state += np.outer(key, value)         # write row 1 only
print("read row 1:", (key @ state).tolist())
print("row 0 unchanged:", state[0].tolist())

new_value = np.array([1.0, 2.0, 3.0])
state += np.outer(key, new_value - (key @ state))  # overwrite row 1
print("read row 1 again:", (key @ state).tolist())
print("row 2 unchanged:", state[2].tolist())

Actual local output:

read row 1: [7.0, 8.0, 9.0]
row 0 unchanged: [0.0, 0.0, 0.0]
read row 1 again: [1.0, 2.0, 3.0]
row 2 unchanged: [0.0, 0.0, 0.0]

The example addresses one row, overwrites it and reads it back. NESformer extends the idea across layers and heads, and nesformer.json records which layer and head holds each part of the machine. The real address space and update rules live in core.py and weights.py; this article does not reimplement them.

Verification compares tokens, not plausible-looking frames

The README reports a layered check. A reference CPU matches nestest's golden log for 3,348 instructions and matches py65 on 5.7 million instructions of game code. The exact circuit predicts every reference token in Qwenputer's evaluator, and the checkpoint in stock transformers predicts 2,651 positions without a mismatch. Free-running traces for nestest, Thwaite and Full Game are reported as identical to the reference CPU.

CodeBridge cloned the repository, unpacked the weights and ran the check command. That command runs the model and the reference CPU side by side, stopping at the first difference. It is a teacher-forced check, not a free-running gameplay run.

$ python -m nesformer check roms/thwaite.nes --tokens 12000
OK: the weights generated 12,000 tokens identical to the reference CPU's trace
    (3 frames, frames identical: True; 4,385 tok/s)

On this machine, 12,000 tokens and three frames matched the reference. This is not a reproduction of the README's full runs (Thwaite at 1,916,527 tokens and 600 frames), so it does not establish that an entire game is identical. It does confirm that the checkpoint loads locally and behaves deterministically over an initial execution window.

Speed: 700,000 tokens per second versus 1,650

The README says a real NES would need about 700,000 tokens per second. The default numba engine ran at about 1,650 tokens/s on one CPU core of the build machine. The raw ratio is:

700,000 / 1,650 ≈ 424.2

By raw throughput alone, that is roughly 424 times slower than the console. Games spend much of their time in idle loops waiting for vblank, however, and NESformer detects an exactly repeating loop with no RAM or I/O writes and skips ahead to the next PPU event. That reduces the token count by 2.6× to 4.2×. Dividing 424 by those factors gives 101–163×, not the README's overall 150–290× figure. The ranges do not reconcile from the published numbers alone. Hardware or measurement conditions may differ, so the README's 150–290× remains a separate author-reported measurement rather than a derived result.

Our local check reached 4,385 tokens/s after warm-up, but that was a short teacher-forced path on different hardware with different cache conditions. It should not be generalized into a product speed. The point is not real-time emulation; it is deterministic execution inside a model.

The limits define the research

The repository lists its boundaries:

• NROM cartridges only (iNES mapper 0). Games that switch banks cannot follow the change because ROM reads happen from a fixed 32 KB table inside the model.
• Official opcodes only. An unofficial opcode emits HALT.
• Code must run from cartridge ROM. A jump into RAM emits HALT.
• No APU frame IRQ or DMC IRQ. BRK and NMI work.
• The PPU and APU are not in the weights. Rendering is scanline-based, mid-scanline effects are not reproduced, sprite-0 hit and overflow are approximate, and there is no sound.

This list is the antidote to "an LLM emulates the NES" summaries. The result is possible because the deterministic CPU core was separated from the peripherals. It is not one model that learned the whole console.

What remains for developers

NESformer is an architecture experiment, not a performance benchmark. Using recurrent state as addressable memory and embedding a compiled circuit in a language-model checkpoint makes an important distinction visible: a model that computes deterministically is not the same as a model that happens to produce a plausible answer. One has a checkable execution path; the other is probabilistic generation.

The practical takeaway is narrow. Most products should use a real CPU or deterministic code, which is cheaper and faster. The broader idea, treating internal model state as external memory, is relevant to long-lived agent state, tool traces and verifiable intermediate computation. Even then, do not merge "the model got it right" and "the circuit guarantees it" into one success metric.

For more on reading evaluation claims, see our guide to AI benchmarks. NESformer's code and weights are MIT-licensed; the test ROMs keep their own licenses.