one year on
A GPT in 60 Lines of NumPy
A single-file implementation clarifies the decoder-only transformer architecture using only NumPy, offering an educational window into how GPT models work under the hood.
Jay Mody’s blog post ‘GPT in 60 Lines of NumPy’ is on Hacker News today, with 1563 points and 146 comments. The post implements the forward pass of a GPT-2 model using only NumPy, loading pretrained weights from OpenAI to demonstrate text generation.
Mody’s guide walks through tokenization, embedding layers, multi-head causal self-attention, and the decoder-only transformer stack. While the implementation is deliberately minimal—no batching, GPU support, KV cache, or training loop—the author shows how swapping in JAX can make the code automatically differentiable and GPU-compatible.
On Hacker News, the post was widely praised as an educational resource. Some commenters noted the assumption of familiarity with neural networks, but the overall reception was positive, with many appreciating the clarity of the explanation for those looking to understand GPT internals.
Called it 'an absolutely fantastic introduction to GPT models' and said, 'I think the clearest I've seen anywhere, at least for the first section that talks about generating text and sampling.'
Called the code 'Nothing left to add, nothing left to take away: perfection.'
Said, 'This looks like something Peter Norvig would write, and that's about the highest compliment I can give.'
One year later — open only if you can handle spoilers
The post remains a widely referenced educational resource. The linked JAX approach for automatic differentiation presaged the trend toward JAX-based training libraries, though PyTorch continued to dominate in practice.
The Weekly Replay · free by email
This week, one year ago — every Sunday.
One email each Sunday: the week's replayed AI news, with the one-year-later annotations included. Written like it's breaking — dated like it isn't.
Free · double opt-in · unsubscribe anytime · privacy