one year on
Developer builds Llama 3 from scratch one matrix multiplication at a time
The HN submission about a GitHub repository walking through the entire Llama 3 implementation line by line draws 1041 points and 269 comments on Hacker News, sparking discussion on how accessible modern LLM architecture really is.
A developer known as naklecha has published a GitHub repository that implements Meta’s Llama 3 from scratch, “one tensor and matrix multiplication at a time.” The project loads weights directly from Meta’s official model file and walks through every step—tokenization, RMS normalization, attention with RoPE, and the full transformer forward pass—in a single Jupyter notebook.
The HN submission has drawn 1041 points and 269 comments. Some commenters compare it to earlier efforts like Karpathy’s llama2.c or GPT-2 in 60 lines of NumPy. One commenter noted that the code is simple enough for an individual to implement in a year, but the real barriers are hardware and data.
Others caution that the math can be opaque without a background in linear algebra and calculus. A commenter with a Calculus 3 background confessed they still got lost. The HN thread this week is animated by the tension between the accessibility of the code and the inaccessibility of the underlying concepts, as well as a side debate on whether such implementations shatter or reinforce the mystique of large language models.
Compared the project to llama2.c and noted that the code for LLMs is not that complicated, but the difficulty lies in access to hardware and data.
Suggested that studying the implementation might shatter the illusion of LLMs.
Asked whether a non-technical person should go through the tutorial or follow a different path.
One year later — open only if you can handle spoilers
The Naklecha repository became a steady reference for newcomers to transformer internals, often cited alongside Karpathy's llama2.c. A year later, the notion that 'the code isn't complicated' is repeated in nearly every educational thread, though the debate over whether that is an empowering or reductive framing continues.
The Weekly Replay · free by email
This week, one year ago — every Sunday.
One email each Sunday: the week's replayed AI news, with the one-year-later annotations included. Written like it's breaking — dated like it isn't.
Free · double opt-in · unsubscribe anytime · privacy