Research

Grokking: What happens when AI suddenly 'gets it'

Small arithmetic models can generalize long after memorizing their training data. Grokking shows why training accuracy alone is not evidence of readiness.

Avala Research විසින්

කියවීමට විනාඩි 9
0
බෙදාගන්න

A model can memorize its training examples long before it succeeds on new ones. In Power and colleagues' grokking experiments, some small networks eventually generalized after an extended period of overfitting. The surprising result was the delay, not proof that every stalled training run will recover.

The name comes from Robert Heinlein's Stranger in a Strange Land. In machine learning, it describes delayed generalization, not a test for human-like understanding. Follow-up work made one small modular-addition network unusually interpretable: researchers could examine the algorithm forming inside it while its headline performance appeared flat.

For teams building Physical AI, the useful distinction is between fitting known examples and demonstrating useful behavior on held-out conditions. Training accuracy alone cannot establish readiness. Interpretability can help explain a result; it does not replace evaluation.

The puzzle: perfect training, terrible testing

The researchers were studying modular arithmetic, which you already do every time you read a clock. If it's 11 AM and you have a 3-hour meeting, it ends at 2 PM: 11 + 3 = 14, which wraps past 12 to leave 2. That is modular arithmetic with a modulus of 12.

In the follow-up study by Nanda and colleagues, a small transformer learned modular addition with a modulus of 113. Given inputs like "17 + 45 =", the model should output "62" (since 17 + 45 = 62, which is less than 113). Given "100 + 50 =", it should output "37" (since 150 mod 113 = 37). Some examples were held out for testing.

Early in training, the model reached 100% accuracy on the training data. That is not a valid stopping or deployment criterion: held-out accuracy was still around chance. It had memorized the training examples without learning modular addition at all.

The researchers continued training beyond this apparent plateau. For thousands more steps, both curves stayed flat, training accuracy at 100%, test accuracy near 0%. Then test accuracy shot up, quickly reaching nearly 100%. The model could now solve problems it had never seen.

What was happening during those flat steps, and what triggered the transition?

The network invented Fourier analysis

In their ICLR 2023 study, Neel Nanda and collaborators reverse-engineered a Fourier-based algorithm in small modular-addition transformers. This explains their studied models, not every instance of grokking.

To solve modular addition, the network uses Fourier analysis and trigonometric identities. Picture a clock face: adding hours is rotating around a circle. 3 o'clock plus 4 hours means rotating from the 3 position (90 degrees) by 4 hours (120 degrees) to reach 7 o'clock (210 degrees). Modular arithmetic with modulus 113 works the same way, on a circle divided into 113 segments.

With no instruction to do so, the network learns to represent inputs as points on circles using sines and cosines, the mathematical language of rotation. For "17 + 45 =", it computes the sines and cosines of 17 and 45 at specific frequencies, then applies the identity cos(x)·cos(y) - sin(x)·sin(y) = cos(x+y) to compute the sum. It was never taught trigonometry. It was given no hint that circles or waves might be relevant. Under the study's training setup, it learned a Fourier-based solution without an explicit trigonometry lesson.

The authors analyzed both activations and weights, then tested their explanation by removing Fourier components. Those interventions matter: a wave-like visualization can suggest a mechanism, but showing what changes when part of the computation is removed provides stronger evidence. The explanation is a model of the studied computation, not a claim that every individual neuron implements a pure trigonometric function.

Three phases behind the apparent plateau

With tools to probe the network's internals, Nanda's team could watch learning unfold in detail. Their main training run has three phases. The epoch boundaries below describe that run; they are not a universal training schedule.

Phase 1: memorization (epochs 0 to 1,400). The model rapidly learns to answer every training example, but its internals show none of the Fourier structure of the final solution. It is building a high-dimensional lookup table from each input to its output. This reaches 100% training accuracy quickly, while held-out results still show that generalization is missing.

Phase 2: circuit formation (epochs 1,400 to 9,400). Headline performance remains nearly flat: training accuracy is high while held-out accuracy remains low. Inside the network, the sine and cosine patterns are slowly building up in the weights. The circuit that will implement modular addition is under construction while the network still answers from memory. To detect this hidden progress, the researchers built a new metric called "excluded loss" that removes the key Fourier frequencies from the logits and measures training loss on what remains. Its rise during this phase reveals increasing reliance on the Fourier solution while ordinary train and test losses remain nearly flat.

Phase 3: cleanup (epochs 9,400 to 14,000). Once the Fourier circuit is fully formed, weight decay (a regularization technique that penalizes large weights) favors shedding the memorization solution. The authors also associate weight decay with the earlier circuit-formation phase. The lookup table dissolves; the more efficient circuit remains. This is when test accuracy spikes. What looks like sudden insight is the unveiling of a solution that had been building for thousands of steps.

Stopping early would have missed the later generalization, but the held-out results already warned against deployment. The additional metrics reveal progress toward a solution, not readiness that the test set somehow failed to detect.

The same geometry shows up in Claude

Grokking in modular arithmetic might look like a curiosity about small networks on toy tasks. The implications reach the frontier models we use every day.

The arithmetic study raises a useful question for larger models: could gradual changes in internal computation precede an apparently abrupt capability? Answering that requires evidence from those models, not extrapolation from a small transformer.

Anthropic's line-breaking study offers a parallel. Studying how Claude 3.5 Haiku decides when to insert line breaks while writing, researchers found the model represents character count on a six-dimensional manifold, a geometric structure remarkably similar to the circles found in grokking. The model tracks its position on the current line with a helical representation in activation space. Multiple attention heads contribute information across different ranges of remaining characters; their combined output resolves ambiguities that a single head leaves. A geometric mechanism called the "QK twist" rotates these representations to compare current position against target line width.

This is a parallel in representation geometry, not evidence that Claude learned line breaking through the same grokking dynamics. A circuit study and a training-history study answer different questions.

Understanding is possible, but still rare

The modular-addition study connects a behavioral result to a concrete algorithm and its development during training. That is a useful example of interpretability at a deliberately manageable scale.

Explaining a particular counting circuit in a larger model does not explain everything that model can do. The scope of an explanation must stay attached to the behavior and experiment it addresses.

That gap matters for safety. Incomplete explanations limit what we can infer about unfamiliar conditions. Behavioral tests and monitoring can still detect failures without a complete mechanistic account. For physical AI systems (autonomous vehicles, robots, medical devices), those failures have real-world consequences. Grokking's contribution is proof of possibility: networks are not random soups of weights, they learn structured solutions that make sense when properly analyzed. The challenge is developing interpretability techniques that scale to models orders of magnitude larger.

What grokking says about training data

For those of us building AI infrastructure, grokking offers practical guidance.

Training dynamics and training data both matter. Two models trained on identical data with slightly different hyperparameters can land in very different places: one memorizing, one generalizing. The conditions that promote genuine learning over memorization should inform how training pipelines are structured.

Dataset size and regularization affected the arithmetic experiments. That does not establish a universal relationship between label quality and the time a robotics model takes to generalize.

Our practical inference is to test the data decisions directly. For a robot-learning program, compare models under a fixed evaluation protocol, separate held-out environments from training examples, and inspect failures rather than relying on aggregate training accuracy. Additional sensors, more diverse scenes and longer sequences are candidate interventions whose value depends on the task. Grokking does not prove that adding LiDAR, radar or temporal context improves a particular policy.

The alien in the network

These findings should make us cautious about anthropomorphizing AI systems. In Animals vs Ghosts, Andrej Karpathy contrasts language-model training with the processes that produce animal intelligence. That is a useful metaphor for the difference, not an experimental explanation of grokking.

The network that learned modular addition reinvented Fourier analysis, but it did not learn our way of doing arithmetic. Most humans would solve 17 + 45 mod 113 by computing 62 directly, then checking whether it exceeds 113. The network converts inputs to wave representations, multiplies sines and cosines, uses trigonometric identities, and recovers the answer from the resulting pattern. Both approaches work. The network's is mathematically beautiful, computationally efficient, and utterly alien to human intuition about arithmetic.

We communicate with AI systems in human language, but beneath that interface they process information in ways we are only beginning to map. We cannot assume they reason as we do, fail as we would, or generalize in ways we'd expect.

Grokking, understanding so complete that you merge with the subject, is ironically named. We grok how this small network grokked arithmetic. For frontier models, we do not yet, and until we do, humility is not a virtue but a requirement for anyone putting these systems to work in the physical world.


This piece draws heavily on Welch Labs' excellent video "Grokking: The Most Understood AI Model," which we recommend watching for its visualizations of these concepts.

Power and colleagues documented delayed generalization on small algorithmic datasets in January 2022 (paper). The anecdote about an accidentally extended training run is not needed to establish the result.

Neel Nanda's mechanistic analysis with collaborators appeared at ICLR 2023 (paper). For more on mechanistic interpretability, Neel's getting started guide is the best entry point. Quanta Magazine also has an excellent accessible overview: How Do Machines 'Grok' Data?

The Claude Haiku character-counting research by Emmanuel Ameisen, Wes Gurnee and team at Anthropic is available at transformer-circuits.pub, and it's in-depth and worth reading in full.

For a robotics program, start with the intended output and the conditions that would demonstrate generalization. Explore robotics workflows or book a working session to discuss the data and evaluation requirements.

  • Mechanistic Interpretability
  • AI Safety
  • Physical AI
  • Machine Learning
  • Founder Insights

ප්‍රවෘත්ති කාමරය

නිර්දේශිත ලිපි

මාතෘකා වලින් තෝරාගත්: Mechanistic Interpretability, AI Safety, Physical AI, Machine Learning, Founder Insights

සියලු ප්‍රවෘත්ති බලන්න