The Practical Guide to Recurrent-Depth Transformer (No Fluff)

If you’ve been tracking the latest shifts in LLM architecture, you’ve likely noticed that the industry is moving away from simply stacking more layers. The real frontier isn't just parameter count; it’s how we handle compute-adaptive reasoning. The OpenMythos project provides a fascinating look at what many suspect is the engine behind Claude Mythos: the Recurrent-Depth Transformer (RDT).

Most models are static. You feed in a prompt, it passes through a fixed number of layers, and you get an output. If the problem requires more "thought," you’re stuck with whatever depth you trained for. RDTs change this by recycling a subset of layers. Instead of a massive, static stack, you have a Prelude, a looped Recurrent Block, and a Coda. You run the recurrent block $T$ times, effectively deepening the model’s reasoning at inference time without adding a single parameter.

Here’s where most people get tripped up: they assume this is just another form of Chain-of-Thought (CoT). It isn't. CoT forces the model to output intermediate tokens, which is slow and consumes context window space. In an RDT, the reasoning happens silently in continuous latent space. Each loop iteration is functionally equivalent to a step of CoT, but because it’s latent, the model can explore multiple reasoning paths simultaneously before converging on an answer.

The biggest hurdle in building these systems is stability. If you’ve ever tried to train a recurrent transformer, you’ve seen the loss spikes. The hidden state $h_t$ tends to explode because the spectral radius of the injection parameters often exceeds one. If $\rho(A) \geq 1$, your model is essentially a runaway train.

To fix this, you have to treat the recurrence as a discrete linear time-invariant system. By parameterizing $A$ as a continuous negative diagonal matrix and enforcing stability through discretization—specifically using ZOH or Euler schemes—you guarantee that $\rho(A) < 1$. This isn't just a theoretical safeguard; it’s the difference between a model that converges and one that diverges into garbage after a few thousand steps.

Why does this matter for your AI development workflow? Because it decouples reasoning depth from model size. You can train a relatively small model and, at inference time, choose how much "compute" to spend on a problem by adjusting the loop count. If you’re solving a simple classification task, run one loop. If you’re tackling complex, multi-step planning, run ten.

This architecture allows for systematic generalization that vanilla transformers simply can't touch. While standard models struggle with OOD (out-of-distribution) compositions, looped models exhibit a phase-transition-like "grokking" process. They move from memorization to in-distribution handling, and finally to systematic generalization, where they can solve novel problems they’ve never seen during training.

If you want to experiment with this, the OpenMythos implementation is the best place to start. It’s a clean, first-principles reconstruction that lets you toggle between MLA and GQA attention types while playing with the loop iterations.

How does this change your approach to model scaling? If you’re still betting everything on parameter count, you’re missing the shift toward compute-adaptive, depth-variable reasoning. Try implementing a basic recurrent block in your next project and observe how the spectral radius impacts your training stability.