Looped language models

1. Introduction

Large language models (LLMs) are a class of machine learning models that can engage in natural language conversations. These models are trained on a vast corpus of data scraped from the internet, and the knowledge gained from this data is stored as numerical values in matrices. These matrices collectively constitute the layers of the LLM. Typically, GPUs are used to run these models. A simplified definition of a GPU divides it into two main components: compute units, where mathematical operations like matrix multiplication occur, and memory units, where the actual model is loaded.

Even though a model might fit within the GPU memory (for instance, the DGX Spark has 128 GB of unified memory, which should comfortably accommodate a 115 GB model), single-user decoding presents a challenge. In scenarios where only one person is interacting with the model, GPU utilization remains very low for classic LLMs due to slow memory bandwidth. In simpler terms, the transfer of model weights from the memory unit to the compute unit is so slow that the compute units often sit idle, having finished their previous tasks while waiting for the next weights to arrive. This inefficiency led us to explore architectures designed to solve this bottleneck.

2. How looped models work

Looped models repeatedly pass the running representation through the same set of layers multiple times. For example, consider a model that is effectively 36 layers deep. Instead of storing 36 unique transformer blocks, you might train a model with only 3 unique layers and apply them 12 times, yielding 36 effective layer applications. In essence, the model becomes more capable without substantially increasing parameter count. Compared to storing a full 36-layer stack, the size increase is nearly zero because the same learned block is reused repeatedly.

We read a few papers in this space, and what makes them compelling is that they are not merely a compression technique; they also provide a story about reasoning and inference-time compute. The paper Reasoning with Latent Thoughts: On the Power of Looped Transformers argues that many reasoning tasks require depth more than unique parameters, showing that a k-layer model looped L times can approach the performance of a kL-layer non-looped model on several reasoning benchmarks.

3. The systems perspective

What made looped language models especially interesting was not only the reasoning aspect, but also the systems aspect. We were running compute-bound experiments on DGX Spark, which is a heavily bandwidth-constrained and also cache-starved machine for single-user decoding.

When analysing whether autoregressive decoding (the process used by models to generate text one token at a time, where each new token is based on all the previous tokens generated) is compute-bound, an important quantity is arithmetic intensity (a metric that measures the ratio of computational work performed to the amount of data memory bandwidth accessed). For standard autoregressive decoding at batch size 1, arithmetic intensity is relatively weak because every generated token requires moving a large number of weights for comparatively little computation. As a result, decoding tends to be memory-bound unless weights can be reused efficiently or operations can be fused aggressively.

That is exactly where looped models become exciting. DGX Spark GB10 offers approximately 273 GB/s of LPDDR5x bandwidth and 24 MiB of shared L2 cache (a small but very fast memory storage area located directly on the processor or GPU, designed to hold data that the processor needs to access frequently, avoiding the slower process of fetching it from the main system memory), and these two numbers define much of the optimisation landscape. If the reused transformer block is sufficiently small to fit inside L2 cache, weights only need to be fetched from DRAM once, and subsequent loop iterations can be served directly from cache. This means additional loop depth adds computation and reasoning without adding proportional memory traffic.

But our goal was to be compute-bound at single-user decode, and that only made sense if we became L2-resident: even with a looped model, if the block doesn’t fit in L2, weights are fetched again and again and arithmetic intensity effectively collapses back to the standard case: you get the reasoning capability, but you are still not compute-bound.

4. Increasing arithmetic intensity through reuse

This is the key conceptual shift: in a conventional transformer, increasing depth generally means storing and streaming more unique parameters. In a looped model, increasing effective depth means reusing the same parameters repeatedly. Once the block is resident in cache, arithmetic intensity increases because more computation is performed per byte fetched.

5. Our looped GPT design

Our own model is built around this idea. It is a 12.1M parameter looped GPT consisting of 3 unique transformer layers applied 12 times, giving 36 effective layer applications per token.

Component	Value
d_model	512
Attention heads	8 query, 2 KV
Head dimension	64
SwiGLU hidden size	1280
Positional encoding	RoPE
Normalisation	RMSNorm
Embeddings	Tied token embeddings, tied LM head

The most important systems detail is that these three unique layers occupy roughly 15 MiB in bf16, allowing them to fit comfortably inside DGX Spark GB10’s 24 MiB L2 cache.

This shape was not accidental. It was deliberately chosen for a machine where DRAM bandwidth is limited but cache reuse can be exploited effectively. If the looped block remains resident in cache, later passes no longer incur the full off-chip weight-fetch cost.

6. Cache residency helps, but looping is not free

One particularly interesting observation from the measurements is that cache residency clearly works, but looping still incurs costs. The resident 15 MiB block contributes essentially zero additional DRAM traffic per extra loop, and rerunning a resident layer saves roughly 12 µs per layer compared to streaming it from DRAM. However, each loop still incurs approximately 22 µs per layer due to synchronisation barriers and compute overhead that cache residency alone cannot eliminate.

Looped models therefore should not be described as providing free depth. They provide cheap depth, provided the reused block remains cache-resident. Once weight traffic is sufficiently reduced, the bottleneck begins shifting toward synchronisation costs, occupancy limitations, and kernel organisation rather than raw memory bandwidth. For anyone trying to make batch-1 decoding compute-bound, this is actually useful, because it means the problem has transitioned from memory movement toward a regime where compute-side optimisations become increasingly important.

7. Implications for quantisation

This perspective also explains one of the more surprising findings from the project. Quantisation alone would not necessarily solve the throughput problem. In the looped regime, weights are already resident in L2 cache, meaning performance is no longer dominated by weight movement in the same way as conventional large-model decoding. Instead, barriers and computation become dominant costs. As a result, shrinking weights to int4 would not automatically produce dramatic throughput gains. This stands in contrast to the standard large-model decoding scenario, where quantisation frequently helps because bandwidth is the primary bottleneck.

8. Why looped models matter

Looped language models are exciting because they directly connect architectural design to inference hardware behaviour. They allow us to:

• trade unique parameter storage for recurrent depth,
• increase useful computation per byte fetched.

If meaningful models can be made L2-resident, we get maximum compute efficiency.

References

1. Universal Transformers: arxiv.org/abs/1807.03819
2. Reasoning with Latent Thoughts: On the Power of Looped Transformers: openreview.net/forum?id=din0lGfZFd
3. Loop, Think, & Generalise: Implicit Reasoning in Recurrent-Depth Transformers: arxiv.org/abs/2604.07822
4. LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning: huggingface.co/papers/2602.11451