Diffusion Beats Autoregressive in Knowledge-Constrained Settings – Machine Studying Weblog | ML@CMU

TLDR:

If you’re compute-constrained, use autoregressive fashions; if you’re data-constrained, use diffusion fashions.

Motivation

Progress in AI over the previous decade has largely been pushed by scaling compute and information. The recipe from GPT-1 to GPT-5 has appeared simple: prepare a bigger mannequin on extra information, and the result’s a extra succesful system.

But a central query stays: will this recipe proceed to carry from GPT-6 to GPT-N?

Many analysts and researchers imagine the reply is not any. As an illustration, Ilya Sutskever, in his NeurIPS 2024 Take a look at-of-Time Award talk, remarked: “Compute is rising—higher algorithms, higher {hardware}, greater clusters—however information will not be rising. We’ve only one web, the fossil gas of AI.” 

This concern is echoed by AI forecasters, who’ve analyzed compute and information progress extra systematically and concluded that compute is outpacing information at an accelerating price.

The above Determine, illustrates this stress by overlaying projections from EpochAI’s analysis. Their research extrapolates historic tendencies in compute, dataset utilization, and internet-scale information availability. The forecast means that by round 2028, we are going to enter a data-constrained regime: much more compute shall be out there than there are coaching tokens to devour.

This paper addresses the problem by asking: how can we commerce off extra compute for much less information? Our central concept is to revisit the foundations of contemporary generative modeling and examine the 2 dominant paradigms for scaling AI.

Broadly, there have been two households of algorithms that formed latest progress in AI:

• Autoregressive fashions, popularized in 2019 within the textual content area with the GPT-2 paper.

• Diffusion fashions, popularized in 2020 within the imaginative and prescient area with the DDPM paper.

Each purpose to maximise the joint probability, however they differ essentially in how they factorize this joint distribution.

The success of diffusion in imaginative and prescient and autoregression in language has sparked each pleasure and confusion—particularly as every group has begun experimenting with the opposite’s paradigm.

For instance, the language group has explored diffusion on textual content:

D3PM launched discrete diffusion through random masking, whereas Diffusion-LM utilized steady diffusion by projecting tokens to embeddings earlier than including Gaussian noise. Since then, quite a few works have prolonged this line of analysis.

Conversely, the imaginative and prescient group has experimented with doing autoregressive modeling on pictures. Fashions equivalent to PARTI and DALLE exemplify this strategy with sturdy outcomes.

This cross-pollination has led to even better uncertainty in robotics, the place each diffusion-based and autoregressive approaches are broadly adopted. As an instance this, OpenAI Deep Analysis has compiled a listing of robotics works throughout each paradigms, highlighting the dearth of consensus within the subject.

This ambiguity raises a elementary query: ought to we be coaching diffusion fashions or autoregressive fashions?

Fast Background:

Autoregressive language fashions:

They mannequin information distribution in a left-to-right method

Diffusion language fashions:

For a extra detailed understanding, with cool animations, please seek advice from this video from Jia-Bin Huang – https://www.youtube.com/watch?v=8BTOoc0yDVA

Prior outcomes with Diffusion Language fashions

Since 2021, diffusion language fashions have sparked vital curiosity, with many works specializing in bettering their design and efficiency.

Within the desk above, we spotlight consultant outcomes from a well-liked work.
The takeaways are as follows:

• Discrete diffusion performs higher than steady diffusion on textual content.
• Autoregressive fashions nonetheless obtain the strongest outcomes general.

A number of works have additionally explored the scaling conduct of diffusion-based language fashions.

Nie et al report that discrete diffusion LLMs require roughly 16× extra compute than autoregressive LLMs to match the identical detrimental log-likelihood. Comparable outcomes have been noticed in multimodal domains—as an illustration, UniDisc finds that discrete diffusion wants about 12× extra compute than autoregression for comparable likelihoods.

Nevertheless, these outcomes conflate information and compute as a result of they’re measured in a single-epoch coaching regime. This raises an necessary ambiguity: do diffusion fashions actually require 16× extra compute, or do they in truth require 16× extra information?

On this work, we explicitly disentangle information and compute. Our objective is to review diffusion and autoregressive fashions particularly in data-constrained settings.

Our Motivation

To grasp why diffusion might behave in a different way, let’s revisit its coaching goal.

In diffusion coaching, tokens are randomly masked and the mannequin learns to recuperate them. Importantly, left-to-right masking is a particular case inside this framework.

Seen this fashion, diffusion will be interpreted as a type of implicit information augmentation for autoregressive coaching. As a substitute of solely studying from left-to-right sequences, the mannequin additionally advantages from many different masking methods.

And if diffusion is actually information augmentation, then its advantages needs to be most pronounced when coaching is data-bottlenecked.

This attitude explains why prior works have reported weaker outcomes for diffusion: they primarily evaluated in single-epoch settings, the place information is considerable. In distinction, our research focuses on eventualities the place information is restricted and compute will be traded off extra successfully.

Our Experiments

On this work, we prepare lots of of fashions spanning a number of orders of magnitude in mannequin measurement, information amount, and variety of coaching epochs to suit scaling legal guidelines for diffusion fashions within the data-constrained setting. We summarize a few of our key findings beneath.

Discovering #1:

Diffusion fashions outperform autoregressive fashions when educated with adequate compute (i.e., extra epochs & parameters). Throughout totally different distinctive information scales, we observe:

• At low compute, Autoregressive fashions win.

• After a certain quantity of compute, efficiency matches—we name this the important compute level.

• Past this, diffusion retains bettering, whereas Autoregressive plateaus or overfits.

Every level within the determine exhibits a mannequin educated to convergence. The x-axis exhibits the full coaching FLOPs of that time, and the y-axis exhibits the perfect validation loss achieved by that mannequin household below that coaching compute price range.

Discovering #2:

Autoregressive fashions start to overfit a lot rapidly, whereas diffusion exhibits no indicators of overfitting even after 10x the variety of epochs. Within the above determine, we confirmed that growing compute finally favors diffusion. However compute will be scaled in two methods: (i) Rising mannequin measurement (ii) Rising the variety of epochs Within the following plot, we separate these axes.

The coloured star marks the 1-epoch level, the place Autoregressive outperforms diffusion. The star (★) denotes the perfect loss achieved by every mannequin.

• Autoregressive hits its finest across the center, then overfits.

• Diffusion retains bettering and reaches its finest loss on the far proper.

Not solely does diffusion profit from extra coaching—it additionally achieves a greater closing loss than Autoregressive (3.51 vs. 3.71).

Discovering #3:

Diffusion fashions are considerably extra strong to information repetition than autoregressive (AR) fashions.

We present coaching curves of fashions educated with the identical complete compute, however totally different trade-offs between distinctive information and variety of epochs.

An “epoch” right here means reusing a smaller subset of information extra occasions(e.g., 4 Ep is 4 epochs whereas utilizing 25% distinctive information, 2 Ep is 2 epochs with 50% and so forth).

• AR fashions start to overfit as repetition will increase—their validation loss worsens and considerably diverges at larger epoch counts.

• Diffusion fashions stay steady throughout all repetition ranges, displaying no indicators of overfitting or diverging—even at 100 epochs.

Discovering #4:

Diffusion fashions exhibit a a lot larger half-life of information reuse (R_D*) —i.e., the variety of epochs after which returns from repeating information begins to considerably diminish.

We undertake the data-constrained scaling framework launched by Muennighoff et al. of their wonderful NeurIPS paper to suit scaling legal guidelines for diffusion fashions. Whereas Muennighoff et al. discovered R_D* ~ 15 for autoregressive fashions, we discover a considerably larger worth of R_D* ~ 500 for diffusion fashions—highlighting their means to learn from much more information repetition.

The above Determine research the Decay price of information worth below repetition: left exhibits diffusion, center AR, and proper the typical decay price for each.

Factors are empirical outcomes (darker coloration = larger FLOPs, lighter coloration =
decrease FLOPs; every line = fastened compute), we discover that fitted curves (represented as strains) intently match the empirical factors, indicating our scaling legal guidelines are consultant. The decay price of worth for repeated information is decrease for diffusion, reflecting its better robustness to repeating. On this experiment 100% information fraction means coaching 1 epoch with 100% distinctive information, whereas 50% means 2 epoch epoch with solely utilizing 50% distinctive information and so forth.

Discovering #5:

Muennighoff et al. confirmed that repeating the dataset as much as 4 epochs is sort of as efficient as utilizing recent information for autoregressive fashions.

In distinction, we discover that diffusion fashions will be educated on repeated information for as much as 100 epochs, whereas having repeated information nearly as efficient as recent information.

Discovering #6:

The compute required for diffusion to outperform AR follows a predictable energy legislation. Above we outlined the important compute threshold as the quantity of FLOPs the place diffusion matches AR efficiency for a given distinctive dataset measurement.

We discover that we are able to derive a easy closed-form analytical expression for this threshold, this enables us to foretell when diffusion will surpass AR given any distinctive information measurement. Within the determine we present each the fitted curve and empirical important threshold factors, which align intently.

Discovering #7:

The information effectivity of diffusion fashions interprets to raised downstream efficiency.

Lastly we consider the best-performing diffusion and AR fashions (educated below the identical information price range) on a spread of language understanding duties.

Throughout most benchmarks, diffusion fashions outperform AR fashions, confirming that diffusion’s decrease validation loss interprets to raised downstream efficiency.

Discovering #8:

Publicity to totally different token orderings helps clarify diffusion’s information effectivity. By including express information augmentations to AR coaching, we discover that diffusion mannequin’s benefit arises from their publicity to a various set of token orderings.

As seen within the above Determine, growing N constantly lowered validation loss and delayed overfitting. At N = 16, the 100-epoch validation lack of AR fashions approached that of diffusion, suggesting that numerous orderings are certainly a key driver of diffusion’s information effectivity. These outcomes assist our interpretation that diffusion fashions outperform AR fashions in low-data regimes as a result of they’re implicitly educated on a richer distribution of conditional prediction duties.

Lastly, this evaluation suggests a pure continuum between the 2 paradigms: by controlling job range via masking or reordering—we may design hybrid fashions that interpolate between compute effectivity (AR-like) and information effectivity (diffusion-like).

For extra experiments and particulars please seek advice from authentic paper –https://arxiv.org/abs/2507.15857

Conclusion

As the supply of high-quality information plateaus, bettering information effectivity turns into important for scaling deep studying. On this work, we present that masked diffusion fashions constantly outperform autoregressive (AR) fashions in data-constrained regimes — when coaching entails repeated passes over a restricted dataset. We set up new scaling legal guidelines for diffusion fashions, revealing their means to extract worth from repeated information far past what AR fashions can obtain.

These outcomes problem the traditional perception that AR fashions are universally superior and spotlight diffusion fashions as a compelling various when information—not compute—is the first bottleneck. Trying forward, environment friendly use of finite information might outline the subsequent frontier in scaling deep studying fashions. Though the research have been carried out within the context of language fashions, we imagine these findings ought to apply throughout any form of sequence modeling information, equivalent to in robotics or healthcare. For practitioners, our takeaway is easy: if you’re compute-constrained, use autoregressive fashions; if you’re data-constrained, use diffusion fashions.

Bibtex:

@article{prabhudesai2025diffusion,
title={Diffusion Beats Autoregressive in Knowledge-Constrained Settings},
creator={Prabhudesai, Mihir and Wu, Mengning and Zadeh, Amir and Fragkiadaki, Katerina and Pathak, Deepak},
journal={arXiv preprint arXiv:2507.15857},
yr={2025}
}