Project page · video distillation

Learning the Autoregressive

Direct Score-Difference Estimation with Amortized Future Credit for Video Distillation

Leta fuses a directly-learned local correction with cross-chunk future credit into one teacher-guided update field, produced in a single estimator forward pass at generator update time.

Zixuan Duan1,2,† Yabo Chen2,† Xunzhi Xiang1,† Xin Zhang2,3 Haibin Huang2 Yin Chen2 Chi Zhang2,* Qi Fan1,*

1Nanjing University 2Institute of Artificial Intelligence, China Telecom (TeleAI) 3Fudan University

†Co-first authors   *Corresponding authors

View video results Paper coming soon

Video results

One prompt. Two update fields.

Twenty matched prompts from the shared evaluation set, each generated under the same seed, rollout procedure, and four-step sampling schedule. Leta sits beside the DMD baseline for a direct temporal comparison.

Leta ours DMD baseline

Videos play when scrolled into view · open a card for the full prompt and a larger view

Method

From a local gap to future-aware credit.

DMD reconstructs the full student score before subtracting it from the teacher's. Leta learns the gap itself, then lets the next chunk send credit back to the current one — both from a single shared estimator.

ConceptTwo signals, one update field
Left: estimating the full student score and subtracting it from the teacher score yields a larger gap error than predicting the gap directly. Right: conventional training ignores the future chunk's influence, while Leta adds a future-credit term to the local correction.
Direct gap prediction, plus future credit. Learning the teacher–student residual directly avoids reconstructing two full scores before subtracting them, which lowers the gap error at fixed capacity. On the right, detached-history training drops the current chunk's influence on later chunks; FutureCredit restores it as an additive term.
01 local correction

ScoreDelta

Rather than learning the student score and subtracting it from the teacher score, Leta regresses the teacher–student denoising residual directly. At the population optimum this recovers the same score difference DMD uses, up to a known timestep scaling; under a fixed-feature rank budget it can reach strictly lower approximation error when the residual is simpler to represent than the student denoiser.

update≈ ∇k

02 cross-chunk credit

FutureCredit

Detaching the generated history hides how the current chunk shapes everything after it, and backpropagating the full rollout costs too much memory. Leta replays the next chunk's final denoising step and takes a vector–Jacobian product against the next-chunk teacher residual, giving a learnable target that carries that influence backward.

update= ∇k + λ Ck

Training loopGenerator updates · estimator updates
Leta's two alternating phases. Left: the generator combines the frozen joint estimator's ScoreDelta correction and FutureCredit credit. Right: the estimator learns both fields from teacher residuals and next-chunk VJP targets.
One estimator pass per generator update. The two phases alternate: the shared bidirectional estimator learns both fields under complementary supervision from teacher residuals and next-chunk VJP targets, then the frozen estimator supplies the fused direction. Teacher evaluations and credit VJPs appear only in estimator training, so generator updates carry no extra teacher cost.
Toy studyMatched parameter budget
Three spiral score fields: the target, the result of estimating the student score and subtracting, and the result of predicting the difference directly, which stays closest to the target.
Shared structure cancels. On a fixed-density spiral with matched trainable parameters, predicting the score difference directly (c) preserves the target geometry (a) far better than estimating the student score and subtracting it from the teacher score (b).

Abstract

Few-step distillation for autoregressive (AR) video generation commonly first initializes an AR student and then refines it with DMD. This refinement process faces two challenges. First, although initialization already yields a reasonably capable student, DMD obtains the refinement direction indirectly by estimating the full student score before forming the teacher–student correction. Second, detached-history training prevents the generator from assigning credit to a current chunk for its influence on subsequent chunks.

We introduce Leta, a few-step distillation framework that addresses these challenges with ScoreDelta and FutureCredit. ScoreDelta directly estimates the teacher–student correction from denoising residuals, while FutureCredit learns an auxiliary cross-chunk update signal from vector–Jacobian products (VJPs) of next-chunk teacher residuals. Leta combines these signals into a joint teacher-guided update field in a single forward pass during generator updates. Teacher evaluations and credit VJPs are used only for estimator training.

For ScoreDelta, we establish population equivalence to score-difference estimation and a conditional approximation advantage under limited capacity in a fixed-feature linear model. In our controlled experiments, Leta improves over DMD by 0.9 points on VBench and 0.66 on VideoAlign. This advantage persists at a higher generator update frequency (G:E = 1:1), where Leta improves over DMD by 1.6 on VBench and 0.35 on VideoAlign.

Paper

Learning the Autoregressive

Direct Score-Difference Estimation with Amortized Future Credit for Video Distillation

Paper coming soon Back to top
BibTeX 
@article{duan2026leta,
  title   = {Learning the Autoregressive: Direct Score-Difference Estimation
             with Amortized Future Credit for Video Distillation},
  author  = {Duan, Zixuan and Chen, Yabo and Xiang, Xunzhi and Zhang, Xin and
             Huang, Haibin and Chen, Yin and Zhang, Chi and Fan, Qi},
  year    = {2026}
}