The second installment of the zero-barrier DDPM series. Here we derive the forward (noising) process in closed form — how to jump from the original image x0x_0 to any timestep xtx_t in a single shot, instead of crawling there one noisy step at a time.

The "experienced driver" framing

There's a nice mental model for diffusion models — think of it as an experienced driver (老司机) theory. To master a diffusion model you need to nail down three things:

  1. The route (路线) — the noising trajectory.
  2. The car (车) — what's actually moving along it.
  3. The driver (司机) — who's steering.

We'll derive DDPM from all three angles across the series. This post tackles the route — the noising process. The next two will cover the car and the driver.

The per-step formula the paper just hands you

The paper, somewhat arbitrarily (you could say willfully), defines a single-step noising formula:

xt=1−βt xt−1+βt εt,εt∼N(0,I).x_t = \sqrt{1 - \beta_t}\, x_{t-1} + \sqrt{\beta_t}\, \varepsilon_t, \qquad \varepsilon_t \sim \mathcal{N}(\mathbf{0}, \mathbf{I}).

A few notes:

  • This is defined, not derived — it's a reasonable, human-chosen construction.
  • βt\beta_t increases with time. Larger βt\beta_t ⇒ more noise injected per step ⇒ the noising amplitude grows as tt grows. Intuitive enough.

Why this formula isn't good enough

The problem: starting from x0x_0, this formula only lets you reach x1x_1. To get x2x_2 you apply it again; to reach xTx_T you apply it TT times. That's a step-by-step crawl — far too slow.

An experienced driver doesn't drive one step and look around, drive one more, look around again. We want to stand at the start (x0x_0) and jump to wherever we like — straight to x2x_2, or xt−1x_{t-1}, or any xtx_t — in one move. So we need to push this formula further into a closed form.

This derivation is, in my opinion, the easiest of the route/car/driver trio: about 90% elementary math, with exactly one trick.

Setup: two definitions

To simplify the algebra, define:

αt:=1−βt,\alpha_t := 1 - \beta_t,

and the cumulative product:

αˉt:=αt αt−1⋯α1=∏s=1tαs.\bar\alpha_t := \alpha_t\, \alpha_{t-1} \cdots \alpha_1 = \prod_{s=1}^{t} \alpha_s.

These are purely for notational convenience — the route designer chose them to make the math cleaner. With αt=1−βt\alpha_t = 1 - \beta_t, note the per-step formula becomes:

xt=αt xt−1+βt εt.x_t = \sqrt{\alpha_t}\, x_{t-1} + \sqrt{\beta_t}\, \varepsilon_t.

The trick: nesting dolls (套娃)

We want to express xtx_t in terms of x0x_0, not xt−1x_{t-1}. The whole derivation is captured by one idea: nesting dolls (套娃 / matryoshka) — repeatedly substitute the formula into itself until x0x_0 appears.

Since indices above 3 get annoying to write, let's work the example x3x_3 (the general TT case follows identically). Start:

x3=1−β3 x2+β3 ε3=α3 x2+β3 ε3.x_3 = \sqrt{1 - \beta_3}\, x_2 + \sqrt{\beta_3}\, \varepsilon_3 = \sqrt{\alpha_3}\, x_2 + \sqrt{\beta_3}\, \varepsilon_3.

Substitute x2x_2 (first nesting):

x2=α2 x1+β2 ε2,x_2 = \sqrt{\alpha_2}\, x_1 + \sqrt{\beta_2}\, \varepsilon_2,

⇒x3=α3(α2 x1+β2 ε2)+β3 ε3=α3α2 x1+α3β2 ε2+β3 ε3.\Rightarrow\quad x_3 = \sqrt{\alpha_3}\big(\sqrt{\alpha_2}\, x_1 + \sqrt{\beta_2}\, \varepsilon_2\big) + \sqrt{\beta_3}\, \varepsilon_3 = \sqrt{\alpha_3 \alpha_2}\, x_1 + \sqrt{\alpha_3 \beta_2}\, \varepsilon_2 + \sqrt{\beta_3}\, \varepsilon_3.

Substitute x1x_1 (second nesting — and x0x_0 finally appears 🎉):

x1=α1 x0+β1 ε1,x_1 = \sqrt{\alpha_1}\, x_0 + \sqrt{\beta_1}\, \varepsilon_1,

⇒x3=α3α2α1 x0⏟①+α3α2β1 ε1+α3β2 ε2+β3 ε3⏟②.\Rightarrow\quad x_3 = \underbrace{\sqrt{\alpha_3 \alpha_2 \alpha_1}\, x_0}_{\textstyle ①} + \underbrace{\sqrt{\alpha_3 \alpha_2 \beta_1}\, \varepsilon_1 + \sqrt{\alpha_3 \beta_2}\, \varepsilon_2 + \sqrt{\beta_3}\, \varepsilon_3}_{\textstyle ②}.

Notice the pattern: the leading coefficients are products of α\alpha's, and the trailing terms are a sum of independent noise terms. Call the first term ① and the noise sum ②.

Collapsing term ①

By the definition of the cumulative product:

①=α3α2α1 x0=αˉ3 x0.① = \sqrt{\alpha_3 \alpha_2 \alpha_1}\, x_0 = \sqrt{\bar\alpha_3}\, x_0.

Clean. The mess is all in ②.

Collapsing term ②: a sum of Gaussians

Each εi∼N(0,I)\varepsilon_i \sim \mathcal{N}(\mathbf{0}, \mathbf{I}). We need two facts from Part 1's prerequisites:

Scaling a Gaussian. If ε∼N(0,I)\varepsilon \sim \mathcal{N}(\mathbf{0}, \mathbf{I}) and we scale by aa:

a ε∼N(0, a2I).a\, \varepsilon \sim \mathcal{N}(\mathbf{0},\ a^2 \mathbf{I}).

Summing independent Gaussians. For two scaled terms:

a1ε1+a2ε2∼N(0, (a12+a22) I).a_1 \varepsilon_1 + a_2 \varepsilon_2 \sim \mathcal{N}\big(\mathbf{0},\ (a_1^2 + a_2^2)\,\mathbf{I}\big).

The means are all zero, so the combined distribution depends only on the sum of the squared coefficients. The squared coefficients in ② are:

α3α2β1⏟ε1  +  α3β2⏟ε2  +  β3⏟ε3.\underbrace{\alpha_3 \alpha_2 \beta_1}_{\varepsilon_1} \;+\; \underbrace{\alpha_3 \beta_2}_{\varepsilon_2} \;+\; \underbrace{\beta_3}_{\varepsilon_3}.

Call this sum △\triangle. So term ② is distributed as N(0, △ I)\mathcal{N}(\mathbf{0},\ \triangle\, \mathbf{I}).

The one trick: add an auxiliary term

Summing △\triangle directly is awkward. The trick is to add a helper term α3α2α1=αˉ3\alpha_3 \alpha_2 \alpha_1 = \bar\alpha_3 and compute △+αˉ3\triangle + \bar\alpha_3 instead — because the relation αt=1−βt\alpha_t = 1 - \beta_t, i.e. αt+βt=1\alpha_t + \beta_t = 1, makes everything telescope.

Compute △+αˉ3\triangle + \bar\alpha_3 by repeatedly factoring out common terms:

△+αˉ3=(α3α2β1+α3β2+β3)+α3α2α1.\triangle + \bar\alpha_3 = \big(\alpha_3 \alpha_2 \beta_1 + \alpha_3 \beta_2 + \beta_3\big) + \alpha_3 \alpha_2 \alpha_1.

Group the α3α2\alpha_3 \alpha_2 terms (the auxiliary term was added precisely to complete β1+α1\beta_1 + \alpha_1):

=α3α2(β1+α1)⏟= 1+α3β2+β3=α3α2+α3β2+β3.= \alpha_3 \alpha_2 \underbrace{(\beta_1 + \alpha_1)}_{=\,1} + \alpha_3 \beta_2 + \beta_3 = \alpha_3 \alpha_2 + \alpha_3 \beta_2 + \beta_3.

Factor α3\alpha_3 from the first two:

=α3(α2+β2)⏟= 1+β3=α3+β3=1.= \alpha_3 \underbrace{(\alpha_2 + \beta_2)}_{=\,1} + \beta_3 = \alpha_3 + \beta_3 = 1.

So:

△+αˉ3=1⟹△=1−αˉ3.\triangle + \bar\alpha_3 = 1 \quad\Longrightarrow\quad \boxed{\triangle = 1 - \bar\alpha_3.}

No squaring needed — the coefficients were already the squared values (they sit under the square roots, and variance uses their squares directly).

Term ② in closed form

Therefore term ② follows N(0, (1−αˉ3) I)\mathcal{N}(\mathbf{0},\ (1 - \bar\alpha_3)\,\mathbf{I}), and via the reparameterization trick we can write it as a single noise variable:

②=1−αˉ3 ε3,ε3∼N(0,I).② = \sqrt{1 - \bar\alpha_3}\, \varepsilon_3, \qquad \varepsilon_3 \sim \mathcal{N}(\mathbf{0}, \mathbf{I}).

(We collapse three separate noise draws into one equivalent standard-normal draw.)

Putting it together

Adding ① + ②:

x3=αˉ3 x0+1−αˉ3 ε3.x_3 = \sqrt{\bar\alpha_3}\, x_0 + \sqrt{1 - \bar\alpha_3}\, \varepsilon_3.

This is the route: from the starting point, jump anywhere you want in one step. Standing at x0x_0, "eyes on all six roads, ears on all eight directions," we can pin down the whole noised state directly.

Generalizing 3 → T

Why is it legitimate to use just three terms? Because TT is a finite value. If you had the patience you could write x4,x5,…,x1000x_4, x_5, \dots, x_{1000} — the pattern is identical (同理可得). So we simply swap the 33 for tt:

 q(xt∣x0)=N(xt; αˉt x0, (1−αˉt) I)⟺xt=αˉt x0+1−αˉt ε,  ε∼N(0,I). \boxed{\,q(x_t \mid x_0) = \mathcal{N}\big(x_t;\ \sqrt{\bar\alpha_t}\, x_0,\ (1 - \bar\alpha_t)\,\mathbf{I}\big) \quad\Longleftrightarrow\quad x_t = \sqrt{\bar\alpha_t}\, x_0 + \sqrt{1 - \bar\alpha_t}\, \varepsilon,\ \ \varepsilon \sim \mathcal{N}(\mathbf{0}, \mathbf{I}).\,}

This is exactly the closed-form forward sampling formula in the paper — and the one you actually use in code.

Summary

  • DDPM defines an arbitrary one-step noising formula xt=1−βt xt−1+βt εtx_t = \sqrt{1-\beta_t}\,x_{t-1} + \sqrt{\beta_t}\,\varepsilon_t, but stepping one timestep at a time is too slow.
  • We want to jump from x0x_0 to any xtx_t directly. The method is nesting dolls (套娃): substitute the formula into itself, peeling xt→xt−1→⋯→x0x_t \to x_{t-1} \to \cdots \to x_0.
  • The result splits into a clean mean term αˉt x0\sqrt{\bar\alpha_t}\,x_0 and a sum of independent Gaussians.
  • Using the Gaussian sum property, that noise sum has mean 0\mathbf{0} and variance equal to the sum of squared coefficients △\triangle.
  • The one trick: add the auxiliary term αˉt\bar\alpha_t, use αt+βt=1\alpha_t + \beta_t = 1 to telescope, and find △+αˉt=1\triangle + \bar\alpha_t = 1, hence △=1−αˉt\triangle = 1 - \bar\alpha_t.
  • We demonstrated with three terms because TT is finite; swapping 3→t3 \to t gives the final closed-form: xt=αˉt x0+1−αˉt ε.x_t = \sqrt{\bar\alpha_t}\, x_0 + \sqrt{1 - \bar\alpha_t}\, \varepsilon.

That completes the route derivation. Next up: the car.

Notes taken from a video tutorial on DDPM (Part 2 of the series).

References

  • Ho, J., Jain, A., & Abbeel, P. (2020). Denoising Diffusion Probabilistic Models. NeurIPS 2020. arXiv:2006.11239