Part 2 of the DDIM series. In Part 1 we diagnosed the problem: DDPM's reverse process is shackled to a Markov chain, forcing 1000 one-step-at-a-time denoising steps. Now we break the shackle — and derive DDIM's reverse distribution and its famous sampling formula by the method of undetermined coefficients, all while keeping the ability to free-load (白嫖) a pretrained DDPM model.

Where exactly was Markov used?

Recall the DDPM reverse derivation. Bayes gave us (Formula 4):

q(xt−1∣xt,x0)=q(xt∣xt−1,x0) q(xt−1∣x0)q(xt∣x0).q(x_{t-1}\mid x_t, x_0) = \frac{q(x_t \mid x_{t-1}, x_0)\,q(x_{t-1}\mid x_0)}{q(x_t \mid x_0)}.

The first numerator term q(xt∣xt−1,x0)q(x_t \mid x_{t-1}, x_0) is unknown — until we invoke the Markov property: any xtx_t depends only on its immediate predecessor, so the distant x0x_0 drops out, q(xt∣xt−1,x0)=q(xt∣xt−1)q(x_t\mid x_{t-1}, x_0) = q(x_t\mid x_{t-1}). That single move turned the unknown into the known and let us simplify Formula 4 into the solvable Formula 5.

DDIM refuses to use the Markov property. So q(xt∣xt−1,x0)q(x_t\mid x_{t-1}, x_0) stays unknown, and Formula 4 can no longer be simplified to Formula 5. We've lost our derivation path. We need a different strategy.

Two requirements for free-loading

DDIM wants the acceleration "for free" — reuse a DDPM-trained model with minimal cost. Beyond dropping Markov, that imposes two requirements:

Requirement 1 — don't change q(xt∣x0)q(x_t \mid x_0)

Look back at the DDPM loss (Formula 8): the only distribution it actually uses is the marginal q(xt∣x0)q(x_t\mid x_0) (Formula 2) — not the joint q(x1:T∣x0)q(x_{1:T}\mid x_0). To free-load DDPM's trained model, the loss must obey a "gradient-equivalence everywhere" principle:

The one distribution DDPM's loss depends on — q(xt∣x0)q(x_t\mid x_0) — must not change at all because of DDIM's step-skipping.

"Equivalence" here is in the broad sense: a leading constant coefficient is fine. If DDIM's loss LσL_{\sigma} differs from DDPM's LγL_\gamma only by a constant multiplier aa (i.e. Lσ=a Lγ+bL_\sigma = a\,L_\gamma + b), the gradient direction is unchanged — you just rescale the learning rate by 1/a1/a and get identical optimization. A constant out front is harmless; a different functional form is fatal. So the distribution q(xt∣x0)q(x_t\mid x_0) is locked.

Requirement 2 — the reverse distribution must be simple (Gaussian)

Our entire goal is to go from xtx_t to xt−1x_{t-1}. For that we must be able to sample, so the reverse distribution q(xt−1∣xt,x0)q(x_{t-1}\mid x_t, x_0) has to be a simple distribution — ideally a Gaussian — which we can draw from via reparameterization.

Study tip: when learning diffusion models, think simple. A lot of things are simply assumed by the authors. Don't agonize over whether the Gaussian is "optimal" or whether some fancier form would be better. The goal is to generate images — the simplest, most direct choice is often a perfectly good one. Since dropping Markov left q(xt∣xt−1,x0)q(x_t\mid x_{t-1}, x_0) unknown, we can't derive the reverse distribution anymore, so we assume it's Gaussian and move on.

Setting up the ansatz (undetermined coefficients)

By analogy with DDPM, assume DDIM's reverse denoising distribution is also a DDPM-like Gaussian:

qσt(xt−1∣xt,x0)∼N(μ(x0,xt), σt2I).(9)q_{\sigma_t}(x_{t-1}\mid x_t, x_0) \sim \mathcal{N}\big(\mu(x_0, x_t),\ \sigma_t^2 \mathbf{I}\big). \tag{9}
  • Mean μ(x0,xt)=k x0+m xt\mu(x_0, x_t) = k\,x_0 + m\,x_t — a weighted combination of x0x_0 and xtx_t (the conditions), with k,mk, m as undetermined coefficients to solve for. (DDPM's mean was likewise a weighted sum of x0x_0 and xtx_t.)
  • Variance σt2≥0\sigma_t^2 \ge 0 is a human-specified, known constant — exactly as DDPM's reverse variance β~t\tilde\beta_t was a constant. We're free-loading all the way: copy the mean form and assume the variance is a known constant too.

Reparameterized sampling then gives:

xt−1=μ(x0,xt)+σt ϵt=k x0+m xt+σt ϵt.(10)x_{t-1} = \mu(x_0, x_t) + \sigma_t\,\epsilon_t = k\,x_0 + m\,x_t + \sigma_t\,\epsilon_t. \tag{10}

One equation, two unknowns (k,mk, m) — not solvable alone. We need another equation, then solve the system.

The second equation: Requirement 1 in action

By Formula 2 and Requirement 1, the marginal of xt−1x_{t-1} is locked to the DDPM form:

xt−1∼N(αt−1 x0, (1−αt−1)I),i.e.xt−1=αt−1 x0+1−αt−1 ϵt−1.(11)x_{t-1} \sim \mathcal{N}\big(\sqrt{\alpha_{t-1}}\,x_0,\ (1-\alpha_{t-1})\mathbf{I}\big), \quad\text{i.e.}\quad x_{t-1} = \sqrt{\alpha_{t-1}}\,x_0 + \sqrt{1-\alpha_{t-1}}\,\epsilon_{t-1}. \tag{11}

The two expressions (10) and (11) for xt−1x_{t-1} must be equal — otherwise Requirement 1 fails:

k x0+m xt+σt ϵt=αt−1 x0+1−αt−1 ϵt−1′.(12)k\,x_0 + m\,x_t + \sigma_t\,\epsilon_t = \sqrt{\alpha_{t-1}}\,x_0 + \sqrt{1-\alpha_{t-1}}\,\epsilon'_{t-1}. \tag{12}

Snag: the left side has xtx_t, the right side doesn't — we can't pin down mm. Fix it by expanding xtx_t with Formula 2 as well, xt=αt x0+1−αt ϵt′′x_t = \sqrt{\alpha_t}\,x_0 + \sqrt{1-\alpha_t}\,\epsilon''_t:

k x0+m(αt x0+1−αt ϵt′′)+σt ϵt=αt−1 x0+1−αt−1 ϵt−1′.(13)k\,x_0 + m\big(\sqrt{\alpha_t}\,x_0 + \sqrt{1-\alpha_t}\,\epsilon''_t\big) + \sigma_t\,\epsilon_t = \sqrt{\alpha_{t-1}}\,x_0 + \sqrt{1-\alpha_{t-1}}\,\epsilon'_{t-1}. \tag{13}

Now both sides are organized in terms of x0x_0 and noise.

⚠️ Why the primes (ϵt\epsilon_t, ϵt−1′\epsilon'_{t-1}, ϵt′′\epsilon''_t)? Each ϵ\epsilon is an independent standard normal. You cannot cancel them across the equals sign or treat them with ordinary algebra. The primes are a deliberate reminder.

Memorize this: random variables can only be handled by analyzing their distributions — never by elementary add/subtract/multiply/divide. (We hammered this in the SDE walkthrough.) Two independent ϵ\epsilon's don't "cancel"; if anything they combine into a new variance.

Solving by matching distributions

Since random-variable equations are solved via distributions, match the distribution on each side of (13). Prerequisite — the Gaussian variance-addition property:

a ϵ′+b ϵ′′∼N(0, (a2+b2)I).a\,\epsilon' + b\,\epsilon'' \sim \mathcal{N}\big(\mathbf{0},\ (a^2 + b^2)\mathbf{I}\big).

Left side of (13) is Gaussian with

mean=(k+mαt) x0,variance=(m2(1−αt)+σt2)I.\text{mean} = (k + m\sqrt{\alpha_t})\,x_0, \qquad \text{variance} = \big(m^2(1-\alpha_t) + \sigma_t^2\big)\mathbf{I}.

Right side of (13) is N(αt−1 x0, (1−αt−1)I)\mathcal{N}\big(\sqrt{\alpha_{t-1}}\,x_0,\ (1-\alpha_{t-1})\mathbf{I}\big).

Equating the two Gaussians gives two equations in two unknowns:

(mean)k+mαt=αt−1,\textbf{(mean)}\quad k + m\sqrt{\alpha_t} = \sqrt{\alpha_{t-1}}, (variance)m2(1−αt)+σt2=1−αt−1.\textbf{(variance)}\quad m^2(1-\alpha_t) + \sigma_t^2 = 1-\alpha_{t-1}.

Solve the variance equation for mm, then back-substitute for kk:

m=1−αt−1−σt21−αt,k=αt−1−αt⋅1−αt−1−σt21−αt.(14)m = \sqrt{\frac{1-\alpha_{t-1}-\sigma_t^2}{1-\alpha_t}}, \qquad k = \sqrt{\alpha_{t-1}} - \sqrt{\alpha_t}\cdot\sqrt{\frac{1-\alpha_{t-1}-\sigma_t^2}{1-\alpha_t}}. \tag{14}

Plugging k,mk, m back into the assumed mean μ=kx0+mxt\mu = kx_0 + mx_t and collecting terms yields the paper's reverse distribution:

 qσt(xt−1∣xt,x0)=N ⁣(αt−1 x0+1−αt−1−σt2⋅xt−αt x01−αt,  σt2I). (15)\boxed{\,q_{\sigma_t}(x_{t-1}\mid x_t, x_0) = \mathcal{N}\!\left( \sqrt{\alpha_{t-1}}\,x_0 + \sqrt{1-\alpha_{t-1}-\sigma_t^2}\cdot\frac{x_t - \sqrt{\alpha_t}\,x_0}{\sqrt{1-\alpha_t}},\ \ \sigma_t^2\mathbf{I}\right).\,} \tag{15}

A small rant: the DDIM paper just states this result — it shows almost no derivation, very "look how clever I am." The undetermined-coefficients method above is the honest way to get kk and mm.

Where's the step-skipping?

The distribution (15) still looks like it relates tt and t−1t-1. So where's the acceleration?

The key: we never imposed the Markov property, so t−1t-1 and tt denote the (t−1)(t{-}1)-th and tt-th sampling steps — which need not be consecutive in DDPM time.

Example: the tt-th DDIM step can correspond to DDPM's 900th timepoint, and the (t−1)(t{-}1)-th DDIM step to DDPM's 800th — a 100-step jump in one move. Allowed, precisely because no Markov assumption was used.

We never eliminated x0x_0 via Markov; instead we posited the whole distribution directly (in order to free-load). Both requirements are met, so the Markov shackle is now completely gone — skip away.

The DDIM sampling formula (Eq 16)

Reparameterize-sample from (15). At inference we don't have the true x0x_0, so we substitute the predicted x0x_0. From xt=αt x0+1−αt ϵx_t = \sqrt{\alpha_t}\,x_0 + \sqrt{1-\alpha_t}\,\epsilon, solving for x0x_0 with the model's noise estimate ϵθ(xt)\epsilon_\theta(x_t) gives x^0=(xt−1−αt ϵθ(xt))/αt\hat{x}_0 = \big(x_t - \sqrt{1-\alpha_t}\,\epsilon_\theta(x_t)\big)/\sqrt{\alpha_t}. The result:

 xt−1=αt−1(xt−1−αt ϵθ(xt)αt)⏟“predicted x0”+1−αt−1−σt2⋅ϵθ(xt)⏟“direction pointing to xt”+σt ϵt⏟random noise. (16)\boxed{\,x_{t-1} = \sqrt{\alpha_{t-1}}\underbrace{\left(\frac{x_t - \sqrt{1-\alpha_t}\,\epsilon_\theta(x_t)}{\sqrt{\alpha_t}}\right)}_{\text{``predicted }x_0\text{''}} + \underbrace{\sqrt{1-\alpha_{t-1}-\sigma_t^2}\cdot\epsilon_\theta(x_t)}_{\text{``direction pointing to }x_t\text{''}} + \underbrace{\sigma_t\,\epsilon_t}_{\text{random noise}}.\,} \tag{16}

Reading the three terms (a geometric picture)

Take xtx_t as the origin (think simple!). Then x^0≈\hat x_0 \approx the estimated clean image, sitting some distance away.

  • "predicted x0x_0" — the model's estimate of the clean image, obtained by inverting Formula 2. Straightforward.
  • "direction pointing to xtx_t" — this term carries the opposite sign from the noise inside the predicted-x0x_0 term. The predicted-x0x_0 piece moves away from xtx_t (along −ϵθ-\epsilon_\theta); this +ϵθ+\epsilon_\theta piece pushes back toward xtx_t. Net effect: it nudges the trajectory back a bit in the xtx_t direction.
  • random noise σtϵt\sigma_t\epsilon_t — a stochastic perturbation that scatters the result.

(This also connects to the earlier intuition for why DDIM can skip steps but not too many — worth revisiting alongside.)

The "I" in DDIM: Implicit

DDIM's signature move: set σt=0\sigma_t = 0. Then the random-noise term in (16) vanishes — the sampling process becomes fully deterministic.

But a true diffusion process is defined by randomness — particles diffuse via Brownian motion. Kill the randomness and, strictly, it's no longer a diffusion model. Yet its underlying network is still a DDPM-trained model. So we have something that is "diffusion and not-diffusion" at once. The author's resolution:

Call it an Implicit Diffusion Model — a hidden diffusion model. Its essence comes from diffusion, but it's not literally a diffusion process, because with σt=0\sigma_t = 0 there's no dWdW term, no Wiener process — it's an ODE.

That deterministic, randomness-free sampler is exactly the "I" — Implicit — in DDIM. The model is rooted in diffusion, but the sampling no longer fits the strict definition of one.

Summary

  • Markov was used only to drop x0x_0 from q(xt∣xt−1,x0)q(x_t\mid x_{t-1}, x_0) (Formula 4 → 5). DDIM forgoes Markov, so it must posit the reverse distribution instead of deriving it.
  • Two free-loading requirements: (1) keep the marginal q(xt∣x0)q(x_t\mid x_0) unchanged (gradient-equivalence), (2) make the reverse distribution a simple Gaussian.
  • Assume μ=kx0+mxt\mu = kx_0 + mx_t with constant variance σt2\sigma_t^2; solve k,mk, m by matching distributions (never canceling the independent ϵ\epsilon's) ⇒ Eq (15).
  • Because no Markov was used, tt and t−1t-1 can be far-apart DDPM timepoints ⇒ step-skipping.
  • Sampling formula (16) = predicted-x0x_0 + direction-toward-xtx_t + random-noise.
  • Setting σt=0\sigma_t = 0 removes randomness ⇒ deterministic ODE sampler ⇒ the Implicit in DDIM.

Notes taken from a video tutorial on DDIM (Part 2 of the series).

References

  • Song, J., Meng, C., & Ermon, S. (2021). Denoising Diffusion Implicit Models. ICLR 2021. arXiv:2010.02502
  • Ho, J., Jain, A., & Abbeel, P. (2020). Denoising Diffusion Probabilistic Models. NeurIPS 2020. arXiv:2006.11239