Part 3 of the DDIM series. In Part 2 we removed the Markov shackle and derived DDIM's step-skipping sampler under two free-loading requirements. But "it looks like the noising process is untouched" isn't a proof. Here we prove it rigorously: under DDIM's setup, the loss is equivalent to DDPM's — which is what licenses free-loading a pretrained DDPM model.

What "equivalent" means — Theorem 1

The DDIM paper states this as Theorem 1:

Theorem 1. For all σ>0\sigma > 0, there exists γ∈R>0T\gamma \in \mathbb{R}^T_{>0} and C∈RC \in \mathbb{R} such that Jσ=Lγ+CJ_\sigma = L_\gamma + C.

Decoding it:

  • JσJ_\sigma — the DDIM loss (under DDIM's sampling setup, variance vector σ\sigma).
  • LγL_\gamma — the DDPM loss (the γ\gamma-weighted noise-prediction MSE we derived in Part 1 / the DDPM series).
  • They differ by only a constant CC. (And note: σ>0\sigma > 0 strictly — remember this, it matters at the end.)

Two losses that differ by a constant (and a constant scale) have the same gradient direction, so optimizing one optimizes the other. That's the whole point: train once with DDPM, sample with DDIM.

Which process does DDIM actually change?

DDPM's loss derivation starts from two joint distributions:

q(x1:T∣x0)=∏t=1Tq(xt∣xt−1),(17)q(x_{1:T}\mid x_0) = \prod_{t=1}^{T} q(x_t \mid x_{t-1}), \tag{17} pθ(x0:T)=p(xT)∏t=1Tpθ(xt−1∣xt).(18)p_\theta(x_{0:T}) = p(x_T)\prod_{t=1}^{T} p_\theta(x_{t-1}\mid x_t). \tag{18}

Formula 17 is the forward (noising) joint (Markov chain). Formula 18 is the reverse (denoising) joint — what the neural network pθp_\theta models.

Quiz: DDIM changes one of these. Which — the forward (17) or the reverse (18)?

Answer (surprising): DDIM changes the forward process, Formula 17.

Why? In Part 2, the term q(xt∣xt−1)q(x_t \mid x_{t-1}) in the Bayes expansion would normally have an x0x_0 that Markov lets us drop. But DDIM is non-Markovian, so x0x_0 cannot be dropped — the forward transition is really q(xt∣xt−1,x0)q(x_t \mid x_{t-1}, x_0). That extra x0x_0 is exactly what changes Formula 17.

Picture the DDIM graphical model: xtx_t is determined jointly by xt−1x_{t-1} and x0x_0 (arrows into xtx_t from both). So:

DDIM modifies the forward noising joint distribution q(x1:T∣x0)q(x_{1:T}\mid x_0). The reverse model pθp_\theta is left untouched — to free-load, you change qq, not pθp_\theta.

Rebuilding the forward joint (Formula 19)

So each factor in Formula 17 picks up an x0x_0: q(xt∣xt−1)→q(xt∣xt−1,x0)q(x_t \mid x_{t-1}) \to q(x_t \mid x_{t-1}, x_0). The paper then "shows off" by jumping straight to the answer — but the honest path is Bayes plus a telescoping product:

q(x1:T∣x0)=∏t=1Tq(xt∣xt−1,x0)=∏t=1Tq(xt−1∣xt,x0) q(xt∣x0)q(xt−1∣x0).q(x_{1:T}\mid x_0) = \prod_{t=1}^{T} q(x_t \mid x_{t-1}, x_0) = \prod_{t=1}^{T} \frac{q(x_{t-1}\mid x_t, x_0)\,q(x_t \mid x_0)}{q(x_{t-1}\mid x_0)}.

(That first step is just Bayes applied to each factor — note the swap of xtx_t and xt−1x_{t-1} between condition and main variable, the tell-tale sign of Bayes.) Split the product:

=∏t=1Tq(xt−1∣xt,x0) ⋅ ∏t=1Tq(xt∣x0)q(xt−1∣x0).= \prod_{t=1}^{T} q(x_{t-1}\mid x_t, x_0)\ \cdot\ \prod_{t=1}^{T}\frac{q(x_t\mid x_0)}{q(x_{t-1}\mid x_0)}.

The second product telescopes — write out t=1,2,3,…,Tt = 1, 2, 3, \dots, T and watch the middle factors cancel:

∏t=1Tq(xt∣x0)q(xt−1∣x0)=q(x1∣x0)q(x0∣x0)⋅q(x2∣x0)q(x1∣x0)⋯q(xT∣x0)q(xT−1∣x0)=q(xT∣x0)q(x0∣x0).\prod_{t=1}^{T}\frac{q(x_t\mid x_0)}{q(x_{t-1}\mid x_0)} = \frac{q(x_1\mid x_0)}{q(x_0\mid x_0)}\cdot\frac{q(x_2\mid x_0)}{q(x_1\mid x_0)}\cdots\frac{q(x_T\mid x_0)}{q(x_{T-1}\mid x_0)} = \frac{q(x_T\mid x_0)}{q(x_0\mid x_0)}.

By the Kunkun rule q(x0∣x0)=1q(x_0\mid x_0) = 1 ("given Kunkun practiced two and a half years, the probability he practiced two and a half years is 1"), the denominator is 11. Hence:

q(x1:T∣x0)=q(xT∣x0)∏t=1Tq(xt−1∣xt,x0).(19)q(x_{1:T}\mid x_0) = q(x_T\mid x_0)\prod_{t=1}^{T} q(x_{t-1}\mid x_t, x_0). \tag{19}

Finally, tag on the human-set variance subscript σ\sigma (the distribution is controlled by the variance), giving the form of Eq. 6 in the original DDIM paper (an external reference — not our lecture-numbered Eq 6):

qσ(x1:T∣x0)=qσ(xT∣x0)∏t=1Tqσ(xt−1∣xt,x0).(20)q_\sigma(x_{1:T}\mid x_0) = q_\sigma(x_T\mid x_0)\prod_{t=1}^{T} q_\sigma(x_{t-1}\mid x_t, x_0). \tag{20}

Why does the product start at t=1t=1, not t=2t=2? Just like DDPM, the t=1t=1 term can be folded into the t≥2t \ge 2 terms (Kunkun rule again), so we don't split out t=1t=1 as a special case — handling one fewer special term makes the rest of the derivation much cleaner.

Why does a "forward" joint look like a sum of "reverse" conditionals q(xt−1∣xt,x0)q(x_{t-1}\mid x_t, x_0)? Because Formula 20 is the result after applying Bayes. It looks incongruous (one side forward-noising, the other reverse-conditioned), but that's exactly the Bayes-rewritten form — and crucially, this DDIM forward step relates tt to its previous step t−1t-1 (which need not be the strictly-adjacent DDPM timestep) plus an x0x_0. The price of non-Markovianity is carrying that x0x_0 along.

Deriving JσJ_\sigma and reducing it

The DDIM loss JσJ_\sigma follows the DDPM recipe: maximum likelihood, then max⁡→min⁡\max \to \min with a log⁡\log and a minus sign. Using Formula 20 for qσq_\sigma and Formula 18 for pθp_\theta:

Jσ=Eq(x0:T) ⁣[−log⁡pθ(x0:T)qσ(x1:T∣x0)]=Eq(x0:T) ⁣[−log⁡p(xT)∏t=1Tpθ(xt−1∣xt)qσ(xT∣x0)∏t=1Tqσ(xt−1∣xt,x0)].J_\sigma = \mathbb{E}_{q(x_{0:T})}\!\left[-\log\frac{p_\theta(x_{0:T})}{q_\sigma(x_{1:T}\mid x_0)}\right] = \mathbb{E}_{q(x_{0:T})}\!\left[-\log\frac{p(x_T)\prod_{t=1}^T p_\theta(x_{t-1}\mid x_t)}{q_\sigma(x_T\mid x_0)\prod_{t=1}^T q_\sigma(x_{t-1}\mid x_t, x_0)}\right].

Flip the minus sign into the ratio (invert numerator/denominator) and turn products into sums of logs:

=Eq(x0:T) ⁣[log⁡qσ(xT∣x0)p(xT)⏟model-free, ignore+∑t=1Tlog⁡qσ(xt−1∣xt,x0)pθ(xt−1∣xt)].= \mathbb{E}_{q(x_{0:T})}\!\left[\underbrace{\log\frac{q_\sigma(x_T\mid x_0)}{p(x_T)}}_{\text{model-free, ignore}} + \sum_{t=1}^{T}\log\frac{q_\sigma(x_{t-1}\mid x_t, x_0)}{p_\theta(x_{t-1}\mid x_t)}\right].
  • The first term involves only qσ(xT∣x0)q_\sigma(x_T\mid x_0) (known: x0x_0 is data, the endpoint is the approximate standard normal) and p(xT)p(x_T) — no network, so it's a constant we drop.
  • Each summand is a log⁡\log ratio under an expectation → a KL divergence:
=Eq(x0:T) ⁣[∑t=1TDKL(qσ(xt−1∣xt,x0) ∥ pθ(xt−1∣xt))].(21)= \mathbb{E}_{q(x_{0:T})}\!\left[\sum_{t=1}^{T} D_{\mathrm{KL}}\big(q_\sigma(x_{t-1}\mid x_t, x_0)\,\|\,p_\theta(x_{t-1}\mid x_t)\big)\right]. \tag{21}

Why the outer E\mathbb{E} survives the KL

A natural worry: doesn't E[log⁡qp]\mathbb{E}[\log\frac{q}{p}] become the KL, consuming the expectation? Not fully. The log⁡\log ratio concerns the xt−1x_{t-1} conditional, but the outer expectation is over the full joint q(x0:T)q(x_{0:T}). Writing it out:

Eq(x0:T) ⁣[log⁡qσ(xt−1∣xt,x0)pθ(xt−1∣xt)]=∫log⁡(⋯ ) q(x0:T) dx0:T.\mathbb{E}_{q(x_{0:T})}\!\left[\log\frac{q_\sigma(x_{t-1}\mid x_t, x_0)}{p_\theta(x_{t-1}\mid x_t)}\right] = \int \log(\cdots)\, q(x_{0:T})\, dx_{0:T}.

Forming the KL only "uses up" the single xt−1x_{t-1} conditional pulled from the joint — the marginals over the other latents (x0,x1,…x_0, x_1, \dots except xt−1x_{t-1}) are still being integrated. So the outer Eq(x0:T)\mathbb{E}_{q(x_{0:T})} cannot be cancelled; it persists. (Recall the KL identity DKL(P∥Q)=∫p(x)log⁡p(x)q(x)dx=EP[log⁡pq]D_{\mathrm{KL}}(P\|Q) = \int p(x)\log\frac{p(x)}{q(x)}dx = \mathbb{E}_P[\log\frac{p}{q}].)

Why we may write pθp_\theta as qσq_\sigma

Next subtlety: in the KL we replace pθ(xt−1∣xt)p_\theta(x_{t-1}\mid x_t) with qσ(xt−1∣xt,x^0)q_\sigma(x_{t-1}\mid x_t, \hat x_0). Is that legal? Yes — because the functional form is already fixed. In DDPM, the model pθp_\theta predicts x^0\hat x_0 (equivalently ϵ^\hat\epsilon — the two are interchangeable), and the reverse distribution is a Gaussian with a known mean form (a weighted combo of x0,xtx_0, x_t) and a fixed variance. In DDIM we worked hard in Part 2 to pin down exactly that qσq_\sigma form. So pθp_\theta is a qσq_\sigma — the only difference is that its x0x_0 is the model's estimate x^0\hat x_0 rather than the dataset's true x0x_0:

Jσ≐Eq(x0:T) ⁣[∑t=1TDKL(qσ(xt−1∣xt,x0) ∥ qσ(xt−1∣xt,x^0))],x^0=model estimate.J_\sigma \doteq \mathbb{E}_{q(x_{0:T})}\!\left[\sum_{t=1}^{T} D_{\mathrm{KL}}\big(q_\sigma(x_{t-1}\mid x_t, x_0)\,\|\,q_\sigma(x_{t-1}\mid x_t, \hat x_0)\big)\right], \quad \hat x_0 = \text{model estimate}.

Here "≐\doteq" means equal up to constant terms.

Collapsing the Gaussian KL to MSE

Both arguments of the KL are the same qσq_\sigma Gaussian, differing only in x0x_0 vs. x^0\hat x_0. Plug into the two-Gaussian KL formula (derived in the DDPM loss post):

DKL=log⁡σ2σ1+σ12+(μ1−μ2)22σ22−12.D_{\mathrm{KL}} = \log\frac{\sigma_2}{\sigma_1} + \frac{\sigma_1^2 + (\mu_1 - \mu_2)^2}{2\sigma_2^2} - \frac12.

Drop all constant terms:

  • log⁡σ2σ1=log⁡1=0\log\frac{\sigma_2}{\sigma_1} = \log 1 = 0 — both variances are the same human-specified σt\sigma_t.
  • σ122σ22=12\frac{\sigma_1^2}{2\sigma_2^2} = \frac12 — cancels the −12-\frac12.

Only the mean-difference survives. The two means differ only through x0x_0 vs x^0\hat x_0 (the kx0+mxtkx_0 + mx_t form, same k,m,xtk,m,x_t), so (μ1−μ2)2∝∥x0−x^0∥2(\mu_1 - \mu_2)^2 \propto \|x_0 - \hat x_0\|^2:

Jσ≐Eq(x0) ⁣[∑t=1T∥x0−x^0∥222σt2].J_\sigma \doteq \mathbb{E}_{q(x_0)}\!\left[\sum_{t=1}^{T} \frac{\|x_0 - \hat x_0\|_2^2}{2\sigma_t^2}\right].

Now substitute x0=xt−1−αt ϵtαtx_0 = \dfrac{x_t - \sqrt{1-\alpha_t}\,\epsilon_t}{\sqrt{\alpha_t}} (invert Formula 2) for both the true and predicted x0x_0, and fold all the constants into a per-step coefficient γt\gamma_t:

Jσ=∑t=1TEϵt∼N(0,I) ⁣[γt ∥ϵt−ϵθ(αt x0+1−αt ϵt, t)∥22]≐Lγ.J_\sigma = \sum_{t=1}^{T}\mathbb{E}_{\epsilon_t\sim\mathcal{N}(\mathbf{0},\mathbf{I})}\!\left[ \gamma_t\,\Big\|\epsilon_t - \epsilon_\theta\big(\sqrt{\alpha_t}\,x_0 + \sqrt{1-\alpha_t}\,\epsilon_t,\ t\big)\Big\|_2^2\right] \doteq L_\gamma.

This is identical in form to DDPM's loss (Formula 8) — same γt\gamma_t out front, same noise-prediction MSE. The two γt\gamma_t's might differ, but we may force them equal; and even if they didn't match, a constant scale is absorbed by the learning rate (LL tripled ⇔ learning rate ×13\times \tfrac13). To be rigorous the paper sets the γt\gamma_t in JσJ_\sigma equal to DDPM's, yielding exactly

 Jσ=Lγ+C. \boxed{\,J_\sigma = L_\gamma + C.\,}

QED. DDIM's change to the forward process does not change the gradient of the DDPM training loss — so a DDPM-trained model can be free-loaded by DDIM's sampler.

The fine print: σ>0\sigma > 0 vs. DDIM's σ=0\sigma = 0

One catch: Theorem 1 requires σ>0\sigma > 0 strictly. Why? Look at the 12σt2\frac{1}{2\sigma_t^2} denominators — at σt=0\sigma_t = 0 they blow up (division by zero). But DDIM's headline trick is exactly σt=0\sigma_t = 0 (the deterministic, Implicit sampler from Part 2). The paper patches this by continuity: the equality holds as σ→0\sigma \to 0 (i.e. for arbitrarily small σ\sigma), so we may treat σ=0\sigma = 0 as the limiting case where Theorem 1 still holds.

Summary

  • Theorem 1: Jσ=Lγ+CJ_\sigma = L_\gamma + C — DDIM's loss equals DDPM's up to a constant, so their gradients agree and the pretrained DDPM model is reusable.
  • DDIM changes the forward joint q(x1:T∣x0)q(x_{1:T}\mid x_0) (Formula 17 → 19/20), not the reverse model pθp_\theta — because non-Markovianity forbids dropping x0x_0 from q(xt∣xt−1,x0)q(x_t\mid x_{t-1}, x_0).
  • Rebuild the forward joint via Bayes + a telescoping product + the Kunkun rule (Eq 20).
  • Derive JσJ_\sigma by max-likelihood → drop the model-free xTx_T term → recognize KL divergences (the outer E\mathbb{E} survives; pθp_\theta rewrites as qσq_\sigma with x^0\hat x_0).
  • The Gaussian KL collapses (equal variances, constants dropped) to the noise-prediction MSE — identical to DDPM's loss.
  • Theorem 1 needs σ>0\sigma > 0; DDIM's σ=0\sigma = 0 is justified as the σ→0\sigma \to 0 limit.

This closes the loop opened in Part 2: changing the forward noising process to allow step-skipping is precisely what lets the reverse sampling skip steps — for free.

Notes taken from a video tutorial on DDIM (Part 3 of the series).

References

  • Song, J., Meng, C., & Ermon, S. (2021). Denoising Diffusion Implicit Models. ICLR 2021. arXiv:2010.02502
  • Ho, J., Jain, A., & Abbeel, P. (2020). Denoising Diffusion Probabilistic Models. NeurIPS 2020. arXiv:2006.11239