---
title: "DDPM from Scratch, Part 3: The Reverse Denoising Process"
date: 2026-06-14
tags: ["Diffusion Models", "Generative Models", "Deep Learning"]
author: "Yiyang Yu"
---

> The third installment. Having built the prerequisites (Part 1) and derived the closed-form
> **forward/noising** process (Part 2 — the "route"), we now tackle the **reverse/denoising**
> process: how to walk a noisy image *backward*, one step at a time, into the image we want. In the
> experienced-driver picture this is the **car** (车).

## Loot from the previous levels

Clearing a level earns rewards you carry into the next. Here's our inventory of known conditions:

**① The (human-defined) one-step noising formula:**
$$q(x_t \mid x_{t-1}) := \mathcal{N}\big(x_t;\ \sqrt{1-\beta_t}\, x_{t-1},\ \beta_t \mathbf{I}\big).$$
Don't ask *why* — the driver just picks a reasonable route.

**② The closed-form jump from $x_0$** (Part 2's nesting-dolls result):
$$
q(x_t \mid x_0) = \mathcal{N}\big(x_t;\ \sqrt{\bar\alpha_t}\, x_0,\ (1-\bar\alpha_t)\mathbf{I}\big)
\ \Longleftrightarrow\
x_t = \sqrt{\bar\alpha_t}\, x_0 + \sqrt{1-\bar\alpha_t}\,\varepsilon,\quad \varepsilon \sim \mathcal{N}(\mathbf{0}, \mathbf{I}).
$$
This is the *reparameterized* form; we'll run it in **reverse** to read off the underlying Gaussian.

**③ The normal PDF**, which we'll write in the "expanded exponent" form (this is the form that makes
the template-matching trick work):
$$
p(x) = \frac{1}{\sqrt{2\pi}\,\sigma}\exp\!\left(-\frac{(x-\mu)^2}{2\sigma^2}\right)
= \frac{1}{\sqrt{2\pi}\,\sigma}\exp\!\left[-\frac12\left(\frac{x^2}{\sigma^2} - \frac{2\mu x}{\sigma^2} + \frac{\mu^2}{\sigma^2}\right)\right].
$$

**④ and ⑤ — the author's bookkeeping definitions:**
$$
\bar\alpha_t = \alpha_t\,\alpha_{t-1}\cdots\alpha_1 = \prod_{i=1}^{t}\alpha_i,
\qquad \alpha_t + \beta_t = 1.
$$

---

## Why we need a *distribution*, not a match

The goal is the **reverse** direction (reverse = backward in **time**): given $x_t$, find $x_{t-1}$.
Specifically we want the distribution $q(x_{t-1} \mid x_t)$, then sample from it.

Why a distribution and not a direct mapping? Because this is a **generative** model — we want to
*create*, including images outside the training set. If we learn a relationship between two
*distributions*, we can sample one point from the prior and generate a corresponding point in the
data distribution $p_{\text{data}}$. A one-to-one sample matching could never work:

- There are infinitely many noise samples — you can't enumerate the pairing.
- Your finite training samples are tiny relative to the space. Sample a fresh noise vector that
  deviates from anything you've seen, and a match-based scheme produces garbage.

So we build a **distribution-to-distribution** relationship. That's what lets us generalize. This is
a *generation* task, not a *matching* task.

---

## Step 1: Bayes' rule to flip time

Look at the known conditions — they all run **forward in time** ($t-1 \to t$, with $t > t-1 > 0$).
We know forward; we're asked for reverse. The first instinct: **flip time with Bayes' rule**.

$$q(x_{t-1} \mid x_t) = \frac{q(x_t \mid x_{t-1})\, q(x_{t-1})}{q(x_t)}.$$

But this introduces a new problem: $q(x_{t-1})$ and $q(x_t)$ are **data distributions we don't
know**. (If we knew those, we wouldn't need any of this.) Dead end — as written.

---

## Step 2: Add the condition $x_0$

The fix: **condition everything on $x_0$.**

$$q(x_{t-1} \mid x_t, x_0) = \frac{q(x_t \mid x_{t-1}, x_0)\, q(x_{t-1} \mid x_0)}{q(x_t \mid x_0)}.$$

**Is adding $x_0$ legitimate?** Two reasons it is:

1. **It makes the problem solvable** — every term on the right becomes something we know (below).
2. **It matches the training setup.** In (unconditional) training, our data *is* a set of original
   images $x_0$. We add noise to $x_0$ and ask the model to restore it. The only thing we genuinely
   have in hand is $x_0$ — so it's exactly the condition we're *allowed* to add. You can only
   condition on what you can actually obtain, and we can obtain $x_0$.

And conditions ① and ② quietly hint that conditioning on $x_0$ is the *only* way to make the
distribution tractable.

### The three terms are all known Gaussians

- $q(x_t \mid x_{t-1}, x_0) \overset{\text{Markov}}{=} q(x_t \mid x_{t-1})$ — by the **Markov
  property** only adjacent timesteps interact, so $x_0$ vanishes. This is **condition ①**.
- $q(x_{t-1} \mid x_0)$ — **condition ②** (at $t-1$).
- $q(x_t \mid x_0)$ — **condition ②** (at $t$).

Explicitly:

$$q(x_t \mid x_{t-1}) = \mathcal{N}\big(x_t;\ \sqrt{\alpha_t}\, x_{t-1},\ \beta_t \mathbf{I}\big),$$
$$q(x_{t-1} \mid x_0) = \mathcal{N}\big(x_{t-1};\ \sqrt{\bar\alpha_{t-1}}\, x_0,\ (1-\bar\alpha_{t-1})\mathbf{I}\big),$$
$$q(x_t \mid x_0) = \mathcal{N}\big(x_t;\ \sqrt{\bar\alpha_t}\, x_0,\ (1-\bar\alpha_t)\mathbf{I}\big).$$

> ⚠️ **Watch the bars.** Some terms use $\bar\alpha$ (cumulative product) and some use plain
> $\alpha$. The single most common derivation error is copying $\bar\alpha_{t-1}$ where you meant
> $\alpha_t$, or dropping a bar. Keep them straight.

---

## Step 3: Plug in the PDFs and collect powers of $x_{t-1}$

Substitute the three Gaussian densities (condition ③). The leading $\tfrac{1}{\sqrt{2\pi}\sigma}$
**coefficients are constants** and can be dropped — they don't affect the resulting mean/variance.
(I checked this by carrying them through too; the final $\mu$ and $\sigma^2$ come out identical, which
is itself a confirmation that **a product/quotient of Gaussians is again Gaussian**.)

Multiplying/dividing exponentials just adds/subtracts exponents. The combined exponent is:

$$
-\frac{1}{2}\left[\frac{(x_t - \sqrt{\alpha_t}\,x_{t-1})^2}{\beta_t}
+ \frac{(x_{t-1} - \sqrt{\bar\alpha_{t-1}}\,x_0)^2}{1-\bar\alpha_{t-1}}
- \frac{(x_t - \sqrt{\bar\alpha_t}\,x_0)^2}{1-\bar\alpha_t}\right].
$$

We're after a distribution over $x_{t-1}$, so expand and **collect by powers of $x_{t-1}$**:

$$
\exp\!\left(-\frac12\left[
\underbrace{\left(\frac{\alpha_t}{\beta_t} + \frac{1}{1-\bar\alpha_{t-1}}\right)}_{\text{coeff. of } x_{t-1}^2}\, x_{t-1}^2
\;-\; \underbrace{\left(\frac{2\sqrt{\alpha_t}}{\beta_t}\,x_t + \frac{2\sqrt{\bar\alpha_{t-1}}}{1-\bar\alpha_{t-1}}\,x_0\right)}_{\text{coeff. of } x_{t-1}}\, x_{t-1}
\;+\; C(x_t, x_0)\right]\right).
$$

Here $C(x_t, x_0)$ collects everything independent of $x_{t-1}$. It doesn't affect $\mu$ or
$\sigma^2$ — in fact, plugging the final $\mu, \sigma^2$ back in reproduces exactly the
"completing-the-square" remainder, another sanity check.

---

## Step 4: "Trace the gourd to draw the ladle" (照葫芦画瓢)

Last time the keyword was *nesting dolls*. This time it's **照葫芦画瓢** — *match the template*. We
line our exponent up against the standard form
$-\frac12\big(\frac{x^2}{\sigma^2} - \frac{2\mu x}{\sigma^2} + \frac{\mu^2}{\sigma^2}\big)$ and read
off $\sigma^2$ and $\mu$.

### Variance — from the $x_{t-1}^2$ coefficient

$$
\frac{1}{\sigma^2} = \frac{\alpha_t}{\beta_t} + \frac{1}{1-\bar\alpha_{t-1}}
= \frac{\alpha_t(1-\bar\alpha_{t-1}) + \beta_t}{\beta_t(1-\bar\alpha_{t-1})}.
$$

Simplify the numerator using $\alpha_t \bar\alpha_{t-1} = \bar\alpha_t$ and $\alpha_t + \beta_t = 1$:

$$\alpha_t - \alpha_t\bar\alpha_{t-1} + \beta_t = (\alpha_t + \beta_t) - \bar\alpha_t = 1 - \bar\alpha_t.$$

Therefore:

$$
\frac{1}{\sigma^2} = \frac{1-\bar\alpha_t}{\beta_t(1-\bar\alpha_{t-1})}
\quad\Longrightarrow\quad
\boxed{\ \sigma^2 = \tilde\beta_t = \frac{1-\bar\alpha_{t-1}}{1-\bar\alpha_t}\,\beta_t.\ }
$$

This is the paper's $\tilde\beta_t$.

### Mean — from the $x_{t-1}$ coefficient

Matching $\dfrac{2\mu}{\sigma^2}$ to the linear coefficient:

$$
\frac{2\mu}{\sigma^2} = \frac{2\sqrt{\alpha_t}}{\beta_t}\,x_t + \frac{2\sqrt{\bar\alpha_{t-1}}}{1-\bar\alpha_{t-1}}\,x_0
\quad\Longrightarrow\quad
\mu = \frac12\,\sigma^2\left(\frac{2\sqrt{\alpha_t}}{\beta_t}\,x_t + \frac{2\sqrt{\bar\alpha_{t-1}}}{1-\bar\alpha_{t-1}}\,x_0\right).
$$

Substituting $\sigma^2 = \tilde\beta_t$ and simplifying:

$$
\boxed{\ \tilde\mu_t(x_t, x_0) = \frac{\sqrt{\alpha_t}\,(1-\bar\alpha_{t-1})}{1-\bar\alpha_t}\,x_t
+ \frac{\sqrt{\bar\alpha_{t-1}}\,\beta_t}{1-\bar\alpha_t}\,x_0.\ }
$$

So the reverse posterior is:

$$q(x_{t-1} \mid x_t, x_0) = \mathcal{N}\big(x_{t-1};\ \tilde\mu_t(x_t, x_0),\ \tilde\beta_t \mathbf{I}\big).$$

The algebra to get $\tilde\mu_t$ is pure elementary manipulation (no trick) — just keep applying
$\alpha_t\bar\alpha_{t-1} = \bar\alpha_t$ and $\alpha_t + \beta_t = 1$, and don't miscopy a term.
$x_t$ is the previous (known) step and $x_0$ is the original training image, so everything in
$\tilde\mu_t$ is available **at training time**. Without conditioning on $x_0$, you genuinely could
not solve this.

---

## Step 5: From $x_0$ to noise — why DDPM predicts $\varepsilon$

Here's the catch. $\tilde\mu_t$ needs $x_0$. At **training** time we have it. At **inference** time
we **don't** — we only have a noisy $x_t$. So we let a model predict it.

Recall from condition ② that $x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\varepsilon$.
Solve for $x_0$:

$$\hat{x}_0 = \frac{x_t - \sqrt{1-\bar\alpha_t}\,\varepsilon}{\sqrt{\bar\alpha_t}}.$$

This says $x_0$ and the noise $\varepsilon$ are interchangeable as the unknown. The model has two
equivalent options:

- **Predict $x_0$ directly**, or
- **Predict the noise $\varepsilon$** — and recover $\hat x_0$ from the equation above.

DDPM chooses to **predict the noise $\varepsilon$**. Knowing $\hat\varepsilon$ gives $\hat x_0$,
which gives $\tilde\mu_t$, which lets us sample $x_{t-1}$ and iterate
$x_T \to x_{T-1} \to \cdots \to x_0$.

### The loss

We want our predicted noise to match the true noise — minimize the gap between two tensors. The
natural choices are **MSE** ($L_2$) or **L1**; DDPM (and most work) uses **MSE**. So the training
objective collapses to: *predict the injected noise accurately*,

$$\mathcal{L} = \big\|\,\varepsilon - \varepsilon_\theta(x_t, t)\,\big\|^2.$$

Interestingly, this same "predict $\varepsilon$ with MSE" target also drops out of the *principled*
route — maximizing the likelihood $\max_\theta \log p_\theta(x)$ (the variational bound from Part 1).
Think of it as two paths to the same technique: the "forward discipline" (likelihood / loss
derivation) and the "reverse discipline" (this posterior derivation). Both land on noise prediction
with MSE.

---

## Summary

- **Goal:** the reverse posterior $q(x_{t-1} \mid x_t)$ — but we only know forward conditionals.
- **Bayes' rule** flips time but introduces unknown data marginals $q(x_t), q(x_{t-1})$.
- **Condition on $x_0$**: $q(x_{t-1}\mid x_t, x_0) = \dfrac{q(x_t\mid x_{t-1})\,q(x_{t-1}\mid x_0)}{q(x_t\mid x_0)}$.
  Legitimate because we *have* $x_0$ at training time, and the Markov property drops $x_0$ from the
  first term.
- All three terms are **known Gaussians**; plug in the PDFs, drop constants, and collect powers of
  $x_{t-1}$.
- **Template-match (照葫芦画瓢)** against the standard Gaussian exponent to read off:
  $$\sigma^2 = \tilde\beta_t = \frac{1-\bar\alpha_{t-1}}{1-\bar\alpha_t}\beta_t,\qquad \tilde\mu_t = \frac{\sqrt{\alpha_t}(1-\bar\alpha_{t-1})}{1-\bar\alpha_t}x_t + \frac{\sqrt{\bar\alpha_{t-1}}\beta_t}{1-\bar\alpha_t}x_0.$$
- At inference $x_0$ is unknown, so the model predicts it — equivalently **predicts the noise
  $\varepsilon$** via $\hat x_0 = (x_t - \sqrt{1-\bar\alpha_t}\,\varepsilon)/\sqrt{\bar\alpha_t}$ —
  trained with **MSE**.

That's the **car**. Next time: the **driver** — the principled likelihood/loss derivation
("forward discipline"), worked step by step.

---

*Notes taken from [a video tutorial](https://www.bilibili.com/video/BV1zz421i7UM/) on DDPM (Part 3 of the series).*


---

## References

- Ho, J., Jain, A., & Abbeel, P. (2020). *Denoising Diffusion Probabilistic Models.* NeurIPS 2020. [arXiv:2006.11239](https://arxiv.org/abs/2006.11239)