---
title: "DDPM from Scratch, Part 1: The Math You Need to Read the Paper"
date: 2026-06-06
tags: ["Diffusion Models", "Generative Models", "Deep Learning"]
author: "Yiyang Yu"
---

> A beginner-friendly walkthrough of the mathematical prerequisites for reading the
> *Denoising Diffusion Probabilistic Models* (DDPM) paper by Jonathan Ho, Ajay Jain, and Pieter Abbeel.
> This first installment doesn't touch the main method — it clears away the "background" hurdles
> so that, by the end, you can read **Section 2 (Background)** without getting stuck.

## Why this series exists

If your math is, let's say, *average* — and if you haven't spent much time around AIGC
(text-to-image, text-to-video, and friends) — then opening the DDPM paper for the first time
is rough. You glance at the title, *Denoising Diffusion Probabilistic Models*, and by the time
you hit the second paragraph of the Background section you're already lost in a thicket of
notation.

Here's the thing: you've almost certainly *learned* all of this math before. You just quietly
handed it back to your teachers somewhere along the way. So when people call DDPM "zero-barrier,"
my honest reaction the first time through was: this isn't zero-barrier — it's **barriers everywhere**.

The goal of this post is to lower that barrier. If you're already an expert, feel free to skip
this entirely — we're only collecting the prerequisite math and the little stumbling blocks you
hit while reading the Background section, all in one place. By the end you'll have the **six
things** you need to read the paper comfortably.

---

## Setting the stage: the notation that trips everyone up

The Background section opens with:

> Diffusion models are latent variable models of the form
> $$p_\theta(x_0) := \int p_\theta(x_{0:T})\, dx_{1:T},$$
> where $x_1, \dots, x_T$ are latents of the same dimensionality as the data $x_0 \sim q(x_0)$.

The first thing that stops you cold is **$x_{0:T}$**. What is that?

It's just shorthand for the whole sequence:

$$x_{0:T} := x_0, x_1, \dots, x_T.$$

Written as a probability, it's the **joint distribution** over $x_0$ through $x_T$:

$$p_\theta(x_{0:T}) = p_\theta(x_0, x_1, \dots, x_T).$$

Once you know that, the defining equation reads cleanly:

- $p_\theta(\cdot)$ — anything with the subscript $\theta$ is produced by our model (a deep neural
  network). We're using the network to **predict a distribution**.
- $p_\theta(x_0) = \int p_\theta(x_{0:T})\, dx_{1:T}$ — here $dx_{1:T}$ means
  $dx_1\, dx_2 \cdots dx_T$. We integrate out every latent variable $x_1, \dots, x_T$, and the
  only variable left standing is $x_0$. So the integral of the joint over all the latents gives
  back the marginal $p_\theta(x_0)$.

This is just the basic **marginalization** rule from probability.

> **A note on rigor:** Anything at or below a high-school level we'll use without proof. For some
> post-high-school results I'll sketch a proof; for the genuinely messy ones I'll skip the
> derivation to keep the pace. If you want a particular formula derived, that can go in a
> follow-up.

That's hurdle one cleared. On to the next.

---

## Prerequisite 1: The reverse process

The joint distribution $p_\theta(x_{0:T})$ is called the **reverse process**, and the paper
defines it as a Markov chain with learned Gaussian transitions starting at
$p(x_T) = \mathcal{N}(x_T; \mathbf{0}, \mathbf{I})$:

$$
p_\theta(x_{0:T}) := p(x_T) \prod_{t=1}^{T} p_\theta(x_{t-1} \mid x_t),
\qquad
p_\theta(x_{t-1} \mid x_t) := \mathcal{N}\big(x_{t-1};\ \mu_\theta(x_t, t),\ \Sigma_\theta(x_t, t)\big).
$$

### What does "reverse" mean here?

It refers to **time**. In DDPM:

- **Forward** direction: $x_0 \to x_T$ (the index increases — clean image gets progressively noisier).
- **Reverse** direction: $x_T \to x_0$ (the index decreases — noise gets progressively denoised).

We start from $p(x_T)$, the **prior distribution**, which we simply *define* to be a standard
Gaussian (pure Gaussian noise). The whole model is the journey from that prior back to the data
distribution.

### Deriving the product form

The transition term $p_\theta(x_{t-1} \mid x_t)$ seems to appear out of nowhere. Let's derive it
on a small example so it stops feeling like magic.

Start from the only tool we need — the **conditional probability** rule you learned in high school:

$$P(A, B) = P(A \mid B)\, P(B) = P(B \mid A)\, P(A).$$

There are two ways to split $P(A,B)$. *Which* split we pick is the trick, and it's dictated by the
**direction of the process**. The reverse process runs **backward in time**, so we want to peel
off variables along *decreasing* time — which means the **conditioning variable should be the
later (larger-index) one**.

Take a three-variable example, $P(x_0, x_1, x_2)$. Group $(x_0, x_1)$ together as one block and
condition on $x_2$:

$$P(x_0, x_1, x_2) = P(x_0, x_1 \mid x_2)\, P(x_2).$$

Now split the block $P(x_0, x_1 \mid x_2)$ the same way — keep time decreasing, so $x_0$ is
conditioned on $x_1$:

$$P(x_0, x_1 \mid x_2) = P(x_0 \mid x_1, x_2)\, P(x_1 \mid x_2).$$

Putting it together:

$$P(x_0, x_1, x_2) = P(x_0 \mid x_1, x_2)\, P(x_1 \mid x_2)\, P(x_2).$$

### Applying the Markov property

DDPM assumes the **Markov property**: a state depends only on the *adjacent* time step. Anything
further back can be dropped. So in $P(x_0 \mid x_1, x_2)$, the $x_2$ condition is non-adjacent to
$x_0$ and can be crossed out:

$$P(x_0 \mid x_1, x_2) = P(x_0 \mid x_1).$$

Which gives:

$$P(x_0, x_1, x_2) = P(x_0 \mid x_1)\, P(x_1 \mid x_2)\, P(x_2).$$

Generalizing from $\{x_0, x_1, x_2\}$ to $x_{0:T}$, you can write it in one step — pull out the
last term $p(x_T)$ and take the product:

$$\boxed{\,p_\theta(x_{0:T}) = p(x_T) \prod_{t=1}^{T} p_\theta(x_{t-1} \mid x_t)\,}$$

This is a reverse process: the **larger time index sits in the condition**, the smaller one is
what we predict.

---

## Prerequisite 2: The forward process

Next comes the **forward process** (`forward process`), also a Markov chain — the paper tells you
so directly:

$$
q(x_{1:T} \mid x_0) := \prod_{t=1}^{T} q(x_t \mid x_{t-1}),
\qquad
q(x_t \mid x_{t-1}) := \mathcal{N}\big(x_t;\ \sqrt{1 - \beta_t}\, x_{t-1},\ \beta_t \mathbf{I}\big).
$$

The forward process is the **noising** direction: we're given a clean image $x_0$ and progressively
add Gaussian noise. Because it's forward, the **Markov dependence points to the *next* time step**,
and crucially the condition $x_0$ is *given*.

### Deriving the product form

Same trick, opposite direction. Consider $P(x_3, x_2, x_1 \mid x_0)$. Ignore the given $x_0$ for a
moment and just split $P(x_3, x_2, x_1)$.

For the forward process we want the **conditioning index to be *smaller*** than the predicted one.
Peel off $x_1$ first (the smallest), conditioning the rest on it:

$$P(x_3, x_2, x_1) = P(x_3, x_2 \mid x_1)\, P(x_1).$$

Split again the same way:

$$P(x_3, x_2 \mid x_1) = P(x_3 \mid x_2, x_1)\, P(x_2 \mid x_1).$$

So:

$$P(x_3, x_2, x_1) = P(x_3 \mid x_2, x_1)\, P(x_2 \mid x_1)\, P(x_1).$$

Now reattach the given $x_0$ as a "carry-along" condition on every term:

$$P(x_3, x_2, x_1 \mid x_0) = P(x_3 \mid x_2, x_1, x_0)\, P(x_2 \mid x_1, x_0)\, P(x_1 \mid x_0).$$

Apply the **Markov property** — each state depends only on the immediately *previous* one
($x_3$ depends only on $x_2$, not on $x_1$ or $x_0$):

$$P(x_3, x_2, x_1 \mid x_0) = P(x_3 \mid x_2)\, P(x_2 \mid x_1)\, P(x_1 \mid x_0).$$

Switching from $P$ to $q$ and generalizing to $T$ terms:

$$\boxed{\,q(x_{1:T} \mid x_0) = \prod_{t=1}^{T} q(x_t \mid x_{t-1})\,}$$

### Reverse vs. forward — the one rule to remember

When you decompose a joint with conditional probability, $P(A, B)$ can become $P(A \mid B)P(B)$ or
$P(B \mid A)P(A)$. **Which one you pick depends on the direction:**

| Process | Direction | Conditioning index | Why |
|---|---|---|---|
| **Reverse** | $x_T \to x_0$ | the **larger** time index | We solve for images going backward, so the larger time sits in the condition. |
| **Forward** | $x_0 \to x_T$ | the **smaller** time index | We're given $x_0$ and add noise, so the smaller time sits in the condition. |

The two split in *opposite* ways — that's the whole subtlety, and it's easy once you've worked the
small examples by hand.

---

## Prerequisite 3: The variational bound (and Jensen's inequality)

Now the paper hits you with the training objective:

$$
\mathbb{E}\big[-\log p_\theta(x_0)\big] \le
\mathbb{E}_q\!\left[-\log \frac{p_\theta(x_{0:T})}{q(x_{1:T} \mid x_0)}\right] =
\mathbb{E}_q\!\left[-\log p(x_T) - \sum_{t \ge 1} \log \frac{p_\theta(x_{t-1} \mid x_t)}{q(x_t \mid x_{t-1})}\right] =: L.
$$

This is the "variational bound on negative log likelihood." At first glance the left and right
sides look unrelated — and that $\le$ sign is doing a lot of quiet work. Let's reconstruct it.

### The objective

In image generation we want the probability of generating *the image we want* to be **large** — we
want our samples to be real images, not noise or garbage. So our goal is to **maximize**
$\log p_\theta(x_0)$. Since this is an optimization problem, maximizing is equivalent to minimizing
the negative:

$$\max\ \log p_\theta(x_0) \iff \min\ \big(-\log p_\theta(x_0)\big).$$

The sign is a matter of convenience; ignore it for now and analyze the quantity itself.

### Reusing earlier results

Good papers are logically tight — every formula they already wrote down tends to get reused. We
have two such formulas sitting around: the marginalization rule, and the $q(x_{1:T}\mid x_0)$
decomposition. Let's force both into play.

**Step 1 — marginalize.** Using $p_\theta(x_0) = \int p_\theta(x_{0:T})\, dx_{1:T}$:

$$\log p_\theta(x_0) = \log \int p_\theta(x_{0:T})\, dx_{1:T}.$$

**Step 2 — multiply and divide by $q$.** We still haven't used the forward decomposition, so let's
"force" it in by multiplying and dividing by $q(x_{1:T} \mid x_0)$. Since we're integrating over
$x_{1:T}$, we need a distribution over $x_{1:T}$ to take an expectation against:

$$
\log p_\theta(x_0)
= \log \int q(x_{1:T} \mid x_0)\, \frac{p_\theta(x_{0:T})}{q(x_{1:T} \mid x_0)}\, dx_{1:T}.
$$

**Step 3 — recognize the expectation.** An integral of the form $\int q(\cdot)\, [\,\cdot\,]\, dx$
is exactly an **expectation** with respect to $q$:

$$
\log p_\theta(x_0)
= \log\ \mathbb{E}_{q(x_{1:T} \mid x_0)}\!\left[\frac{p_\theta(x_{0:T})}{q(x_{1:T} \mid x_0)}\right].
$$

> **The subscript on $\mathbb{E}$ matters enormously.** Always be clear *which distribution* you're
> taking the expectation over — here it's $q(x_{1:T} \mid x_0)$.

We're now *almost* at the paper's form. The only difference is the **order of $\log$ and
$\mathbb{E}$**. Can we just swap them? **No** — and that's where Jensen's inequality comes in.

### Jensen's inequality (with the trap that catches everyone)

Start with the definition of a **convex** function (Chinese: *下凸*, "convex from below"). A
real-valued function $f$ on a convex subset $C$ is convex if for any two points $x_1, x_2 \in C$
and $0 \le t \le 1$:

$$t\, f(x_1) + (1 - t)\, f(x_2) \ge f\big(t x_1 + (1 - t) x_2\big).$$

(The weighted average of the function values lies *above* the function of the weighted average —
the chord sits above the curve.)

Generalizing to many points gives **Jensen's inequality** for a convex $f$, with weights
$\lambda_i \ge 0$ (in fact $\lambda_i > 0$) and $\sum_{i=1}^{M} \lambda_i = 1$:

$$f\!\left(\sum_{i=1}^{M} \lambda_i x_i\right) \le \sum_{i=1}^{M} \lambda_i f(x_i).$$

Because $\sum_i \lambda_i = 1$ with $\lambda_i > 0$, the weights act exactly like a probability
distribution (the discrete analogue of $\int p(x)\,dx = 1$). So this is precisely:

$$f\big(\mathbb{E}[X]\big) \le \mathbb{E}\big[f(X)\big] \qquad (f \text{ convex}).$$

**Here is the trap.** Our $f$ is $\log$, and **$\log$ is *concave*** (Chinese: *上凸*, "convex from
above"), not convex. A lot of blog posts quote only the convex version — and don't even tell you
their $f$ is convex. If you sketch $\log$ you'll see immediately it curves the other way, so the
inequality **flips**:

$$f\big(\mathbb{E}[X]\big) \ge \mathbb{E}\big[f(X)\big] \qquad (f \text{ concave, e.g. } \log).$$

Geometrically: take $x_1, x_2$ on the $\log$ curve and $t = \tfrac12$. The point on the *curve*
above the midpoint $\tfrac12 x_1 + \tfrac12 x_2$ is **higher** than the midpoint of the chord
$\tfrac12 f(x_1) + \tfrac12 f(x_2)$. That's concavity, and it's the reason the inequality reverses.

### Putting it together

Apply the concave (flipped) Jensen to our expression — $\log$ moves inside the expectation and the
relation becomes $\ge$:

$$
\log p_\theta(x_0)
= \log\ \mathbb{E}_{q}\!\left[\frac{p_\theta(x_{0:T})}{q(x_{1:T}\mid x_0)}\right]
\ \ge\ \mathbb{E}_{q}\!\left[\log \frac{p_\theta(x_{0:T})}{q(x_{1:T} \mid x_0)}\right].
$$

This already matches the paper's right-hand side — except for a missing negative sign. Easy: negate
both sides (which **flips** $\ge$ to $\le$), consistent with "maximize $\Rightarrow$ minimize the
negative":

$$
\boxed{\,-\log p_\theta(x_0)
\ \le\ \mathbb{E}_{q}\!\left[-\log \frac{p_\theta(x_{0:T})}{q(x_{1:T} \mid x_0)}\right] =: L\,}
$$

That's the variational bound. The paper never tells you it relies on Jensen's inequality — so
without knowing it, you simply can't reproduce this step. The rest of the bound's expansion into
per-step terms we'll cover next time.

---

## Prerequisite 4: The normal distribution PDF

For a Gaussian $X \sim \mathcal{N}(\mu, \sigma^2)$, the probability density is:

$$p(x) = \frac{1}{\sqrt{2\pi}\,\sigma}\, \exp\!\left(-\frac{(x - \mu)^2}{2\sigma^2}\right).$$

You've seen this since high school / early college. It shows up everywhere downstream — including
the eventual derivation of the posterior mean $\mu$.

---

## Prerequisite 5: Sum of independent Gaussians

If

$$X_1 \sim \mathcal{N}(\mu_1, \sigma_1^2), \qquad X_2 \sim \mathcal{N}(\mu_2, \sigma_2^2),$$

then their sum is again Gaussian, with **means adding** and **variances adding**:

$$X_1 + X_2 \sim \mathcal{N}\big(\mu_1 + \mu_2,\ \sigma_1^2 + \sigma_2^2\big).$$

That the mean and variance add follows from the basic linearity/variance properties of expectation.
The harder part — proving that the *sum* is itself normally distributed — uses the **convolution
formula** (the same convolution you know from CNNs):

$$p_{X_1 + X_2}(z) = \int p_{X_1}(x)\, p_{X_2}(z - x)\, dx.$$

---

## Prerequisite 6: KL divergence

The **Kullback–Leibler (KL) divergence** measures how *similar* two distributions are:

$$D_{\mathrm{KL}}\big(P \parallel Q\big) = \int p(x)\, \log \frac{p(x)}{q(x)}\, dx.$$

Key intuitions:

- **Smaller KL $\Rightarrow$ more similar** distributions (not the other way around).
- $D_{\mathrm{KL}} \ge 0$ always.
- If $P = Q$, the ratio is always $1$, $\log 1 = 0$, so $D_{\mathrm{KL}} = 0$.
- The further apart $P$ and $Q$ are, the larger the value.

So in training we **want this value to be small** — we're driving two distributions together. When
both $P$ and $Q$ are Gaussian, substituting the normal PDF yields a clean closed-form expression
(which we'll use later).

---

## The six things — your checklist

If you're solid on these six, you can read the DDPM Background section with **zero barriers**:

1. **Joint-distribution notation & marginalization** — $x_{0:T} = x_0,\dots,x_T$, and
   $p_\theta(x_0) = \int p_\theta(x_{0:T})\,dx_{1:T}$.
2. **Reverse process decomposition** — conditional probability + Markov property; condition on the
   *larger* time index.
3. **Forward process decomposition** — same tools, condition on the *smaller* time index.
4. **Variational bound** — maximize $\log p_\theta(x_0) \iff$ minimize its negative, plus
   **Jensen's inequality** (remember: $\log$ is *concave*, so the inequality flips).
5. **Normal distribution PDF** and the **sum-of-Gaussians** rule (via convolution).
6. **KL divergence** — definition, non-negativity, and "smaller = more similar."

Everything after this in the paper is mostly elementary math — the formulas get long, but they no
longer require clever tricks.

---

## What's next

Next time we'll tackle the **method** itself: the tractable forward posterior

$$q(x_{t-1} \mid x_t, x_0) = \mathcal{N}\big(x_{t-1};\ \tilde{\mu}_t(x_t, x_0),\ \tilde{\beta}_t \mathbf{I}\big),$$

with

$$
\tilde{\mu}_t(x_t, x_0) := \frac{\sqrt{\bar\alpha_{t-1}}\,\beta_t}{1 - \bar\alpha_t}\, x_0
+ \frac{\sqrt{\alpha_t}\,(1 - \bar\alpha_{t-1})}{1 - \bar\alpha_t}\, x_t,
\qquad
\tilde{\beta}_t := \frac{1 - \bar\alpha_{t-1}}{1 - \bar\alpha_t}\,\beta_t,
$$

where $\alpha_t := 1 - \beta_t$ and $\bar\alpha_t := \prod_{s=1}^{t} \alpha_s$. We'll see *why* we
must condition on $x_0$ to turn the unknown into the known, derive the mean and variance, and then
use **reparameterized sampling** to iterate $x_t \to x_{t-1}$ all the way from pure noise $x_T$ back
to a clean image $x_0$.

See you in the next one.

---

*Notes taken from [a video tutorial](https://www.bilibili.com/video/BV1zz421i7UM/) on DDPM (Part 1 of the series).*


---

## References

- Ho, J., Jain, A., & Abbeel, P. (2020). *Denoising Diffusion Probabilistic Models.* NeurIPS 2020. [arXiv:2006.11239](https://arxiv.org/abs/2006.11239)