DDPM from Scratch, Part 1: The Math You Need to Read the Paper

A beginner-friendly walkthrough of the mathematical prerequisites for reading the Denoising Diffusion Probabilistic Models (DDPM) paper by Jonathan Ho, Ajay Jain, and Pieter Abbeel. This first installment doesn't touch the main method — it clears away the "background" hurdles so that, by the end, you can read Section 2 (Background) without getting stuck.

Why this series exists

If your math is, let's say, average — and if you haven't spent much time around AIGC (text-to-image, text-to-video, and friends) — then opening the DDPM paper for the first time is rough. You glance at the title, Denoising Diffusion Probabilistic Models, and by the time you hit the second paragraph of the Background section you're already lost in a thicket of notation.

Here's the thing: you've almost certainly learned all of this math before. You just quietly handed it back to your teachers somewhere along the way. So when people call DDPM "zero-barrier," my honest reaction the first time through was: this isn't zero-barrier — it's barriers everywhere.

The goal of this post is to lower that barrier. If you're already an expert, feel free to skip this entirely — we're only collecting the prerequisite math and the little stumbling blocks you hit while reading the Background section, all in one place. By the end you'll have the six things you need to read the paper comfortably.

---

Setting the stage: the notation that trips everyone up

The Background section opens with:

Diffusion models are latent variable models of the form $$p_\theta(x_0) := \int p_\theta(x_{0:T})\, dx_{1:T},$$ where $x_1, \dots, x_T$ are latents of the same dimensionality as the data $x_0 \sim q(x_0)$.

The first thing that stops you cold is $x_{0:T}$. What is that?

It's just shorthand for the whole sequence:

$$x_{0:T} := x_0, x_1, \dots, x_T.$$

Written as a probability, it's the joint distribution over $x_0$ through $x_T$:

$$p_\theta(x_{0:T}) = p_\theta(x_0, x_1, \dots, x_T).$$

Once you know that, the defining equation reads cleanly:

• $p_\theta(\cdot)$ — anything with the subscript $\theta$ is produced by our model (a deep neural network). We're using the network to predict a distribution.
• $p_\theta(x_0) = \int p_\theta(x_{0:T})\, dx_{1:T}$ — here $dx_{1:T}$ means $dx_1\, dx_2 \cdots dx_T$. We integrate out every latent variable $x_1, \dots, x_T$, and the only variable left standing is $x_0$. So the integral of the joint over all the latents gives back the marginal $p_\theta(x_0)$.

This is just the basic marginalization rule from probability.

A note on rigor: Anything at or below a high-school level we'll use without proof. For some post-high-school results I'll sketch a proof; for the genuinely messy ones I'll skip the derivation to keep the pace. If you want a particular formula derived, that can go in a follow-up.

That's hurdle one cleared. On to the next.

---

Prerequisite 1: The reverse process

The joint distribution $p_\theta(x_{0:T})$ is called the reverse process, and the paper defines it as a Markov chain with learned Gaussian transitions starting at $p(x_T) = \mathcal{N}(x_T; \mathbf{0}, \mathbf{I})$:

$$p_\theta(x_{0:T}) := p(x_T) \prod_{t=1}^{T} p_\theta(x_{t-1} \mid x_t), \qquad p_\theta(x_{t-1} \mid x_t) := \mathcal{N}\big(x_{t-1};\ \mu_\theta(x_t, t),\ \Sigma_\theta(x_t, t)\big).$$

What does "reverse" mean here?

It refers to time. In DDPM:

• Forward direction: $x_0 \to x_T$ (the index increases — clean image gets progressively noisier).
• Reverse direction: $x_T \to x_0$ (the index decreases — noise gets progressively denoised).

We start from $p(x_T)$, the prior distribution, which we simply define to be a standard Gaussian (pure Gaussian noise). The whole model is the journey from that prior back to the data distribution.

Deriving the product form

The transition term $p_\theta(x_{t-1} \mid x_t)$ seems to appear out of nowhere. Let's derive it on a small example so it stops feeling like magic.

Start from the only tool we need — the conditional probability rule you learned in high school:

$$P(A, B) = P(A \mid B)\, P(B) = P(B \mid A)\, P(A).$$

There are two ways to split $P(A,B)$. Which split we pick is the trick, and it's dictated by the direction of the process. The reverse process runs backward in time, so we want to peel off variables along decreasing time — which means the conditioning variable should be the later (larger-index) one.

Take a three-variable example, $P(x_0, x_1, x_2)$. Group $(x_0, x_1)$ together as one block and condition on $x_2$:

$$P(x_0, x_1, x_2) = P(x_0, x_1 \mid x_2)\, P(x_2).$$

Now split the block $P(x_0, x_1 \mid x_2)$ the same way — keep time decreasing, so $x_0$ is conditioned on $x_1$:

$$P(x_0, x_1 \mid x_2) = P(x_0 \mid x_1, x_2)\, P(x_1 \mid x_2).$$

Putting it together:

$$P(x_0, x_1, x_2) = P(x_0 \mid x_1, x_2)\, P(x_1 \mid x_2)\, P(x_2).$$

Applying the Markov property

DDPM assumes the Markov property: a state depends only on the adjacent time step. Anything further back can be dropped. So in $P(x_0 \mid x_1, x_2)$, the $x_2$ condition is non-adjacent to $x_0$ and can be crossed out:

$$P(x_0 \mid x_1, x_2) = P(x_0 \mid x_1).$$

Which gives:

$$P(x_0, x_1, x_2) = P(x_0 \mid x_1)\, P(x_1 \mid x_2)\, P(x_2).$$

Generalizing from $\{x_0, x_1, x_2\}$ to $x_{0:T}$, you can write it in one step — pull out the last term $p(x_T)$ and take the product:

$$\boxed{\,p_\theta(x_{0:T}) = p(x_T) \prod_{t=1}^{T} p_\theta(x_{t-1} \mid x_t)\,}$$

This is a reverse process: the larger time index sits in the condition, the smaller one is what we predict.

---

Prerequisite 2: The forward process

Next comes the forward process (forward process), also a Markov chain — the paper tells you so directly:

$$q(x_{1:T} \mid x_0) := \prod_{t=1}^{T} q(x_t \mid x_{t-1}), \qquad q(x_t \mid x_{t-1}) := \mathcal{N}\big(x_t;\ \sqrt{1 - \beta_t}\, x_{t-1},\ \beta_t \mathbf{I}\big).$$

The forward process is the noising direction: we're given a clean image $x_0$ and progressively add Gaussian noise. Because it's forward, the Markov dependence points to the next time step, and crucially the condition $x_0$ is given.

Deriving the product form

Same trick, opposite direction. Consider $P(x_3, x_2, x_1 \mid x_0)$. Ignore the given $x_0$ for a moment and just split $P(x_3, x_2, x_1)$.

For the forward process we want the conditioning index to be smaller than the predicted one. Peel off $x_1$ first (the smallest), conditioning the rest on it:

$$P(x_3, x_2, x_1) = P(x_3, x_2 \mid x_1)\, P(x_1).$$

Split again the same way:

$$P(x_3, x_2 \mid x_1) = P(x_3 \mid x_2, x_1)\, P(x_2 \mid x_1).$$

So:

$$P(x_3, x_2, x_1) = P(x_3 \mid x_2, x_1)\, P(x_2 \mid x_1)\, P(x_1).$$

Now reattach the given $x_0$ as a "carry-along" condition on every term:

$$P(x_3, x_2, x_1 \mid x_0) = P(x_3 \mid x_2, x_1, x_0)\, P(x_2 \mid x_1, x_0)\, P(x_1 \mid x_0).$$

Apply the Markov property — each state depends only on the immediately previous one ($x_3$ depends only on $x_2$, not on $x_1$ or $x_0$):

$$P(x_3, x_2, x_1 \mid x_0) = P(x_3 \mid x_2)\, P(x_2 \mid x_1)\, P(x_1 \mid x_0).$$

Switching from $P$ to $q$ and generalizing to $T$ terms:

$$\boxed{\,q(x_{1:T} \mid x_0) = \prod_{t=1}^{T} q(x_t \mid x_{t-1})\,}$$

Reverse vs. forward — the one rule to remember

When you decompose a joint with conditional probability, $P(A, B)$ can become $P(A \mid B)P(B)$ or $P(B \mid A)P(A)$. Which one you pick depends on the direction:

Process	Direction	Conditioning index	Why
Reverse	$x_T \to x_0$	the larger time index	We solve for images going backward, so the larger time sits in the condition.
Forward	$x_0 \to x_T$	the smaller time index	We're given $x_0$ and add noise, so the smaller time sits in the condition.

The two split in opposite ways — that's the whole subtlety, and it's easy once you've worked the small examples by hand.

---

Prerequisite 3: The variational bound (and Jensen's inequality)

Now the paper hits you with the training objective:

$$\mathbb{E}\big[-\log p_\theta(x_0)\big] \le \mathbb{E}_q\!\left[-\log \frac{p_\theta(x_{0:T})}{q(x_{1:T} \mid x_0)}\right] = \mathbb{E}_q\!\left[-\log p(x_T) - \sum_{t \ge 1} \log \frac{p_\theta(x_{t-1} \mid x_t)}{q(x_t \mid x_{t-1})}\right] =: L.$$

This is the "variational bound on negative log likelihood." At first glance the left and right sides look unrelated — and that $\le$ sign is doing a lot of quiet work. Let's reconstruct it.

The objective

In image generation we want the probability of generating the image we want to be large — we want our samples to be real images, not noise or garbage. So our goal is to maximize $\log p_\theta(x_0)$. Since this is an optimization problem, maximizing is equivalent to minimizing the negative:

$$\max\ \log p_\theta(x_0) \iff \min\ \big(-\log p_\theta(x_0)\big).$$

The sign is a matter of convenience; ignore it for now and analyze the quantity itself.

Reusing earlier results

Good papers are logically tight — every formula they already wrote down tends to get reused. We have two such formulas sitting around: the marginalization rule, and the $q(x_{1:T}\mid x_0)$ decomposition. Let's force both into play.

Step 1 — marginalize. Using $p_\theta(x_0) = \int p_\theta(x_{0:T})\, dx_{1:T}$:

$$\log p_\theta(x_0) = \log \int p_\theta(x_{0:T})\, dx_{1:T}.$$

Step 2 — multiply and divide by $q$. We still haven't used the forward decomposition, so let's "force" it in by multiplying and dividing by $q(x_{1:T} \mid x_0)$. Since we're integrating over $x_{1:T}$, we need a distribution over $x_{1:T}$ to take an expectation against:

$$\log p_\theta(x_0) = \log \int q(x_{1:T} \mid x_0)\, \frac{p_\theta(x_{0:T})}{q(x_{1:T} \mid x_0)}\, dx_{1:T}.$$

Step 3 — recognize the expectation. An integral of the form $\int q(\cdot)\, [\,\cdot\,]\, dx$ is exactly an expectation with respect to $q$:

$$\log p_\theta(x_0) = \log\ \mathbb{E}_{q(x_{1:T} \mid x_0)}\!\left[\frac{p_\theta(x_{0:T})}{q(x_{1:T} \mid x_0)}\right].$$

The subscript on $\mathbb{E}$ matters enormously. Always be clear which distribution you're taking the expectation over — here it's $q(x_{1:T} \mid x_0)$.

We're now almost at the paper's form. The only difference is the order of $\log$ and $\mathbb{E}$. Can we just swap them? No — and that's where Jensen's inequality comes in.

Jensen's inequality (with the trap that catches everyone)

Start with the definition of a convex function (Chinese: 下凸, "convex from below"). A real-valued function $f$ on a convex subset $C$ is convex if for any two points $x_1, x_2 \in C$ and $0 \le t \le 1$:

$$t\, f(x_1) + (1 - t)\, f(x_2) \ge f\big(t x_1 + (1 - t) x_2\big).$$

(The weighted average of the function values lies above the function of the weighted average — the chord sits above the curve.)

Generalizing to many points gives Jensen's inequality for a convex $f$, with weights $\lambda_i \ge 0$ (in fact $\lambda_i > 0$) and $\sum_{i=1}^{M} \lambda_i = 1$:

$$f\!\left(\sum_{i=1}^{M} \lambda_i x_i\right) \le \sum_{i=1}^{M} \lambda_i f(x_i).$$

Because $\sum_i \lambda_i = 1$ with $\lambda_i > 0$, the weights act exactly like a probability distribution (the discrete analogue of $\int p(x)\,dx = 1$). So this is precisely:

$$f\big(\mathbb{E}[X]\big) \le \mathbb{E}\big[f(X)\big] \qquad (f \text{ convex}).$$

Here is the trap. Our $f$ is $\log$, and $\log$ is concave (Chinese: 上凸, "convex from above"), not convex. A lot of blog posts quote only the convex version — and don't even tell you their $f$ is convex. If you sketch $\log$ you'll see immediately it curves the other way, so the inequality flips:

$$f\big(\mathbb{E}[X]\big) \ge \mathbb{E}\big[f(X)\big] \qquad (f \text{ concave, e.g. } \log).$$

Geometrically: take $x_1, x_2$ on the $\log$ curve and $t = \tfrac12$. The point on the curve above the midpoint $\tfrac12 x_1 + \tfrac12 x_2$ is higher than the midpoint of the chord $\tfrac12 f(x_1) + \tfrac12 f(x_2)$. That's concavity, and it's the reason the inequality reverses.

Putting it together

Apply the concave (flipped) Jensen to our expression — $\log$ moves inside the expectation and the relation becomes $\ge$:

$$\log p_\theta(x_0) = \log\ \mathbb{E}_{q}\!\left[\frac{p_\theta(x_{0:T})}{q(x_{1:T}\mid x_0)}\right] \ \ge\ \mathbb{E}_{q}\!\left[\log \frac{p_\theta(x_{0:T})}{q(x_{1:T} \mid x_0)}\right].$$

This already matches the paper's right-hand side — except for a missing negative sign. Easy: negate both sides (which flips $\ge$ to $\le$), consistent with "maximize $\Rightarrow$ minimize the negative":

$$\boxed{\,-\log p_\theta(x_0) \ \le\ \mathbb{E}_{q}\!\left[-\log \frac{p_\theta(x_{0:T})}{q(x_{1:T} \mid x_0)}\right] =: L\,}$$

That's the variational bound. The paper never tells you it relies on Jensen's inequality — so without knowing it, you simply can't reproduce this step. The rest of the bound's expansion into per-step terms we'll cover next time.

---

Prerequisite 4: The normal distribution PDF

For a Gaussian $X \sim \mathcal{N}(\mu, \sigma^2)$, the probability density is:

$$p(x) = \frac{1}{\sqrt{2\pi}\,\sigma}\, \exp\!\left(-\frac{(x - \mu)^2}{2\sigma^2}\right).$$

You've seen this since high school / early college. It shows up everywhere downstream — including the eventual derivation of the posterior mean $\mu$.

---

Prerequisite 5: Sum of independent Gaussians

If

$$X_1 \sim \mathcal{N}(\mu_1, \sigma_1^2), \qquad X_2 \sim \mathcal{N}(\mu_2, \sigma_2^2),$$

then their sum is again Gaussian, with means adding and variances adding:

$$X_1 + X_2 \sim \mathcal{N}\big(\mu_1 + \mu_2,\ \sigma_1^2 + \sigma_2^2\big).$$

That the mean and variance add follows from the basic linearity/variance properties of expectation. The harder part — proving that the sum is itself normally distributed — uses the convolution formula (the same convolution you know from CNNs):

$$p_{X_1 + X_2}(z) = \int p_{X_1}(x)\, p_{X_2}(z - x)\, dx.$$

---

Prerequisite 6: KL divergence

The Kullback–Leibler (KL) divergence measures how similar two distributions are:

$$D_{\mathrm{KL}}\big(P \parallel Q\big) = \int p(x)\, \log \frac{p(x)}{q(x)}\, dx.$$

Key intuitions:

• Smaller KL $\Rightarrow$ more similar distributions (not the other way around).
• $D_{\mathrm{KL}} \ge 0$ always.
• If $P = Q$, the ratio is always $1$, $\log 1 = 0$, so $D_{\mathrm{KL}} = 0$.
• The further apart $P$ and $Q$ are, the larger the value.

So in training we want this value to be small — we're driving two distributions together. When both $P$ and $Q$ are Gaussian, substituting the normal PDF yields a clean closed-form expression (which we'll use later).

---

The six things — your checklist

If you're solid on these six, you can read the DDPM Background section with zero barriers:

1. Joint-distribution notation & marginalization — $x_{0:T} = x_0,\dots,x_T$, and $p_\theta(x_0) = \int p_\theta(x_{0:T})\,dx_{1:T}$.
2. Reverse process decomposition — conditional probability + Markov property; condition on the larger time index.
3. Forward process decomposition — same tools, condition on the smaller time index.
4. Variational bound — maximize $\log p_\theta(x_0) \iff$ minimize its negative, plus Jensen's inequality (remember: $\log$ is concave, so the inequality flips).
5. Normal distribution PDF and the sum-of-Gaussians rule (via convolution).
6. KL divergence — definition, non-negativity, and "smaller = more similar."

Everything after this in the paper is mostly elementary math — the formulas get long, but they no longer require clever tricks.

---

What's next

Next time we'll tackle the method itself: the tractable forward posterior

$$q(x_{t-1} \mid x_t, x_0) = \mathcal{N}\big(x_{t-1};\ \tilde{\mu}_t(x_t, x_0),\ \tilde{\beta}_t \mathbf{I}\big),$$

with

$$\tilde{\mu}_t(x_t, x_0) := \frac{\sqrt{\bar\alpha_{t-1}}\,\beta_t}{1 - \bar\alpha_t}\, x_0 + \frac{\sqrt{\alpha_t}\,(1 - \bar\alpha_{t-1})}{1 - \bar\alpha_t}\, x_t, \qquad \tilde{\beta}_t := \frac{1 - \bar\alpha_{t-1}}{1 - \bar\alpha_t}\,\beta_t,$$

where $\alpha_t := 1 - \beta_t$ and $\bar\alpha_t := \prod_{s=1}^{t} \alpha_s$. We'll see why we must condition on $x_0$ to turn the unknown into the known, derive the mean and variance, and then use reparameterized sampling to iterate $x_t \to x_{t-1}$ all the way from pure noise $x_T$ back to a clean image $x_0$.

See you in the next one.

---

Notes taken from a video tutorial on DDPM (Part 1 of the series).

---

References

• Ho, J., Jain, A., & Abbeel, P. (2020). Denoising Diffusion Probabilistic Models. NeurIPS 2020. arXiv:2006.11239