Score-based Generative Models

ODEs & SDEs

Ordinary Differential Equations (ODEs)

Ordinary Differential Equations (ODEs): Given an unknown function $y(x)$, if the equation involves not only the variable $x$ but also the derivative of the function $y(x)$, then it is called an ODE. The general form of an ODE is given by:

\[ F(x, y, y', y'', \ldots, y^{(n)}) = 0 \]

where $y^{(n)}$ denotes the $n$-th derivative of $y(x)$. Here $n$ is the order of the ODE. For example, we can consider a first-order ODE as follows:

\[ \frac{dy}{dx} = 2x \]

To solve the above ODE, we need to find a function $y(x)$ such that its derivative is equal to $2x$. By integrating both sides of the equation, we can find the solution to be $y(x) = x^2 + C$, where $C$ is the constant of integration. Here to obtain a unique solution, we need an initial condition, i.e., $y(0) = 1$. An ODE along with an initial condition is called an initial value problem (IVP).

Partial Differential Equations (PDEs): If the equation involves partial derivatives of the unknown function $u(x, y)$, then it is called a PDE. The general form of a PDE is given by:

\[ F(x, y, u, \frac{\partial u}{\partial x}, \frac{\partial u}{\partial y}, \ldots, \frac{\partial^2 u}{\partial x^2}, \frac{\partial^2 u}{\partial y^2}, \ldots) = 0 \]

where $\frac{\partial u}{\partial x}$ denotes the partial derivative of $u(x, y)$ with respect to $x$.

Stochastic Differential Equations (SDEs)

Stochastic Differential Equations (SDEs): SDEs are used to model stochastic processes, where the evolution of the process is not deterministic but random and we only have the probability distribution of the process at any given time. The most common stochastic process is the Brownian motion (or Wiener process).

Wiener process is a continuous-time stochastic process that has the following properties:

We can model the change in the process over a small time interval $\Delta t$ as a Gaussian random variable with mean $0$ and variance $\Delta t$, i.e., $\Delta Z = \epsilon \sqrt{\Delta t}$, where $\epsilon \sim \mathcal{N}(0, 1)$. Here $\Delta Z\sim \mathcal{N}(0, \Delta t)$, which means its variance grows linearly with time.
For time $T$, we have: $$ Z(T) = Z(0) + \sum_{i=1}^n \Delta Z_i $$

Here we could find the mean will always be $0$

Generalized Wiener process: The Wiener process can be generalized to have a drift term $\alpha$ and a variance rate term $b$ as follows:

\[ dX = \alpha dt + b dZ \]

Here $\alpha$ controls the change of the mean while $b$ controls the variance change

Ito process (Diffusion): The Ito process is a stochastic process has a more general form than the Wiener process. The Ito process is defined as follows:

\[ dX = \alpha(X, t) dt + b(X, t) dZ \]

where we could find now the drift term $\alpha$ and the variance rate term $b$ are functions of $X$ and $t$.

ODE solver

Euler method: The Euler method is the simplest numerical method to solve ODEs. Given the following ODE:

\[ \frac{dy}{dx} = f(x, y), \quad y(x_0) = y_0 \]

The Euler method approximates the derivative as follows:

\[ \frac{dy}{dx} \approx \frac{y_{n+1} - y_n}{h} \]

where $h$ is the step size. The Euler method updates the solution as follows:

\[ y_{n+1} = y_n + h f(x_n, y_n) \]

where $x_n = x_0 + nh$ and $y_n$ is the solution at $x_n$.

DDPM and SDEs

Consider the forward process of a DDPM as follows:

\[ x_{i} = \sqrt{\beta_i} \epsilon_{i-1} + \sqrt{1-\beta^2} x_{i-1} \]

We can derive its continuous form as follows:

\[ dx = -\frac{1}{2}\beta(t)x dt + \sqrt{\beta(t)} d\omega \]

where $\omega$ is the Wiener process. This is a SDE that is variance preserving, i.e., the variance of the process remains bounded.