Volume Rendering and NeRF

Properties of Volumes

Volumes, in our context, can be considered as a cloud of particles that ranges from atoms to any particle size. When a photon passes through such a volume, it has the possibility to collide with the particles inside the volume. Such collisions formulate the radiance distribution of the volume. Note that in order to satisfy the statistically independent collisions, the size of each particle should be small enough in comparison to their average distance.

The properties of volumes that impact the radiance, i.e., the radiant flux travel through a differential beam, can be divided into absorption, scattering, and emission:

Absorption: Absorption means some particles can absorb the photons when a collision appears. Let us consider a radiance beam \(L(\mathbf{x}, \omega)\), which starts at \(\mathbf{x}\) with direction \(\omega\), passes through a segment of the volume. The radiance would be reduced, and the rate of such reduction depends on the absorption coefficient \(\sigma_a(\mathbf{x})\), which determines the probability density of a photon being absorbed per unit distance \(dz\):

\[ \frac{dL}{dz} = -\sigma_a(\mathbf{x}) L (\mathbf{x}, \omega). \]

Scattering: Scattering describes a collision which scatters the photons into different directions that defined by phase function \(f_p(\mathbf{x}, \omega, \bar{\omega})\) that is normalised over the sphere:

\[ \int_{S^2} f_p(\mathbf{x}, \omega, \bar{\omega}) d\theta = 1. \]

When scattering appears, some of the photons will be scattered out, and cause the loss of the radiance. The rate of such loss is proportional to the scattering coefficient \(\sigma_s(\mathbf{x})\), which determines the probability density of a photon being scattered per unit distance \(dz\):

\[ \frac{dL}{dz} = -\sigma_s(\mathbf{x}) L (\mathbf{x}, \omega). \]

Since the photons can out-scattering, there are also chances that the photons can scatter into the direction \(\omega\) of the radiance beam:

\[ \frac{dL}{dz} = \sigma_s(\mathbf{x}) L_s (\mathbf{x}, \omega), \]

where the in-scattering radiance \(L_s (\mathbf{x}, \omega)\) is as follows:

\[ L_s (\mathbf{x}, \omega) = \int_{S^2} f_p(\mathbf{x}, \omega, \bar{\omega}) L (\mathbf{x}, \bar{\omega}) d \bar{\omega}. \]

Emission: Emission describes that the radiance can also increase as a result of emission from the volume. Given the emitted radiance \(L_e (\mathbf{x}, \omega)\), the emission should be the product of the absorption coefficient:

\[ \frac{dL}{dz} = \sigma_a(\mathbf{x}) L_e (\mathbf{x}, \omega). \]

Volume Rendering Equations

The radiance distribution within a volume can be defined by radiative transfer Equation (RTE) that combines absorption, scattering, and emission:

\[ \frac{dL(\mathbf{x}, \omega)}{dz} = -\sigma_a(\mathbf{x}) L (\mathbf{x}, \omega) -\sigma_s(\mathbf{x}) L (\mathbf{x}, \omega) +\sigma_s(\mathbf{x}) L_s (\mathbf{x}, \omega) +\sigma_a(\mathbf{x}) L_e (\mathbf{x}, \omega). \]

As both absorption coefficient \(\sigma_a(\mathbf{x})\) and out-scattering operates on the same radiance function \(L (\mathbf{x}, \omega)\), we could merge them into one term:

\[ \frac{dL(\mathbf{x}, \omega)}{dz} = -\sigma_t(\mathbf{x}) L (\mathbf{x}, \omega) +\sigma_s(\mathbf{x}) L_s (\mathbf{x}, \omega) +\sigma_a(\mathbf{x}) L_e (\mathbf{x}, \omega), \]

where \(\sigma_t(\mathbf{x}) = \sigma_a(\mathbf{x}) + \sigma_s(\mathbf{x})\). The RTE describes the change of the radiance beams during the traveling. By integrating both sides of the equation with finite length \(z\), we could obtain the integral form of the RTE:

\[ L(\mathbf{x}, \omega) = \int_{t=0}^z T(t)[\sigma_a(\mathbf{x}_t)L_e(\mathbf{x}_t, \omega) + \sigma_s(\mathbf{x}_t)L_s(\mathbf{x}_t,\omega)]dt, \]

where transmittance \(T(t)=exp(-\int_{s=0}^t \sigma_t(\mathbf{x}_s))ds\) is the probability of a photon that travels from \(\mathbf{x}\) to \(\mathbf{x}_t\) without hitting any particles. In order to model the radiance at the end of the beam, we could add \(L_z(\mathbf{x}_z, \omega)\) to RTE and formulate the volume rendering equation (VRE):

\[ L(\mathbf{x}, \omega) = \int_{t=0}^z T(t)[\sigma_a(\mathbf{x}_t)L_e(\mathbf{x}_t, \omega) + \sigma_s(\mathbf{x}_t)L_s(\mathbf{x}_t \omega)]dt + T(z)L_z(\mathbf{x}_z, \omega). \]

The derivation of \(T(t)\) is as follows:

\[ \begin{aligned} P(\text{survive at } t+dt)&=P(\text{survive before } t) \times P(\text{survive at } t)\\ \Rightarrow T(t+dt) &= T(t) \times (1-\sigma(t)dt)\\ \Rightarrow \quad T(t)+T^{\prime}(t) d t &=T(t)-T(t) \sigma(t) dt\\ \Rightarrow \quad \frac{T^{\prime}(t)}{T(t)} d t &=-\sigma(t) d t \\ \Rightarrow \quad \ln T(t) &=-\int_{t_{0}}^{t} \sigma(s) ds \end{aligned} \]

Volume Rendering for NeRF

The exact interaction between light and the particle is quite a complex problem to solve. Thus, NeRF adopts a volume density scattering model that assumes there is only one scattering of the radiation from the light source to eye. Volume density \(\sigma(\mathbf{r}(t))\) can be considered as the probability of a photon stopping at a small interval around \(t\). And \(T(t)\) is the probability that the ray travels from origin to \(t\) without hitting any other particle. (\(\sigma\): prob of stopping here, \(T(t)\): prob of travelling here) Given the color \(c(\mathbf{r}(t))\) at each point of a radiance beam, the expected color \(C(\mathbf{r})\) of a ray \(\mathbf{r}\) can within near \(t_n\) and far \(t_f\) bound is the sum of the contribution of each volume element:

\[ C(\mathbf{r}) = \int_{t_n}^{t_f} T(t) \sigma(\mathbf{r}(t))c(\mathbf{r}(t), \omega)dt, \hspace{0.25em} \text{where} \hspace{0.25em} T(t) = \exp(-\int_{t_n}^{t}\sigma(\mathbf{r}(s))ds). \]

​

To solve such an integration, NeRF divides \([t_n, t_f]\) into \(N\) equal partitions and uniformly sample point \(t_i\) within each partition:

\[ t_i \sim \mathcal{U}[t_n + \frac{i-1}{N}(t_f-t_n), t_n + \frac{i}{N}(t_f-t_n)]. \]

These samples are then adopted for the estimation of \(C(\mathbf{r})\) using quadrature rule}:

\[ \hat{C}(\mathbf{r}) = \sum_{i=1}^N T_i (1-exp(-\sigma_i(t_{i+1}-t_i)))\mathbf{c}_i, \quad \text{where} \hspace{0.25em} T_i = exp(-\sum_{j=1}^{i-1}\sigma_j(t_{j+1}-t_j)) \]

We can write this equation as:

\[ \hat{C}(\mathbf{r}) = \sum_{i=1}^N T_i \alpha_i\mathbf{c}_i \]

The opacity \(\alpha_i\) describle the differential probability of lights stop here. Lower value means empty space. The transmittance \(T_i\) describle the probability of lights travel to here. Lower value means occluded space.

The derivation is as follows:

\[ \begin{aligned} C(\mathbf{r}) &=\sum_{i=1}^N C(\mathbf{r})_i \\ &=\sum_{i=1}^N\int_{t_i}^{t_{i+1}} T(t) \sigma(\mathbf{r}(t_i))c(\mathbf{r}(t_i), \omega)dt \end{aligned} \]

For each \(C(\mathbf{r})_i\), we get:

\[ \begin{aligned} C(\mathbf{r})_i &= \sigma_i c_i \int_{t_i}^{t_{i+1}} T(t)dt \\ &= \sigma_i c_i \int_{t_i}^{t_{i+1}} \exp(-\int_{t_n}^{t}\sigma(\mathbf{r}(s)) ds) dt\\ &= \sigma_i c_i \int_{t_i}^{t_{i+1}} \exp(-\int_{t_n}^{t_i}\sigma(\mathbf{r}(s)) ds)\exp(-\int_{t_i}^{t}\sigma(\mathbf{r}(s)) ds) dt\\ &=\sigma_i c_i T_i \int_{t_i}^{t_{i+1}} \exp(-\int_{t_i}^{t}\sigma(\mathbf{r}(s)) ds) dt\\ &=\sigma_i c_i T_i \int_{t_i}^{t_{i+1}} \exp(-\sigma_i(t-t_i)) dt\\ &=\sigma_i c_i T_i \frac{\exp(-\sigma_i(t-t_i))}{-\sigma_i}|_{t_i}^{t_{i+1}}\\ &=c_i T_i (1-e^{-\sigma_i\delta_i}) \end{aligned} \]

Reference

https://cs.dartmouth.edu/wjarosz/publications/novak18monte-sig-slides-2-fundamentals-notes.pdf