From NeRF to Surf

NeuS

S density function

Propose to use a S-density function (Derivative of Sigmoid):

\[ \phi_s(x) = \frac{se^{-sx}}{(1+se^{-sx})^2} \]

Note the std is given by \(1/s\), which would approaches to 0 as the training convergences. That is to say, the function will be more concentrated to surface point.

import numpy as np
import matplotlib.pyplot as plt
def s_density(x,s=1):
    return s*np.exp(-s*x)/(1+np.exp(-s*x))**2

s_vals = np.linspace(0, 10, 200)
sdf_vals = np.linspace(-5, 5, 200)

sdf_grid, s_grid  = np.meshgrid(sdf_vals, s_vals)
heat = s_density(sdf_grid, s_grid)

plt.imshow(heat, cmap="hot", extent=[sdf_vals[0], sdf_vals[-1], s_vals[-1], s_vals[0]])
plt.xlabel("SDF")
plt.ylabel("s")

Naive solution for S density

Set \(\sigma\) in NeRF as S-density, and result in biased solution...

(Note: \(t^\star\) is surface, \(f(t)\) is a SDF function, \(\sigma(t)\) is now set to be S-density function, \(w(t)\) is the weight function for rendering)

Proposed method

Propose a weight function as follows:

\[ w(t) = \frac{\phi_s(f(\mathbf{p}(t)))}{\int_0^{\infty}\phi_s(f(\mathbf{p}(u)))du} \]

The problem of this weight function is that it is not occlusion aware... (This is similar to Neural RGBD weight function). Thus, NeuS proposes to get the weight function in a similar way to NeRF (NeRF's alpha compositing is actually occulusion-aware):

\[ w(t) = T(t)\rho(t), \text{where} \hspace{0.25em} T(t) = \exp(-\int_0^t\rho(u)du). \]

How to get \(\rho?\)

We can firstly get sdf function

\[ f(\mathbf{p}(t)) = -|\cos(\theta)| \cdot (t-t^\star) \]

Then, the weight function can be rewritten as:

\[ \begin{aligned} w(t) &= \frac{\phi_s(f(\mathbf{p}(t)))}{\int_{-\infty}^{\infty}\phi_s(f(\mathbf{p}(u)))du}\\ &= \frac{\phi_s(f(\mathbf{p}(t)))}{\int_{-\infty}^{\infty}\phi_s(-|\cos\theta|\cdot(u-t^\star))du}\\ &= \frac{\phi_s(f(\mathbf{p}(t)))}{|\cos\theta|^{-1}\cdot\int_{-\infty}^{\infty}\phi_s(u-t^\star)du}\\ &= |\cos\theta|\phi_s(f(\mathbf{p}(t))) = T(t)\rho(t)=-\frac{dT}{dt}(t). \end{aligned} \]

Note that:

\[ |\cos\theta|\phi_s(f(\mathbf{p}(t))) = -\frac{d\Phi_s}{dt}(f(\mathbf{p}(t))) = -\frac{dT}{dt}(t) \]

We can get

$$ T(t) = \Phi_s(f(\mathbf{p}(t))) $$ Then we get \(\rho(t)\):

\[ \int^t_{-\infty} \rho(u)du = -\ln (\Phi_s(f(\mathbf{p}(t))))\\ \Rightarrow \rho(t) = \frac{\frac{-d\Phi_s}{dt}(f(\mathbf{p}(t)))}{\Phi_s(f(\mathbf{p}(t)))} \]

For multiple surface intersection, \(f(\mathbf{p}(t))\) might be smaller than 0, then, we need clip it:

\[ \rho(t) = \max(\frac{\frac{-d\Phi_s}{dt}(f(\mathbf{p}(t)))}{\Phi_s(f(\mathbf{p}(t)))}, 0) \]

VolSDF

Transformed learnable SDF

Propose to model the density using a transformed learnable SDF:

\[ \Psi_\beta(s)= \begin{cases} \frac{1}{2} \exp \left(\frac{s}{\beta}\right) & \text { if } s \leq 0 \\ 1-\frac{1}{2} \exp \left(-\frac{s}{\beta}\right) & \text { if } s>0 \end{cases} \]

UniSurf

Unifying Surface and Volume Rendering

The rendering equation of NeRF can be written as:

\[ \hat{C}(\mathbf{r}) = \sum_{i=1}^N \alpha(\mathbf{x}_i)\prod_{j<i}(1-\alpha(\mathbf{x}_j))c(\mathbf{x}_i, \mathbf{d}) \]

UniSurf re-write the equation as:

\[ \hat{C}(\mathbf{r}) = \sum_{i=1}^N \mathcal{o}(\mathbf{x}_i)\prod_{j<i}(1-\mathcal{o}(\mathbf{x}_j))c(\mathbf{x}_i, \mathbf{d}), \]

where \(\mathcal{o}(\mathbf{x}_i)\) is a continous occupancy function parameterized with an MLP.

Reference

[NeurIPS'21] NeuS: NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction

[NeurIPS'21] VolSDF: Volume Rendering of Neural Implicit Surfaces

[ICCV'21] UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction