From NeRF to Surf

NeuS

S density function

Propose to use a S-density function (Derivative of Sigmoid):

$${\varphi }_s(x)=\frac{s{e}^{-sx}}{(1+s{e}^{-sx}{)}^2}$$

Note the std is given by $1/s$, which would approaches to 0 as the training convergences. That is to say, the function will be more concentrated to surface point.

import numpy as np
import matplotlib.pyplot as plt
def s_density(x,s=1):
    return s*np.exp(-s*x)/(1+np.exp(-s*x))**2

s_vals = np.linspace(0, 10, 200)
sdf_vals = np.linspace(-5, 5, 200)

sdf_grid, s_grid  = np.meshgrid(sdf_vals, s_vals)
heat = s_density(sdf_grid, s_grid)

plt.imshow(heat, cmap="hot", extent=[sdf_vals[0], sdf_vals[-1], s_vals[-1], s_vals[0]])
plt.xlabel("SDF")
plt.ylabel("s")

Naive solution for S density

Set $\sigma$ in NeRF as S-density, and result in biased solution...

(Note: $t^{\star }$ is surface, $f(t)$ is a SDF function, $\sigma (t)$ is now set to be S-density function, $w(t)$ is the weight function for rendering)

Proposed method

Propose a weight function as follows:

$$w(t)=\frac{{\varphi }_s(f(\mathbf{p}(t)))}{{\int }_0^{\mathrm{\infty }}{\varphi }_s(f(\mathbf{p}(u)))du}$$

The problem of this weight function is that it is not occlusion aware... (This is similar to Neural RGBD weight function). Thus, NeuS proposes to get the weight function in a similar way to NeRF (NeRF's alpha compositing is actually occulusion-aware):

$$w(t)=T(t)\rho (t),\text{where}T(t)=\exp (-{\int }_0^t\rho (u)du).$$

How to get $\rho ?$

We can firstly get sdf function

$$f(\mathbf{p}(t))=-|\cos (\theta )|\cdot (t-t^{\star })$$

Then, the weight function can be rewritten as:

$$\begin{array}{rl}w(t)& =\frac{{\varphi }_s(f(\mathbf{p}(t)))}{{\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{\varphi }_s(f(\mathbf{p}(u)))du}\\ & =\frac{{\varphi }_s(f(\mathbf{p}(t)))}{{\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{\varphi }_s(-|\cos \theta |\cdot (u-t^{\star }))du}\\ & =\frac{{\varphi }_s(f(\mathbf{p}(t)))}{|\cos \theta {|}^{-1}\cdot {\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{\varphi }_s(u-t^{\star })du}\\ & =|\cos \theta |{\varphi }_s(f(\mathbf{p}(t)))=T(t)\rho (t)=-\frac{dT}{dt}(t).\end{array}$$

Note that:

$$|\cos \theta |{\varphi }_s(f(\mathbf{p}(t)))=-\frac{d{\mathrm{\Phi }}_s}{dt}(f(\mathbf{p}(t)))=-\frac{dT}{dt}(t)$$

We can get

$$ T(t) = \Phi_s(f(\mathbf{p}(t))) $$ Then we get $\rho (t)$:

$$\begin{gathered} {\int }_{-\mathrm{\infty }}^t\rho (u)du=-\ln ({\mathrm{\Phi }}_s(f(\mathbf{p}(t)))) \\ \Rightarrow \rho (t)=\frac{\frac{-d{\mathrm{\Phi }}_s}{dt}(f(\mathbf{p}(t)))}{{\mathrm{\Phi }}_s(f(\mathbf{p}(t)))} \end{gathered}$$

For multiple surface intersection, $f(\mathbf{p}(t))$ might be smaller than 0, then, we need clip it:

$$\rho (t)=max(\frac{\frac{-d{\mathrm{\Phi }}_s}{dt}(f(\mathbf{p}(t)))}{{\mathrm{\Phi }}_s(f(\mathbf{p}(t)))},0)$$

VolSDF

Transformed learnable SDF

Propose to model the density using a transformed learnable SDF:

$${\mathrm{\Psi }}_{\beta }(s)=\{\begin{array}{ll}\frac{1}{2}\exp \left(\frac{s}{\beta }\right)& \text{ if }s\le 0\\ 1-\frac{1}{2}\exp (-\frac{s}{\beta })& \text{ if }s>0\end{array}$$

UniSurf

Unifying Surface and Volume Rendering

The rendering equation of NeRF can be written as:

$$\hat{C}(\mathbf{r})=\sum _{i=1}^N\alpha ({\mathbf{x}}_i)\prod _{j<i}(1-\alpha ({\mathbf{x}}_j))c({\mathbf{x}}_i,\mathbf{d})$$

UniSurf re-write the equation as:

$$\hat{C}(\mathbf{r})=\sum _{i=1}^N\mathcal{o}({\mathbf{x}}_i)\prod _{j<i}(1-\mathcal{o}({\mathbf{x}}_j))c({\mathbf{x}}_i,\mathbf{d}),$$

where $\mathcal{o}({\mathbf{x}}_i)$ is a continous occupancy function parameterized with an MLP.

Reference

[NeurIPS'21] NeuS: NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction

[NeurIPS'21] VolSDF: Volume Rendering of Neural Implicit Surfaces

[ICCV'21] UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction