Neural Radiance Fields (NeRF) in Unity

I'll use this thread to share my progress on porting the results of some research papers to Unity, which explore ways to combine neural radiance fields with other representations suitable for real-time rendering.
______________________

MobileNeRF in Unity
(Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architectures)


Unity Package on GitHub
Original research paper: mobile-nerf.github.io

 

Sparse Neural Radiance Grids (SNeRG) in Unity
(Baking Neural Radiance Fields for Real-Time View Synthesis)


Unity Package on GitHub
Original research paper: phog.github.io/snerg

 

looks rather amazing, what's going on, is it generating a LEGO from a few reference states, or, is it a way of storing a ton of information in a few ' bricks ', etc etc . . . .

 

SNeRG is essentially doing raymarching / volume rendering. Here's a good example of doing this in Unity using medical datasets.

The novel thing with SNeRG is, that they bake all the information from a NeRF to multiple volume textures. In addition to color & density, it's also accumulating a specular color during raymarching, so you still get the view-dependend effects that NeRFs are good at learning. That specular term is actually calculated by evaluating a small, view-dependent MLP (Multi Layer Perceptron - a type of neural network) in the fragment shader, which still seems crazy to me.

 

Super cool! Do you know of a way to natively render a pre-trained NeRF from Nerfstudio in Unity? Or is that something that can be done with the SNeRG approach?

 

I'm not too familiar with Nerfstudio, but from their documentation and some GitHub issue it seems, that right now they mostly focus on NeRF approaches that represent the entire scene in a neural network (as opposed to approaches like MobileNeRF, SNeRG, MERF, PlenOctree, Plenoxels, BakedSDF and many others that extract/optimize more traditional 3D representations suitable for realtime rendering).

Supported by Nerfstudio (at the time of writing this post):

• NeRF
• Mip-NeRF
• Nerfacto
• Instant-NGP
• Instruct-NeRF2NeRF
• LERF
• Semantic NeRF-W

So I'm afraid that right now there isn't an easy way to bring a NeRF trained in Nerfstudio into Unity.

Unless you're fine with exporting to a mesh or a point cloud, but that kinda defeats the purpose.

 

Gold. Plz keep updating

 

This is a fantastic start. I haven't had time to dig into all of the guts of it yes, but it looks very feature-rich compared to the other NeRF stuff I've seen. Thank you, and please keep us updated on this work!

 

Thanks for this thread.
Did you try https://docs.nerf.studio/en/latest/nerfology/methods/tetranerf.html ?
I'm willing to experiment this for a real estate interior realtime visit.

 

Hi, any news on any updates?

 

Some recent updates
- The MobileNeRF implementation got some performance improvements making it feasible to run in VR.
- For SNeRG, I added Single Pass Stereo rendering and depth testing
- I ported another technique to Unity: MERF: Memory-Efficient Radiance Fields, an improvement to SNeRG that uses a lot less VRAM. It's available on GitHub, but not yet fully working: https://github.com/julienkay/MERF-Unity-Viewer

On a related note, someone else did an implementation of BakedSDF for both Unity and Unreal:
- https://github.com/AyoubKhammassi/BakedSDF2FBX
- https://github.com/AyoubKhammassi/UnityBakedSDF
- https://github.com/AyoubKhammassi/UnrealBakedSDF

 

Thanks for the update

There are implementations to run bakedsdf models from the paper in-engine but there is no implementation to train your own bakedsdf model yet, right?

 

I have not tried it and it seems not very well documented, but from this issue, it seems sdfstudio has an implementation to train BakedSDF models. I'm not sure if they are supporting all that would be necessary to export the models though. Might require some work to get into the proper format.

 

this seems to be the main nerf related thread,
so adding this gaussian splat repo from Aras:

https://github.com/aras-p/UnityGaussianSplatting

 

Ah exactly, sdfstudio is what I was referring to.

 

Snap built a Neural Light Fields implementation that runs in their AR framework on mobile devices. The output consists of ONNX files which could be usable in Sentis.
I'll see if I can rewrite the render code for Unity.

See: https://github.com/snap-research/MobileR2L