This neural net stitches together still photos into absurdly realistic 3D scenes

Neural networks work a little like human brains, insofar as they allow for connections between different data points in a similar way that neurons are interconnected. That makes it possible for them to make inferences about data that isn’t there from data that is. Why is this useful for 3D imaging? Because it means with the right neural network it’s possible to create 3D models — complete with accurate reflections of the sort ray tracing will enable on next-gen gaming consoles — from 2D images alone.

Turning stills into 3D motion — Matthew Tancik, a PhD student at UC Berkley, posted the video below to YouTube that shows the possibilities of what he and his research partners call Neural Radiance Fields (NeRF). What makes NeRF remarkable is the amount of detail, the smooth camera motions, and the insanely accurate rendering of reflections, specularities, and translucence it’s able to achieve from sets of 30 to 100 photos or other 2D images alone.

Better than existing methods — As the video demonstrates, NeRF seriously outperforms the existing, dominant methods of 3D rendering from 2D input: Scene representation networks (SSNs), local light-field fusion (LLFF), and neural volumes (NVs). While each of them are able to achieve some level of inference, each suffers from either flickering, an inability to preserve detail and handle high-resolution images, problems with complex occlusion effects, or a combination of all of the above. Even photogrammetry, which we've previously seen used to amazing effect, can't manage the same calibre results.

Mobile tech could harness it — For now, NeRF is still very processor intensive and the results can take minutes or hours to render, depending on how many inputs there are and how detailed each is. But as processing power continues to increase and the method is refined, it’s possible to imagine how the technology could be combined with the LiDAR sensor found on the new iPad Pro, or the ToF (time of flight) cameras on recent smartphones for incredible results.

The sort of results that are going to make today’s 3D pictures on Facebook appear comparatively prehistoric and laughably crude. But, more importantly, it’s easy to extrapolate to far more utilitarian applications for the technology than merely better 3D images of your lunch.

Self-driving cars, games, you name it — Obvious applications include game production, CGI for TV and movies, and anything else where you want wholly photo-realistic textures or need to replicate complex lighting effects on 3D objects while also allowing for motion. But the technology could also be used to help self-driving cars make predictions about the world around them, guide surgeons removing tumors, or help flying taxis navigate crowded airspace.

But, in the interim, we'll gladly settle for hyper-realistic video games where we can see our character's face in a puddle... or an approaching zombie in the reflection of a broken store window. Or, failing that, the ability to turn photos of our houseplants into insanely realistic 3D models.