Latin America. Despite years of hype, VIRTUAL reality headsets have yet to overtake TV or computer screens as reference devices for video viewing.
One reason: VIRTUAL reality can make users feel bad. Nausea and eye strain can occur because virtual reality creates an illusion of 3D visualization, although the user is in fact looking at a fixed-distance 2D screen. The solution for better 3D visualization could lie in a 60-year-old technology remade for the digital world: holograms.
Holograms offer an exceptional representation of the 3D world around us. In addition, they are beautiful. Holograms offer a changing perspective based on the viewer's position, and allow the eye to adjust the focal depth to alternately focus on the foreground and background.
Researchers have long sought to make computer-generated holograms, but the process has traditionally required a supercomputer to perform physics simulations, which is time-consuming and can produce less than photorealistic results. Now, MIT researchers have developed a new way to produce holograms almost instantaneously, and the deep learning-based method is so efficient that it can run on a laptop in the blink of an eye, the researchers say.
"People previously thought that with existing consumer hardware, it was impossible to do real-time 3D holography calculations," says Liang Shi, lead author of the study and a doctoral student in MIT's Department of Electrical Engineering and Computer Science (EECS). "It has often been said that commercially available holographic displays will be available in 10 years, yet this claim has been around for decades."
Shi believes the new approach, which the team calls "tensor holography," will finally put that elusive 10-year goal within reach. The breakthrough could fuel an overflow of holography in fields such as virtual reality and 3D printing.
Shi worked on the study, published today in Nature, with his advisor and co-author Wojciech Matusik. Other co-authors include Beichen Li of EECS and MIT's Computer Science and Artificial Intelligence Laboratory, as well as former MIT researchers Changil Kim (now on Facebook) and Petr Kellnhofer (now at Stanford University).
The search for a better 3D
A typical lens-based photograph encodes the brightness of each light wave; a photo can faithfully reproduce the colors of a scene, but ultimately produces a flat image.
In contrast, a hologram encodes both the brightness and phase of each light wave. That combination offers a more faithful description of the parallax and depth of a scene. So while a photograph of Monet's "Water Lilies" can highlight the color palette of the paintings, a hologram can bring the work to life, representing the unique 3D texture of each brush stroke. But despite their realism, holograms are a challenge to make and share.
First developed in the mid-twentieth century, the first holograms were optically recorded. That required splitting a laser beam, with half of the beam used to illuminate the subject and the other half used as a reference for the phase of light waves. This reference generates the unique sense of depth of a hologram. The resulting images were static, so they couldn't capture the movement. And they were just hard copies, which made reproduction and sharing difficult.
Computer-generated holography avoids these challenges by simulating optical configuration. But the process can be computational hard work. "Because each point in the scene has a different depth, you can't apply the same operations to everyone," Shi says. "That increases complexity significantly." Running a clustered supercomputer to run these physics-based simulations could take seconds or minutes for a single holographic image. In addition, existing algorithms do not model the occlusion with photorealistic accuracy. So Shi's team took a different approach: letting the computer teach physics to itself.
They used deep learning to accelerate computer-generated holography, allowing for real-time hologram generation. The team designed a convolutional neural network, a processing technique that uses a chain of trainable tensors to roughly mimic how humans process visual information. Training a neural network usually requires a large, high-quality dataset, which did not previously exist for 3D holograms.
The team built a custom database of 4,000 pairs of computer-generated images. Each pair matched an image, including the color and depth information of each pixel, with its corresponding hologram. To create the holograms in the new database, the researchers used scenes with complex and variable shapes and colors, with the depth of the pixels evenly distributed from background to foreground, and with a new set of physics-based calculations to handle occlusion. That approach resulted in photorealistic training data. Next, the algorithm got to work.
By learning from each pair of images, the tensor network modified the parameters of its own calculations, successively improving its ability to create holograms. The fully optimized network operated orders of magnitude faster than physics-based calculations. That efficiency surprised the team itself.
"We're amazed at how well it works," Matusik says. In just milliseconds, tensor holography can create holograms from images with depth information, which is provided by typical computer-generated images and can be calculated from a multi-camera setup or a LiDAR sensor (both are standard on some new smartphones). This breakthrough paves the way for real-time 3D holography. In addition, the compact tensor network requires less than 1 MB of memory. "It's insignificant, considering the tens and hundreds of gigabytes available on the latest cell phone," he says.
The research "shows that true 3D holographic displays are practical with only moderate computational requirements," says Joel Kollin, Microsoft's chief optical architect who was not involved in the research. He adds that "this document shows a marked improvement in image quality over previous work," which "will add realism and comfort to the viewer." Kollin also hints at the possibility that holographic screens like this can even be customized to a viewer's ophthalmic prescription. "Holographic screens can correct aberrations in the eye. This makes possible a sharper screen image than the wearer might see with contact lenses or glasses, which only correct for low-order aberrations such as focusing and astigmatism."
"A considerable leap"
Real-time 3D holography would improve a plethora of systems, from virtual reality to 3D printing. The team says the new system could help immerse VR viewers in more realistic scenarios, while eliminating eye strain and other side effects of prolonged VR use. The technology could easily be implemented in displays that modulate the phase of light waves. Currently, more affordable consumer displays modulate only brightness, although the cost of phase modulation displays would decrease if they were widely adopted.
Three-dimensional holography could also drive the development of volumetric 3D printing, the researchers say. This technology could prove faster and more accurate than traditional layer-by-layer 3D printing, as volumetric 3D printing allows simultaneous projection of the entire 3D pattern. Other applications include microscopy, medical data visualization, and surface design with unique optical properties.
"It's a considerable leap that could completely change people's attitude towards holography," says Matusik. "We feel that neural networks were born for this task."
The work was supported, in part, by Sony.
Text written by Daniel Ackerman of MIT.
