How Google made a 3D video calling booth
May 20, 2021 - Google has developed a “magical” video calling booth that renders a lifelike 3D image of each person – making video conferences more personal than ever before.
“Project Starline” was unveiled in California at Google's annual I/O developer conference.
Still very much at an experimental stage, Starline does work well, as was demonstrated when friends who hadn't seen each other for a long time due to pandemic lockdowns, were allowed to talk with one another through the system – leading to comments like “It really, really felt like she and I were in the same room.”
Alphabet CEO Sundar Pichai says the system uses high-resolution cameras, and is likely the culmination of years of research and company acquisitions made by the tech behemoth.
Although a detailed explanation of the technology was lacking, one can extrapolate that about a dozen cameras and sensors, hidden around the display screen, capture the person from multiple angles to figure out their exact shape and create a realtime 3D model of them.
Steve Seitz, Starline's Director of Engineering, says three breakthroughs came together to make it possible. First, the ability to capture people as they are. Second, to be able to compress the data so it can be sent over existing networks. And third, the technology to render a person realistically on a 3D screen.
Perhaps the most impressive breakthrough in creating such realism comes from its new light field cameras and display – an extremely complex way of tracing all light rays in 3D space that flow through every point and in every direction. The display in particular creates a magical window effect, whereby the other caller looks like they are just on the other side of the glass.
The Covid-19 pandemic has created huge interest in how video conferencing can be improved, so a project like this, if it can be commercialised into an affordable set-up, could pave the way for the future of working-from-home – possibly alleviating the debilitating effects of "Zoom fatigue".
Other applications for Starline, such as in the field of medicine, could have far reaching implications.