As the 2018 FIFA World Cup kicks off today in Russia, millions of people around the world will watch the matches on their TVs, computers, and smartphones. But what if you could watch it on your kitchen table in the form of a 3D hologram?
That was the goal of a recent project between researchers from the University of Washington, Facebook, and Google. The team developed a first-of-its-kind deep learning-based system that was able to transform videos of a soccer game into a “moving 3D reconstruction, in which the players and field can be rendered interactively with a 3D viewer or through an Augmented Reality device.”
Past experiments have shown it’s possible to reconstruct, say, a soccer match in a 3D hologram by outfitting the field with many high-tech cameras. But that’s expensive and difficult. That’s why the researchers set out to accomplish the same thing by using only a single YouTube video to reconstruct the soccer game in 3D.
Stills from video game converted into 3D imagery
“FIFA, similar to most games, uses deferred shading during gameplay,” the team wrote. “Having access to the GPU calls enables capture of the depth and color buffers per frame. Once depth and color are captured for a given frame, we process it to extract the players.”
The next step was to try rendering 3D holograms from real-life soccer matches on YouTube. The results were impressive.
Stills from real-life soccer games converted into 3D imagery
But there are still kinks to work out: the hologram didn’t reproduce a moving ball, its image quality was somewhat low compared to the 4k-resolution source videos, it wasn’t produced in real time, and it could only be viewed from one side of the field.
Still, it’s possible researchers will be able to develop the technology in time for the 2022 World Cup, which will be held in Qatar.