Watching sport on a 2D screen is never satisfying. A full 3D 360 surround vision is what is required to see what is going on and hence the idea that sport is one TV event that might benefit from special cameras and viewing devices. Now we have an alternative that might mean no changes are needed. Watch soccer on your table top in realistic AR "hologram" - it's as good as being there.

Instead of bringing new cameras to the playing field, the idea is to use AI to construct a depth field from the standard monocular view of a TV camera. This sounds comparatively possible these days, but it is worth remembering that only recently such an approach would have been considered hopelessly optimistic. The key is that now we are confident enough to try to train a neural network to extract the depth field.

But where to get the labeled data for the training.

What is needed are some videos with players labeled with their depth in the image. While there are lots of sports videos not many, if any, are labeled and hence suitable as a training set. The solution that the team from the University of Washington, Google and Facebook thought up was to use the well known FIFA soccer game. RenderDox was used to intercept calls from the game engine to the GPU. This provides depth and color information for each player. The players were separated from the noise of the pitch and surroundings and identified as individual players. Next the neural network was trained to identify players and depth.

The complete game reconstruction pipeline is more complicated than just depth estimation. First the camera pose has to be estimated to allow the figures to be placed on the pitch. Next players are identified and a smoothed track computed. From the depth data, a 3D mesh for each player is constructed and this is used in the AR presentation.

The system was tried out on YouTube videos and it seems to work. Take a look:

It is a shame that there is only a short view of how the games look on a AR tabletop at the start of the clip.

Before you get carried away and start to imagine watching your favorite spot on a table top. it is worth listing some of the problems:

First, only a depth map is reconstructed per player currently, which provides a satisfactory viewing experience from only one side of the field. Further, occluded portions of players are not reconstructed. Hallucinating the opposite sides (geometry and texture) and occluded portions of players would enable viewing from any angle. Second, further improvements in player detection, tracking, and depth estimation will help reduce occasional artifacts and reconstructing the ball in the field will enable a more satisfactory viewing of an entire game.

Did you notice the important part?

There is no ball. The team hasn't as yet got round to detecting and rendering the ball, so at the moment what you have is a 3D "spot the ball" competition.

This is the first step on the road to something that is likely to be important in the future. With more development you might well be able to opt to view a standard broadcast of a soccer game as it if was happening in front of you on a tabletop - and yes complete with the ball and even the referee.

The code is available on Github.

More Information

Soccer on Your Tabletop

Konstantinos Rematas, Ira Kemelmacher-Shlizerman, Brian Curless, and Steve Seitz to appear in CVPR 2018!

Related Articles

Walk A Thousand VR Miles And Never Leave The Room

Holoportation - Another Step for Microsoft Research

Is Real Reality The Next Big Thing?

Slalom Skiing New Robot Sport

RoboCup In Its 21st Year

RoboCup 2016 Introduces Outdoor Robot Soccer

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info