Using Artificial Intelligence to Freeze Key Super Bowl Action
A 3D reconstruction of the moment when Kansas City receiver Kadarius Toney slipped past Philadelphia Eagles receiver Zach Pascal in Super Bowl LVII
As part of The Times's coverage of Super Bowl LVII, R&D and The New York Times Graphics Desk collaborated to offer readers a 3D perspective on a decisive moment of the game — when Kansas City receiver Kadarius Toney slipped past Philadelphia Eagles receiver Zach Pascal during a late game punt return. An artificial intelligence machine learning model was used to generate 3D models of athletes from a single photograph for the key moment.
This work builds on methods we initially used to analyze winning goals for our coverage of the 2022 FIFA World Cup. You can read a detailed explanation of the process here.
We used the regulation dimensions of an N.F.L. field to calculate the position of our photographer’s camera in 3D space by annotating key points on-field in a series of photographs. This allowed us to quickly generate accurate 3D geometry that we aligned the player models to.
Our method for estimating pose works across many applications and many sports, but we took a few specifics into consideration when optimizing the workflow for the Super Bowl instead of the World Cup:
- Football fields have more markings than soccer fields, which means our computer vision techniques calibrated the camera position quicker and more accurately than our previous, fully manual approach.
- The machine learning model we used, ICON, worked even with football helmets and padding, but it required more modeling by hand in post-production to ensure the final 3D model accurately represented the action.
- Compared to soccer, there were more players in each photograph and more overlap of their positions, so we focused on modeling the key athletes in a given play instead of every athlete.
Camera alignment was adjusted using regulation field lines as guidance.
Publishing WebGL stories this quickly wouldn’t have been possible without Threebird, our collaborative, web-based visual tool for designing 3D interactive web graphics. By bringing the editing of animations, shaders, camera paths and more into the browser, we were able to compose, edit and visualize the interactive experience all at the same time. This reduced the amount of times we needed to export files from 3D authoring software, while maintaining compatibility with the tools we use to produce 3D assets.
Under the hood in Threebird, our collaborative, web-based visual tool for designing 3D interactive web graphics.
About This Work
R&D has been exploring the use of computer vision to analyze form and movement in sport for several years. Some other experiments in this area are described in previous posts on estimating 3D poses from athletes and using computer vision to extract speed data from photographs.
Our Super Bowl interactive was created and produced by Weiyi Cai, Or Fleisher, Karthik Patanjali, Kenan Davis, Malika Khurana, Mark McKeague, Yuliya Parshina-Kottas, Pablo Robles, Bedel Saget and Jenny Vrentas.
The camera calibration tool was created and developed by Or Fleisher, Don McCurdy, and Mint Boonyapanachoti.
The player models were created using a machine learning model called ICON, which was created by Yuliang Xiu, Jinlong Yang, Dimitrios Tzionas, Michael J. Black and the Max Planck Institute. The models were refined manually using photographs as reference.
The same process used during the Super Bowl was used previously in our World Cup coverage:
- How Pulisic Crafted the U.S. Goal in Its World Cup Opener
- Belgium’s Long-Ball Goal Sinks a Determined Canada
- Germany’s Late Equalizer Revives Its World Cup Hopes
- Watch Christian Pulisic Send the U.S. Past Iran and Into the Knockout Rounds
- Richarlison, Messi and Pulisic: Three Stunning Goals Frozen in Time
If you’re working on similar problems or are interested in working together, reach out to rd@nytimes.com.