[PYTHON] Motion analysis of figure skating with VideoPose 3D

background

Recently, I was watching figure skating on TV, and I thought that the trajectory at the time of jumping was visualized three-dimensionally and the number of revolutions could not be measured from the image, so I tried to analyze it. After a light survey, I found the following article. A splendid encounter between a former figure skater and deep learning: Challenge of a graduate student who wants to help the player

CNN + SVM is used to determine whether or not there is insufficient rotation. Recently, I suddenly remembered that there is a 3D version of OpenPose in the attitude estimation area. I wondered if it would be possible to count the number of rotations directly from the results of 3D attitude estimation. When I surveyed about 3D attitude estimation, I found a repository called VideoPose3D of FAIR. Immediately adopted because there was an example of figure skating in the demo. VideoPose3D(Facebook AI Research)

environment

• Google Colabolatory
  • PyTorch
  • Detectron2

Execute

Regarding the introduction of the execution environment, refer to the following because it is summarized in the notebook (when using it, it is necessary to change the git clone statement and the path surroundings). When executing with colab, I changed the loading part of coco's pkl file in inference/infer_video_d2.py and placed coco's pkl directly in the same hierarchy (By default, it was read from the web, but why? Or failed to read).

I used a video of Kihira's quadruple salchow There were twists and turns, such as cropping to eliminate telops and converting from 30 fps to 50 fps, but as a result, fairly good estimation results were obtained (the above figure is displayed at 10 fps). What is worrisome here is the number of revolutions. From the viewpoint in the above figure, it is difficult to understand whether the joints are correctly obtained at the moment of jumping. Therefore, only the waist (Figs. 2 and 5 above) is plotted from the viewpoint directly above. Looking at the above figure, the red and black lines are not interchanged, only vibrating at the moment of jumping. After all, the rotation at the moment of jump (both sides of the body) is not estimated correctly. Therefore, it seems difficult to count the number of revolutions from this information.

However, what was interesting was the moment of rotation. Looking at only both arms and shoulders (joint ID = 9 to 17) from above, it can be seen that the moment of jump is closer to the center. Since this 3D plot is plotted around the pelvis (joint ID = 1), it seems that the deviation of the rotation axis can be calculated from this information. VideoPose3D itself was researched about two years ago, so I think it has already been done, but it seems to be a source of various research.

A set of programs used this time

https://github.com/kuboyoo/VideoPose3D