[PYTHON] Motion analysis of figure skating with VideoPose 3D

background

Recently, I was watching figure skating on TV, and I thought that the trajectory at the time of jumping was visualized three-dimensionally and the number of revolutions could not be measured from the image, so I tried to analyze it. After a light survey, I found the following article. A splendid encounter between a former figure skater and deep learning: Challenge of a graduate student who wants to help the player

CNN + SVM is used to determine whether or not there is insufficient rotation. Recently, I suddenly remembered that there is a 3D version of OpenPose in the attitude estimation area. I wondered if it would be possible to count the number of rotations directly from the results of 3D attitude estimation. When I surveyed about 3D attitude estimation, I found a repository called VideoPose3D of FAIR. Immediately adopted because there was an example of figure skating in the demo. VideoPose3D(Facebook AI Research)

environment

Execute

Regarding the introduction of the execution environment, refer to the following because it is summarized in the notebook (when using it, it is necessary to change the git clone statement and the path surroundings). Open In Colab When executing with colab, I changed the loading part of coco's pkl file in inference/infer_video_d2.py and placed coco's pkl directly in the same hierarchy (By default, it was read from the web, but why? Or failed to read).

I used a video of Kihira's quadruple salchow output.gif There were twists and turns, such as cropping to eliminate telops and converting from 30 fps to 50 fps, but as a result, fairly good estimation results were obtained (the above figure is displayed at 10 fps). What is worrisome here is the number of revolutions. From the viewpoint in the above figure, it is difficult to understand whether the joints are correctly obtained at the moment of jumping. images.jpeg Therefore, only the waist (Figs. 2 and 5 above) is plotted from the viewpoint directly above. output_roll.gif Looking at the above figure, the red and black lines are not interchanged, only vibrating at the moment of jumping. After all, the rotation at the moment of jump (both sides of the body) is not estimated correctly. Therefore, it seems difficult to count the number of revolutions from this information.

output_shoulder.gif However, what was interesting was the moment of rotation. Looking at only both arms and shoulders (joint ID = 9 to 17) from above, it can be seen that the moment of jump is closer to the center. Since this 3D plot is plotted around the pelvis (joint ID = 1), it seems that the deviation of the rotation axis can be calculated from this information. VideoPose3D itself was researched about two years ago, so I think it has already been done, but it seems to be a source of various research.

A set of programs used this time

https://github.com/kuboyoo/VideoPose3D

Recommended Posts

Motion analysis of figure skating with VideoPose 3D
Equation of motion with sympy
3D skeleton structure analysis with Python
2D FEM stress analysis program with Python
Example of 3D skeleton analysis by Python
Sentiment analysis of tweets with deep learning
[Python] limit axis of 3D graph with Matplotlib
Solve A ~ D of yuki coder 247 with python
Static analysis of Python code with GitLab CI
Add information to the bottom of the figure with Matplotlib
[OpenCV / Python] I tried image analysis of cells with OpenCV
Calculate the regression coefficient of simple regression analysis with python
Challenge principal component analysis of text data with Python
Planar skeleton analysis with Python (4) Handling of forced displacement