Tennis Shot Classifier
Turning tennis broadcasts into body-pose sequences and classifying each shot.
I wanted to see how far you could get classifying tennis shots from ordinary broadcast footage, with no special camera setup and unknown camera parameters. The pipeline takes a full recording of a match as input, segments every single shot, converts the short clip into a sequence of poses, and uses them to learn to distinguish between the main tennis strikes. Solo project for the Sport Data Science course at Leiden University.
How it works
It automatically finds the shots by listening rather than looking. The audio has a sharp spike at the moment the racket hits the ball, so it detects those transient onsets using librosa and cuts a short clip around each one. Next, YOLOv8 finds and crops the player hitting the ball, and then a small labelling tool is used to manually label each clip as one of eight different tennis strikes. However, the three most common shots are the serve, forehand, and backhand, and after going through the labelling process for many hours, covering four different broadcast recordings, I did not end up with enough data for the remaining five classes, so I decided to omit them from training.
I then run the cropped shot clips through Sports2D with RTMPose to get 2D keypoints for every frame, followed by centering on and scaling them from the hips so the player on the far side of the court is indistinguishable from the player on the near side. Finally I augment the pose sequences by mirroring them across the vertical axis. The classifier is a two layer bidirectional LSTM that reads the pose sequence and predicts the shot.
I intentionally chose matches between players with rarer shot styles, so as to test the model’s generalization capabilities. For example, Roger Federer is in two of the broadcasts chosen specifically to test if the model can distinguish between a standard two-handed backhand and his one-handed backhand, and Rafael Nadal is in one of the broadcasts to test how the model handles left-handed players.
Results
Trained on roughly 3000 labelled shots from four hard court grand slam matches, the model reaches 98% on a held out test set of 100 clips. That number comes with a caveat: the test clips come from the same matches as the training clips, so it says more about the concept working than about how the model would generalize across unseen settings and players.