Skip to main content

Teaching a Computer to Watch Tennis

·3 min read
Trevor McCormick
Trevor McCormick
Data Product @ Disney+

I'm going to try to teach a computer to chart tennis matches from broadcast video. This devlog is where I write down what happens — including the parts that don't work.

The itch​

The Match Charting Project is one of my favorite datasets in sports. Volunteers watch professional tennis matches and hand-code every single shot: serve direction, rally length, shot types, court positions, error types. Thousands of matches, charted stroke by stroke, freely available. Nearly every deep tennis stat you've ever read traces back to this kind of shot-level data.

The catch is in the word hand-code. Charting one match takes hours of a volunteer's full attention. That means coverage is uneven — big names and big tournaments get charted, everyone else mostly doesn't. The dataset's bottleneck isn't enthusiasm or tooling. It's human hours.

I've wanted to attack that bottleneck for years, and until recently the honest assessment was that the computer vision was the hard part — you'd need custom-trained detectors for the ball, the players, the court, all brittle, all a research project in themselves before you ever got to the actual charting logic.

What changed​

In late 2025 Meta released SAM 3, the third generation of their Segment Anything Model, with open weights. The headline capability: you give it video and a text prompt — literally the string "tennis ball" — and it finds every matching object, masks it pixel-perfectly, and tracks it across frames.

No custom training. No hand-labeled dataset. The part of this project that used to be a research career is now, at least in theory, an API call.

That "in theory" is what this project is about. Broadcast tennis is a hostile environment for vision models: the ball is a handful of pixels moving at 200+ km/h with motion blur, players occlude each other, cameras cut between angles, replays interleave with live points. I don't know where SAM 3 breaks on this footage. Finding out is the fun part.

The plan​

The destination is a pipeline that takes broadcast video in and emits Match Charting Project notation out. Between here and there:

  • M0 — Track one rally. Run SAM 3 on a single broadcast rally clip and get clean tracks for the ball and both players. This is the feasibility gate for everything else.
  • M1 — Find the court. Detect court lines, compute the homography, and convert pixel positions into real court coordinates.
  • M2 — Detect hits and bounces. A tennis ball's trajectory is a series of smooth arcs interrupted by discontinuities. Every discontinuity is a hit or a bounce. That's the raw material of charting.
  • M3 — Chart points. Segment rallies, classify shots, and emit MCP notation.
  • M4 — Take the test. Chart a match that a human volunteer already charted, diff the two, and publish the agreement rate — whatever it is.

M4 keeps me honest. It's very easy to demo a highlight clip with pretty masks and claim victory. Agreeing with a careful human across a full match is a different bar, and the number will be public even if it's embarrassing.

Why write it down​

Because I've learned the hard way that projects I don't write about are projects I quietly abandon, and because the failure modes are usually more interesting than the wins. Expect short entries, real artifacts — clips, trajectories, wrong turns — and no authority voice. I'm figuring this out as I go.

First up: M0. Can SAM 3 actually hold onto a tennis ball at broadcast speed? Next entry will have the answer.