Court Vision
The goal: point a computer at broadcast tennis video and get back a point-by-point chart of the match — every serve, every shot, every rally — in the notation of the Match Charting Project.
Why
The Match Charting Project is one of the best open datasets in sports: thousands of pro matches hand-charted shot by shot by volunteers. But it's bottlenecked by exactly that — humans watching matches and coding every shot by hand. A single match takes hours to chart. Coverage is thin outside the top players and big tournaments.
Computer vision keeps getting closer to removing that bottleneck. SAM 3, Meta's open-weights segmentation model released in late 2025, can find and track objects in video from a plain text prompt — "tennis ball", "tennis player" — no custom training required. That changes the feasibility math for auto-charting enough that it's worth finding out where it breaks.
How it would work
Milestones
| # | Milestone | Status |
|---|---|---|
| M0 | Track ball and players through one rally with SAM 3 | 🟡 In progress |
| M1 | Court keypoints → homography → real court coordinates | ⚪ Not started |
| M2 | Detect hits and bounces from ball trajectory | ⚪ Not started |
| M3 | Rally segmentation → shot sequences → MCP notation | ⚪ Not started |
| M4 | Validate against a human-charted MCP match | ⚪ Not started |
M4 is the honest test: chart a match the volunteers already charted, diff the output, and publish the agreement rate — whatever it turns out to be.
Following along
Everything gets written up in the devlog as it happens — including the dead ends. Subscribe via RSS if you want the updates.