Teaching a Computer to Watch Tennis
1. Teaching a Computer to Watch Tennis
Kicking off Court Vision: an attempt to auto-chart tennis matches from broadcast video with SAM 3, built in public.
Build in public
Court Vision
Teaching a computer to chart tennis matches from broadcast video — ball tracking, shot detection, and point-by-point data with SAM 3.
About the projectDevlog
Working notes, written as the work happens.
2. M0: Can SAM 3 Hold Onto a Tennis Ball?
Four runs, two dead ends, and one undocumented response format later: SAM 3 tracks the game ball through a full broadcast rally, for about sixty cents.
3. M1: From Pixels to Meters
The court found itself: a median frame, 48 Hough segments, and four corners put the M0 ball track on a real court diagram — no clicks, no API calls, no dollars.
4. M2: The Court Doesn't Return Serve
The hit/bounce detector was wrong twice before it was right thirteen times: every event in the rally found, classified, and frame-verified. The pipeline's first proto-chart.
5. M3: Who Hit That?
Two box prompts came back as one blob, the line judge deleted himself from the data, and the rally learned its letters — every forehand and backhand called and frame-verified.
6. M3: The Cutting Room
A 24-minute highlights reel, a homography moonlighting as a camera fingerprint, and sixty complete points extracted — serves included, courtesy of the broadcast editor.
7. M3: The Static Camera Isn't
Player tracking for zero dollars, a broadcast camera that turns out to pan mid-point, ghosts in the clean plates, ballkids impersonating players — and forty serves called with the confidence to refuse the other twenty.
8. M3: The Toss Is the Bootstrap
M0's one-manual-click problem solves itself, the score bug impersonates a tennis ball, the pre-serve dribble charts itself as rally — and the pipeline writes its first complete point, asterisks included.
9. M3: The Ball Gets a Vote
Proximity dies twice, the homography turns out to have been lying about airborne balls since M1, a weighted election picks every striker — and the ball wins an argument against the serve detector, on camera.
10. T1: A Test Set With Answers in the Back
The dev reel gets demoted to training data, Jeff Sackmann's Match Charting Project supplies 225 hand-charted answers for the same court — and the homography refuses to transfer, failing plausibly three times under floodlights before four manual corners and some arithmetic settle it.
11. T1: Right Hardware, Wrong Sign
The frozen pipeline meets a human chart: server ends and rally lengths hold up, exact letters crater at 2/11 — and 9 of 11 are perfect mirrors, because a right-hander assumption met two lefties and inverted almost every call.
12. The $0 Specialist and the Load-Bearing Bug
A landscape review finds tennis-ball tracking is a solved, free, open problem and nobody anywhere charts from broadcast video — so SAM gets fired from ball duty by a 6 MB specialist, which promptly regresses the best metric on the board, because SAM's habit of losing the ball had been quietly detecting the end of every point.
13. The Score Bug Is the Point ID
The benchmark goes adversarial — clay, grass, both tours — and the headline failure is upstream of every metric: segment ≠ point, in opposite directions per broadcaster. The detector was on screen the whole time: the score bug changes exactly at point boundaries, and reading it takes alignment to 108/108.
14. Within One Token of a Human
The event detector gets inverted — net crossings are the rally skeleton, and raw crossings+1 beat the whole old detector before a line was written — then the report card gets a north star: a point is accepted at ≤1 token edit from the human chart. First reading 2.2%. The decomposition names direction digits as 39% of the edit budget, and the fix is a coordinate bug: the compass was mirrored.
15. The Draft Grades Itself
The letters pass closes the component grind — a $12 buy-vs-build says SAM boxes are worth 59%→67% on one tree, and the default stays $0 — then fourteen posts of experiment sprawl consolidate into one package behind a byte-for-byte gate. On top ships the actual MVP: a two-tier confidence layer, calibrated leave-one-match-out, where the tier that failed is on the record and HIGH means start from the draft with 93% reliability.
16. The Benchmark Grows Up
Three finals land in one pass — Melbourne night, New York, indoor Turin — all through package configs, zero new scripts, and the ground truth grows 135 → 491. The recalibration corrects the headline from 93% to 88%, which is what more data is for. Then the weak fold gets its autopsy: a serve gate that structurally couldn't fire on Wimbledon's feed was passing every point unexamined, two whole-point gates replace the free pass, and the honest number climbs to 94% with disasters halved.
17. The 33rd Charter
Nothing in the pipeline changed this session — the project turns to face the people it was built for. The Match Charting Project ran on 32 volunteer charters in 2025, and its founder published a plea for more. Here are seven draft charts in the MCP's own columns — 508 points, 99 flagged trustworthy — published for a real charter to tear apart, with a stopwatch and three questions attached.
The road to auto-charting
Each milestone is a devlog entry. Here's where things stand.
- ✓M0 — Track ball and players through one rally with SAM 3
- ✓M1 — Court keypoints → homography → real court coordinates
- ✓M2 — Detect hits and bounces from ball trajectory
- ●M3 — Rally segmentation → shot sequences → MCP notation
- ○M4 — Validate against a human-charted Match Charting Project match