Teaching a Computer to Watch Tennis
1. Teaching a Computer to Watch Tennis
Kicking off Court Vision: an attempt to auto-chart tennis matches from broadcast video with SAM 3, built in public.
Build in public
Court Vision
Teaching a computer to chart tennis matches from broadcast video — ball tracking, shot detection, and point-by-point data with SAM 3.
About the projectDevlog
Working notes, written as the work happens.
2. M0: Can SAM 3 Hold Onto a Tennis Ball?
Four runs, two dead ends, and one undocumented response format later: SAM 3 tracks the game ball through a full broadcast rally, for about sixty cents.
3. M1: From Pixels to Meters
The court found itself: a median frame, 48 Hough segments, and four corners put the M0 ball track on a real court diagram — no clicks, no API calls, no dollars.
4. M2: The Court Doesn't Return Serve
The hit/bounce detector was wrong twice before it was right thirteen times: every event in the rally found, classified, and frame-verified. The pipeline's first proto-chart.
5. M3: Who Hit That?
Two box prompts came back as one blob, the line judge deleted himself from the data, and the rally learned its letters — every forehand and backhand called and frame-verified.
6. M3: The Cutting Room
A 24-minute highlights reel, a homography moonlighting as a camera fingerprint, and sixty complete points extracted — serves included, courtesy of the broadcast editor.
7. M3: The Static Camera Isn't
Player tracking for zero dollars, a broadcast camera that turns out to pan mid-point, ghosts in the clean plates, ballkids impersonating players — and forty serves called with the confidence to refuse the other twenty.
8. M3: The Toss Is the Bootstrap
M0's one-manual-click problem solves itself, the score bug impersonates a tennis ball, the pre-serve dribble charts itself as rally — and the pipeline writes its first complete point, asterisks included.
9. M3: The Ball Gets a Vote
Proximity dies twice, the homography turns out to have been lying about airborne balls since M1, a weighted election picks every striker — and the ball wins an argument against the serve detector, on camera.
10. T1: A Test Set With Answers in the Back
The dev reel gets demoted to training data, Jeff Sackmann's Match Charting Project supplies 225 hand-charted answers for the same court — and the homography refuses to transfer, failing plausibly three times under floodlights before four manual corners and some arithmetic settle it.
11. T1: Right Hardware, Wrong Sign
The frozen pipeline meets a human chart: server ends and rally lengths hold up, exact letters crater at 2/11 — and 9 of 11 are perfect mirrors, because a right-hander assumption met two lefties and inverted almost every call.
12. The $0 Specialist and the Load-Bearing Bug
A landscape review finds tennis-ball tracking is a solved, free, open problem and nobody anywhere charts from broadcast video — so SAM gets fired from ball duty by a 6 MB specialist, which promptly regresses the best metric on the board, because SAM's habit of losing the ball had been quietly detecting the end of every point.
13. The Score Bug Is the Point ID
The benchmark goes adversarial — clay, grass, both tours — and the headline failure is upstream of every metric: segment ≠ point, in opposite directions per broadcaster. The detector was on screen the whole time: the score bug changes exactly at point boundaries, and reading it takes alignment to 108/108.
14. Within One Token of a Human
The event detector gets inverted — net crossings are the rally skeleton, and raw crossings+1 beat the whole old detector before a line was written — then the report card gets a north star: a point is accepted at ≤1 token edit from the human chart. First reading 2.2%. The decomposition names direction digits as 39% of the edit budget, and the fix is a coordinate bug: the compass was mirrored.
The road to auto-charting
Each milestone is a devlog entry. Here's where things stand.
- ✓M0 — Track ball and players through one rally with SAM 3
- ✓M1 — Court keypoints → homography → real court coordinates
- ✓M2 — Detect hits and bounces from ball trajectory
- ●M3 — Rally segmentation → shot sequences → MCP notation
- ○M4 — Validate against a human-charted Match Charting Project match