Court Vision in Plain English

The devlog assumes you know tennis cold and computer vision not at all. That's the right assumption, but it means the posts lean on a handful of ideas that deserve to be explained properly once instead of badly fifteen times. This page is that once. Every entry in the series links back here; nothing below needs more math than a chair umpire does in their head.

Pixels and frames​

Video is a flipbook. A broadcast is 25 to 30 still pictures per second — each picture is a frame — and each frame is a grid of colored dots called pixels. A 720p frame is 1280 dots wide by 720 tall, about 900,000 pixels, and at broadcast distance the game ball occupies maybe eight of them. That's the raw material: not "a rally," just a very fast stack of photographs.

Two consequences run through the whole series. First, when a post says "frame 240," it means the 240th photograph — eight seconds into a 30fps clip. Frames are the pipeline's clock. Second, there is no motion anywhere in the data. Motion is something you infer by comparing one photograph to the next — like a line judge who never gets to watch the ball fly, only to study a strip of stills and work out what must have happened between them. Everything the pipeline does — tracking, hit detection, serve calls — is built on top of that inference. When a fast-moving ball travels a long way between two consecutive photographs, or smears into a blur inside one, the inference gets hard. That's most of why broadcast tennis is a hostile environment for this stuff.

Homography: the TV picture vs the court diagram​

The broadcast camera looks at the court from a raised angle behind the baseline, so the far half of the court appears tiny and squashed and the near half looms. A charting diagram is the opposite: top-down, to scale, every meter equal. A homography is the conversion rule between the two — a recipe that turns "the ball is at this spot on your TV screen" into "the ball is two feet inside the baseline."

You already run this conversion by instinct: watch a match on TV and you can mark, on a paper court diagram, roughly where a ball landed. The software needs the recipe written down, and it gets it from the court's own painted lines — match the lines on screen to the official dimensions (10.97 by 23.77 meters) and the conversion falls out. Four corners turn out to be enough.

The catch that echoes through the whole series: the recipe only works for things on the ground. It maps the court's surface, not the air above it. An airborne ball's screen position converts to where its shadow would be, not where the ball is — which is how a perfectly fit conversion can report a ball "18 meters past the baseline" mid-flight. Feet are on the ground, so player positions convert truthfully. The ball only tells the truth at the moment it bounces.

Background subtraction: compare everything to an empty court​

Take a photo of the court with nobody on it. Now compare each broadcast frame against that photo, pixel by pixel: whatever's different is something that moved in — a player, the ball, a ballkid. That's background subtraction, and it's the series' zero-dollar player detector.

The empty-court photo comes free, via a trick worth admiring: for each pixel, take the median value across all the frames of a clip. Anything that moves — players, ball — is only ever briefly at any given pixel, so the median votes it out of existence. The court lines, which never move, stay razor sharp. The result is a photograph of a court where nobody is playing tennis, distilled from footage where two people very much are.

The failure modes are exactly what you'd guess. Anyone who stands still long enough becomes part of the "empty court" — a line judge at parade rest literally erases himself from the data. Anyone who moves near the court reads as a player — ballkids, umpires, a flapping banner. And the whole comparison assumes the camera holds still; a mid-point pan smears the empty-court photo like a shaken Etch A Sketch, so frames have to be aligned to each other before anything else happens.

Detection vs tracking​

Two different ways to follow a ball, with opposite personalities.

Detection looks at every frame fresh: "is there a ball in this picture, and where?" Thirty times a second, no memory, no idea what happened in the previous frame. Tracking is the opposite: lock onto one specific object and follow it from frame to frame, carrying memory forward — the way you watch a rally without blinking.

Tracking gives you continuity: it knows the ball in frame 100 is the same ball as in frame 99. But memory is a liability. Blink at the wrong moment — occlusion, motion blur, a camera cut — and a tracker can lose its object or latch onto the wrong one, and everything after that is confidently wrong. A detector can't lose the plot because it never had a plot; it just answers the same question again next frame. The price is that it doesn't know which ball is which, or that the "ball" it found this frame is the one from last frame.

The series uses both. The early milestones tracked (SAM following one prompted ball); the later ones detect (a specialist finding the ball fresh every frame) and stitch the per-frame answers into a trajectory afterward. The switch — and why it happened — is cv-12.

Ball-finders: tiny specialists vs promptable generalists​

Two very different kinds of model appear in this series, and the contrast matters.

SAM 3 is a generalist — a huge model trained on everything, that you prompt. Give it video plus the words "tennis ball," or a box drawn around one specific ball, and it finds, outlines, and follows the object. Enormous range: the same model segments players, courts, umpires, anything you can name or point at. The costs: it needs to be told what to follow, it bills by the frame on a hosted API, and as a tracker it can lose the object and keep going as if it hadn't.

WASB is a specialist — a 6-megabyte network that has only ever looked at tennis balls. For each frame it outputs a heatmap: a picture the same shape as the frame where brightness means "probability the ball is here." Take the hottest spot, done. It answers exactly one question — "where's the ball in this picture?" — about thirty times a second, fresh each time. It never gets lost, because it never has to remember anything. It can't be asked anything else, and it doesn't need to be.

The tennis version: the generalist is a brilliant all-court player you have to coach before every point; the specialist is a line judge who has watched one line their entire career and is nearly unbeatable on it.

Reading events out of a trajectory​

Once the ball's positions are strung into a path over time, the charting events live in the path's shape. A hit reverses the ball's direction of travel — the ball goes back where it came from, as sharp a signature as a signal gets. A bounce reverses nothing: the ball keeps traveling the same way, just lower and slower. The series' shorthand for the distinction: the court doesn't return serve. After a hit the ball leaves fast; after a bounce, its speed collapses.

The camera angle complicates this (see the homography note — the software really places the ball's ground shadow, and an airborne ball's shadow moves at nonsense speeds), and the later posts rebuild the detector around a blunter piece of tennis sense: a live rally sends the ball across the net every shot. Count the net crossings and you have the rally's skeleton — each crossing-to-crossing slice contains exactly one hit, and the delicate cusp-hunting is demoted to locating the hit inside its slice rather than deciding whether it exists. Hits are inferred from geometry; nobody ever sees a racket touch a ball.

The score bug is metadata​

The score bug — the little scoreboard graphic in the corner — is not part of the scene. It's an overlay drawn on top of the picture, so a camera pan never moves it, lighting never changes it, and it behaves with a discipline nothing on the court can match: it is pixel-for-pixel constant for the duration of a point, and it changes at exactly one kind of moment — when a point ends.

That discipline makes it two tools. First, a point ID: if two stretches of footage show the same unchanged score bug, they are fragments of the same point, even if the director cut away in between; if the bug changes mid-stretch, that stretch holds two points. Second, a join key: read the score off the bug and you can look up the exact same point in a human-charted record of the match, which is what makes machine-vs-human grading possible point by point.

It's the least glamorous pixels on the screen and the most reliable witness in the building — the one thing every broadcaster in every country agrees to keep on camera at all times. The pipeline learning to read it is cv-13.

Ground truth and benchmarks​

Ground truth is an answer key created independently of the system being graded. For this project it's the Match Charting Project: volunteers who watched real matches and hand-coded every shot. When a post says a match "has answers in the back," it means a human already charted it, point by point, in the exact notation the pipeline is trying to produce.

Why it matters: for the first stretch of the series, the person tuning the pipeline was also the person grading it — checking his own frame calls with his own eyes. That's calling your own lines. It's fine for practice and worthless as evidence, because the same eyes that tuned the thresholds will forgive them. A benchmark is the discipline that fixes it: freeze the pipeline (no constants change, written down before the run), point it at matches somebody else charted, count the agreement, publish the number even when it's embarrassing.

One more rule rides along: anything tuned on a match can't be graded on it. The dev match is a practice court; the test matches are tournaments. The moment a test match gets used to fix a bug, it quietly becomes practice too — which is why the series keeps having to go find fresh matches.

Precision, coverage, calibration​

The finished tool doesn't just draft a chart — it flags each drafted point HIGH or LOW confidence. Three words do the work of describing whether those flags mean anything.

Coverage is how often the system is willing to say HIGH — the share of points it stamps. Precision is how often it's right when it says HIGH. The two trade off exactly like line-calling does: a judge who only calls the balls that land a foot out is nearly always right and nearly useless; one who calls everything close is helpful and often wrong. A confidence flag is only worth something if you know both numbers.

Calibration is how the flags get honest: take points that humans have already graded, look at the signals the pipeline computes about itself — did the serve call commit? does the ball track have holes? did the striker votes conflict? — and fit those signals to the actual outcomes. Then the crucial hygiene, called leave-one-match-out: each match gets flagged by a model that never saw that match during fitting. The system never grades its own homework. When cv-15 says HIGH means "start from the draft" with 93% reliability, that 93% survived exactly this test — and the fancier tier that didn't survive it is reported dead rather than quietly deleted.

Token edit distance​

The acceptance metric — the number the whole project is climbing toward.

A finished chart of a point is a string of tokens: the serve and its zone, then each shot's letter and direction, then the ending. Both charts — the machine's and the human's — get written as token lists, and the question asked is brutally practical: how many tokens would an editor have to fix — change, insert, or delete — to turn the machine's chart into the human's? That count is the token edit distance. Zero means the machine matched the volunteer exactly. A point is accepted when the distance is at most one — the chart a charter would sign after fixing at most one thing.

Two properties make it the honest metric. It grades the deliverable, not the components — server calls, rally lengths, and letters can all inch upward while no single point is actually right, and this number refuses to be fooled by that. And a refusal (?) costs exactly what an error costs, because the editor has to fix both: a shrug is an edit too. The pipeline doesn't get credit for hedging. When cv-14 reports 2.2%, that's the fraction of points a human would sign — the honest distance to the mountain top, measured in the only unit a volunteer charter cares about: corrections.