M0: Can SAM 3 Hold Onto a Tennis Ball?

That clip up top is SAM 3 riding the game ball through a full broadcast rally — the box is the model's output, not mine.

The kickoff post ended on a question: can SAM 3 actually hold onto a tennis ball at broadcast speed? Short answer: yes — with a catch. Getting there took four runs, two dead ends, and one undocumented response format.

The setup

I grabbed a 16-second broadcast rally — Zverev–Gasquet in Montreal — at 720p/30fps, 480 frames, pulled from a highlights compilation with yt-dlp. That sentence hides the first lesson, because the first clip I grabbed was unusable: highlights reels cut away from the broadcast angle after about five seconds. Rally, player close-up, crowd shot. Finding a continuous, single-angle rally took two tries. Clip sourcing is its own pipeline step. Compilations are a minefield of camera cuts; full-match VODs will be the real source going forward.

For compute I'm using fal.ai's hosted SAM 3 endpoints — sam-3/video, which returns a masked overlay video, and sam-3/video-rle, which returns the actual data: RLE masks and per-frame boxes. Half a cent per 16 frames. No GPUs to babysit.

Run 1: just ask for what you want

Text prompt: "tennis ball, tennis player", both endpoints.

The overlay came back looking great — for about eight seconds. Players segmented cleanly... along with the line judges and the ball kids, which is fair, because they're dressed like players. Then at around frame 240 the mask abruptly flipped to segmenting the entire court.

The logs had the clue: Completed chunk 2/2. fal processes video in chunks, and the second chunk apparently re-ran concept detection from scratch and latched onto something else entirely. Chunk boundaries are concept-drift boundaries.

The RLE endpoint with the same prompt did something different but no more useful: exactly one box per frame — a single giant region merging everything it matched — and every confidence score null.

Run 2: fine, just the ball

Text prompt: "tennis ball" alone, detection threshold 0.3. This returned 480 boxes — one per frame! Very exciting until I plotted the trajectory:

The tracked "ball," per frame. Long dead-flat plateaus. Tennis balls don't plateau.

A ball in play never sits still, so a flat line means the tracker is glued to something stationary. Rendering the boxes back onto the video explained it: the box is the union of every tennis ball in frame. Ball kids hold spare balls. Broadcast tennis has something like six balls in frame at any moment, and SAM 3 was dutifully boxing all of them as one concept.

The actual game ball. This is the thing we're asking a model to hold onto at 200 km/h.

The game ball — a smudge maybe eight pixels across — sat dwarfed inside a half-court-sized union box, contributing almost nothing to it. Open-vocabulary text prompts match ALL instances of a concept. "Tennis ball" is not "THE tennis ball."

Run 3: point at it

New hypothesis: skip text, prompt visually. Draw a box around the specific ball in a specific frame and let SAM 3 track that one object, SAM 2-style. I found the game ball by eye in frame 240 — pixel (622, 462) — and sent a box prompt anchored there.

470 of 480 frames came back with boxes. Then the first parse produced negative box widths.

That was the tell. After decoding frame 240's raw box and finding it exactly ball-sized at exactly the spot I'd prompted, it clicked: box-prompted responses come back as [cx, cy, w, h] — normalized center-plus-size — not the [x1, y1, x2, y2] corners you'd expect. As far as I can tell this is documented nowhere. Discovered exclusively by staring at impossible numbers.

Run 4: reparse, and there it is

Game ball centroid, box prompt at frame 240, propagated both directions.

Frames 0 through 290: ball-sized boxes tracing clean flight arcs. The y-position plot is the money chart — roughly three full baseline-to-baseline round trips, smooth parabolas with a sharp reversal at every hit. Those discontinuities are exactly the raw material M2 (hit and bounce detection) needs, and they're plainly visible in a first-pass track.

Two things I didn't expect. First, SAM 3 propagated the frame-240 prompt backwards to frame 0 as well as forwards — bidirectional tracking works on the hosted endpoint. Second, track death has a clean signature: when the rally ends around frame 300 and the ball leaves play, the box goes giant and freezes in place. A dead track looks like a frozen, oversized box. I dropped anything wider than 0.05 normalized and the garbage vanished.

Frame 225, with the tracked path drawn in. The box is on the game ball — not a ball kid's spare.

The verdict

M0: achieved — for the ball, which was the part I genuinely doubted. Total cost of the milestone, across all four runs including the failures: about $0.60 in API calls.

Two pieces of unfinished business, carried forward honestly rather than quietly dropped:

• Players. My M0 definition said "ball and both players." Run 1's player masks looked good until the chunk boundary ate them, and I stopped chasing players once the ball — the actual feasibility risk — worked. Players are bigger, slower, and hundreds of pixels tall; I expect box prompts to handle them, but that's a claim, not a result.
• The click. The pipeline currently needs a human to draw one box around one ball, once per rally. That's a bootstrap, not automation. Candidate fix when M3 forces the issue: text-prompt all the balls, then keep the one that moves.

Next up, M1: finding the court. A ball track in pixel coordinates means nothing until I can say where those pixels are on the court — line detection and a homography. A different kind of problem: less "will the model cooperate," more geometry homework.