Object detection on a moving background with registered subtraction

A scrolling striped backdrop turns plain frame differencing into confetti: cv2.absdiff flags the fish and every stripe edge it crosses. The fix is to stop treating the pattern as noise and treat it as a known motion instead. Measure the backdrop's shift in pixels per frame, then subtract a background image re-registered by that shift, mask the predicted stripe bands, or collapse the pattern with a temporal median. This guide gives you the four numbers to measure, a worked 30 fps example with every step of the arithmetic, the three pipelines as ordered steps, and the optical moves that make the fish separable before any code runs.

Why a moving background breaks frame differencing

Frame differencing answers one question: which pixels changed intensity since the last frame? Over a static backdrop the answer is the swept region around the moving object and nothing else. Over a scrolling stripe pattern every stripe edge sweeps too, and the answer becomes two overlapping signals.

The stripe signal is deterministic, not noise. If the backdrop advances n pixels per frame, the difference between consecutive frames shows vertical bands of width n: a white-to-black band and a black-to-white band, alternating, one pair per pattern period. The whole band train advances n pixels every frame, and neighbouring bands sit one period P apart. You can draw next frame's stripe difference before the frame arrives.

The object signal is a swept blob along the object's path. Both signals reach connected-components analysis looking alike, which is exactly why the centroid tracker locks onto stripes. The three methods below separate them by registration, by geometry, or by time.

Three fixes and when each wins

Default to registered subtraction: one frame of latency, no assumption beyond sideways motion, and the reference image doubles as your segmentation background. Use the band mask when the drive is speed-locked and you want the cheapest pipeline. Use the median when no clean reference frame exists or the speed jitters. If you control the cell itself, an optical move later in this guide may remove the problem outright.

The numbers to measure first

Four numbers drive every method:

• m, millimetres per pixel, from a ruler placed in the object plane: m = field width ÷ sensor width in pixels.
• V, backdrop speed in mm/s, from a stopwatch or the drive's step rate.
• f, frame rate in frames per second.
• P, the stripe period: p millimetres on the backdrop, P = p ÷ m in pixels.

From these:

n is the number that decides everything downstream: it is the width of each stripe band in the difference image and the shift your registration must reproduce. t_exp takes the fastest mover in the scene, which is the backdrop here, not the fish. Resolution is rarely the binding constraint: a 25 mm fish at 20 pixels across only needs m ≤ 1.25 mm/px. Run your field width through the calculator below; exposure and shift, not pixel count, are what this task lives on.

Worked example: a fish tank at 30 fps

The scene: a 200 mm-wide field, a striped backdrop scrolling at 100 mm/s, 30 fps, a 20 mm stripe period, a 25 mm fish. The arithmetic, step by step:

• m = 200 mm ÷ 4000 px = 0.05 mm/px
• n = 100 mm/s ÷ (30 fps × 0.05 mm/px) = 100 ÷ 1.5 = 66.7 px per frame, worked at 67
• P = 20 mm ÷ 0.05 mm/px = 400 px
• Difference bands: 67 px wide, alternating white-to-black and black-to-white, repeating every 400 px
• t_exp ≤ 0.05 mm/px ÷ 100 mm/s = 0.0005 s = 0.5 ms
• Median window N ≥ 400 ÷ 67 + 1 = 6.97, so 7 frames: 7 × 4000 × 3000 bytes = 84 MB of mono 8-bit frames
• Median latency ≈ 7 ÷ 2 ÷ 30 ≈ 0.12 s; a fish crossing 5 px per frame smears 35 px across the window
• The fish spans 25 mm ÷ 0.05 mm/px = 500 px, comfortable for a centroid

A USB3 camera reading 4000 × 3000 at 30 fps carries this example as written; the LS-CMR-0166 is a 12 MP, 1/1.7" rolling-shutter model in that class. At 67 px of shift per frame, the row-by-row skew a rolling shutter adds is a few pixels and sits inside the band margin.

Raise the backdrop to several hundred mm/s and rolling-shutter skew bends the bands inside a single frame faster than a 5 px margin absorbs; that is global-shutter territory. The MV-CH120-11UC is a 12 MP USB3 camera with a global shutter, sold as a high-speed version.

Method 1: subtract a background registered by the measured shift

This is the default pipeline. Five steps:

1. Grab a reference frame B0 with the cell empty. If the fish cannot be removed on demand, take the per-pixel median of 20-50 frames captured while it swam elsewhere; the median erases it.
2. Measure the shift per frame: convert the previous and current frames to float32 grayscale and run phase correlation. On a striped field the pattern dominates the correlation surface, so the estimate holds even with the fish in view.
3. Accumulate: s_k = s_(k−1) + (dx, dy). Warp B0 by s_k with cv2.warpAffine as a pure translation, interpolating cubically for the sub-pixel part.
4. absdiff the live frame against the warped reference, threshold with Otsu, then open with a 3×3 kernel. Sub-pixel registration error leaves residual bands 1-3 px wide and the opening erases them, while a fish blob hundreds of pixels wide survives. Where illumination varies across the tank, local mean adaptive thresholding beats a single global cut.
5. Take the largest connected component, read its centroid from the image moments, and append it to the trajectory.

Prove the pipeline on the empty recording before the real one: with no fish in view, the foreground count must sit near zero, or the registration is wrong, not the fish.

The model is pure translation. Any direction in the image plane works, the camera must stay fixed, and the pattern must keep its distance from the lens. An encoder-driven backdrop makes step 2 trivial: grab one frame per encoder step and the shift is a known constant; the hardware trigger grab loop in Python shows that acquisition pattern. Frame differencing grows artifact blobs beyond stripes too; the image subtraction false defects guide covers the contour artifacts that the morphology step above cleans up.

Method 2: mask the predicted stripe bands

Is the drive speed-locked? Skip the background image entirely.

1. Measure n once on an empty recording: the median estimate over 100 frame pairs of phase correlation. In the worked example, 67 px.
2. Read the stripe-edge x positions at frame 0 from a column-sum profile of the reference frame.
3. At frame k, every edge sits k × n pixels from its frame-0 position. Build a mask of vertical bands of width n plus a 3-5 px margin at those positions, wrapping positions modulo P, and merge all bands into one mask.
4. Foreground = difference mask AND NOT stripe mask, then morphology, largest component, centroid as before.

The margin absorbs speed jitter, and the decision is numerical: take the standard deviation of the measured shift across 100 frame pairs on the empty cell. In the worked example, a 5 px margin absorbs a spread of 2 px comfortably. If your spread exceeds 2 px, drop the mask and register per frame instead, because widening margins eventually eat the fish's own difference lobes.

The mask costs no stored background and no float math, which makes it the natural fit on embedded targets. Its honesty rests on the constant-speed assumption, so an encoder-gated grab, as in Method 1, is what keeps it true.

Method 3: a temporal median over one scroll period

No reference frame and jittery speed? Build a rolling median over the last N frames and segment the median image itself.

The window must cover one full scroll period. N ≥ P/n + 1 frames guarantees every pixel spends part of the window black and part white, so its median collapses to mid-grey. The fish occupies its pixels for the whole window, keeps its grey level, and stands as the only blob on a uniform field — no phase alignment needed, which is exactly why this method tolerates jitter that breaks the band mask.

Costs at N = 7 in the worked example: 84 MB of memory, about 0.12 s of latency, and smear equal to the fish's own travel across the window — a fish crossing 5 px per frame smears 35 px on a 500-px body. For trajectory plots that is acceptable; for a fast reject gate it is not.

Optics: separate the fish before writing code

Every pipeline above gets easier if the sensor sees less of the stripes. Three moves, cheapest first.

Defocus the backdrop. Place the stripes 50-100 mm farther from the lens than the fish plane and open the aperture until the backdrop sits beyond the depth-of-field limit: the stripes wash out to grey while the fish stays sharp. Treat it as a depth-of-field problem with a target blur of 3-5 px at the backdrop plane — the depth of field calculator does the f-number arithmetic for your working distance and pixel size.

Cross the polarizer if the pattern comes from a display behind the tank. A liquid-crystal panel emits linearly polarized light, so a polarizer crossed to its axis cuts the pattern hard, while light scattered by the fish arrives partially depolarized and survives. Rotate the filter for extinction, then re-check the fish's contrast.

Control the light instead. Light the fish plane and keep the backdrop unlit: exposure set by the lit fish drops the stripes toward black, and the task becomes bright-object segmentation. The bright lamp this takes is also what buys a 0.5 ms exposure at a working f-number.

Stripes are black and white, so a mono sensor turns every photon into signal instead of colour redundancy and halves the data. The MV-CS200-10UM is a 20 MP mono USB3.0 option for small targets. At the example's 0.05 mm/px a 25 mm fish spans 500 px; halve the field width and that doubles.

Checks before you commit

• Empty-cell gate: run the full pipeline on 200 fish-free frames. In the worked example, foreground below 0.1% of the frame — about 12,000 pixels on a 4000 × 3000 sensor — is the pass mark. A fail means registration or masking is wrong, not the fish.
• Shift spread: the standard deviation of measured n across 100 frame pairs should stay at or under 2 px in the example; above that, register per frame.
• Contrast audit: compare the fish's grey level against both stripe states. Within about 20 counts of either state, no algorithm will recover it — fix the optics.
• Latency budget: about 0.12 s of median latency suits a trajectory plot and fails a reject gate; decide which one this tracker feeds before choosing a method.
• Calibrate m with a ruler in the object plane, never from the lens's nominal field: parallax between the fish plane and the backdrop plane shifts m and n together.

When the usual answer is wrong

Adaptive per-pixel models look tempting because a multimodal background model can hold both stripe states as background and let the pattern fade away on its own. The costs are quiet: the model wants a fish-free warm-up, any fish that pauses longer than the adaptation horizon becomes background, and the learned multi-state model leaves you no grey reference to segment against. Registered subtraction keeps one frame of latency, zero training, and an image you can inspect.

A learned detector handles overlapping fish better than blob logic, but it needs labels drawn on your own striped footage, and the stripes were never the hard part. Start deterministic; reach for a detector when overlap defeats connected components.

If the camera itself moves across a static scene, that is the inverse problem with different mathematics — the moving object detection with a moving camera guide covers depth-based approaches to it.

Budget exposure and shift first, with t_exp ≤ m/V and an n small enough to mask, then choose the camera on shutter and frame rate rather than resolution. The camera finder filters by shutter, resolution and interface, the area scan cameras category lists the series, and the configurator below walks the whole chain. For a second opinion on your own numbers, contact the applications team; more application write-ups sit on the applications topic page.