Video stabilization with phase correlation and optical flow

Footage from a camera bolted to a vehicle, a robot arm or a vibrating fixture drifts and jitters, and a stabilizer that only watches a fixed region in the middle of the image quietly locks onto the scene instead of the camera. The fix is a two-stage estimator: phase correlation for the large whole-pixel shift between frames, then pyramidal Lucas-Kanade optical flow for the residual sub-pixel shift and rotation. The second half of the job is proving it works: you cannot score stabilization quality on a live stream, so you build a synthetic sequence with known ground truth and accumulate the variance of the estimation errors. This guide gives you the algorithm split, the parameter numbers, a worked variance calculation and the checks that catch the mistakes before they reach a production line.

Why a fixed region and phase correlation alone fail

Phase correlation aligns two frames by whitening their cross-power spectrum, and the peak of the inverse transform sits exactly at the integer-pixel translation between them. Three properties make it the right first stage for stabilization: it needs no starting guess, it tolerates global brightness changes, and its search range is the entire frame.

It also has three hard limits. It returns translation only, with no rotation term. The peak lands on the discrete grid of the transform, so the raw answer is whole pixels. And a region fixed in image coordinates does not track the camera: on a moving platform the content passing through your centre window changes every frame, so the estimate mixes camera motion with scene change, and a foreground object crossing the window corrupts the peak outright.

What you actually want is the camera's global motion, one translation plus one small rotation per frame pair. That takes a second stage.

Two stages: global shift, then sub-pixel and rotation

Stage 1: global integer shift. Downscale both frames by two (a 4000 × 3000 frame becomes 2000 × 1500, about 4 times fewer pixels to transform) and run phase correlation across the overlapping area. The output (dx, dy) can be hundreds of pixels, which no differential method can follow from zero.

Stage 2: sub-pixel refinement and rotation. Warp frame 2 by (dx, dy) with bilinear interpolation, then run pyramidal Lucas-Kanade from a grid of textured patches. Lucas-Kanade is the standard differential flow method: it assumes a patch keeps its brightness while it moves and solves for the motion from the image gradients (Lucas–Kanade method).

Fit (dx, dy, θ) to the patch displacements by least squares, drop patches whose residual exceeds about 1 px, and refit. Settings that hold up on real footage: a 3-level pyramid, 15 × 15 to 21 × 21 patch windows, patches on a 40-60 px grid, keeping only patches with enough texture. A single 21 × 21 window converges for shifts up to roughly half its width, about 10 px, and each pyramid level extends that by a factor of two, so three levels reach roughly 80 px. The ±20 px of jitter left after stage 1 sits well inside that range. If host bandwidth is tight, crop the estimator's region at the source; ROI cropping before the host shows the same trick that trims the stream.

Keep the pan, remove the jitter

The raw global motion curve mixes two things: the intentional pan or scan you want to keep, and the vibration you want gone. Correct with (m - s), where m is the measured motion and s its exponential moving average, s(t) = α·m(t) + (1 - α)·s(t-1).

The split point follows from α. For small α the moving average is a low-pass with a cutoff near α × fps / (2π); at α = 0.1 and 30 fps that is about 0.5 Hz. Camera pans sweep below that line, while motor and structural vibration sits above it, from a few hertz upward. At 10 Hz the smoothed path retains only about 6% of the jitter amplitude, from α / √(α² + 4·sin²(π·f/fps)) = 0.1 / √(0.01 + 3.0) = 0.1 / 1.73, so the correction removes roughly 94% of it.

Score the stabilizer with synthetic ground truth

A live stream gives you no truth, so build the test. Take one still master image with texture like your target scene. In a loop, cut a tile the size of the window your estimator actually matches, say 1000 × 1000, with a random shift and rotation inside your expected range, here ±10 px and ±0.5°, sampling with bilinear interpolation. Nearest-neighbour crops snap the ground truth to whole pixels and hide every sub-pixel error you are trying to measure.

The master must be big enough. A 1000 × 1000 tile rotated by 0.5° needs a 1000 × (cos 0.5° + sin 0.5°) = 1008.7 px square to be cut from, and the ±10 px shift adds 20 px more, so each placement needs about a 1030 px square. On a 4000 × 3000 master that leaves roughly 2970 × 1970 px of valid placements. If the estimator matches full 4000 × 3000 frames instead, the master must grow to about 4060 × 3060 px, so either source a larger master or evaluate on the working window. Rotated crops are built exactly as in arbitrary-angle image rotation in machine vision.

Run the estimator on each crop the way production will run it frame to frame: match the crop against the previous crop, and accumulate every parameter's error, estimate minus truth. Test the chained mode on purpose; matching against the previous frame accumulates error over time, matching against the master does not, and real operation is chained. To exercise the estimator on realistic footage later, save camera frames to disk for offline testing and enrollment and replay them through the same harness.

One honest limit: this harness scores the estimator on static content, and moving foregrounds bias both stages. The FAQ below covers how to extend the test to them. More capture-side techniques like this live on the machine-vision systems topic page.

Worked example: variance of the shift and rotation errors

Ten trials from such a harness (the values are illustrative; your harness produces its own):

Shift errors: the sum is 0.0, so the mean bias is 0.00 px. The squared errors sum to 0.09 + 0.04 + 0.01 + 0.16 + 0.04 + 0.00 + 0.01 + 0.09 + 0.09 + 0.01 = 0.54 px², so the variance is 0.54 / 10 = 0.054 px² and the standard deviation is √0.054 = 0.23 px.

Rotation errors: the sum is again 0.000°, the squared errors sum to 6.16 × 10⁻⁴ deg², the variance is 6.16 × 10⁻⁵ deg², and σ = 0.008°.

Cross-check the two. A 0.008° rotation moves the frame edge, 2000 px from the centre, by 2000 × tan(0.008°) = 2000 × 1.40 × 10⁻⁴ = 0.28 px, the same order as the shift σ, so a worst-case corner lands near 0.5 px of total residual. At a scene scale of 2 mm/px, that is 0.46 mm of residual drift against ±20 mm of injected jitter.

Accuracy targets and how many trials

First, how well do you know σ itself? For Gaussian errors, the standard deviation estimated from N trials carries a relative uncertainty of about 1 / √(2(N-1)): 24% at 10 trials, 7% at 100, 5% at 200. Run 200 or more; the loop costs seconds on a modern processor.

Commit to a pass line before you tune anything. For monitoring video, residual jitter below about 1 px reads as steady on a normal display. For an overlay that must track a feature, size the residual against the measurement: with a ±2 px tracking tolerance, a σ of 0.4 px or less keeps the stabilizer from being the limiting factor, using tolerance / 5 as the budget. A workable pass rule: σ within target and mean bias within 0.1 px. A bias above that points at interpolation asymmetry, not noise.

Gate every frame on confidence. Use the phase-correlation peak height against the strongest side peak, calibrate the acceptance level on the harness, and hold the previous motion estimate below it. A bad correction is worse than no correction.

Size the stages to the real motion

Translate platform speed into pixels before choosing parameters. Take a camera on a vehicle at 50 km/h = 13.9 m/s, optics at 2 mm/px, 30 fps: the scene moves 13,900 / 30 = 463 mm per frame, which is 463 / 2 ≈ 232 px per frame. Stage 1 must own that shift; phase correlation returns it in one shot, while differential flow seeded at zero cannot follow it. After the warp, the remaining ±2 to ±5 px of jitter sits comfortably in stage 2's range.

Then budget exposure. Blur in pixels equals platform speed times exposure divided by the pixel footprint, so for 1 px of blur or less the exposure must stay under 2 mm ÷ 13,900 mm/s = 0.14 ms. At a 1 ms exposure the blur is 13.9 px, both stages bias, and no post-processing recovers it. Short exposures need light and contrast; the LED colour contrast guide covers the illumination side, and video exposure and frame rate explains why video modes change exposure behaviour.

Cameras and compute for the pipeline

On moving equipment the camera needs a global shutter. The MV-CU020-90GC is a 2.5 MP GigE camera with a global shutter, a fit for platform-mounted stabilization at moderate resolution.

When the platform moves slowly and you want maximum scene detail, the LS-CMR-0166 delivers 12 MP frames of 4000 × 3000 at 30 fps over USB 3, and the synthetic harness can run on stills of exactly that size.

The estimator itself runs at the edge next to the camera. The MV-VC2140-128G40-1T vision controller pairs an Intel N-series processor with 8 GB of memory, a 1 TB HDD and 2 USB 3 ports, so a USB 3 camera streams straight into it.

Browse the area scan cameras in the camera finder, configure a set in the camera configurator, or compare the vision controllers range for the compute side.

When the usual answer is wrong

If the part is fixed and the camera shakes, fix the mount first. No amount of software correction beats a rigid mechanical assembly, and a stabilizer on a static station only adds latency and interpolation.

If you measure from the video, do not resample the frames. Warping every frame blurs edges slightly; applying the inverse motion to the measured coordinates keeps the original pixels and the sensor's own uncertainty. The same sub-pixel discipline applies as in sub-pixel centroid estimation.

If the scene has little texture, sky or bare walls, the phase-correlation peak collapses and no threshold saves it. Add fiducial marks to the scene, or fuse the camera with an inertial unit; verifying how the IMU and camera frames relate is covered in IMU optical frame transforms.

If the lens zooms during the take, translation plus rotation is no longer the right model. Scale changes need a similarity transform, four parameters instead of three, fitted the same way from the patch displacements.

Pre-flight checklist

• Master image textured like the target scene; flat regions starve the correlation peak.
• Bilinear crops in the harness, never nearest-neighbour, or the sub-pixel truth is quantized away.
• Master margin checked with the shift-plus-rotation arithmetic before generating placements.
• 200 or more trials, so σ is known to about ±5%.
• Rotation range covering the worst mount flex, not just the nominal pan.
• Chained frame-to-frame mode tested, because production runs chained.
• Exposure budget met: at most 1 px of blur at top platform speed.
• Confidence gate calibrated on the harness and wired to hold the last motion estimate.
• Global shutter confirmed for any platform moving faster than a few pixels per frame.

If you are sizing hardware around this pipeline, start at the camera configurator and work forward, or ask the applications team to review your motion budget before you order.