Phase correlation video stabilization: rotation rejection

A phase-correlation stabilizer returns dx and dy for every frame, and live footage gives you no ground truth to hold it against. Camera roll quietly drags the correlation peak, the correction drifts one-sided, and the picture starts to swim. The fix is a synthetic test loop: crop one large master image into frames with known shifts and small rotations, run your estimator on every pair, and accumulate the errors into numbers you can act on. This walkthrough builds that harness step by step, works a 100-frame example end to end, and ends with the numeric trip that rejects rotation-induced peak drift. It is written for engineers running their own correlation code on a vision controller or a bench machine.

Why rotation moves the correlation peak

Phase correlation reads translation straight out of the Fourier shift theorem: a shift by (dx, dy) adds a linear phase ramp to the spectrum, so the normalized cross-power spectrum of two frames transforms back into a single sharp peak at (dx, dy). The phase correlation article on Wikipedia covers the derivation. Rotation does not fit that model. A roll of θ about the frame center keeps the center fixed and moves every feature at radius r by roughly r·θ, with θ in radians.

In a 1024 × 1024 frame the corners sit at r = 512 × sqrt(2) ≈ 724 px from the center. A 1° roll (0.01745 rad) displaces them by 724 × 0.01745 ≈ 12.6 px, and the ±1.5° bound used below displaces them by up to 18.9 px, while the true translation stays within ±5 px. The estimator sees corner motion several times larger than the shift, the peak smears, and its location biases in one direction as roll accumulates. The harness measures how much of that leaks into the translation estimate; the tracker catches the drift before it reaches the output.

What you need

• One master image, textured and non-periodic. Chessboards, gratings and repeating conveyor texture return several comparable peaks, an ambiguity the phase-correlation literature calls out directly; natural scene texture keeps a single dominant peak.
• Your estimator: the self-written correlation function under test, returning dx, dy and a peak response per pair.
• A reference implementation. OpenCV's phaseCorrelate returns a sub-pixel shift and a response value, so it doubles as a sanity baseline for your own function.
• A ground-truth table, one row per frame: dx and dy in pixels, θ in degrees, written before any estimating happens.
• A machine for the offline harness, with enough memory to hold the master image and the frame sequence; the deployed loop that later runs in the cell is what a fanless industrial mini PC is sized for, not the test itself.

Build the synthetic sequence

Sizes first. Work with 1024 × 1024 px frames cut from a 2048 × 2048 px master, and check the margin before writing code: a 1024 px crop rotated by 1.5° spans 1024 × (cos 1.5° + sin 1.5°) ≈ 1050 px, and ±5 px of crop offset adds another 10 px, so the master needs to be at least a 1060 px square. The 2048 px master leaves about 490 px of slack on each side.

Then the loop:

1. Draw the ground truth for frame i: dx and dy uniform in ±5 px, θ uniform in ±1.5°. These bounds encode the working assumption for this class of stabilizer, shifts of a few pixels and rotation near or under a degree, and they go into the truth table before anything is estimated.
2. Warp the crop out of the master with a 2 × 3 affine matrix: rotation θ about the frame center plus translation (dx, dy), bilinear interpolation. The interpolation is what makes the shift sub-pixel; nearest-neighbour sampling quantizes it back to whole pixels and ruins the test. The matrix arithmetic is laid out step by step in the guide on arbitrary angle image rotation in machine vision.
3. Convert each warped frame to single-channel float32 and apply the same Hann window to both members of every pair.
4. Run your estimator frame by frame and log what it returns: dx, dy, the peak response, and θ if your implementation estimates one.
5. Repeat the whole sequence with every θ forced to 0. That control pass isolates how much rotation alone inflates the shift error.

Score every frame against the ground truth

Three numbers per axis tell the whole story. The error is e = estimate − truth in pixels. The RMS punishes large per-frame mistakes, the mean exposes bias that integrates into visible drift, and the response tracks how much you should trust each estimate.

Decision lines for this frame size and shift range: an RMS at or below 0.1 px passes, and a mean within ±0.01 px per frame passes. The 0.1 px line has a documented basis: windowed phase correlation on clean synthetic pairs is a sub-pixel method, with published results clustering around 0.01–0.1 px RMS, so a number above it points at windowing, rounding or rotation bias rather than at the method itself. If the rotated pass more than doubles the control pass, rotation is the dominant error source, and you need the trip in the next section or a rotation-estimating stage. What a small bias does over a full clip is what the worked example below puts a number on.

Worked example: 100 frames at ±5 px shift and ±1.5° rotation

Inputs, with their sources:

• N = 100 frames from one random seed; the FAQ gives the precision behind that choice.
• True shifts uniform in ±5 px and true rotation uniform in ±1.5°, the bounds set in the build step.
• Frames 1024 × 1024 px, float32, Hann window on both members of each pair.
• Illustrative measured sums from the run: Σ e_x² = 0.28 px², Σ e_y² = 0.35 px², Σ e_x = +1.6 px; control pass at θ = 0: Σ e_x² = 0.12 px².

The rule, in words: RMS is the square root of the accumulated squared error divided by the frame count, and bias is the plain average error.

The arithmetic:

• rms_x = sqrt(0.28 / 100) = sqrt(0.0028) = 0.053 px
• rms_y = sqrt(0.35 / 100) = sqrt(0.0035) = 0.059 px
• bias_x = +1.6 / 100 = +0.016 px per frame
• control rms_x = sqrt(0.12 / 100) = sqrt(0.0012) = 0.035 px

Result: translation RMS lands at 0.053–0.059 px, inside the 0.1 px pass line, and rotation lifts the shift RMS from 0.035 px to 0.053 px, a factor of about 1.5. Rotation is present but not dominant. The failure is the bias: +0.016 px per frame integrates to 16 px of drift over a 1000-frame clip (0.016 × 1000), well past the ±0.01 px per frame line.

Decision: the translation estimator passes on per-frame accuracy and fails on bias, so leave the correlation math alone and fix the drift. Enable the re-anchor trip below, which fires long before 16 px can accumulate, then re-run with two more seeds before signing off. These pass lines are set for 1024 px frames with ±5 px shifts; recompute them if your geometry differs.

The trip that rejects rotation drift

The peak's position tells you whether roll is steering it. Three rules turn that observation into code:

1. Jump guard: if the peak moves more than 10 px between consecutive frames, double the ±5 px bound, treat it as a lost or relocated peak. Re-anchor the reference to the current frame, zero the accumulated correction, and skip this frame's estimate. That is the accept-the-move-once, do-not-chase-it behavior.
2. Drift trip: keep the last 10 peak increments. When at least 8 share one sign and their sum exceeds 1 px, the peak is creeping rather than jittering; re-anchor and reset the accumulator. Translation jitter changes sign frame to frame, roll does not.
3. Trip margin from geometry: under a steady 0.1° of roll per frame, the corners of a 1024 px frame move 724 × 0.001745 ≈ 1.3 px per frame in the same direction, so the accumulator sees eight same-sign increments totalling about 10 px and fires after roughly eight frames, long before peak smearing at the 1.5° bound corrupts the translation estimate.

Be clear about what rejection does and does not do. It stops the stabilizer from following roll, and a false trip costs one dropped estimate because re-anchoring only redefines where zero is. It does not level a rotating scene. If the roll is real content that the viewer must see level, estimate it instead of rejecting it: resampling the log-magnitude spectra into log-polar coordinates turns rotation and scale into shifts, the Fourier–Mellin extension of phase correlation, and the control pass from your harness tells you whether that stage is needed.

Where the test lies: windowing, texture, rounding

Four more ways a harness lies to you:

• No window, or two different windows. Without a Hann window the DFT treats frame edges as wrapping around, and that wraparound smears the peak. Apply the same window to both frames, once.
• Periodic master texture. Chessboards, gratings and repeated patterns yield several comparable peaks and an ambiguous answer. Use natural texture, and treat a periodic real scene as outside what this harness can certify.
• Testing at the wrong resolution or dtype. An error measured on half-size 8-bit previews does not transfer to a full-resolution float32 deployment path; validate exactly what ships.
• Shifts approaching the window edge. A 5 px shift on a 1024 px frame is about 0.5% of the width, deep inside the unambiguous range, while circular-shift ambiguity begins as a shift approaches half the window. If real footage moves hundreds of pixels per frame, enlarge the frames or re-anchor more often.

Take it to real footage, and where it runs

Re-run the scorer on the stabilized output of a real clip: estimate motion between consecutive stabilized frames. A healthy run hovers around zero with no one-sided trend, which is the same frame-shift tracking the guide on detecting camera vibration with OpenCV frame tracking uses to catch vibration. Optical flow then refines what correlation starts; the walkthrough on video stabilization with phase correlation and optical flow covers that complete loop.

Budget before you deploy. A 1024 × 1024 real FFT costs on the order of 2 × 10⁷ operations, three transforms per estimate, two forward and one inverse, so a 30 fps stream lands near 2 GFLOPS. That is light for a desktop-class CPU, but it multiplies with stream count, and the controller also runs acquisition, lighting and your application. The correlation stage itself runs on a windowed ROI of each frame, so ROI size, not sensor size, sets the FFT cost.

The VC5000-class controller below carries 32 GB of memory, 5+ LAN ports and PoE camera ports, which suits a multi-camera line where every stream adds its own FFT budget:

The VC2000-class unit below covers bench validation and single-stream cells:

Live capture in the cell comes from the area scan cameras range. A global shutter sensor keeps frame geometry intact during pans, and the 12 MP USB3 model below fits this pipeline:

Pre-flight checklist

• Master margin: covers the rotated crop, about a 1060 px square for 1024 px frames at ±1.5° with ±5 px offsets.
• Texture: natural and non-periodic, one dominant structure per window.
• Pipeline: warped frames in float32, the same Hann window applied once to each pair member.
• Truth table: written before estimating; estimates diff against it, never against each other.
• Metrics: RMS and mean per axis, plus a θ = 0 control pass.
• Pass lines for this geometry: RMS at or below 0.1 px, mean within ±0.01 px per frame.
• Trip settings: 10 px jump guard, 1 px same-sign drift across the last 10 increments.
• Real footage: residual-motion scorer on the stabilized output, watching for one-sided trends.

To size the machine that runs the deployed loop, work through the vision controllers range or start the system configurator; related systems walk-throughs sit under the systems topic page.