Verify recorded stereo frames are truly rectified

Recorded stereo pairs are easy to mishandle because the file name or topic rarely tells you whether the two frames have been rectified. If a matcher runs on a raw pair, it produces noisy disparity with no obvious error message, and the fault surfaces hours later as bad depth. This guide shows you how to confirm, with three practical tests, whether a saved or streamed stereo pair is rectified, how to read the metadata that hints at it, and what to do when the frames turn out to be raw. It is written for engineers building or debugging depth pipelines from stereo modules, structured-light cameras and paired area scan cameras.

What rectified actually means

Two cameras looking at the same scene from slightly different positions see every point along an epipolar line. In a raw pair those lines are usually slanted or curved, because the sensors are not perfectly coplanar and the lenses are not distortion-free. Rectification warps both images so that every epipolar line becomes horizontal and lies at the same row in both frames. After that, a search for a match in the right image only has to scan one row, and disparity becomes a pure horizontal offset.

That formula only holds for a rectified pair. With a raw pair the vertical component of the match offset is non-zero, so a row-constrained block matcher picks wrong correspondences. The result is not a clean failure: depth values still come out, they are just wrong in patches, typically strongest near the image borders where residual distortion is largest.

Why recorded frames differ from the live view

The most common trap is that a live viewer applies rectification as a post-processing step for display, while the recorded stream holds the raw sensor pair. The recording tool may also select a different pixel format than the viewer showed, and some formats carry rectification while others do not from the same module. So the preview looked correct, and the file quietly contains something else.

Three things decide what a saved pair contains:

• The stream you recorded: a dedicated rectified topic or output versus the raw synchronised pair.
• The pixel format chosen at capture: some infra-red formats are rectified outputs, others are raw sensor data.
• Where rectification runs: on the camera, in the driver, or in your own code using factory calibration.

The lesson generalises beyond any one vendor: verify the data you actually stored, not what a tool displayed at capture time. The same discipline applies when you replay recorded video through a vision pipeline for regression testing.

Check 1: the epipolar line overlay

The fastest visual test. Open the left and right frames side by side, pick a high-texture feature near the image edge, and draw a horizontal line through it in both frames. If the same texture sits on the same row in the right image, the pair is consistent with rectification at that point. Repeat at five or six positions: image centre, all four corners, and mid-edges.

Residual vertical offsets under about half a pixel are normal after rectification, since the warp is an estimate. Offsets of several pixels at the corners mean the frames are raw, or the wrong calibration was applied. A live binary image preview is a quick way to render such overlays during development, because high-contrast edges make row misalignment obvious.

Check 2: row-correlation sweep

The quantitative version of the overlay. Take the left frame, and for vertical shifts s from -3 to +3 pixels compute the normalized cross-correlation between the left image and the right image shifted by s. Average over several textured regions.

For a rectified pair, the correlation peaks at s = 0 for every region. For a raw pair, the peak row shifts systematically across the image, for example s = -2 near the top and s = +3 near the bottom. That systematic drift is the signature of uncorrected rotation between the cameras. You can run this sweep on a handful of 100 × 100 patches in a script in minutes, and it gives you a number to log in your test report rather than an eyeball judgement.

Check 3: metadata and pipeline flags

Before the pixel tests, read what the capture pipeline itself claims:

• Topic, stream or channel names: names containing "rect" indicate a rectified output, while "raw" usually means sensor data.
• Pixel format: some infra-red and mono formats are produced after rectification, others before; the SDK documentation for your module states which.
• Calibration records stored alongside the frames: presence of a rectification transform or projection matrix alongside the intrinsics suggests the pipeline rectifies on the host.

Treat all of it as a hint, not proof. Metadata tells you what the pipeline was configured to do; the correlation sweep tells you what the pixels actually contain. When both agree, you can trust the pair.

Choosing a camera whose output you can trust

When the verification burden itself is a problem, it pays to pick a module that states its output explicitly and keeps its calibration on board. A projected structured-light 3D camera such as the MV-DPS200P-02 publishes a processed depth stream alongside its stereo inputs, so the question of raw versus rectified applies to fewer stages of the pipeline.

For scanning tasks that need denser, better-conditioned matching, a laser galvanometer stereo camera changes the geometry entirely: the pattern is projected rather than passive texture, and the MV-DLS300P-04 is built around that principle. Verification effort moves from "is this pair rectified" to "is the projected pattern synchronised with capture".

Both sit in the 3D sensors category; if instead you build the stereo rig yourself from two area scan cameras, you own the rectification step and the live stereo rectification workflow applies directly. The camera finder filters modules by interface and resolution, and the configurator walks the rest of the chain.

If the frames turn out to be raw

You have two honest options. First, rectify the pair yourself using the module's calibration data: undistort each frame with the intrinsics, then apply the stereo rectification rotation. Do this once per frame pair, before matching, and cache the warp maps so the cost is two lookups per pixel.

Second, re-record. If the original capture is still possible, reconfigure the source to emit its rectified output or the format your pipeline expects, then repeat the row-correlation sweep on the new recording to confirm. Never rectify the same pair twice, and never feed an already-rectified pair through a second rectification: the double warp degrades resolution near the borders and introduces its own vertical residuals. When depth looks wrong downstream, stereo disparity map troubleshooting is the next step, and a systematic vertical-offset signature there often traces back to this exact assumption.

When the usual answer is wrong

Rectification does not fix everything, and assuming it does leads to two recurring mistakes:

• Rectified does not mean matched exposure and gain. A rectified pair can still have one frame darker than the other, which hurts matching more than small geometric error does. Verify per-frame exposure metadata; the per-frame exposure and gain check shows how.
• Rectification narrows the effective field of view. The warp crops toward the image interior, so a rectified frame covers slightly less of the scene than the raw frame. Plan for this margin; the same effect is why depth image field of view is smaller than the colour view on many modules.

On very dark or featureless surfaces, passive stereo fails regardless of geometric quality; stereo depth lighting optimization addresses the illumination side of that problem.

Quick verification checklist

• Confirm stream names and pixel format against the SDK documentation before recording.
• Overlay horizontal lines at five or more image positions; note the vertical offsets.
• Run the row-correlation sweep (-3 to +3 pixels) on several textured patches; the peak must sit at zero.
• Cross-check the metadata claim against the pixel result; trust the pixels.
• If raw, rectify once with the stored calibration, cache the warp maps, and re-run the sweep.
• Log the sweep result with every recording so the assumption is auditable later.

Run these checks once per capture configuration, not once per project. A driver update, a changed pixel format or a swapped mount can silently turn a rectified stream into a raw one, and the sweep takes minutes to catch it. If you are selecting hardware for a new depth cell, start from the 3D sensors category or contact us for help matching the module to the task.