Why uncalibrated rectification gives far objects large disparity

You rectify a stereo pair, apply the rule that closer means larger disparity, and find a far object — the car at the far end of the lot, the wall behind the line — sitting at a disparity that belongs a few meters ahead. The pair is usually not broken. Uncalibrated rectification, the kind built from a fundamental matrix estimated on matched feature points, only makes the epipolar lines horizontal; it does not force the two camera axes parallel, so the rectified pair stays slightly verged and infinity keeps a few pixels of disparity. This guide gives the geometry behind that, the pixel numbers to expect, a field test, the correction that keeps depth valid, and the signs that a different fault is the real cause.

Why does a far object still show a large disparity?

Because uncalibrated rectification only aligns the epipolar lines, it does not force the two camera axes parallel, so the rectified pair stays slightly verged and objects at infinity keep a constant offset of a few pixels of disparity. The rule "closer means larger disparity" still holds — every measured value is simply shifted:

Here f is the focal length in pixels, B the baseline, Z the object distance and d∞ the disparity this particular pair assigns to infinitely far points. The sign of d∞ follows the vergence direction, so far objects can even come out negative. That is a nuisance, not a fault: the map stays self-consistent, only its zero is not at infinity. If instead the correspondences or the matrix itself are unstable, read why the fundamental matrix fails uncalibrated stereo rectification first — that failure looks different.

What is the difference between calibrated and uncalibrated rectification?

Calibrated rectification uses measured intrinsics and extrinsics from a target to rotate both images into a truly parallel pair, in which the plane at infinity is the zero-disparity locus; uncalibrated rectification recovers only the epipolar geometry, with no scale and no vergence constraint. The fundamental matrix tells your code where a match may lie, not how far apart it must be. Rectification built from it picks two homographies out of a large family, and the default choice in most vision libraries leaves a residual convergence angle in the pair. Specialized quasi-Euclidean methods push the zero-disparity locus back toward infinity, but they are not what a standard rectification call returns. Two freedoms survive into your map: an unknown additive constant, and an unknown scale until you pin f·B on a measured distance — the scale issue worked through in stereo disparity scale and calibration for depth estimates.

How large can the disparity at infinity be?

Roughly the focal length in pixels times the residual vergence angle in radians, f·θ, so a quarter degree of vergence at f = 2000 px puts about 8.7 px of disparity on every infinitely far point in the scene. Half a degree at the same focal length gives about 17 px. Across the middle of the image the offset is close to constant; below about one degree of vergence the variation toward the edges stays under a pixel or two.

Worked example, with two identical cameras on a 100 mm baseline and f = 2000 px (a 6.9 mm lens on 3.45 µm pixels), so f·B = 200 px·m:

• True parallax of a car at Z = 20 m: d = f·B/Z = 200 / 20 = 10.0 px
• Residual vergence left by the rectification: θ = 0.25° = 0.00436 rad
• Offset: d∞ = f·θ = 2000 × 0.00436 = 8.7 px
• Measured disparity: 10.0 + 8.7 = 18.7 px
• Depth the naive conversion reports: 200 / 18.7 = 10.7 m
• Depth after removing the offset: 200 / (18.7 − 8.7) = 20.0 m

The far car comes back 9.3 m too close, and nothing in the pair is broken.

How do I test whether my rectified pair is verged?

Point the pair at the farthest textured background within reach, compute the disparity map, and compare the median disparity there with the parallax f·B/D that distance D predicts — the excess is the vergence offset, and it repeats on every far surface. For the example pair, a wall at 50 m should show 200 / 50 = 4 px of true parallax; a median of 12.7 px there leaves 8.7 px of offset. Sky and blank walls return mismatches rather than measurements, so use a distant building or the far end of a corridor.

Two more checks close the case. Sample ten strong corner matches and confirm they share a row after rectification; a vertical residual means the rectification itself is poor, and stereo disparity map troubleshooting after rectification covers that repair. Then confirm scale against one tape-measured distance, where d should land on f·B/Z.

How much does the offset damage depth estimates?

It biases every depth low by d∞/(d + d∞), a fraction that grows with distance — with the example pair the error is 4% at 1 m, 18% at 5 m and 47% at 20 m. Ordering survives, because nearer still means larger measured disparity, so detection and sorting keep working; metric depth does not. The distance at which the bias passes 5% sits near B/(19·θ), which is 1.2 m for the example pair; beyond it, treat uncorrected depths as relative only.

Far-field reading is coarse regardless: one disparity pixel spans Z²/(f·B) of depth, 2 m at 20 m here. That is why fixed mm-per-band assumptions fail in disparity map to depth conversion: bands are not fixed mm, and why fine height steps suffer the quantization shown in stereo depth height overestimation from disparity quantization.

How do I remove the disparity offset?

Subtract the measured far-field disparity from the map and convert with Z = f·B/(d − d_min), or remove the cause outright by calibrating the pair so rectification is Euclidean and infinity maps to zero disparity. In the field:

1. Capture a frame pair with far textured background in view, at your production exposure, gain and matcher window.
2. Read the median disparity over that background and call it d_min. Use the median of the far region, not the raw map minimum — isolated mismatches drag the minimum below the true offset.
3. Shift the map, d′ = d − d_min. In matcher terms, move the minimum-disparity bound until the far background lands at zero; if the far field reads negative, extend the range below zero first or the clip deletes it. The same range-shifting move appears for a depth camera in disparity shift: set the minimum range in your script and for the conversion side in stereo disparity to depth conversion: set the depth window.
4. Pin the scale on one known distance — the offset fix positions the zero, it does not set f·B.
5. Prove the fix on two targets: the far background must return a depth beyond its true distance, and a near tape-measured target must land on its number. With the example pair, the 1 m target reads (208.7 − 8.7) = 200 px, and 200 / 200 = 1.00 m.

The subtraction carries one assumption: the offset is constant across the field, which holds to a fraction of a pixel for vergence under about a degree inside the central two-thirds of the image.

When is the vergence explanation wrong?

When the large disparity belongs to one isolated object instead of the whole far field: a constant offset shows up on every distant surface, while a single far object with high disparity is usually a matching error, a reflection, or a smaller object than you assumed. Work the causes in order of probability:

Matching failure deserves a paragraph of its own: on weak texture the matcher copies the disparity of a nearby strong edge, which looks exactly like "a far object with big disparity". The window search behind it is explained in SSD block matching for stereo depth, and boundary artifacts have their own analysis in flying pixels between object and background. Learned matchers bring their own noise patterns, covered in troubleshooting noisy DNN stereo disparity depth.

How do I prevent the offset on the next build?

Calibrate the pair with a target and run calibrated rectification in production, and buy two identical bodies with locked, identical settings — that removes both the vergence and the scale freedom that uncalibrated rectification leaves open. On the hardware side, build from two of the same model with the same lens. In the area scan cameras line, the camera finder filters by shutter and interface. The MV-CU020-90GC is a 2.5 MP GigE camera with a global shutter and a C-mount, and two of them make a compact pair for conveyor-range work.

When far detail sets the resolution, the MV-CH120-11UC is a 12 MP USB3 camera on the IMX304 sensor in a high-speed version.

Both are color models. Where faint texture rules, a mono body matches more cleanly: the MV-CS200-10UM is the 20 MP mono USB3 option, and its rolling shutter confines it to static scenes, so trigger both cameras together and keep the part still. If the metric must be right the day the cell starts, move to a purpose-built stereo unit, where rectification is the maker's job rather than yours: among 3D sensors, the stereo and structured-light class includes the MV-DLS300P-04, a laser galvanometer stereo camera. Its illuminator is a laser, so honor the class rating and keep eyes out of the beam during alignment.

Prevention checklist:

• Calibrate intrinsics and extrinsics with a target; use calibrated rectification in production.
• Two identical bodies, the same lens, focus locked.
• Exposure, gain and white balance locked to the same values on both cameras.
• One hardware trigger line to both cameras; global shutter whenever anything moves.
• After mounting, record the far-background median next to the camera serial numbers.

More stereo geometry pieces sit on the 3D topic page, and the camera step of the configurator walks a pair choice end to end.