Why the fundamental matrix fails uncalibrated stereo rectification

You found a chessboard, matched corners, estimated a fundamental matrix, and the rectified pair came back black, zero-filled or full of infinities. That output is the pipeline answering a metric question with a projective result. It is the classic failure of uncalibrated stereo rectification, and it hits anyone wiring two industrial cameras into a rig — a depth head beside an existing camera — with image pairs in hand but no camera matrices. This guide shows why the fundamental matrix alone cannot rectify, what cvStereoRectifyUncalibrated actually returns, and the calibrated path from the same chessboard set to R1, R2, P1, P2 and Q, plus the three numbers that tell you the result is sound before you build a depth map on top of it.

Why do rectified stereo images come out black, zero-filled or infinite?

Your maps are degenerate because the pipeline mixes two incompatible geometries: cvStereoRectifyUncalibrated returns projective homographies, while initUndistortRectifyMap expects a rotation plus calibrated camera matrices, and that mismatch sends every mapx and mapy to 0.00 or ±inf.

Each signature points at a different broken input:

• Every map entry exactly 0.00: the homography or the composed transform came out singular, which usually indicts an F built from pooled or planar points.
• Maps of ±inf across the frame: a projective term survived into the sampler, and its denominator crosses zero near the epipole.
• Values sane in the centre, exploding toward one edge: the same singularity, with the epipole just outside the frame.

In order of probability on a rig in this state: first, no per-camera calibration exists, so the code substitutes identities or guesses for K1 and K2; second, F was estimated from correspondences describing more than one geometry; third, a general homography was pushed through a pipeline that only accepts rotations. One trap hides in the composition itself: cvMul, the multiply these wrappers reach for, works element-wise, while composing a homography with a camera matrix calls for the true matrix product — check that call before debugging anything else. All three causes fall to the same fix, which is the rest of this guide: calibrate the heads, then rectify from R and T.

Can you rectify a stereo pair from the fundamental matrix alone?

No. The fundamental matrix F encodes only epipolar geometry in raw pixels and is defined up to scale, so it cannot tell you a focal length, a principal point or a baseline; without those metric quantities the rectified pair has consistent rows but no usable camera model.

F is a 3 × 3 matrix of rank 2 that converts a pixel in the left view into the epipolar line on which its match must lie in the right view: x2ᵀ·F·x1 = 0. It needs no lens knowledge, which makes it excellent for rejecting bad matches and useless for metric rectification. Focal length, principal point, distortion and baseline are exactly what a depth-capable rig requires, and none is recoverable from F: the same correspondence set fits a 60 mm baseline and a 600 mm one equally well, and since depth scales with the baseline, every distance differs by a factor of 10 between the two. Scale is a free parameter that only a physical measurement or a calibration target pins down. The bridge from the pixel-level F to that calibrated world is one matrix multiply per side — the essential matrix — and building it is where the fix starts. Wikipedia's fundamental matrix article keeps the algebra in one place.

What does cvStereoRectifyUncalibrated actually return?

It returns H1 and H2, a pair of 3×3 projective homographies that align epipolar lines horizontally for one specific scene pair; they carry an arbitrary projective component tied to the epipole, so unlike a rotation R they cannot be written as Knew·R·K⁻¹ and slotted into a calibrated remap.

The inputs are matched points from one two-view set, the image size and F; the outputs are the two homographies that send every epipolar line to a common row. The standard method achieves that by pushing the epipole to infinity, and the price is a projective term in the last row of each H. A calibrated rectification is the special case with that term zero: a pure rotation about the camera centre, always expressible as Knew·R·K⁻¹. Hand the general form to initUndistortRectifyMap and the sampler evaluates the homography near the epipole's image position, the denominator crosses zero, and the source coordinate runs to infinity — the ±inf maps from the first section. The 0.02 scale that travels with a lot of copy-pasted code multiplies every entry equally; it shrinks the numbers without touching the projective term, so it cannot rescue the pipeline.

How do you get the camera matrices the pipeline is missing?

Run one cv::calibrateCamera per camera on the chessboard set you already have: a 9×6 inner-corner pattern yields 54 points per view, and 25 pose pairs per head are enough to solve the 3×3 matrix K (fx, fy, cx, cy) and the distortion coefficients D that rectification needs.

Detect corners once per view, keep the ordering consistent between the two heads, and define the square side in millimetres: print a target with 20.0 mm squares and confirm the print with calipers, because that one number sets the units of everything downstream, including the translation the stereo step returns. The per-device case is the same math with or without a partner head — the single camera calibration guide walks it. Then hand K1, D1, K2, D2 and the per-pose corner lists to cv::stereoCalibrate; it returns R, T and an RMS reprojection error in pixels.

Four conditions decide whether those matrices are usable:

• 20 to 40 poses per head, the board visiting every region of the frame, tilted 30-45° about both axes, at two or three distances.
• Both heads exposed at the same instant; a board that moves between two asynchronous exposures corrupts T. Trigger the pair in hardware — the USB stereo trigger set-up covers the wiring, and if hardware sync seems to score worse than software, the trigger timing fix names the usual cause.
• Focus and iris locked before the first frame. A focus bump after calibration invalidates every matrix in the set, and nothing downstream detects it.
• Distortion coefficients come out of the same run; when their signs and magnitudes look strange, the distortion coefficient guide decodes them field by field.

Check streams and metadata once before blaming the math — the raw stereo calibration check is that pre-flight. When a flat target cannot enter the scene at all, calibration from surveyed scene points replaces the chessboard entirely.

Should all 25 image pairs go into findFundamentalMat?

No. F describes one rigid two-view geometry, so its correspondences must come from a single simultaneous stereo pair; pooling corners from 25 chessboard poses mixes many relative poses, and a planar target on top of that leaves F ambiguous, so RANSAC returns a matrix that maps everything to zero.

The RANSAC settings that usually travel with the call — a 1.0 px threshold at 0.99 confidence — can only select one consensus model; when the input describes 25 geometries, the selected model fits none of them, and its F drives the degenerate maps of the first section. A calibration board adds a second, deeper problem: every corner lies on one plane, and planar correspondences leave F ambiguous — any matrix consistent with the plane-to-plane homography reproduces the matches, including ones that rectify to nonsense.

The data flow therefore changes shape. The 25 pairs stop being findFundamentalMat input and become calibration poses for cv::calibrateCamera and cv::stereoCalibrate. If you still want an F — for match filtering, say — estimate it per capture from non-planar scene points, or build the essential matrix directly from the calibrated R and T you now own. On this rig, with intrinsics in hand, the uncalibrated rectification function has no remaining job.

How do you turn F into R, t and a proper rectification?

Once both K matrices exist, compute the essential matrix E = K2ᵀ·F·K1, decompose it by SVD into R and t, keep the candidate with positive depth, then call cv::stereoRectify to produce the R1, R2, P1, P2 and Q the standard remap expects.

Three steps, all documented library calls:

1. Build E from F with E = K2ᵀ·F·K1. If you reached this point through cv::stereoCalibrate, skip the step: R and T came back from the calibration directly, with the scale already fixed in chessboard units.
2. Decompose E by SVD when F is all you have: two rotation candidates times two signs of t give four solutions, and the cheirality test keeps the one that triangulates scene points to positive depth in front of both cameras.
3. Rectify with cv::stereoRectify(K1, D1, K2, D2, imageSize, R, T). It returns R1, R2, the projection matrices P1 and P2 — a shared new focal length with aligned principal points — and Q for disparity-to-3D conversion.

Then build the maps with initUndistortRectifyMap(K1, D1, R1, P1, ...) and remap both streams. The new-camera-matrix argument takes P1, not K1: P1 carries the shifted principal point and the valid-pixel focal length that makes rows truly correspond.

From here the rig behaves like any calibrated pair: the live rectification guide streams the result, and the disparity-to-3D conversion turns one matched point pair into XYZ coordinates.

What numbers tell you the calibration is good before you trust depth?

Check three numbers: an RMS reprojection error from cv::stereoCalibrate under 0.5 px, matches landing on the same row within about 0.5 px after rectification, and a disparity map that is smooth on textured surfaces instead of striped.

The first number comes straight out of cv::stereoCalibrate; two fixed industrial cameras with sound lenses land far below 1 px, and a worse value sends you back to corner ordering and simultaneous exposure, not to the rectification code. The second test takes a minute: draw horizontal lines through a dozen hand-picked matches in one scene pair, and a correct rectification keeps every pair on its line. The third runs the stereo matcher over a textured target at a known distance and checks that disparities grow smoothly as the target approaches.

A worked example with the arithmetic

Illustrative rig: two identical 1440 × 1080 global shutter cameras, 3.45 µm pixels, a 6 mm fixed-focus lens, baseline B = 120.0 mm measured between lens centres.

• Expected focal length in pixels: fx = f / pixel pitch = 6 mm / 0.00345 mm = 1739 px. A calibration returning close to 1740 px passes; a value far from it indicts the square size or the corner ordering, not the lens.
• Principal point: cx ≈ 720 px and cy ≈ 540 px, the centre of 1440 × 1080. A centred lens drives both toward the middle; large offsets point to a pose set that never tilts the board.
• Depth from disparity with the new focal f' = 1739 px and B = 120 mm: Z = f'·B/d, so d = 40 px gives Z = 1739 × 120 / 40 = 5217 mm ≈ 5.2 m, and d = 200 px gives Z = 1739 × 120 / 200 = 1043 mm ≈ 1.0 m.
• Depth precision: δZ ≈ Z²·δd/(f'·B). At Z = 1.0 m with δd = 0.2 px: 1.00 m² × 0.2 px / (1739 px × 0.120 m) = 0.2 / 208.7 m = 0.00096 m ≈ 1 mm.

F alone supplies none of it: f' comes from the calibration, B from the ruler, and the uncalibrated homographies carry no scale whatsoever. Save the matrices once and reuse them every session — the stored calibration route shows the load-and-rectify path. If the Q-generated point cloud comes out holey, the missing-XYZ checks separate geometry problems from sensor problems.

When is the uncalibrated fundamental-matrix route still the right choice?

Only when you need epipolar matching without metric output: the uncalibrated homographies straighten rows for a matching pass on a scene where no calibration target can go, but they never yield a baseline, so every distance they imply is an arbitrary scale.

Match filtering with F survives on rigs you cannot recalibrate, and row-aligned rectification survives for one-off captures where nothing downstream converts disparity to distance. Everything measurement-shaped belongs to the calibrated path.

If the rig is yours to specify, the hardware prevents this failure class: two copies of one global shutter model on a rigid bar, triggered together, calibrated once, focus taped. Pick the pair from the area scan cameras category. The MV-CU020-90GC is a 2.5 MP GigE camera with a 1/1.7" global shutter sensor and C-mount; the MV-CH120-11UC moves to 12 MP on a 1.1" USB3 sensor when close-range depth needs the extra pixels.

When the deliverable is depth rather than a rig, stereo and structured-light devices in the 3D sensors category handle the geometry themselves. The camera finder filters both routes by interface and resolution, the camera configurator walks sensor, mount and interface in order, and the 3D topic page collects the geometry fixes. To size the baseline against a depth-precision target, send the working distance and tolerance through the contact page and we will run the numbers with you.