Stereo rectified images with no overlap: what the wide baseline did

You pointed two cameras at a plate from 60-70 cm, ran the checkerboard calibration in the Stereo Camera Calibrator, and the app detected every corner cleanly. Then the rectified view came back with one image pushed completely left and the other completely right, and the anaglyph refuses to fuse. The horizontal lines sit at the same heights in both views, and that is the part that says the calibration worked. What is broken is the geometry: the baseline at that working distance produces a disparity larger than the width the two views share, so after rectification the scene lands in exclusive columns. This walkthrough gives you the measurement that proves it, the arithmetic that sizes the fix, and the decision between a shorter baseline, a taller rig, and a shifted matching range.

The symptom: rows match, columns do not

rectifyStereoImages projects both views onto a common image plane so that corresponding points land on the same row. That is its definition of success, and this pair achieves it: the horizontal lines sit at identical heights, and only the x-ranges are exclusive. Extra calibration images tighten the estimate of the camera parameters; they cannot move a column offset that the baseline and the depth have set. Re-shooting three series of seven to eight boards at different sizes and orientations changes exactly nothing here, which is the strongest signal that the calibration was never the fault.

The decision is immediate: stop recalibrating and start measuring. Keep the contrast in mind, because a pair whose rows are broken as well is a different fault — stereo rectification errors that show as horizontal lines trace back to camera order and baseline sign, not to disparity scale.

The geometry that splits the views

Rectification reprojects the pair as if both cameras were parallel. After that transform, the disparity of any point is fixed by three numbers:

Here f_px is focal length in pixels, B the baseline, Z the depth, and W_fov the width of one camera's footprint on the plate. Two consequences follow for a rig like this.

First, vergence does not rescue you. Toe the cameras in as far as the mounts allow: after rectification every pair is parallel, and a point on a plate 0.65 m below carries f_px·B/0.65 pixels of disparity whether the cameras stared straight down or crossed at the plate. Convergence only forces a heavier warp inside rectifyStereoImages, which is why the function's notes warn that cameras far apart and pointing in different directions need a greater adjustment.

Second, the two footprints must intersect. When B grows to W_fov, the left and right views of the plate stop overlapping, the anaglyph splits into a red half and a cyan half, and no matcher can repair what the cameras never both saw. The overlap rule above is the parallel-case estimate, and it errs on the optimistic side for converged cameras. The same shrinkage seen from the other side is why a depth image's field of view is smaller than either camera's; sizing the spacing against the working distance in the first place is the stereo baseline choice from distance decision this rig skipped.

Measure the disparity before touching the rig

Put a number on the offset before you loosen anything:

1. Open the rectified pair side by side, or the anaglyph from stereoAnaglyph.
2. Find one sharp feature you can identify in both views: a board corner, a scratch, a printed edge. If no feature appears in both, overlap is already zero and only the geometry fixes apply.
3. Read the feature's column in the left view and in the right; disparity is the left column minus the right column. The imageViewer app gives the pixel coordinate, and on the anaglyph you can measure straight between the red and cyan copies of one edge.
4. Compare that value with the image width, and with 256.

On a rig like this one — two cameras 60-70 cm over a plate — the measurement lands near 500-550 px. Set against a 256 px matching ceiling and a finite image width, that single number predicts everything the rectified view showed you.

Worked example: 500-550 px of disparity at 0.65 m

Take the rig's own inputs. Depth Z = 0.65 m, the midpoint of the 60-70 cm camera height. Measured disparity d = 525 px, the midpoint of the 500-550 px range. Matching ceiling 256 px, the documented limit on MaxDisparity in disparityBM.

The rule in words: disparity equals focal length in pixels times baseline over depth, so the product f_px·B equals d × Z. The arithmetic:

• f_px·B = 525 px × 0.65 m = 341 px·m. You never need f and B separately to fix this — the product is what the disparity sees.
• Baseline cut to reach 256 px at the same depth: B_new = B × 256 / 525 = 0.49 B. Roughly halve the spacing.
• Depth step per disparity pixel, from Z = f_px·B / d: before the cut, 0.65² / 341 = 0.4225 / 341 = 0.0012 m, about 1.2 mm.
• After halving the baseline: 0.4225 / 171 = 0.0025 m, about 2.5 mm per pixel.
• Raising the rig instead of cutting the baseline: Z_new = 341 / 256 = 1.33 m, with a depth step of 1.33² / 341 = 5.2 mm per pixel.

Decision: cut the baseline to about half. It reaches the 256 px target at half the depth cost of doubling the height — 2.5 mm against 5.2 mm per disparity pixel — and it keeps each camera's footprint on the plate. For any rebuild, hold the budget f_px·B ≤ 256 px × 0.65 m = 166 px·m. Take a 6 mm lens on 3.45 µm pixels: f_px = 6 mm / 0.00345 mm = 1739 px, so the baseline cap is 166 / 1739 = 0.096 m, about 96 mm centre-to-centre.

The fixes, and what each one costs

The matcher rows read as follows. disparityBM takes MaxDisparity up to 256 and no higher, so 500-550 px sits outside the window it searches; how block matching computes a disparity map shows the search that would have to span it. disparitySGM puts no ceiling on where the search window sits, but the span you pass — MaxDisparity minus MinDisparity — must stay within 128 pixels. So shift the window instead of widening it: a measured 500-550 px disparity fits disparityRange = [423 550], a span of 127, and the returned map comes back dense.

The one thing SGM cannot lift is shared content. If the rectified pair genuinely shares no columns, there is nothing to match and the geometry changes first. And a wide baseline is not a mistake everywhere: at long range it is the deliberate choice, as wide-baseline stereo camera selection and the depth accuracy math for long-range stereo lay out. At 0.65 m it is simply the wrong tool.

When it is not the baseline

Three faults imitate this symptom. Test them in this order:

1. The cameras are swapped in the pair list. Feeding the right camera's files as I1 and the left camera's as I2 flips the baseline sign and mirrors the split. Test: open the stereoParams object and check the camera-to-camera translation vector against the spacing you measured with a ruler; a vector pointing the wrong way means the input order, not the optics. The full failure is written up in negative baseline after stereo calibration.
2. The rows are broken too. Leaning or wavy horizontal lines put you in the row-mismatch case named at the top of this page, where camera order and baseline sign are the suspects and disparity scale is not.
3. The calibration is genuinely thin. Two image pairs at nearly the same board orientation cannot constrain a stereo rig; ten or more, at varied angles and positions, is the working minimum. Its signature is noisy corners and wavy rows rather than clean exclusive columns — troubleshooting a stereo disparity map after rectification covers that path.

If you rebuild the pair

Pick the spacing from the budget above, B ≤ 256 × Z / f_px — 96 mm centre-to-centre at 0.65 m with the 1739 px example lens.

Then build from two identical global-shutter cameras on one trigger line, so both sensors expose at the same instant and the rows hold still while you fix the columns:

A matched pair on one trigger removes synchronization as a variable. The camera finder filters by shutter type and mount, and the configurator walks camera, lens and trigger in one pass.

When the height and the spacing are both locked by the machine, the way out is a unit that does its own matching:

A projected-light stereo unit replaces the rectify-then-match pipeline with depth computed inside the sensor. These units are laser emitters: respect the class labelling on the housing and never look into the aperture while it runs. Browse 3D sensors for that class of unit, or area scan cameras for the two-camera route.

Checks before you change the rig

• Rows align in the rectified view. If they do not, this page is the wrong page; start from the row-mismatch fault.
• Disparity measured on one shared feature, then compared with the image width: at or above the width, no matcher setting recovers it.
• Disparity compared with 256 for block matching, and with a 128-px-span shifted window for disparitySGM.
• Baseline sign checked in stereoParams before any metal moves.
• Rebuild specced as two identical global-shutter cameras on one trigger.

Work out the footprint one camera throws on the plate before you cut anything, and compare it with the spacing:

If the numbers leave you between two configurations, send the working distance and the plate size and we will size the pair with you.