Depth from a matched stereo point pair: disparity to 3D coordinates

You can identify one feature in both images of a stereo pair - a hole center, a fiducial, a corner - and you want its 3D position, not a dense depth map. That is the situation when you build a pair from two area scan cameras, bypass a depth engine, or debug one point in a 3D camera's output. The math is four lines: disparity from the two pixel positions, depth from disparity, then deprojection of one pixel. What breaks it is units, rectification and integer disparity. This guide walks the whole path for one matched pair of points, shows the arithmetic with real numbers, and names the failure modes behind most wrong results, with the check that catches each one.

The depth formula and the units that break it

In a rectified pair, the two image planes are coplanar and the rows align. One world point then projects to (u_L, v) in image 1 and (u_R, v) in image 2 - same row, different column. Disparity is that column difference, in pixels, and depth is focal length in pixels times baseline divided by it:

fx is the focal length in pixels of camera 1's rectified intrinsics; b is the distance between the two camera centers. The baseline's unit becomes the depth's unit - b in meters returns Z in meters - so a depth off by exactly 1000× means the baseline unit and the unit you expected disagree. Near points produce large disparity, and a point at infinity drives disparity toward zero, which is why far range is where stereo starves. Confirm the sign convention once with a near object: it must give larger d and smaller Z than a far one, because some pipelines define disparity as u_R - u_L.

For scaling a whole depth image rather than one point, the companion piece is stereo disparity scale and calibration for depth estimates.

From depth to X, Y, Z

Depth is one axis. The other two come from deprojecting the pixel in image 1 with the same rectified intrinsics - a pinhole model with zero distortion, because rectification has already removed the lens distortion:

cx and cy are the principal-point coordinates, in pixels, of camera 1's rectified intrinsics. The result sits in camera 1's frame: X runs right, Y runs down when cy counts from the top row, so flip Y's sign for a Y-up world frame. An SDK's deprojection call does exactly this from a pixel, a depth value and one intrinsics block. Which intrinsics block belongs to which pixel is where deprojection usually fails - aligned depth point cloud intrinsics works that case in detail.

One matched point at 6 m: fx 600 px, baseline 50 mm, disparity 5 px

Inputs, each with its unit and its source:

• fx = fy = 600 px, cx = 640 px, cy = 512 px - camera 1's rectified intrinsics, from the pair's calibration block.
• b = 50 mm - the baseline between the camera centers, from the pair's extrinsics; the geometry of a compact stereo module.
• P = (700, 500) px in image 1 and P' = (695, 500) px in image 2 - sub-pixel matched positions (whole pixels here, so the arithmetic stays visible).
• The rows agree at 500 px, so the pair can be treated as rectified.

The rule in words: depth is focal length in pixels times baseline, divided by disparity in pixels; each transverse coordinate is the pixel's offset from the principal point, scaled by depth over focal length.

• d = 700 - 695 = 5 px
• Z = 600 × 50 / 5 = 30,000 / 5 = 6,000 mm = 6.0 m
• X = (700 - 640) × 6,000 / 600 = 60 × 10 = 600 mm
• Y = (500 - 512) × 6,000 / 600 = -12 × 10 = -120 mm

The point is (600, -120, 6,000) mm in camera 1's frame: 0.6 m right of the optical axis, 0.12 m above it after flipping Y, 6.0 m out.

How good is that number? Depth resolution follows the disparity step:

At 6,000 mm, one pixel of disparity is 6,000² / 30,000 = 36,000,000 / 30,000 = 1,200 mm of depth. A 1/8 px sub-pixel match narrows that to 1,200 / 8 = 150 mm. The decision: a 50 mm baseline at 6 m is a presence-and-pose sensor with ±0.15 m of depth at best, not a gauge. Holding 50 mm of depth resolution at 6 m needs fx × b ≥ 6,000² / 50 = 720,000 px·mm - a 1,200 mm baseline at fx = 600 px, or fx = 14,400 px at a 50 mm baseline. That job belongs to a wide-baseline pair or a dedicated 3D sensor.

Step by step: two pixels to X, Y, Z

You need a rectified pair (or your calibration maps), camera 1's rectified intrinsics, the baseline, and the two matched pixels.

1. Confirm both images are the rectified pair. If they are raw, rectify the two points with the calibration maps first, or take the triangulation route in the last section.
2. Read P = (u_L, v_L) in image 1 and P' = (u_R, v_R) in image 2. Check that |v_L - v_R| stays within 1 px; a larger row offset means a bad match or drifting rectification.
3. Refine both positions to sub-pixel - a dot centroid or a corner fit on the local patch - and keep the float values.
4. Compute d = u_L - u_R in pixels. A nearer object must yield larger d and smaller Z; if it yields the opposite, flip the sign to your pipeline's convention.
5. Compute Z = fx × b / d with b in millimeters, so Z lands in millimeters.
6. Compute X = (u_L - cx) × Z / fx and Y = (v_L - cy) × Z / fy with camera 1's rectified cx and cy.
7. Reproject the 3D point into image 2 with camera 2's rectified projection; it must land within about 1 px of P'. This one check exposes mixed intrinsics, sign errors and bad matches together.
8. Once per set-up, cross-check with a triangulation routine on the same two points; the two answers should agree inside the depth-step band.

Why the result comes out nonsense

The zero-coefficients trap earns its own paragraph. A rectified stream reports every distortion coefficient as zero because rectification has already removed the distortion; using those coefficients with a rectified pixel is correct. The damage comes from mixing domains - a pixel picked on the rectified image fed to the original camera's coefficients, or the other way round. The fix is to read the parameter set you were actually given; depth camera intrinsics and distortion parameters maps out where each set lives.

Integer disparity is the quiet one. At 6,000 mm with this pair, whole-pixel matching quantizes depth in 1,200 mm jumps - the same stairstep effect that overstates heights near disparity edges, only now you are staring at a single point of it.

Hardware that makes the number trustworthy

Depth step shrinks with the product fx × b and grows with the square of distance. The same 6 m point under three pair geometries:

Double the baseline, or the focal length in pixels, and the depth step halves; double the distance and it quadruples. Long range stereo depth accuracy with a wide baseline works that trade at distance, and wide baseline stereo camera selection for depth mapping turns it into a camera choice.

Two hardware routes. Build the pair yourself from board-level cameras and own the calibration: a binocular module such as the LS-BCM-0043 carries both cameras - 1.3 MP, USB3, a 1280 x 1024 image and a global shutter - and the pipeline, this arithmetic included, stays yours.

Or buy the depth engine ready-made: 3D sensors return computed depth instead of a pair you must match. The MV-DLS300P-04 is a laser galvanometer stereo camera; the MV-DPS200P-02 is a projected structured-light 3D camera.

Checks before you trust the number

• Row offset within 1 px after rectification; more means the match or the rectification is bad.
• Sub-pixel floats kept: 1/8 px is a realistic floor for patch matching, and a symmetric dot centroid reaches about 1/16 px.
• A tape-measured target whose computed Z lands inside the depth-step band Z² × dd / (fx × b).
• Reprojection into image 2 within about 1 px of P'.
• Unit and sign confirmed with one near and one far object.
• One triangulation cross-check per set-up, agreeing inside the same band.

When the one-line shortcut does not apply

The closed form lives on rectified images. If the two points come from raw images, u_L - u_R is not a disparity, and no unit choice repairs it. Two legitimate routes remain: rectify the points with the calibration maps and continue as above, or intersect the two projection rays directly with a triangulation routine - cv::triangulatePoints is the common one - using the full extrinsics. Triangulation is the general tool; the disparity formula is its closed form after rectification, so on the same points the two must agree. What the coefficients mean before that rectification is covered in unrectified camera images and distortion coefficients.

The same product fx × b also brackets the usable range. With 30,000 px·mm, the last full pixel of disparity sits at Z = 30,000 mm = 30 m, and the nearest geometric limit, where disparity spans a whole 1280 px row, is 30,000 / 1280 ≈ 23 mm. Both limits move linearly with fx × b, which is one more reason long range wants a long baseline.

To turn this into a system rather than a one-off calculation, start from the hardware: the camera finder filters area scan and board-level cameras by sensor, mount and interface, and the configurator walks camera, lens, light and controller in order. More measurement guides sit on the 3D topic page, and if a number here will not match your rig, the contact page reaches the applications team.