Stereo triangulation world origin: choosing where Z is measured from

You calibrated a stereo pair against a chessboard, built projection matrices from per-view poses, triangulated the board's corners, and got back exactly the object points you defined: X and Y on a grid, Z near zero. The triangulation is working. The points are simply coming out in the chessboard's coordinate system, because that is the world frame your projection matrices were built in. This guide shows how to rebuild the matrices so camera 1 sits at the origin and Z reports camera-to-object distance, with the arithmetic shown so you can verify each step.

Why the frame follows the projection matrices

Triangulation solves for the 3D point X that projects to both observed image points. It has no idea of cameras or boards; it only sees two 3×4 matrices P1 and P2. Whatever world frame those matrices encode, the recovered points live in.

When you run a per-view pose solver on the chessboard corners, the returned rotation rvec and translation tvec describe the camera in the board's frame (equivalently, they map board coordinates into camera coordinates, depending on the convention your code applies). Chain that pose into a projection matrix and the board becomes the world: its plane is Z = 0, and your cameras sit somewhere at a large Z. Triangulating the board's own corners then trivially returns your grid with Z ≈ 0 — a sign of consistency, not a bug.

If you instead want Z to be the depth of the object from the stereo pair, define the world frame at camera 1:

Here K1 and K2 are the intrinsic matrices, I is a 3×3 identity, and [R | t] is camera 2's extrinsic expressed in camera 1's frame. The usual stereo convention also makes X right, Y down, Z forward along the optical axis of camera 1.

What you need

• A calibrated stereo pair: intrinsics K1, K2 and distortion coefficients for each camera. Two matched area scan cameras such as the LS-CMR-0053 or the LS-CMR-0166 work well for bench-scale depth; a larger sensor like the MV-CS200-10UM gives more pixels per millimetre at the same field of view.
• Per-view poses of the calibration board in each camera (or a stereo calibration that outputs R, T directly).
• Matched image points, undistorted with each camera's coefficients before triangulation.
• A rigid mount, so the relative pose does not drift between calibration and measurement. A camera stand with a slide stage makes the baseline adjustable and repeatable.

If you are still capturing calibration images, read stereo depth camera calibration: target capture and stream setup first; good target coverage decides whether any of this works.

Build a camera-centred world frame step by step

1. Calibrate intrinsics and distortion for both cameras from the chessboard views, exactly as you already do. Nothing changes here.

2. Compute per-view poses of the board in each camera with a PnP solver, giving rvec1, tvec1, rvec2, tvec2 for the reference view. Note the convention: these describe the board relative to each camera (or the camera relative to the board, depending on your formulation — check with a reprojection).

3. Invert camera 1's pose so you can travel from camera 1 to the board and then to camera 2:

4. Compose the two legs of the path. Camera 1 to board, then board to camera 2, gives camera 2's pose in camera 1's frame:

composeRT chains rotations and translations correctly, including the cross term most hand-rolled compositions forget.

5. Build the projection matrices with camera 1 at identity:

6. Undistort the matched image points with each camera's coefficients before triangulating. Feed P1, P2 and the normalized points to your triangulation function.

7. Verify with a known standoff. Place the board at a measured distance from camera 1 and check the mean Z of the triangulated corners.

Worked example: composing the pose

Suppose the reference view gives, in the board frame, camera 1 at t1 = (−40, 30, 640) mm with a small rotation, and camera 2 at t2 = (−140, 28, 642) mm. The baseline is therefore about 100 mm along X, which matches the rail.

Invert camera 1's pose: R1inv = R1ᵀ and t1inv = −R1ᵀ · t1 ≈ (−640-ish in the board's Z after rotation). Compose with camera 2's pose:

• t12 ≈ t2 − R12 · t1 in the small-rotation limit, which lands near (−100, −2, 2) mm: camera 2 sits 100 mm to the left of camera 1, roughly level. That is the sanity check — the composed translation should reproduce your measured baseline.

Now triangulate a board corner that sits 640 mm in front of camera 1. With the chessboard-frame matrices it returned Z ≈ 0. With P1 = K1 · [I | 0] it returns roughly X ≈ 0, Y ≈ 0, Z ≈ 640 mm — the camera-to-object distance you wanted, up to the board's tilt.

For how those XYZ values map to real measurements once you have them, see bounding box depth: mapping image pixels to 3D points.

Chessboard frame or camera frame: which do you want?

Each is one rigid transform away from the others. Once your points are in camera 1's frame, prepend the camera-to-robot transform to reach robot coordinates — the mechanics are the same as in multi camera extrinsic calibration in one shared coordinate frame. And if your points later travel between OpenCV, MATLAB and other tools, the frame conventions differ per format; import stereo calibration parameters across software formats lists the gotchas.

Choosing hardware that makes Z trustworthy

The frame choice costs nothing, but depth precision does depend on parts:

• Baseline and depth set the triangulation angle. Longer baselines give better Z precision at range, at the cost of a smaller shared field of view — the trade-offs are laid out in wide baseline stereo camera selection for depth mapping.
• A matched lens pair keeps the two focal lengths identical, so the composed pose stays clean. Browse machine vision lenses for fixed-focal C-mount pairs.
• A rigid stand, not two flexible tripods, keeps [R | t] valid for more than a day. The camera stands finder lists slide stages that set the baseline by scale.

Checks before you trust the new frame

• Mean Z of a board at a measured standoff matches the tape measure within your expected precision.
• Reprojecting the triangulated points through P1 and P2 lands on the detected corners to well under a pixel.
• The composed translation's norm equals your measured baseline.
• Tilting the board changes its Z spread but not its mean distance by much — if the mean jumps, rotation leaked into the translation.
• Camera 1's extrinsic block is exactly [I | 0] in code, not the leftover inverted pose.

When the usual answer is wrong

If your application actually wants board-relative measurements — flatness of a part sitting on a known plane, for instance — the chessboard frame you already had may be the right one. Moving to a camera frame and then re-projecting onto the part plane adds a transform and a chance for error. Choose the frame the downstream consumer of the points wants, and change it in one place: the extrinsics. Never patch Z afterwards by subtracting a constant, because tilt and off-axis points will not shift uniformly.

Ready to assemble the pair? Start at the camera finder or the system configurator to match cameras, lenses and a stand in one pass, or contact us with your working distance and depth tolerance and we will size the baseline for you.