3D visionHow-to

Camera projection matrix from calibration for stereo triangulation

Assemble the 3×4 camera projection matrices P = K[I|0] and P' = K'[R|t] from a stereo calibration, run a DLT triangulation, and verify the result to sub-pixel.

LumaSpec Vision11 min read Share on LinkedIn

On this page 10 sections
  1. 01What the triangulation solver needs from you
  2. 02What you need from the calibration run
  3. 03Assemble the two projection matrices
  4. 04Undistort the points before they touch P
  5. 05Solve for the 3D point
  6. 06Worked example, end to end
  7. 07Verify before production
  8. 08Common mistakes, most frequent first
  9. 09Calibrate a pair or buy the calibration done
  10. 10Questions

You have run a stereo calibration and hold two 3×3 intrinsic matrices, two distortion vectors, a rotation R and a translation t — while your triangulation routine asks for two 3×4 projection matrices instead. The assembly is direct once you fix the world frame: put the origin at camera 1 and write P = K[I|0] and P' = K'[R|t]. This guide assembles both matrices step by step, prepares the matched points, solves the linear system, and works a complete numeric example you can reproduce in a spreadsheet. It is written for vision engineers implementing direct linear transform (DLT) triangulation on their own calibrated pair, and it ends with the checks that separate a correct implementation from one that returns plausible nonsense.

What the triangulation solver needs from you

A DLT triangulation takes, per 3D point, two matched pixel measurements and two camera projection matrices. A projection matrix P has 3 rows and 4 columns and maps a homogeneous world point X (4×1) to homogeneous pixel coordinates (3×1); the pixel is the first two entries divided by the third. Text that says 4×3 means the same matrix counted columns first.

Your intrinsics K cannot serve alone. K maps points that already sit in camera-1 coordinates onto pixels, but it says nothing about where the camera sits in the world — that is what the fourth column carries. Assembling K together with the extrinsics is camera resectioning: building a projection matrix from intrinsic and extrinsic parameters. Because you calibrated the rig, R and t are explicit, so you avoid the scale ambiguity that comes with factorising a fundamental matrix instead.

What you need from the calibration run

  • K and K', each 3×3 with entries in pixels: focal lengths on the diagonal, principal point (cx, cy) in the last column.
  • One distortion vector per camera, the radial and tangential terms your calibrator prints.
  • R (3×3) and t (3×1), defined so that a point in camera-1 coordinates maps to camera-2 coordinates as X2 = R·X1 + t. Confirm which direction your calibrator uses — this convention is the most common trip in the whole assembly.
  • All of it from one calibration session. K from one run with R, t from another gives matrices that are each sane and jointly wrong; the input-side failure modes are catalogued in camera calibration input errors: matrices and point arrays.
  • The two matched pixel points, one per image, undistorted before use.
  • Any routine that computes a singular value decomposition (SVD).

The essential and fundamental matrices are not needed on this path. t carries the metric scale: its unit is the unit you measured the calibration squares in, and the triangulated points come back in that unit. When the unit slips, the whole cloud scales with it — the same failure seen later as a point cloud with the wrong scale.

Assemble the two projection matrices

Work through these five steps once per rig:

  1. Fix the world frame at camera 1. Camera 1's extrinsics are then identity rotation and zero translation, so P is K with a column of zeros appended. Where Z is measured from follows from this choice; choosing where Z is measured from in a stereo frame covers the consequences downstream.
  2. Build the 3×4 block [R | t]: R in columns 1–3, t as the fourth column. If your calibrator returned the rotation as a three-number vector, convert it to a matrix first; rotation vector to rotation matrix, and the units of t shows the conversion.
  3. Check the direction of R and t. They must map camera-1 points into camera-2 coordinates, X2 = R·X1 + t. If your library reports the inverse pose instead, invert before use: R becomes Rᵀ and t becomes −Rᵀt. Applied backwards, the reconstruction mirrors through the origin.
  4. Multiply: P' = K'·[R | t]. The product is 3×4, the same shape as P.
  5. Check the rotation. det(R) must equal +1. A determinant of −1 means a transcribed sign or swapped rows; fix it before triangulating anything.
P = K [ I | 0 ] (3 x 4) P' = K' [ R | t ] (3 x 4), X2 = R X1 + t

Undistort the points before they touch P

Distortion coefficients describe the lens; P and P' describe ideal pinhole cameras. Push each detected point through its own camera's undistort routine with that camera's vector, and use the corrected pixel coordinates everywhere after. Two conventions to keep straight: if your pipeline produces normalised image coordinates rather than pixels, set cx and cy to zero in K so the matrix matches the points; if you triangulate from rectified images, use the calibration the rectification produces, not the originals — feeding rectified pairs with their matching calibration is that problem in detail.

Solve for the 3D point

The method is the direct linear transformation: stack linear equations from the projective relations and read the point out of the null space. Each camera contributes two independent rows. With (x, y, 1) the homogeneous pixel and P1…P3 the rows of P, the cross product x × (P·X) = 0 gives:

row 1: x3*P1 - x1*P3 row 2: x3*P2 - x2*P3 row 3,4: same pattern with P' and (x1', x2', 1) A (4 x 4) Xh = 0 Xh = right singular vector of A for the smallest singular value X = ( Xh1/Xh4, Xh2/Xh4, Xh3/Xh4 )

Stack the four rows, solve, dehomogenise; the point lands in t's unit. Two practical notes. For numerical conditioning, centre and scale the pixel coordinates before building A — subtract the mean, scale so the mean distance from the origin is √2, undo on the result — because with focal lengths near 800 px the raw system is badly scaled. And each additional camera just adds two rows: three views give a 6×4 system solved the same way.

DLT minimises algebraic, not geometric, error. That bias stays small against pixel noise when corners are detected to sub-pixel; refine only if the residuals in the next section tell you to.

Worked example, end to end

Take a 1280×960 pair with identical intrinsics fx = fy = 800 px, cx = 640 px, cy = 480 px, parallel axes (R = identity) and t = (−200, 0, 0) mm, so camera 2 sits 200 mm along camera 1's x-axis. A part corner sits at X = (100, 0, 800) mm in camera-1 coordinates.

Quantity Value
Image size 1280 × 960 px
K = K' fx = fy = 800 px, cx = 640, cy = 480
R identity
t (−200, 0, 0) mm, baseline B = 200 mm
World point X (100, 0, 800) mm

The two matrices come out as:

P = [ 800 0 640 0; 0 800 480 0; 0 0 1 0 ] P' = [ 800 0 640 -160000; 0 800 480 0; 0 0 1 0 ]

The −160000 entry is 800 px × (−200 mm): the translation absorbed into the homogeneous column.

Project X = (100, 0, 800, 1) through both. Camera 1: first entry = 800·100 + 640·800 = 592000, second = 480·800 = 384000, third = 800; pixel = (592000/800, 384000/800) = (740, 480). Camera 2: X2 = (100−200, 0, 800) = (−100, 0, 800) mm; first entry = 800·(−100) + 640·800 = 432000; pixel = (432000/800, 384000/800) = (540, 480). Disparity = 740 − 540 = 200 px, and the rectified shortcut Z = f·B/d = 800·200/200 = 800 mm confirms the input depth. That shortcut is the special case of parallel axes and a shared principal point; disparity scale and calibration for depth estimates covers the general pairing.

Now solve it as the DLT would. The four rows work out to:

[ 800 0 -100 0 ] [ 0 800 0 0 ] [ 800 0 100 -160000 ] [ 0 800 0 0 ]

Rows 2 and 4 force the second coordinate to zero. Row 1 gives 800·X = 100·Z; row 3 gives 800·X + 100·Z = 160000. Substituting the first into the second: 100·Z + 100·Z = 160000, so Z = 800 mm and X = 100 mm. The solve returns exactly (100, 0, 800) mm — a zero-residual recovery, as it must be with noise-free pixels.

Noise behaves differently. Differentiating Z = f·B/d gives δZ = Z²·δd/(f·B). With δd = 0.2 px of matching noise on this rig: at Z = 800 mm, δZ = 800²·0.2/(800·200) = 0.8 mm; at Z = 1600 mm, δZ = 1600²·0.2/160000 = 3.2 mm. Doubling depth quadruples depth noise; doubling baseline or focal length halves it. So buy depth accuracy with baseline and focal length, keep the working distance within a few multiples of the baseline, and stop expecting extra megapixels to fix a thin baseline.

Verify before production

  • Reprojection residual. Push the triangulated X back through P and P'. With decent sub-pixel corners both residuals should land under about 0.5 px; several pixels means distorted points went in or a matrix is wrong.
  • Known point. Place a target corner at a measured stand-off — 800.0 mm on a steel rule, say — triangulate it, and expect Z within several times the 0.8 mm noise floor computed above, not within the noise floor itself.
  • Rotation and units. det(R) = +1, and the reconstruction in the same unit as t. An answer uniformly 25.4× off points at inches versus millimetres in the target squares.
  • Cloud attitude. A point cloud that leans or shears is the signature of a wrong extrinsic; the checks in why a triangulated point cloud comes out tilted apply to any triangulated rig.

Common mistakes, most frequent first

  1. Skipping undistortion. Points carry the lens distortion; P does not. The result is a smooth depth bias that grows toward the image edges.
  2. Using the inverse pose unchanged. If X1 = R·X2 + t is your convention, triangulating with P' = K'[R|t] mirrors the scene. Invert first: Rᵀ, −Rᵀt.
  3. Mixing calibration sessions. K from one run with R, t from another: residuals look random and reproduce consistently.
  4. Convention mismatch on points. cx, cy left in K while feeding normalised coordinates, or the reverse, shifts every triangulated point by a constant amount.
  5. Single precision through the solver. The null-space vector degrades quietly; carry points and matrices in double precision — see point data types in stereo triangulation.
  6. Forgetting the homogeneous division. The raw output of P·X is (592000, 384000, 800), not a pixel; the division by the third entry is the step.
  7. Rectified images with the original K. Rectification remaps the pixels; the intrinsics must be remapped with them.

Calibrate a pair or buy the calibration done

For your own rig, choose two cameras with global shutter — rolling shutter skews anything that moves during the exposure, and the triangulation inherits the skew — plus a hardware trigger so both exposures land together, and a printed target whose square size you measure carefully in the unit you want t in. GigE C-mount area scan cameras in the 2–3 MP class are the workhorse choice; sort candidates by shutter, mount and interface in the area scan cameras category, or narrow a pair quickly with the camera configurator.

One representative global-shutter, GigE, C-mount part for such a pair:

Area scan camerasHikrobot MV-CU020-90GC 2.5 MP Area Scan Camera, GigE, GMAX4002, ColorMV-CU020-90GC
Sensor format
1/1.7"
Lens mount
C-mount
Interface
GigE
Resolution
Up to 3 MP
$309.00View partBuild a system with it

Mount two of them on a rigid beam at your baseline — 200 mm in the worked example — trigger together, calibrate, and the five steps above are the whole integration.

If owning calibration is not what your team should be doing, buy it done. The 3D sensors line covers laser profilers and stereo or structured-light cameras that ship factory-calibrated and hand you depth maps or point clouds directly, with the projection matrices inside maintained for you. Depth that jitters on such a module usually traces to synchronisation and inertial timing rather than your matrices; stereo depth-jitter troubleshooting guide walks that diagnosis on the module side. Three representative families:

Specification MIS-DP3850-01P MV-DLS300P-04 MV-DPS200P-02
Repeatability (Z-axis) 1.68 μm @ data that sensor tests gauge block on optical platform1 0.03 mm @ 600 mm 0.02 mm @ 300 mm
Laser wavelength 405 nm 450 nm –
Laser safety class Class3R Class 2 –
Data interface Gigabit Ethernet (1000 Mbit/s), compatible with Fast Ethernet (100 Mbit/s) Gigabit Ethernet (1000 Mbit/s) Gigabit Ethernet (1000 Mbit/s)
Digital I/O 12-pin M12 interface provides power and I/O, including opto-isolated input × 2 (Line 0/9), differential input × 2 (Line 3/6), and differential output × 1 (Line 1) 12-pin M12 connector provides power supply and I/O, including opto-isolated input × 3 (Line 0/3/6), opto-isolated output × 3 (Line 1/4/7) 12-pin M12 connector provides power supply and I/O, including opto-isolated input × 3 (Line 0/3/6), opto-isolated output × 3 (Line 1/4/7)
Power supply 24 VDC 24 VDC 24 V DC
Power consumption 17.2 W @ 24 VDC 15 W @ 24 V DC 11 W @ 24 V DC
Dimension 144.5 mm × 96 mm × 58 mm 236.3 mm × 79.8 mm × 98.5 mm 126 mm × 90 mm × 50 mm
Weight Approx. 950 g (2.1 lb.) Approx. 1.3 kg 0.74 kg
Ingress protection IP67 IP65 IP67
Temperature Working temperature: 0 °C to 45 °C (32 °F to 113 °F); Storage temperature: –30 °C to 80 °C (–22 °F to 176 °F) Working temperature: 0 °C to 45 °C (32 °F to 113 °F); Storage temperature: –30 °C to 80 °C (–22 °F to 176 °F) Working temperature: 0 °C to 45 °C (32 °F to 113 °F); Storage temperature: –30 °C to 80 °C (–22 °F to 176 °F)
Humidity 20% RH to 85% RH (no condensation) 20% to 85% RH (non-condensing) 20% to 85% RH (non-condensing)
Price $12,379.00 · quoted $16,579.00 · quoted $8,829.00 · quoted

Values as printed in each product's data sheet; open a part for its drawing and full specifications.

Decisions, then. Implementing on your own calibrated pair: fix the world origin at camera 1, build P = K[I|0] and P' = K'[R|t] with the direction check in step 3, undistort first, and hold the implementation to sub-half-pixel reprojection residuals on a known point. Buying instead: take the factory-calibrated 3D sensor route and spend your effort on trigger timing and point-cloud handling — applying per-axis scale factors when a depth image becomes a point cloud is the next conversion down that path, and decomposing a projection matrix back into its intrinsics is the inverse of the assembly this guide walks. More on both routes lives on the 3D topic page, and the contact page is where a proposed camera pair or sensor pick gets reviewed before you order it.

Questions

Why is the projection matrix 3×4 when K is 3×3?

K maps a point that already sits in camera coordinates onto pixels, once you divide by depth. P must also place the camera in a world frame, and the extra fourth column carries the translation, so a 4-entry homogeneous world point maps straight to homogeneous pixels. Texts that write 4×3 mean the same matrix counted columns first; the maths is identical.

What units is the triangulated 3D point in?

The unit you measured the calibration target in. Pass the checkerboard square size in millimetres and X comes back in millimetres, because t inherits that unit; K stays in pixels regardless. A reconstruction uniformly 25.4× too large or small almost always means inches went in one side, or a square size was passed in pixels.

Do I still need to undistort points if distortion looks small?

Yes. Near the image centre the correction can be a fraction of a pixel, but at the edges of a wide C-mount lens it reaches several pixels, and each pixel of image error becomes millimetres of depth error at working distances. Running the undistort routine costs almost nothing and removes the largest systematic term.

Can the solver handle more than two cameras?

Yes. Each view contributes two independent rows, so N cameras give a 2N×4 system solved by the same SVD null-space step. Extra views average down matching noise and break ties when the point sits close to one camera's ray, provided every camera is calibrated into the same world frame and gets its own P.

Is DLT accurate enough without the optimal refinement?

DLT minimises algebraic error rather than geometric image distance, so the estimate carries a small bias under noise. With sub-pixel corners, undistorted points and a baseline comparable to the working distance, the bias stays well inside the depth noise itself: 0.2 px of matching noise on the 200 mm example puts 0.8 mm of depth noise at 800 mm. Let reprojection residuals decide whether to add refinement.