Calibrate two depth cameras for combined 3D point clouds

You have two depth cameras aimed at the same scene from different angles, and each one returns a point cloud in its own optical frame. Fusing them into a single 3D reconstruction needs one thing: the rigid 4×4 transform between those frames. You get it from fiducial boards both cameras can see. This procedure places two AprilTag-style boards in the shared view, solves each camera's pose against them, chains the two poses into one world frame, and finishes with an ICP refinement over the overlapping surfaces. Done this way, two depth cameras land within 2-3 mm of each other at a 1 m range, and a gauge check proves it before the merged cloud goes anywhere near a production decision.

The transform you are solving for

Every depth frame a sensor hands you lives in that sensor's own coordinate system: Z along the optical axis, origin at the depth reference, not the mount and not the housing. If your code assumes otherwise, read transforming depth point clouds between optical and camera frames first, because a wrong frame convention quietly ruins every later step.

A tag seen by both cameras is the relay. solvePnP on the tag's four corners returns the tag's pose in each camera, T_C1_tag and T_C2_tag, and the camera-to-camera transform follows directly:

Choose the world frame once — camera 1's depth frame is the simplest, which makes camera 1's extrinsic the identity — and every cloud becomes p_w = T_w_ci * p_ci. The entire calibration output is one 4×4 matrix per camera plus the sync setup that makes the frames simultaneous. For the wider problem of hanging many cameras off one frame, see multi-camera extrinsic calibration in a shared coordinate frame.

What you need

• Two depth cameras with per-pixel depth and published intrinsics. A Time-of-Flight unit, a projected structured-light camera, or a laser galvanometer stereo unit all work, because the tag chain does not care how depth was produced:

• Rigid mounts. A 0.1° knock to a bracket shifts points at 1.2 m by about 2 mm, which is your whole error budget. The LS-MNT-0040 slide-table stand carries both cameras on one platform, with a 700 mm height and a 400 x 450 mm base. Lock every knob, then scribe-mark the positions so a future bump is visible.

• Two tag boards. Print AprilTag-family markers on matte paper and mount them flat on aluminium composite or foam PVC. Glossy laminate reflects the projector and punches holes in the depth map exactly where you need corners.
• Sync. If both cameras emit infrared patterns, plan hardware sync from the start; the setup is its own section below.
• Software. Open3D for clouds and ICP, a tag detector, and solvePnP — the pose side of this is the same pipeline described in fiducial tag pose estimation from detected corners.

Calibration procedure

1. Fix the geometry. Aim the cameras so their fields of view overlap by 30-50% of the smaller one: enough shared surface for ICP, and every point on the object measured twice. Set both depth range windows to cover the whole target zone; a cloud that ends abruptly at close range is a range-window setting, not a dead sensor (why your point cloud is cut off at close range). Placement rules of thumb are in the 3D camera working distance and positioning guide.

2. Place the boards. Both cameras must see at least one tag per frame; two tags at 30-60° to each other is the working minimum. Move the pair through 10-15 positions: centre and corners of the shared field of view, ranges from 0.6 to 1.5× your working distance, tilts up to about 30°.

3. Synchronise. Hardware-sync pattern-projecting cameras so they expose in alternating windows. Confirm in a live viewer: with both free-running, the overlap tears and doubles; synced, it settles.

4. Capture. Record 10-15 synced frame pairs with the boards in those positions. Boards and cameras stay still during each exposure pair.

5. Solve and chain. For each frame pair, run solvePnP per camera per tag, build the relative transform from the formula above, and take the median of all estimates. If the spread across your samples exceeds 5 mm in translation, a detection is bad — drop that frame rather than averaging it in.

6. Merge and refine. Turn each depth frame into a cloud in its own camera frame, apply the transforms, then let ICP finish: voxel-downsample at 2-5 mm, estimate normals, point-to-plane ICP with the correspondence limit starting at 10 mm and tightened to 2 mm. ICP should shift each cloud by 1-3 mm at most; a 10 mm correction means the tag poses were weak, so redo step 2 with bigger boards.

For the mechanics of combining clouds from several sensors into one gap-free model, see how to stitch point clouds from multiple cameras and merging multiple depth frames into one gap-free point cloud. A second end-to-end walkthrough of merging two depth camera point clouds runs the same pipeline with a different sensor pair.

Worked example: tag size and expected accuracy

Take a depth stream of 640 x 480 pixels with a 70° horizontal field of view. The focal length in pixels is:

A tag spans f_x × size / Z pixels. At Z = 1.2 m:

• 300 mm tag: 457 × 0.30 / 1.20 ≈ 114 px. A tag36h11 layout is 8 grid units across, so each black-white cell covers about 14 px — comfortably above the ~10 px per cell where corner detection is stable.
• 100 mm tag: 457 × 0.10 / 1.20 ≈ 38 px, under 5 px per cell. Corners jitter and poses swing; do not use it at this range.

Accuracy on the same numbers: 0.5 px of corner noise on a 114 px tag is an angular uncertainty of about 0.5 / 57 ≈ 9 mrad, so one tag locates a camera at 1.2 m to roughly 1200 × 0.009 ≈ 11 mm. Average 20 independent observations (2 boards × 10 positions) and the statistical component falls to 11 / √20 ≈ 2.5 mm; ICP then aligns surfaces down to the sensor's own noise floor. That is the basis of the 2-3 mm figure: it assumes tags above 100 px, a static rig, and both range windows covering the boards.

Sync two projectors without interference

Two cameras that both project infrared — speckle, coded light, or ToF modulation — read each other's pattern as their own. The overlap fills with flying pixels and doubled edges, and no calibration survives that input. The fix is hardware sync: master camera on the odd trigger pulses, slave on the even ones, each exposing only while the other's projector is dark. Trigger at half the single-camera frame rate so each keeps its full exposure time.

If a model has no trigger input, aim the two projector cones so they do not cross inside the measurement volume and accept a noisier overlap, or pair the projector camera with a passive stereo or laser-profiler sensor that emits nothing the other can see.

One streaming pitfall remains: with two depth streams on one host, frames can arrive swapped or mislabelled, which makes a good calibration look broken. The diagnosis is in mixing depth frames from two cameras: find the buffer bug.

Verify the merged cloud

• Gauge object. Lay a machined bar or ground box across the overlap, spanning at least 500 mm. Measure it in the merged cloud: within 3 mm of the calipered length at about 1 m passes; beyond that, recalibrate.
• Plane test. Hold a flat plate in the overlap and fit a plane. Points from the two cameras should interleave into one sheet; two parallel sheets a few millimetres apart is classic extrinsic error, and no filter hides it.
• Trust the geometry, not the score. A low reprojection error says the tag fit was self-consistent, not that the transform is right in the world — the argument is laid out in camera extrinsic calibration accuracy beyond reprojection error.

Run the gauge check after any knock, remount, or guard change, and before every shift during the first week of operation.

When the usual method fails

• No shared field of view. Tags must be visible to both cameras; non-overlapping rigs need a moving target or rotation-only methods, and both are fragile. Aim the cameras so they share at least a corner of the volume — the geometry is covered in choosing a stereo camera baseline distance.
• Glossy or wavy boards. Laminate glare and curled paper both destroy corners at the tag edge. Matte print on aluminium composite, checked against a straightedge.
• Tags under 100 px. Everything in the worked example scales with range: at 3 m a 300 mm tag is 46 px and single-frame error grows past 20 mm. Print tags at least 100 × Z / f_x across — 0.65 m at 3 m with f_x = 457 px.
• Springy mounts. A 0.1° knock costs about 2 mm at 1.2 m. Scribe marks turn an invisible bump into a visible one.
• Moving scene, sequential captures. The extrinsic holds, but clouds from different instants smear wherever the object moved. Sync the pair and merge only simultaneous frames.
• ICP-only fallback. With no boards at all, ICP on a rich static scene works when the overlap is 30-50% and geometry is varied; it drifts on flat walls and repeated shapes. Markers first, ICP second, always.

Pre-flight checklist

• Field-of-view overlap 30-50% of the smaller camera
• Both depth range windows cover the whole target zone
• Two or more tags visible to both cameras, each spanning at least 100 px
• 10-15 synced capture pairs across varied range, tilt, and field positions
• Spread of chained transforms ≤5 mm; ICP correction ≤3 mm
• Gauge bar within 3 mm at about 1 m; one plane sheet, no doubling
• Sync confirmed with both projectors running; mounts locked and scribe-marked

For the next rig, the camera finder filters depth and stereo sensors by type, and the 3D vision topic page gathers the related calibration and registration guides. To lay out the mounting first, start at the stand step of the configurator.