Multi-camera point cloud fusion with depth camera extrinsics

Two depth cameras see one part from two sides, but each cloud arrives in its own camera frame: same room, different origin. Fusing them is not a registration trick you run at the end of the pipeline — it is one 6-DOF extrinsic transform per camera, measured once against a target both cameras see, then applied to every depth frame. This guide walks a vision or controls engineer from rig layout through shared-target calibration, frame synchronization, transform and merge, with the error arithmetic for a two-camera rig around a 900 mm casting. You finish with a calibration that holds the seam to a few millimeters, and the checks that prove it.

What fusion actually requires

Point cloud fusion between two separate cameras is solved by one number set: a 3×3 rotation and a 3-vector translation, written together as a 4×4 matrix per camera, that maps its depth frame into a shared world frame. Once those matrices exist, fusion is matrix multiplication on every frame. Everything that can go wrong lives in getting them right and keeping them valid.

Factory calibration files will not do it. A factory file describes one unit: intrinsics, lens distortion, and the rectification of its internal stereo pair. Two cameras bolted a meter apart share nothing in those files — the transform between the two housings exists only after you measure it, in the final mounting. This is also why multi-device fusion pipelines load one extrinsics file per camera at startup: when that np.load(path) call finds no file, the script stops before the first frame. The fix is to run the calibration that writes the file, not to edit the fusion code.

The matrix does not survive rough handling. A 0.1° rotation slip of one camera shifts its cloud 1.7 mm at a part 1 m away, because 1000 × tan 0.1° = 1.7 mm — the size of the defects you are fusing images to find. Mount both cameras on one plate and recalibrate whenever a mount is loosened. The calibration routine itself is covered in calibrating two depth cameras for one combined point cloud; the rest of this guide is the fusion pipeline and its error budget.

The math you apply to every frame

Each camera's depth image becomes XYZ points through that camera's own intrinsics; the extrinsics then move the points into the world frame:

R is the 3×3 rotation, t the translation in millimeters, one pair per camera. Choose the world frame once and write it down. Anchored to camera A's depth frame, only camera B needs a stored matrix. Anchored to a fixture datum or robot base, both cameras get one, and the cloud survives either camera being remounted. For a two-camera inspection cell, anchor to camera A; when a robot consumes the cloud, anchor to the robot base.

Two more formulas set expectations before you buy anything:

The first is stereo depth noise: z in millimeters, σ_disp the disparity error in pixels, f the focal length in pixels, B the internal stereo baseline in millimeters. The second is motion error across the seam: v the part speed, Δt the skew between the two exposures.

Building the per-pixel XYZ in the first place means transforming depth point clouds between the optical frame and the camera frame. If the merged cloud needs color, per-camera registration of RGB onto depth is a separate step — the color source can be the depth head's own sensor or a separate area scan camera, as covered in fusing depth, color and infrared into one point cloud.

Keep one unit system. Depth pipelines return millimeters or meters; a meter-valued cloud fed into a millimeter world frame is scaled by 1000, and the merged result collapses to a dot or explodes off screen. Convert at the capture node, before the transform.

If no shared target is possible — long standoff, no common line of sight — iterative closest point on the overlap is the fallback, at the price of needing rich surface geometry; point-set registration explains the algorithm family.

What you need before you start

• Two depth cameras with factory intrinsics loaded, and a trigger input on each if the part moves.
• One rigid checkerboard or asymmetric circle target, 25-40 mm features, flat within 0.5 mm, feature size known to 0.1 mm.
• One mount plate: 8 mm aluminum is the practical floor for a rig that gets handled.
• A capture program that saves depth frames from both cameras with per-frame timestamps.
• A check artifact for the verification pass: a machined flat plate, a 5.00 mm step, or a 50 mm gauge sphere.

Worked example: two stereo cameras around a 900 mm casting

The part is a 900 mm aluminum casting with a 120 mm boss that hides the right flank from any single view. Two active-stereo depth cameras sit at ±40° around the surface normal, 1.0 m from the part, each with a 70° horizontal field of view and a 640-pixel-wide depth image.

Coverage and spacing. Each camera's footprint at 1.0 m is 2 × 1000 × tan 35° = 1400 mm, so 900 mm fits easily — the second camera exists for the occlusion, not the width. Lateral point spacing is 1400 / 640 = 2.2 mm, and the focal length in pixels follows from the same geometry: f = 320 / tan 35° = 457 px.

Depth noise. With a 75 mm internal baseline and 0.15 px disparity error: σ_z = (1000² × 0.15) / (457 × 75) = 150,000 / 34,275 = 4.4 mm. Double the internal baseline to 150 mm and it halves to 2.2 mm; double the distance to 2 m and it quadruples to 17.5 mm.

Extrinsics error. Calibrate with a 400 mm target held 0.6 m from the cameras. Its image spans 400 × 457 / 600 = 305 px; with 0.3 px reprojection error its orientation resolves to 0.3 / 305 = 0.00098 rad = 0.056°, which is 1.0 mm at the part, and its position to 0.3 px × 600 / 457 = 0.4 mm. The extrinsic contribution to the seam is √(1.0² + 0.4²) = 1.1 mm — an order below the 4.4 mm depth noise on each side.

Merge settings. Voxel leaf 3 mm, slightly coarser than the 2.2 mm spacing so the leaves fill; statistical outlier filter at 20 neighbors and 2σ after downsampling. Expected step across the seam on a flat wall: 3-5 mm, dominated by depth noise, with the 1.1 mm extrinsic offset underneath it.

The procedure, step by step

1. Freeze the rig. Both cameras on one plate, aimed so the fields overlap by at least 30% of each frame, with the seam crossing the feature you inspect. Torque the mounts, then mark their positions on the plate.

2. Fix units and intrinsics. Convert every depth frame to millimeters in the capture node, before anything else touches the data, and load each camera's factory intrinsics for the per-pixel XYZ projection.

3. Capture the shared target. Move the target through the overlap volume: 15-30 poses, tilts up to 45°, distances spread across the working range, the full pattern visible to both cameras in every pose. Vary the angles — one pose orientation cannot recover six degrees of freedom, the same reason one calibration point cannot recover two axes.

4. Solve the pair. Standard multi-view calibration returns each camera's target poses; the extrinsics you want carry camera B's frame onto camera A's across all of them. The solving algebra is laid out in multi-camera extrinsic calibration in one shared coordinate frame.

5. Save one file per camera. Write each 4×4 next to the camera's serial number. The fusion pipeline loads exactly these files; a startup failure inside that loader means this step was skipped.

6. Synchronize. Static part: per-frame timestamps are enough. Moving part: one hardware trigger line to both cameras, with skew budgeted as v × Δt — at 200 mm/s, 10 ms of skew smears the seam by 2 mm; at 1 m/s the skew must stay under 2 ms for the same 2 mm.

7. Transform and crop. Apply each camera's 4×4 to its cloud, then crop to the region where the view stays within 45° of the local surface normal. At 60° incidence a 2.2 mm footprint stretches to 2.2 / cos 60° = 4.4 mm and the surface goes sparse.

8. Downsample, filter, merge. Voxel-filter in the world frame (3 mm leaf in this rig), remove statistical outliers (20 neighbors, 2σ), then concatenate. Holes from a glossy surface are a capture problem, not a fusion one — start with missing points in a 3D camera point cloud. Filling one camera's own dropout from extra exposures is covered in merging multiple depth frames into one gap-free cloud.

How to verify the fused cloud

Span the seam with a machined flat plate and fit one plane to the overlap region. Read two numbers from the fit. The per-point RMS reflects depth noise — about 4 mm at 1 m for the worked rig. The difference between the two cameras' mean plane offsets isolates the extrinsics, because random noise averages out over the thousands of overlap points: pass below 1.5 mm, recalibrate between 1.5 and 3 mm, and treat anything above 3 mm as a mount that moved.

Then measure a known quantity. A 5.00 mm step machined into the check plate should read 5.0 ± 0.5 mm as the difference of two fitted plane means; a 50 mm gauge ball centered on the seam should hold its diameter within 1 mm. Reprojection error alone tells you none of this — extrinsic calibration accuracy beyond reprojection error lists the physical checks worth adding to an acceptance test.

Choosing the cameras for the rig

The depth method sets the seam you can reach. Active stereo with an IR pattern is the default from 0.3 to 3 m; projected structured light buys detail at short range; time-of-flight trades precision for volume; laser galvanometer stereo heads hold sub-millimeter depth at close range in fixed installations.

For a stationary fusion rig, the usual industrial starting points are the MV-DLS300P-04 laser galvanometer stereo camera and the MV-DPS200P-02 projected structured-light camera.

For room-scale coverage of static scenes, a time-of-flight head such as the MV-DT01WMU trades depth precision for covered volume, and in some cells a single head replaces the pair.

Before ordering, confirm that each candidate's depth image separates your smallest feature. The calculator turns feature size, field of view and working distance into the pixel count you need.

When two cameras are the wrong answer

One camera covers it. If a single head reaches your point spacing and sees past the occlusion, use one camera and delete the extrinsic maintenance. In the worked example the second camera exists only because of the boss; without it, one 640 px head already delivered 2.2 mm spacing over 1400 mm.

The part moves and you cannot trigger. Fusion without sync smears the seam by v × Δt, and no filter recovers it. A single triggered camera has nothing to sync against.

Conveyor lines suit line-scan 2.5D instead of two triggered area heads: the MV-HLVS-16K100-L60 covers 600 mm of line at up to a 120 kHz sampling rate with 16K resolution, and motion becomes resolution instead of blur. For the wider question of which method fits a task, read 3D scanning method selection for inspection; for robot picking off a belt, see 2D or 3D camera for conveyor pick and place.

Pre-flight checklist

• Both cameras on one rigid plate, positions marked after torquing — a knocked camera invalidates the extrinsics silently.
• Overlap at least 30% of each frame, seam crossing the inspected feature.
• Depth frames converted to millimeters before the transform; mixed units scale one cloud by 1000.
• 15-30 shared-target poses captured, full pattern in both views, angles varied.
• One 4×4 file per camera, named by serial number.
• Hardware trigger wired for moving parts, skew budget computed as v × Δt.
• Voxel leaf at or above the point spacing; outlier filtering after downsampling, in the world frame.
• Check artifact ready at the seam: flat plate, 5.00 mm step or 50 mm gauge sphere.
• Weekly five-pose spot check on the schedule.

Start the shortlist in the 3D sensors category, narrow heads in the camera finder, or run the pair through the configurator from the camera step. More 3D set-up guides sit in the Learn hub, and a part drawing sent through the contact page gets a rig review with numbers.