3D visionHow-to

Fuse two depth camera point clouds into one map

Fuse two depth camera point clouds: extrinsic calibration into one frame, a rigid transform per cloud, real-time merging, a worked example and checks.

LumaSpec Vision12 min read Share on LinkedIn

On this page 10 sections
  1. 01Why won't two depth cameras share one 3D map?
  2. 02What transform connects two depth cameras?
  3. 03How do you measure the transform on a real rig?
    1. Which cameras suit a fixed two-camera rig?
  4. 04What is the calibration procedure, step by step?
  5. 05How do you apply the transform to each point?
  6. 06How accurate is a fused two-camera map?
  7. 07Where should two depth cameras overlap?
  8. 08How do you keep the merged map stable in real time?
  9. 09What should you check before you go live?
  10. 10Questions

Two fixed depth cameras — one at the front of a vehicle, one at the rear, or two walls of a pick cell — do not share a 3D map just because they stream side by side. Each camera instance builds its own map in its own world frame, and the result flickers and misaligns. The fix is one rigid-body calibration per camera into a single world frame, then your own merging code. This guide gives you the transform, a measuring procedure, a worked example with the arithmetic shown, the accuracy to plan for, and the checks to run before you trust the map in real time.

Why won't two depth cameras share one 3D map?

Because each camera instance builds its own map in its own world frame, anchored at its own imager: one device's mapper cannot write into another's map, so alternating updates make chunks flicker in and out of the scene.

Depth-camera SDKs open every device as an independent instance. Each instance keeps its own coordinate origin, its own chunk indexing and its own map storage, and nothing in the API lets instance B append to instance A's map. Run two mappers over one scene and the viewer shows parts appearing and vanishing as the two update passes interleave — the classic dual-camera flicker.

The fix is architectural, not a parameter. Treat the cameras as depth sensors: ignore each one's built-in mapper, export the raw point clouds, put both clouds in one world frame, and merge in your own process. Two frame subtleties come first: why point cloud axes differ from the camera frame, and how to move a cloud between the optical frame and the camera frame — ten minutes of reading that saves an afternoon of debugging.

What transform connects two depth cameras?

One rigid-body transform per camera: three rotation and three translation parameters — six degrees of freedom, no scale term — because depth cameras already report metric points, so only the pose between the two imagers is unknown.

Write it once and apply it to every point of that camera's cloud, every frame, for as long as the mount stays bolted:

p_world = R · p_cam + t T = [ R t ] [ 0 1 ]

R is a 3×3 rotation and t a 3×1 translation in meters when the clouds are in meters. Both clouds share the same unit and scale, which is what separates this from monocular SLAM: there is nothing to estimate beyond the six pose numbers. The order in the formula matters — rotating after translating misplaces the whole cloud by the full translation — and the rotate-then-translate walkthrough breaks the algebra down rule by rule.

How do you measure the transform on a real rig?

Rank the methods by what the rig gives you: a vendor multi-device utility if one exists, a planar target seen by both cameras second, mount measurement plus ICP third, and a helper camera when the two fields of view never overlap.

Vendor multi-device utility. Several depth-camera vendors ship a tool that estimates where the devices sit relative to one another so their data lands in one coordinate system: connect both cameras to one host, let it discover them, and walk slowly through the whole covered area while it watches a target move between the views. If your vendor ships one, it is minutes of work — use it first.

Planar target. Place a checkerboard or coded board where both cameras see it, detect the pattern in each camera's RGB or IR stream, and solve the registration between the two point sets. On a stiff indoor mount the residual lands in the low millimeters; the usual failure is target size, not the math — see why checkerboard calibration fails at distance.

Mount measurement plus ICP. Caliper or CAD the mount, seed the transform with those numbers, then refine with ICP on the overlapping surfaces. The seed only has to be within a few centimeters for ICP to converge, so tape-measure accuracy is enough to start.

No shared view. When front and rear cameras see nothing in common, fix a target beside each depth camera, add a helper camera that sees both targets, solve the two pairs, and chain the transforms through the rig frame. The helper is needed only during calibration; remove it afterwards. This beats mirror-based setups on set-up effort and accuracy, and the two-camera calibration walkthrough shows the chaining step by step.

Which cameras suit a fixed two-camera rig?

Any depth camera that exports raw points fuses — the transform does not care who made the sensor. Structured-light models give dense clouds indoors on diffuse surfaces; ToF models keep measuring when texture is poor. Two fixed-mount options from the 3D sensors range, one of each type:

3D sensorsHikrobot MV-DPS200P-02 Projected structured-light 3D cameraMV-DPS200P-02
Type
Stereo / structured light
$8,829.00 · quotedView and quote
3D sensorsHikrobot MV-DT01WMU ToF sensing cameraMV-DT01WMU
Type
Depth / ToF camera
Price on requestView and quote

If one station in the cell is a plain 2D camera instead, the fusion math is identical; the area scan cameras range covers that side of the shop.

What is the calibration procedure, step by step?

Ten minutes of bench work once the mount is rigid: one planar target in the shared view, ten synchronized frame pairs, a six-parameter solve, ICP refinement, and a known-length check before anything consumes the map.

  1. Bolt both cameras to one rigid frame. Confirm the mount does not move during capture, because stillness during capture decides image quality.
  2. One host, production modes. Connect both cameras to the same host and run the depth mode and resolution you will ship; a calibration taken in another mode shifts with it.
  3. Target in the overlap strip. For standoffs of 1.5–3 m, a 0.6–1.0 m board with 50–100 mm squares keeps every square at 25 px or more in both views, which is what corner detection needs.
  4. Grab at least ten synchronized frame pairs. Discard blurred frames; detect corners on the RGB or IR streams, where the pattern is crispest.
  5. Solve the six parameters. Register target to target; aim for under 2 px RMS corner reprojection — at a 2 m standoff that is about 5 mm on the cloud for a 700 px focal length.
  6. Refine with ICP on the overlapping surfaces and drop residuals beyond two standard deviations.
  7. Store the extrinsic as R and t in one configuration file, with units and frame names written next to the numbers.
  8. Validate on a known length. Stand a 1.000 m bar where both cameras see it; a rigid indoor rig should measure it within ±10 mm after ICP. Re-run this check after every remount — the same dimension-trueness discipline used to verify a depth camera on the line applies to fused clouds.

How do you apply the transform to each point?

Rotate each point, then translate it: p_world = R·p_cam + t. Reversing the two operations misplaces the entire cloud by exactly the translation vector — 4 m in the example below.

Take the surround-view rig: camera A at the front defines the world frame, and calibration put the rear camera B at t = (−4.000, 0.000, 0.000) m, yawed 180° about the vertical axis, so R = [[−1, 0, 0], [0, −1, 0], [0, 0, 1]].

A point in B's cloud at p_B = (−0.500, −0.300, 1.200) m — half a meter in front of B, 0.3 m off-axis, 1.2 m up:

  • Rotate: x = (−1)(−0.500) = 0.500, y = (−1)(−0.300) = 0.300, z = (1)(1.200) = 1.200.
  • Translate: p_world = (0.500 − 4.000, 0.300 + 0.000, 1.200 + 0.000) = (−3.500, 0.300, 1.200) m.

Sanity check: B sits 4 m behind A and the point sat 0.5 m in front of B, so world x must land at −3.5 m — it does. Run the same two operations on every point of every frame: a full-HD depth cloud is about two million 4×4 multiplies, single-digit milliseconds on a GPU and a few tens of milliseconds on one CPU thread.

How accurate is a fused two-camera map?

Budget 2–5 mm of misalignment from the calibration at 1–3 m standoff, and expect depth noise to dominate beyond that — roughly 5 mm at 1.5 m and 21 mm at 3.0 m for a typical indoor stereo depth camera, because depth error grows with distance squared.

The calibration share sits on a stiff mount, and an ICP pass on the overlapping surfaces trims the overlap residual toward 1–2 mm. The walkthrough on merging two depth-camera point clouds with affine transforms and ICP shows that refinement loop end to end.

Depth noise then takes over with distance:

σ_z ≈ z² · σ_disp / (f · b)

with f the focal length in pixels, b the baseline in meters and σ_disp the disparity noise in pixels. For a representative indoor stereo camera take f = 700 px, b = 0.12 m, σ_disp = 0.2 px:

  • At z = 1.5 m: σ_z ≈ (1.5² × 0.2) / (700 × 0.12) = 0.45 / 84 ≈ 0.005 m ≈ 5 mm.
  • At z = 3.0 m: σ_z ≈ (3.0² × 0.2) / 84 = 1.8 / 84 ≈ 0.021 m ≈ 21 mm.

Doubling the distance quadruples the noise. The decision that follows: the fusion step adds almost nothing to the error budget, so put the geometry that matters inside 3 m and let the calibration, not the merge, set the accuracy floor. A given sensor's own noise figures live on its data sheet; the formula gives you the shape to expect.

Where should two depth cameras overlap?

Keep at least a 0.5 m deep strip visible to both cameras at working distance — enough for the calibration target and an ICP surface — and mind the surround geometry: two opposing 90° FOV cameras leave two 90° wedges at the sides blind.

The FOV arithmetic decides coverage. A 90° horizontal FOV spans a quarter of the circle, so front and rear cameras aimed along one axis cover the two end quarters and blind the flanks; four cameras at 90° spacing close the ring only when each FOV is at least 90°. If the flanks matter and the budget stops at two cameras, a single camera on a rotating mount covers 360° another way, fusing frames over the turn — see the 360° point cloud fusion walkthrough.

The overlap strip does double duty: it hosts the calibration target during set-up, then gives the merged map a region where both cameras measure the same surface — where ICP has grip and voxel deduplication has to arbitrate. Aim it at mid-range, 1.5–2.5 m on a vehicle-scale rig, where depth noise is still single-digit millimeters.

How do you keep the merged map stable in real time?

Pair frames by timestamp within half a frame period — ±33 ms at 15 fps — keep the extrinsics fixed, and merge into one voxel grid; at 10 km/h a 33 ms mismatch smears surfaces by about 92 mm.

  • One capture thread or process per device. Two cameras grabbed from a single thread serialize each other and produce interleaved partial updates; give each device its own thread and its own error handling.
  • Pair before you transform. 10 km/h is 2.78 m/s, so a 33 ms mismatch moves the scene 2.78 × 0.033 ≈ 0.092 m. Hardware-trigger both cameras when the budget allows; otherwise match nearest timestamps and reject pairs wider than half a period.
  • Never re-estimate the extrinsics online. Scene-based drift correction warps the map; the mount is rigid, so the six numbers stay constant until the mount is touched.
  • Merge at 20–50 mm voxels for vehicle-scale maps — keep the voxel two to three times the depth noise at your far edge (21 mm at 3 m above, so 50 mm is comfortable). Nearest return wins inside the overlap strip; newest frame wins elsewhere. The recipes for merging multiple depth frames into one gap-free cloud and merging multiple viewpoints of one object slot in here.
  • Publish one cloud, keep the raws. The merged cloud feeds navigation; raw frames with timestamps feed the next calibration. Scaling past two cameras changes nothing in the math — the multi-camera point cloud fusion guide extends the same extrinsics to any number of cameras.

What should you check before you go live?

Five checks, five minutes: a known length within ±10 mm, no flicker over 100 static frames, extrinsics re-validated after every remount, end-to-end latency under 200 ms at vehicle speeds, and no saturated IR frames in your real lighting.

  • Known length. The 1.000 m bar spanning both views is the arbiter — one number that catches a wrong transform order, a unit mix-up and a slipped mount at once.
  • Static flicker. Hold the scene still for 100 frames; merged points stay inside one voxel. Flicker here means a built-in mapper is still writing.
  • Remount discipline. R and t describe the mount, not the camera; any service or knock invalidates them until the bar check passes again.
  • Latency. Capture-to-merged-cloud under 200 ms for a 10 km/h vehicle — 0.2 s × 2.78 m/s ≈ 0.56 m of travel at that latency.
  • Exposure. Check for saturated IR at the brightest lighting you will run; saturation costs depth quality before it costs the image.

For a new rig, shortlist sensors in the camera finder, size the pair in the build-a-system configurator, or send the mount sketch through the contact page and we will check the FOV overlap with you. More 3D recipes sit on the 3D topic page.

Questions

How should I store the rig extrinsic between two depth cameras?

Store one 4×4 matrix, or a quaternion plus a translation, per camera, with a units line — meters and radians — and the frame names. Do not trust Euler angles alone: the convention order differs between tools. Validate the file by transforming one point whose world position you measured with a tape; if it lands within centimeters, the file is correct.

How often do the extrinsics need re-checking on a vehicle?

Knocks move them, not temperature: a 0.5 m aluminum arm expands about 0.2 mm over a 20 °C swing (23×10⁻⁶/K × 0.5 m × 20 K), far below a millimeter-class accuracy budget. Re-run the known-length check after every impact, after every remount, and weekly at first; it takes minutes, so make it a shift-start habit.

How many depth cameras can one host fuse in real time?

Two to four is the practical range. Budget one USB3 or GigE port per camera, since each carries roughly one HD depth stream, and leave GPU headroom for the voxel merge. Past four cameras, split capture across two hosts and fuse the already-transformed clouds over the network, which needs nothing but the configuration files.

Does structured light or ToF work better outdoors?

Neither holds its indoor range in direct sun: ambient IR washes out the projected pattern, and ToF pixels lose contrast, so both shorten range and drop points. Stereo depth with controlled exposure tolerates daylight best. For an outdoor vehicle, plan a shorter effective range, avoid aiming into the low sun, and validate the merged cloud against known geometry.

Why do the fused clouds still flicker after a good calibration?

Calibration cannot fix a mapper conflict. Check that no built-in mapping module is still running on either camera, that both clouds are transformed before the merge rather than after, that frame pairs are timestamp-matched, and that the overlap strip has one deduplication rule. Re-run the 100-frame static test after each change; the flicker should be gone.