Merge multiple depth frames into one gap-free point cloud

A single depth frame always leaves holes: occlusions, shiny patches and noisy returns mean the mesh you build from one shot has gaps and ragged edges. The fix is to capture a fixed set of frames, fuse them into one point cloud, and export that cloud as a single PLY file. This guide walks through the capture loop, the alignment step, the averaging arithmetic, and the checks that tell you the fusion worked, using nothing beyond your camera's SDK and a point cloud library.

Decide how the frames relate to each other

Everything downstream depends on one question: did the camera or the part move between frames?

• Camera and part both static. Every frame is already in the same coordinate frame. You can average the depth images directly and skip alignment entirely. This is the simplest and most accurate case.
• Camera moves on a robot, stage or scanner gantry. You know the transform for each pose from the robot controller or encoder. Apply the known transform to each cloud, then merge.
• Handheld or free-moving camera. You do not know the transforms. Estimate them with ICP (iterative closest point) between consecutive frames, then chain them. Expect more error and more cleanup.

If you are combining views from several cameras rather than several frames from one camera, that is a calibration problem first: see multi camera extrinsic calibration in one shared coordinate frame for the shared-frame set-up.

Capture an exact number of frames

Write the capture as a counted loop, not a while-loop with a stop condition. A fixed count makes the acquisition reproducible: the same five or thirty frames every run, easy to diff when something changes.

The pattern is plain: start the pipeline, request N frames in a for-loop, convert each depth frame to a point cloud, and accumulate. Between two and five frames per second of scene time is plenty for a static part; more frames only help if the camera moves to new viewpoints.

For camera choice, a structured-light model such as the MV-DPS200P-02 projects a known pattern, which gives dense depth on bland surfaces where passive stereo struggles. For volume and logistics work where the part moves on a belt, the logistics 3D camera range pairs naturally with frame fusion across a short travel window.

Choose the fusion method

Static scene: average the depth images

Keep one camera pose and average in depth space. For each pixel, take the median or the trimmed mean of valid depths across frames. The median rejects the occasional bad return, which a plain mean does not.

A useful estimate of the noise reduction:

Ten frames cut random depth noise to about a third of a single frame (1 / sqrt(10) ≈ 0.32). That is the whole reason fusion helps meshing: edges get crisper and thin gaps from noisy returns close up.

Known motion: apply the transform, then merge

When the camera rides a robot or a stage, transform each cloud into the base frame with the pose recorded at capture time. The standard form of a rigid transform:

where R is the 3x3 rotation and t the translation of that pose. Apply it per point, then concatenate the clouds. Accuracy here is limited by the pose source: a repeatability of ±0.05 mm on the robot means fused features finer than that are not trustworthy.

Unknown motion: ICP between frames

With a free-moving camera, register consecutive clouds with ICP: find closest-point pairs, solve for the best-fit rigid transform, iterate until the match error stops dropping. Chaining pairwise transforms gives a full trajectory. Run ICP only after voxel-grid downsampling, and always check the residual; a residual several times larger than the single-frame noise means the registration failed.

For a walkthrough of what misalignment looks like in the result, see why a laser triangulation point cloud is tilted: calibration checks — the same overlay checks apply to ICP output.

Worked example: fusing five frames of a static part

Say you inspect a machined bracket with a MV-DPS200P-02, camera and part both fixed:

1. Capture 5 depth frames at the working distance, roughly 500 ms apart.
2. For each pixel, collect the valid depths across frames. Where fewer than 3 of the 5 frames return a valid depth, mark the pixel invalid.
3. Take the median of the valid depths per pixel.
4. Convert the fused depth map to a point cloud with the camera intrinsics.
5. Export as a binary PLY.

If the single-frame depth noise is 0.6 mm, the fused noise is about 0.6 / sqrt(5) ≈ 0.27 mm. Pixels that were marginal in every frame — a dark chamfer, a specular face — stay invalid, which is correct: no fusion recovers depth the sensor never returned. For surfaces like that, change the lighting rather than the frame count.

Export one PLY

Write all fused points to a single binary PLY with XYZ plus any per-point properties you need (confidence, colour, normal). Keep the header simple and consistent; most meshing tools want one cloud with one origin.

Two practical notes:

• Store confidence or validity counts per point if you can. Later stages can then down-weight the pixels averaged from few frames instead of treating all points equally.
• For very large fusions, voxel-grid downsample before export. Our guide on how to store point cloud data in a voxel grid for 3D inspection covers the grid size trade-off; a voxel edge of 1–2× your target mesh resolution is the usual starting point.

Point density per area is worth computing before you commit to a frame count, especially if the fused cloud feeds a tolerance check: work the same arithmetic shown in the noise formula above, dividing your minimum feature size by the fused point spacing to confirm the fused cloud actually resolves the feature. The overlay and flat-surface checks below then confirm it in practice.

Verify the fusion before meshing

• Overlay check. Render two individual frames and the fused cloud in the same viewer. The fused cloud should be smooth where frames agree, not doubled or blurred. Doubling means a bad transform.
• Flat-surface fit. If the part has a known flat face, fit a plane to it in the fused cloud. The RMS residual should be close to the fused noise estimate from the sqrt(N) formula; if it is much larger, alignment error has crept in.
• Validity map. Render the validity count per pixel. Large dark regions mean the sensor never returned depth there — a lighting or reflectivity problem, not a fusion problem. Stereo point cloud visualization: check missing XYZ points shows how to spot and diagnose those missing regions.
• Edge crispness. Slice through a sharp edge. One frame gives a sloped, noisy edge; the fused cloud should show a narrow transition.

Common mistakes

• Fusing frames of a moving part. If the part moves between frames without a trigger and known timing, you are averaging several different shapes. Synchronize capture to the motion; the timestamp checks in depth and color synchronization: check the right timestamps apply to any multi-frame capture.
• Plain mean instead of median. One bad frame with 20% outlier depths drags the mean visibly; the median barely notices.
• ICP with default settings on sparse clouds. Loose correspondence thresholds happily converge to a wrong alignment that still looks plausible in a viewer. Always check the residual against your single-frame noise.
• Mixing resolutions. If you changed the depth resolution between captures, the frames do not share a pixel grid. Fuse in 3D space after conversion, or recapture at one resolution — see change 3D camera depth resolution and field of view at capture time.

Ordering and set-up checklist

• Confirm the scene is genuinely static, or that you have a pose source per frame.
• Pick a frame count: 5–30 for a static scene; more only with new viewpoints.
• Capture after warm-up, with fixed exposure and light.
• Use a median or trimmed mean for static fusion, known transforms for staged motion, ICP with residual checks for free motion.
• Export binary PLY with per-point validity or confidence.
• Run the overlay, flat-surface and validity checks before meshing.

When fusion is the wrong answer

If a single frame already covers the part and your only problem is noise on a few edges, fix the illumination or exposure first — it is cheaper and faster than adding a fusion pipeline. If the gaps come from occlusion, moving the camera to a second viewpoint and fusing two clouds beats pushing frame counts at the same viewpoint, where occluded geometry never appears in any frame. And if the part never stops moving, a single well-triggered capture synchronized to the motion is usually more reliable than frame-averaging a blurred sequence.

If you are specifying the sensor and capture set-up from scratch rather than tuning code, the 3D sensors range covers laser profilers, structured-light and ToF cameras, and the camera finder filters them by type and series. To match sensor, mount and processing into one build, start at the system configurator, or send the part details through the contact page and the applications team will review the capture plan with you.