Fuse depth frames into one 360° point cloud scan

A depth camera pointed at a part sees, at best, half of it. Move the camera around the part — or spin the part on a turntable under a fixed camera — and every frame adds surface, but each frame arrives in the camera's own coordinates and the frames refuse to line up. Fusion is the fix: estimate a camera pose for every frame, pour each depth image into one shared voxel volume, and read out a single gap-free cloud or mesh at the end.

Below are the pipeline stages — frame-to-model ICP, TSDF integration, loop closure — and the three numbers that decide whether a 360° scan closes inside tolerance: the 30% overlap floor, a drift budget of about half the depth noise per view, and a voxel size matched to that same noise. A worked 16-view example runs the arithmetic end to end.

What a fused scan gives you

The output is one coordinate frame holding every surface the camera saw, not a stack of frames to align afterwards. One run yields three artifacts: a fused point cloud for inspection and CAD comparison, a mesh for reverse engineering or printing, and the camera trajectory itself, which you can replay to rescan the same part next week.

Volumetric fusion differs from post-hoc alignment. It merges frames as they arrive, so each new view corrects the model already built rather than fighting it. If your views come from fixed stations instead — a scanner moved between mounts, or two cameras on a bracket — that is static multi-view merge, and merging point clouds from multiple viewpoints of one object is the article for it. Aiming one fixed camera repeatedly at the same scene to fill gaps and average noise is simpler again; merging multiple depth frames into one gap-free point cloud covers that case.

The pipeline: acquire, register, integrate, extract

The same four stages run whether the code is a vendor demo or your own Python:

1. Acquire. Stream depth frames at planned poses: indexed turntable steps, or a slow continuous orbit with the camera fired per increment. Clamp the depth range first so background surfaces never enter the volume; clamping depth range on a 3D camera to crop the point cloud shows the setting.
2. Register. Align each new frame to the model built so far with frame-to-model ICP. Poses live in the camera's optical frame; keep the optical frame versus camera frame transform straight before handing poses to other software.
3. Integrate. Project each depth frame into a TSDF volume — a grid where every voxel stores its signed distance to the nearest surface plus a confidence weight. A surface seen from several views averages together, so noise and small registration errors wash out instead of stacking.
4. Extract. Marching cubes turns the volume into a mesh; sampling voxel centers gives a plain cloud. This is also where loop closure lands: add the constraint that the last view matches the first, and a pose-graph optimization spreads the leftover error around the whole circle instead of dumping it at the seam.

Frame-to-model ICP sets the drift budget

Iterative closest point registration takes two clouds, finds nearest-neighbour correspondences between them and solves the rigid transform that best aligns the pairs — a least-squares fit, the same estimator behind plane fitting on 3D point clouds. Two choices dominate the accuracy you end up with.

Register every frame against the accumulated model, not the previous frame. Frame-to-frame chaining stacks each error on the last, and drift grows in proportion to the view count. Frame-to-model re-anchors every view to a surface that already averages many frames, so the per-view residual lands below the raw depth noise: half the noise is a realistic planning figure, because the least-squares fit averages thousands of correspondences per frame.

Independent registration errors add in root-sum-of-squares, like a random walk:

with r the per-view residual and N the number of views in the loop. More overlap means more correspondences and pulls r down; a part with good geometric features holds r near its floor; sparse, featureless surfaces push it up until tracking is lost outright.

TSDF integration: voxel size and memory

The voxel size is the resolution ceiling of the whole scan, and no amount of frames gets past it. Set it at the depth noise for your stand-off, or half that for finely featured parts: coarser voxels discard detail the frames carry, finer voxels mostly integrate noise at a higher memory bill. Keep the truncation distance — the band around the surface inside which voxels update — at 3 to 5 voxel widths, wide enough to absorb moderate registration error, narrow enough that thin walls do not bleed into each other.

L, W and H are the part's bounding box, m a margin (10 mm is generous for a bench part) and v the voxel size; 8 bytes per voxel covers the stored distance and its weight. The worked example below runs the numbers: even at half-millimetre voxels a bench-scale volume costs tens of megabytes, so memory never drives the hardware choice — registration compute does.

Worked example: 16 views around a 100 mm housing at a 2 mm seam budget

The inputs: a housing 100 mm tall and 60 mm in diameter on an indexed turntable, a typical bench-scale first-article job. The depth camera covers 60° horizontally — a common close-range field of view; substitute your sensor's value — with depth noise around 1 mm at the 0.4 m stand-off, typical of structured-light and ToF heads at bench range and measurable with the flat-target check below. The target: a seam no wider than 2 mm where the scan closes.

Step 1 — view count from the overlap floor. Adjacent views must share at least 30% of their surface for ICP to hold correspondences, so the angular step cannot exceed 70% of the field of view:

step ≤ 0.70 × 60° = 42°, so N ≥ 360° / 42° = 8.57 → 9 views minimum

Sixteen views at 22.5° steps gives overlap of 1 − 22.5/60 = 62.5% — comfortably above the floor, with correspondences to spare for steady tracking.

Step 2 — drift against the seam budget. Per-view residual r ≈ σ/2 = 0.5 mm. The random-walk sum around the loop:

gap ≈ 0.5 mm × √16 = 0.5 mm × 4 = 2.0 mm

Sixteen indexed views with frame-to-model ICP land exactly on the 2 mm budget, so the scan closes without help. Add loop closure anyway: it redistributes the 2 mm across all 16 view pairs, about 0.125 mm per pair — an order of magnitude below the surface noise, invisible in the fusion. Chained frame to frame instead, the same scan opens 0.5 mm × 16 = 8 mm, four times over budget.

Step 3 — volume size. Bounding box 60 × 60 × 100 mm plus 10 mm margin per side = 80 × 80 × 120 mm. At v = 1 mm: 80 × 80 × 120 = 768,000 voxels × 8 B ≈ 6 MB. At v = 0.5 mm: 160 × 160 × 240 = 6,144,000 voxels × 8 B ≈ 49 MB. Both trivial.

Decision. For this part and budget: 16 indexed views, frame-to-model ICP, 0.5–1.0 mm voxels, loop closure on. If the drawing tightens to a 0.2 mm seam, no view count reaches it on 1 mm raw noise — fusion averages noise but cannot invent detail. Cut σ first: shorter stand-off, better surface contrast, controlled ambient light. Beyond that, the part belongs on a profiler-class sensor.

Picking the depth camera and the compute

Let part size and required detail choose the camera; the fusion pipeline is identical underneath. For bench-scale parts, projected structured light delivers the densest, lowest-noise depth at close range:

When the subject outgrows the bench — totes, pallets, a robot cell — a time-of-flight head holds the whole volume in one frame, trading close-range detail for reach:

Both families sit in 3D sensors, and the camera finder filters them by type and working range. Where the drawing cares about one profile repeated around the part, a laser profiler swept once per rotation is the other classic route: it scans profiles instead of fusing frames and skips registration altogether.

The fusion process itself is modest on memory and heaviest on CPU while registering. A vision controller sized for multi-camera lines carries the camera links, the lighting outputs and the fusion process in one enclosure:

The MV-VC5525-256G70-NN pairs a 12th-generation Core i5 with 32 GB of memory and PoE camera ports, plus 8 built-in light channels for the illumination side of the cell. The remaining vision controllers cover smaller and larger stations, and the system configurator walks the chain from camera to controller in order.

When frame fusion is the wrong tool

Fusion answers "one camera, many poses". Reach for something else when:

• One face carries the inspection. A fixed camera and a height map finish in one shot; turning a 2.5D height map into a 3D point cloud covers that path.
• The drawing puts a profile tolerance on the scan. Certifying against a tolerance needs a defensible point density, which the CMM point cloud density guide for profile tolerances works through.
• The part cannot rotate, or it flexes. Simultaneous views from a calibrated rig replace the sweep; calibrating two depth cameras for combined point clouds starts that build.
• You would rather hand-hold the camera. Free-hand capture swaps the turntable for a SLAM tracking front end; when that returns no cloud at all, the ROS visual SLAM point cloud failure walkthrough lists the usual causes.
• The surface defeats depth. Transparent, mirror-finish and deep-recessed faces return weak or misleading depth; changing the finish or the lighting beats any amount of fusion.

Checks before the first full scan

Run these quick checks in this order; each one protects a number from the sections above.

1. Measure single-frame noise at your stand-off. Aim at a flat matte white target, record 50 frames, take the standard deviation as σ. Every budget in this guide scales with it.
2. Project the overlap. For each planned pose, confirm at least 30% of the previous view's surface returns; push toward 60% on featureless parts.
3. Watch the residual live. A mean ICP residual that creeps upward means tracking is slipping or the model is drifting; stop and re-anchor before integrating further.
4. Verify scale. Measure a gauge block in the fused cloud. Scale error integrates into a wrong-sized model, and depth scale mismatch explains clouds that cut off or shrink at close range.
5. Close the loop and measure the seam. Compare the last-to-first overlap region against the drift budget before trusting the mesh.
6. Check the exported numbers. Validate the finished cloud against a known dimension in your tooling; measuring distance and bend angle from point clouds in Python shows the measurement patterns.

When the rehearsal scan passes, the production loop is set: same poses, same settings, one fused volume per part. The 3D vision guides carry the surrounding steps — noisy stereo point cloud fixes if the raw frames are rough — and the contact page is open if you want a second opinion on the view plan before committing hardware.