Noisy depth point cloud: hardware vs software registration

A depth cloud that looks crisp from the front and falls apart from the side is rarely a bad camera. In many RGB-D stacks the depth is resampled before your point-cloud node ever sees it: software registration reprojects depth into the color camera's frame while the hardware-registered stream is already in the chain. Every resampled pixel interpolates between its neighbors, and the damage stays hidden until you rotate the view. This guide shows how to recognize double registration, set hw_registered_processing and sw_registered_processing to the right combination, rebuild the cloud from native depth when the flags are not enough, and judge leftover noise with view-angle arithmetic.

Symptom: clean head-on, ragged from the side

From the front, surfaces read solid and edges sit where they should. Rotate the same cloud 60° to 90° and every edge grows a fringe of points that belong to no surface, while flat areas turn into fine fuzz.

Two signatures separate this from ordinary depth dropouts. Resampling fuzz tracks edges and thin structures and grows with the view angle; dropouts leave holes where the sensor measured nothing, head-on or not. If your cloud shows gaps rather than fringes, work through the missing points in a 3D camera point cloud fixes first, because no registration change will fill them. Judge raw exported points rather than a smoothed viewer render: smoothing can hide the fringe entirely.

Cause 1, most likely: depth registered twice

RGB-D wrappers expose two launch arguments that choose where depth-to-color alignment happens. Hardware registration aligns depth to color inside the camera or the vendor SDK, once, on the camera's own pixel grid. Software registration repeats the job on the host with a CPU node. Launch files for this camera class shipped with software registration enabled by default, so a pipeline consuming the registered stream while the hardware path was also active resampled every depth pixel twice. Interpolation at each resampled step blends depths across object edges and manufactures points that sit off any real surface: the flying pixels that fringe every boundary.

The fix costs one relaunch. In the RGBD nodelet launch file, set hw_registered_processing to true and sw_registered_processing to false, restart the driver and rebuild the cloud from the native depth topic. When the software path was the offender, the edge fringes disappear and the side view cleans up immediately. If the points never need color, leave both paths off and build geometry from native depth alone, as shown below.

Cause 2: a stale driver build

If the flags are already right and noise survives, suspect the driver build. In the reported case the distro-packaged stack lagged the camera's SDK generation, and the same camera streamed clean clouds under a current SDK sample. Test it exactly that way: run the maker's current SDK sample viewer against the same camera, and if its cloud is clean where the wrapper's is not, the wrapper build is the suspect. Pair versions deliberately: check the firmware generation the camera runs, install the matching SDK release from the maker's official download, and rebuild the wrapper against that SDK rather than against the distro package. A packaged driver against a newer firmware is the combination that produced unusable depth here.

Cause 3: real sensor noise, amplified by geometry

When the native depth stream itself is speckled, the fault sits upstream of any registration: stereo matching or structured-light decoding at the edge of its range, exposure set for the color camera instead of the projector, or a depth range that swallows background clutter. Those are sensing settings with their own fix list of thresholds, sensing modes and depth ranges. Dark, distant or mirror-like surfaces decode badly; a metal clamp on bare concrete sits well inside the reliable zone. Test head-on first: scatter within about 0.5% of the standoff is normal for this class, and anything larger belongs to the settings, not the pipeline.

Worked example: the floor clamp at 0.80 m seen at 80°

The case is the reported one: a metal clamp on a concrete floor under a short-range structured-light camera. Inputs: standoff Z = 0.80 m from lens to clamp, a typical desk-rig throw for this camera class; depth noise σ_z = 2.0 mm at that standoff, about 0.25% of Z and in line with published structured-light accuracy at close range, taken here as an estimate; and a viewing angle θ = 80° between the line of sight and the depth axis, the grazing side view where the cloud looked worst.

In words: a point that floats δz off the true surface appears, in a view angled θ from the depth axis, displaced sideways by δz × tan θ. Head-on, θ = 0°, tan θ = 0, and the depth error hides inside the viewing direction.

Arithmetic: tan 80° = 5.67, so the spread is 2.0 mm × 5.67 = 11.3 mm. The 2 mm error the front view forgives reads as 11 mm of lateral fuzz from the side.

Decision: amplification is tan θ, so a grazing view multiplies any depth fault about six-fold at 80°. Set the pass limit head-on instead. For this camera class, treat head-on scatter beyond about 0.5% of the standoff, here 4 mm at 0.80 m, as a sensing problem, and scatter below it as a pipeline problem. The clamp cloud stayed under that limit head-on and fell apart at 80°, which convicts the registration path, not the sensor.

Register once: build the cloud from native depth

When the launch arguments are out of reach, or when you want the cleanest pipeline, skip color registration for geometry entirely. Subscribe to the native depth image and deproject it with the depth camera's own intrinsics: fx, fy, cx and cy from the depth sensor, never from the color camera. The data layout guide sorts out which intrinsic set applies, and the deprojection guide covers the frame choice. Each pixel (u, v) with depth Z becomes a point directly:

Keep units consistent: Z in meters puts X and Y in meters. When the depth image lands in an array, watch the per-axis scale factors that expand or shrink raw values. Attach color afterwards, if a viewer needs it, by nearest-neighbor lookup: the geometry is never resampled, and a wrong color at an edge costs nothing. When a message genuinely needs per-point color, request the hardware-registered output once and fuse color and intensity from that single alignment.

Hardware paths without the reprojection

For a new station, sensor choice removes the fault class altogether. A 3D laser profile sensor builds depth from triangulated laser profiles on its own optics, so there is no color frame to reproject onto and no registration flag to get wrong:

For area depth where color is optional, a projected structured-light camera hands you the native depth stream this whole fix relies on; deproject it as above and registration never enters the pipeline:

Where the job is volume, occupancy or dimensioning on a conveyor, a ToF sensing camera measures range per pixel directly, which keeps the cloud native by construction:

Browse the 3D sensors category or the camera finder to filter by type, series and availability.

Prevention checklist

• Set hw_registered_processing to true and sw_registered_processing to false in every launch file you ship, and re-check both after each wrapper update, because updates restore the shipped configuration.
• Pin the SDK and firmware pair; never run a distro-packaged driver against a newer firmware generation.
• Build production clouds from native depth and depth intrinsics; treat registered streams as a viewer convenience.
• Let the camera stream about 30 frames after start-up before you judge noise.
• Measure noise head-on; convert side-view scatter back to depth noise by dividing by the tangent of the view angle.
• If the viewer looks clean but the exported file is ragged, compare the raw export against the render as the exported-cloud guide explains before touching settings.

If the noise survives single registration, a current driver and settled frames, point the system configurator at your standoff, target size and interface, and pick a sensor whose native depth meets your head-on budget. The 3D camera guides continue from there with deprojection, intrinsics and filtering.