Depth point cloud not filtered: which topic and frame to use

You turned the depth filters on, the depth image looks clean, and the color-aligned point cloud still shows holes, speckle at the edges and frame-to-frame flicker. The cloud is not broken. In most depth camera drivers, the color-aligned cloud publisher deprojects the raw depth frame before the decimation, spatial, temporal and hole-filling filters ever touch the stream. Here is the pipeline order that causes it, which topic carries a cloud built from the filtered depth image, how to make the driver filter before alignment, and how to pair the intrinsics with the frame named in the cloud's header. The same wiring applies to any depth or ToF camera streaming into ROS, whether the job is bin picking or volume checking.

The pipeline order behind an unfiltered cloud

A depth camera driver runs its stages in a fixed order: raw depth frame, filter chain, optional alignment to the color image, then deprojection into a point cloud. The filters — decimation, spatial, temporal and hole filling — work on the depth image. They close holes, suppress flying pixels and average out flicker before anything is converted into 3D points.

The color-aligned cloud topic skips that order on many drivers. Its publisher takes the raw depth frame, converts raw values to metres with the depth scale, and deprojects with the color intrinsics. The filter chain runs in parallel on the depth image and never reaches the cloud. That is why the depth image in RViz2 looks clean while the cloud above it still flickers.

The registered cloud published by a separate RGB-D node is built from the filtered depth image, which is why it looks better the moment you switch topics. Nothing on the camera changed; you changed the pipeline stage the cloud was built from.

Two topics, two sources: pick the one fed by the filtered image

Once you know what each cloud is built from, the choice is mechanical.

The deciding rule: subscribe to the cloud whose source is the filtered depth image. If your driver publishes one, check its header and you are done. If the color-aligned cloud is the only one and the driver cannot filter before alignment, build the cloud yourself from the filtered image.

Make the driver filter before the cloud is built

Drivers that expose the filters as node parameters let you close the gap without writing code:

1. Enable the depth filters on the driver node: decimation, spatial, temporal and hole filling.
2. Put decimation first. At 2×2 it keeps every fourth pixel, so each later filter and the alignment step run on a quarter of the data.
3. Point the cloud generation at the filtered depth stream. Where the driver offers a stream selector for the cloud, choose the filtered image; where alignment to color is enabled, confirm alignment runs after the filters.
4. Restart the driver and recount points (the check below). A drop of about 4× after 2×2 decimation confirms the filters now feed the cloud.

Build the cloud yourself from the filtered depth image

If the driver offers no hook, or you want the cloud in the depth frame with no color attached, subscribe to the filtered depth image and deproject it in your own node. The math is exactly what the publisher skipped:

Here (u, v) is the pixel coordinate, (cx, cy) the principal point, fx and fy the focal lengths in pixels — all from the intrinsics of the stream you deproject — and raw is the depth value converted to metres by the depth scale. The step-by-step version with a worked example is in Depth map to point cloud: the deprojection calculation explained.

The classic trap is which intrinsics to load: an aligned image takes the color intrinsics, the plain depth image takes the depth intrinsics. Aligned depth point cloud intrinsics: which camera to deproject with settles that choice, and aligned depth streams: data layout and which intrinsics apply covers the stream layouts behind it. For geometry cleanup after deprojection, a voxel grid and statistical outlier removal from PCL run with little overhead on your own cloud.

Frames and intrinsics: read the header, then match

A deprojected cloud is stamped with the optical frame of the stream that produced it. Optical frames follow the ROS convention: +Z forward along the viewing axis, +X right, +Y down. The driver also publishes a static transform from each optical frame to the camera's mechanical frame; take that rotation from TF rather than composing it by hand.

The pairing rule that prevents most warped clouds: the intrinsics must belong to the stream whose optical frame is in the header.

• A cloud built from depth aligned to color carries camera_color_optical_frame — pair it with the color intrinsics.
• A cloud built from the plain depth image carries the depth optical frame — pair it with the depth intrinsics.
• The registered cloud's frame depends on the node that does the registration, so echo the header once and match the stream that frame names.

Is depth image_raw aligned to RGB? Check registration shows how to confirm the alignment itself, and depth to color registration: raw versus rectified frames untangles the frame variants. To read the header without dumping a million points, run ros2 topic echo /camera/depth/color/points --field header --once on ROS 2, or rostopic echo --noarr on ROS 1, which suppresses the data array.

The depth scale check that catches a 10x-off cloud

The publisher converts raw depth to metres by multiplying the depth scale. A typical depth stream counts millimetres, so a raw reading of 2,500 means 2.500 m; some sensors count 0.1 mm steps and the scale value differs accordingly. A 1 mm scale applied to a 0.1 mm stream puts every point ten times too far away — a clean-looking cloud the robot cannot use.

Test it in two minutes: aim at a flat board at a known distance, convert one cloud, histogram the Z values. On a healthy stream at 1 m, the median lands within about 1% of the true distance (±10 mm at 1 m for most depth cameras; your camera's own accuracy figure may be tighter). A median at 10 m for a board at 1 m is the scale factor. If instead the cloud waves and bends across the field of view, that is intrinsics, not scale — depth filter camera intrinsics: diagnose wavy point clouds walks that diagnosis.

Worked example: point count and load before and after decimation

Take a typical 1280×720 depth stream at 30 Hz and the unfiltered color-aligned cloud:

• Points per frame: 1280 × 720 = 921,600.
• Memory per cloud: 921,600 × 32 B = 29.5 MB, using the 32-byte stride PCL's padded point types occupy in memory. The serialized message on the wire is denser, but every copy your process makes touches the full stride.
• Rate into each subscriber: 29.5 MB × 30 Hz ≈ 885 MB/s, before your own processing adds anything.

Now enable 2×2 decimation at the driver so the filtered image feeds the cloud:

• Points per frame: 640 × 360 = 230,400 — exactly 4× fewer.
• Memory per cloud: 230,400 × 32 B = 7.4 MB.
• Rate into each subscriber: 7.4 MB × 30 Hz ≈ 221 MB/s — again 4× less.

A second doubling divides the load by 16, which is why decimation belongs at the front of the chain. Each extra subscriber, PCL pass and planner copy multiplies the memory traffic, so cutting points at the source is the lowest-cost performance change in this pipeline. The geometric side effects are covered in point cloud downsampling distortion: why decimation shifts points.

A host absorbing several such streams at once needs LAN ports and memory to match. A vision controller in the 32 GB memory class with 5+ LAN ports and PoE camera ports, such as the MV-VC5525-256G70-NN below, keeps the camera links, deprojection and PCL passes on one machine:

When the depth principle, not the filter, is the problem

Filters clean noise; none invents geometry where the sensor returned none. Shiny metal, black rubber and other low-return surfaces leave permanent holes that temporal and hole-filling filters cannot close honestly, and averaging frames of a moving part smears the result. If you need a denser capture of a still scene, merge frames deliberately instead — merge multiple depth frames into one gap-free point cloud sets the alignment conditions.

Where the holes follow the material, change the depth principle: ToF, projected structured light and laser profiling suit different surfaces, ranges and cell layouts, and the system-level choice for conveyor work is laid out in 2D or 3D camera for conveyor pick and place: choosing the system. Two depth sources from the range, for different materials and cells:

Reconsider alignment itself while you are at it. Aligning depth to color resamples every depth pixel onto the color geometry and opens holes along depth edges. If no consumer needs a color value per point, work in the depth frame and skip alignment: the cloud is denser and the pipeline shorter.

Checks before you trust the cloud

1. Echo the header once and note the frame_id, with the commands above.
2. Enable 2×2 decimation and recount: about 4× fewer points confirms the filters feed the cloud; no change means the raw frame still does.
3. Flat board at a known distance: the median Z should land within about 1% (±10 mm at 1 m). Off by a round factor means depth scale; wavy means intrinsics.
4. Open RViz2 with the fixed frame set to the cloud's own frame. A blank display while the topic publishes is usually the QoS mismatch fixed in RViz2 image topic published but not displaying: fix the QoS mismatch.
5. Compare the cloud topic's rate with the depth stream's: 10 Hz of cloud from 30 Hz of depth means the publisher cannot keep up, and driver-side decimation is the lowest-cost fix.

Sourcing the parts and where to go next

For a new cell rather than a debugged one, choose the depth camera by surface and range first, then wire the topics as above. The 3D sensors category lists laser profilers, structured-light stereo cameras and ToF cameras, and the camera finder filters them by type, resolution and interface. If the host is the constraint, the vision controllers category covers machines sized for multi-camera cloud loads, and the camera configurator walks the whole chain from camera to mount. Every published guide on 3D capture and point clouds is collected on the 3D topic page. If a cloud still misbehaves after these checks, contact the applications team with the header, a point count and one captured frame — those three pin the cause quickly.