Point cloud downsampling distortion: why decimation shifts points

You enable a decimation filter to lighten the load, and the point cloud breaks: the histogram of X and Y that was centered on zero collapses toward positive values, and whole regions of the scene vanish. Nothing in the scene moved and the camera did not move, so the fault is in the math between the depth image and the 3D points. In nearly every case the cause is the same: the image got smaller and the camera intrinsics did not. This article explains why that mismatch shifts points, how to confirm it in ten minutes, and three ways to fix it, ordered by how much code they cost you.

The symptom

A stereo point cloud visualization looks correct with the decimation factor set to 1. Turn it to 2 or 4 and three things happen at once:

• Valid points cluster in the positive X and Y quadrant instead of spreading around the optical axis.
• The Z histogram still looks plausible, so depth itself seems fine.
• Points disappear, as if a chunk of the field of view were cropped away.

That combination is the fingerprint of an intrinsic mismatch, not a sensor fault, a calibration drift, or a filter bug in the depth data.

Why decimation shifts points off-axis

A point-cloud node converts each depth pixel to a 3D point with the pinhole model:

Here u, v are the pixel coordinates, fx, fy the focal lengths in pixels, and cx, cy the principal point, all taken from the calibration of the full-resolution image. A decimation filter keeps every d-th pixel, so the image shrinks from, say, 1280 × 800 to 640 × 400. If the projection keeps using the old matrix, pixel (320, 200) in the small image is treated as if it were (320, 200) in the large one. It is really the pixel that used to sit at (640, 400), close to the image center.

The error grows with distance from the origin of the small image, which is its top-left corner. That is exactly why every point drifts toward positive X and Y: the projection thinks all pixels sit in the lower-right quadrant of the original frame. The correct matrix for a decimated image is:

The half-pixel terms keep the pixel-center convention consistent. Skip them and you add a systematic half-pixel offset per decimation step, small but measurable at close range.

Confirm the cause in ten minutes

1. Capture one frame of a static scene with the decimation factor at 1 and save the cloud.
2. Capture the same scene with the factor at 2, 4, and 8.
3. For each capture, compute the median X, median Y, and median Z of points within 100 mm of the cloud center.
4. Compare against the full-resolution capture.

If Z stays constant while X and Y grow roughly in proportion to the decimation factor, the intrinsics are the problem. A flat target such as a white board makes the test unambiguous: its corner coordinates should land within a millimeter or two of the full-resolution values at any factor. For background on how decimation also changes the coverage of the depth image itself, see depth image field of view: why stereo coverage is smaller.

Fix 1: let the software scale the intrinsics

The cleanest fix costs no code: use a pipeline where the point-cloud stage receives the decimated image size along with the calibration and scales fx, fy, cx, cy itself. Camera SDKs have been converging on this, but versions and containers lag. If the vendor has published a fix, updating the SDK, firmware, and any ROS or container image you run is the first move. If you pin a container tag such as a rolling latest, check its build date against the vendor's issue tracker before assuming you have the fix.

Fix 2: project on the host with your own matrix

If you cannot update, stop using the on-device point-cloud stage and project the decimated depth image yourself. The computation is trivial and runs comfortably in real time for factors of 2 or more:

Take fx, fy, cx, cy from the device calibration for the full resolution, apply the scaling formulas above for your factor, and iterate over the decimated image. This gives you exact control and doubles as a diagnostic: if your host-built cloud matches the full-resolution cloud, you have proven the on-device stage was the fault. The mapping from image pixels to 3D points is the same one behind bounding box depth: mapping image pixels to 3D points, so the same code serves both tasks.

Fix 3: decimate after projection

The simplest robust design is to not decimate the depth image at all. Project the full-resolution cloud with the untouched intrinsics, then subsample the points:

Every retained point is exact, because it never passed through a mismatched projection. For storage and visualization, voxel grids are usually better than thinning anyway; see store point cloud data in a voxel grid for 3d inspection and voxel grid comparison: align scans to identify new coverage. The only cost is bandwidth: the full-resolution depth stream travels over USB or Ethernet before it is reduced. If that link is your bottleneck, prefer Fix 2.

Choosing the right 3D sensor for dense clouds

If you are fighting bandwidth because your application genuinely needs millions of points per frame, the fix may be hardware. Structured-light and laser-profile sensors deliver dense, calibrated clouds without you managing the projection pipeline at all.

The MV-DPS200P-02 is a projected structured-light 3D camera: it projects a known pattern and triangulates from two views, producing a registered cloud with the intrinsics handled in the factory calibration.

For laser line profiling, the MV-DP4180-03P builds the cloud from a calibrated laser plane, so each profile lands in 3D coordinates directly and there is no per-pixel projection to get wrong.

Browse the full range in the 3D sensors category, narrow it with the camera finder, or start at the system configurator and let it match sensor, lens, and controller. Where point density feeds a downstream tolerance check, the guide on point cloud density for profile tolerance covers how density maps to ASME profile verification.

Other causes worth ruling out

Work down this list in order of likelihood:

1. Intrinsic mismatch after decimation (most likely). Test: the median-X/Y drift test above. Fix: Fixes 1-3.
2. Depth aligned to the wrong camera. If depth is aligned to a different sensor than the intrinsics assume, the same shift appears even without decimation. Disable alignment and compare.
3. Rectification and decimation order. Some stacks decimate before rectification. If the rectification maps assume full resolution, the geometry skews exactly as described. Check the order of the stages in your pipeline graph.
4. Genuine data loss. A stream that drops frames or half-frames produces different symptoms: torn clouds, flicker, or crashes. That is a transport problem, covered in depth camera black image: fix USB bandwidth and power issues and stereo depth stream disconnect: diagnose startup crashes.
5. Calibration error. A tilted or warped cloud with decimation at 1 points at extrinsic calibration instead; the checks in why a laser triangulation point cloud is tilted: calibration checks apply.

Prevention checklist

• Record the image size alongside every intrinsic set you store, and assert they match before projecting.
• Treat any change to resolution, binning, ISP scaling, or decimation as a change to the camera matrix, not just to the image.
• Keep one golden scene: capture it at full resolution and after any pipeline change, and diff the median X, Y, Z of a flat target.
• Project once, with one matrix, and do all reduction downstream of the projection.
• Pin and date-stamp SDK and firmware versions in your deployment notes so a known fixed bug cannot quietly return in a container rebuild.

When the usual answer is wrong

If your cloud shifts but Z also changes with decimation, the intrinsics are not the whole story: you are likely also changing exposure or confidence behavior, and losing points asymmetrically. Check the validity mask per factor. And if you decimate the texture or color stream instead of the depth stream, the projection is unaffected; a misregistration between color and geometry there is a different problem, covered in camera intrinsic parameter conversion for depth and texture images.

The takeaway: a depth image and its intrinsics are a matched pair. Break the pairing and the cloud breaks with it. Keep them together, and decimation is a safe, reversible tool for lighter processing and rendering.

If you are specifying a sensor for a new cell and want a second opinion on resolution, density, and processing load, send us the scene description through the contact page — the consultation is free with any order. More practical 3D material is on the 3D topic page and across Learn.