3D visionExplainer

Why aligned point clouds keep residual lens distortion (and the fix)

Why depth-to-color alignment leaves residual lens distortion in an aligned point cloud, how the error grows at close range, and the two fixes.

LumaSpec Vision10 min read Share on LinkedIn

On this page 10 sections
  1. 01What the align filter actually builds
  2. 02Why the error grows at the edges and up close
  3. 03Worked example: a straight edge at 0.4 m
  4. 04Fix 1: undistort the color frames first
  5. 05Fix 2: drop the filter and reproject yourself
  6. 06Which path to pick
  7. 07When the usual answer is wrong
  8. 08The camera side of the problem
  9. 09Checklist before you ship
  10. 10Questions

You align depth to color, generate the colored point cloud, and the straight edge of a box near the camera still bends as if the lens were distorting it. Nothing is wrong with the sensor or with your calibration: the pipeline is mixing two image geometries. The align filter in common depth camera SDKs reprojects the depth through an ideal pinhole model — in at least one widely used SDK the distortion coefficients are explicitly zeroed by default — while the color frames you sample from stay raw and distorted. Below is the mechanism, the numbers that decide whether the error matters at your working distance, and two fixes: undistort the color frames before sampling, or do the depth-to-color reprojection yourself with full calibration on both cameras.

What the align filter actually builds

An align filter gives every depth pixel a color. It turns each depth measurement into a ray with the depth camera's intrinsics, moves that ray into the color camera's coordinate system with the extrinsics, and lands it on the color image to pick up RGB. The output is an aligned depth frame — depth resampled onto the color camera's pixel grid — which is why aligned depth to color keeps the depth resolution: the depth samples, not the color pixels, set the point count.

The landing needs a camera model for the target frame. In at least one widely used SDK, the aligned profile is created with every distortion coefficient set to zero unless you switch a flag — named addTargetDistortion there — to copy the color camera's real coefficients. Zero coefficients describe an ideal rectilinear pinhole. The color frames you subscribe to are raw sensor images, and whatever barrel distortion the color lens has is still in them; what the distortion coefficients mean sets how wide the gap between the two geometries becomes.

So the pair you color with lives in two geometries that differ by exactly the color lens's distortion. You sample color at pixel (u, v) and paint it on the point the ideal model places at (u, v) — but the raw pixel at (u, v) was imaged along a different ray, so the color belongs to a neighboring surface point. The cloud inherits the difference as residual barrel distortion: straight world edges bend, most at the frame borders. If you have compared the two color streams an SDK can deliver, image_raw versus image_rect_color is this same distinction.

Why the error grows at the edges and up close

Distortion is the departure from rectilinear projection — a projection in which straight scene lines stay straight in the image. Radial distortion dominates on wide-angle color imagers, and its size follows a cube law in image radius. Write r for the normalized radius, the pixel distance from the principal point divided by the focal length:

fractional shift = k1 · r² (first order, one radial coefficient) pixel shift = r_px · |k1| · r² (r_px = pixel radius from the principal point) lateral error = Z · r · |k1| · r² (small angles, radians; Z = range)

Double a point's distance from the principal point and its mis-sampling grows eightfold. Barrel distortion (negative k1) pulls image points toward the center, pincushion pushes them out; the sign is irrelevant to the mismatch, only the magnitude matters. Wide-angle color lenses typically put a few percent of radial shift at the frame corners, while the image center is nearly clean.

Range enters through image size. An object's image radius scales as 1/Z, so bringing a part four times closer moves its edges four times further out and multiplies the mis-sampling by 4³ = 64. A distant part images near the principal point, where the shift is a fraction of a pixel and everything looks correct. The fault did not appear — it was always there; the part simply reached the radius where you can see it.

Worked example: a straight edge at 0.4 m

Inputs, typical for a wide-angle color imager on a depth camera:

  • Color stream 1280 × 720 px, fx = fy = 640 px, principal point (640, 360): atan(640 / 640) = 45° per side, a 90° horizontal field of view.
  • One radial coefficient k1 = −0.05, a barrel value typical of wide-angle color lenses; tangential terms ignored.
  • A box edge whose outer side images at (1120, 360), 480 px from the principal point.
  • The box at Z = 0.4 m; the same box at Z = 1.6 m puts that edge at 120 px.

Rule: pixel shift = r_px · |k1| · r², with r = r_px / fx.

At 0.4 m:

  • r = 480 / 640 = 0.75
  • fractional shift = 0.05 × 0.75² = 0.05 × 0.5625 = 0.0281, about 2.8% of radius
  • pixel shift = 480 × 0.0281 = 13.5 px
  • angular error = 0.75 × 0.0281 = 0.0211 rad ≈ 1.2°
  • lateral offset in the cloud = 0.4 m × 0.0211 = 8.4 mm

At 1.6 m: r = 120 / 640 = 0.1875 and the pixel shift is 120 × 0.05 × 0.1875² = 0.21 px, about 0.5 mm of bend at that range. One quarter the radius, one sixty-fourth the shift.

Decision from these inputs: at 0.4 m the colored edge carries 13.5 px of mis-sampling and roughly 8 mm of bend at the frame border — visible in any render and large enough to disturb edge-based picking or measurement. Anything imaging inside a 160 px radius stays under 0.5 px of error, the central quarter of this sensor's width, and parts beyond about 2 m are sub-pixel. If your task measures, matches, or picks on colored edges inside roughly 1 m at this field of view, undistort the color frames; display-only clouds past 2 m will not show the difference.

Fix 1: undistort the color frames first

The aligned depth already lives in ideal pinhole geometry — meet it there. Remove the color lens's distortion from every color frame before sampling:

  1. Read the color camera's intrinsics and distortion coefficients for the exact profile you stream, from the SDK's calibration or your own checkerboard calibration. A reported set of all zeros means an idealized profile, not a perfect lens.
  2. Build the undistortion pixel maps from those coefficients once at start-up; standard calibration libraries turn intrinsics plus coefficients into forward maps.
  3. Apply cv::remap to every color frame with those maps, then sample colors into the cloud exactly as before. After the remap, pixel (u, v) names the same ideal ray in both frames.

Use the full coefficient set your calibration returns. Dropping higher-order radial terms to save a step leaves precisely the edge error you are removing — the role of the k3 coefficient in calibration shows how those terms share the work.

Fix 2: drop the filter and reproject yourself

The other path skips alignment entirely. Deproject each depth pixel with the depth camera's own intrinsics — which camera's intrinsics to deproject with settles that choice — transform the point into the color frame with the extrinsics, then project it into the raw color image with the color camera's full model, distortion included, and read the color there. The geometry never passes through the color lens's distortion, so the cloud stays rectilinear; the distortion only decides where to sample.

You then own a per-point projection loop instead of a black box, which pays off when you filter, downsample, or combine clouds. The projection step is the same one behind registering RGB and depth frames, and it carries over to merging point clouds from two depth cameras. Before assuming this costs more than the built-in path, check where your SDK's "hardware" alignment actually runs: on at least one major SDK it turned out to be vectorized host code on x86-64, not work done in the camera.

Which path to pick

Path Color sampling Cloud geometry Use it when
Do nothing off by the k1·r² fraction rectilinear display only, parts near the center or beyond about 2 m
Undistort color, keep aligned depth correct rectilinear the default for measurement, matching, and picking
Reproject yourself correct rectilinear you need per-point control, filtering, or multi-camera work
Enable the target-distortion flag consistent carries barrel distortion you texture-map onto the raw color image itself

Row 4 is the trap. The flag does not remove distortion, it moves it: the aligned depth is redistorted into the color lens's geometry, so sampling lines up with the raw image — and any consumer that treats that frame as an ideal pinhole, as point-cloud generators typically do, bakes the bend into the geometry.

When the usual answer is wrong

  • "Turn on the distortion flag." That aligns sampling by bending the depth into the color lens's distortion. Colors stop smearing; geometry stops being rectilinear. Right for overlays drawn on the raw image, wrong for anything that measures.
  • "It looked fine in the viewer." Viewers take their own rendering path and can differ from your exported data — why exported point clouds differ from the viewer render covers the gap. Judge the points your code writes.
  • "It's occlusion." Close range also grows parallax bands at depth discontinuities — occlusion bands in depth-to-color alignment. Bands hug depth edges and point one way; lens distortion bends straight edges everywhere off-center and follows the cube law. Both can be present at once, so test with a straight bar on a flat background.

The camera side of the problem

Any rig that fuses depth with a separate color imager can show this, and the color imager is an ordinary area-scan camera — machine vision lens distortion applies to it like any other lens. On a custom rig, a narrower field of view on the color lens cuts corner distortion steeply, at the price of coverage. Integrated units stream depth and color from one housing:

3D sensorsHikrobot MV-DT01WMU ToF sensing cameraMV-DT01WMU
Type
Depth / ToF camera
Price on requestView and quote
3D sensorsHikrobot MV-DLS300P-04 Laser galvanometer stereo cameraMV-DLS300P-04
Type
Stereo / structured light
$16,579.00 · quotedView and quote
3D sensorsHikrobot MV-DPS200P-02 Projected structured-light 3D cameraMV-DPS200P-02
Type
Stereo / structured light
$8,829.00 · quotedView and quote

3D sensors group the depth engines — ToF, stereo structured light, laser profiling — and the camera finder filters by type. The color imager of a custom rig comes from area scan cameras, where the lens mount and sensor format decide how much distortion you will be correcting.

Checklist before you ship

  • Print the distortion coefficients for the exact color profile you stream. An all-zero set points to an idealized profile — the trap above.
  • Undistort one frame and measure the map's largest displacement inside your working radius. Under 0.5 px you can skip the fix; at 13 px as in the worked example, fix it first.
  • Straight-edge test: a bar across the frame corner at your minimum working distance, a line fit to its cloud points, and the maximum deviation recorded.
  • Repeat that test on the uncolored cloud. If it still bends, the problem sits on the depth side, not the color lens.
  • Judge exported point data, never the viewer render.
  • Recalibrate the color intrinsics after refocusing or changing resolution; coefficients belong to one stream profile.

When the software path is settled, specify the hardware in one pass:

Build the whole station

The configurator matches a camera, lens, light, light controller and cables to each other, step by step, and lists only parts that fit what you already picked.

Start with the camera

Questions

Does the align filter undistort the color frames for me?

No. Alignment moves depth into the color camera's pixel grid; it does not rewrite the color image. The frames you subscribe to stay raw unless the SDK also offers a rectified color profile — some list both under different stream names, so check the profile list before writing your own remap.

Why does the bend almost disappear when I move the object away?

Image radius scales inversely with distance and mis-sampling grows with the cube of that radius, so four times further away means one sixty-fourth the shift. Far parts also sit near the principal point, where the shift is a fraction of a pixel. Cropping to the central region is a legitimate fix when you cannot remap.

Should I enable the SDK's target-distortion flag instead?

Only for display work. The flag makes the aligned depth live in the color lens's distorted geometry, so colors stop smearing but the cloud geometry carries the barrel distortion with it. For overlays drawn on the raw image that is fine; if downstream code measures or picks on the points, keep the geometry rectilinear and undistort the color frames instead.

How do I prove the fix worked on real data?

Place a straight bar across the frame corner at your minimum working distance, fit a line to its cloud points, and record the maximum deviation before and after the remap. Expect the bend to fall from the millimetres of the worked example to the level of the depth noise you already measure on a flat plate. Judge exported points, not the viewer render.

Are the factory coefficients good enough, or must I recalibrate?

Start with the coefficients your SDK reports for the exact color profile you stream; they belong to that resolution. Recalibrate against a checkerboard when the straight-edge test still bends after the remap, when you refocus or change resolution, or when the reported set is all zeros — a zero set points to an idealized profile, not a perfect lens.