Lines in aligned depth images: why artifacts follow edges

Thin lines that trace object edges in a depth image aligned to the color frame can appear in RGB-D pipelines. They follow depth discontinuities, and their values may differ from the surfaces on either side by millimetres or centimetres. This article explains how alignment can create those lines, how to isolate the processing stage responsible, and which fixes to apply in which order. It is for anyone building point clouds, overlays or measurements from a registered depth and color pair, whether from a stereo and structured light camera or a time-of-flight module.

The symptom

You enable a depth stream and a color stream, pass both through the SDK's alignment stage, and the resulting registered depth image shows narrow lines along object silhouettes and internal depth steps. The rest of the image looks normal. The lines often show up more clearly when you convert the depth image into a point cloud and view it from an angle: thin ridges or valleys hang off edges.

Two details point at alignment or its associated processing as the cause:

• The same frames, viewed without alignment, show no lines. The raw depth image is clean.
• The lines coincide with borders in the depth data, not simply with texture borders in the color image, and move with those depth borders.

If either of those fails, look elsewhere first: exposure banding, flickering illumination, or a synchronization problem such as the one described in depth and color synchronization: check the right timestamps.

Why interpolation at depth edges creates the lines

Alignment maps measurements between camera coordinate systems using camera intrinsics and the extrinsic transform. Depth-to-color registration commonly deprojects depth samples, transforms their 3D positions, and projects them onto the color pixel grid. How the implementation resolves collisions, empty target pixels and occlusions determines the resulting edge behavior. Alignment does not necessarily blend depth values; inspect the SDK's documented resampling and hole-filling behavior.

Interpolation can be harmless on smooth surfaces. At a depth discontinuity it can create false measurements. Consider a foreground edge in front of a background wall, a step of 300 mm. Projection can leave target pixels without a source measurement, particularly around occlusions. If interpolation or hole filling combines samples from opposite sides of that step, the result represents neither surface.

Here, w is the foreground contribution, between 0 and 1. With a 300 mm step and w = 0.9, the error relative to the foreground is approximately 0.1 × 300 mm = 30 mm. Deprojecting that value places a point between the surfaces, creating an apparent ridge or valley. The seam width depends on camera geometry, scene depth, output resolution and processing settings; it is not necessarily one pixel. Raw depth avoids this particular remapping step, although it can still contain sensor and processing errors of its own.

For the transform conventions behind this mapping, see transform depth point clouds between optical frame and camera frame; the registration concepts themselves are covered in depth to color registration: raw versus rectified frames.

Confirm the alignment stage is the cause

Work through these tests in order. Each one isolates a different part of the pipeline.

1. Bypass the alignment. Take both frames straight from the capture call, before your processing, and build the point cloud from the raw depth. If the lines vanish, investigate remapping and downstream processing before changing camera settings. If they remain, investigate depth acquisition, illumination and any processing already enabled in the SDK.
2. View the aligned depth as an image, not a point cloud. Color-map the aligned depth frame, but also inspect its numerical values. Look for narrow bands of intermediate depths beside a discontinuity. Record their width rather than assuming a fixed size. Wider bands can also involve scaling or decimation, as described in depth image resolution smaller than expected: check decimation first.
3. Check the vendor viewer. Run the manufacturer's own viewer with matched stream profiles and processing settings. If its aligned output is clean while yours shows lines, compare frame selection, calibration, filters and resizing. An interpolating resize can mix depths across an edge. Keep measurement data at its registered resolution unless your resampling method explicitly handles invalid values and discontinuities.
4. Check timestamps. Ask for the depth and color frames that belong to the same capture instant and verify the match explicitly. A mismatched pair can produce moving edge offsets, particularly when the camera or object moves.

Fixes, in the order to try them

Reorder so depth stays raw

The simplest fix is often to avoid aligning depth at all. If your output is a colored point cloud, deproject from the raw depth image and fetch each point's color by projecting its 3D position into the color image. This preserves the source depth geometry, although color assignment still needs bounds and visibility checks. It also sidesteps the scale confusion covered in depth scale mismatch: why your point cloud is cut off at close range.

Filter before alignment, not after

A temporal or spatial filter on the raw depth frame can reduce noise before remapping. It can also introduce motion trails or alter boundaries, so compare filtered and unfiltered captures of a known step edge. After alignment, an ordinary averaging filter can mix the very discontinuities you care about. If you must filter the registered image, use an edge-aware filter, or evaluate a small median, and verify that the measured depth on either side has not moved.

Handle the seam explicitly

Mark uncertain boundary pixels and exclude them from precision measurements. One starting point is a local depth-range test: flag neighborhoods whose valid maximum minus minimum exceeds your chosen discontinuity threshold. This identifies an uncertain region, not proof that every pixel within it is wrong. Choose the threshold above ordinary surface noise and below the depth step you need to detect.

In a point cloud, neighborhood depth differences can support similar rejection, which overlaps with the diagnostics in stereo point cloud visualization: check missing XYZ points. If downstream segmentation tolerates gaps better than wrong values, leave the seam invalid. Do not fill an occlusion by averaging foreground and background. For scenes needing additional coverage, consider the observations and registration checks in merge multiple depth frames into one gap-free point cloud; additional frames help only when they contain valid observations of the missing surface.

Update the SDK through the manufacturer

Review the manufacturer's release notes for registration, calibration and filtering changes. If an update is appropriate, obtain the SDK and compatible device firmware through the manufacturer's documented channels, then rerun the same saved-frame and live-capture tests. Keep the update out of your validated production pipeline until edge behavior and measurement results meet your acceptance limits.

When the usual answer is wrong

Not every edge-following line is an interpolation seam. Check these before committing to the explanation above:

• Lines that flicker under artificial light. Some lamps modulate their output, which can interact with exposure timing and produce banding. Compare captures under steady illumination and change exposure settings to isolate that effect.
• Lines at fixed image rows or columns. Investigate data integrity, image stride, pixel-format interpretation and sensor output. Transport problems may also be relevant, but a fixed line alone does not establish a bandwidth or cable fault.
• Lines on dark or glossy surfaces only. Structured-light and stereo sensors can return sparse or noisy depth on low-texture, dark or specular material. Subsequent processing may extend those errors around holes. Check lighting and exposure, as covered in missing points in a 3D camera point cloud: fix the depth holes.

Choosing hardware for clean registered pairs

Sensor choice affects the registration workflow before any code runs. Confirm whether a candidate provides a color stream, calibration between streams, and a documented registration method. A depth camera's sensing technology alone does not establish those capabilities. The MV-DPS200P-02 is a projected structured-light 3D camera.

For volume-measurement applications, consider a multi-DOE speckle structured-light 3D camera. Confirm its supported output streams and processing functions against your application requirements rather than assuming a particular seam behavior.

For a different sensing approach, the MV-DT01WMU is a ToF sensing camera. Request representative raw and processed captures before judging edge quality or measurement suitability.

Whatever the technology, evaluate edge behavior on your own parts before ordering: a step edge test piece with a known height tells you more about your pipeline than a general technology comparison.

Pre-order checks

• Record raw and aligned depth of a step edge with a known height, and measure the seam error.
• Verify the SDK's alignment runs on your host without frame drops at your target resolution and rate.
• Test your darkest and glossiest part; check the depth density before alignment, since interpolation cannot invent valid observations.
• Confirm the depth scale and units your SDK reports, so the seam threshold is set in millimetres, not code values.

Short checklist for the pipeline

• Deproject from raw depth; project into color for per-point RGB, with visibility checks.
• Avoid interpolating across depth discontinuities when resizing or decimating; see point cloud downsampling distortion: why decimation shifts points.
• Evaluate filtering before alignment; after alignment, preserve invalid pixels and depth boundaries.
• Mark uncertain seam pixels explicitly and decide per application: invalid, filled from supported observations, or kept.
• For fixed-image-point measurements, verify which camera grid the point belongs to; see center pixel depth measurement: read a fixed image point.
• Re-verify after every SDK update; alignment behavior can change between releases.

An edge-following seam can reflect resampling, occlusion handling or another processing error; it is not automatically a camera defect. Preserve source depth where possible, attach colors per point when appropriate, and validate every operation that changes boundary values. When selecting a sensor for a registered-pair application, use the 3D camera finder to compare technology and resolution, or the system configurator to work through camera, lens, light and host choices. For help reviewing the proposed workflow, contact our applications team.