3D visionTroubleshooting

Aligned depth all zeros: fix the clipping distance, not the camera

Aligned depth reads all zeros while raw depth looks fine? The script's clipping mask is the usual cause. Test with frame stats, then set the clip distance.

LumaSpec Vision10 min read Share on LinkedIn

On this page 10 sections
  1. 01Alignment resamples depth, it does not erase it
  2. 02Two-minute test: is the aligned depth actually empty?
  3. 03Cause 1: the clip mask is shorter than your scene
    1. Worked example: one constant turns a full frame gray
  4. 04Cause 2: the aligned frame expired before you read it
  5. 05Cause 3: the clip and the depth speak different units
  6. 06Cause 4: snapshot order and frame size
  7. 07When it really is the camera
  8. 08Make it fail fast next time
  9. 09The decision, case by case
  10. 10Questions

Your depth preview looks healthy, you switch on depth-to-color alignment, and the depth half of the display goes flat gray — the matrix underneath reads zero in every element. That combination almost never means a broken camera, and it never means alignment lost the data. In nearly every case the example script you started from carries a background-removal line that paints over every pixel beyond a clip distance, and its default clip is shorter than your scene. Below: a two-minute statistics test that shows where the zeros come from, the four causes in the order to check them, the arithmetic that sets the clip distance from your own working distance, and the fix for each outcome.

Alignment resamples depth, it does not erase it

A correct alignment takes each depth pixel, projects it onto the color camera's pixel grid with the pair's factory extrinsics, and writes the depth value into whichever aligned pixel it lands on. Pixels the color frame sees but the depth sensor cannot — normally a thin border — receive the invalid value 0. That is the only zeroing alignment does: if the raw frame holds 700,000 valid pixels, the aligned frame holds close to the same number, redistributed. An aligned matrix that is zero in every element therefore means the values were discarded after alignment, not during it. Three things live between the aligned frame and your eyes: the demo script's background-removal mask, the lifetime of the frame object, and the units you compare against. Work through them in that order.

Two-minute test: is the aligned depth actually empty?

Print four statistics on the aligned frame before you colorize anything:

import numpy as np

depth_raw = np.asarray(aligned_depth)   # your SDK's aligned depth frame as an array, 16-bit
valid = depth_raw[depth_raw > 0]        # pixels with a return; 0 marks no measurement

print(valid.size, int(valid.min()), int(valid.max()), float(np.median(valid)))

The variable names are mine; the NumPy calls are standard. A 1280 x 720 color frame holds 1280 x 720 = 921,600 pixels, so a scene with a real background should report several hundred thousand valid values. Read the first number:

  • A large valid count with a plausible median — a camera 2.4 m over a belt prints a median near 2400 at the common 1 mm depth scale — means the depth survived alignment. The display path is the bug: go to the clip mask.
  • A count of 0 means the aligned frame itself is empty. Check the frame lifetime, then the pipeline order, and only then suspect the sensor.

Cause 1: the clip mask is shorter than your scene

The most common zeroing line in alignment demos is a background remover. The widely copied align-depth2color.py demo carries it, with the variable clipping_distance_in_meters set to 1:

depth_m = depth_raw.astype(np.float32) * 0.001   # 1 mm depth scale -> meters
clipping_distance_m = 1.0                        # the demo default
gray = np.full_like(color, 110)                  # color: the aligned color frame, uint8
bg_removed = np.where((depth_m > clipping_distance_m) | (depth_m <= 0), gray, color)

Every pixel more than 1 m from the camera is painted gray, and so is every pixel with no return. A rig looking at a scene 2 m or farther shows a completely gray image while the depth array underneath is healthy — which reads exactly as alignment lost my depth. The usual first move is raising the value to 4; the next section shows why that can still print gray, and what value to compute instead.

Worked example: one constant turns a full frame gray

Take a camera mounted 2.4 m above a conveyor, depth scale 1 mm, color frame 1280 x 720 = 921,600 pixels. The belt surface returns 2300-2500 raw units (2.3-2.5 m), and 82% of pixels have a return.

  • Valid pixels: 0.82 x 921,600 = 755,712
  • No-return pixels: 921,600 - 755,712 = 165,888

With the demo default, clip = 1 m = 1000 mm: every valid pixel fails the test because 2400 > 1000, and every zero fails the other branch. Gray pixels: 921,600 of 921,600, the whole frame. Raise the clip to 4 m = 4000 mm: now 2400 < 4000, so all 755,712 valid pixels keep their real color, and gray drops to 165,888 / 921,600 = 18%, which is precisely the pixels with no return. Nothing about the depth changed between the two runs; one constant did.

Set the clip from a measurement, not from a forum reply:

clip = working distance + headroom clip_raw = clip / depth scale

For the conveyor: 2.4 m + 0.6 m headroom = 3.0 m, and 3.0 / 0.001 = 3000 raw units. The headroom absorbs a tilted scene and someone stepping to the belt edge. Measure along the optical axis the way your viewer reports distance, not with a tape to an off-center target: for off-axis pixels the ray distance exceeds Z, as depth value vs true distance explains.

Cause 2: the aligned frame expired before you read it

Many SDK pipelines hand you the aligned frame as a view into buffer memory that the next grab overwrites. An array kept by reference then reads as zeros or stale values one frame later — the debug watch shows zeros even though the display was fine for an instant. Take a copy immediately after alignment: keep = np.array(depth_raw, copy=True). The test is direct: print the statistics on the copy right after the grab, then again after the next frame arrives. A healthy first print and a zero second print is the lifetime bug; fix it by copying before the next wait-for-frame call. The same reference rule explains point clouds that come out null after alignment.

Cause 3: the clip and the depth speak different units

Raw depth values are counts, not meters; the device's depth scale converts one into the other, and 1 mm (a factor of 0.001) is the common case. The mask compares whatever sits in the array, so the clip must be in the same currency. Raise the demo value from 1 to 10 while the array still holds raw counts and you have asked for a 10-count clip: 10 mm, so 2400 > 10 still blanks everything. That is why try 4 or try 10 sometimes changes nothing. Convert once, with the scale printed at start-up:

depth_scale_m = 0.001                    # read this from the device at start-up
clipping_distance_m = 3.0                # from your measured working distance
clipping_distance = clipping_distance_m / depth_scale_m   # 3000 counts at a 1 mm scale

Check it against the statistics print: a median of 2400 against clipping_distance 3000 passes (2400 < 3000); the same median against a clip left at 3.0 fails (2400 > 3.0) and the frame goes gray. The depth scale to meters conversion works the scale factor through, and the same unit trap returns in hand-eye calibration with meters in and millimeters out.

Cause 4: snapshot order and frame size

Alignment needs both frames from the same snapshot, with the color stream enabled, and it runs on frames the pipeline has already delivered. Feed it a depth frame from one grab and a color frame from the next and the mapping lands wherever the extrinsics guess: sparse, shifted or empty. Print the aligned frame's shape once — it must match the color frame, because alignment maps depth onto the color camera's geometry; if you see the depth frame's dimensions you are holding the wrong frame, as resolution changes after alignment describes. When you later deproject the aligned frame to a point cloud, use the color camera's intrinsics; which intrinsics to deproject with matters exactly here. With sizes right, a fresh copy and sane units, the remaining software suspect is the factory extrinsic calibration between the two sensors — the same calibration a stereo RGB alignment fix addresses.

When it really is the camera

If raw depth is healthy, the copy is fresh, the units check out, the shapes match and the mask is out of the path — and the aligned matrix still prints zeros — the sensor or its factory extrinsics are what remain. Two checks take a minute each. Open the vendor's desktop viewer, switch to its 3D point-cloud view and map color onto the cloud: a correctly colored cloud there proves the hardware aligns, and the difference is in your code. Then run the viewer's on-device depth calibration routine, which the major depth-camera vendors ship. Sparse returns are a different story: glass, black matte surfaces and strong sunlight starve the sensor, and the zeros concentrate where the IR image shows a weak signal — scene failure, not alignment failure. If the cloud still comes out empty in the viewer, the module goes back to the vendor; start from the contact page.

Make it fail fast next time

Four habits turn this bug class from an afternoon into a printout:

  1. Assert a valid fraction on every aligned frame: assert (depth_raw > 0).mean() > 0.2, 'aligned depth empty - check mask, copy, order'. A silent gray frame becomes a loud failure at the grab.
  2. Name the unit in every depth variable — depth_raw, depth_m, clipping_distance_m — and convert once, from the scale printed at start-up.
  3. Compute the clip from a configured working distance plus headroom, never from a bare constant copied out of a demo.
  4. Copy the aligned frame before the next grab, and log its shape once at start-up so a wrong-frame bug cannot hide.

Keep the demo's background removal out of measurement code: colorize with the depth colormap for humans, and measure on the raw values, where 0 is missing data rather than a color choice.

The decision, case by case

What you observe Where the zeros come from What to change
Raw depth fine, aligned stats print a healthy count The demo's clip mask Clip = working distance + 0.5-1 m headroom, converted with the depth scale
Stats healthy on grab 1, zeros on grab 2 Frame lifetime Copy the aligned frame immediately after alignment
Clip raised to 4 or 10, nothing changed Units: clip in meters, depth in counts clipping_distance = clip_m / depth_scale
Aligned frame shaped like the depth frame Wrong frame or mixed snapshot Align frames from one snapshot; expect the color frame's size
Every check passes, viewer cloud also empty Sensor or factory extrinsics On-device calibration, then return the module

For the case that opens this guide — raw depth fine, aligned frame all gray, 82% valid in the stats — the fix is one line: clip at the measured working distance plus headroom, 3.0 m for the 2.4 m conveyor, converted to 3000 counts at the 1 mm scale. If you are standing up a station rather than rescuing one, start in the 3D sensors category or the camera finder. The configurator walks the rest of the chain, and the 3D topic page collects the alignment and calibration guides.

3D sensorsHikrobot MV-DPS200P-02 Projected structured-light 3D cameraMV-DPS200P-02
Type
Stereo / structured light
$8,829.00 · quotedView and quote
3D sensorsHikrobot MV-DT01WMU ToF sensing cameraMV-DT01WMU
Type
Depth / ToF camera
Price on requestView and quote

The MV-DPS200P-02 is a projected structured-light 3D camera; the MV-DT01WMU is a ToF sensing camera.

Questions

Why does my aligned depth image show gray instead of zeros?

The demo's background-removal line paints two pixel groups with a solid gray color: pixels with no return (depth 0) and pixels beyond the clip distance. The array underneath can still hold valid depth; only the display is gray. Run the statistics test on the aligned array: a large valid count with a sensible median means the depth arrived and the gray is your mask, not the camera.

What value should the clipping distance be?

Just past the farthest surface you want to keep. Measure the camera-to-scene distance along the optical axis, add 0.5-1 m of headroom for tilt, and convert with the device's depth scale before comparing. A default of 1 m blanks any scene farther away; 4 m is the usual test value for rigs a few meters back, not a universal setting.

Why are the border pixels zero after a correct alignment?

Alignment maps depth onto the color camera's grid. The color frame usually sees a little more than the depth sensor, and stereo depth also breaks down at object edges, where mixed returns produce flying pixels. Pixels with no trustworthy source get the invalid value 0. That is normal: treat 0 as missing data, mask it, and never read it as a 0 mm distance.

Can I avoid the mask entirely by aligning color to depth instead?

Yes. Reversing the direction maps color onto the depth grid, so every depth value stays put - nothing is resampled and nothing can be clipped away. You lose some color detail at depth edges and work at the depth camera's resolution, but for measurement, bin picking and volume work the depth-native result is usually the safer target.

How do I prove the camera is at fault before sending it back?

Check four things in code first: the statistics print shows a healthy valid count on the raw frame, the aligned copy is taken before the next grab, the clip conversion uses the printed depth scale, and the aligned frame's shape matches the color frame. Then open the vendor's viewer, map color onto the 3D point cloud, and run its on-device calibration. If the cloud is still empty there, the module is the problem.