Depth low confidence values: mask invalid pixels with the IR image

A depth map can look full and still lie: the pixels where the camera had no signal hold numbers anyway, and every measurement taken across them drifts. The sensor marks those pixels internally as low confidence, but not every SDK hands you the flag. This piece is for vision engineers integrating stereo, structured-light, ToF and laser profile sensors who need per-pixel validity inside their own pipeline. You will learn what the depth low confidence value is, how to rebuild an equivalent signal from the registered IR image, a starting threshold with its arithmetic, and the cases where bright IR still hides bad depth.

What is a depth low confidence value?

A depth low confidence value is the number the depth source writes into every pixel it could not measure reliably, and in nearly all stereo, structured-light and ToF cameras that number is 0, fixed by the sensor rather than chosen by you.

It is not the same setting as the confidence threshold. The threshold decides which pixels count as weak; the low confidence value decides what those pixels get filled with. Older SDK generations exposed both as separate properties, and several current cross-platform SDKs expose only the threshold — some expose neither, which is why engineers end up rebuilding validity in software. The fill value matters downstream: most deprojection code treats 0 as no point, which is exactly why a raw depth frame turns into a cloud with holes at the origin. That behavior, and how to diagnose it, is the subject of depth pixels that deproject to zero.

How does a camera decide that a pixel is low confidence?

A camera flags a pixel when its matching evidence is weak: too little texture for passive stereo, a faint projected return for structured light, too few returned photons for ToF. The strength of the IR signal tracks that evidence pixel by pixel.

Each technology fails in its own way. Passive stereo needs texture to match between two imagers, so blank surfaces fail even at decent brightness. Structured light needs pattern contrast to decode, so weak returns fail even on bright surfaces. Time of flight needs enough photons to time, so distant or dark surfaces fail. The published IR, amplitude or intensity stream is the physical evidence behind all three judgments, not a marketing extra.

Registration is the other half of the deal. Everything below assumes the IR image arrives on the depth camera's pixel grid, one IR pixel per depth pixel, so a mask built in one frame lands exactly on the other. If your camera rectifies the IR to a different resolution, resample first — otherwise the mask lands a pixel or two off, and edges fail validation for the wrong reason.

How do you mask invalid depth pixels without a confidence stream?

Threshold the IR image that is registered to the depth map: build a binary mask where each IR pixel sits at or above your low cut and below saturation, keep depth only inside that mask, and write 0 everywhere else.

In practice the loop is five steps:

1. Grab depth and the registered IR (or intensity) image in the same trigger cycle.
2. Histogram the IR frame: find the background floor and the good-part median.
3. Set the low cut from that median and a high cut at saturation.
4. Multiply the mask into the depth frame.
5. Clean the mask with a 3×3 majority filter so isolated survivors disappear.

The multiply is one array operation — the same trick as zeroing pixels outside an ROI without a per-pixel loop — so it costs nothing on the line, and a downstream deprojection inherits the mask for free.

What IR threshold should you start with?

Start the low cut at 30 percent of the median IR value of a known-good part at working distance, and cut saturated pixels at 250 or above in an 8-bit stream; both numbers assume the exposure puts good matte returns in the upper third of the range.

The derivation takes one frame. Point at the part, histogram the IR, take the median over the part's pixels, multiply by 0.3 and round. A matte aluminium target reading 140 gives 42; a dark rubber seal reading 70 gives 21. The rule scales with the part; it is not a universal gray level. What moves it: doubling the working distance drops diffuse returns to roughly a quarter, so calibrate at the real standoff; changing exposure rescales the whole histogram; a dirty window or lens pulls the median and the threshold down together.

The rule is ratio-based, so it survives bit depth: 0.3 of the median means the same in an 8-bit, 10-bit or 12-bit stream. What breaks scaling is a 12-bit signal packed into a 16-bit container — values then top out near 65,520, not 65,535, and half-scale assumptions go wrong. Check the true bit depth before scaling; 12-bit data inside a 16-bit frame shows what those containers actually hold.

What does the arithmetic look like on one frame?

On an illustrative 1,280 × 800 depth frame with an 8-bit registered IR image, a good-part median of 140 sets the low cut at 42, the mask removes 131,072 pixels, and 892,928 pixels — 87.2 percent of the map — survive as valid depth.

Here is the arithmetic, line by line:

1. Frame: 1,280 × 800 = 1,024,000 depth pixels (illustrative structured-light sensor at a 600 mm standoff).
2. Calibration: matte aluminium target fills the field; IR median 140; unlit background reads 12.
3. Cuts: T_low = round(0.30 × 140) = 42; T_high = 250.
4. Histogram: 122,880 pixels (12.0 percent) below 42; 8,192 pixels (0.8 percent) at or above 250.
5. Valid depth: 1,024,000 − 122,880 − 8,192 = 892,928 pixels, or 87.2 percent of the frame.

Decision for this frame: window gauges are sound at 87.2 percent overall coverage; single-pixel reads are not. Average every measurement over a window of valid pixels — averaging depth over a segmentation mask shows the selection step — and require at least 90 percent valid pixels per window; a window that misses fails the part. Below about 80 percent overall valid coverage, fix the optics first — exposure, projection power, standoff — not the threshold.

How does this differ across sensor types?

The brightness test transfers to every 3D technology, but the second channel means something different on each: structured light maps IR brightness to pattern contrast, stereo adds texture to the judgment, ToF reports amplitude, and a laser profiler grades each point by returned line intensity.

Structured light: a projected-pattern camera throws coded light and reads it back; where the part returns little, correlation fails and depth goes invalid. The projector's own IR return is therefore a near-direct signal-strength meter, which makes the threshold method work almost as printed. Watch both ends of the histogram: dark rubber starves the low end, polished metal glares into the high end. Projected-pattern and speckle variants differ in the pattern, not in the mask:

For volume measurement on parcels and totes, the same logic runs on speckle: a multi-DOE speckle structured-light camera built for volume measurement produces depth through the same mask logic. How a projected pattern becomes coordinates is its own subject — projector pixel mapping to 3D points covers it.

Passive stereo: matching compares texture between two imagers, so brightness alone lies — a uniform gray tote wall can be bright and unmatchable. Use the rectified reference image and add a local texture floor, typically a 3×3 standard deviation of 2 to 5 gray levels in 8-bit, calibrated exactly like the brightness cut. The image-side differences between projected patterns and passive matching are compared in structured light versus stereo depth:

Time of flight: a ToF camera counts returned photons, and amplitude is the signal meter. Mask on amplitude, then treat concave corners, glass and far edges as suspicious even when amplitude passes, because photons arriving off the direct path still get counted. For time of flight, the amplitude channel is the one to request:

Laser profilers: a sheet-of-light sensor triangulates a laser line against a known baseline, and its intensity image grades every point of every profile. Dark rubber, deep chamfers and steep angles thin the line until points drop out, so threshold intensity per point and reject profile segments rather than whole frames. For sheet-of-light work, the intensity image ships with every profile:

When is an IR threshold the wrong test?

A bright IR pixel can still carry wrong depth, because saturation, specular flare and multi-path returns all pass a brightness cut while corrupting the measurement, so treat IR validity as necessary and never as sufficient.

Saturation is the quiet one: a flooded pixel returns a huge value but loses the pattern contrast the matcher needs, which is why the 250 cut sits at the top of the mask. Specular highlights return plenty of light along a reflected path, so depth lands on the wrong surface. Stereo adds occlusion edges, where one imager sees a surface its partner cannot. None of these yields to a lower threshold; they yield to geometry — tilt the part, kill the glare, or keep inspection to surfaces that face the camera.

When the camera publishes its own per-pixel confidence bitmap, prefer it and read it properly — the bit meaning beats any brightness guess, as the confidence image bitflags breakdown shows. And before a surviving Z becomes a measurement, remember the depth source hands you a per-pixel value that still needs the pixel-to-XYZ step; the height map to point cloud conversion covers how pixels become coordinates.

What should you check before ordering a 3D camera?

Before ordering, confirm the sensor exposes depth plus a second aligned channel — confidence map, IR, amplitude or intensity — at matching resolution, and that the SDK returns both in one grab; without that pair, per-pixel validity stays guesswork.

Ask four questions of any shortlisted sensor:

• Second channel: which one, at what resolution and bit depth, captured with depth in the same trigger cycle.
• Registration: same pixel grid as depth, or documented rectification you can resample.
• Fill behavior: what invalid depth pixels carry, and whether per-pixel flags survive into your API.
• Bit depth: the true value out of the sensor, not the container it is packed into.

The depth camera SDK support check turns those into vendor questions, the 3D sensors category groups laser profilers, structured-light and ToF cameras by channel type, and the camera finder filters them. For a custom rig, a mono area scan camera can carry the registered second channel itself.

Size the shortlist in the camera configurator, then send part material, finish and working distance through the contact page and we will say which channel set the job needs. More depth-pipeline guides sit on the 3D topic page.