3D camera confidence image values: decode bitflags and mask bad pixels

You request a depth frame from a 3D camera, and the SDK hands you a second image labeled confidence, full of values like 577, 513 and 16417. None of them matches the single flags in the manual, and treating them as brightness silently corrupts the point cloud built on top of them. These numbers are 16-bit bitflags: one unsigned integer per pixel packing up to sixteen sensor checks. This page decodes any value by hand, combines the cross-talk and flying-pixel flags into one mask constant, drops bad pixels in NumPy, and sorts the flags by task for picking, volume measurement and overlays.

A confidence image is a bitfield, not a grayscale map

Decode a confidence value with a bitwise AND, one flag at a time, and never compare the whole number to the manual's table. Each bit is the answer to one sensor check: range inside the valid window, signal strong enough, geometry plausible, no cross-talk from a neighboring pixel. A flag-word camera transmits only the combined integer, so splitting it back into flags happens in your code.

Bits add up by OR, not by meaning. A pixel that fails three checks reports the sum of three bit values, which is why 577, 513 and 16417 show up while the table lists only powers of two. A value of 0 is a fully clean pixel; 65535 (2^16 − 1) sets every flag at once.

Time-of-flight and projected structured-light sensors commonly stream this channel beside depth, registered per pixel at the same resolution. Time-of-flight is the classic case:

The flag table behind 577

Each device family assigns its own bits and prints them in its confidence description. The table below is that description for one widely used 16-bit ToF platform, and every example value in this page decodes against it. The layout — one bit per check, values combined by OR — and the arithmetic that follows are identical on any camera that uses flag words.

In that reference, 2, 4, 4096, 16384 and 32768 form the cross-talk group: the bits that light up on flying pixels, the smeared points hanging in space where a single pixel sees both the box and the background behind it. The everyday named bits are 1 invalid, 64 invalid range, 32 exposure index, 512 edge or isolated pixel and 1024 implausible.

Decoding is addition run backwards. Every word is a sum of distinct powers of two from the table, so subtract the largest value that fits and repeat until you reach 0:

• 577 = 512 + 64 + 1 — edge or isolated pixel + invalid range + invalid
• 513 = 512 + 1 — edge or isolated pixel + invalid
• 97 = 64 + 32 + 1 — invalid range + exposure index + invalid
• 4129 = 4096 + 32 + 1 — cross-talk + exposure index + invalid
• 16417 = 16384 + 32 + 1 — cross-talk + exposure index + invalid

On a box-on-pallet frame those sums concentrate along box edges and around bright windows, which is exactly where flying pixels and range errors live.

Read the flags in code: bitwise AND, not equality

The whole test is one operator. AND a pixel with a candidate flag: any nonzero result means that flag is set. Applied to a full image, it returns either the flag value or 0 at every pixel — a filter, not a lookup.

import numpy as np

# confidence: uint16 array, rows x cols, straight from the camera driver
confidence = np.array([[577, 513], [4608, 97]], dtype=np.uint16)

# the cross-talk group, OR-ed into one constant
flags = np.uint16(32768 | 16384 | 4096 | 4 | 2)     # 53254

bad = np.not_equal(confidence & flags, 0)            # boolean mask
invalid = np.not_equal(confidence & 1, 0)            # invalid flag alone

print(flags)              # 53254
print(bad)                # [[False False], [ True False]]
print(invalid)            # [[ True  True], [False  True]]
print(confidence[~bad])   # [577 513 97]

Each line checks out against the table: 577 carries none of the cross-talk bits, 4608 = 512 | 4096 carries cross-talk, and 97 carries only invalid range, exposure index and invalid.

Mask flying and cross-talk pixels

OR the cross-talk group into one constant, once, in one place: 2 | 4 | 4096 | 16384 | 32768 = 53254. Every frame then reduces to a single AND:

import numpy as np

# confidence: uint16 (rows, cols); depth: float32, same shape, same frame
flags = np.uint16(32768 | 16384 | 4096 | 4 | 2)      # 53254

bad = np.not_equal(confidence & flags, 0)             # boolean mask
depth_clean = np.ma.masked_array(depth, mask=bad)     # flying pixels hidden

removed = np.count_nonzero(bad)
share = removed / bad.size * 100
# 640 x 480 frame with 9,824 flagged pixels:
# 9824 / 307200 * 100 = 3.2 %

A 3.2% share is ordinary for a pallet scene with the full cross-talk group, so watch the trend instead of a fixed limit. When the share climbs frame over frame, something moved: a new reflective bag, a shifted mount, an exposure that no longer fits.

Feed the masked array, not the raw depth, into your point-cloud conversion: flagged pixels are excluded there, so no flying point reaches pose estimation or volume. The mapping of image pixels to 3D points then runs on clean input, and any stray invalid pixel that still reaches deprojection shows up as a zero at the origin — the diagnosis in depth pixels that deproject to zero.

The flying-pixel artefact itself — why it forms where object and background share a pixel — is the same physics in stereo depth, and flying pixels between object and background in stereo depth edges works through it.

Keep the invalid flag separate from the cross-talk flags

Value 1 and the cross-talk group fail for different reasons, so give them two masks and two policies. Invalid means no usable measurement — the depth there is not inaccurate, it is absent. Cross-talk means a measurement exists but was corrupted by mixed returns from two surfaces in one pixel.

• Picking: demand both masks clear on the grasp surface. One flying pixel on a box corner tilts a fitted plane, and the error is bounded by the depth gap the pixel straddles — a foreground at 300 mm over a background at 700 mm can put that point up to 400 mm off the true edge.
• Volume and counting: drop both masks, then average over the surviving object pixels — the approach in selecting valid object pixels with a segmentation mask.
• Overlay and teaching: keep everything and color-code the flags, so you can see windows and edges while you position the sensor. Frame-edge isolation has the same origin as the invalid band at the edges of stereo depth images; here it arrives as the 512 edge flag instead of an empty strip.

What a confidence image does not fix

The flags certify the sensor's own checks, not your set-up. A dark, polished or steeply angled surface can pass every check and still return a weak, noisy distance; exposure time, range window and mounting geometry remain your levers, and the 3D camera working distance and positioning guide walks through them.

A mirror-flat part can also return a plausible-looking wrong distance that no flag catches — multipath the sensor did not detect. Prove critical geometry against a gauge block or a second viewing angle, and let the mask handle the rest.

A minority of sensors stream a plain score, 0–255 or a 0–1 probability, instead of bit words. Threshold those directly — 0.5 is the usual first cut. Never threshold a flag word: the words have no order, and 512 (an edge) sitting above 33 (exposure index + invalid) says nothing about which pixel is worse.

Choose a camera whose confidence channel you can read

Before you order, put four things on your list:

• The confidence description for the exact model, not the series flyer.
• SDK access to the raw 16-bit words, per pixel, registered with depth — same rows and columns.
• Exposure and range-window settings under your control, because most flags are best fixed at the source: raise exposure before you grow the mask.
• A 90-day return window, so you can bench-test the flag table on your own parts.

Both families sit in the 3D sensors range, filtered by type — laser profiler, stereo or structured light, logistics volume and ToF — in the 3D camera finder. Structured-light and stereo options for scenes where a projected pattern or a second view fits better than single-sensor time-of-flight:

Decode checklist

• Get the confidence description for your exact model and write the bit values into one constants block.
• OR your bad flags into one mask and note the decimal value: 2 | 4 | 4096 | 16384 | 32768 = 53254 on the example table.
• Test with &, never with ==.
• Keep the invalid mask separate from the cross-talk mask and apply them per task.
• Print np.unique(confidence) on commissioning frames and re-check it whenever the scene changes.
• Track the removed-pixel share per frame — 3.2% in the example above — and react to the trend, not to a fixed limit.
• Verify masking on a gauge part: masked and unmasked readings must agree within the sensor's accuracy spec on clean surfaces.
• Pass the masked array, not raw depth, to deprojection and pose estimation.

That is the whole workflow: one constants block, one AND, one masked array. If you would rather have the flag table checked against your scene before you commit code, send the part drawing and tolerances through the contact page. To step through sensor, mounting and controller choices yourself: