The fix comes in three moves, and you can apply the first two today. Cover the wall and floor in matte light-gray tape so the projector's 850 nm pattern comes back, and run a 10 to 20 mm matte black band around the wall base so the subject's outline can never merge with the wall. On footage already recorded, fit the wall as a cylinder and cut everything protruding more than 5 mm inward — at 600 mm a stereo camera carries about 1.7 mm of depth noise, so a 5 mm cut is three sigma and a 15 mm subject clears it three times over. Below: the mechanism behind those numbers, a five-minute test that separates absorption dropout from wall aliasing, the scene changes, the Python for the radial cut, and the hardware that removes the problem outright.
Two failures that look like one
A dropout is a pixel where the camera wrote zero because it measured nothing there. The render paints it black. Dark, matte and steeply angled surfaces produce them in solid blocks, and the block follows the material, not the geometry: the same shape on a light surface returns depth. For filling and filtering strategies around those holes, see missing points in a 3D camera point cloud.
Wall aliasing is the opposite. Every pixel has valid depth, but subject and wall differ by less than the camera's noise, so the color scale paints both the same color. The outline you need is still in the geometry, and only a geometric cut or a learned mask will find it.
The decision is short: zero-valued pixels mean a reflectance problem; valid depth that merges means a segmentation problem. The rest of this page fixes both.
Why dark and matte surfaces return nothing
Every projector-based depth camera measures its own light. Active stereo units project an infrared speckle pattern near 850 nm — 940 nm in models built to resist sunlight — and correlate it between two sensors. Structured-light cameras decode a coded pattern at the same wavelengths, and time-of-flight cameras time a flood pulse. All three need photons to come back.
Reflectance at that wavelength decides. Matte black paint and black flocking return a few percent of incident near-infrared; matte white and light-gray card return on the order of 80 to 90 percent. A correlator looking at a few-percent return sees two flat gray patches with no speckle contrast, finds no match, and writes zero. A pattern decoder cannot read what never reached the sensor. Visible color tells you nothing: the near-infrared reflectance of tapes and plastics varies independently of how they look to your eye.
Motion adds dropouts of its own. In a dark arena the auto exposure lengthens the integration time, a moving subject smears the projected pattern in both images, and correlation fails exactly where the subject is. The exposure behavior behind that is the same one covered in depth camera frame rate drops: exposure settings cap the FPS.
Tell the two apart in five minutes
- Capture one still frame with the subject against the wall and export the raw depth array — 16-bit values in millimetres — not the colored render.
- Count zero-valued pixels inside the wall band. A solid block of zeros over the dark material is absorption dropout. Zeros hugging only the subject's silhouette are stereo occlusion at the disparity edge, which is normal at a wall base and needs no fix.
- Tape a sheet of matte white paper over the failing area and recapture. If the paper returns depth, absorption is confirmed. If depth returns but subject and wall still merge, you have aliasing.
- Read the depth values on both surfaces. A difference under about twice the noise figure from the worked example below means no color scale will ever separate them — go straight to the height cut.
- Preview any new material before committing: most compact cameras and front-facing phone cameras see near-infrared, and a sample where the projected dots stand out bright gives the correlator contrast to work with.
Fix the scene before the next recording
You cannot change footage already shot, but every future capture improves with two changes.
Make the wall and floor light and matte. Light-gray matte tape over the full wall height lifts the near-infrared return from a few percent into the range where the correlator always works, and it shortens the exposure the camera asks for, which helps a moving subject.
Add a narrow separator band where the outline matters most. A matte black band, 10 to 20 mm wide, run around the wall base and partway up the side, returns nothing and renders as a hard zero-depth gap. A subject in front of that gap is bounded by no-depth on one side and its own depth on the other, so its outline cannot merge with the wall. Keep the band narrow: whatever stands on it loses the depth under its feet, and that cannot be recovered later.
Avoid glossy finishes everywhere. A glossy surface mirrors the pattern away from the sensor and drops out like a black one, but no amount of reflectance fixes it.
Before re-finishing a whole arena, pre-screen the candidate materials in the band the projector lives in. A short-wave infrared area scan camera such as the LS-IRC-0029, with 15 μm × 15 μm pixels, shows you which tapes and paints still reflect beside the projector's wavelength.
- Sensor format
- 1/1.2"
- Lens mount
- C-mount
- Interface
- GigE
- Resolution
- Up to 3 MP
Segment the footage you already recorded
With past recordings fixed, work the geometry: the wall is a cylinder, the subject protrudes inward, and protrusion is measurable.
- Open the recording through your SDK's playback path. In the Python SDK used for these files, playback starts when you place an
enable_device_from_filecall on the line beforepipe.start. - Convert each depth frame to metric XYZ with its per-axis scale factors — the conversions and their failure modes are covered in depth image to point cloud: apply per-axis scale factors and numpy depth image to point cloud. A depth frame is already a height map in this form; see 2.5D height map to 3D point cloud: pixels to XYZ.
- Fit the wall. With the camera looking straight down the arena axis, the wall sits at one radial distance and a percentile of the radius finds it; off-axis, fit a cylinder by least squares as in measuring surface height difference from a 3D point cloud.
- Mask everything closer to the axis than the fitted wall by your threshold.
This simple cut assumes the camera looks down the arena axis and the frame is filled by the arena:
import numpy as np
depth = np.asarray(depth_frame, dtype=np.float32) # 16-bit millimetre depth in, 0 = no return
depth[depth == 0] = np.nan # dropouts must not average in
h, w = depth.shape # 480, 848 in the worked example
f_px = 447.0 # from the horizontal field of view
yy, xx = np.mgrid[0:h, 0:w]
X = (xx - w / 2) / f_px * depth
Y = (yy - h / 2) / f_px * depth
r = np.sqrt(X**2 + Y**2) # radial distance from the camera axis
wall_r = np.nanpercentile(r, 90) # the wall is the outermost annulus
mask = (r < wall_r - 5.0) & np.isfinite(depth) # 5 mm cut, about 3 sigma at 600 mm
print(wall_r, int(mask.sum())) # sane run: wall_r near the wall radius
assert 0 < mask.sum() < 0.3 * h * w # mask covers a fraction, never all
If the arena is crowded or subjects cross each other, the learned route is the right escalation: annotate masks and train on the infrared and depth streams rather than color. Overlay 2D image on depth map covers the alignment, and 3D camera 2D template matching the locate-first pattern that pairs well with the cut.
Worked example: numbers for a 600 mm arena
Assume an active-stereo depth camera with an 848 × 480 depth image, an 87° horizontal field of view, a 95 mm stereo baseline and a 0.2 pixel disparity error, mounted 600 mm above the arena floor. The subject presses against the wall, protrudes 15 mm inward, and moves at 100 mm per second.
Depth noise at 600 mm is about 1.7 mm one-sigma, and it grows with the square of distance: the same camera at 1000 mm gives 1000² × 0.2 / (447 × 95) ≈ 4.7 mm. Two decisions fall out:
- Cut at 5 mm above the fitted wall — about three sigma, so wall pixels almost never cross it, and a 15 mm protrusion clears it three times over.
- The 15 mm outline spans 15 / 1.34 ≈ 11 pixels, enough contour for a mask and a tracker.
Exposure gets its own limit. One pixel of blur at 100 mm per second allows an integration time of 1.34 / 100 ≈ 13 ms, so cap exposure at 10 ms and win the light back through wall reflectance. Where those controls live is covered in 3D camera 2D stream exposure, iris and gain, and smoothing the depth before the cut in stereo depth map noise.
For the 2D stream you would train a network on, the same pixels-per-feature arithmetic is the camera resolution calculator:
Change any input and the same three formulas re-scale every decision above.
Hardware that removes the problem outright
If the arena must stay dark to the animal and light matte tape is off the table, change the sensing mechanism, not the settings.
| Method | What it needs from the surface | On matte black | The fix that works |
|---|---|---|---|
| Active IR stereo | speckle contrast in two images | flat gray, no match, hole | raise near-IR reflectance |
| Structured light | a decodable projected pattern | pattern too faint, hole | raise reflectance or close the distance |
| Time of flight | enough return photons per pixel | noisy or invalid range | raise reflectance, avoid gloss |
| Laser-line profiling | one bright line returning | usually returns — power concentrated in a line | matte finish; glossy black still fails |
Projected structured light, laser-galvanometer stereo and ToF sensing each attack dark surfaces with a different mechanism, and all three sit in the 3D sensors range.
- Type
- Stereo / structured light
- Type
- Stereo / structured light
- Type
- Depth / ToF camera
For dark matte parts on a tray, a laser profiler builds the height map line by line, and the segmentation becomes exactly the radial cut from the section above. The workflow in 3D surface defect detection from point cloud height maps is built on that pipeline.
- Type
- Laser profiler
Compare the mechanisms side by side in the camera finder, or start a build at the camera step.
Check these before blaming the surface
- Focus and calibration first. A stereo pair out of focus or poorly calibrated loses depth everywhere, and dark surfaces simply go first — the diagnosis sequence is in poor stereo depth quality: fix calibration and focus mismatch first.
- Exposure ramp. Read the integration time the auto loop settles at in your dark arena; if it climbs past your motion limit of about 10 ms, the scene is too dark for the settings, not the camera.
- Occlusion versus absorption. Zeros that hug the subject's silhouette and move with it are stereo occlusion at the disparity edge — normal, and untouched by any reflectance fix.
- Playback path. A recording read with live-camera defaults can drop frames or return stale depth; the playback call order in the section above matters.
When the usual answer is wrong
The black separator band is the standard trick, and it fails when the measurement you need is the contact itself. A subject standing on the band shows no floor depth under its feet; if grip or posture matters, keep the wall light and rely on the height cut.
A learned mask is the wrong first move when the depth image already holds the answer. The wall is a fitted cylinder and the subject is everything protruding inward — geometry beats a network you must annotate, and it re-tunes in one number when the threshold changes.
Depth differencing against a background frame fails for the same reason aliasing does: at the wall, subject and background differ by less than the noise. That is why the cut is radial, not temporal.
Range is the last limit. Stereo noise grows with the square of distance, and at ranges of metres a wider baseline or a different method wins. For large dark scenes, spinning LiDAR versus a stereo depth camera in a long dark tunnel compares the two directly.
Checklist for the next recording
- Matte, light finish on every surface the subject can reach; qualify materials with the paper test or a near-infrared preview.
- One 10 to 20 mm black band at the wall base, only where you do not need contact depth.
- Exposure capped near 10 ms for a subject moving about 100 mm per second.
- Camera axis on the arena axis, so the percentile wall fit holds.
- Raw depth frames logged, not rendered video, with playback checked end to end on one known frame.
- The 5 mm radial cut scripted and run on that frame before the first real session.
Bring the arena dimensions and one recorded frame to the contact page and we will size the cut with you.
Questions
Why do depth map dropouts appear only over dark surfaces?
The projector works in near-infrared, usually near 850 nm. Matte black materials return a few percent of that light, the correlator or pattern decoder sees no contrast, and the pixel is written as zero depth. Light matte surfaces return on the order of 80 to 90 percent and hold depth reliably. Qualify any material with the paper test: tape matte white paper over it, and if the paper returns depth, absorption is the cause.
Can the missing depth values be recovered in software?
No. A zero-depth pixel was never measured, so no filter can reconstruct it honestly. Fill holes only for display. For measurement, segment around them: the height-above-wall cut uses the valid pixels on the subject and the fitted wall cylinder and simply ignores the holes, which is why it survives dropouts that defeat background subtraction on depth.
Is black tape at the wall base a good fix?
Yes, as a separator. A 10 to 20 mm matte black band returns no depth and leaves a hard zero gap between the subject's outline and the wall, so the subject stands out whenever it is in front of the band. Keep it narrow: whatever stands on the band loses the depth under its feet, and that data cannot be recovered from the recording later.
How close in depth can two surfaces be before the color scale merges them?
Within roughly twice the camera's depth noise. At 600 mm the worked example gives 1.7 mm one-sigma, so surfaces less than about 3.5 mm apart can render as the same color. No recolorization fixes that permanently. Separate them geometrically with the radial cut, or change the reflectance so the outline lands on clearly different depth values.
Which depth method handles black matte parts best?
A laser-line profiler usually returns on matte black that defeats speckle correlation, because the optical power is concentrated into one line. Structured light and time-of-flight both need the surface to return the pattern or the pulse, so they fail the same way stereo does. Glossy black is the shared failure case: it mirrors the light away from the sensor no matter how much power you apply.
Does motion cause dropouts too?
Yes. In a dark arena the auto exposure lengthens the integration time, and a moving subject smears the projected pattern in both images, so correlation fails exactly on the subject. Cap exposure at one pixel of motion — about 10 ms for a subject moving 100 mm per second with a 1.34 mm pixel footprint — and recover brightness through wall reflectance, not through exposure.




