Flying pixels between object and background in stereo depth edges

You point a stereo depth camera at an object and ask for its edges. Close up, the point cloud is clean: the object ends, the background begins. Push the same part out to about two metres and a curtain of points appears, gluing the silhouette to the wall behind it. Nothing exists there — the camera invented it. This guide explains the mechanism, puts a number on how fast it grows with distance, gives you a four-step test that separates raw matching error from post-processing artifacts, and ends with the filter sequence and the hardware choices that fix it.

Why do points appear between my object and the background?

They are flying pixels: edge pixels whose matching window covers part object and part background, so the matcher reports a disparity between the two true planes — a step from 1.5 m to 2.3 m produces a false shelf of points near 1.8 m.

Stereo matching finds, for every pixel in the left image, the pixel in the right image showing the same scene point, using a support window around it — 5×5 to 21×21 pixels in common implementations. On a flat surface the window sees one plane and the disparity is unambiguous. At a depth step the window sees object on one side and background on the other, so the left and right windows never line up cleanly; the best-scoring shift lands between the two true disparities. Depth is computed from disparity, so the intermediate shift becomes an intermediate coordinate: a point in mid-air.

Pixels the second camera cannot see at all get forced onto one plane or the other. The net effect at the silhouette is a ramp or shelf about one window wide, bridging object to background. The occlusion side of this is covered in why stereo depth images have an invalid band at the edges, and the same mixed-window physics also rounds real edges at range, which stereo depth edge accuracy: why distance deforms edges measures in detail.

How much worse does stereo depth get at two metres?

Four times worse, and the bridge widens with it: stereo depth error grows with the square of distance, so a camera holding about 13 mm at 1 m holds about 50 mm at 2 m and 113 mm at 3 m.

Z is depth, B the baseline between the two cameras, f the focal length in pixels at the resolution you run, d the disparity, and Δd the disparity resolution — the subpixel step the matcher resolves, a quarter pixel for a typical implementation.

Worked example: a compact stereo module with B = 50 mm and f = 400 px, matching to Δd = 0.25 px.

• At 1 m: ΔZ = 1000² × 0.25 / (400 × 50) = 250,000 / 20,000 = 12.5 mm
• At 2 m: ΔZ = 2000² × 0.25 / 20,000 = 1,000,000 / 20,000 = 50 mm
• At 3 m: ΔZ = 3000² × 0.25 / 20,000 = 2,250,000 / 20,000 = 112.5 mm

Double the distance and you quadruple the error. The bridge width follows the same trip, because the window footprint on the scene is Z × 15 / f for a 15-pixel window:

Now the ghost itself. Put an object at 1.5 m in front of a wall at 2.3 m. Object disparity: d = 400 × 50 / 1500 = 13.3 px. Wall disparity: 400 × 50 / 2300 = 8.7 px. A mixed window scores best near 11 px, which converts to Z = 400 × 50 / 11 ≈ 1,820 mm. That is your bridge: a plane of points about 320 mm behind the object and 480 mm in front of the wall, belonging to neither.

Are the bridge points in the raw depth or made by post-processing?

Switch every post-processing filter off — hole filling first — and render the cloud as points rather than a mesh: a bridge that survives is flying pixels from the matcher, and a bridge that vanishes was manufactured by your own fill and smoothing stages.

1. Disable hole filling, temporal smoothing and any edge-preserving spatial filter in your viewer or SDK.
2. Switch the 3D view from surface or mesh rendering to point rendering. A mesh draws triangles straight across the gap you are measuring and exaggerates the bridge.
3. Walk the raw depth image along the silhouette where the object meets the background and note where the intermediate depths begin.
4. Re-enable one filter at a time. The one that recreates or widens the bridge is the one to reconfigure or mask.

Hole filling copies the nearest valid depth or averages neighbours. Inside a flat plane that is exactly right; across a depth step it plants mid-air points by design. Handle genuine holes as missing points in a 3D camera point cloud describes, but mask the discontinuity first so the fill cannot cross it.

How do you filter flying pixels out of the depth image?

Reject every pixel whose local depth range exceeds three times the expected error at that range — 150 mm at 2 m for the example camera — then cut on matcher confidence and, on a live stream, on temporal agreement; the bridge goes and both true planes stay.

1. Depth-discontinuity filter. For each pixel take the depth range over a 5×5 neighbourhood and reject the pixel when max − min > 3ΔZ at that range. At 2 m that is 3 × 50 = 150 mm. A ghost at 1.8 m between planes at 1.5 m and 2.3 m differs from both by more than 300 mm and cannot survive; a pixel on a flat surface varies by noise far below the cut. You lose one or two true edge pixels with it — the correct trade, because you locate the edge as the jump between the two clean planes that remain.
2. Confidence cut. A mixed window cannot agree between left and right images, so matchers score it low. Keep pixels above 0.5 confidence, or use the edge-confidence map your SDK exposes.
3. Temporal agreement. Require a depth to repeat within ±3ΔZ over three frames, or take the median of five. One-frame mismatches disappear. Jitter grows with range like everything else; the settings that steady a long-range stream are collected in stereo camera depth jitter and IMU drift.
4. Order. Discontinuity and confidence first, hole filling last and inside planes only. Then erode the object mask by half the correlation window — 8 px in our example — before you measure anything from it.

Once the depth is clean, take the object edge as the boundary between the two surviving planes, not the last remaining pixel. When you average depth over the object afterwards, segmentation mask average depth: select valid object pixels keeps the number honest, and dropping the whole far plane up front — the same move as removing ground points from a depth point cloud — removes most false edge candidates in one pass.

Why does high-accuracy mode help only a little?

Because it refines the disparity, not the geometry: error still climbs with the square of distance, so a mode that halves Δd halves the error — 50 mm becomes 25 mm at 2 m — and the bridge shrinks but never closes.

Accuracy presets spend frame time on a wider disparity search and finer subpixel interpolation, moving Δd from 0.25 px toward 0.125 px. That is worth having, and it cannot unmix a window that physically sees two planes. The moves that change the physics:

• Cut the distance. 2 m to 1 m quarters the error and halves the window footprint, 75 mm to 38 mm.
• Add texture. An untextured object against an untextured wall is the worst case for a mixed window; a static projected pattern turns it into the best case. A stereo camera with its own galvanometer laser projector, such as the MV-DLS300P-04, makes its own texture where the scene has none.
• Widen the baseline or raise the resolution. Doubling either doubles the f·B term in the denominator and halves ΔZ at every distance.

If you build your own rig from two area scan cameras to set the baseline yourself, choose global-shutter sensors and hardware sync; a rolling shutter smears the very edge you are measuring.

When should you switch away from stereo for edge finding?

When the part can be brought inside one metre, or when you need edge position to a few millimetres beyond it: encode the geometry in the light instead of the texture, with projected structured light or a laser profiler, whose depth error grows linearly with range.

• Static part, edges to the millimetre: projected structured light. A projected structured-light 3D camera such as the MV-DPS200P-02 codes each pixel by pattern, so an edge pixel decodes to one plane instead of averaging two.
• Moving part, cross-sections on a conveyor: a laser profiler such as the MIS-DP3850-01P triangulates a line of points per profile; its own edge artifact is a short tail at height steps, bounded and calibratable.
• Navigation, volume, logistics over several metres: stay with stereo. The bridge is filterable there because you follow planes and obstacles, not sub-millimetre edges. The wider method trade-offs are laid out in 3D scanning method selection for inspection.

One safety note belongs here: treat every laser projector and profiler as an eye hazard. Never look into the beam or its mirror reflection, and keep the laser class label on the housing readable.

What should you check before buying a stereo camera for edge work?

Compute ΔZ at your farthest edge before anything else: the baseline, the focal length in pixels at the resolution you will run, and the subpixel step the SDK claims together give the number that decides whether the job is possible at all.

Say your edge tolerance is 20 mm of depth separation. Invert the error formula for the example camera: Z = √(ΔZ × f × B / Δd) = √(20 × 400 × 50 / 0.25) = √1,600,000 ≈ 1,265 mm. Past about 1.3 m that camera can no longer separate your edges, no filter will change that, and the decision is to move closer or change method.

• Baseline. A compact module carries around 50 mm; each doubling halves ΔZ at every distance. Mounting space buys accuracy.
• Disparity resolution. Quarter-pixel is standard; eighth-pixel is what high-accuracy modes chase. Ask for the number, not the label.
• Filter access. The SDK must expose a discontinuity or edge filter, a confidence map, and per-region control of hole filling. If it does not, plan to filter in your own code with the thresholds above.
• Worst-case scene test. Dark object against a bright wall, at the farthest working distance you will accept, before you commit to the part.

Browse the 3D sensors range, open the camera finder, or start the system configurator at the camera step. More depth-camera fixes are collected under 3D vision guides, and if your scene is unusual, bring your numbers to us through the contact page and we will run the error math with you.