Depth image edge overlay from a single acquisition stream

A depth image and its edge overlay can come from the same acquired frame. You do not need to alternate camera presets or open another acquisition session just to create a second visualization. Keep the received depth data intact, calculate discontinuities in a separate software branch, and combine the results only for display. This procedure is for developers building live depth viewers who need stable frame correspondence without sacrificing the original measurements.

1. Decide what the overlay should represent

1. Define an edge as a depth discontinuity, not simply a change in display color.

A depth discontinuity is a sufficiently large range difference between neighboring valid samples. It can reveal an object silhouette, a recess, or a raised feature. It will not reliably reveal a painted marking on a flat surface, because that marking may have no measurable depth change.

Keep three outputs conceptually separate:

Here, retained depth means the depth delivered by the acquisition interface. It does not mean unprocessed sensor measurements or data from before the camera's internal processing.

A software discontinuity map is also not necessarily equivalent to a camera's edge-oriented preset. A preset may change depth generation itself. If your problem is missing or incorrect reconstructed depth, start with stereo disparity diagnosis after rectification, rather than trying to repair it through an overlay.

2. Acquire once and retain an owned frame

2. Give one acquisition owner responsibility for receiving depth frames, then distribute each frame to processing branches.

Use the manufacturer's documented acquisition interface. Do not assume that opening another session on the same depth stream provides an independent processing configuration. Access rules vary, and a second session can conflict with the existing owner.

For each received frame, retain:

• The depth samples and their documented scale or units.
• A frame identifier, when available.
• The capture timestamp and its clock domain, when documented.
• Image dimensions and any crop or orientation information.
• Validity or confidence information supplied with that frame.
• The processing configuration used to derive the overlay.

Check buffer ownership before passing frames to another thread. Some interfaces recycle acquisition buffers after a frame object is released. Hold a documented reference or make an owned copy; a pointer into a recycled buffer is not a retained frame.

Branch the owned frame into measurement, edge processing and display work. Each branch must treat the retained depth array as read-only. If one branch needs to filter or normalize it, that branch creates a separate array.

Attach the source frame identifier to the edge result. The renderer must combine that result with its corresponding depth frame, not whichever frame happens to be newest.

3. Mask invalid depth before comparing neighbors

3. Build a validity mask using the interface's documented invalid-value rules.

Depending on the data format, invalid samples may use a sentinel value, a non-finite floating-point value, or separate validity flags. Apply the documented representation rather than assuming every device uses the same convention. Also reject samples outside the application's accepted measurement interval.

For each neighboring pair, require both samples to be valid before calculating a depth difference. Otherwise, an invalid zero beside a valid surface can appear to be a large physical step.

For example, a valid sample at 1.20 m beside an invalid zero would produce an apparent difference of 1.20 m. That is a missing-data boundary, not evidence of an object edge.

Maintain separate masks if you want to display both:

• Depth discontinuities: valid depth compared with valid depth.
• Validity boundaries: transitions between valid and invalid samples.

Give these different display styles. An amber missing-data boundary and a white depth edge communicate different conditions without pretending that missing data has a measured distance.

Do not fill holes before the initial diagnostic test. Interpolation can create plausible-looking slopes and boundaries where no measurement exists. If the underlying problem is incomplete geometry, use the point-cloud missing-data checks to investigate it separately.

4. Set a threshold above the observed noise

4. Compare neighboring depths in physical units, using thresholds derived from representative data.

Start with horizontal and vertical neighbors. For valid samples at pixels p and q, a simple rule is:

Here, T_abs is an absolute depth-change threshold and r is a dimensionless relative threshold. Both depth values and T_abs must use the same units. The relative term is a heuristic, not a calibrated camera uncertainty model.

If you have measured per-pixel noise, a noise-aware alternative is:

The noise expression assumes independent errors. Neighboring depth errors can be correlated, especially after spatial processing, so validate the threshold against recorded frames rather than treating the formula as a guarantee.

Worked threshold example

Suppose a stationary, approximately front-facing surface shows a depth standard deviation of 5 mm at each neighboring pixel. Under the independent-error estimate:

An initial absolute threshold of 25 mm would sit above that estimate. With a relative threshold of 0.01:

• At 1 m, the relative term is 0.01 × 1000 = 10 mm, so the threshold remains 25 mm.
• At 3 m, the relative term is 0.01 × 3000 = 30 mm, so the threshold becomes 30 mm.
• A measured neighbor difference of 40 mm exceeds either threshold in these examples.

These are illustrative settings, not product specifications or universal recommendations. Repeat the stationary test at the operating distances, exposures and surface conditions you expect.

A slanted surface can also exceed a neighbor threshold without containing a sharp boundary. If slopes dominate the display, evaluate local plane residuals or a wider neighborhood instead of simply raising the threshold until all edges disappear.

5. Generate and render the paired result

5. Produce the edge mask and depth visualization from the same retained frame.

The following pseudocode shows the data flow. It deliberately avoids device-specific function names:

for each received_depth_frame:
    owned_frame = retain_reference_or_copy(received_depth_frame)
    frame_key = source_identifier(owned_frame)

    Z = convert_to_metres(owned_frame.depth)
    valid = documented_validity(owned_frame)
    valid = valid AND application_range_check(Z)

    edges = false_array_with_same_shape(Z)

    for each horizontal_or_vertical_neighbor_pair(p, q):
        if valid[p] AND valid[q]:
            threshold = max(abs_threshold_m,
                            relative_threshold * min(Z[p], Z[q]))
            if abs(Z[p] - Z[q]) > threshold:
                edges[p] = true
                edges[q] = true

    base = colorize_depth(Z, valid, fixed_display_limits)
    composite = paint_edges(base, edges)
    publish_paired_display(frame_key, composite)
    retain_or_record_original_if_required(owned_frame)

Marking both endpoints makes boundaries easier to see but produces a thicker mask. Marking only the nearer endpoint can give a thinner foreground outline, provided that convention is documented and used consistently.

Keep the display depth limits fixed while tuning. Automatic rescaling can make an unchanged scene look different when a distant object enters the view. That changes the visualization, not necessarily the measured depth.

If you rotate, resize or crop the displayed image, apply the corresponding transformation to the edge mask. Use nearest-neighbor resizing for a binary mask when you want to preserve its discrete labels.

No depth-to-color registration is needed when both layers remain on the original depth grid. If you later place the overlay on a separate color image, that becomes a calibration and coordinate-mapping task. See mapping image pixels to depth points and intrinsic parameters for depth and texture images.

6. Verify frame correspondence and processing time

6. Test correctness before adjusting the appearance of the overlay.

Start with a stationary scene containing a flat region, a clear object boundary and an area of missing depth. Confirm that invalid regions are excluded from the depth-edge calculation and that the flat region does not sparkle excessively.

Then move an object across the image. Its displayed boundary and highlighted edge should move together. A trailing outline usually points to mismatched frames, temporal filtering history, or an accumulating processing queue.

At an illustrative acquisition rate of 30 frames/s, the frame interval is:

If processing takes 45 ms per frame on a single serial worker, it cannot sustain that input rate indefinitely. The backlog grows unless acquisition is slowed, processing is accelerated, or some visualization frames are dropped.

For a live viewer, a bounded latest-frame queue is often appropriate. If you discard a visualization job, discard its paired base image and edge result together. A recording branch may need a different policy because retaining every acquired frame can be part of the requirement.

If frames themselves time out or arrive inconsistently before edge processing begins, follow the camera SDK build and timeout checks. Overlay tuning cannot fix an acquisition failure.

7. Know when software edges are insufficient

7. Separate visualization limitations from sensor-selection requirements.

A discontinuity detector cannot recover geometry the depth stream never measured. Very small surface features may fall below spatial sampling or depth repeatability. Reflective, dark, transparent or occluded regions may provide insufficient usable depth for a stable boundary.

Likewise, retaining an acquired frame does not expose an earlier internal processing stage. If you need data before a camera's internal filter, check its documented output modes. Do not infer access from a preset name or assume a second acquisition session reveals that data.

Changing hardware is not normally necessary merely to create an overlay. If the underlying depth measurement is inadequate, compare sensing approaches in the 3D sensors category and the camera finder.

The MV-DPS200P-02 is a projected structured-light camera. It is a sensing-approach candidate to evaluate, not a claim of support for the software workflow described here.

The MV-DT01WMU is a ToF sensing camera. Confirm its documented output access and suitability for your measurement task before selection.

For either approach, request confirmation of depth units, validity representation, buffer lifetime, frame metadata and access to the required stream. The catalog descriptions alone do not establish those interface details.

Checks before deployment

Use this short acceptance checklist before relying on the viewer:

• Original data retained: measurement and recording branches receive unchanged acquired depth samples.
• Ownership verified: asynchronous consumers cannot read recycled acquisition buffers.
• Units consistent: depth differences and absolute thresholds use the same physical units.
• Invalid data separated: missing-depth boundaries are not presented as measured surface steps.
• Thresholds validated: stationary noise, expected slopes and required features have all been checked.
• Frames paired: the base image and edge mask carry the same source identifier.
• Latency bounded: queue behavior remains acceptable during sustained operation.
• Configuration recorded: saved results identify the thresholds, optional filters and display settings used.

Save a representative depth sequence and replay it when changing the algorithm. This separates software changes from scene variation and makes visual comparisons more meaningful.

For help choosing a depth source or checking the required data interface, send the operating distance, smallest relevant depth change, surface types and latency target through application support. Start with the measurement requirement; the overlay should explain the data, not conceal its limitations.