Segmentation mask average depth: select valid object pixels

A segmentation boundary tells you which image pixels belong to an object, but it does not make those pixels valid depth measurements. Timing errors, different camera viewpoints, inference resizing, and background pixels can all produce a plausible but incorrect average. This procedure is for engineers connecting instance segmentation to a depth-processing pipeline. You will build a per-object sampling region, calculate defensible depth statistics, and flag measurements that lack enough usable surface data.

Decide what the depth statistic should mean

Start with one object instance, not every pixel carrying its class label. A class such as “container” can describe several objects at different distances. Combining their masks creates a class-level statistic, not an individual object's depth.

Use an instance record containing its identifier, class, mask, and capture timestamp. The identifier should remain associated with the mask throughout processing. If identity must persist across captures, the image sequence segmentation and tracking guide explains the additional tracking problem.

Choose the statistic according to the decision:

These are image-sampled surface statistics. They are not automatically the distance to an object's geometric center or an area-weighted average over its physical surface.

Prepare synchronized, spatially registered inputs

You need the segmentation source image, its per-instance masks, a corresponding depth image, and the calibration information describing that depth grid. Also retain the documented depth scale, invalid-value conventions, and any available validity or confidence output.

Matching image dimensions is necessary for direct indexing, but it is not proof of registration. Separate imaging channels can have different viewpoints even when their arrays have identical shapes. Follow the depth-to-color registration checks before reading depth at a segmentation pixel.

Prefer documented registration functions supplied with the acquisition system. Registration may leave holes where one channel sees a surface that the other cannot observe. Those holes are missing measurements, not permission to substitute background depth.

Pair frames using capture timestamps or documented synchronization metadata rather than arrival order. Transport and processing delays can make consecutive callbacks refer to different capture times. The multi-camera buffer diagnostic is useful when valid images occasionally appear paired with unrelated depth.

As a timing estimate, an object moving at 200 mm/s travels 2 mm during a 10 ms mismatch:

That displacement can shift a narrow mask onto another surface. Establish the allowable mismatch from your measurement tolerance and expected motion.

Build the sampling region in numbered steps

1. Keep each instance separate. Use one binary mask per detection, or an instance-ID image accompanied by class metadata. A class-only image cannot distinguish touching objects of the same class. Connected components may separate disconnected regions, but they do not reliably recover object identity.

2. Obtain a filled mask. If the annotation contains polygon vertices, rasterize the polygon interior rather than sampling only its outline. Confirm how the format represents holes, multiple components, and overlapping instances. The segmentation polygon drawing guide covers the label-construction side; this procedure begins with the filled object region.

3. Reverse preprocessing transforms. Remove inference padding and reverse resizing or cropping before sampling the target depth grid. Preserve the actual transform applied to each image, including unequal horizontal and vertical scale factors where relevant.

4. Use the correct resampling method. Resize binary masks and instance-ID images with nearest-neighbor interpolation. Interpolating IDs creates values that are not valid identifiers. Probability maps can be interpolated before thresholding, but the threshold and transform must remain explicit. The image resampling guide explains why continuous images and discrete labels need different treatment.

5. Exclude uncertain boundaries. Consider eroding the final binary mask by 1–3 pixels as an initial engineering trial. Evaluate the loss of useful surface area before adopting that setting. Erosion is especially risky for thin features, small objects, and masks with narrow connecting regions.

6. Apply measurement validity rules. Intersect the instance region with finite depth, documented sensor validity, an appropriate operating range, and any justified confidence threshold. Count accepted pixels and retain their fraction of the sampling region.

For normalized polygon coordinates, many formats define continuous image coordinates using image width and height. For example, a normalized horizontal coordinate of 0.25 maps to 160 in an image 640 pixels wide: 0.25 × 640 = 160. Confirm the annotation format's boundary and pixel-center conventions before rasterizing; they are not universal.

For a letterboxed inference image, the inverse coordinate transform commonly has this form:

This reverses preprocessing only. It does not perform geometric registration between separate cameras.

Calculate statistics from accepted pixels

Represent the object region as a Boolean array rather than a dictionary of coordinate-to-class entries. The array directly selects samples and avoids storing a separate coordinate pair for every foreground pixel.

Let M identify the selected instance, V identify valid depth samples, and R identify samples within the intended measurement range. Their intersection defines the accepted set:

The following Python example uses array operations. Supply an already registered depth image and an already transformed, optionally eroded mask. The validity image must come from documented validity rules for your acquisition output.

import numpy as np


def instance_depth(mask, raw_depth, sensor_valid, depth_scale,
                   z_min, z_max, min_count=100, min_fraction=0.60):
    mask = np.asarray(mask, dtype=bool)
    raw_depth = np.asarray(raw_depth)
    sensor_valid = np.asarray(sensor_valid, dtype=bool)

    if mask.shape != raw_depth.shape:
        raise ValueError("Mask and depth grids must match")
    if sensor_valid.shape != raw_depth.shape:
        raise ValueError("Validity and depth grids must match")

    region_count = int(np.count_nonzero(mask))
    z = raw_depth.astype(np.float64) * depth_scale
    accepted = (
        mask & sensor_valid & np.isfinite(z)
        & (z >= z_min) & (z <= z_max)
    )
    samples = z[accepted]
    count = int(samples.size)
    fraction = count / region_count if region_count else 0.0

    output = {
        "sample_count": count,
        "region_count": region_count,
        "valid_fraction": fraction,
    }
    if count == 0 or count < min_count or fraction < min_fraction:
        output["status"] = "insufficient_depth"
        return output

    output.update({
        "status": "ok",
        "mean_z": float(np.mean(samples)),
        "median_z": float(np.median(samples)),
        "p10_z": float(np.percentile(samples, 10)),
        "p90_z": float(np.percentile(samples, 90)),
    })
    return output

The sample-count and valid-fraction defaults are illustrative acceptance criteria, not sensor specifications. Set them from object size, required tolerance, and validation data. The fraction here uses the post-erosion region as its denominator; separately record the original mask area to reveal excessive erosion.

Convert depth units once. For example, if documented output uses 0.001 m per count, a raw value of 600 represents 0.600 m. Do not assume that scale applies to another acquisition mode or device.

Work through contamination and missing data

Suppose an instance mask contains 1,200 pixels. Erosion leaves 1,000 pixels, of which 150 fail sensor validity and another 50 fail the selected depth range. That leaves 800 accepted samples:

The count passes the example acceptance criteria, but it does not prove that every sample belongs to the intended surface. Consider 760 accepted samples at 600 mm and 40 background samples at 900 mm:

A small contaminated region shifts the mean by 15 mm. The median resists this particular contamination, but it is not a universal correction. If the object genuinely has surfaces at different depths, replacing the mean with the median changes the measurement definition.

Inspect the accepted-pixel overlay when mean and median disagree. Check registration, segmentation boundaries, and visible surface geometry before applying statistical rejection. Blindly removing distant samples could discard a legitimate rear surface.

The bounding-box depth guide provides the closest alternative when a reliable mask is unavailable. A mask narrows the sampling region, but both methods still depend on valid coordinate mapping.

Distinguish axial depth from camera-to-point distance

A depth image may report axial Z, Euclidean range, or another documented quantity. Read that definition before labelling an output “distance.” For a rectified pinhole image carrying axial depth, deprojection uses:

Use intrinsics belonging to the actual sampling grid. Cropping, decimation, registration, and resizing can change which calibration parameters apply. The depth and texture intrinsic conversion guide explains this dependency.

If you need mean camera-to-point distance, deproject every accepted sample, calculate its distance, and average those distances. Do not convert mean Z using a single mask-center ray. The axial-depth versus true-distance explanation develops that distinction.

For grasping or dimensional measurement, an aggregate depth alone may be inadequate. You may instead need a local surface fit, a selected contact patch, or a calibrated three-dimensional point with an explicit uncertainty budget.

Select acquisition hardware around the required outputs

Start with the 3D sensors category and use the camera finder to narrow acquisition candidates. Confirm registered depth availability, validity information, calibration access, timestamp behavior, and the depth quantity before selecting hardware.

The MV-DPS200P-02 is catalogued as a projected structured-light camera. Treat it as an acquisition candidate, not evidence that a particular segmentation pipeline is supported.

The MV-DT01WMU is catalogued as a ToF sensing camera. Verify its required data outputs and integration path for your application.

If segmentation needs a separate texture image, review area scan cameras. Adding another camera also adds synchronization and geometric-registration work; an ordinary intensity image does not supply depth by itself.

Do not choose acquisition technology from the aggregation formula alone. Test representative surfaces, distances, orientations, and motion using the intended capture mode. Unsupported assumptions about depth access cannot be repaired by downstream statistics.

Verify the pipeline before deployment

Use a static target first, then introduce motion and additional objects. Keep overlays of the instance boundary, accepted pixels, and rejected pixels beside the numerical output.

• Known plane: compare reported depth with a reference using the same axial or range definition.
• Two instances: place same-class objects at different depths and verify independent outputs.
• Image edges: check registration and validity across the full operating region, not only near the center.
• Resolution changes: confirm mask dimensions and calibration after changing processing modes. Use the depth decimation diagnostic when the output grid changes unexpectedly.
• Missing surface data: verify that insufficient valid coverage produces an explicit failure state rather than zero depth.
• Boundary sensitivity: compare the statistic before and after erosion, while recording retained area.
• Motion: inspect timestamp pairing and accepted-pixel overlays at the intended operating speed.

Save the instance ID, class, timestamp, sample count, valid fraction, mean, median, units, and coordinate-frame definition with each measurement. These fields make later discrepancies traceable.

For acquisition planning, start with the camera configuration step. If registration or output availability remains uncertain, send your image dimensions, capture sequence, surface examples, and required tolerance through applications support before committing to hardware.