A feature detected in a color image does not automatically identify a depth pixel. The cameras can have different viewpoints, resolutions, distortion, and visibility. For inspection software, the practical task is to find measured depth samples that project near the detected feature, then decide whether those samples belong to its surface. This workflow shows you how to perform that lookup, retain the original depth coordinates, and reject misleading matches near boundaries.
Choose the output before choosing the mapping
A color pixel defines a viewing ray, not a unique point in space. Without a measured distance or another geometric constraint, transforming that ray cannot identify a unique depth pixel. Assigning an arbitrary distance merely chooses one possible point along it.
For color pixel to depth lookup, start with measured depth samples instead. Deproject them into space, transform them into the color camera frame, and project them onto the color image. Search those projected locations for a suitable match.
Decide which output your application requires:
- Original depth pixel: return its source column and row, plus its measured depth.
- Point in space: return coordinates and explicitly name the coordinate frame and units.
- Depth along the color camera axis: return the transformed point's color-frame Z coordinate.
These outputs are related, but they are not interchangeable. A source depth value can differ from the transformed color-frame Z value.
If you already have depth registered onto the color grid, you may be able to sample that output directly. First establish its geometry using raw versus rectified depth-to-color registration. Do not apply the same registration twice.
Gather calibrated streams and preserve their metadata
You need a color frame, a corresponding depth frame, and the calibration describing both active stream profiles. Confirm the depth representation: some interfaces report axial depth, while others report distance along a ray. Use the documented deprojection method for that representation.
Record the following with each acquisition:
- Image dimensions, timestamps, and stream identities.
- Intrinsic parameters and distortion models for the images being processed.
- The depth-to-color rigid transform, including its direction and translation units.
- Depth scale, invalid-value rules, and available confidence information.
- Any crop, resize, rectification, or decimation applied before detection.
Follow raw stereo image calibration checks when stream identities or calibration metadata are uncertain. For moving objects, use timestamp-based frame matching to avoid pairing a detection with a different scene state.
The 3D sensors category and camera finder are starting points for hardware selection. The MV-DPS200P-02 is a projected structured-light camera.
- Type
- Stereo / structured light
The MV-DT01WMU is a ToF sensing camera.
- Type
- Depth / ToF camera
These catalog classifications do not establish support for a particular registration interface, color stream, or calibration export. Confirm those requirements before selection. Adding a separate camera from the area scan cameras category requires calibration and appropriate timing between devices; a shared connection alone does not provide registration.
Use this processing order:
- Restore the detection to the calibration image coordinates.
- Project valid measured depth samples into the color view.
- Resolve competing samples and search a bounded neighborhood.
- Return the source sample and a clearly defined geometric output.
- Validate correspondence across position, distance, and motion.
1. Restore the detected point to the correct pixel grid
Treat pixel coordinates as a documented convention, not an assumption. Common image-array access uses row first, while geometric projection uses horizontal coordinate first. For example, image[row, column] commonly corresponds to a projection coordinate (u, v) = (column, row).
Check whether your library places pixel centers at integer coordinates or uses another convention. Keep that convention consistent through deprojection, projection, rasterization, and display overlays. Adding a half-pixel correction without checking the interface can introduce an error rather than remove one.
Undo detector preprocessing before using camera calibration. For a simple resized crop without padding, the relationship is:
This expression assumes both coordinate systems use the same origin and pixel-center convention. A resize implementation can require additional center offsets. Letterboxing also requires removing padding before reversing the scale.
For example, suppose a crop begins at column 100 and is resized by a factor of 0.5. A detector coordinate of 80 maps to 80 / 0.5 + 100 = 260 in the original horizontal coordinate system, under those assumptions.
Keep the detection as floating-point coordinates until you must address an array. Early rounding discards useful information and makes small coordinate errors harder to diagnose. If processing changes the depth dimensions, review depth resolution and decimation before reusing earlier intrinsics.
2. Project measured depth samples into the color view
For each valid depth sample, obtain its point in the depth camera frame. In the ideal pinhole case, with axial depth and an already undistorted pixel, the calculation is:
Here, R_dc and t_dc transform depth-frame coordinates into the color frame. Translation and point coordinates must use identical length units. Reject points with nonpositive color-frame Z before projection.
These equations are not a replacement for distortion handling. Use documented projection and deprojection functions for the active image models. Project into raw color coordinates when the detector sees a raw image, or into the corresponding rectified coordinates when it sees a rectified image.
Retain a record for every projected sample:
projected color coordinate
original depth column and row
original depth value
point in the depth camera frame
point in the color camera frame
available validity or confidence information
Keeping this provenance avoids confusing a resampled output pixel with an original measurement. For broader region-based association, mapping bounding boxes to depth points explains the related selection problem.
3. Resolve visibility before selecting a nearby match
Several depth samples can project into the same color neighborhood. Some may describe the front surface; others may describe a background surface that is hidden from the color camera. Simply taking the nearest projected coordinate does not resolve this ambiguity.
A practical first pass uses a visibility buffer in the color grid. Where samples compete for an output cell, retain the smallest positive color-frame Z. This approximates front-surface visibility at that raster resolution. It is not proof of visibility: sparse samples, different viewpoints, and thin objects can still leave an incorrect or incomplete representation.
Next, search around the detected color coordinate. Define the acceptance radius in color-image pixels, not depth-image pixels. Different resolutions and focal lengths make those tolerances physically different.
for each valid source depth sample:
deproject using the source image model
transform from depth frame to color frame
reject nonpositive color-frame Z
project using the target color image model
reject projections outside the target image
update the visibility buffer using color-frame Z
retain the source sample identity
collect visible candidates within the search radius
reject candidates inconsistent with the target surface
if no acceptable candidate remains:
return NO_MATCH
otherwise:
return the selected sample and its projected distance
Use the documented pixel convention when assigning projections to buffer cells. Consider projected sample footprints when the two image grids differ substantially, but recognize that spreading samples can bridge an actual boundary.
Start with a small radius justified by validation. A radius of 2 pixels may be a useful initial test setting; it is not a guaranteed accuracy specification. Expanding to 10 pixels can recover more candidates while also admitting neighboring objects.
If an object mask is available, use it as an additional candidate constraint. The guidance on selecting valid depth pixels with a segmentation mask helps distinguish object-supported samples from unrelated neighbors.
4. Check the arithmetic with a simple example
Consider a hypothetical rectified pair with parallel optical axes. Both horizontal focal lengths are 600 pixels, and both principal points are at (320, 240). The depth-to-color transform has no rotation and adds 50 mm along the horizontal axis.
Take a depth sample at (320, 240) with axial depth 1000 mm. Its depth-frame coordinates are (0, 0, 1000) mm. After transformation, the color-frame coordinates become (50, 0, 1000) mm.
Suppose the detector reports (350.4, 239.7). The projected distance is:
That sample passes a 2-pixel geometric gate. It still needs the visibility and surface-consistency checks described above.
Now place a hypothetical point on the same depth-camera ray at 2000 mm. Its horizontal color projection becomes 600 × 50 / 2000 + 320 = 335 pixels. The distance changed the correspondence by 15 pixels.
This is why deprojecting a color pixel at an arbitrary distance cannot generally solve the lookup. Translation between viewpoints makes the mapping depend on the actual scene distance.
The example uses ideal pinhole geometry, not measured performance for any catalog product. A field-of-view or camera-resolution calculator does not perform this correspondence calculation.
5. Verify the lookup before trusting feature depth
Use a stationary target with identifiable features on a continuous surface. Test the image center and corners, then repeat at several distances within the intended measurement range. Record projected error, rejected matches, and depth stability separately.
Inspect these failure patterns before increasing the search radius:
| Observed pattern | Likely check | Practical test |
|---|---|---|
| Offset changes with target distance | Arbitrary assumed depth or incorrect translation | Repeat with measured depth at several distances |
| Nearly constant horizontal or vertical shift | Crop origin, padding, or pixel convention | Overlay known image coordinates before projection |
| Error grows toward image corners | Distortion model or incorrect intrinsics | Compare raw and rectified processing paths |
| Stationary target works, moving target fails | Timing mismatch or motion during acquisition | Compare synchronized stationary and moving captures |
| Matches fail near object boundaries | Occlusion or inadequate sample coverage | Display source sample identities over the color view |
| Image center works but edges have no samples | Different sensor coverage | Inspect the projected depth coverage mask |
A missing match near the image boundary may be expected. Depth image field of view and stereo coverage explains why color coverage does not guarantee depth coverage.
If you use an aligned output, verify its target grid and read data from the processed output rather than the original acquisition. Follow checks for depth alignment to RGB before treating matching array coordinates as corresponding observations.
For each accepted match, log the projected pixel error and source depth coordinate. For rejected matches, retain a reason such as invalid measurement, outside coverage, excessive distance, or ambiguous surface. This makes failures diagnosable without disguising them as valid measurements.
Checks before deployment and when to choose another method
Full-frame projection is a straightforward reference implementation. For repeated feature queries within one acquisition, build the visibility buffer or spatial index once and reuse it. Do not rebuild the geometry independently for every feature unless the processing load is acceptable.
An already registered depth image may be simpler when the interface documents its coordinate grid, value semantics, and invalid-pixel behavior. A single-feature lookup remains useful when you need the original depth sample identity or explicit control over candidate rejection.
Neither approach can recover a measurement from a surface the depth sensor cannot observe. A feature on a thin edge, reflective patch, or occluded background may require a different viewpoint, another sensing method, or an explicit no-measurement outcome. Interpolating across a hole does not establish correspondence.
Before deployment, check:
- Coordinate system: detector preprocessing is reversed, and row/column order is verified.
- Geometry: calibration matches the active profiles, transform direction is correct, and units agree.
- Timing: color detections and depth samples describe the same scene state.
- Validity: invalid measurements and points behind the color camera are rejected.
- Visibility: competing surfaces are considered before selecting a nearby candidate.
- Acceptance: the radius is expressed in color pixels and validated against known features.
- Output: source depth, transformed Z, and point coordinates are labeled distinctly.
- Failure behavior: unsupported or ambiguous features return no match rather than an invented distance.
For hardware selection, begin with the camera configuration step. If correspondence requirements remain uncertain, send the stream profiles, calibration availability, target distances, and acceptable no-match rate through application support before choosing components.
Questions
Can I convert a color pixel directly into a depth pixel?
Not generally. A color pixel identifies a ray, and different distances along that ray can project to different depth pixels when the cameras have separated viewpoints. You need measured depth, a known surface constraint, or a documented registration process that uses depth measurements.
How large should the pixel search tolerance be?
Define the tolerance in the color image where the feature was detected, then validate it using known targets. The appropriate radius depends on calibration, sampling, preprocessing, and feature location. A larger radius increases candidate availability but also increases the chance of selecting an unrelated surface.
Should I return the original depth or the transformed Z coordinate?
Return the quantity your application explicitly requires. The original measurement belongs to the depth sensor's representation, while transformed Z belongs to the color camera coordinate frame. Preserve both when useful, and label the frame, units, and source pixel so downstream processing cannot confuse them.
Why can a visible color feature have no reliable depth match?
The feature may be visible only to the color camera, lie between valid depth samples, or sit beside a depth discontinuity. Invalid measurements and timing differences can also prevent association. Preserve a no-match outcome instead of filling the gap with a neighboring surface without justification.
Can I use an aligned depth image instead of projecting every sample?
Yes, if the output is registered onto the color image grid and its depth semantics are documented. Confirm that detection and registration use the same raw or rectified geometry. Registration can simplify indexing, but it does not eliminate holes, occlusion, or uncertainty at boundaries.