Sub-pixel depth sampling: bilinear interpolation or triangulation

Your corner detector reports the target at pixel (512.37, 402.61) on the left infrared image, and then the depth frame hands you a distance only at whole pixels — get_distance(int x, int y) — so the sub-pixel work looks lost. It is not. Two fixes exist: bilinear interpolation of the four depth pixels around the point, or re-triangulating the feature from matched left and right detections. This page gives the formula, a worked example at 1 m with the arithmetic shown, the guards for no-data pixels and depth edges, and the decision rule for when triangulation earns its extra code. The short version: interpolate the depth and deproject with the fractional pixel by default; triangulate only to beat the map's own depth noise on textured features.

What the integer depth lookup costs

Depth cameras answer what is the range at pixel (512, 402), never at (512.37, 402.61). Combine that integer lookup with a deprojection that also receives integers, and the detector's sub-pixel work is discarded before it ever reaches a millimeter number.

The example frame used throughout: 1280 × 720 px, fx = fy = 640 px (a 90° horizontal field of view on a 1280-px-wide sensor), baseline B = 55 mm, target at Z = 1000 mm. Pixel scale is Z/fx = 1000/640 = 1.56 mm per pixel. Rounding the keypoint to the nearest pixel moves it by up to 0.5 px per axis: 0.78 mm per axis, or √2 × 0.5 × 1.56 ≈ 1.1 mm radial in the worst corner case. A good corner or centroid detector locates to 0.1–0.2 px, which is 0.16–0.31 mm at this scale, so integer sampling is 3–5× coarser than the measurement you already made. What a detection error of that size does downstream is worked through in edge position accuracy from image sampling; for blob-type keypoints, the inverted grayscale centroid reaches the same sub-pixel class on infrared images.

The depth value itself suffers least on smooth surfaces: reading one pixel away shifts Z by about the map's own noise. The dangerous case is the depth edge, and it gets its own guards below.

Bilinear sampling at the fractional pixel

Read the four neighbors of the point — Z00 at (x0, y0), Z10 at (x0+1, y0), Z01 at (x0, y0+1), Z11 at (x0+1, y0+1) — and weight each by the fractional offsets. The weights are the fractional areas of the sampling cell, per the standard bilinear interpolation definition, and they sum to 1.

Deproject with the fractional (u, v) and the depth sensor's own intrinsics (fx, fy, cx, cy in pixels; on a rectified stereo pair fx = fy, and y grows down the image with v). That is the whole method: two lines of arithmetic, no re-matching.

Worked example: one keypoint at 1 m

The keypoint sits at u = 512.37, v = 402.61. The four surrounding depth pixels read:

Offsets dx = 0.37 and dy = 0.61 give weights (1−0.37)(1−0.61) = 0.2457, 0.37×0.39 = 0.1443, 0.63×0.61 = 0.3843 and 0.37×0.61 = 0.2257, which sum to 1.0000. The interpolated depth:

Z = 0.2457×1002 + 0.1443×1004 + 0.3843×1000 + 0.2257×1003 = 246.2 + 144.9 + 384.3 + 226.4 = 1001.7 mm

Deprojection with the fractional pixel: X = (512.37 − 640) × 1001.7/640 = −127.63 × 1.5652 = −199.8 mm, and Y = (402.61 − 360) × 1.5652 = 42.61 × 1.5652 = 66.7 mm.

The nearest-pixel path returns the (512, 403) values: X = (512 − 640) × 1000/640 = −128 × 1.5625 = −200.0 mm, Y = (403 − 360) × 1000/640 = 43 × 1.5625 = 67.2 mm, Z = 1000 mm. That path misses by 0.23 mm in X and 0.50 mm in Y — 0.55 mm radial here, 1.1 mm in the worst case. The two Z reads differ by 1.7 mm, inside the map's own noise, so on a smooth surface the depth location is a minor term: lateral position is where integer sampling hurts.

When re-triangulation is worth it

The alternative skips the map: detect the same physical feature in the right rectified infrared image and triangulate. On a rectified pair both detections share the row, so the disparity is d = u_L − u_R:

At the example settings the disparity is 640 × 55/1000 = 35.2 px, and one pixel of disparity error costs Z²/(fx·B) = 1000²/(640 × 55) = 28.4 mm of depth. An unrefined map, whose matching error runs 0.2–0.5 px, therefore carries 5.7–14.2 mm rms of depth noise at this distance. A patch refinement dedicated to your keypoint — search along the epipolar line, fit the cost curve — typically lands at 0.05–0.2 px, which is 1.4–5.7 mm. That 2–4× on Z is the entire payoff, and it exists only while both images lock the same physical micro-edge; on untextured plastic the map and your matcher fail together. Judge matchability after exposure: the depth camera gain and exposure guide for near range covers the infrared side.

On lateral position, triangulation changes little on-axis — X and Y come from u_L, the same detector — but off-axis the map's Z noise leaks into X at the rate (u − cx)/fx, which is 300/640 = 0.47 mm per millimeter of Z error at 300 px off-center. The Z² term also bites fast: at Z = 2000 mm the same 0.1 px refinement is ±11.4 mm, the regime where a wider baseline starts to pay, as the long-range stereo accuracy guide works through. The disparity-to-3D arithmetic behind that conversion is the same one the map runs internally; you are only concentrating it on one patch.

Decision rule: bilinear plus fractional deprojection for point positions; add re-triangulation when the task measures a height or a step at that feature and the surface is textured.

Guards: no-data pixels and depth edges

Two conditions corrupt an otherwise valid interpolation. The first is no-data: most depth SDKs mark missing pixels as 0 or negative, and a zero averaged into the weights puts the point at the lens. Return invalid instead; renormalizing over the valid three is acceptable for overlays but biased for gauging, because three pixels straddling an edge tilt the average.

The second is the depth edge. Take the spread of the four samples, max − min, and reject the sample when it exceeds 2% of Z — 20 mm at 1 m. The line has room on both sides: a machined surface tilted 45° moves Z by about 1.6 mm across the cell at 1.56 mm pixels, and map noise on a textured part sits in the few-millimeter band, while real object steps measure tens of millimeters or more.

If the camera publishes a confidence or validity mask, decode and apply it before sampling anything: the confidence bitflag walkthrough shows how the flags pack the failure modes.

Implement it and prove it

The whole method in a dozen lines, with both guards riding along:

# Illustrative pseudo-code; these names are ours, not any SDK's.
def sample_point(depth, u, v, fx, fy, cx, cy):
    x0 = floor(u); y0 = floor(v)        # ints
    dx = u - x0;   dy = v - y0          # floats in [0, 1)
    z00 = depth[y0][x0]                 # float, mm
    z10 = depth[y0][x0 + 1]
    z01 = depth[y0 + 1][x0]
    z11 = depth[y0 + 1][x0 + 1]
    if min(z00, z10, z01, z11) <= 0:
        return None                     # no-data pixel: never average a zero
    if max(z00, z10, z01, z11) - min(z00, z10, z01, z11) > 0.02 * z00:
        return None                     # depth-edge guard: 2% of Z
    z = (1 - dx) * (1 - dy) * z00 + dx * (1 - dy) * z10 + (1 - dx) * dy * z01 + dx * dy * z11
    x = (u - cx) * z / fx               # mm, fractional pixels in
    y = (v - cy) * z / fy               # y grows down the image
    return x, y, z

Two assertions prove the weights: fill all four neighbors with 1000.0 and call at any fractional (u, v) — the result must be exactly 1000.0, because the weights sum to 1. Call with u and v exactly on a pixel center and the returned depth equals that pixel's value, because dx and dy are zero. A pipeline-level check: run five keypoints through this function and through the SDK's integer lookup, and require every difference to stay inside the 0.78 mm per-axis bound from the worked example.

Sample on the depth's native grid

Detect on the left infrared stream rectified to the stereo pair, and sample depth that shares that geometry. A depth frame aligned to the color camera has already been resampled once — silhouettes shift and smooth by up to about a pixel of the target grid — so your bilinear pass would interpolate interpolated data. Align color to depth instead of depth to color, or request depth in the infrared geometry; the SDK alignment utilities exist to build overlays, and the per-point metric read stays your job. If you must work on an aligned frame anyway, widen the spread guard, because every silhouette arrives pre-mixed.

When the depth map is the wrong tool

At 1 m the example frame resolves laterally at 1.56 mm per pixel, and its depth floor is millimeter-class. A specification that asks for ±0.05 mm on a height is out of reach of any sampling method; change the sensor instead of the sampler. For profile gauging, a laser profile sensor such as the MV-DP4180-03P triangulates each laser profile directly on the sensor. For full-area depth at close range, a projected structured-light camera such as the MV-DPS200P-02 rebuilds the whole scene in one capture. Both still end in a 3D map you sample, so the interpolation code on this page carries over unchanged — but the depth floor at short range is a different class.

Weigh the two families against each other in the 3D sensors category, and against the area scan cameras that supply the infrared and color images your detector runs on; the camera finder filters both by resolution, interface and sensor format.

Before you ship: the checks

• Intrinsics come from the depth sensor's calibration, in pixels (fx, fy, cx, cy); on a rectified pair fx = fy.
• Scale check with a ruler: a 100 mm target at 1 m must span 100/1.56 = 64 px on the example frame. Off by more than a pixel means the wrong intrinsics.
• Detector proven: a synthetic corner shifted by a known 0.5 px is recovered within 0.1 px.
• No-data handling: assert all four samples are positive before weighting; a zero never enters an average.
• Spread guard tuned: log max − min over a known flat part; the 2% of Z line must sit above that spread and below the smallest real step the task must resolve.
• One geometry: detection stream, rectification and depth frame all native to the same grid.
• Cross-check: five keypoints through both the interpolated sampler and the SDK's integer lookup; differences inside the 0.78 mm per-axis bound.

If the numbers in your specification do not close — lateral scale too coarse, depth floor too high — the camera configurator walks sensor, lens and interface in order, and the 3D guides topic page collects the calibration and integration pieces around this one.