3D visionHow-to

Object 3D position from depth map and camera intrinsics

Compute an object's 3D position from a depth map and intrinsics: back-projection formulas, the Z-depth vs ray-range decision, outlier rejection, an example.

LumaSpec Vision11 min read Share on LinkedIn

On this page 12 sections
  1. 01What you can report, and what you cannot
  2. 02One grid: mask, depth map and intrinsics
  3. 03The depth source and the color side
  4. 04Where fx comes from
  5. 05Z-depth or ray range: the convention picks the formula
  6. 06Seven steps from mask to metric position
  7. 07Worked example: a case on the floor at 2.5 m
  8. 08No intrinsics? The depth-only fallback
  9. 09Checks before you trust the numbers
  10. 10When the usual answer is wrong
  11. 11Where to go next
  12. 12Questions

You have a detector drawing masks around objects and a depth camera producing a depth map of the same scene. The question is what you can honestly report from the two together: a metric position in camera coordinates, observed 3D bounds, or only a depth summary. This guide gives the decision, both back-projection formulas — optical-axis Z and ray range — the filtering that makes the numbers trustworthy, and a worked example at 2.5 m with the arithmetic shown. Metric XYZ needs a calibrated intrinsic matrix; without one, you get a strictly depth-only report, defined below. The steps work for any depth map plus instance masks, whatever produced the depth.

What you can report, and what you cannot

Commit to the rule up front. Intrinsics plus a known depth convention buy you the camera-frame position and the observed bounds; without intrinsics you report depth statistics and nothing lateral.

You have You can report You cannot report
Mask and depth map only median depth, p90 − p10 spread, valid fraction metric X or Y, any lateral extent
Intrinsics, Z-depth map position (X, Y, Z), observed bounds, origin distance hidden extents, physical centre
Intrinsics, range map the same, after ray normalization positions from the Z formulas

The position is the per-axis median of the back-projected points; the observed bounds are their per-axis minimum and maximum. Both describe the visible surface, never hidden faces or the physical centre of the body. For the smaller job of per-mask depth statistics alone, segmentation mask average depth: select valid object pixels covers the selection rules.

One grid: mask, depth map and intrinsics

Three arrays decide everything, and they must describe the same image. The depth map arrives in metres at the resolution its intrinsics were calibrated for; depth in millimetres or in vendor-scaled units is converted first, or every coordinate comes out wrong — aligned depth point cloud wrong scale: fix unit conversion shows that failure.

The mask must land on exactly that grid. If your detector letterboxes, crops or resizes, undo that geometry before indexing depth; the stream side of the problem is covered in aligned depth streams: data layout and which intrinsics apply. When depth has been aligned to a second camera, deproject with the intrinsics of the camera those pixels belong to — aligned depth point cloud intrinsics: which camera to deproject with settles which matrix that is.

The depth source and the color side

Any head that outputs a per-pixel depth map works with these steps. A ToF sensing camera is the simplest source — one housing, one depth map:

3D sensorsHikrobot MV-DT01WMU ToF sensing cameraMV-DT01WMU
Type
Depth / ToF camera
Price on requestView and quote

Close-range station work suits projected structured-light and laser galvanometer stereo heads instead:

3D sensorsHikrobot MV-DPS200P-02 Projected structured-light 3D cameraMV-DPS200P-02
Type
Stereo / structured light
$8,829.00 · quotedView and quote
3D sensorsHikrobot MV-DLS300P-04 Laser galvanometer stereo cameraMV-DLS300P-04
Type
Stereo / structured light
$16,579.00 · quotedView and quote

Detection can also run on a separate color camera such as the MV-CS200-10GC, a 20 MP GigE model, aligned to the depth frame before anything else:

Area scan camerasHikrobot MV-CS200-10GC 20 MP Area Scan Camera, GigE, IMX183, Color, BasicMV-CS200-10GC
Sensor format
1"
Lens mount
C-mount
Interface
GigE
Resolution
16-25 MP
$739.00View partBuild a system with it

Where fx comes from

fx, fy, cx and cy come from a board calibration. Ahead of it you can estimate the pixel focal length from the lens and the sensor, and the estimate starts at the lens — the LS-FFL-0567 is a 16 mm C-mount lens for 1.1" 20 MP sensors.

Machine vision lenses16 mm, 1.1", 20 MP, C-mount machine vision lens, F2.8-22, <0.3% distortion (LS-FFL-0567)LS-FFL-0567
Type
Machine vision fixed
Sensor format
1.1"
Mount
C-mount
Resolution
16-25 MP
$679.00View partBuild a system with it
fx [px] = f [mm] ÷ pixel pitch [mm]

On a sensor with 3.45 µm pixels, fx = 16 mm ÷ 0.00345 mm = 4638 px. Treat that as a starting value only: a calibration target hands you cx, cy, both focal lengths and the distortion model in one pass, and it overrides the estimate. The same arithmetic holds across machine vision lenses of any focal length.

Z-depth or ray range: the convention picks the formula

Depth is one number per pixel under two conventions. Optical-axis depth Z is measured along the camera's principal axis; range r is measured along the ray through the pixel. The pinhole camera model is written for the first, and the familiar back-projection assumes it:

X = (u − cx) · Z ÷ fx Y = (v − cy) · Z ÷ fy Z = D(u, v)

When the map stores range, normalize the ray first:

kx = (u − cx) ÷ fx ky = (v − cy) ÷ fy s = √(1 + kx² + ky²) Z = r ÷ s, X = kx · Z, Y = ky · Z

The size of the mistake shows why this deserves its own section. On a camera calibrated fx = fy = 800 px, cx = 640 px, cy = 360 px, take pixel (840, 300): kx = 0.25, ky = −0.075, s = √(1 + 0.0625 + 0.0056) = 1.0335. A range of 2.50 m at that pixel is a Z of 2.50 ÷ 1.0335 = 2.42 m, so feeding range into the Z equations overstates Z by 0.08 m and X by 0.02 m. At the corner of the same image the ratio reaches 1.36, and with a wide lens the convention becomes the dominant error, ahead of sensor noise.

One test settles it in a minute: aim the camera square at a flat wall. A Z map reads nearly constant across the wall; a range map reads lowest at the centre and climbs toward the corners. Whichever behaviour you see is the convention your pipeline assumes from then on.

Seven steps from mask to metric position

  1. Put mask, depth and intrinsics on one grid. The mask and depth arrays must share height and width; refuse the frame when they do not rather than resample silently.
  2. Collect the depth inside the mask and drop non-finite values and anything at or below zero. Valid fraction = valid depth pixels ÷ all pixels inside the mask. An empty result is reported as unavailable, never deleted.
  3. Reject outliers with a median absolute deviation cut and apply the same index filter to the pixel coordinates. Contamination survives even then: flying pixels between object and background in stereo depth edges shows where those come from, and a confidence map tightens the gate — depth low confidence values: mask invalid pixels with the IR image.
  4. Back-project every retained pixel with the formula your convention picked above.
  5. Position = the per-axis median of X, Y and Z over the retained points.
  6. Observed bounds = per-axis minimum and maximum; extents = max − min per axis.
  7. Return one record per mask: class, detector confidence, position in metres, distance from the camera origin, bounds, retained count, valid fraction and a geometry-validity flag. Attach it to a tracker ID if you run one — the position travels with the track and claims nothing more.
MAD = median(|d − median(d)|) keep |d − median(d)| ≤ 3 × 1.4826 × MAD

The cut equals three standard deviations for Gaussian noise, so honest surface depth survives while silhouette mixtures go.

# Illustrative pseudocode: names are ours, not any SDK's API.
import numpy as np

fx: float = 800.0        # pixels, from your calibration
fy: float = 800.0
cx: float = 640.0
cy: float = 360.0
# depth: H x W array in metres; mask: H x W boolean array
assert depth.shape == mask.shape

v, u = np.nonzero(mask)          # v = pixel rows, u = pixel columns
z = depth[mask]
valid = np.isfinite(z) & (z > 0.0)
z, u, v = z[valid], u[valid], v[valid]

med = float(np.median(z))
mad = float(np.median(np.abs(z - med)))
keep = np.abs(z - med) <= 3.0 * 1.4826 * mad
z, u, v = z[keep], u[keep], v[keep]

x = (u - cx) * z / fx            # optical-axis Z convention
y = (v - cy) * z / fy
position = np.array([np.median(x), np.median(y), np.median(z)])
lo = np.array([x.min(), y.min(), z.min()])   # observed bounds
hi = np.array([x.max(), y.max(), z.max()])

# Test on one retained pixel: u = 840, v = 300, z = 2.50 m
assert (840 - cx) * 2.50 / fx == 0.625      # X, metres
assert (300 - cy) * 2.50 / fy == -0.1875    # Y, metres

Worked example: a case on the floor at 2.5 m

A 1280 × 720 depth map, calibrated fx = fy = 800 px, cx = 640 px, cy = 360 px. The mask covers a case on the floor and holds 5,000 pixels. Step 2 keeps 4,860 of them, a valid fraction of 4,860 ÷ 5,000 = 0.97. Step 3 measures a median of 2.50 m and a MAD of 0.035 m, so the cut is 3 × 1.4826 × 0.035 = 0.156 m and the accepted band runs from 2.34 m to 2.66 m; 4,812 pixels survive.

The per-axis medians land at pixel (840, 300) with Z = 2.50 m:

  • X = (840 − 640) × 2.50 ÷ 800 = 200 × 2.50 ÷ 800 = 0.625 m
  • Y = (300 − 360) × 2.50 ÷ 800 = −60 × 2.50 ÷ 800 = −0.1875 m
  • Z = 2.50 m
  • distance from the camera origin = √(0.625² + 0.1875² + 2.50²) = √6.676 = 2.58 m

The observed bounds come from the retained span — columns 800 to 880, rows 260 to 340, depth 2.35 m to 2.65 m:

  • X from (800 − 640) × 2.35 ÷ 800 = 0.47 m to (880 − 640) × 2.65 ÷ 800 = 0.80 m
  • Y from (260 − 360) × 2.35 ÷ 800 = −0.29 m to (340 − 360) × 2.65 ÷ 800 = −0.07 m
  • Z from 2.35 m to 2.65 m
  • extents 0.33 m × 0.23 m × 0.30 m — a visible surface about 330 mm wide

One pixel of mask error at this distance moves a lateral bound by Z ÷ fx = 2.50 ÷ 800 = 3.1 mm. Quoting bounds finer than that is quoting the mask, not the part.

No intrinsics? The depth-only fallback

No intrinsic matrix means no metric coordinates — that is the honest output, and it is still useful. Per mask, report the median depth, the spread as p90 − p10 over the valid pixels, and the valid fraction from step 2. The example mask gives median 2.50 m, p10 2.42 m, p90 2.60 m, spread 0.18 m, valid fraction 0.97. Keep one row per mask, including an explicit unavailable entry when nothing valid survives. The spread is a variation summary, not a calibrated uncertainty, and no depth statistic yields a lateral coordinate.

Checks before you trust the numbers

  • Shape: mask, depth and intrinsics must refer to the same image size. This single assertion catches every letterbox and resize mistake.
  • Convention: run the flat-wall test before the first object crosses the camera. A range map through the Z formulas is the largest single error on this list.
  • Round trip: project the reported position back to pixels, u′ = fx · X ÷ Z + cx and v′ = fy · Y ÷ Z + cy, and confirm it lands inside the mask. Here 800 × 0.625 ÷ 2.50 + 640 = 840 and 800 × (−0.1875) ÷ 2.50 + 360 = 300. Project a 3D point to pixel coordinates with a calibrated camera gives the full procedure.
  • Known part: measure a box of known width against the observed X extent and log the error against the 3.1 mm/px floor from the example.
  • Stillness: log a fixed object's position over a few hundred frames; the span is your repeatability. Depth jitter, not geometry, usually widens it — the stereo camera depth jitter and IMU drift guide diagnoses that side.
  • Calibration: re-run the board after any refocus or lens swap; for metric ground truth, depth map to real distance calibration for XYZ point clouds is the reference workflow.

When the usual answer is wrong

  • Range fed to the Z formulas. The overstatement is about 3 % of Z at the example pixel and grows to its maximum at the image corner; wide-lens cameras suffer most.
  • Masks in the wrong coordinates. A mask computed on a letterboxed, resized frame and indexed against the full-resolution depth map lands on neighbouring surfaces; positions shift by the padding offset and extents stretch.
  • Bounding boxes instead of masks. A box around a small object is mostly background, so the depth median lands behind the object. Use masks; with boxes only, shrink to the central region and expect contamination.
  • Treating the visible surface as the object. The position and bounds track what the camera sees. Occlusion or rotation moves the estimate of a stationary object, and a moving camera needs known poses before positions compare across frames.
  • Publishing the spread as accuracy. p90 − p10 says how the depth varies; it certifies nothing.

Where to go next

Pick the depth source in 3D sensors, keep the color side in area scan cameras, and take the whole scene through the camera step of the configurator. More depth workflows sit on the 3D topic page, and the contact page puts scene-level questions in front of our applications team.

Cameras3D sensors

Laser profilers and stereo 3D cameras

170 models · from $699.00

Questions

Can I compute an object's metric position without camera intrinsics?

No. Without fx, fy, cx and cy a pixel offset has no metric scale: the same pixels and depth values produce different lateral coordinates at different focal lengths, so any X or Y you publish would be arbitrary. You can still report the mask's median depth, its p90 − p10 spread and the valid fraction. Those are depth summaries; lateral coordinates and extents need a camera model, supplied by calibration or by the vendor.

How do I tell whether my depth map stores Z-depth or ray range?

Point the camera square at a flat wall. A Z-depth map reads almost constant across the wall; a range map reads lowest at the centre and rises toward the corners — on a 1280 × 720 map with fx = fy = 800 px, the corner range is about 1.36 × the corner Z. Then commit to the matching formula and record which convention the pipeline assumes.

Why take the median instead of the mean of the back-projected points?

Because the median tolerates the contamination your filters miss: background bleeding through mask edges, flying pixels at depth discontinuities, holes from specular surfaces. A mean shifts with every surviving outlier; a per-axis median moves only when half the retained points move. Publish the retained count beside the median, so a position built on a dozen pixels is visible as such.

The detector resizes or letterboxes images. How do I keep the mask aligned with the depth map?

Undo the resize geometry and restore masks to the depth map's coordinates before indexing, then check that the mask and depth arrays share the same shape. If your detector can keep masks in original-image coordinates, enable that. When the shapes disagree, reject the frame instead of silently resampling one side — a silent resample moves every back-projected coordinate.

Is the estimated position the object's physical centre?

No. Every retained pixel is a visible surface point, so the position and bounds describe what the camera sees, not the full body: hidden faces, the physical centre and true orientation remain unknown. A stationary object can shift its estimate when visibility changes, so treat these as visible-surface measurements and compare them under a fixed camera.