Depth map to real distance calibration for XYZ point clouds

Your depth camera hands you a 16-bit image full of numbers such as 10,247, and the robot needs millimetres. The gap between the two is smaller than it looks: one scale factor turns raw values into Z, and four intrinsics numbers turn Z plus pixel position into X and Y. This guide is for engineers wiring a depth image into a measurement, a robot pick or a volume check. You will calibrate the depth scale against targets at two known distances, take the intrinsics that belong to your exact resolution, convert every pixel to X, Y and Z, and prove the result on a gauge block before it touches production data.

What you need

A short bench list; nothing here needs special equipment.

• The raw depth image exactly as your code reads it: bit depth, value range, and whether the SDK claims units or hands you processor units.
• The intrinsics for your camera at its current resolution: fx, fy, cx and cy, all in pixels, from the vendor file or your own calibration.
• A flat target of known size, such as a checkerboard plate or a machined blank.
• Two ways to hold that target at known distances: machined standoffs, a gauge block, or a rail with a scale.
• Your conversion code, which is a few lines of array arithmetic once the numbers exist.

The two numbers: depth scale and intrinsics

A depth image stores one value per pixel: depth along the optical axis, in units the camera's processor chose — commonly 1 mm, sometimes 0.1 mm, sometimes something else entirely. The first conversion multiplies by a scale factor s and adds an offset z0, both measured against targets rather than assumed. The second spreads Z across the image: each pixel's column u and row v, shifted by the principal point cx and cy and scaled by the focal length in pixels fx and fy, become X and Y.

The last line estimates fx from nominal focal length and optical format, for the first hour of a project before any calibration exists. Treat it as good to a few percent: focusing shifts the effective focal length and format dimensions are rounded. The calibrated fx from the vendor file or a checkerboard session is the number to run in production. Count u and v from zero with row 0 at the top. The pinhole geometry behind all four lines:

If you are still choosing the lens that sets your millimetres per pixel, check the focal length against your field of view first:

Calibrate and convert, step by step

1. Record the raw format. Bit depth, maximum value, origin corner and the SDK's claimed unit. You are about to replace that claim with a measurement, so write the claim down.
2. Take intrinsics at your exact resolution. fx, fy, cx and cy move when binning or cropping changes the pixel grid, so pull them at the resolution you will run; rescaling intrinsics after binning shows the arithmetic when you must derive them.
3. Calibrate the scale with two distances. Put the flat target at a near distance d1 and a far distance d2 inside the working range, read the mean raw values r1 and r2 from a small central patch, then s = (d2 − d1) / (r2 − r1) and z0 = d1 − s × r1. Keep the target large enough for its footprint at d2; undersized targets at range fail on their own (checkerboard target size at distance).
4. Convert every pixel to Z. Mask pixels with raw 0 or the vendor's invalid flag, then multiply by s and add z0. Unmasked zeros pile up as false points at the origin.
5. Deproject to X and Y. Build a grid of u and v, subtract cx and cy, multiply by Z, divide by fx and fy. The deprojection calculation explained expands that line by line with code.
6. Assemble the cloud. Stack X, Y and Z as three same-size single-channel images or one N × 3 array; most 3D viewers accept either. Keep millimetres from here to the robot.
7. Store the calibration with the camera settings. s, z0, fx, fy, cx, cy, resolution and firmware version in one record beside the capture code, because any of them changing silently invalidates every cloud after it.

Worked example: 8 mm lens on a 2/3-inch sensor, targets at 500 and 1,500 mm

Inputs. A 2/3-inch sensor, 8.8 mm wide and 6.6 mm high, with an 8 mm nominal lens produces a raw 16-bit depth image of 1280 × 960 px. A flat plate held at 500.0 mm reads raw 5,000; at 1,500.0 mm it reads raw 15,000. The pixel of interest sits at u = 900, v = 700 with raw value 10,000. These are measured and estimated numbers, not data-sheet promises.

Scale. s = (1500.0 − 500.0) / (15000 − 5000) = 1000.0 / 10000 = 0.1 mm per raw unit. The offset is z0 = 500.0 − 0.1 × 5000 = 0.0 mm, so raw values are proportional to distance across this pair.

Depth. Z = 10,000 × 0.1 + 0.0 = 1,000 mm along the optical axis.

Intrinsics. fx = 8 × 1280 / 8.8 = 1163.6 ≈ 1,164 px. On the vertical, fy = 8 × 960 / 6.6 = 1163.6 px, which matches, as it should when pixels are square. For a centred optical axis, cx = (1280 − 1) / 2 = 639.5 px and cy = (960 − 1) / 2 = 479.5 px.

Lateral position. Z / fx = 1000 / 1164 = 0.859 mm per pixel. X = (900 − 639.5) × 0.859 = 260.5 × 0.859 = 224 mm right of the axis. Y = (700 − 479.5) × 0.859 = 220.5 × 0.859 = 189 mm below the centre row, below because rows count downward. The point sits 1,000 mm ahead, 224 mm right, 189 mm down.

Range check. One 16-bit frame at s = 0.1 mm per unit spans 65,535 × 0.1 = 6,553.5 mm, about 6.5 m, so the whole 500 to 1,500 mm band fits with the top of the range to spare.

Decision. At 1,000 mm the lateral scale is 0.859 mm per pixel, so a 100.0 mm gauge block spans 100.0 / 0.859 ≈ 116 px and its measured width carries ±1 px per edge, about ±1.7 mm worst case. These optics deliver ±1.7 mm on a 100 mm width at 1 m. For a ±0.5 mm specification at the same distance the pixel must be 0.5 mm or better: fx ≥ 1000 / 0.5 = 2,000 px, which at this sensor width takes a focal length of 2000 × 8.8 / 1280 = 13.75, about 14 mm, or a sensor with more pixels. Keep the 8 mm lens for the ±1.7 mm job; change the optics for the ±0.5 mm job.

Prove the cloud before production

• Plane fit. Fit a plane to the flat target's cloud at mid-range. The residual should sit at the sensor's stated Z noise for that distance; tilt or waviness points at calibration, not at the part.
• Known width. The 100.0 mm block must measure 100 mm within about ±1.7 mm for the worked example's optics: two edges, ±1 px each, at 0.859 mm per pixel.
• Third distance. Predict the raw value at 1,000 mm from the two-point line, then measure it. A two-point fit cannot expose non-linearity; a third point can.
• Centre versus corner. Measure the block once near the centre and once near a corner. A difference of several millimetres is lens distortion, so undistort coordinates before deprojecting.

Mistakes that move every point

• Z is not the distance to the camera. Most depth images store depth along the optical axis; the true range to an off-axis pixel is Z / cos θ, which at θ = 20° is 6.4 % more. Know which value your SDK writes before you send coordinates to a robot — the depth value versus true distance conversion shows the difference.
• Cropping without shifting the principal point. Cut away w columns and h rows and cx, cy move by w and h. A stale cx shifts every X by Z / fx millimetres per cropped pixel, and a principal point that lands outside a heavily cropped frame deserves a second look (is the calibration still fine?).
• Unit drift after updates. A firmware update or a changed depth-unit setting can redefine what one raw unit means, and the image looks identical. The fixture gauge block catches it in seconds; scale-versus-format disagreement is also how clouds end up truncated at close range (depth scale mismatch).
• Nominal numbers past day one. The estimate above is for the first hour of a project. Swap in calibrated fx, fy, cx and cy before anyone acts on a measurement.

The laser-plane case: one scale is not enough

A line laser projects a light plane and the camera images the stripe, so depth comes from the geometry between the image and that plane rather than from one scale factor. The extrinsics [R, T] of the laser plane in camera coordinates, multiplied by the intrinsics, form the homography that maps each stripe pixel to a metric point on the plane. Calibrate it by imaging the stripe across three or more points of known position on the plane — a target slid along a rail works — and solving for the plane pose, then compose. When the cloud reads right distance-wise but comes out tilted, the plane pose is the suspect: why a laser triangulation point cloud is tilted walks that diagnosis.

If you would rather buy that plane calibration than solve it, a factory-calibrated laser profiler hands over the finished metric profile instead of a raw depth image:

When the sensor hands you the cloud

Every step above is also a place to inherit a unit mistake, and factory calibration closes most of them. A laser profiler triangulates each profile against its factory geometry and outputs metric points; a projected structured-light camera returns a full-field cloud from calibrated stereo; a time-of-flight camera meters depth per pixel on the sensor. The 3D sensors category spans all three types, and the 3D camera finder filters them by type and series.

Even with a calibrated sensor, keep the gauge block in the fixture: the two-minute check that catches a wrong scale factor on a hand-built converter catches a bumped mount just as fast.

To set the distance that fixes your millimetres per pixel in the first place, the 3D camera working distance and positioning guide covers the mounting geometry, and the 3D topic page collects the rest of the depth-to-cloud workflow. When you are ready to specify hardware, start the configurator at the camera step: