Depth map to point cloud: the deprojection calculation explained

You call the SDK's point-cloud function, hand it a depth map, and get back an array of X, Y, Z vertices. The code you can read stops at that function call; the calculation runs inside the library. This guide opens it up: the pinhole formula every SDK implements, what fx, fy, px and py mean and why they are in pixels, a worked example you can reproduce on a calculator, and a test that proves your own conversion matches the SDK's vertex for vertex. The math is identical for time-of-flight, structured-light and stereo depth cameras, so it applies to any area depth sensor in the 3D sensors category.

What the SDK hands you

A depth map is a width × height array of integers. Each element is range along the optical axis, stored in fixed units — commonly 1 mm or 0.1 mm per count — and the stream metadata states the scale. A point cloud is an array of 32-bit float triples in metres.

The SDK produces it with one loop over the depth pixels and hands it back through a getter. In one common SDK this is get_vertices() on the points frame, which returns a pointer to the vertex array; the arithmetic never appears in your code because the library has already done it. The frame's documentation names the axes and units — read that comment once, because the sign of Y is the convention that differs most between SDKs. Which frame the vertices live in matters the moment a robot consumes them: transform depth point clouds between the optical frame and the camera frame covers that step.

The deprojection formula

Projection maps 3D points onto pixels; deprojection runs the same geometry backwards. For a camera described by the pinhole camera model, a point (X, Y, Z) lands at the pixel:

u = fx·X/Z + px, v = fy·Y/Z + py

Two equations, three unknowns — which is why a single 2D image cannot give you depth. A depth map supplies the missing equation, because Z is known for every pixel. Solve each line for the lateral coordinate and the whole conversion falls out:

Two practical notes. Depth sensors almost always have square pixels, so fx and fy come out equal and one focal number serves both lines. The principal point sits near the image centre but rarely exactly at (W/2, H/2); a 1 px error in px shifts every point sideways by Z/fx, which is 0.78 mm at 0.5 m with fx = 639. Use the calibrated value, not the centre.

Focal length in pixels, not millimetres

fx is the physical focal length divided by the pixel pitch:

A 2.2 mm lens over 3.0 µm pixels gives fx = 2.2 / 0.003 = 733 px. Bin the sensor 2×2 and the effective pitch doubles, so the same optics give fx = 2.2 / 0.006 = 367 px on the 640×480 stream. This is why intrinsics belong to a stream profile, not to the camera as a whole.

Worked example: deprojecting one pixel from a 1280×720 depth stream

Inputs. The stream is 1280×720. Its depth calibration reports fx = fy = 639.0 px, px = 319.5 px, py = 239.5 px — example values for this class of sensor, consistent with a horizontal field of view of 2·atan(640/639) ≈ 90°. The pixel of interest is u = 800, v = 400, right of and below the centre. The raw count is 4500 at 0.1 mm per count, so Z = 4500 × 0.1 mm = 450 mm = 0.450 m.

Rule. Apply X = (u - px)·Z/fx and Y = (v - py)·Z/fy; Z passes through unchanged.

Arithmetic.

X = (800 - 319.5) × 0.450 / 639.0 = 480.5 × 0.450 / 639.0 = 216.225 / 639.0 ≈ 0.338 m

Y = (400 - 239.5) × 0.450 / 639.0 = 160.5 × 0.450 / 639.0 = 72.225 / 639.0 ≈ 0.113 m

Z = 0.450 m

Result. The surface point sits 0.338 m right of the optical axis, 0.113 m below it and 0.450 m in front of the camera. The signs assume image rows grow downward and the camera frame follows the sensor; check the SDK's axis comment once per camera.

What moves the answer. One pixel of lateral spacing at this depth is Z/fx = 0.450 / 639.0 ≈ 0.70 mm. Depth error leaks sideways at (u - px)/fx = 480.5 / 639.0 ≈ 0.75, so 10 mm of depth error moves X by about 7.5 mm. Back the camera off to 1.5 m and the same pixel spans 1500 / 639.0 ≈ 2.35 mm.

Decision. Against a ±1 mm lateral tolerance, the cloud at 0.45 m meets the budget with single pixels; at 1.5 m one pixel already spans 2.35 mm, so geometry alone breaks the budget and the fix is a shorter working distance, not a better formula. Whether that pixel spacing can carry a profile tolerance is a sampling question — the guide on point cloud density for profile tolerances works it through against ASME practice.

Where the calculation hides in the SDK

get_vertices() does not compute anything in your thread. The library has already run the loop — read each count, multiply by the depth scale, apply the two formula lines with the depth stream's intrinsics, store three floats — and the getter hands you the pointer. Texture-mapped clouds add a second projection on top: each vertex is pushed into the colour camera with the extrinsics, then mapped with the colour intrinsics. Camera intrinsic parameter conversion for depth and texture images walks that exchange.

You can prove the formula is what the SDK does with a five-line test. Take the worked-example pixel: in the row-major vertex array its index is v·W + u = 400 × 1280 + 800 = 512,800. Compare your hand-computed (0.338, 0.113, 0.450) m with vertex 512,800 from get_vertices(). Agreement to single-precision rounding confirms the model; disagreement means wrong intrinsics or the wrong depth scale, not exotic math.

Read the intrinsics from the SDK for the exact stream profile you opened — they arrive with width, height, fx, fy, px, py and the distortion model. Most SDKs also print them with no code at all: an enumerate-devices tool run with a calibration flag lists the values for every stream the device offers.

Limits: what the formula does not fix

Zero depth comes first. Pixels with no return carry Z = 0, and both formula lines collapse them onto the origin — the whole invalid set stacks into one phantom point at (0, 0, 0) in the middle of your scene. Filter zero or no-return counts before any measurement touches the cloud; depth pixel deprojection zero diagnoses the symptom and missing points in a 3D camera point cloud covers the wider gaps.

Distortion comes second. The formula assumes rectified pixels, so check the distortion field in the intrinsics: when it reports none, the plain formula is exact for that stream; when it reports a model, remap (u, v) through the coefficients first.

Scale comes third. Deprojecting raw counts without the 0.1 mm-per-count factor stretches the entire cloud by 10; depth scale mismatch is the reference case.

Family is the fourth. A laser profiler needs none of this per pixel — its on-sensor triangulation already outputs calibrated profile points. Multi-camera rigs combine clouds with extrinsics instead: calibrate two depth cameras for combined 3D point clouds.

Write it yourself or call the SDK

Per pixel the cost is two subtractions, two multiplies and two divides. A full 1280×720 frame is 1280 × 720 = 921,600 pixels, so the win comes from vectorising: the whole frame deprojects in a few tens of milliseconds on a desktop CPU, and the traps there are stride and dtype — numpy depth image to point cloud is the fix-it guide.

Call the SDK when you want the full cloud every frame; it is optimised and on some platforms GPU-backed. Write the formula yourself in three cases. A handful of pixels — one readout that becomes a robot target, the case in center pixel depth measurement. A cropped region — deprojecting an ROI after cropping the XY region of interest saves most of the work. And any cross-check of vendor output against your own math.

Cameras and the computer that runs the conversion

Every area depth camera feeds this pipeline; the families differ only in where Z comes from. The MV-DT01WMU is a ToF sensing camera. The MV-DPS200P-02 is a projected structured-light 3D camera. The MIS-DP3850-01P is a 3D laser profile sensor that scans one line at a time.

The deprojection loop is per-pixel work over roughly a million pixels at frame rate, so it belongs on dedicated compute rather than a shared desktop. A vision controller pairs camera ports with CPU, memory and built-in light channels in one unit; the MV-VC5525-256G70-NN carries 32 GB of memory, PoE camera ports and 8 built-in light channels.

Next step: shortlist the depth sensor in the camera finder or start the configurator at the camera step. The Learn hub collects the rest of the depth-cloud guides, and the contact page reaches the applications team when the geometry question is yours alone.