Aligned depth point cloud intrinsics: which camera to deproject with

You aligned the depth stream to colour, built the point cloud, and the geometry is wrong: the cloud looks stretched, measurements disagree with the colour image, and a pixel lookup lands on the wrong feature. The fault is usually one line deep — the deprojection used the depth camera's intrinsics on a frame that now lives on the colour camera's pixel grid. This guide is for anyone wiring a depth camera into a measuring, sorting or pick-and-place cell that also uses colour. You will get the rule and the reason behind it, the size of the error worked out in metres, the array-shape trap that breaks pixel lookups, four checks that catch the bug before it ships, and the cases where texture-mapping colour onto the native depth cloud is the better build.

Why is the point cloud stretched after aligning depth to colour?

The cloud was deprojected with the depth camera's intrinsics. Once the depth frame is aligned to colour, every depth sample sits at a colour-image pixel position, so only the colour camera's fx, fy, cx and cy describe where it is; depth values stretch and shear the whole cloud.

Alignment does not change what the depth sensor measured. It reprojects each depth sample through the depth-to-colour extrinsics onto the colour camera's pixel grid, so pixel (u, v) in the aligned frame means the depth along the ray that colour pixel looks down. A grid defined by the colour camera can only be turned into metres with the colour camera's calibration.

Deprojecting with the depth constants instead applies a focal length that can differ by a factor of about two and a principal point hundreds of pixels away, so the cloud reads zoomed and slanted, worst at the image edges. Not every distortion is this bug: ripples that follow the surface come from smoothing filters, and depth filter camera intrinsics: diagnose wavy point clouds covers that separate fault.

Which intrinsics deproject an aligned depth frame?

Use the colour camera's intrinsics: fx and fy from the colour calibration, cx and cy from the colour principal point, and one vertex per colour pixel in row-major order. The aligned frame lives on the colour camera's grid, so the colour profile is the only calibration that describes it.

Here fx_c and fy_c are the colour focal lengths in pixels, cx_c and cy_c the colour principal point, (u, v) the colour-image pixel, and Z the aligned depth value there, converted to metres. Read the four values from the colour stream profile in your SDK after requesting the aligned frame — the aligned depth frame's own profile should carry them. If it still reports the depth values after alignment, you have met a wrapper bug of exactly this class; the fix arrives as an SDK update, so update the SDK before writing a workaround.

The points come out in the colour camera's optical frame. If downstream code expects the robot or table frame, transform them deliberately rather than assuming shared coordinates: transform depth point clouds between optical frame and camera frame covers the two frames. For how the pixel buffers themselves are laid out once streams are aligned, see aligned depth streams: data layout and which intrinsics apply.

How far off is a vertex deprojected with the depth intrinsics?

At working range the error is metres, not millimetres: in the worked example below, the same colour pixel deprojects to x = 0.522 m with the colour intrinsics and to x = 1.860 m with the depth ones — a 1.338 m mistake from two wrong constants.

Assume a rig streaming depth at 1280 × 720 and colour at 1920 × 1080, with factory values of the order these resolutions report: depth fx_d = 645.0 px and cx_d = 640.0 px; colour fx_c = 1380.0 px and cx_c = 960.0 px. A weld bead sits 480 pixels right of the colour principal point, at colour pixel (1440, 540), and the aligned depth there reads Z = 1.500 m.

The 1.338 m error is not a subtle offset. The focal-length ratio 1380.0 / 645.0 ≈ 2.14 stretches every lateral coordinate, and the 320 px principal-point difference shears it; on the y axis the same two mechanisms apply, with cy off by 180 px (540 versus 360). When the two streams differ in resolution, expect metre-scale lateral errors at working range, not pixel-scale ones. Read your own four values from the SDK and run this two-line check on any rig you inherit.

Why does the vertex array not match the colour image?

The array is sized to the frame the cloud was built from, not to your colour image: a cloud from a 1280 × 720 depth frame has 921,600 vertices, so colour pixel (u, v) in a 1920 × 1080 image indexes vertex v × 1280 + u — the wrong row.

A cloud built from the wrong frame fails twice at once. The geometry is deprojected on the depth grid, and the array holds the depth resolution's vertex count while the colour image holds its own pixel count, so every lookup past the first rows lands on the wrong part. Build the cloud from the aligned frame instead; then index = v × W_c + u, with W_c the colour width, puts exactly one vertex behind every colour pixel.

Guard the borders too. Alignment cannot invent depth where the colour ray found none, so an aligned frame carries invalid pixels along its border and at occlusion edges — test the zero or invalid flag before trusting a vertex. The same pixel-to-vertex indexing underlies feature lookups; bounding box depth: mapping image pixels to 3D points works it through for detection boxes.

How do you check which intrinsics the pipeline actually used?

Three checks settle it: the vertex count against colour width × height, a reprojection overlay drawn on the colour snapshot, and one known target read back within a few millimetres. Log fx, fy, cx and cy at start-up so a silent profile switch cannot hide.

1. Count the vertices. A cloud from the aligned frame holds exactly colour width × colour height vertices; a count matching the depth resolution means the pipeline built it on the depth grid.
2. Overlay the cloud. Reproject every vertex with u = fx_c·x/Z + cx_c and v = fy_c·y/Z + cy_c, then draw the dots on a colour snapshot. They must sit on their features at the corners as well as the centre.
3. Read back a known target. Place a target 300 mm right of the principal point at 1.0 m range; the vertex under its colour pixel should read x ≈ +0.300 m within a few millimetres.
4. Log the calibration. Print fx, fy, cx, cy and both resolutions at every start-up. A resolution or binning change rescales them; camera intrinsics after binning: scale them correctly gives the rule.

If the overlay lands correctly but the surface still wobbles frame to frame, the intrinsics are right and the fault is depth stability. That has its own playbook: this stereo depth jitter and IMU drift troubleshooting guide covers the symptom and its usual causes.

Can a few colour pixels be deprojected without building the full cloud?

Yes. Take Z from the aligned depth frame at each pixel of interest and run the pixel-to-point formula with the colour intrinsics — a 10 × 10 patch is 100 evaluations, not two million — and no vertex array at all.

This is the fastest fix for a measurement cell: no cloud object, no array-length assertion, just the formula. Two details matter. Check the depth unit — many SDKs deliver depth scaled in millimetres, so divide by 1000 before the formula, or the result lands a thousand times too far out. And skip pixels flagged invalid, where Z reads zero.

Two live guides take this further: colour pixel to depth lookup for detected image features handles lookups under detected features, and numpy depth image to point cloud: fix strides and dtypes shows the array arithmetic when you do want the whole cloud in your own code.

When should colour be texture-mapped onto the native depth cloud instead?

Texture-map when geometry accuracy leads: the native depth cloud keeps the depth sensor's own grid and edge positions, while alignment resamples every pixel and smears edges by one to two pixels. Keep the aligned, colour-deprojected cloud when your code indexes vertices by colour pixel.

Alignment moves every depth value to a new pixel centre, so a thin feature — a pin, a fillet radius, a gap — loses a pixel of edge on each side, and occlusion bands grow. That resampling error is measurement error you do not need. The native depth cloud deprojected with the depth intrinsics keeps the sensor's own geometry, and colour rides along as per-vertex texture coordinates computed from the extrinsics and the colour intrinsics.

Texture coordinates have their own failure mode: vertices the colour camera cannot see get no valid UV, and they must be filtered before they reach a mesher — filter invalid texture coordinates in a point cloud shows the test. Once the cloud is right, measure distance and bend angle from depth point clouds in Python is the worked follow-on for geometry.

Which depth cameras avoid the alignment problem entirely?

Pick a camera whose deliverable is already a calibrated cloud: structured-light, stereo and ToF depth cameras output 3D data from factory calibration, and a laser profiler computes every profile from its own calibrated laser-camera geometry, so no two pixel grids ever need registering in your code.

Where the deliverable is the cloud itself — volume, pose, presence — let the vendor's SDK own registration. Integrated depth cameras of these types carry the factory intrinsics for both paths, so the decision in the rule section is handled inside the SDK rather than in your code. The 3D sensors category groups laser profilers, structured-light and stereo units, and the camera finder filters them by type; the camera configurator narrows the field by resolution and interface.

If your rig instead pairs a separate depth camera with a separate colour camera — any area scan camera with the resolution you need — nothing is factory-registered. You calibrate the extrinsics yourself against a planar target, and then every rule above applies to your pair exactly as it does to an integrated unit.

Metrology at production rate is a different shape again. A laser profiler returns each profile already in calibrated sensor coordinates, computed from the factory laser-camera geometry, so there is no colour grid and no deprojection step; for inline profile work a sensor such as the MV-DP4180-03P removes the whole question.

How do you keep the wrong-intrinsics bug out of the next build?

Read the intrinsics from the stream you actually deproject, assert the vertex count against the colour resolution before the first lookup, and log all four values with both stream resolutions at every start-up. The checklist below turns those habits into integration steps.

• At start-up: read fx, fy, cx and cy from the profile actually deprojected — the colour profile for an aligned frame — never from a cached depth profile.
• At start-up: assert vertex count equals colour width × height before the first pixel lookup, and fail loudly when it does not.
• At integration: run the overlay and known-target checks on a rigged target, not on the first production part.
• At integration: confirm the depth unit once, in code, against a target at a known range.
• At maintenance: repeat the overlay test after every SDK or firmware update; alignment fixes arrive inside SDK releases, and any resolution change rescales the intrinsics.

If a cloud still fails the overlay after all of this, send the intrinsics log, both resolutions and the overlay image through the contact page and we will work through it with you — support is included with every order. The rest of the depth guides, from sensor choice to calibration, sit on the 3D topic page.