How to align depth and color at different resolutions

The fix is a per-pixel projection: deproject each 1280×720 depth pixel to metres with the depth intrinsics, carry it across the depth-to-color extrinsics, and project it into the 1920×1080 color frame with the color intrinsics. Run the depth stream at 1280×720 and color at 1920×1080, call a wrapper that only copies pixels, and the depth texture comes back stretched, sheared or half black — 921,600 depth pixels pasted into a frame of 2,073,600 fill just 44.4% of it. The same code runs cleanly with both streams at 1280×720, so the camera looks like the fault. It is not. The projection works at any pair of resolutions; below are the reason a pixel-copy wrapper fails, the three-step math, a worked 1280×720-to-1920×1080 example, and when matching the stream sizes is the better choice.

Why mixed resolutions tear a copied depth map

A depth frame is a grid of distances; a color frame is a grid of colors. Alignment answers one question for every depth pixel: where does this same surface point land in the color image? Geometry gives the answer, through the depth intrinsics, the depth-to-color extrinsics and the color intrinsics. A pixel copy assumes the answer is always the same row and column, which holds only when both grids share size and viewing geometry.

Written forward, a copy fails immediately at mixed sizes. The 1280×720 depth map holds 921,600 pixels; the 1920×1080 color frame holds 2,073,600. Pasting rows of 1280 into rows of 1920 fills the top-left corner: 921,600 / 2,073,600 = 44.4% of the frame, with the remainder black or stale. Reading the other way — code that samples the color texture at depth coordinates — every sample comes from too close to the top-left. With matched fields of view, the true position of depth column u is 1.5 × u. The copy misses by 0.5 × u pixels: 320 px at the frame centre, 640 px at the right edge.

In a rendered mesh the same mistake shows as a torn or doubled depth texture, because UVs computed for one texture size get applied to another. Equal sizes hide the defect; mixed sizes expose it. When alignment works only at equal resolutions, the mapping in play is a copy, and the cure is the projection below or a current wrapper — not a permanent restriction to matched modes. To get a rig running right now, matching the stream sizes is the fastest fix; Depth to color alignment breaks up: match stream resolution first covers that route and its cost in color detail.

The projection that works at any resolution pair

Three steps run per depth pixel, each using numbers in the pixels of its own stream:

1. Deproject the depth pixel to metres with the depth intrinsics fx_d, fy_d, ppx_d, ppy_d.
2. Rotate and translate the point into the color frame with the extrinsics R and t.
3. Project it with the color intrinsics fx_c, fy_c, ppx_c, ppy_c.

fx_d counts depth pixels and fx_c color pixels, so each stream's resolution already sits inside its own constants; no special case for mixed sizes exists or is needed. To build a color overlay shaped like the depth map, run the projection once per depth pixel and fetch the color at (u_c, v_c). To draw depth on color, compute the same target and write the depth value there, interpolated — never rescaled, for the reason in the fixes section below.

One trap sits after the math: a registered frame is described by the intrinsics of the grid it inherited, not always by the color camera's own set. Aligned depth streams: data layout and which intrinsics apply sorts out which set applies where. And if your SDK ships no alignment helper, this mapping is exactly what you build by hand, as walked through in Align depth and color without the camera SDK pipeline.

Intrinsics scale with resolution — and when they do not

fx is the focal length measured in pixels: physical focal length divided by pixel pitch. Open the same sensor view at 1920×1080 instead of 1280×720 and the pixels shrink by 1.5, so fx, fy, ppx and ppy each grow by 1920/1280 = 1.5. The constants absorb the resolution change; that is the whole reason the projection needs no special case.

The ratio stops holding when the two imagers view different fields of view, which is common: many depth cameras ship a color imager that sees a narrower scene than the depth imager. Then fx_c / fx_d is not 1.5, the per-stream intrinsics the camera reports are the numbers to use, and the projection itself is untouched. The shortcut has one legitimate job: correcting for your own post-capture changes. Whatever your code does to the width of a stream, do the same to fx and ppx — binning 2×2 halves the width and must halve fx, exactly the correction covered in Camera intrinsics after binning: scale them correctly. Converting between the parameter formats different SDKs print is set out in Camera intrinsic parameter conversion for depth and texture images.

Worked example: placing one 1280×720 depth pixel into a 1920×1080 color frame

Inputs, with their source:

• Stream sizes 1280×720 (depth) and 1920×1080 (color): the mixed pair behind the failure this article answers.
• Pixel to place: (u_d, v_d) = (896, 432), with depth z = 2.000 m read from that pixel.
• Depth intrinsics: fx_d = fy_d = 640 px, principal point (640, 360) px — illustrative values declared here (centred, square pixels, roughly 90° field of view), not quoted from any datasheet.
• Color intrinsics: fx_c = fy_c = 960 px, principal point (960, 540) px, from the same-FOV scale rule.
• Extrinsics: R = identity, t = 0 — co-located cameras. A real rig adds the factory rotation and baseline, and the same three steps carry them.

Reading z straight out of a depth pixel, without an SDK helper, is its own small procedure: Read distance from a depth image pixel without an SDK.

Rule in words: deproject with the depth intrinsics, carry the point across the extrinsics, project with the color intrinsics. With R = I and t = 0 the middle step passes the point through unchanged:

Result: the surface seen at depth pixel (896, 432), 2.000 m out, owns the color at (1344, 648). Cross-check with fractions: 896/1280 = 0.700 = 1344/1920 and 432/720 = 0.600 = 648/1080, which agree because the fields of view agree. The pixel copy would have written the value at (896, 432) — 448 px left of and 216 px above the true spot — and at the right edge of the frame that horizontal error grows to 640 px.

Decision: run the projection and keep the mixed pair — 720p depth plus all 2,073,600 color pixels. If your wrapper insists on equal sizes, matching both streams at 1280×720 is an acceptable stopgap: 921,600 color pixels, 1,152,000 fewer than 1080p, until the copy is replaced by this projection or the wrapper is updated. Run the fraction test once at bring-up; it is the pass/fail check on your own rig.

Matched streams or intrinsic mapping: which to choose

For a display-only overlay on a bench rig, match the modes and move on. When the inspection reads color detail — print, text, surface finish — project, because the 1,152,000 extra color pixels are the entire point. For dimensional measurement, measure in the depth frame and let color aim and document; then either route stays metric, as long as z is never rescaled. Whether RGB and the point cloud belong in one device or in two is a separate decision, set out in 2D and 3D capture strategy: combine or split RGB and point cloud.

Two resolution facts complete the picture. A registered frame keeps the depth grid's resolution unless the call resamples it — Why aligned depth to color keeps the depth resolution. And inflating depth to the color grid buys appearance, not accuracy — Upsampled depth frames: alignment resolution vs accuracy.

Three fixes that corrupt depth instead

Resize the depth image and call it aligned

An image resize keeps the depth grid's own viewing geometry, so unless both sensors share field of view and principal point the resized map is still misregistered. A nearest-neighbour stretch by 1.5 also duplicates pixels in an irregular 1-2 pattern that paints jagged depth edges. Resize for display if you like; align with the projection.

Scale z to match the frame sizes

Depth values are metres. The 1.5 ratio lives in pixel coordinates through fx and fy, never in z. Dividing the worked example's 2.000 m by 1.5 reports 1.333 m for the same surface — a 33% measurement error from one well-meant line.

Share one intrinsics set across both streams

Projecting with fx_d = 640 px into the 1080p frame lands the worked-example pixel at (1216, 612) instead of (1344, 648). Off-centre content collapses toward the principal point by 640/960 = 0.667 — here 128 px left and 36 px up, and a third of the offset everywhere else.

When the color camera is a separate unit with no factory depth-to-color transform, calibrate R and t once yourself and then reuse the projection unchanged; Align depth to a third RGB camera: fix missing extrinsics covers the calibration path.

Checks before you trust the overlay

• Frame size. Read the width and height your library returns for the aligned frame instead of assuming them; a mismatch here is the most common source of a torn texture.
• Intrinsics per mode. Pull intrinsics after opening the exact stream modes, and if your code crops, bins or resizes a stream afterwards, scale fx, fy, ppx and ppy by the same ratios.
• Fraction test. Map the depth frame's centre pixel and one corner pixel. The centre lands on the color centre, and with matched fields of view the corner's width and height fractions match on both sides, as 0.700 does above.
• Feature test. Aim at a printed target at 2.000 m. The color fetched at the mapped coordinate must show the target, not its neighbour. Drift at the edges only points to wrong focal lengths. Uniform drift points to the translation.
• Magnitude test. Depth values at the same surfaces must equal the inputs in metres. Any change means z was rescaled, and that step has to go.
• Link budget. Check the interface carries both modes at your frame rate before blaming alignment for dropped frames.

Cameras for depth-plus-color work

Any depth camera that also delivers a texture stream runs this alignment, whatever the depth technology — stereo, structured light or time of flight. Examples from the 3D sensors range:

When the color side needs more detail than a depth camera's texture stream delivers, split the job: depth camera for geometry, separate color camera for detail, registered by a calibrated transform rather than matched pixels. The color unit comes from the area scan cameras range; a 20 MP GigE color camera such as the MV-CS200-10GC covers the detail side:

To shortlist hardware, start at the cameras finder or step through the camera configurator; both filter by interface, resolution and shutter. Pairing a depth unit with a color camera is the kind of choice the applications team handles daily — free expert support comes with every order, and the contact page is the way in. More worked 3D topics sit on the 3D vision topic page.