Aligned depth streams: data layout and which intrinsics apply

An aligned depth frame that arrives at the wrong resolution or decodes as striped noise is almost always a layout mistake, not a broken camera. The synthetic streams are simple once stated plainly: each is a single, ordinary image that lives on one camera's pixel grid. This guide gives you the buffer organization of depth_aligned_to_color and color_aligned_to_depth, the intrinsics each stream projects with, the byte counts and OpenCV types to expect, and the decision between the two alignments for measurement, texturing and robot guidance. It is written for anyone fusing depth and color from a depth camera in C++ or Python.

Is an aligned depth stream interleaved with color pixels?

No — a synthetic aligned stream is never interleaved. depth_aligned_to_color is a pure single-channel depth map, one 16-bit value per pixel, resampled onto the color camera's pixel grid. The color data stays in its own native stream.

The rgb, rgb, rgb … d, d, d picture does not exist in these APIs because no consumer wants it. Tracking, overlays and measurement all need the two images separately, each with one known pixel format. The SDK runs the registration inside its acquisition pipeline and hands you two ordinary image buffers, each with its own header. Drivers that deliver raw, unregistered pairs leave registration to you; synthetic aligned streams are the SDK's way of having already done it.

Both aligned views are produced for exactly the tracking-and-mapping use: registered geometry and appearance, consistent within one acquisition, with no extra work on your side. Dense RGB-D tracking consumes depth on one grid plus the registered color image — either alignment direction serves that, provided you project with the matching intrinsics in the next section.

Which camera's intrinsics does each aligned stream use?

The aligned frame uses the intrinsics of the camera whose grid it lives on. depth_aligned_to_color is projected with the color camera's fx, fy and principal point, and color_aligned_to_depth with the depth camera's.

A frame's intrinsics describe its pixel grid, not the origin of its data. depth_aligned_to_color behaves like a virtual range camera sitting at the color imager's pose, so its pixels unproject only with the color focal lengths. Forgetting the swap is the most common cause of a point cloud that is subtly scaled and shifted.

On distortion: the usual five-element coefficient set is laid out k1, k2, p1, p2, k3 — the radial pair first, then the tangential pair, then a third radial term — which is the convention most SDK headers document. The header comment beside coeffs is the contract, and fisheye models extend the set with more radial terms. Most depth cameras deliver rectified streams anyway, where the distortion model reads none and every coefficient is zero, so alignment needs only fx, fy and the principal point.

Which native streams do you enable to get the aligned pair?

Enable the two native streams, depth and color, and the aligned variants are simply there. The SDK computes each synthetic frame on demand from the natives, so no third stream is opened and no separate registration call is written.

That is why vendor alignment samples only assert that the natives are enabled — an is_stream_enabled check on depth and color — before requesting synthetic frames; there is no stream ID to open for depth_aligned_to_color itself. The SDK computes the aligned view when you request it, so its cost lands wherever you request it: ask for the 1280 × 720 remap only at the rate you consume it. An aligned frame that comes back empty means a native stream is not running — fix the enable, not the request.

What resolution and pixel format does each aligned frame keep?

An aligned frame takes the width and height of the camera it is aligned to and keeps its own pixel format. Depth stays one 16-bit channel and color stays three 8-bit channels on either grid.

depth_aligned_to_color from a 1280 × 720 color stream arrives 1280 × 720 and 16-bit; color_aligned_to_depth from a 640 × 480 depth stream arrives 640 × 480 and 8-bit × 3. Depth values follow the usual depth-camera convention: 16-bit unsigned millimetres — a stored 1500 reads as 1.500 m — and 0 marks no measurement, a hole rather than zero distance.

Read each frame's width and height from its own header instead of hard-coding one pair: wrap a 1280-pixel-wide buffer at a 640-pixel stride and every row starts mid-line, shearing the image. Bin or decimate a grid afterwards and every intrinsic scales with the factor — see Camera intrinsics after binning: scale them correctly.

How do you wrap the buffer in a Mat without corrupting it?

Wrap each buffer in a Mat of that stream's own width, height and channel depth — CV_8UC3 for color, CV_16UC1 for depth — and the frame decodes correctly; reuse another stream's Mat type and you get interleaved garbage.

The byte counts are worth memorising because they catch a wrong wrap immediately. For a 1280 × 720 color stream beside a 640 × 480 depth stream:

The wrap itself is one line, the same pattern the vendor samples use for color:

cv::Mat depth_a2c(720, 1280, CV_16UC1, (void*)frame_data);

Color buffers from most depth SDKs arrive in RGB byte order; OpenCV's imshow assumes BGR, so convert with cvtColor and the CV_BGR2RGB flag as those samples do, or red and blue swap.

A doubled or tripled color frame is the same class of error on the other stream — see Colour frame comes out doubled: fix Mat type pixel format mismatch.

Where does the depth value under a color pixel come from?

You unproject each aligned depth pixel with the color camera's focal lengths: X = (u − pp_x) × Z / fx and Y = (v − pp_y) × Z / fy give millimetres when Z is in millimetres, because fx and fy are in pixels.

Worked example, with example intrinsics for a 1280 × 720 color grid: fx = fy = 1000 px, pp_x = 640, pp_y = 360. The aligned depth at pixel (896, 360) reads 1500 (mm), so:

• X = (896 − 640) × 1500 / 1000 = 256 × 1.5 = 384 mm
• Y = (360 − 360) × 1500 / 1000 = 0 × 1.5 = 0 mm

The point sits 384 mm right of the optical axis, on the vertical centerline, 1.500 m from the color camera. The inverse check runs the same model backwards: u = 1000 × 384 / 1500 + 640 = 256 + 640 = 896, which returns the original pixel. The basis is the pinhole model on a rectified stream — the case most depth cameras ship, as above. Whatever unit Z is in, X and Y come out in; reading a millimetre stream as metres scales the cloud by exactly 1000.

When do you align depth to color, and when the other way?

Align depth to color when the decision is made on the color image and needs a height value under each pixel. Align color to depth when the workflow lives in the point cloud, because keeping depth as the target preserves every measured value.

Concretely, four cases:

• Inspection on texture — label print, surface stains, presence checks judged on the color image but gated by height: consume depth_aligned_to_color, one depth sample under every color pixel. Values resampled across a depth edge are interpolated, so gauge on the native grid, not the resampled copy.
• Point clouds, robot picking, volume — consume color_aligned_to_depth: the depth grid is untouched, so every XYZ value stays exactly as measured, and color is resampled onto it for an XYZRGB cloud, as in Full resolution color with depth point cloud.
• Feature lookups — a detection in the color image that needs its distance: keep both grids and look the color pixel up in depth, per Color pixel to depth lookup for detected image features.
• Gauging — keep depth native and resample only the color; resampling depth is the one direction that touches your measurement.

When the texture must out-resolve the depth camera's own RGB, plants add a separate color camera and align it to the depth map with calibrated extrinsics — the synthetic-stream idea, executed by you. A 20 MP color sensor such as the MV-CS200-10GC over GigE gives that side of the pair far more detail than any onboard RGB; the area scan cameras category filters by resolution and interface.

Where does alignment cost time, and where does it fail?

Alignment costs one remap per target pixel — about 0.9 million samples for a 1280 × 720 color grid — and it fails predictably at depth edges, where the color camera sees a surface the depth camera cannot.

The cost scales with the target grid, so run the remap for the frames you consume, not on every acquisition. The failures have three classic shapes. First, occlusion holes: the color and depth imagers sit tens of millimetres apart, so at a silhouette edge the color pixel sees background where the depth camera has no return — a 0-valued band, widest exactly where surfaces step. Second, resampling artifacts along those edges: nearest-neighbour fill leaves staircases and spurious depth steps that follow contours, the pattern behind Lines in aligned depth images: why artifacts follow edges. Third, calibration drift: a constant offset of a few pixels between the two grids that shows on every edge — diagnose and fix it in Depth to RGB registration offset in pixels: fix the rectified stream.

How much valid data survives is decided at the sensor, not in the remap: depth resolution and technology set the fill rate. The three families below cover short-range projected structured light, laser galvanometer stereo and ToF.

How do you check an aligned pair before you build on it?

Check an aligned pair with a physical target, not by eye: project a measured corner through the stream's own intrinsics and expect it inside one to two pixels, then inspect the hole band at edges before you trust depth values there.

1. Type and size. Print the Mat type and dimensions and compare against the stream header: CV_16UC1 at the target grid for aligned depth, CV_8UC3 for aligned color.
2. Edge overlay. Overlay depth edges on the color image from one acquisition; a straight machined edge should land on itself within a pixel or two, as set out in Depth image edge overlay from a single acquisition stream.
3. Corner projection. Measure a corner in the scene, project it with the formulas above and compare. More than a couple of pixels means stale calibration — first confirm whether the frames you request are raw or rectified (Depth to color registration: raw versus rectified frames, Is depth image_raw aligned to RGB? Check registration).
4. Zero policy. Decide before the first measurement how 0-valued pixels are handled: mask them out or fill from a bounded neighborhood, and never let them average into a statistic.
5. Decision. Measuring on texture: enable depth and color, consume depth_aligned_to_color with the color intrinsics. Texturing a cloud: consume color_aligned_to_depth on the untouched depth grid. Gauging: native depth only, color resampled.

If you are still choosing hardware, the 3D sensors category and the camera finder filter by depth technology and interface, and the camera configurator assembles the rest of the chain. More registered-depth material lives on the 3D vision topic page.