Depth and color at the same resolution are not aligned

You set depth_width and color_width to 640 and both heights to 480, the driver publishes two 640×480 streams, and the tempting conclusion follows: pixel (320, 240) in depth is pixel (320, 240) in color. It is not. Matching sizes only makes the two buffers equal; each stream still carries its own sensor's geometry — a different focal length in pixels, a different principal point, lens distortion on the color side and a physical offset between the imagers.

This guide separates what size matching does from what alignment does, shows why a 640×480 depth frame can be a padded 628×468 measurement, and works through the projection that produces one-to-one (u,v) correspondence, with arithmetic you can repeat on your own intrinsics.

Same resolution matches buffers, not pixels

The size parameters control one thing: how many rows and columns the driver allocates and delivers. They do not touch geometry. A depth pixel at (u, v) is one ray through the depth rig's optical center; the color pixel at the same (u, v) is a different ray through the color sensor's center. Two imagers mounted centimeters apart with different lenses cannot produce identical ray grids by coincidence.

Depth adds a twist of its own. Stereo depth is computed from two infrared images that the driver rectifies first, so the depth frame already lives on the first infrared camera's rectified grid. The color frame is usually the raw sensor image, barrel distortion included, unless you request the rectified variant. That makes three possible grids — rectified depth/IR, rectified color, raw color — and matching sizes collapses none of them into the others. The mechanics are laid out in depth to color registration: raw versus rectified frames, and the direction you pick decides which stream keeps its original values.

One buffer size, two fields of view

Focal length in pixels links a pixel index to a viewing direction, and it changes with stream size. The field of view follows:

Take illustrative intrinsics typical of a compact depth module, with c_x = 320 px and c_y = 240 px on both sides: 640×480 output with f = 380 px on the depth side and f = 620 px on the rectified color side. Depth: 2 × atan(640 / 760) = 2 × 40.1° = 80.2°. Color: 2 × atan(640 / 1240) = 2 × 27.3° = 54.7°. Equal buffers, scenes about 25° apart in coverage: the color frame is a central crop of what depth sees, and its pixels resolve about 1.6× finer in angle (620 / 380). Identical sizes, different physics — an overlay built on equal angles drifts by tens of pixels near the frame edges.

The depth frame may be padded, not native

Hardware decides which depth sizes are real. On one legacy stereo depth line the native modes were 628×468, 480×360 and 320×240; 640×480 was never a sensor mode. Ask the driver for 640×480 anyway and it obliges: native depth sits six pixels inside the first infrared frame on all four sides and is otherwise pixel-aligned with it, so the driver pads the 628×468 measurement out to 640×480 (628 + 12 = 640, 468 + 12 = 480) or crops the infrared frame instead. Request matching depth and infrared sizes and both grids share the exact same intrinsics.

The general lesson: a buffer size is a driver contract, not a sensor readout. The border carries no measurement — typically zeros — so any minimum, maximum or blob statistic that touches it is reading padding, not the part.

The projection that actually aligns them

Alignment is a per-pixel projection through the color camera's rectified intrinsics — the standard camera-matrix model described on Wikipedia's camera resectioning page. For each depth pixel:

Z is the depth value, which on most cameras measures distance along the optical axis rather than slant range — the conversion is covered in depth value vs true distance: convert a pixel's Z to camera distance.

R sits close to identity for boresight-aligned modules and t is the small baseline between imagers. Both come from the per-unit factory calibration, never from a nominally identical unit: otherwise matching cameras have shown per-unit stereo scale differences, as stereo depth scale differs between identical cameras: verify and fix documents. Run the loop over all 307,200 pixels and the output is a color-indexed depth image — one-to-one (u,v) correspondence by construction.

Worked example: projecting one depth pixel

Illustrative numbers for the class; your SDK's intrinsics call prints the real ones. Both streams run 640×480 with c_x = 320 px and c_y = 240 px, while f_x = f_y is 380 px on the depth side and 620 px on the rectified color side. R = identity, t = (−0.025, 0.000, 0.000) m — a 25 mm baseline between imagers.

Center pixel: u_d = 320, v_d = 240, Z = 1.500 m.

The depth-frame center lands at color pixel (310, 240): 10.3 pixels left of center, from the 25 mm offset alone. At Z = 0.500 m the same depth pixel shifts 620 × 0.025 / 0.500 = 31.0 px. Parallax scales as 1/Z, so close parts move far more than far ones.

Edge pixel, same frame: u_d = 639, v_d = 240, Z = 1.500 m.

Column 830 does not exist in a 640-wide color frame: that depth pixel has no color counterpart. A correct registration writes a hole there, never a clamped or wrapped value. Expect a hole band around the border wherever depth coverage exceeds color coverage.

Where one-to-one correspondence is impossible

Four effects leave some pixels permanently unpaired:

• Field-of-view mismatch. In the worked example, color covers ±27.3° horizontally and depth ±40.1°; every depth direction beyond ±27.3° maps outside the color frame and must be dropped.
• Baseline occlusion. The 25 mm offset means a part edge at 0.100 m range shifts 620 × 0.025 / 0.100 = 155 px between grids. Surfaces visible to one imager are genuinely hidden from the other; the shadow cannot be interpolated back.
• Resolution ratio. Registered depth is resampled, and upsampling a smaller depth grid into a larger frame adds interpolation, not information — upsampled depth frames: alignment resolution vs accuracy quantifies the trade.
• Exposure timing. Equal frame rates are not synchronized shutters. A part moving at 0.5 m/s with a 2 ms offset between depth and color exposure smears the correspondence by 0.5 m/s × 0.002 s = 1 mm; judge timestamps, not fps.

Doing it in practice

Prefer the SDK's registration call over a hand-rolled loop: depth SDKs ship one that applies intrinsics, extrinsics and the padding or cropping rules in a single pass. The reverse direction is the same machinery run backwards — color pixel to depth projection: one point instead of a whole frame shows the single-point form and map RGB to depth: color-to-depth projection, step by step the full-frame version.

If you project yourself, read the intrinsics after setting stream sizes, because focal length in pixels changes with resolution. Keep meters on both sides of the extrinsics, and vectorize the loop — 307,200 pixels per frame is trivial as an array operation and slow one pixel at a time.

Depth cameras that pair depth with color span projected structured light, stereo and galvanometer-scanned types, and all of them register the same way. Some of these units steer laser light across the scene, so respect the class label on the housing and power the projector down before reaching into the field of view. Examples from the shop:

Pick the direction by where the measurement lives. Depth-to-color when detections run on the RGB image or the overlay is human-facing; color-to-depth when metrology runs on the depth grid and you only want texture carried across, leaving depth values and their resolution untouched — why aligned depth to color keeps the depth resolution explains what survives the trip.

When the on-board color imager is not sharp enough, mount a separate area scan camera beside the depth head, calibrate the extrinsics once per station, and the same projection applies with better color data.

Checks before you trust the aligned pair

• Project a known point. Place a marker at a measured (X, Y, Z) and check the projected (u_c, v_c) lands within 1–2 px of the marker in the color image.
• Straight-line test. A straight edge must stay straight in rectified color; residual bow means raw and rectified frames got mixed.
• Border audit. Echo one border row and column of the depth frame; zeros or one constant invalid value mean a padded frame, so mask it.
• Overlap check. Compute both fields of view from the intrinsics and confirm the band you actually inspect lies inside both.
• Timestamp gap. Keep depth and color stamps inside the motion budget: at 0.5 m/s, 2 ms of gap is 1 mm of smear.
• Hole count. Depth pixels without color at the frame border are expected geometry, not a defect. Count the band once and exclude it from scoring.

The decision

Same size is not alignment: resolution matching equalizes buffers, and one-to-one pixels exist only after the projection. Run registration on every fused capture, choose the direction by where the measurement lives, mask padded borders, and verify each station with one known point. A 640×480 pair that passes those checks is a measurement; the same pair without them is two unrelated images that happen to share a size.

For hardware, filter the camera finder by interface and resolution or start at the camera step of the configurator; the 3D topic page gathers the alignment guides, including depth to RGB registration offset in pixels: fix the rectified stream for pairs that land close but shifted.