You enabled the SDK's depth-to-color registration mode, the property write failed or the frames are recorded, and the fallback everyone reaches for - cv2.stereoRectify with the printed calibration - still leaves the overlay shifted. Neither cause is a bug in your matrix arithmetic. stereoRectify builds a virtual rectified pair, and the depth stream's pixel grid is not exactly the grid the extrinsics describe. This guide replaces the rectify step with a per-pixel projection of depth into the color camera, adds one measured constant, and proves the result with a phase-correlation test. The worked example uses a real 640 × 480 calibration print-out with a 25.1 mm baseline; after the fix the overlay lands within 1 px.
Why stereoRectify leaves the shift
cv2.stereoRectify computes two new projection matrices and re-projects both cameras into a virtual rectified pair. With CALIB_ZERO_DISPARITY both new principal points are forced onto a shared center, and the alpha parameter rescales and crops the result. Rectified depth and rectified color align with each other, but they live in neither camera's real pixel grid, so overlaying rectified depth on the original color frame inherits the principal-point difference: cy_d - cy_c = 251.533 - 229.077 = 22.5 px of vertical shift before any other error contributes. The stream-level side of this - what raw versus registered output means - is covered in depth to color registration: raw versus rectified frames.
The second cause sits outside the calibration block. Depth is triangulated on the infrared image, so the printed rotation and translation relate the IR camera to the color camera. The depth engine then outputs its own crop of that IR grid; the vendor trims IR and depth slightly differently for throughput, and that difference appears in neither printed matrix. It shows up as a constant horizontal shift - 11 px in the recorded case at 640 × 480 - and no choice of R, T, alpha or flag inside stereoRectify can remove it. A crop also rescales whatever intrinsics travel with it, the same trap that breaks pose estimation after an ROI crop; see SolvePnP returns wrong results when a crop rescales the intrinsics.
The rotation is not the problem: with off-diagonal terms of 0.003 to 0.005 it moves points by under 0.1 px at 1 m. The 25.097 mm translation and the two principal points do essentially all the work. So skip rectification and reproject.
What you need
- Depth and color frames of the same scene, live or recorded; depth as 16-bit millimeters with 0 marking invalid pixels.
- The printed calibration block: both 3 × 3 intrinsic matrices, both distortion sets, the rotation matrix and the translation in millimeters. On this print-out 25.097 mm is the physical IR-to-color lens spacing, which is how you confirm the unit.
- OpenCV 4.x, for cv2.undistortPoints, cv2.projectPoints, cv2.remap and cv2.phaseCorrelate.
- A test scene with a strong vertical edge at a known distance; 1 m is a good anchor.
Build the mapping in five steps
- Enter the calibration in OpenCV order. The five distortion values on these print-outs read (k1, k2, k3, p1, p2); OpenCV wants (k1, k2, p1, p2, k3). The tangential pair is always the two smallest values, so the 0.2995-magnitude entry belongs in k3, not p1 - fed there it shears the remap visibly.
- Lift depth pixels to 3-D. Undistort the pixel grid to normalized rays and multiply by the depth value. These streams are planar depth (Z along the optical axis), so a point is Z times its ray, in millimeters.
- Transform into the color frame. Apply X_c = R·X + T with the printed R and T, everything in millimeters.
- Project with the color camera. cv2.projectPoints with zero rotation and translation applies the color intrinsics and its own distortion to the transformed points.
- Warp and mask. Feed the projected coordinates to cv2.remap as the source map for the color frame, add the measured crop shift to the x map, and black out pixels where depth is 0. A zero depth would otherwise project onto the camera origin; depth pixel deprojection zero: diagnose invalid 3D points covers that failure.
import numpy as np
import cv2
# Calibration print-out of the example camera: 640 x 480 streams, millimeters
K_d = np.array([[578.938, 0.0, 318.496],
[0.0, 578.938, 251.533],
[0.0, 0.0, 1.0]]) # depth (IR) camera, 3x3 float64
K_c = np.array([[517.138, 0.0, 319.184],
[0.0, 517.138, 229.077],
[0.0, 0.0, 1.0]]) # color camera
# OpenCV order (k1, k2, p1, p2, k3); the two ~1e-3 values are tangential
D_d = np.array([-0.094342, 0.290512, -0.000318, -0.000279, -0.299526])
D_c = np.array([0.044356, -0.174023, 0.001794, -0.003853, 0.077324])
R_dc = np.array([[1.000, -0.003, 0.002],
[0.003, 1.000, 0.005],
[-0.002, -0.005, 1.000]]) # depth frame -> color frame
T_dc = np.array([-25.097, 0.288, -1.118]) # mm, same direction as R_dc
crop_dx = 11.0 # px, measured once (see the test)
# depth_mm: uint16 HxW in millimeters, 0 = invalid; colour_bgr: uint8 HxWx3
h, w = depth_mm.shape
uu, vv = np.meshgrid(np.arange(w, dtype=np.float32), np.arange(h, dtype=np.float32))
grid = np.stack([uu, vv], axis=-1).reshape(-1, 1, 2)
rays = cv2.undistortPoints(grid, K_d, D_d).reshape(-1, 2)
Z = depth_mm.reshape(-1, 1).astype(np.float64)
X = np.hstack([rays.astype(np.float64) * Z, Z])
Xc = X @ R_dc.T + T_dc
uv_c, _ = cv2.projectPoints(Xc.reshape(-1, 1, 3), np.zeros((3, 1)), np.zeros((3, 1)), K_c, D_c)
map_c = uv_c.reshape(h, w, 2).astype(np.float32)
map_c[..., 0] += crop_dx
color_reg = cv2.remap(color_bgr, map_c[..., 0], map_c[..., 1], cv2.INTER_LINEAR)
color_reg[depth_mm == 0] = 0
Once frames live in this registered layout, track which intrinsics each consumer needs; aligned depth streams: data layout and which intrinsics apply walks that bookkeeping.
Worked example: one pixel, the arithmetic shown
Distortion is left out of the by-hand numbers; near the optical axis it contributes well under a pixel. Take the depth pixel (u, v) = (320, 240) at Z = 1000 mm:
- x_n = (320 - 318.496) / 578.938 = 0.0026, y_n = (240 - 251.533) / 578.938 = -0.0199
- X_c = 1000 × 0.0026 - 25.097 = -22.50 mm; Y_c = 1000 × (-0.0199) + 0.288 = -19.63 mm; Z_c = 1000 - 1.118 = 998.88 mm
- u_c = 319.184 + 517.138 × (-22.50 / 998.88) = 307.5; v_c = 229.077 + 517.138 × (-19.63 / 998.88) = 218.9
The pixel lands 12.5 px left and 21.1 px up in the color frame. At two more distances the horizontal shift follows the baseline rule fx_c · B / Z = 517.138 × 25.097 / Z, while the vertical shift stays near -21 px because it comes from the 22.5 px principal-point difference, not from the baseline:
| Z (mm) | u_c (px) | horizontal shift (px) | v_c (px) |
|---|---|---|---|
| 500 | 294.5 | -25.5 | 219.1 |
| 1000 | 307.5 | -12.5 | 218.9 |
| 2000 | 314.0 | -6.0 | 218.8 |
The parallax rule predicts 26.0, 13.0 and 6.5 px at those three depths, within 0.5 px of the full model, which is the cross-check to run on your own print-out. To turn registered pixels into object extents, continue with bounding box depth: mapping image pixels to 3D points.
Prove it: phase correlation at two distances
# Test: phase correlation of edge images, before and after the crop shift
win = cv2.createHanningWindow((w, h), cv2.CV_32FC1)
e_c = cv2.Canny(cv2.cvtColor(color_reg, cv2.COLOR_BGR2GRAY), 60, 180).astype(np.float32)
depth_u8 = cv2.normalize(depth_mm, None, 0, 255, cv2.NORM_MINMAX).astype(np.uint8)
e_d = cv2.Canny(depth_u8, 60, 180).astype(np.float32)
(dx, dy), response = cv2.phaseCorrelate(e_d, e_c, win)
print(dx, dy) # recorded scene: -11.2, 0.4 before crop_dx; 0.2, 0.3 after
assert abs(dx) < 1.5 and abs(dy) < 1.5
On the recorded scene this test returned dx ≈ -11.2 px before the crop shift and 0.2 px after it. Then repeat the measurement at a second distance. A shift that stays within about half a pixel of the first is the crop constant; a shift that scales with 1/Z means the translation's unit or sign is wrong, so flip the sign and rerun before touching anything else. If the residual still scales with distance after that, the calibration itself is stale and stereo camera RGB calibration: fix depth-to-color alignment errors walks the redo. Frames that a tool already rectified and that simply sit offset are a different failure; depth to RGB registration offset in pixels: fix the rectified stream covers those.
Common mistakes and their fixes
- Rectifying anyway, just to be safe. Rectified output belongs to a virtual pair, not to the color grid. Use stereoRectify only when you actually want disparity, and then read depth from a match with depth from a matched stereo point pair: disparity to 3D coordinates.
- Mixing units. Translation and depth are both millimeters here; reading one as meters scales every projection by 1000 and throws all points off the image.
- Wrong transform direction. If the overlay moves twice as far off instead of converging, your block maps color to depth: apply X_c = R^T (X_d - T) and rerun.
- Trusting the sync flag. setDepthColorSyncEnabled, the depth-color sync call in the device API, tightens capture timing; it moves no pixel and cannot fix a geometric shift.
- Recalibrating to absorb the crop. A per-stream crop cannot be recalibrated away; measure it once and keep it a named constant next to the mapping.
When the on-device registration is still the answer
If your SDK build exposes the depth-to-color registration mode (DEPTH_TO_COLOR in the device API) and the property write succeeds, use it. The on-device mapping includes the vendor's crop formula and returns frame pairs that are already time-aligned, which matters on anything that moves. Your own pipeline is the right path for recorded streams, for builds where the mode reports supported but the write fails, and when the color comes from a separate camera. One platform quirk worth knowing: some SDK versions attach the registration property to the depth stream rather than the device object, so if a device-level call fails, try the stream-level property before writing your own mapping.
The same projection registers depth against a second, external color camera: use its intrinsics in step 4 and measure the new baseline with a checkerboard. That is the route to color beyond the built-in sensor's native frame - any of the area scan cameras can serve as the second camera.
If you are specifying hardware rather than fixing software, shortlist depth cameras that publish the full calibration block and stream depth and color natively. Two structured-light options in the 3D sensors category:
- Type
- Stereo / structured light
- Type
- Stereo / structured light
Both project structured light; their product pages list depth range and accuracy. For the color channel or a different interface, the camera finder filters by resolution, shutter and interface.
The decision
- Recorded frames, or the registration mode unavailable on your build: build the five-step mapping and compensate the measured crop constant. Under 1 px residual at 1-2 m is the target the worked example hits.
- Registration mode available and the write succeeds: use it, and keep the phase-correlation test as your commissioning check.
- You need disparity rather than registered pairs: rectify - that is the one job stereoRectify is for.
- A constant shift survives everything: it is the crop, measure and add it. A shift that scales with distance is calibration, redo it.
More depth-stream integration fixes sit on the 3D topic page, and the configurator below starts at the camera step if you are still choosing hardware.
The configurator matches a camera, lens, light, light controller and cables to each other, step by step, and lists only parts that fit what you already picked.
Questions
Why does my depth to color overlay drift at close range but look fine at 2 m?
Baseline parallax scales with 1/Z. With this calibration, fx = 517.138 px and B = 25.097 mm, so fx·B/Z gives 6.5 px at 2 m, 13.0 px at 1 m and 26.0 px at 0.5 m. That drift is correct geometry, not an error. The crop offset is the part that must stay constant with distance; if the offset itself scales with 1/Z, the translation unit or its sign is wrong.
Does enabling depth-color sync fix misregistration?
No. A depth-color sync flag only tightens capture timing so both frames expose the same instant; it moves no pixel. Registration is the geometric mapping from the depth grid to the color grid. On a moving scene you need both: sync removes temporal skew, about 8 ms of motion per half-frame at 30 fps, while the reprojection in this guide removes the spatial shift.
How do I register depth into the color image instead of color onto depth?
The mapping in the guide warps color onto the depth frame, which suits pipelines that index results by depth pixel. For depth inside the color image, scatter the projected depth points into a color-sized map instead, then fill the single-pixel holes the forward projection leaves. Do not invert the remap table directly: occluded color pixels have no depth point landing on them, and the inversion is undefined there.
How accurate is a self-built registration compared with the on-device one?
Sub-pixel to about 1 px at 1-2 m when the crop offset is measured properly, which matches the vendor viewer alignment on the recorded scene. The residual grows with calibration age: a bumped camera or swapped mount shifts the extrinsics, and the depth-dependent error doubles every time the distance halves. If two-distance phase correlation still shows that component after the sign check, redo a stereo calibration with a checkerboard from 0.5 m to 2 m.
The images look almost undistorted. Can I skip the distortion coefficients?
No. Radial error grows with the square of the radius. At the 640 × 480 corner of this depth camera the normalized radius is 0.68, so r(k1 + k2·r^2 + k3·r^4) = 0.68 × (-0.094 + 0.134 - 0.064) = -0.016, about 9 px at fx = 578.9. Even a clean-looking frame carries that warp, and phase correlation will report part of it as a few pixels of shift near the edges.