Align near-infrared streams to RGB: extrinsic projection how-to

A stereo depth camera hands you a color stream and one or two near-infrared streams, and most pipelines want the same surface landing on the same pixel in all of them. The SDK's align function pairs depth with color, but it refuses an infrared-to-color pair, because alignment needs depth as an input. This how-to shows the way through: read the factory extrinsics off the device, project every RGB pixel into the infrared image, and sample there. The math is four lines and the code a dozen; the bench check then tells you within a couple of pixels whether you got it right. The worked numbers assume a 720p stereo module and Python with OpenCV, but the projection runs in any language with a matrix library.

Why alignment needs depth in the pair

An infrared-to-color mapping is not a fixed 2D shift. The two lenses sit tens of millimetres apart, so the same object lands at different pixel positions depending on how far away it is: the closer the object, the larger the offset. Mapping pixels between two perspective views needs a 3D position for each one, and that comes from the depth stream. That is why the align function insists on depth as one of its two streams. Depth-color works, depth-infrared works, and an infrared-color pair has nothing to drive the reprojection.

The escape is to supply the missing dimension yourself. An RGB pixel plus its depth is a 3D point. The stored rotation and translation move that point into the infrared camera's frame, and the infrared intrinsics project it back down to a pixel. Everything else in this how-to is bookkeeping around those three operations.

Read the calibration the device already carries

Ask the SDK for three things, per stream: the pinhole matrix — fx, fy, cx, cy in pixels — the distortion coefficients, and the extrinsic between the pair as a 3×3 rotation R and a translation t. Factory-calibrated modules store all of it on the device and print it at runtime. No calibration board is needed for this job.

Sanity-check two values before use. The magnitude of t should equal the mechanical spacing of the two lens centres, roughly 15 to 100 mm on stereo depth modules; metres or microns mean the units are wrong. R should sit within a few tenths of a degree of the identity, because both cameras share one chassis. A 0.2° rotation is worth fx·θ ≈ 645 × 0.0035 ≈ 2.3 px on a 645-pixel-focal-length camera, so the stored R goes into the math even when it looks trivial.

If the stream list or calibration output looks wrong, confirm the streams and their metadata first; the raw stereo image calibration check covers what to verify before trusting any of it.

The projection, step by step

Z is the pixel's depth in millimetres, read from the depth stream after depth-to-color alignment. Five steps, run once per frame on the pixel grid:

1. Pull the intrinsics, the distortion coefficients and the color-to-infrared extrinsic from the SDK's calibration.
2. Undistort each RGB pixel with the color coefficients to get its normalized ray (x_n, y_n).
3. Scale the ray by the pixel's depth Z to get the 3D point in the color frame.
4. Move the point with P_ir = R · P_rgb + t into the infrared frame.
5. Project with the infrared intrinsics and sample the infrared image there — bilinear, never nearest neighbour, because the map lands on fractional pixels.

For a whole image, run the same steps on a meshgrid of pixels and reshape the output into two float32 maps for a remap call. With a fixed-distance Z the maps are built once; with per-pixel depth they rebuild every frame, which is still a handful of matrix operations at 720p.

import numpy as np
import cv2

# Example device, 1280 x 720 streams. Replace every value with your SDK's calibration.
K_rgb = np.array([[900.0, 0.0, 640.0],        # 3x3 float64, color pinhole matrix
                  [0.0, 900.0, 360.0],
                  [0.0, 0.0, 1.0]])
d_rgb = np.zeros((1, 5), np.float64)          # 1x5 float64, color distortion coefficients
K_ir = np.array([[645.0, 0.0, 640.0],         # 3x3 float64, infrared pinhole matrix
                 [0.0, 645.0, 360.0],
                 [0.0, 0.0, 1.0]])
d_ir = np.zeros((1, 5), np.float64)           # 1x5 float64, infrared distortion coefficients
R = np.eye(3)                                 # 3x3 float64, rotation, color -> infrared
t = np.array([[-18.0], [0.0], [0.0]])         # 3x1 float64, mm, color origin in IR frame

uv = np.array([[[640.0, 360.0]]], np.float64) # 1x1x2 float64, RGB pixel under test
Z = 600.0                                     # scalar float, mm, aligned depth at that pixel

ray = cv2.undistortPoints(uv, K_rgb, d_rgb)             # 1x1x2, normalized ray
p_rgb = ray[0, 0] * Z                                   # (3,) float64, mm, color frame
p_ir = R @ p_rgb + t.ravel()                            # (3,) float64, mm, IR frame
uv_ir, _ = cv2.projectPoints(p_ir.reshape(1, 1, 3),
                             np.zeros(3), np.zeros(3), K_ir, d_ir)
print(uv_ir[0, 0])                                      # [620.65 360. ]
assert np.allclose(uv_ir[0, 0], [620.65, 360.0], atol=0.05)

If your pipeline also builds point clouds from the same device, the same three calibration sets decide which matrix deprojects which stream; which camera to deproject with sorts that out. Whether the depth image shares pixels with the left infrared image is itself a checkable claim; is depth image_raw aligned to RGB? shows the check.

Worked example: where the color centre lands

With the example values — fx_rgb = fy_rgb = 900 px, cx_rgb = 640, cy_rgb = 360, fx_ir = fy_ir = 645 px, cx_ir = 640, cy_ir = 360, R = identity, t = (−18, 0, 0) mm, Z = 600 mm:

1. Ray: x_n = (640 − 640)/900 = 0 and y_n = (360 − 360)/900 = 0, because the test pixel is the color image's optical centre.
2. Point: P_rgb = 600 × [0, 0, 1] = (0, 0, 600) mm.
3. Transform: P_ir = (0 − 18, 0, 600) = (−18, 0, 600) mm.
4. Project: u_ir = 645 × (−18/600) + 640 = −19.35 + 640 = 620.65, and v_ir = 645 × (0/600) + 360 = 360.0.

The color centre lands on infrared pixel (620.65, 360.0), a shift of 19.35 px. Cross-check with the shortcut formula, shift ≈ fx_ir × |t_x| / Z = 645 × 18 / 600 = 19.35 px.

Depth sets the shift, and it scales as 1/Z: the same centre pixel maps 38.7 px away at Z = 300 mm and 9.7 px away at Z = 1,200 mm. No constant pixel offset can follow that curve, which is exactly why the depth term exists.

Choosing Z: per pixel, once, or from a plane

Three ways to source Z, in descending accuracy:

• Per pixel, from aligned depth. Run the depth-to-color alignment first and read the depth under each color pixel. The mapping is then limited only by the module's calibration quality — expect residuals near one to two pixels on a factory unit.
• One distance for the scene. When the part sits at a fixed stand-off, set Z to that distance and build the maps once. The error term is fx_ir × |t| × ΔZ / Z²: with the example numbers, a 50 mm distance error at 600 mm costs 645 × 18 × 50 / 600² ≈ 1.6 px, and 100 mm costs about 3.2 px.
• A fitted plane. For a flat scene at unknown distance, fit a plane to the point cloud and intersect each color ray with it. Exact on the plane, wrong wherever the scene leaves it.

The 1/Z scaling cuts both ways: the same 50 mm error at 1,200 mm costs 645 × 18 × 50 / 1,200² ≈ 0.4 px. Coarse depth is forgivable at range and expensive close in.

Verify it on the bench

Set a printed checkerboard, or any high-contrast flat target, at the working distance. Run the mapping, toggle the remapped infrared image over the color image, and the edges should hold still within one to two pixels. Then make the result earn trust:

• Distance scaling. Repeat at half the distance; the offset should roughly double. If it does not, the Z source or the extrinsic direction is wrong.
• Both channels. Map left and right infrared with their own extrinsics; each lands at its own offset, and copying one to the other doubles the error.
• Synchronization. The streams must be time-synchronized before mapping, or a moving target smears the overlay even with perfect calibration.

A constant few-pixel residual that survives all three checks is a registration problem, not a projection bug; the rectified-stream offset fix covers that case.

Mistakes that cost hours

• Inverted extrinsic. The overlay drifts opposite to the lens offset: you applied color-to-infrared where the inverse was needed. Use P_rgb = Rᵀ (P_ir − t).
• Mixed units. A translation in millimetres fed to a metric point cloud is a 1,000× error. Pick millimetres and convert once, at the top of the pipeline.
• Skipped undistortion. Non-zero color coefficients left unused bend the overlay toward the frame edges; run the undistort step whenever they are not all zero.
• Nearest-neighbour sampling. The map lands on fractional pixels, and nearest neighbour steps every edge by a full pixel. Bilinear keeps the edges where the math put them.
• One extrinsic for both infrared channels. Left and right sit at different mechanical positions, and each carries its own pair calibration; project the streams separately.
• Resolution mismatch. The printed intrinsics belong to one stream resolution. Running 640 × 360 from a 1280 × 720 calibration means scaling fx, fy, cx and cy all by 0.5 before projecting.

Picking the hardware

Nothing in the projection cares which camera produced the streams, only that color and infrared arrive together with their calibration exposed. The depth side comes from the 3D sensors range. When the application needs a color channel beyond the module's own, an area scan camera bolted beside it works the same way after one stereo calibration of the pair; the residual targets to hit are in the calibration residuals guide.

Examples from the 3D sensor range:

The decision, case by case

If only the left infrared channel matters and your module registers depth to it pixel-for-pixel, the depth-to-color align output is already your mapping — done. For the right infrared channel, for modules without that registration, or for an external color camera, run the five-step projection with per-pixel aligned depth; that is the accurate default. Reserve the fixed-distance shortcut for flat scenes at a known stand-off, accepting about 1.6 px per 50 mm of distance error at 600 mm with the example's numbers.

To choose the camera itself, the camera finder filters the range by interface, resolution and sensor, and the configurator below walks a full station into a parts list. More depth-camera reading sits on the 3D vision topic page, and the contact page reaches the applications team when a specific module needs a specific answer.