Rectify color stereo images yourself with a remap and calibration

You built a stereo rig with color sensors, opened the rectified pair the depth module outputs, and both views came back grayscale. Nothing is broken. Stereo matching scores pixel intensity, so the module converts every frame to one channel before it rectifies and matches. The calibration underneath is pure geometry, though, and geometry does not care how many channels a pixel carries. This guide shows you how to rectify the color stereo images yourself, on the device where the firmware allows it and on the host where it does not, and how to prove your color pair lands on the same rectified grid as the depth. You finish with a working remap, the mesh setting that stops block glitches, and a parity test that runs in one line.

Why does the stereo depth block only output grayscale rectified images?

Because stereo matching compares pixel intensity, the pipeline converts every frame to single-channel grayscale before rectification and matching, so the rectified outputs are mono even when the sensors are color.

The matcher, whether block matching or semi-global matching, scores candidate correspondences by intensity difference inside a window. Three channels would triple that work and add nothing to the geometry, so depth APIs of this class hand back grayscale rectified images even when you attach color cameras. Two consequences follow. The rectified mono pair is a by-product of the depth stage, free and always in register with the disparity map. And the color you want never enters the matcher at all: it lives in a parallel stream, either from your sensors before conversion or from separate mono sensors the module actually matches on.

If you are still choosing hardware, the 3D sensors category puts the depth side in a single housing:

Can the device rectify the color pair itself?

Yes, when the module's firmware lets a warp node consume a rectification mesh: build the mesh from the same calibration the depth pipeline uses and set meshCellSize = 20, because 16 px cells leave block-shaped glitches in the warped frame.

The warp runs as a second branch beside the depth branch: color frames in, mesh-driven resample, rectified color out, depth untouched. The mesh is a coarse grid of source coordinates that the firmware interpolates between, which is what makes the on-device path fast and what makes the cell size matter. Treat 20 as the starting point; go finer only on a firmware release where the block artifact is gone. If your firmware exposes no mesh-consuming warp, the device cannot do this job, and the host path below is the answer with the same maps.

How do you rectify the color pair on the host?

Stream the raw color pair and the calibration off the module, then remap each frame through maps built from the rectification rotations and projection matrices, one resampling pass per frame, and the depth branch never knows about it.

1. Enable the raw, unrectified color streams for both sensors, the same outputs you would use to stream left and right stereo image pairs from a depth camera.
2. Read the calibration for the color pair: each sensor's camera matrix and distortion coefficients, plus R1, R2, P1 and P2. If the module only calibrates its mono pair, calibrate the color pair yourself as described in the next section.
3. At startup, build the two maps with initUndistortRectifyMap: the left map from the left sensor's matrix and distortion with R1 and P1, the right from the right sensor's with R2 and P2, at your chosen output size. Build once; the maps are constants.
4. Remap every frame as it arrives: each output pixel reads the source coordinate stored for it, interpolated between map entries. Rectifying live stereo video on the host is this same loop.
5. Pair the frames by device timestamp, not arrival order; the rule that fixes RGB and depth frame sync holds for left and right color as well.
6. Feed the rectified pair to whatever consumes color geometry. If that is your own matcher, stereo depth from host images covers exactly what to feed it.

# Pseudocode. initUndistortRectifyMap is a documented library call;
# every other name below is ours.

K_left  = None  # 3x3 float camera matrix, left color sensor, from calibration
D_left  = None  # float distortion coefficients, left color sensor
R1      = None  # 3x3 float rectification rotation, left view
P1      = None  # 3x4 float projection matrix, left rectified view
K_right = None  # same four values, right color sensor
D_right = None
R2      = None
P2      = None
out_size = (1280, 800)  # (width, height) in px: the output grid you choose

map_l_x, map_l_y = initUndistortRectifyMap(K_left, D_left, R1, P1, out_size)
map_r_x, map_r_y = initUndistortRectifyMap(K_right, D_right, R2, P2, out_size)

frame_l = None  # uint8 array, shape (800, 1280, 3): raw left BGR frame
frame_r = None  # uint8 array, shape (800, 1280, 3): raw right BGR frame

rect_l = resample(frame_l, map_l_x, map_l_y)  # resample = your toolkit's
rect_r = resample(frame_r, map_r_x, map_r_y)  # inverse-map remap call

Which intrinsics go into the rectification maps?

Use the raw-view calibration for what goes in and the rectified-view parameters for where it lands: each sensor's own camera matrix and distortion coefficients, combined with R1 and P1 for the left map, R2 and P2 for the right.

The map is an inverse mapping: for every pixel of the output grid it stores the coordinate to read in the raw frame, computed through the projection P, the rotation R, and back through the raw K and D. Feed it the raw sensor's K and D, because those describe the frames you are resampling. The rectified stream's intrinsics are a different object, and which intrinsics apply to rectified versus raw stereo images sorts that difference once. P1 and P2 also carry the rectified focal length and the baseline, so a pair rectified with them is consistent with the module's disparity output by construction.

When the color sensors are separate from the module's matching pair, calibrate them yourself. A stereo calibration with each sensor's intrinsics already known and only the extrinsics free is enough, and stereo calibration with fixed intrinsics works through that case. Shoot a target with sharp dots at a known pitch, such as the LS-LXR-0029 board with 0.5 mm dots across its 76 × 76 mm grid, from lens accessories:

How do you remap three color channels at once?

You hand the interleaved BGR frame to the same remap once: the map is pure geometry, all three channels resample with the same coefficients, and nothing can drift between channels.

The single-channel technique this comes from generalizes by changing the frame you pass in, not the maps: a three-channel array in, a three-channel array out. Where a toolkit only accepts single-channel input, split the frame, remap each plane with the same two maps, and merge. Three passes, one geometry, the same result. Use the library's undistort-rectify call rather than re-deriving the sampling by hand; the convention for where distortion is applied lives inside that call, and hand-rolled copies are the recurring source of small mismatches.

Order matters once: debayer before the remap. A Bayer pattern is a mosaic of color sites, and resampling it mixes sites that demosaicing later treats as neighbors, so demosaic first and remap the finished BGR.

How do you verify the color pair matches the depth geometry?

Convert your rectified color frame to grayscale and difference it against the module's grayscale rectified output from the same instant, and check that matched feature points share a row to within about half a pixel.

Test one, row parity: detect corner features in both rectified color frames and compare the vertical coordinate of each match.

u_l, v_l = 320.0, 240.0      # a matched corner in the left rectified frame, px
u_r, v_r = 288.4, 240.3      # the same physical corner in the right frame, px
row_error = abs(v_l - v_r)   # 0.3 px, passes a 1.0 px per-point gate

Average the error over a few dozen corners. Below 0.5 px on average, the pair satisfies the row constraint the module's own matcher works under. A systematic offset of a pixel or more means the map inputs are wrong; reread the warning two sections up. This is the same check used to verify recorded stereo frames are truly rectified before trusting downstream stereo, and row errors that survive into matching are the first place disparity map troubleshooting after rectification looks.

Test two, grayscale parity: take the module's rectified mono frame and your color frame converted to gray from the same timestamp, and subtract them. A mean absolute difference of one or two gray levels is demosaic and tone rounding, a pass. A difference that jumps across one region of the frame is a wrong map: usually the mono pair's calibration applied to color sensors, or R1 and R2 swapped.

What bandwidth does a rectified color pair cost?

At 1280 × 800 BGR and 30 fps, two cameras produce about 184 MB/s, close to 1.47 Gbit/s: past what one GigE link carries, comfortable inside USB3.

Walk the arithmetic: 1280 × 800 is 1,024,000 pixels; at three bytes of BGR that is 3,072,000 bytes per frame per camera; at 30 fps, 92,160,000 bytes per second per camera; across both cameras, 184,320,000 bytes per second. A single 1 Gbit/s GigE link sits below that figure, so on GigE you rectify on the device, halve the frame rate, or carry the color as mono. USB3 has the headroom. Interface bandwidth, cabling and distance are their own subject; the 5GBASE-T camera interface guide works one such link end to end.

The remap itself is one interpolated pass per frame. Budget it against the 33.3 ms frame period at 30 fps, not against the depth pipeline's budget, which it never touches.

When is rectified color the wrong answer?

When you only need color painted onto the depth map for an operator display, register the depth to the color stream instead: rectified color earns its bandwidth only when the color pair itself is the deliverable.

Registration projects depth points into the color camera's geometry through the extrinsics between them, and the color frames stay raw and untouched. Mapping RGB to depth, step by step covers the projection, and a fixed pixel offset after registering is the symptom the depth-to-RGB registration offset fix names the stream for. Choose rectified color when the consumer needs a true stereo color pair: your own matcher running on color, color features measured in rectified coordinates, or a recording format that stores the pair. Whether color and point cloud should travel together at all is worth deciding first, and combine or split RGB and depth capture lays out that trade.

To own the whole chain instead, build the pair from two matched area scan cameras and calibrate it yourself, so the rectification, the matching and the color are all yours:

The decision, one line per case: you need the color pair, so run a self-rectified color branch beside the depth; you need the overlay, so register depth to color; the link is too slow, so warp on the device at meshCellSize = 20. Size the cameras in the camera finder, start a system build at the camera step, and bring the module's calibration numbers to the contact page if parity will not close. More stereo and depth write-ups sit on the 3D topic page.