Depth-to-RGB rectification with your own calibration data

Factory depth-to-RGB registration tends to pass a center-frame glance and still bleed color across the edges of every object. If you need color and depth aligned to within a pixel or two — defect overlays, pick points on colored parts, pixel-accurate fused images — the stored calibration is not writable. The camera's built-in alignment applies it whether it fits your cell or not. The practical fix is to stop depending on it: keep the depth stream in the left imager's frame, stereo-calibrate the imager-to-RGB pair yourself, and run the registration on your own host with your own numbers. This how-to gives the calibration capture, the five-line projection, a worked example of what each calibration error costs in pixels, and the checks that prove the result before you ship it.

Why the built-in registration misses at the edges

A stereo depth camera computes depth in the frame of its left infrared imager. The on-sensor pipeline rectifies the left and right IR images against the factory IR-pair calibration, runs stereo matching, and emits a depth frame in which every pixel is a ray through the left imager's lens with a range attached. None of that pipeline knows an RGB sensor exists.

Registration happens afterwards, on the host: back-project each depth pixel, transform it into the RGB camera's frame, project it through the RGB lens. The rotation, translation and intrinsics it applies are the values stored at the factory — one measurement, per unit, of a board-level geometry that drifts with temperature, shock and lens tolerances.

The error hides on smooth surfaces and shows at boundaries. A rigid misregistration displaces every projected pixel by a similar amount. That reads as nothing on uniform material and as a color halo where a foreground object meets a differently colored background: an 8-pixel shift draws an 8-pixel halo along every silhouette. Distortion residuals also grow with image radius, so the stored-versus-real disagreement is worst in the frame corners — often exactly where the part sits. The same boundary signature appears inside stereo depth itself; the stereo depth edge-bias guide covers that variant.

If your acceptance criterion is color sitting on depth within 1-2 px at object boundaries, plan on your own registration from the start.

What you can and cannot replace

Split the pipeline at the depth frame.

Fixed on the sensor: the left/right IR rectification and the stereo matching. They run on the factory IR-pair calibration, and depth cameras of this kind offer no interface that injects your own numbers into that stage. You do not need one. Those stages only produce a range image in the left imager's frame, which is exactly the input your own registration consumes — asking the camera to build depth with your calibration is answering the wrong question.

Yours on the host: everything after the depth frame. The depth intrinsics, the RGB intrinsics and the depth-to-RGB rotation and translation are ordinary pinhole parameters, and the vendor's alignment function is just code that applies them. Re-implement it with your own values and nothing upstream changes: the depth stream stays untouched and stays comparable frame to frame.

Read the depth stream's intrinsics from the camera rather than reconstructing them; the depth camera intrinsics guide shows where they live. If your SDK offers several depth layouts — raw, aligned to RGB, color-registered — register from the native one and label your own output; the aligned depth streams guide maps each layout to the intrinsics that belong to it.

Calibrate the imager-to-RGB pair yourself

You need the camera with both streams live, a rigid flat target, a ruler for the target size, calipers for the baseline check, and your vision library's calibration module. The job is a stereo calibration between the camera that owns the depth frame — the left IR imager — and the RGB camera. Five steps:

1. Choose a rigid, flat target. A checkerboard or circle grid 200-400 mm across, mounted on plate or glass so it cannot flex. Board warp enters the solution directly as reprojection error.
2. Capture 20-30 synchronized pairs. Cover every quadrant of the frame, tilt the target up to about 45°, and include your working range — for a short-range pairing that means captures near 0.3 m, 0.5 m and 1.0 m. Both frames must see the same instant; a target that moves between the two exposures corrupts the translation.
3. Run the stereo solve. Feed the pairs to your library's stereo-calibration routine (the calib3d module in OpenCV is the common route), with each camera's own intrinsics as initial values and the target size measured from the board, not read from the print file. Keep the depth-side stream native — never run the SDK's alignment on it first.
4. Sanity-check the translation. Its magnitude should match the measured spacing between the two optical centers; on a compact module, expect caliper agreement within a millimeter or two. A translation off by a clean factor means the target size was entered wrong.
5. Accept on RMS. A good capture lands at 0.3-0.5 px RMS reprojection error. Above about 1 px, recapture with a flatter board and better tilt coverage instead of tuning anything else.

No left IR stream exposed by your SDK? Calibrate the RGB camera against planes and scene points measured in the depth data instead; the extrinsic calibration without a checkerboard guide gives the method.

The projection, in five lines

With the depth frame D(u, v) holding range Z in meters, the depth intrinsics (fx_d, fy_d, cx_d, cy_d) in pixels, the RGB intrinsics (fx_r, fy_r, cx_r, cy_r), and your rotation R and translation t from the depth frame to the RGB frame:

Keep the units straight and the math is hard to get wrong: f and c values are pixels, Z and t are meters — solve the calibration in meters and keep it there. Drop any point with Z' below zero, because it projects behind the RGB camera; drop or clamp points that land outside the RGB frame.

For a registered depth image, loop over all depth pixels and write each one's Z into the RGB pixel it lands on, then fill single-pixel holes from their neighbors. That loop is exactly what the built-in alignment computes, which hands you a free regression test: run both mappings over a grid of pixels and diff them. The differences are precisely the calibration disagreement you are correcting.

Two details bite people. The depth frame belongs to the rectified left imager, so use the rectified intrinsics the SDK reports for the depth stream — the rectified vs raw IR guide sorts out which parameters apply to which stream. And if you also want the color image drawn back onto the depth map, the same parameters run in reverse; the overlay 2D image on depth map guide walks that direction.

Worked example: a 0.5° extrinsic error at 0.5 m

Inputs

Depth and RGB streams of 1280×720 px with a 0.5 m working distance — the short-range pairing this sensor class is built for. Your calibration disagrees with the stored extrinsic by 0.5° of rotation and 1 mm of translation, magnitudes a knocked mount or a lens swap produces. One assumed input: the RGB lens covers 65° horizontally across its 1280 columns, because the answer scales directly with your real field of view. Question: how many pixels does the wrong registration move an edge at the frame border?

The rule

A rotation error about the imager's optical center swings the whole set of rays by its angle, so its pixel shift is f × θ and does not depend on range. A translation error δt moves projections by f × δt / Z and shrinks as the part moves away.

Arithmetic

The decision

Against a 1-2 px color-registration tolerance, an uncorrected 0.5° disagreement fails by a factor of five to ten — and no working distance rescues it, because the rotation term ignores Z. Even with translation solved perfectly, the residual is about 9 px. The scaling sets your budget: at this focal length every 0.1° of extrinsic disagreement is worth roughly 1.8 px, so a measured 2 px edge offset points to about 0.1° of rotation error, not a translation nudge. If your overlay check shows offsets of that size, building your own registration pays for itself within the first hour of data; if it stays under 1 px everywhere, keep the stored values.

Verify the registration on real data

• Flat-target sweep. A flat board with a strong color/depth boundary at 0.3 m, 0.5 m and 1.0 m: overlay your registered depth boundary on the RGB edges and measure the mean offset along all four frame sides as well as the center. A sound own-calibration lands near 1 px; several pixels means the calibration is wrong, not the projection code.
• Corner symmetry. Offsets that grow toward the corners are distortion residuals; offsets uniform in one direction are extrinsic. The pattern tells you which parameter to fix.
• Diff against the built-in mapping. Run the vendor alignment once on the same frame and subtract the two mappings. The difference field should match the f × θ prediction from the worked example; if it is under tolerance everywhere, skip your own pipeline and keep the built-in one.
• Live part. Confirm on the actual part, at the actual working distance, against the acceptance criterion you will ship.

For the device-side routes — forcing a re-registration or re-running the camera's own checkerboard check — see the depth to RGB registration offset guide and the depth-to-RGB alignment checkerboard guide. When those leave a residual at the boundaries, the host-side pipeline above is what removes it; the stereo camera RGB calibration guide covers the same pair from the RGB side.

When to keep the built-in alignment

Rebuilding registration costs a calibration session and a per-pixel loop, so spend it where it earns:

• Coarse overlays. If the registered pair feeds a human viewer or a coarse region of interest, 5-10 px of bleed is harmless and the built-in path costs nothing.
• Verified-good units. If the overlay check passes your tolerance on the cameras you already own, keep the built-in alignment — measured, not assumed.
• Host budget. A 1280×720 mapping is about 0.92 M projections per frame, roughly 27.6 M/s at 30 fps. That is small for a vectorized CPU loop, but the hole-filling pass is where the time goes; on a weak host, a GPU-accelerated built-in path can win. Measure before deciding.

The deciding number is your tolerance at the boundaries. Inside about 1-2 px across the whole frame, budget the half-day for your own calibration and registration; looser than that, use the camera's own pipeline. The 3D topic page collects the surrounding guides on stream layouts, rectification and calibration storage.

Sensors this workflow runs on

Before ordering, confirm two things in a sensor's SDK material: the depth stream publishes its intrinsics, and raw depth plus RGB can be pulled without forced alignment. Depth sensors of the classes this pipeline fits, from the 3D sensors category:

All three output depth in their own sensor frame, which is exactly the input the five-line projection consumes. One safety note: galvanometer-laser and structured-light sensors project strong illumination on purpose, so point the aperture only at the inspected volume and follow the laser-safety label on the housing.

For volume and logistics cells, where depth feeds dimensioning more than defect overlays, the same registration math carries the depth into any RGB camera you calibrate it against. The multi-DOE speckle structured-light 3D camera for volume measurement is that class in the catalog.

Put your working distance and tolerance into the camera finder and the configurator to size the sensor and the rest of the station. If the geometry or the tolerance is unusual, contact us with the part and the distance, and we will check the calibration approach against it before you commit.