Depth-to-RGB alignment off? Run your own checkerboard calibration

You enable the aligned depth-in-color stream, overlay it on the RGB frame, and the texture edges sit a finger's width off the object. Re-running the vendor viewer on another PC moves the error with the camera, so the factory parameters shipped with your unit are the problem, not your code. The decision: measure the offset in pixels, calibrate the color intrinsics and the depth-to-color extrinsic yourself with a printed checkerboard, and register depth with your own values. This guide is for engineers running a stereo or structured-light depth camera with a separate color sensor - bin picking, dimensioning, overlay inspection. It gives the pose counts that constrain the solve, the arithmetic that turns a pixel offset into millimetres, and the checks that tell you the new parameters are good.

Measure the offset and rule out the simple fixes

Overlay the aligned depth stream, drawn as an edge image, on the RGB frame, and read the offset in pixels at a sharp vertical feature. Note the stand-off distance too - you need measurements at two distances before deciding anything.

Then run the two checks that cost nothing. First, update to the current SDK and repeat the overlay: one camera family has shipped depth-to-color translations in different unit scales across library versions - alignment only lined up after multiplying the reported translation by 10,000 on one build, and after the upgrade the same factor overshot in the opposite direction. If an update moves the offset, the factory calibration was never wrong and the defect lives in the library. Second, confirm both streams run at the resolutions your pipeline assumes: depth-to-color alignment breaks up when stream resolutions do not match, and a constant pixel offset in the rectified stream is its own, separately fixed fault.

With both ruled out, classify the error. Measure at 0.3 m and at 0.9 m: an offset that shrinks in proportion to distance points at the extrinsic translation; a constant pixel offset at both distances points at the color intrinsics, mainly the principal point.

What you need

• A printed checkerboard glued flat to aluminum or glass. An odd-by-even grid such as 9 x 6 inner corners works with every common library. Measure the printed square with calipers and use the measured value - printer scaling drifts the print from nominal, and the square size sets your translation units.
• Diffuse, even lighting with no specular hotspot on the board. Matte print, not photo paper.
• A rigid mount. Hand-held capture moves the baseline between cameras and shows up as a distance-dependent error.
• SDK access to raw color frames, raw depth, and the depth camera's factory intrinsics.
• Auto exposure and auto focus off, fixed for the whole session. Most chessboards that will not detect are lighting or focus problems, not corner-detection bugs, and a board with the wrong grid pitch biases every number downstream.

If you are also specifying hardware: structured-light cameras such as the MV-DPS200P-02 pair stereo depth with a separate color texture stream - the same two-camera geometry this procedure recalibrates.

Mount the camera once and do not move it until the last frame is captured. The LS-MNT-0040 provides a 700 mm column height and a 400 x 450 mm base. That height sweeps the board from about 0.3 m to 1.2 m without re-clamping, and the mass of the base keeps the rig still while you flip the board around.

Capture 10-20 good poses at each of 3-5 distances

1. Clamp the camera on the stand and lock exposure and focus. Nothing on the camera moves after frame one.
2. Start at the nearest distance, about 0.3 m. Hold the board tilted 20-45 degrees off-axis and capture 10-20 poses, walking the board so its corners visit every part of the frame, including near all four image corners.
3. Step back and repeat at two to four more stand-offs, up to the far edge of the working range - 3-5 distances in total. Give each distance its own board-to-frame fill: a target that fails at distance because it is too small drops out of the solve and quietly skews what remains.
4. Reject frames with blurred, clipped or misordered corner detection. Only accepted poses count toward the 10-20 per distance.
5. For the stereo step, capture the same board in the depth camera's own view - the left IR image on stereo modules - with the IR projector disabled through the SDK's emitter control for these frames. The solve uses the board, not depth.
6. Write down the measured square size in metres. Object points defined in metres make cv::stereoCalibrate return the translation T in metres.

Calibrate the color camera, then the stereo pair

1. Detect and refine corners on every color frame with cv::findChessboardCorners and cv::cornerSubPix.
2. Run cv::calibrateCamera on the color set. It returns the color intrinsics - fx, fy, cx, cy - and the distortion vector in OpenCV's order (k1, k2, p1, p2, k3). Expect a reprojection RMS around 0.3-0.5 px for a fixed-focus module; treat anything above about 1 px as a capture problem and drop the worst frames. The residual values worth aiming for are the same ones you would accept on any precision job.
3. Leave the depth camera's factory intrinsics in place, read straight from the SDK - read them, never guess them.
4. Detect the board in the depth camera's view and run cv::stereoCalibrate with both point sets, the depth view as the first camera and the color view as the second. It returns R, a 3 x 3 rotation, and T, a 3 x 1 translation, that map points from depth-camera coordinates into the color camera. A stereo RMS near or under 1 px is the target.

The projection the whole pipeline rests on is the pinhole model, and it is also the diagnostic:

A lateral translation error dX displaces a projected point by du pixels at depth Z. The arithmetic follows OpenCV's camera calibration tutorial.

Worked example: a 12 pixel edge offset at 0.60 m

Inputs:

• du = 12 px - the offset read between the aligned-depth edge and the RGB edge at a vertical feature (your overlay measurement).
• Z = 0.60 m - the stand-off where you measured it (your rig).
• fx = 600 px - the color camera's focal length in pixels from your calibration; a typical value for a 640 x 480 wide-angle color module, illustrative here - substitute your own.

Rule in words: a lateral translation error dX in the depth-to-color extrinsic displaces every projected point by du = fx * dX / Z pixels; inverted, dX = du * Z / fx.

Arithmetic:

dX = 12 px x 0.60 m / 600 px = 7.2 / 600 m = 0.012 m = 12 mm

Signature check: the same 12 mm error predicts du = 600 px x 0.012 m / 0.30 m = 24 px at 0.30 m, and du = 600 px x 0.012 m / 1.20 m = 6 px at 1.20 m.

Result: your 12 px offset at 0.60 m is exactly what a 12 mm translation error produces.

Decision: if your measurements at 0.30, 0.60 and 1.20 m land near 24, 12 and 6 px, the extrinsic translation is the fault and the stereoCalibrate replacement will remove it. If instead the offset stays near 12 px at all three distances, recalibrate the color intrinsics - a principal-point error moves pixels, not in proportion to distance. After recalibration, accept edge residuals within about 1-2 px at all three stand-offs; a residual that keeps one direction at the far distance means T still is not right.

Register depth with your own parameters

Do not write the OpenCV distortion vector into the SDK's own calibration fields: the two formats differ in coefficient order, meaning and translation units, and a pasted vector produces a different, cleaner-looking misalignment. Your values live in your code.

The registration loop is short. For each depth pixel with a valid depth Z: deproject the pixel through the depth camera's factory intrinsics to a 3-D point, apply R and T, and project the point through your new color intrinsics and distortion with cv::projectPoints. Because the intrinsics are fixed, precompute the result as a lookup map with cv::initUndistortRectifyMap and cv::remap, and the per-frame cost is one table lookup. A 640 x 480 map holding two float coordinates costs 640 x 480 x 2 x 4 bytes = 2.5 MB, built once per calibration.

Get the direction and the stream right - which camera's intrinsics deproject an aligned depth pixel, raw versus rectified frames in depth-to-color registration and aligning without losing depth precision each have their own failure modes. If your stack consumes registered topics, point a pipeline such as the ROS depth_image_proc package at the new parameters instead of writing the loop. More stream geometry lives on the 3D camera topic page.

Verify and version the result

Re-run the overlay at 0.30, 0.60 and 1.20 m. Accept the calibration when the mean edge offset is within about 1-2 px with no consistent direction; a direction bias at the far distance means T is still short, and an offset that grows toward the frame edges means the distortion terms do not match the lens.

Store the parameters as a file per camera serial number with the date, the square size and the stand-off list. The calibration belongs to the mechanical assembly, not the camera model: any bump, bracket swap or transport invalidates it, and a two-minute overlay re-check is faster than a bin pick that misses. For multi-camera rigs, fold each pair into one common frame with multi-camera extrinsic calibration in a shared coordinate frame.

Common mistakes

• Moving or re-clamping the camera between capture sessions - the extrinsic you solved describes a baseline that no longer exists.
• A board that is not flat. A warped print tilts every solution by a fraction of a degree.
• Defining squares in millimetres and reading T as metres. The units of T follow your object points.
• Calibrating at one distance only. The solve passes near and drifts far.
• Pasting OpenCV coefficients into vendor fields. Keep the formats apart and the projection in your own code.

When recalibration is the wrong tool

• The offset moved after an SDK update or after matching stream resolutions: the factory data was never wrong, so keep it.
• The mount flexes with temperature: a calibration locks in one thermal state, and focus drift with temperature shows how to tell a thermal shift from a parameter fault.
• The cameras move relative to each other by design: no static extrinsic applies, so register per frame or re-anchor the pair mechanically.
• Aligned color is only needed for display: a rough factory alignment may be good enough, and the bench hour is better spent elsewhere.

Measure first - the two-distance overlay takes fifteen minutes and tells you which fault you have. If it confirms stale factory parameters, the capture set above is about an hour on a bench and ends with a registration you can version alongside your code. When hardware is on the order list instead, start at the 3D sensors category or the camera finder, put the rig on one of the camera stands, and if the residuals will not settle, send the overlay images through the contact page.