Depth to RGB registration offset in pixels: fix the rectified stream

You publish a registered point cloud and the colors sit visibly off the depth surface — fingers held horizontal come out mostly white, a box edge takes on the color of the surface behind it — while the vendor's bundled pointcloud example looks clean on the same scene. The offset is real, constant at roughly 9 pixels, and it is not the sensor. What follows is the diagnosis in the order that finds the cause fastest: measure the offset and how it scales with distance, decide which camera's calibration is wrong, find the stream the pipeline actually used, then switch the registration to the rectified color stream. The arithmetic needs a checkerboard, a tape measure and about ten minutes, and it separates a software mix-up from a hardware fault before anyone orders parts.

Why does your registered point cloud have a color offset of about 9 pixels?

Because the registration colored the depth points through the unrectified color camera's calibration while setting the rotation between color and depth to identity. That leaves a constant shift of roughly 9 pixels which averaging, filtering and recalibrating the depth side will never remove.

The signature is specific. Colors bleed across every depth discontinuity along one image axis — here the rows — and the bleed is the same size at 0.5 m and at 2 m. The geometry of the cloud stays correct: planes are flat, edges are sharp, only the color lookup lands in the wrong place. Correct geometry plus a constant pixel-domain color shift points at the calibration constants in the registration chain, not at optics or depth quality. One production ROS driver carried exactly this mix: the registration nodelet read the plain color topic while every rotation constant was hard-coded to identity.

How do you measure the offset before changing anything?

Put a straight high-contrast edge in front of the camera. Read the gap between the depth step and the color boundary in color-image pixels, and repeat at 0.5, 1.0 and 2.0 m — the failing case reads about 9 pixels at all three distances.

Stand a checkerboard, or a white card on black cloth, square to the camera. Project the registered cloud back into the color image (most viewers and SDKs do this), and count pixels between the depth step and the color boundary. Record the number at the image centre and near a corner, on a horizontal edge and on a vertical one. Then read the behaviour:

The failing case reads 9 px at all three distances, which convicts the rotation term. With the numbers:

A 640 × 480 color stream with a 56° horizontal field of view has a focal length near 600 px, so the unapplied rotation is about 0.86°. Cross-check the scale: a misapplied 70 mm stereo baseline at 1 m would shift pixels by 600 × 0.070 / 1.0 = 42 px, about five times the observed offset. Nine constant pixels is rotation-sized, not translation-sized.

On a discrete rig built from a machine-vision camera the focal length in pixels is much larger, and the same 9 px means a smaller angle: a 4 mm lens over 2.2 µm pixels gives 4000 / 2.2 ≈ 1818 px, so 9 px = 9 / 1818 = 0.005 rad = 0.28°. The method does not change.

Which camera is wrong: the depth sensor or the color camera?

The color calibration path, in nearly every case with these numbers: the depth geometry is right, the vendor's own projection example is clean, and only the driver's constants for color are mixed up — prove it before you order parts or a return.

Run three checks in this order:

1. Run the maker's own pointcloud example on the same scene. A clean reading means the calibration stored in the device is intact and the fault is downstream.
2. Back-project the depth cloud and project it into the color image with the SDK's projection call. If the offset appears there too, the device's stored color calibration is suspect. If it appears only in your published cloud, the driver's constants are the mismatch.
3. Print both calibration sets the driver can hold for color: the unrectified set, with non-zero distortion and a real rotation to depth, and the rectified set, with zero distortion and identity rotation. If the registration path loads the unrectified values while the rotation stays identity, the bug is found.

The depth side almost never fails this way: a depth fault moves geometry, drops ranges or bends planes. A color-side calibration mix leaves geometry untouched and displaces color, which is what you see. For the same family of stream-level checks, see is the depth image aligned to RGB?.

Why does mixing rectified and unrectified calibration cause the offset?

Rectification rotates the color camera's image plane until its rows align with the depth frame. The rectified color stream therefore carries identity rotation by construction, and feeding the unrectified matrix into that same chain leaves the physical rotation — here 0.86° — unapplied.

Every RGB-D pair gets rectified: both images are remapped onto a common image plane with rows aligned and distortion removed, and the depth frame is normally the reference. The rectified color camera and the depth camera therefore share an image plane by design — the rotation between them is identity, and only the physical translation remains. The chain your registration runs:

Feed K_color from the unrectified stream into that chain and set R to identity, and two corrections vanish at once: the small physical rotation between the cameras, and the lens distortion. The rotation term costs f_px × θ pixels on every point — about 9 px here — at any distance, because a rotation does not care how far away the point is. What each stream contains is covered in depth to color registration: raw versus rectified frames, and the worked workflow in how to register RGB and depth images for a colored point cloud. If back-projection itself returns zeros, that is a separate failure — depth pixel deprojection returning invalid points.

How do you fix the offset in the registration pipeline?

Register against the rectified color stream: subscribe to the rectified topic, load that stream's intrinsics from the device, keep the physical translation, and set the rotation to identity — the measured offset drops from about 9 pixels to about one.

1. Subscribe to the rectified color stream — commonly published as image_rect_color — inside the registration node, and point every consumer downstream at it.
2. Query the device for that stream's camera matrix; its distortion coefficients are zero by construction.
3. Keep the depth-to-color translation the device reports, and set the rotation to identity. That is correct for the rectified stream only.
4. Back-project each valid depth pixel with the depth intrinsics, add the translation, and project with the rectified color matrix. Skip the undistortion step: the image is already rectified.
5. Republish, and re-measure on the same target at 0.5, 1.0 and 2.0 m.
6. If the driver is off-limits, rectify in your own node: remap the raw frame with the vendor's rotation using OpenCV's initUndistortRectifyMap, then run steps 3 to 5 there.

Go consistent, not half-way. Leaving the rest of the stack on the raw color topic works until someone republishes the wrong stream and the shift returns. When the color image must stay at full sensor resolution through this, full resolution color with a depth point cloud covers the mapping.

How do you verify the fix with a flat target?

Point at a checkerboard held square at 1 m: the color boundary must sit on the depth step within 1 to 2 pixels, at centre and corners, on horizontal and vertical edges alike, and the residual must not grow at 0.5 m or 2 m.

Check four things on the target: the near side and the far side of each edge, a horizontal and a vertical edge, the centre and a corner, and both working distances. Expect a systematic residual of one pixel or less, and one to two pixels of total edge bleed where depth noise smears the boundary. A residual that triples toward the corners means the distortion handling is still wrong — go back to the behaviour table.

What registration error should a healthy system show?

Plan on 0.1 to 0.5 pixels RMS reprojection error from a careful checkerboard calibration of one camera across 15 to 20 poses, and 1 to 2 pixels of depth-to-color edge bleed on a healthy unit — a constant 9 pixels is roughly twenty times that and is never noise.

Checkerboard practice sets the floor: 15 to 20 poses of a board, well spread in angle and distance, land a single camera's intrinsics at 0.1 to 0.5 px RMS reprojection error. Depth-to-color registration then adds 1 to 2 px of edge bleed from depth noise at discontinuities, and that is the healthy ceiling.

Convert the residual to part units for the buy-off. One pixel at 1 m with a 600 px focal length spans 1000 mm / 600 ≈ 1.7 mm on the target, so the unfixed 9-px offset paints color 15 mm from the surface it belongs to — enough to color the wrong face of a 10 mm feature. Two guides carry the numbers deeper: camera extrinsic calibration accuracy beyond reprojection error and stereo depth camera calibration: target capture and stream setup.

What should you check before buying a camera for registered color?

Buy a unit whose SDK publishes the calibration of every stream it offers, rectified and raw, with the rotation between them — and for a discrete rig, a global-shutter color camera, because rolling-shutter row skew on moving parts imitates this offset.

Ask two questions of any unit. Does the SDK expose the calibration of each stream it publishes, with the rotation between them? Is there a documented projection call you can test against a target on the bench? Integrated options in 3D sensors span the technologies — projected structured light like the MV-DPS200P-02, laser galvanometer stereo like the MV-DLS300P-04, and ToF sensing with the MV-DT01WMU:

For a discrete rig, pair the depth unit with a global-shutter color camera from area scan cameras so all rows expose together even when the part moves:

Rolling-shutter color adds a row-direction skew on anything moving, which mimics this bug and wastes an afternoon. Size the color sensor so the smallest edge you will grade spans at least three to four pixels:

When is the offset a hardware fault instead of a software mix-up?

When the rectified-stream fix changes nothing and the SDK's own projection example shows the same offset, the stored calibration is stale or wrong — update firmware and SDK together, recalibrate the depth-to-color pose yourself, and only then ask for a return.

Two real causes remain once the stream fix fails. First, firmware and SDK reading the stored calibration block in different layouts after an update: align both to one release before judging the hardware — the failure pattern is described in 3D camera header version mismatch: firmware and SDK alignment. Second, a unit that took a knock: the stored rotation is then wrong by degrees, the offset runs to tens of pixels, and straight lines in the color image look bent. Recalibrate the depth-to-color pose with a checkerboard seen by both sensors, or send the unit back.

Prevention checklist for the next commissioning:

• Run the maker's pointcloud example once at bench time and save the screenshot.
• Archive both color calibration sets next to the firmware version they came from.
• Re-measure the three-distance offset after every firmware or SDK update.
• Keep the whole stack on the rectified stream, publisher to consumer.

The camera finder filters by interface, shutter and resolution, and configure a system walks camera, lens and light in one pass. More RGB-D workflow pieces sit on the 3D vision topic page; if a live stream still shows the shift, send the before-and-after target shots and the driver version to the applications team.