Is depth image_raw aligned to RGB? Check registration

You have a depth stream on a ROS-style topic such as camera/depth/image_raw and an RGB stream beside it, and overlays of the two do not line up. Before you touch extrinsics, answer one question: is the depth already registered to the color camera, or is it still in the depth sensor's own frame? This article shows how to tell the two cases apart in a few minutes, with a projected-edge test and a UV-map check, and what to do in each case. It applies to integrated depth cameras such as the structured-light and ToF units in our 3D sensors range as well as to any depth-plus-RGB pair you have calibrated yourself.

What image_raw actually tells you

The suffix image_raw says nothing about registration. It is a transport convention: the frame arrives as raw pixel data rather than compressed or color-converted. Whether those pixels are in the depth sensor's frame or the color camera's frame is a separate property that the driver decides, and drivers differ.

Three arrangements exist in practice:

1. Unregistered by default. The driver publishes depth straight from the depth sensor and leaves the depth-to-color mapping to you. This is the most common case and the one that produces large, distance-dependent offsets.
2. Registered on request. The driver offers a registration parameter. Until you enable it, depth arrives in the depth sensor's frame even though the topic name looks the same.
3. Registered by default. Some drivers publish already-registered depth and provide the unregistered stream on a second topic, or not at all.

Check the driver's documentation first, then verify on your bench. Documentation tells you the intent; the test below tells you what is actually arriving.

The two-frame geometry in one formula

Registration is a reprojection. A depth pixel (u, v) with value z lives in the depth camera's frame. To paint it onto the color image you transform it into the color camera's frame and re-project:

Here K_d and K_c are the two intrinsic matrices and [R | t] is the depth-to-color extrinsic. If the driver has already applied this mapping, the depth image and the color image share the color camera's perspective, and you can index them with the same pixel coordinates. If it has not, every depth pixel corresponds to a slightly different viewing ray, and the disagreement grows away from the image center.

Our article on camera intrinsic parameter conversion covers how to keep the two intrinsic sets straight, which is the second most common source of confusion after registration itself.

The projected-edge test

This is the fastest reliable check. Point the camera at a scene with strong depth edges: a flat board on a patterned background, a box on a desk, a door frame.

1. Capture one depth frame and one color frame at rest, with matching timestamps. If the part moves between exposures, even a perfect registration will look wrong; our note on depth and color synchronization explains which stamps to compare.
2. Colorize the depth frame and overlay it on the RGB image without any mapping step, using the color image's dimensions for both.
3. Look at a vertical depth edge, such as the side of the box.

Interpretation:

• Edges within a few pixels. The stream is registered, or the sensor pair is so close that the baseline error is negligible. Small residuals mean calibration noise, not missing registration.
• Edges offset by tens of pixels, worse at the borders. The depth is in the depth sensor's own frame. Registration has not been applied.
• Offset roughly equal everywhere. Suspect a wrong principal point or a cropped image rather than registration.

The UV map check

The projected-edge test tells you whether registration happened. The UV map tells you how the driver maps depth to color, which you need before you fuse anything. A UV map is, for every depth pixel, the color-image coordinate that corresponds to it. Many SDKs expose it directly; if yours does, request it and inspect it:

• A UV map that is close to the identity (u' ≈ u, v' ≈ v) means the streams are already aligned, or the sensors share an optical axis.
• A UV map with a smooth, distance-dependent warp means the driver expects to deliver registered depth and the mapping is available to you.
• No UV map at all means you compute it yourself from K_d, K_c and the extrinsic, using the formula above.

If you build the mapping yourself, work per depth pixel, not per color pixel: registration is a forward warp from depth to color, and the reverse direction leaves holes. Our guide on raw versus rectified depth-to-color registration goes deeper into when to map before or after rectification.

Worked example: what offset to expect

Take a typical integrated depth camera with 60 mm between the depth and color sensors, viewing a target at 0.6 m with a color focal length of about 900 pixels (a rough figure for a 2 MP-class color sensor; use your own calibration values).

At 0.6 m the depth edge should land about 90 pixels away from the color edge if the stream is unregistered. At 1.8 m the same geometry gives about 30 pixels. So an overlay offset that shrinks from roughly 90 to roughly 30 pixels as you move the target back by a metre is the fingerprint of parallax, not of bad extrinsics. If instead you see a near-constant offset of 3-5 pixels, the stream is registered and the residual is calibration quality.

This is also why a registered stream matters for anything metric. If you read a bounding box in the color image and want the depth behind it, an unregistered depth stream makes you sample the wrong column of the scene by up to tens of pixels; the workflow in mapping image pixels to 3D points assumes registered or explicitly mapped frames.

Applying registration yourself

If the test shows an unregistered stream, apply the mapping once, at the source, and republish:

1. Get K_d, K_c and the depth-to-color extrinsic from the camera's factory calibration. For integrated cameras such as the projected structured-light 3D camera MV-DPS200P-02, check the manufacturer's documentation for the calibration-access procedure; never substitute nominal values.
2. For each depth pixel, compute the 3D point, transform it with [R | t], and project into the color image.
3. Keep the depth value attached to the depth pixel; write it into the output image at the projected color coordinate. Handle collisions by keeping the nearest depth.
4. Republish on a clearly named topic such as depth_registered/image_raw, and leave the original stream untouched for debugging.

If the driver or SDK offers a built-in registration mode, use it instead of rolling your own: it uses the factory values, handles sensor-specific quirks, and costs no calibration effort on your side.

For time-of-flight designs the same logic applies, with one extra caveat: ToF depth is a direct range measurement, so the 3D point comes from the range value along the pixel ray rather than from a stereo-style disparity. The ToF sensing camera MV-DT01WMU is an example of the type; the registration step is still the extrinsic transform plus reprojection above.

When extrinsics really are the problem

Extrinsics earn suspicion only after you have ruled out the cheaper explanations. Work down this list:

• Stream not registered (most likely, per the test above).
• Swapped or mixed intrinsics, for example color intrinsics applied to the depth image; see the intrinsic-conversion article linked earlier.
• Timestamp mismatch, where the geometry is fine but the frames describe different moments.
• Factory extrinsic error, typically 1-3 pixels of residual after registration.

Only if a careful per-pixel extrinsic fit beats the factory values consistently, and the residuals are systematic rather than random, is re-calibration worth the effort. For fused color-plus-geometry outputs such as the workflows in fusing depth, color and IR intensity into a point cloud, a residual of a pixel or two is usually invisible; chasing it costs more than it returns.

Choosing hardware with alignment in mind

If you are specifying a camera for a depth-plus-color task, prefer integrated units whose SDK performs depth-to-color registration in hardware or in the driver, over two separate cameras plus a hand calibration. Browse the 3D sensors category for integrated stereo, structured-light and ToF options, or the area scan cameras category if you will build the pair yourself and calibrate it. The camera finder filters both groups by interface and resolution, and the system configurator walks lens, cable and controller choices for the color side.

Quick checklist

• Confirm from the driver documentation whether depth is registered, and on which topic.
• Capture colorized-depth-over-RGB with matched timestamps at two distances.
• Offsets shrinking with distance: unregistered; near-constant few-pixel residual: registered.
• Request or compute the UV map before fusing color and depth.
• Apply registration once, at the source, and republish under an explicit name.
• Blame extrinsics only after intrinsics and timestamps are ruled out.

If the overlay test leaves any doubt, send us your two saved frames and calibration file through the contact page and we will help you read them. Expert support is included with every order.