Depth to color registration: raw versus rectified frames

A depth overlay can look almost correct while remaining several pixels wrong around every edge. Before changing calibration values, check whether the images, intrinsics and transforms describe the same optical frames. A rectified image represents rays in a virtual camera frame; it is not merely a cleaner raw image. This troubleshooting procedure helps you separate metadata mismatches, omitted rotation, reversed transforms and ordinary visibility effects without compensating for one error with another.

1. Start with a stationary, repeatable symptom

Use a stationary target containing a flat surface, visible corners and a foreground object. Keep the cameras and target fixed, then save the depth image, color image, calibration metadata and transform chain from the same acquisition configuration.

Inspect several locations across the image rather than one central point. A central feature can align even when focal length, distortion or rotation is wrong elsewhere.

These patterns suggest tests; they do not uniquely identify a cause. For example, a principal-point error can resemble a small rotation near the image center.

If the stationary test passes but moving targets fail, follow the depth and color timestamp checks before changing geometry.

2. Build a ledger for every image and optical frame

Write down what each stream actually contains. Labels such as “color” or “registered” are not enough to identify its geometry.

Raw and rectified streams can have different intrinsics even when their dimensions match. Rectification may change focal length, principal point, usable field of view and optical-frame orientation.

The rectified-image intrinsics guide explains why a processed image needs its corresponding camera model. The depth and texture intrinsic conversion guide addresses differing image grids.

Do not assume a displayed image has the same pixel coordinates as the acquisition buffer. If an application rotates images, check the image rotation and acquisition guide and document the resulting coordinate mapping.

3. Verify the basic projection before adding rectification

Define one transform convention and keep it throughout the investigation. Here, the transform from depth optical coordinates to color optical coordinates is:

Here, X_d and X_c are column vectors representing the same physical point. The translation t_cd is expressed in the destination color frame and must use the same length unit as the point coordinates.

For an optical-axis depth measurement, first recover the normalized depth ray. For a raw distorted pixel, use the documented inverse-distortion operation. For an undistorted pinhole image with matching intrinsics:

This expression assumes the final component of q_d is one and Z_d is depth along that optical frame's forward axis. A radial range measurement requires a different scaling:

Transform the point, reject points behind the destination camera, and project into the destination image. For an undistorted pinhole color image:

The coordinate symbols in these equations refer to components of the transformed point. If the destination image is raw and distorted, apply its documented forward-distortion model before converting normalized coordinates to pixels.

A remapped depth image needs particular care. Moving depth samples onto a rectified grid does not automatically convert their values into depth along the new optical axis. Use the processed stream's documented depth definition or reconstruct and transform the original points.

The pixel-to-point mapping guide provides additional checks for indexing, invalid depth and pixel correspondence.

4. Account for rectification rotation exactly once

Rectification rotates image rays into a virtual optical frame and resamples the image onto a chosen pixel grid. That rotation must agree with the frame used by the projection pipeline.

Define the rectification rotations explicitly:

The subscripts dr and cr indicate rectified depth and rectified color frames. Substituting these definitions into the physical depth-to-color transform gives:

This assumes the rectified frames retain their respective camera origins. If processing defines a different virtual origin, use the complete documented rigid transform instead.

Three mistakes are common:

• Omitted rotation: raw depth points receive translation only, although the physical camera axes are not parallel.
• Double-applied rotation: points are already expressed in a rectified frame, but the raw-frame rotation is applied again.
• Mixed destination: points are projected using rectified intrinsics, then drawn onto the raw color image.

An identity relative rotation can be correct when the published rectified optical frames share the same orientation. It is not justified merely because a stream is called rectified.

Estimate whether the missing rotation matters

For a small angular error near the optical axis, pixel displacement is approximately focal length in pixels multiplied by angle in radians.

These are illustrative application values, not product specifications. A rotation that looks small in a matrix can therefore create an obvious overlay error.

For comparison, assume parallel cameras with a horizontal baseline of 0.05 m and a horizontal focal length of 600 pixels:

This simplified example shows why distance changes help distinguish translation-related effects from small rotational errors. Real systems can contain both.

5. Check transform direction, axes and matrix handling

A calibration file may describe color-to-depth geometry while your code needs depth-to-color geometry. Invert the rigid transform correctly:

Negating translation alone is not the inverse unless the relevant rotation makes that simplification valid.

Next, separate optical coordinates from mechanical or robot-base coordinates. An optical convention commonly places the forward axis along the viewing direction, while a mechanical frame may use different axes. An axis permutation or sign change must be applied consistently to the full transform, not just its translation.

Inspect matrix storage and multiplication conventions. Reading column-major data as row-major data can effectively transpose a rotation. Code written for row vectors also requires a different multiplication arrangement from the column-vector equations above.

Useful mathematical checks include:

• transpose(R) R should be approximately the identity matrix.
• The determinant of a proper rotation should be approximately positive one.
• Forward transformation followed by its inverse should recover the original point within numerical precision.

These tests catch implementation mistakes, but they cannot prove that a valid rotation belongs to the correct pair of frames.

6. Verify correspondence without mistaking occlusion for error

Use a sparse set of valid depth points first. For each point, record the source pixel, measured depth, reconstructed point, transformed point and projected destination pixel. This makes the first incorrect stage visible.

Check the following in order:

1. Confirm the depth sample belongs to the expected physical surface.
2. Confirm the reconstructed point has plausible units and forward distance.
3. Confirm the transformed point remains plausible in the color frame.
4. Confirm the destination pixel uses the correct image dimensions and calibration.
5. Compare the result with a visible feature away from depth discontinuities.

When multiple depth points project into one color pixel, retain the closest visible point using a destination-depth test. Otherwise, a background sample can overwrite a foreground sample and create a false registration defect.

Some color-visible surfaces are hidden from the depth camera, and some depth-visible surfaces are hidden from the color camera. Those missing correspondences cannot be corrected by changing rotation.

Use the single-stream edge overlay procedure to distinguish an overlay construction problem from cross-camera alignment. Once the coordinate chain is consistent, the extrinsic calibration accuracy guide helps evaluate remaining physical calibration error.

7. When the apparent fix is the wrong fix

Switching to a rectified color stream can improve an overlay because that stream matches the geometry your code already assumes. That does not establish that translation-only registration is valid for every mode or image profile.

Similarly, adding an arbitrary image shift may hide an error at one distance while making another distance worse. Avoid fitting compensation offsets until the frame ledger and projection chain agree.

Other limits need separate treatment:

• A stereo projection matrix may contain a baseline-related term in its fourth column. Follow its documented meaning rather than applying that translation again.
• Depth already aligned to color must not automatically pass through the original depth-to-color transform again.
• Cropping and resizing change the pixel camera model, but do not necessarily represent a physical camera rotation.
• Transparent, reflective or poorly measured surfaces can produce incorrect depth even with correct registration mathematics.

If switching profiles changes the result, compare complete metadata sets rather than copying only the focal lengths.

8. Check integration requirements before ordering hardware

For a new system, evaluate geometry access as carefully as sensor resolution. Browse 3D sensors for depth acquisition and area scan cameras when a separate color view is required. Use the camera finder to narrow the hardware options.

The MV-DPS200P-02 is a projected structured-light camera.

The MV-CU020-90GC is a color area scan camera with a global shutter.

These examples identify different acquisition roles, not a confirmed compatible registration pair. The catalog descriptions alone do not establish shared timing, cross-camera calibration or access to rectification transforms.

Before selecting a pair, confirm that the documented software interface exposes the required stream intrinsics, distortion models, depth convention and coordinate transforms. Separate cameras also require a stable mount, cross-camera calibration and a suitable timing strategy. A global shutter does not itself synchronize two devices.

9. Prevent the mismatch from returning

Keep a short acceptance checklist with the acquisition configuration:

• Record active image profiles and retrieve their matching calibration metadata.
• Label raw, undistorted, rectified and aligned outputs separately.
• State the source and destination of every transform.
• Record translation units and the meaning of depth values.
• Keep image processing and coordinate-frame processing in the same configuration record.
• Apply each rectification rotation and physical transform exactly once.
• Retain static reference captures at near, middle and far working distances.
• Repeat registration checks after profile, firmware, software or mounting changes.

Save intermediate numerical results alongside the final overlay. A small set of traceable points is more useful for regression testing than a screenshot alone.

For a new build, start with the camera configuration step. For an existing offset, send the frame ledger, active profiles and representative static captures through application support so the next decision starts from a defined coordinate chain.