How to register RGB depth images for a colored point cloud

Receiving separate color and depth images does not automatically provide a colored point cloud. You need a calibrated geometric relationship between the sensors, compatible coordinate conventions, and frames captured close enough together for the scene. This workflow is for engineers combining independently acquired streams. You will learn how to turn valid depth samples into spatial points, project those points into the color image, reject unreliable associations, and verify the result before using it for inspection.

Decide what output you actually need

A combined preview is not evidence that the same representation is available through an acquisition interface. Check the manufacturer's documented stream formats and supported examples before implementing your own registration.

Three outputs are commonly confused:

For a colored cloud, preserve the measured geometry and attach color to each visible point. You do not need to create a dense, color-sized depth image first.

The closest companion is the guide to raw versus rectified depth-to-color registration. Establish which image geometry you have before applying the equations below.

1. Collect calibration and stream definitions

Gather the following information from manufacturer-supported documentation, calibration exports, or a documented calibration procedure:

1. Depth-camera intrinsics, including distortion parameters and the associated image dimensions.
2. Color-camera intrinsics and distortion parameters for the active color mode.
3. Rotation and translation mapping the depth coordinate frame into the color coordinate frame.
4. Depth encoding, invalid-value rules, and the conversion from stored values to physical distance.
5. Capture timestamps, their clock domains, and any available synchronization mechanism.

Record the calibration's resolution, cropping, binning, and rectification settings. Changing an image mode can change the required intrinsics. A centered crop, for example, changes the principal point relative to the cropped image origin.

Do not replace distortion coefficients with zeros simply because an image looks reasonably straight. Residual errors often become most visible near the edges.

Also distinguish camera-axis depth from radial range. Axis depth measures distance along the optical axis; radial range measures distance along the viewing ray. The back-projection equations differ.

2. Pair frames and standardize units

Match images by capture time rather than the order in which acquisition calls return. Network transport, buffering, and processing can delay the streams differently.

If timestamps use independent clocks, establish their relationship before comparing them. Matching similar timestamp numbers is not sufficient when the clock origins or rates differ. The multi-camera calibration and synchronization workflow covers the broader pairing problem.

Estimate the effect of timing error before selecting a tolerance:

This estimate assumes approximately constant lateral motion. It is not a complete model for rotation, rolling-shutter readout, or motion during a depth acquisition sequence.

Convert depth and translation into the same physical units. If depth coordinates use millimeters while translation uses meters, the projection may look almost correct at some distances and fail badly elsewhere.

Start verification with a stationary target. Once static alignment passes, introduce motion and measure whether timing errors become visible.

3. Back-project depth into spatial points

The following equations assume rectified pinhole images, camera-axis depth, and calibration expressed in the corresponding rectified coordinate frames. For raw distorted images, use the documented distortion model to recover viewing rays, or use the manufacturer's supported deprojection function.

For each valid depth pixel, convert the stored value into physical depth and back-project it:

Here, fx_d and fy_d are focal lengths in pixels, while cx_d and cy_d locate the principal point. The pixel coordinates and intrinsics must describe the same image grid.

If the sensor reports radial range instead, normalize the viewing ray and multiply that unit vector by range. Do not substitute range directly for Z_d in these equations.

Reject invalid samples before creating points. Depending on the documented encoding, invalid measurements may appear as reserved values, nonfinite values, or samples rejected by a confidence mask. Do not interpret every zero or saturated value without checking its definition.

Next, transform the point into the color camera's coordinate frame:

R_cd and t_cd must map depth coordinates into color coordinates, not the reverse. The guide to optical-frame and camera-frame point transforms explains why frame names and axis directions need explicit treatment.

4. Project each point into the color image

For a rectified color image, project the transformed coordinates using its intrinsics:

Reject points with nonpositive Z_c, then check whether the projected location lies inside the color image. If sampling requires neighboring pixels, check the complete sampling footprint rather than only its center.

For an unrectified color image, apply the color camera's distortion model during projection. Do not project with rectified intrinsics and then sample an unrelated raw image.

Worked projection example

Assume a hypothetical rectified depth image with focal lengths of 600 pixels and principal point (320, 240). A valid sample at pixel (380, 270) has axis depth of 1,000 mm.

For this example only, assume identity rotation and a depth-to-color translation of (50, 0, 0) mm:

With color focal lengths of 1,000 pixels and principal point (640, 360), the projected location becomes:

The point can therefore receive color sampled around (790, 410), provided it passes visibility and image-boundary checks. These numbers illustrate the calculation; they are not specifications for any listed product.

5. Handle occlusion before assigning color

A projected pixel inside the image is not automatically a valid correspondence. The color camera may see a foreground surface where the depth camera measured a background surface.

Build a color-view depth buffer using transformed Z_c. When multiple depth samples project into the same output pixel, retain the nearest valid sample. This resolves competition among the available samples, but it cannot detect every occluder missing from the depth data.

Expect uncertainty around foreground edges, thin features, and gaps in depth coverage. Mark questionable points as uncolored rather than borrowing a nearby color indiscriminately.

Nearest-neighbor sampling provides a simple starting point for verification. Bilinear sampling produces smoother colors, but it can blend foreground and background colors across boundaries. Choose the method after checking edge behavior, not merely overall appearance.

Keep depth filtering separate from registration. Filling holes or smoothing boundaries changes the geometry that you are projecting. If filtering changes the image grid, verify the corresponding intrinsics; the article on depth filtering and camera intrinsics addresses that failure mode.

6. Build the cloud and verify it quantitatively

Keep geometry, color, and validity status together throughout processing:

1. Read a matched depth and color frame pair.
2. Reject invalid depth measurements and back-project the remaining samples.
3. Transform points into the color frame and project them into the color image.
4. Apply boundary and visibility checks, then sample color.
5. Store spatial coordinates, color values, and a color-valid flag.
6. Label the output coordinate frame and physical units explicitly.

You may retain coordinates in the original depth frame while using transformed coordinates for color sampling. Alternatively, publish the cloud in the color frame. Either choice is valid when the output metadata and downstream interpretation agree.

Use a rigid target with distinctive color features and depth boundaries. Inspect it near the center, near the image edges, and at several distances. A single centered target can hide calibration errors.

For independently identified corresponding landmarks, calculate pixel residuals:

Report the median, a high percentile such as the 95th percentile, and the maximum residual. Also record rejected-point counts. A visually tidy cloud can conceal excessive rejection.

Set acceptance limits from your task. A texture preview and a measurement tied to a narrow color boundary do not have the same tolerance.

Choose parts without assuming software compatibility

Start with the 3D sensors category for depth acquisition and the area scan cameras category when a separate color stream is required. The camera finder helps narrow acquisition choices, but registration requirements extend beyond interface and resolution.

The MV-DPS200P-02 is a projected structured-light 3D camera. Confirm its available depth representation, calibration access, and supported integration methods against your application requirements.

The LS-CMR-0166 is a USB3 area scan camera with a rolling shutter. Evaluate motion and frame-timing requirements before selecting it for a separate color stream.

These are component candidates, not a documented matched pair. The supplied catalog information does not establish shared triggering, factory camera-to-camera calibration, or a combined cloud interface.

Before ordering, establish who will calibrate the assembled rig and how calibration will be retained after mounting adjustments. More color pixels do not create additional measured depth samples; the guide to full-resolution color with a depth point cloud explains that distinction.

When the usual registration workflow is not enough

For a stationary scene and rigid sensor mounting, calibrated projection is often the appropriate starting point. Other conditions require additional work:

• Moving objects: improve capture synchronization before compensating through image processing.
• Changing sensor positions: update the camera-to-camera transform; a fixed calibration no longer represents the assembly.
• Limited shared field of view: accept partially colored geometry or change the sensor arrangement.
• Reflective or transparent surfaces: establish whether the depth measurements are trustworthy before evaluating color alignment.
• Persistent edge errors: inspect occlusion, distortion, and temporal mismatch separately instead of applying one global image offset.

If only a manufacturer's combined visualization is needed, a supported rendering interface may be simpler. If downstream software requires spatial coordinates and color values, request documented data access rather than extracting geometry from a preview.

Final checklist and next step

Before accepting the registration pipeline, confirm:

• Calibration matches the active image sizes, crops, and rectification settings.
• Depth encoding, scale, and invalid-value handling are documented.
• The transform direction and translation units are verified.
• Frame pairing uses meaningful capture timestamps or documented synchronization.
• Projected points pass positive-depth, image-boundary, and visibility checks.
• Output coordinates have a declared frame, and color validity is preserved.
• Static and moving-scene tests meet your task's error limits.

Use the camera configurator to organize acquisition choices, or contact the applications team with your image modes, timing requirements, calibration access, and sample data before selecting the sensor combination.