Map RGB to depth: color-to-depth projection, step by step

Your color image and your depth image look at the same scene from two different pinholes, so identical pixel coordinates point at different surfaces. Mapping RGB to depth answers the question the built-in depth-to-color path never asks: for this color pixel, how far away is the surface behind it? The mapping runs in three steps: unrectify the color pixel into a ray, carry the ray into the depth frame with the color-to-depth extrinsic, then intersect it with the depth map and read the value. Everything below runs offline from a single synchronized pair: the formulas, a 60 mm-baseline worked example with the arithmetic shown, the intrinsics mistakes that shift results by tens of pixels, and the checks that prove the map before it feeds a robot.

Why RGB and depth never line up on their own

Depth cameras and color cameras are two separate optical systems even when they share one housing. Their sensors sit tens of millimetres apart behind different lenses, so one scene point lands on different pixels in the two images. The horizontal displacement follows the pinhole rule:

with fx_d the depth camera's focal length in pixels, b the baseline between the two lens centers and Z the stand-off. A rig with fx_d = 800 px and b = 60 mm splits the same surface by:

Skip the transform and every color pixel carries exactly that error. The two fields of view also differ, which shifts pixels even at zero baseline; the depth and color field of view mismatch guide covers that separate effect. SDKs ship the easy direction, projecting depth into color to colorize a cloud; the deprojection calculation guide shows that math. Color into depth you write yourself, and the same three steps do it.

The projection, in three steps

Every color pixel needs one ray, one rigid-body transform and one depth lookup. The depth map is never resampled; you only find which of its pixels sits behind each color pixel.

Step 1: color pixel to ray

For (u, v) = (940, 460) with fx_c = 1000 px, fy_c = 1000 px, cx_c = 640 px and cy_c = 360 px, the ray is ((940 - 640)/1000, (460 - 360)/1000, 1) = (0.30, 0.10, 1). If the calibration lists distortion coefficients, undistort the pixel first with the model those coefficients belong to; the library call takes the K matrix and the coefficient vector. Machine-vision lenses sit near pinhole behavior, typically under one percent of distortion, while a fisheye is far beyond that and the undistort step is not optional.

Step 2: ray into the depth frame

R rotates the color frame onto the depth frame and t places the color camera's center in the depth frame, in millimetres. A ray from the color camera reads P_c = s × d_c, so the transformed ray is P_d(s) = s × (R × d_c) + t: direction R × d_c, origin t. Swap the direction by mistake and the error returns as the baseline shift from the table above, 46 px at 1050 mm here. The optical-frame and camera-frame transform guide covers the convention traps, and the two-depth-camera calibration guide shows how R and t are solved on a bench.

Step 3: intersect with the depth map

The depth is unknown until you sample, so the intersection iterates:

Start s at the middle of your working range, project the ray point into the depth image, read z_sample and recompute s. Each pass costs one depth lookup; two passes converge on flat and gently curved scenes, three or four on stepped parts. Read z_sample in the map's stored scale, most often one millimetre per 16-bit count; the depth-to-real-distance calibration guide shows how to confirm the scale before you trust it. A sample of zero or an invalid flag means no surface sits behind that pixel, so drop it and flag the color pixel; the deprojection-zero diagnosis guide lists the causes. Round (u_d, v_d) once, at the end.

Which intrinsics go with which image

K must describe the exact pixels you feed it. A raw frame keeps its distortion model; a rectified or undistorted frame gets new intrinsics, because rectification moves the focal length and the principal point as it straightens the rays, and the raw K no longer fits. Resizing scales fx, fy, cx and cy by the width ratio: feed 1920 px intrinsics to a frame resized to 960 and every projection lands twice as far from the center as it belongs. Cropping shifts the principal point by the crop offset and changes nothing else.

A factory-calibrated device publishes a K, a distortion vector and a color-to-depth extrinsic per stream pairing, so take the set that matches your recorded resolution. Modern tools can solve intrinsics and extrinsics jointly, even without a target, but a flat chessboard shot from ten to twenty poses gives a bench answer you can check: a sound solve reprojects the corners well under one pixel.

Worked example: 60 mm baseline, 1.05 m stand-off

Setup. Color camera: fx_c = 1000 px, fy_c = 1000 px, cx_c = 640 px, cy_c = 360 px, a 1280 × 720 frame with distortion under one percent. Depth camera: fx_d = 800 px, fy_d = 800 px, cx_d = 640 px, cy_d = 360 px, a 1280 × 720 map stored in millimetres. Extrinsic: R = identity, t = (-60, 0, 0) mm, the color center sitting 60 mm off the depth axis. Color pixel to map: (u, v) = (940, 460).

Ray: d_c = ((940 - 640)/1000, (460 - 360)/1000, 1) = (0.30, 0.10, 1).

Pass 1, s = 1000 mm: P_d = (0.30 × 1000 - 60, 0.10 × 1000, 1000) = (240, 100, 1000) mm, so u_d = 800 × 240/1000 + 640 = 832 and v_d = 800 × 100/1000 + 360 = 440. The depth at (832, 440) reads 1050 mm.

Pass 2, s = 1050 mm — Z on this ray equals s, because (R × d_c)_z = 1 and t_z = 0: P_d = (0.30 × 1050 - 60, 0.10 × 1050, 1050) = (255, 105, 1050) mm, so u_d = 800 × 255/1050 + 640 = 834.3 and v_d = 800 × 105/1050 + 360 = 440. The depth at (834, 440) reads 1051 mm.

Pass 3, s = 1051 mm projects to (834.3, 440.1), which rounds to the same pixel, and the depth still reads 1051 mm: converged.

Result: color pixel (940, 460) maps to depth pixel (834, 440) at 1051 mm. At that stand-off one depth pixel covers 1051/800 = 1.31 mm of surface and one color pixel covers 1051/1000 = 1.05 mm, so each depth pixel carries 1.31² / 1.05² ≈ 1.6 color pixels; the depth map is the coarser master grid, as it should be. The baseline check closes it: drop the extrinsic and the answer moves by 800 × 60/1050 = 46 px, the exact error the transform removes.

Aligned stream or offline mapping

SDKs ship the depth-to-color direction as a ready call: every depth pixel is projected into the color image and resampled onto a color-grid copy of the depth map. That keeps the color image as master and interpolates every depth pixel. Mapping color into depth keeps the depth map as master instead: no depth pixel is interpolated and the output grid stays native for robot guidance or volume work. The math is identical and only the starting image changes; the depth to color alignment precision guide quantifies what the resample costs in the other direction.

Parts that carry the two streams

Two hardware patterns produce the synchronized pair. A factory-calibrated 3D device with a second sensor prints the intrinsics and the color-to-depth extrinsic for you: browse the 3D sensors category for stereo, structured-light and ToF families such as the MV-DPS200P-02 projected structured-light 3D camera and the MV-DT01WMU ToF sensing camera.

For the color half, mount a color area scan camera rigidly to the depth sensor and calibrate the pair once. The MV-CS200-10GC is a 20 MP color GigE camera with a 1-inch sensor in C-mount, and a short chessboard session on the bench gives the fx, fy, cx and cy the ray formula consumes. Fire both cameras from one trigger source so the pair stays synchronized within the exposure.

Checks that prove the map

Run four checks before the mapping feeds anything downstream.

• Round trip. Project a depth pixel into color and back; start and end must agree within one pixel. A constant offset near the baseline shift means the extrinsic is inverted — the inverse of P_d = R × P_c + t is P_c = Rᵀ (P_d - t).
• Flat wall. Aim at a plain surface at a known stand-off, say 1000 mm; the mapped depth across the color image must sit within the sensor's depth noise of 1000 mm.
• Edge overlay. Blend color edges onto the depth map; silhouettes must land within one to two pixels, the depth-pixel footprint at your stand-off, 1.31 mm in the worked example. The 2D overlay on the depth map guide gives the projection you need.
• Calibration residual. Accept a solve that reprojects under 0.5 px; above one pixel, remount and reshoot.

Where the mapping breaks

Occlusion comes first: the color camera sees surfaces the depth camera cannot, so those rays sample a hole or a background value — drop them and flag the pixel. Depth edges come second: a footprint straddling two surfaces returns a mixed value, which shows up as a thin band of wrong color along every silhouette. Surface physics comes third: dark, transparent or mirror-like patches return no depth, and no transform recovers it. Motion adds a fourth on live rigs: capture the pair on a common trigger, or the scene moves between the two exposures and the parallax itself shifts.

The decision, per case

Both streams from one factory-calibrated device and you want depth as the output grid: write the three-step loop, run two iterations, and expect pixel-level agreement on flat scenes. You want depth on the color grid instead: take the SDK's built-in aligned-depth call, because that direction starts from depth and is already solved for you. Building your own rig: a factory-calibrated depth sensor, a color camera on a rigid bracket, one chessboard solve to under 0.5 px residual, then the same loop. To pick the hardware, the camera finder filters by interface and resolution, the configurator assembles the chain, and the contact page reaches the applications team if you want the extrinsic plan checked before ordering. The 3D topic page collects the rest of the depth-map guides.