Overlay 2D image on depth map: project the cloud back to pixels

A depth camera hands you two pictures of the same pallet: a depth map that knows how far every pixel sits, and a 2D image that shows what each pixel looks like. On their own they answer half a question. Combined, they let you show the cartons at one height in full colour and hand a clean, textured cloud to your picking logic. This guide settles the three decisions that stall the project: whether the two cameras need calibrating to each other, how to keep only the pixels inside a depth band such as 0.8 m to 1.2 m, and how to build the Open3D point cloud straight from NumPy arrays instead of writing and re-reading a .ply file every cycle. The worked example uses a registered 1920 × 1200 pair and a carton face at 1.0 m.

Do you need to calibrate the 2D camera to the depth camera?

No — not when the sensor is a factory-fused RGB-D unit. It stores the 2D image already registered to the depth map, so both share one pixel grid and the SDK's own intrinsics do any projection. You only calibrate when the colour camera is a separate device.

You can tell the cases apart in one capture. A registered pair has the same pixel count, arrives from a single capture call, and projects with the depth camera's intrinsics alone. The call you tried that returns a rotation and a translation between the images, CameraIntrinsics(), is an extrinsic: it exists for the separate-camera case, and applying it to a fused pair adds error rather than removing any. If your SDK ships an example named Mapping2DImageToDepthMap, run it once and keep its output as the reference overlay.

A colour stream from a separate area scan camera is the different case: you solve the extrinsics once on a checkerboard and bake them into the program. The RGB and depth registration guide covers that solve, and which intrinsics to deproject with settles the common mix-up of depth and colour intrinsic sets. An overlay that lands offset from the boxes is usually registration, not your maths — the depth to RGB offset fix walks that case end to end.

How do you keep only the 2D pixels at a chosen distance?

Threshold the registered depth map into a band mask and apply the mask to the 2D image: for a band of 0.8 m to 1.2 m stored as 16-bit millimetres, keep the pixels whose raw value lies between 800 and 1200 and black out the rest.

# depth: uint16, shape (1200, 1920), 1 mm per count, 0 = no return
# rgb:   uint8 BGR, same shape, from the same capture call
z_m = depth.astype(np.float32) * 0.001        # counts -> metres
band = (z_m >= 0.8) & (z_m <= 1.2)            # True inside the band
overlay = rgb.copy()
overlay[~band] = 0

The mask is two comparisons over one array, so it costs nothing per cycle. Pixels with no return read zero, fall below 0.8 m and drop out on their own. Put the band edges in the empty air between layers, never on a surface. With layers 250 mm apart, a 150 mm band centred on one layer top leaves about 175 mm of clear air (250 - 75) before the next layer, so a few millimetres of sensor noise cannot pull a neighbouring layer into the mask. From the band, a per-carton height is a short step — the segmentation mask average-depth method shows the robust way to read it back.

How do you project a 3D point back to a pixel?

Use the depth camera's pinhole intrinsics: u = f_x·X/Z + c_x and v = f_y·Y/Z + c_y, with Z in metres, and round each result to the nearest pixel before you sample the colour there.

Vectorise it rather than looping. With X, Y, Z as float arrays of shape (N,), compute u = (f_x * X / Z + c_x).astype(np.int32), then clip u to 0–1919 and v to 0–1199 before indexing. Guard Z: divide only where Z > 0, or the zero-depth holes throw infinities into the cast. The inverse direction — from a 2D box to the 3D points it covers — is the bounding box depth mapping problem.

How many millimetres does one pixel cover on the carton?

At depth Z a pixel spans Z / f_x laterally, so with f_x = 960 px a pixel covers 1.04 mm on a carton at 1.0 m, and a 600 mm carton face spans 576 pixels (0.600 × 960 / 1.000).

Every masking and edge error scales with that figure. At 2.0 m the same face spans 288 pixels, so both the band fringe and any calibration residual double in millimetres; mount the sensor as close as the field of view allows. Work backwards from the smallest feature you must resolve — a 20 mm gap between cartons needs better than 20 mm per pixel at the far layer — and let the camera resolution calculator turn it into a pixel count:

What are the steps from capture to finished overlay?

Five steps take you from one capture call to a colour overlay and a masked cloud, and everything after step one is NumPy arithmetic on arrays that already share a pixel grid.

1. Capture once with the SDK call named TexturedPointCloud() and keep the returned points_xyz_bgr, an (N, 6) array of X, Y, Z then B, G, R, casting with np.asarray(..., dtype=np.float32) if your build returns raw bytes.
2. Read f_x, f_y, c_x, c_y from the capture object, not from a constants file, so a lens or mount change cannot silently leave them stale.
3. Split xyz = points_xyz_bgr[:, 0:3], project to u, v with the formulas above, round and clip to the image bounds.
4. Build the band mask from the depth map (0.8 m to 1.2 m here) and black out overlay = rgb.copy() outside it. Keep cloud and band in one unit: if the cloud arrives in millimetres, divide Z by 1000 first.
5. If N matches the depth map's pixel count, reshape to (h, w, 6) as a cross-check — channels 3 to 6 are then a registered colour image to compare against your projection — and build the Open3D cloud in memory as below.

How do you build the Open3D point cloud without saving a .ply file?

Skip the file: o3d.utility.Vector3dVector() wraps a NumPy array straight into the cloud's points and colors, so the capture buffer becomes a geometry in memory with no .ply write, read or parse in the cycle.

import numpy as np
import open3d as o3d

pts = np.asarray(points_xyz_bgr, dtype=np.float32)   # (N, 6): X Y Z B G R
xyz = pts[:, 0:3].astype(np.float64)
rgb = pts[:, 3:6].astype(np.float64) / 255.0         # Open3D wants 0-1 floats
rgb = rgb[:, ::-1]                                    # BGR -> RGB

cloud = o3d.geometry.PointCloud()
cloud.points = o3d.utility.Vector3dVector(xyz)
cloud.colors = o3d.utility.Vector3dVector(rgb)

assert len(cloud.points) == pts.shape[0]              # test: no point lost
assert np.allclose(np.asarray(cloud.colors)[0], rgb[0])  # test: RGB, not BGR

Three details decide whether the cloud renders correctly. Divide by 255.0 or every colour saturates; flip BGR to RGB or red and blue swap; keep float64, because the vector wrapper rejects float32. The two asserts are the proof: the first shows every point arrived, the second shows the first stored colour equals the converted RGB triple, not the raw BGR one.

To carry the depth band into the cloud, filter before construction — keep = (xyz[:, 2] >= 0.8) & (xyz[:, 2] <= 1.2), then cloud.select_by_index(np.flatnonzero(keep)) — or colour the discarded points black and see them in the viewer. The old path, writing TexturedPointCloud.ply and calling o3d.io.read_point_cloud('TexturedPointCloud.ply'), pays disk latency on every cycle and can alter attributes on the way through the file: why exported clouds differ from the viewer shows what a round trip can change, and the NumPy depth-to-cloud guide covers the stride and dtype traps on the array side.

What breaks the overlay at carton edges?

Depth edges are where overlays lie. Between a carton top at 1.0 m and the layer behind at 1.25 m the sensor averages both surfaces, leaving a fringe, often a few pixels wide, that lands inside the band carrying colour from the wrong box.

Fix it in this order. Erode the mask by one or two pixels with a 3 × 3 kernel (cv2.erode) so the fringe falls outside the kept area. Use the sensor's confidence or validity image, where the SDK provides one, to zero weak returns before the band test — the confidence bitflag guide decodes the flags. Expect holes where dark or glossy cartons give no return and read as zero; the depth holes fix covers the causes. A fringe inside the band also biases any height you measure from it, which is the second reason the band edges belong in empty air.

Which camera suits cartons on a pallet?

Choose a sensor that saves the registered 2D image together with the depth map and covers the whole pallet from its mounting height, because that pairing removes calibration work and per-cycle registration error from your program.

For a static pallet bay, projected structured light and ToF sensors return the pair in one call; a laser profiler suits a pallet moving under a line, at the cost of assembling profiles into the map yourself. Mounting height and coverage follow the same maths either way — the 3D camera working distance guide walks them, and the 2D or 3D conveyor guide settles when depth earns its keep at all.

Candidates for a static pallet bay, each returning a registered pair in one call:

The MV-DPS200P-02 projects structured light for a static bay, the MV-DT01WMU is the ToF option, and the MV-DLS300P-04 combines a steerable laser with stereo cameras for larger volumes. Size the sensor from that millimetre-per-pixel figure: open the camera finder, start the camera configurator from your mounting height, or browse the 3D sensors category. More depth-pipeline fixes sit on the 3D topic page, and the contact page reaches an applications engineer when the geometry fights back.