Align depth and color without the camera SDK pipeline

Depth and color from the robot-arm camera arrive as RTSP frames, so the SDK pipeline that normally aligns them never sees the data. You can still align them pixel-accurately: alignment is reprojection, reprojection needs only the factory intrinsics, the extrinsics and the depth scale, and every depth SDK prints all three. This guide gives you the capture format that preserves depth, the calibration constants to store once, the reprojection with a worked example, the synthetic-device route that reuses the SDK's own align function, and the checks that prove the overlay. It is for anyone processing streamed or saved RGBD frames in Python on arm cells and multi-camera hosts.

Why doesn't the SDK alignment work on RTSP frames?

The align stage runs inside the vendor capture pipeline, so it only ever sees framesets that pipeline delivers; an RTSP or saved frame is a plain NumPy array carrying no calibration, and there is nothing in it for the SDK to align.

The pipeline's align component reads calibration from its own driver, synchronizes the streams and rewrites depth into the color camera's frame. An RTSP capture hands you pixels only: no intrinsics, no extrinsics, no scale, no device timestamps. The pipeline object also enumerates cameras on its own transport and cannot accept a network stream as input, so there is no bridge to configure.

That leaves two routes, both starting from the same three constants. Hand your frames to the SDK through its synthetic-device interface and keep your existing align calls, or run the reprojection yourself in NumPy. The rest of this guide builds both.

How do you save depth without destroying it?

Save depth as raw 16-bit or 16-bit PNG at the sensor's native resolution, never through an 8-bit lossy codec: depth is one raw unit per pixel, and 8 bits collapses a 4 m range into 255 levels, about 16 mm of quantization.

Depth is a single-channel 16-bit range image (the OpenCV CV_16UC1 layout) with a separate depth scale converting raw units to metres. At the common 1 mm per unit the 16-bit ceiling sits at 65.5 m, far above any arm cell. Store the scale beside the data; it cannot be recovered from pixels afterwards.

Color is ordinary 8-bit BGR and survives lossy video; depth does not, because the quantization is baked in permanently and no filtering recovers it.

Pair each depth frame with its color frame by timestamp before aligning. A static bench forgives a mismatched pair; a moving arm does not — 250 mm/s across a 15 ms exposure gap moves the scene 3.75 mm, more than the baseline shift you are correcting. The timestamp pairing method and the drift fix for growing offsets cover that capture side.

Which calibration values does the stream leave out?

Three constants, all fixed at the factory and printed by the SDK the moment the camera is connected directly: each stream's intrinsics (fx, fy, ppx, ppy), the depth-to-color extrinsics (rotation R and translation t), and the depth scale in metres per raw unit.

Query them once during commissioning, while the camera is still on its native link, and store them in a config file beside the frames; the stream will never carry them. Intrinsics belong to one resolution — the values at 640 × 480 are not those at 1280 × 720 — so keep a block per stream resolution and run both streams at the same resolution, as the stream-resolution fix explains. Extrinsics do not change with resolution: they fix where the two sensors sit inside one housing.

The values below drive every number in this guide. They are an illustrative calibration of the kind an SDK prints at 640 × 480; substitute your camera's own before running anything.

What is the reprojection math, step by step?

Each depth pixel unprojects to a 3D point with the depth intrinsics, the extrinsics move that point into the color camera's frame, and the color intrinsics project it back to a pixel; in the worked example the depth pixel (400, 300) at 1.250 m lands on color pixel (421, 330).

1. Convert to metres: Z = raw × scale = 1250 × 1 mm = 1.250 m.
2. Unproject: x = (400 - 320)/400 × 1.250 = 0.250 m, y = (300 - 240)/400 × 1.250 = 0.1875 m, giving the point (0.250, 0.1875, 1.250) m in the depth camera's frame.
3. Transform: with R = identity and t = (-40.0, 0, 0) mm, P' = (0.210, 0.1875, 1.250) m.
4. Reproject: u' = 600 × 0.210/1.250 + 320 = 420.8 and v' = 600 × 0.1875/1.250 + 240 = 330.0, so rounded the pixel lands at (421, 330). The 40 mm baseline contributes 600 × 0.040/1.250 = 19.2 px of the horizontal move; the color focal being 1.5× the depth focal contributes the rest.
5. Copy color[330][421] into the aligned image at row 300, column 400 and keep Z. Pixels with no color get zero.

import numpy as np

# Illustrative calibration of the kind your SDK prints at 640x480.
fx_d, fy_d = 400.0, 400.0        # depth focal length, px
px_d, py_d = 320.0, 240.0        # depth principal point, px
fx_c, fy_c = 600.0, 600.0        # color focal length, px
px_c, py_c = 320.0, 240.0        # color principal point, px
t_dc = np.array([-0.040, 0.0, 0.0])  # depth-to-color translation, m
depth_scale = 0.001              # metres per raw unit

depth = np.zeros((480, 640), dtype=np.uint16)   # 16-bit raw depth, row-major
depth[300, 400] = 1250                          # test pixel: 1250 x 1 mm

z = depth.astype(np.float32) * depth_scale
u, v = np.meshgrid(np.arange(640, dtype=np.float32),
                   np.arange(480, dtype=np.float32))
x = (u - px_d) / fx_d * z
y = (v - py_d) / fy_d * z
zc = np.where(depth > 0, z + t_dc[2], np.float32(1.0))  # guard divide-by-zero
xc = x + t_dc[0]
yc = y + t_dc[1]
u_c = np.rint(fx_c * xc / zc + px_c).astype(np.int32)
v_c = np.rint(fy_c * yc / zc + py_c).astype(np.int32)
inside = (depth > 0) & (u_c >= 0) & (u_c < 640) & (v_c >= 0) & (v_c < 480)

# Test: the worked pixel must land at color (421, 330) with Z = 1.250 m.
assert abs(float(z[300, 400]) - 1.250) < 1e-6
assert int(u_c[300, 400]) == 421 and int(v_c[300, 400]) == 330
print(int(u_c[300, 400]), int(v_c[300, 400]), float(z[300, 400]))
# -> 421 330 1.25

The same projection run live inside the SDK pipeline is what the depth-to-color precision guide optimizes; here you own every step of it.

Expect two limits. The border first: fx_c is 1.5× fx_d, so color sees a narrower field, and depth pixels with |u - 320| > 213 px project outside the 640-px color width — the outer 107 columns on each side, about a third of the image. The inside mask zeroes them; wrapping the index instead paints garbage.

Can you inject saved frames into the SDK's synthetic device?

Yes — most depth SDKs ship a synthetic (software) device that accepts your own raw frames plus calibration, and once the frames are registered the SDK's align and distance calls behave exactly as they do on live capture.

The pattern: register a depth stream and a color stream with their resolutions, attach each stream's intrinsics, attach the extrinsic between them, set the depth scale, then submit matched frame pairs. Your existing code — the same align calls, the same distance queries — then runs unchanged on recorded or streamed data.

Pick that route when your codebase already speaks the SDK's API and you want no refactor, or when you need the SDK's filters to behave identically to production. Run the NumPy reprojection instead when the deployment must be SDK-free, or when the synthetic device rejects your resolution or format.

Two capture details decide whether injected frames come out sane. Copy buffers out of your capture path before handing them over — a reference into a streaming buffer is overwritten by the next frame, the exact failure the saved-frame buffer fix walks through. And check your SDK exposes the synthetic route at all: it is a documented feature of the major depth SDKs, but the class names differ, so search your SDK reference for its software-device entry point.

How much compute does per-frame alignment cost?

About 4.6 MFLOP per 640 × 480 frame — 307,200 pixels at roughly 15 floating-point operations each — which vectorized NumPy turns around in 3-10 ms on one desktop-class core, well inside the 33.3 ms budget of a 30 fps stream.

Scale by pixel count: a 1280 × 720 frame triples both figures to 13.8 MFLOP and 9-30 ms, which still fits one frame interval but leaves little room for detection or matching on the same core. Time the snippet above at your own resolution and trust that measurement over any table.

The stream weighs more than the math. A raw depth frame is 0.6 MB, so 30 fps means 18 MB/s of depth plus roughly 28 MB/s of 8-bit color — 365 Mbps total, comfortable on a gigabit link, not on Wi-Fi. Size the host for the stream first, then the arithmetic.

If alignment runs on dedicated hardware beside the cell, a vision controller puts the camera LAN ports, PoE options and light-output channels in one unit mounted next to the robot. For a multi-camera cell, one box covers stream handling and the application:

Smaller cells step down the same line:

How do you verify the alignment is pixel-accurate?

Project a target with known geometry and require the overlay to land within 1-2 px; at fx_d = 400 px and 1.25 m one pixel of registration error is about 3.1 mm on the part, so the 1-2 px target is a 3-6 mm band.

Run four checks:

• Overlay board. A printed-circle target at a fixed distance: circles land inside circles within 1-2 px across the central two-thirds of the image.
• Shift law. Horizontal shift at the image center should track fx_c × baseline / Z: for the 40 mm example that is 48 px at 0.5 m, 19.2 px at 1.25 m, 12 px at 2 m. A shift that does not grow as 1/Z means a wrong t or a wrong focal length.
• Border. In the example geometry, columns 0-106 and 534-639 must be zero. Wrapped values mean the mask is missing.
• Range read-back. Compare raw × scale against a tape measure at the same three distances. A centimetre-scale offset there is the sensor's own range error or a wrong scale, not the reprojection.

Mount the camera at the distance the working distance and positioning guide recommends and both error sources stay in their comfort zone.

When should you not rebuild the alignment yourself?

When the camera can sit on the host's own bus, use the SDK pipeline and its built-in align stage — you get factory calibration handling, hardware timestamps and lower latency without writing any of this — and keep the rebuild for frames that exist only as network or recorded data.

Two cases need more than this guide. A separate RGB camera has no factory extrinsics, because R and t describe sensors inside one housing: calibrate your own pair first, following the missing-extrinsics fix and the RGB calibration walkthrough. And no static transform corrects depth and color captured at different instants on a moving arm — hardware-sync the exposures or capture while the arm holds still.

If you are still choosing hardware, prefer a camera class whose SDK delivers synchronized framesets and prints its calibration over its own link — the 3D sensors range is the place to start:

To size an alignment host or confirm a camera's calibration access before ordering, send the camera model, resolution and stream rate through the contact page. The remaining capture-side fixes sit on the 3D vision topic page.