Align depth to a third RGB camera: fix missing extrinsics

You have a stereo pair producing depth and a third camera producing color, and you want one aligned overlay instead of two frames to register by hand. The first attempt fails the moment the align call names the third socket. The second attempt, pointed at the RGB camera socket, gets past the API and then stops on a missing calibration. The sections below take the blockers in the order they bite: socket choice, device calibration, focus position, output size. Each comes with a test and a fix, plus a worked example of what an uncorrected offset costs in pixels.

Three log lines, three different failures

In the rig behind these logs, CAM_B is left, CAM_C is right, CAM_D carries the third color module, and CAM_A is the socket the align stage accepts as the RGB camera. Calling stereo.setDepthAlign(dai.CameraBoardSocket.CAM_D) stops the pipeline build at once. Calling stereo.setDepthAlign(dai.CameraBoardSocket.CAM_A) gets further, into a size warning and then a calibration refusal. Read the messages in that order and each one eliminates a different cause.

What the warp needs, in pixels

The align stage does not blend two pictures; it reprojects. Each depth pixel is deprojected through the left camera's intrinsics into a 3D point, moved by the stored left-to-RGB extrinsic, then projected into the RGB frame with the RGB camera's intrinsics. The three ingredients — left intrinsics, RGB intrinsics, left-to-RGB extrinsic — are written into the device calibration, per camera socket, when you run the vendor's device calibration. Take any one away and there is nothing to warp with, which is exactly what the refusal means.

Here p_mm is the RGB pixel pitch, d_mm the lateral part of the left-to-RGB translation, and Z_mm the stand-off to the feature. The second line estimates how many pixels of misregistration that offset leaves at that distance.

The same arithmetic sets the remount tolerance: 2 mm of mechanical offset at a 600 mm stand-off costs 640 × 2 / 600 ≈ 2 px.

Cause 1: alignment to the third socket is not implemented

The refusal fires before calibration is even opened, so calibrating CAM_D cannot clear it. The test is the socket ID in the call: alignment is implemented toward the RGB camera socket and the stereo inputs only. That leaves two workable layouts. Move the color module to the RGB camera socket, calibrate the device with it mounted there, and align on-device at full frame rate. Or leave the module on CAM_D as an independent stream and warp the depth cloud into its frame on the host — the section near the end covers that path. Pick the first for a live overlay, the second when the mounting is fixed and the overlay is for review.

Cause 2: the device calibration has no RGB entries

Device calibration is stored on the device itself, organized per camera socket. The usual history behind the refusal: the stereo pair was calibrated before the color module was connected, or the module later moved to a different port, so the RGB socket has no stored intrinsics and no extrinsic to the left camera.

Test it in one step before changing anything: read the calibration back with readCalibration2, or dump it fully with the SDK example script calibration_reader.py. If the RGB socket's entries are absent or empty, the device has never been calibrated with that camera in that socket.

The fix is the vendor's device calibration procedure, run with all three cameras connected and mounted exactly as they will run. The stereo camera RGB calibration guide for depth-to-color alignment errors walks the RGB-specific steps, and it pays to size the checkerboard target for your working distance before you start. Then re-read the calibration and confirm the entries exist before touching pipeline code.

Recalibrate after any remount. From the worked example, 2 mm of mechanical offset already costs about 2 px at a 600 mm stand-off.

Cause 3: the RGB focus is not at the calibrated lens position

The vendor's sample code carries the requirement as a comment: the RGB camera must run fixed focus, at the lens position that was used during calibration. Intrinsics describe one lens geometry. Let autofocus refocus and the effective focal length changes, so the warp projects with a magnification the lens no longer has, and the overlay drifts worst away from the image center where the projection model is least forgiving.

The fix is two lines at pipeline build. Read the stored position with calibData.getLensPosition(dai.CameraBoardSocket.CAM_A) and hold it with rgb.initialControl.setManualFocus(lensPosition). Keep autofocus off for the life of that calibration. If you later move the rig to a different working distance, recalibrate there and hold the new stored position.

Cause 4: the output size fell back to the mono resolution

The resolution warning is the least dangerous of the three and the easiest to miss. With a ColorCamera configured on the align socket, the node inherits that stream's size automatically, setIspScale scaling included. Without one, it cannot know the target size, logs the fallback, and produces the aligned output at the mono resolution. Your color frame then never overlays it 1:1.

Either configure the ColorCamera on the align socket, or set the size yourself with stereo.setOutputSize(width, height), passing the width and height of the RGB stream you actually consume. The background on why size settings look ignored lives in stereo depth output resolution: why size settings seem ignored.

The corrected pipeline, end to end

1. Mount all three cameras in their final positions, with the color module in the RGB camera socket, CAM_A.
2. Run the vendor's device calibration with all three connected, following its checkerboard procedure.
3. Read the calibration back with readCalibration2 or calibration_reader.py and confirm the RGB intrinsics and the left-to-RGB extrinsic exist.
4. Build the pipeline: a ColorCamera on CAM_A, and the two mono-side cameras on CAM_B and CAM_C linked into stereo.left and stereo.right.
5. Call stereo.setDepthAlign(dai.CameraBoardSocket.CAM_A).
6. Hold focus with setManualFocus(getLensPosition(CAM_A)). If the RGB stream size differs from the mono size, set it with setOutputSize.
7. Verify on a checkerboard at 600 mm and again at 1,200 mm: overlay the aligned depth on the RGB frame and require edge residual under 2 px at both distances.

import depthai as dai

pipeline = dai.Pipeline()

# Color module on the RGB camera socket (CAM_A), not the third socket (CAM_D)
rgb = pipeline.create(dai.node.ColorCamera)
rgb.setBoardSocket(dai.CameraBoardSocket.CAM_A)

calib = dai.Device().readCalibration2()
lensPosition = calib.getLensPosition(dai.CameraBoardSocket.CAM_A)
rgb.initialControl.setManualFocus(lensPosition)  # fixed focus at the calibrated value

left = pipeline.create(dai.node.ColorCamera)     # left module
left.setBoardSocket(dai.CameraBoardSocket.CAM_B)
right = pipeline.create(dai.node.ColorCamera)    # right module
right.setBoardSocket(dai.CameraBoardSocket.CAM_C)
# left.isp -> stereo.left, right.isp -> stereo.right

stereo = pipeline.create(dai.node.StereoDepth)
stereo.setDepthAlign(dai.CameraBoardSocket.CAM_A)
# Only when the RGB stream size differs from the mono pair's size:
# stereo.setOutputSize(1920, 1080)  # example values; use your stream's width and height

When the module must stay where it is: warp on the host

If the color module is bolted where it is and the overlay is for offline review, skip the on-device path. Calibrate the left-to-RGB extrinsic yourself with a checkerboard seen by both cameras, following the multi-camera extrinsic calibration workflow in one shared coordinate frame. Then, per frame, deproject the depth with the correct camera's intrinsics — which camera to deproject with matters more than most people expect — apply your extrinsic, and project into the RGB frame. Host-side registration also lets you control the precision trade, covered in depth to color alignment without losing depth precision. Budget for it: the on-device warp costs the pipeline nothing, while the host version runs on every frame you process.

Buy the registration instead of building it

On a new build, the cleanest fix is a 3D camera that settles depth-to-texture registration at the factory:

The MV-DPS200P-02 projects structured light and returns depth with its texture hardware in one calibrated unit, so the alignment problem never reaches your bench. The same holds for a galvo-steered stereo rig:

The 3D sensors category lists the stereo, structured-light and profiling families side by side, and the camera finder filters them by type. Where the color inspection is a separate stand-alone camera rather than a board module, area scan cameras cover the GigE, USB3 and 10GigE color sensors, and the system configurator pairs the camera with a lens and light before you order.

Pre-flight checklist

Calibration

• All three cameras connected and in final positions during device calibration — entries are stored per socket, so CAM_D entries do not help an align to CAM_A.
• Calibration read back and confirmed with readCalibration2 or calibration_reader.py — the RGB intrinsics and the left-to-RGB extrinsic must both exist.
• Checkerboard sized for the working distance — a target too small at distance is the top cause of a failed calibration session.

Pipeline

• Color module on the RGB camera socket — the socket ID in setDepthAlign decides between an instant refusal and a working warp.
• Manual focus held at getLensPosition — autofocus silently invalidates the RGB intrinsics.
• Output size settled — configure the ColorCamera on the align socket, or pass the stream size to setOutputSize.

Verification

• RGB and depth captured on the same timestamps — run a depth and color synchronization check first, because alignment cannot fix frames that were never simultaneous.
• Checkerboard residual under 2 px at 600 mm and 1,200 mm — the worked example above sets that scale.

Every fix here is a short bench job once you know the order: socket, calibration, focus, size. For the surrounding articles on stereo geometry, calibration and point clouds, the 3D vision topic page groups them, and the knowledge base hub holds the rest.