Align depth and tracking cameras: extrinsic calibration how-to

A depth camera and a tracking camera on one rig make a strong pair. The depth camera returns metric 3D points in its own frame; the tracking camera reports where the rig itself sits in the world as 6-DoF pose. Bolted to the same bracket, they still speak two coordinate languages, and neither stream says how the frames relate. The procedure below fixes that: mount the pair rigidly, capture a calibration target seen by both cameras, solve one rotation and one translation, and verify the solve before a single depth point reaches production.

What you are solving

Everything you need is one rigid-body transform. A 3×3 rotation matrix R carries the angular alignment between the two optical frames, and a 3×1 translation vector t carries the offset between the optical centres — typically the tens of millimetres set by your bracket. With it, every depth point maps into the tracking frame in one multiply and one add:

Solve both once, while the cameras are bolted together, and the pair stays solved for as long as the mount stays rigid. Watch frame conventions when you apply the result: most cameras publish points with z forward in an optical frame, and the flip to a robot or world convention is where sign errors live. The point cloud frame transform guide walks that conversion.

What you need

• A depth camera with a calibrated depth stream. Stereo, structured light or ToF all work; the 3D sensors category holds the shop's range.
• A tracking camera: the wide-angle fisheye type that reports its own 6-DoF pose from visual-inertial odometry.
• A rigid mount: one aluminium spine or plate carrying both cameras, plus a bench stand to hold the assembly during capture.
• A flat calibration target: a chessboard or ChArUco board glued to glass or aluminium, never loose paper.
• A vision library with a stereo-calibration routine; OpenCV and most camera SDKs ship one.

Any calibrated stereo, structured-light or ToF depth camera serves on the depth side, and these three cover the technologies:

On the bench, a rigid slide-table stand holds the pair at a fixed height while the target moves. The LS-MNT-0040 professional stand is built for exactly that job, with a slide for setting capture distances.

A manual XY and rotation stage under the target board gives repeatable tilts without re-clamping; the LS-MNT-0220 fills that role.

Mount the pair so it stays rigid

The transform is only as good as the mount. Bolt both cameras to one stiff spine so the offset between them cannot change, with the tracking camera's fisheyes and the depth camera's emitter and receiver unobstructed. Printed brackets are convenient but they creep under cable tension and temperature swings, and every bit of flex reappears as error at the target. Route and strain-relieve all cables before calibration, not after.

Some camera makers publish a bracket design together with the transform measured for that exact mount. That published transform is valid only for copies built to its tolerances; a self-printed version needs its own solve, with the published numbers as a sanity check. The camera stands category covers stands, brackets and stages if you are building the mount from shop parts.

Capture the target in 15 to 20 poses

1. Solve intrinsics for each camera first, in the same session and at the focus you will run. A stale intrinsic set biases the rotation, and the intrinsics-first calibration guide shows how that failure looks downstream.
2. Light the board evenly and kill specular glare; a matte print on a flat backing detects corners cleanly.
3. Hold each pose dead still and grab frames from both cameras while nothing moves. A static target makes stream synchronization irrelevant during calibration.
4. Collect 15 to 20 poses: tilt up to ±45°, cover the centre and all four corners of the overlap, and sweep distance from the depth camera's near limit out to roughly 1.5 times your working distance.
5. Inspect corner detection on every frame and discard blurred, clipped or partially occluded views; one bad frame pulls the whole solve.
6. Measure the printed square size with calipers and enter the measured value, not the nominal one — the square size sets the metric scale of the solve.

Solve the rotation and translation

1. Build matched corner lists: for every accepted pose, the same physical board corners detected in the depth-side stream and in the tracking stream.
2. Run the stereo-calibration routine with the depth-side camera as camera one and the tracking camera as camera two. The mechanics are the same as a depth-to-RGB pairing; the stereo RGB calibration guide works through the identical mathematics.
3. Read out R and t, then apply the frame-convention fixes from the first section before you transform any points.
4. Sanity-check the translation. |t| is the physical offset between the two optical centres, so it should land within one to two millimetres of the bracket measurement. A result off by centimetres points to a wrong square size or wrong intrinsics, not to the solver.

The metric scale comes from the known square size and from the depth camera's own metric depth, which is what lets the pair self-check.

Worked example: a 0.3 px residual at a 600 mm standoff

Inputs, illustrative but class-typical — substitute your own values: a depth stream 1280 columns wide across a 70° horizontal field of view; a careful solve at 0.3 px RMS reprojection error, mid-band for the usual 0.2 to 0.5 px result; a working standoff of 600 mm.

The rule: one pixel of residual means the solved reprojection can miss by one pixel's angular subtense, so convert pixels to angle with the field of view, then angle to millimetres with the standoff.

• Pixels per degree: 1280 ÷ 70° = 18.3 px/°
• Angle per pixel: 1 ÷ 18.3 = 0.0547°
• Residual angle: 0.3 px × 0.0547° = 0.0164° = 2.86 × 10⁻⁴ rad
• Lateral error at the target: 600 mm × 2.86 × 10⁻⁴ = 0.17 mm

Result: the rotation half of the transform contributes about 0.17 mm of alignment error at 600 mm. Add ±0.5 mm of mount-metrology uncertainty and the combined figure is √(0.17² + 0.5²) ≈ 0.53 mm.

Decision: set the acceptance gate at 0.5 px RMS. On this camera, 1.0 px already means 0.57 mm at 600 mm, so anything above the gate says recapture rather than tune. At these levels the solve is not the weak link — the bracket measurement is, so spend the effort on calipers and a stiff spine.

Choosing the depth stream for a new rig runs the same chain in reverse — how many pixels a field of view needs for a target accuracy:

Verify before you trust the numbers

1. Check the RMS figure: 0.5 px or below is the pass line for this kind of solve.
2. Reproject the board corners from the tracking frame into the depth image and overlay them; where the residual halo points tells you which axis of the rotation is off.
3. Run the planarity test. Aim the depth camera at a flat board, transform the cloud into the tracking frame, fit a plane, and read the peak deviation. Under about a millimetre at your standoff is a healthy result.
4. Treat the residual as necessary but not sufficient: depth scale bias and mount metrology hide behind it. The extrinsic accuracy guide lists the checks beyond the residual value.
5. Repeat a three-pose quick check after thermal soak and after any knock. A transform that passed on the bench and fails on the machine has a flexing mount, not a bad solve.

Depth holes skew the planarity fit, so keep the test board matte and evenly lit; the missing depth points guide explains where the holes come from.

Common mistakes that corrupt the transform

• Reusing intrinsics from another session or another focus setting.
• A paper target that wrinkles: a 1 mm wrinkle biases the translation directly.
• Cable tension applied to a printed bracket after calibration.
• Ten near-axis poses with the board square to both cameras; the residual looks fine while the rotation stays under-constrained.
• Applying R and t without fixing the optical-frame convention first.
• Trusting a published mount transform on a bracket you printed yourself.

When a fixed transform is the wrong tool

Two cases break the rigid-pair assumption. If the cameras live on separate fixtures — one over the infeed, one in the cell — there is no single transform to solve, and placement rules take over; see the conveyor tracking camera placement guide. If the rig itself moves on a robot wrist, the depth-to-tracking offset changes with every joint, and you need a hand-eye solve instead: the robot-mounted calibration guide covers the capture side, and the eye-in-hand AX=XB extrinsic guide explains the classic formulation.

Adding a third camera? Solve each pair into one shared frame; the multi-camera extrinsic guide shows the bookkeeping.

Build the rig

Pick the depth camera in the 3D camera finder and the mounting hardware in the camera-stand finder, or jump straight to the stand step of the configurator. The 3D topic page collects the related guides, and the applications team will look over a proposed mount before you print or machine it.