Depth camera robot calibration with three point pairs

A depth camera bolted above the workspace reports every object in camera coordinates: millimetres from the sensor, axes aligned to the housing. The robot wants base coordinates. The bridge is one rigid transform, and because the camera already returns metric XYZ you can measure it directly — no calibration plate, no pose sweep. Touch three non-collinear points in the cell with a pointed tool, record each one as camera XYZ and robot XYZ, and solve tgt = [R|T]src. This guide covers the frames to log, the touch-off itself, the arithmetic of R and T, and the check point that says the result is good.

What you are actually solving

You want a 3 × 3 rotation matrix R and a 3 × 1 translation vector T, in millimetres, that turn any camera-frame point into a base-frame point:

R is orthonormal with det R = +1; T carries the offset between the two origins. The matrix stays constant while the camera and the robot base stay put: solve it once, then publish it as a static camera transform on the controller. At pick time the same matrix converts each detected object into a base-frame target, the step described in eye-to-hand object pose transform for robot picks.

Plate-and-pose hand-eye solvers reach the same matrix through many robot poses; camera to robot transform calibration: stop measuring by hand covers that route. When the camera already hands you metric 3D points, three touch-offs are the short path to the same answer.

What you need on the bench

Five things, in order of how often they get skipped:

• A depth camera with metric XYZ output. Stereo structured light, projected structured light, or a ToF camera with calibrated depth. If your pipeline starts from a raw depth image, convert pixels to XYZ first; depth map to real distance calibration for XYZ point clouds covers that step. A stereo structured-light unit such as the MV-DLS300P-04 or a projected structured-light model such as the MV-DPS200P-02 is the usual fixed-mount choice for a robot cell.

• A rigid mount. Every 0.1 mm the camera shifts after calibration becomes 0.1 mm of permanent pick error. A heavy camera stand such as the LS-MNT-0040 machine vision camera stand holds its position against knocks. Lock the optics too: calibrate a camera with autofocus by locking the focus first, because refocusing after calibration invalidates the transform.

• Coverage. Check the field before you commit to marks. A camera at 900 mm with a 60° horizontal field covers 2 × 900 × tan(30°) ≈ 2 × 900 × 0.577 ≈ 1,040 mm, so a 700 × 500 mm workspace fits with about 170 mm of margin per side. All three marks, the robot approach, and the parts you will pick must sit inside that footprint.

• A pointed tool tip with a taught TCP. A nylon or brass cone, sharpened so the contact point repeats better than your pick tolerance. Teach the TCP on it with the pendant's multi-point method before you log anything.

• One program for both devices. Drive the robot and read the camera from the same script — C#, Python, whatever the cell uses — so poses are logged, never transcribed by hand.

Capture the three point pairs

Placement decides the quality. Spread the three marks with legs at least a third of the workspace diagonal — the 700 × 500 mm example gives sqrt(700² + 500²) ≈ 860 mm, so legs of 290 mm or more — and keep every interior angle of the triangle above 30°. The reason is conditioning: 2 mm of lateral touch error on a 300 mm leg becomes 2 ÷ 300 = 0.0067 rad of rotation, which displaces picks by 0.0067 × 800 ≈ 5.3 mm at 800 mm from the centroid. Doubling the leg length halves that term.

1. Teach the TCP on the pointed tip and select that tool as active on the controller.
2. Mark three points in the workspace: centre-punched dimples, scribed crosses, or small adhesive rings on rigid surfaces the robot can reach.
3. Place them wide: legs at least a third of the diagonal, angles above 30°, and none of the three close to the line through the other two.
4. Fix the frame convention. Log the TCP position in base coordinates at every touch. If your interface exposes only the flange pose, add the TCP offset before saving; mixing the two frames is the most common way this calibration fails, as matching tool0 to the sample shows.
5. Read the camera side. Click the pixel at the centre of each mark in the depth image and take the median of a 5 × 5 pixel patch, per axis, in millimetres.
6. Touch each mark with the tip at reduced speed and record the robot XYZ of the TCP in the same file.
7. Save the session — pairs, robot poses, camera settings — so you can replay it later, the workflow in hand-eye calibration pose capture: sample once, replay.

Solve R and T from the pairs

Three exact pairs have a closed-form solution you can check by hand: build a right-handed frame at the first point of each set, shown below. Three or more pairs with noise go through the Kabsch algorithm — the unique closed-form rigid fit in the least-squares sense, built on one singular value decomposition:

With noise-free pairs the fit is exact and the root-mean-square deviation is zero. With six or more pairs, Kabsch minimises that deviation and hands you a per-point residual — the number you will use to accept or reject the calibration. The SVD machinery is the same as in plane fitting on 3D point clouds: least squares vs RANSAC; if one touch is badly off, drop that pair or fit with RANSAC instead of averaging the mistake in.

Worked example: three pairs over a 700 × 500 mm workspace

Camera at 900 mm looking down, marks on the fixture plate, all values in millimetres:

Build a local frame at P1 in camera coordinates. The x-axis runs from P1 to P2: (300, 0, 0) ÷ 300 = (1, 0, 0). The z-axis is the triangle normal: (300, 0, 0) × (−300, 300, 0) = (0, 0, 90,000), normalised to (0, 0, 1). The y-axis completes the triad: (0, 0, 1) × (1, 0, 0) = (0, 1, 0). Stacked as columns, M_src is the identity.

The same construction at Q1 in robot coordinates gives the axes. x: (Q2 − Q1) ÷ 300 = (0, −300, 0) ÷ 300 = (0, −1, 0). z: (0, −300, 0) × (300, 0, 0) = (0, 0, 90,000), again (0, 0, 1). y: (0, 0, 1) × (0, −1, 0) = (1, 0, 0). Stacked as columns, M_tgt = [[0, 1, 0], [−1, 0, 0], [0, 0, 1]].

R = M_tgt · M_src^T = M_tgt, a rotation of −90° about the z-axis. Then the translation:

Check it back through the pairs. R · P2 + T = (100, −700, 900) + (300, 1600, −850) = (400, 900, 50), which is Q2. R · P3 + T = (400, −400, 900) + (300, 1600, −850) = (700, 1200, 50), which is Q3. The fit closes exactly, as it must with three noise-free pairs.

Now the honest test: touch a fourth mark the fit never saw. The camera reads P4 = (550, 250, 900), so the transform predicts (550, 1050, 50). The robot measures (551, 1048, 52). The residual is sqrt(1² + 2² + 2²) = 3 mm. With a 10 mm pick tolerance and the acceptance rule of residual at most one third of tolerance (3.3 mm), this calibration ships.

Check the result before the first pick

1. Fourth-point check. Touch a mark that was not in the fit and compare, as above. Accept when the residual is at most one third of your pick tolerance.
2. Round trip on a real part. Place a part, detect it in the depth image, transform the pick point, and send the robot there with the gripper open. The tip should land on the feature you selected in the image.
3. Publish and re-check. Load the matrix as a static frame — publishing an eye-to-hand calibration as a static camera transform — then let base-frame picks from the calibration results run the cell. Re-run the fourth-point check weekly and after any collision.

Common mistakes and their fixes

When three points are not enough

Three pairs give you no quality metric: the fit is exact by construction, so it hides its own noise. Move to six or more pairs when your pick tolerance is tighter than about three times the fourth-point residual you measured. Least squares averages the random part, and going from 3 to 12 pairs roughly halves it, because independent noise shrinks with the square root of the pair count. Systematic errors do not average out — a mismeasured TCP or a flexing stand shifts every pair the same way, so fix those at the source.

Two neighbouring cases take different tools. If the camera can move or the intrinsics may change, run a full hand-eye solve over robot poses instead; camera above the robot base: eye-to-hand calibration setup covers placement and coverage planning. And if the cell has only a 2D area scan camera, there is no third coordinate to touch with: map image points onto one known plane with a homography, or run a target-based hand-eye solve, and keep the point-pair method for the depth camera.

Checklist before the first autonomous pick

• TCP taught on the pointed tip; the logged frame is the TCP, not the flange
• All values in millimetres, one file, robot poses included
• Legs at least a third of the workspace diagonal; interior angles above 30°
• Focus and exposure locked before the first touch-off
• Camera XYZ read as the median of a 5 × 5 pixel patch
• R orthonormal with det R = +1; every calibration pair reproduces within rounding
• Fourth-point residual at most one third of the pick tolerance
• Transform published as a static frame; weekly check touch scheduled

The camera side starts in 3D sensors and the 3D camera finder. Send your workspace numbers through the contact page and we will run the coverage arithmetic and stand choice with you. More depth-camera workflows sit on the 3D vision topic page.