Camera intrinsics for hand-eye calibration: fix the pose first

You sample the board five times, the solver announces that the camera is calibrated, and the base-to-camera transform it prints lands the camera tens of millimetres — or a tenth of a radian — away from where the mount actually holds it. The hardware did not move. In cases like this the robot, the board and the frame tree are usually innocent: the stack is solving with wrong camera intrinsics, most often the identity matrix it falls back to when no calibration file is loaded.

This guide walks the failure in the order it bites: the symptom visible in the pose overlay, a test for each likely cause, the fix in order, and the checks that keep the result honest. It is written for anyone running eye-to-hand calibration with a 6-DOF arm and a fixed 2D camera ahead of a pick-and-place launch.

The symptom: a pose that cannot be the camera

The tell-tale shows up in the target overlay long before the final matrix disappoints. Open the live pose view and look at the axes drawn on the board. With sound intrinsics, the target frame's three axes come out the same length at right angles to one another, with Z pointing straight out of the board. With intrinsics missing, the Z axis sweeps off toward a corner of the image. The reason is geometric: an identity camera matrix puts the principal point at pixel (0, 0), so the solver yaws the board until its depth axis appears to leave from the corner. The numeric symptom follows: the solved translation disagrees with the measured mount by tens of millimetres, the rotation by tenths of a radian, and the numbers wander as samples are added.

Read the overlay first; it costs nothing and separates an intrinsics fault from everything else on this list.

Cause 1: intrinsics missing or wrong

Check this first because it is both the most common fault and the cheapest to test. When no intrinsics file is loaded, many hand-eye stacks start from an identity camera matrix, fx = fy = 1 with cx = cy = 0, and every pose estimate then inherits a camera model wrong by three orders of magnitude. Open the intrinsics file the camera driver reads (camera_info.yaml on common open-source stacks) and read the matrix block. Values of 1 and 0 mean no calibration is loaded. Plausible numbers for a USB camera are hundreds to a few thousand pixels, with cx and cy near the centre of the image.

A fast sanity value comes from the pinhole relation fx ≈ (W/2)/tan(FOV_h/2): a 720p USB camera with a horizontal field of view near 60° works out at fx ≈ (1280/2)/tan(30°) ≈ 1100 px. If the file says a number tens of times smaller or larger, it is stale, belongs to another camera, or was produced at another resolution.

Why a wrong focal length drags the pose off

Depth and focal length are tied together in the pinhole model:

Depth comes out proportional to whatever focal length the solver believes. Enter an fx about 18% below the true value, 900 px against a true 1100 px, and every target depth reads short: a board held 600 mm from the lens reports 600 × 900 / 1100 ≈ 490 mm. Each hand-eye sample then pairs a correct robot pose with a wrong camera-to-target pose, and the least-squares solution absorbs the inconsistency as a bogus camera location. That is how a converging optimizer still prints a base-to-camera result tens of millimetres and tenths of a radian away from the physical mount.

Cause 2: a target with too little to look at

A single fiducial marker produces a pose, but a poor one: four corners carry little redundancy, and the error grows quickly with distance. Give the solver more features in view. Use a ChArUco board of 5×5 markers or larger; where the fixed camera cannot fit a large board, shrink the marker size and keep the count high, so the whole board stays in frame while carrying more corners to refine the target pose. Keep the board flat and enter its square size exactly, because the solver trusts those dimensions absolutely.

Board and marker size both interact with stand-off distance: the ChArUco board dimensions and distance errors guide shows how distance problems appear when a board is too small, and the checkerboard target size at distance article covers the same failure for checkerboards.

Cause 3: motion without two non-parallel rotations

The hand-eye equations only pin down the camera pose when the sample motion contains rotations about two non-parallel axes; two are necessary, and on a 6-DOF arm they are easy to get. Compact machines need care. On an R-θ-φ arm, if the camera rides a joint that spins about the same axis as the other revolute joint, every sample is a rotation about one axis and the solver has no unique answer; it converges, but to values metres away.

Test by listing which joint moves in each sample. If only one rotation axis appears across the whole set, drive a second, non-parallel revolute joint through its range and resample. Five samples is the bare minimum that produces a result; fifteen with varied orientation gives the solver something worth fitting. The robot positions that carry calibration information guide shows which poses add constraints and which are near-duplicates.

The fix, in order

1. Print or obtain a board with a known square size. Measure the pitch with a caliper; the solver scales the world by that number, and a mis-entered pitch biases every pose the same way. The grid-pitch calibration error guide shows the symptom.
2. Lock focus and iris. Autofocus hunting changes the focal length between frames, so intrinsics come out as an average of different lenses. Calibrate once with focus locked and keep it locked; the lock-the-focus-first procedure covers the order of operations.
3. Capture about 20 board views spread across the frame and tilted both ways, run the intrinsics calibration, and check the residual against the targets in the calibration residual values guide.
4. Write fx, fy, cx, cy and the distortion coefficients into camera_info.yaml, the intrinsics file the driver reads, at the resolution you run. Restart the driver and confirm fx is no longer 1.
5. Re-run the overlay test. The target axes should now stand equal and square, with Z pointing out of the board.
6. Re-take the hand-eye samples, at least fifteen, covering two non-parallel rotation axes, with the board fully in view in every pose.
7. Check the final matrix against the measured mount, then run the formal verification described below.

None of this survives a camera that creeps between sessions. A fixed eye-to-hand camera wants a rigid column it cannot drift along:

The LS-MNT-0040 slide-table stand gives a 700 mm column over a 400 x 450 mm base, which holds the camera pose steady through a long sampling session and leaves the slide for fine framing. For a bench-scale rig, a portable optical stand carries camera and target without a full frame:

Worked example: a 6-DOF cobot bench with the camera 550 mm off the base axis

The inputs come from the bench described above, where the fixed camera hangs about 550 mm to the side of the base axis and 710 mm above the base plane; positions are read off the physical mount with a rule:

• Measured camera position in the base frame: x = 10 mm, y = 550 mm, z = 710 mm.
• Solved translation before intrinsics are loaded (typical of the failure): x = -40 mm, y = 615 mm, z = 660 mm.
• Solved translation after a good intrinsics file is in place: x = 9 mm, y = 552 mm, z = 708 mm.

The rule in words: compare solved and measured positions with the straight-line distance, e = sqrt(Δx² + Δy² + Δz²).

Before the fix: Δx = -40 - 10 = -50 mm, Δy = 615 - 550 = +65 mm, Δz = 660 - 710 = -50 mm, so e = sqrt(2500 + 4225 + 2500) mm = sqrt(9225) mm ≈ 96 mm.

After the fix: Δx = 9 - 10 = -1 mm, Δy = 552 - 550 = +2 mm, Δz = 708 - 710 = -2 mm, so e = sqrt(1 + 4 + 4) mm = 3.0 mm.

The decision: 96 mm out, with a corner-sweeping Z axis, means a broken input — stop tuning and reload intrinsics. Three millimetres total, no axis more than 2 mm out, is as close as a rule-read mount can adjudicate, so sign the inputs off and move to formal verification. Apply the same reading to rotation: the solved optical axis must still pierce the board centre, and offsets of tenths of a radian point to an input error, not to tuning. One honest caveat: a rule read over half a metre is itself good to a millimetre or two, so this check separates sound inputs from broken ones; it does not certify millimetre accuracy. The hand-eye accuracy verification procedure measures that properly, and the eye-to-hand result verification walks the base-to-camera case end to end.

Five samples or fifteen: which matrix to use

The result that appears after the fifth sample is fitted to five poses only, which is why the matrix keeps changing as samples six through fifteen arrive. The final matrix is refitted to all of them, so it is the one to export — not the first valid one and not an average you compute yourself. Changing values between samples are normal; values still moving tens of millimetres per sample after fifteen poses point back at causes 1 to 3 rather than at sample count. Export once sampling stops, then verify: the 1-2 cm translation question puts typical result magnitudes in context before you chase precision.

Still wrong? The remaining suspects

With sound intrinsics, a proper board and varied motion, work down this list.

• Frames. The four selections — camera frame (usb_cam), target frame (handeye_target), end-effector frame (link6) and base frame (base_0) — must all resolve to frames the driver actually publishes; a stale or missing target frame silently corrupts the solve. The target frame checklist covers the recovery.
• Initial guess. Measure the guess off the physical mount in metres and radians, and enter it in the reference frame the tool expects; a guess expressed in the wrong frame drags the optimizer to a wrong basin. The initial-guess frame guide sorts out which frame that is.
• Duplicate publishers. Two nodes publishing the same transform fight each other, and the solver reads whichever wins that instant. See find and fix duplicate calibration publishers.
• A moved camera. If the camera or the board was nudged after intrinsics or between samples, every stored pose is stale; recalibrate after any physical change. The static camera drift explainer shows why.

Prevention checklist

Before intrinsics:

• Lock focus and iris. Autofocus motion changes the focal length mid-session, and even a locked lens drifts with temperature over a long day; the focus-drift diagnosis covers the symptoms.
• Measure the board pitch and enter it exactly; the solver scales the world by that number.
• Bolt the camera to a mount it cannot creep along.

Before sampling:

• Read the intrinsics file: plausible fx and fy, centred cx and cy, never an identity matrix.
• Calibrate intrinsics at the resolution you will run; fx scales with image width.
• Pass the overlay test: equal-length axes at right angles, Z out of the board.
• Enter the initial guess in metres and radians, measured off the mount, in the correct reference frame.

After solving:

• Use the final matrix, refitted to all samples.
• Check the solved translation against the measured mount; agreement within about 2 mm per axis passes a rule-read bench.
• Finish with formal verification before the first production pick.

Parts and the next step

The decision, in one line: if the target's Z axis sweeps to a corner, fix the intrinsics before touching anything else; with sound intrinsics, a 5×5-or-bigger board and two non-parallel rotation axes, fifteen samples produce a final matrix that lands within a millimetre or two of the measured mount.

If the sensor at the end of this exercise is a depth camera rather than a USB webcam, the same intrinsics-first discipline applies, and the 3D camera working-distance guide covers the positioning side. Browse camera stands for the rigid mount, 3D sensors for depth cameras, or start the system configurator to spec the station. More calibration and integration fixes live on the systems topic page, and the applications team is reachable through the contact page if a bench refuses to converge.