You re-ran the solver, and the transform came back several hundred millimetres from the URDF guess you started from. The temptation is to blame the least-squares code. Don't. In bench practice a grossly wrong eye-in-hand result comes from the inputs, and two faults account for nearly all of it: the recorded motion rotates about fewer than two independent axes, so the rotational part of the problem is under-determined, or the camera intrinsics never reached the solver, usually as an empty distortion vector.
This guide gives you the numbers for both checks. You will count the rotational degrees of freedom in your own chain, measure the pairwise rotation spread of your pose set with one trace formula, read back the intrinsics the solver actually stored, and fix whichever input the arithmetic indicts. Both checks take minutes, and both fail silently, which is why they get skipped.
Why does an eye-in-hand calibration land way off its initial guess?
Two input faults produce almost every grossly wrong result: the pose set rotates about fewer than two independent axes, so the rotational part of AX = XB is under-determined, or the intrinsics reach the solver with empty distortion parameters and every corner lands in the wrong place.
A solver returns the transform that best explains the samples it was handed. In the bench case behind this guide, the URDF guess placed the camera at (21, 33, 165) mm from the head frame, and the solved result came back at (−256, 124, 528) mm. The translation moved 466 mm — sqrt(0.277² + 0.091² + 0.363²) = sqrt(0.2168) ≈ 0.466 m — and two of the three Euler angles moved about 16°. No weighting bug does that; degenerate inputs do. The case ran on a ROS hand-eye plugin (MoveIt Calibration), but the failure modes below are solver-agnostic.
Rank the causes in this order before you touch anything:
- Motion about a single axis. The head nods but the torso never yaws (or the reverse), so a whole family of rotations explains the same samples.
- Intrinsics lost between driver and solver. Correct values ride the CameraInfo stream while the plugin's internal copy carries empty distortion parameters.
- Frame mix-ups. You solved head → camera_mount when you meant camera_mount → head, or the sensor-frame setting points at the optical frame while you compare against the mount.
- Too few poses. A five-pose result moves every time you re-solve it.
- The solver itself. Genuine solver faults exist and are testable with the synthetic check in the fourth section below.
How much rotation must the calibration motion contain?
Plan 10 to 15 poses whose pairwise relative rotations span at least two non-parallel axes; with motion about one axis only, the rotational part of the unknown transform has a whole family of solutions, and five or six poses can land tens of degrees apart inside that family.
The eye-in-hand problem pairs each robot motion A_i with the camera's observed motion B_i and solves A_i X = X B_i for the flange-to-camera transform X. Closed-form solutions to the coupled robot-world form AX = YB build the rotation first from the A_i and B_i pairs, and that construction locks only when the motions rotate about at least two non-parallel axes (one classical construction is the NIST Kronecker-product solution). About one axis, the components of the rotation can trade against each other; about two, the family collapses to a single transform.
Measure the set instead of trusting it. For each pose pair i, j, the relative rotation angle is:
Take three poses as a worked example. Pose 1 is the zero pose, pose 2 nods the head 30° about x, pose 3 yaws the torso 40° about z. Pairs 1–2 and 1–3 give θ = 30° and 40° by inspection. Pair 2–3 composes both: Rz(40°)·Rx(30°) has diagonal entries 0.766, 0.663, 0.866, so trace = 2.295 and θ = arccos((2.295 − 1) / 2) = arccos(0.648) ≈ 50°. The set spans two axes with a smallest pair angle of 30°. Delete the yaw poses and every pair collapses onto the nod axis, and no pose count fixes that set.
A healthy 12–15 pose set shows θ_ij values spread through the 25–50° band, with rotation axes drawn from both joints and enough translation that the camera orbits the target rather than spinning in place. If your chain has three rotational joints — the bench chain had torso yaw plus a two-axis head — write the pose script so every pose moves at least two of them. The eye-in-hand pose selection guide builds pose lists this way, and the AX=XB degeneracy guide walks the single-axis diagnosis end to end.
Are the camera intrinsics actually reaching the solver?
Read the intrinsics back out of the solver, not off the wire: a documented bench failure ran with correct values on the CameraInfo stream while the plugin's internal copy held empty distortion parameters, so every outer target corner entered the fit displaced by pixel-scale errors.
Here is the mechanism. The target detector undistorts each detected corner with the distortion vector it is handed. If that vector arrives as zeros, distorted corners get treated as ideal ones, and the fault grows with the cube of the corner's distance from the image centre:
Illustrative numbers for a wide RGB module: true k1 = −0.15, focal length 900 px, a corner at normalized radius r = 0.5. The solver's copy of that corner sits off by 0.15 × 0.125 = 0.0188 in normalized units, which is 0.0188 × 900 ≈ 17 px at the detector. Every outer corner of every sample carries a fault of that size, and the least-squares fit absorbs it into a wrong X.
Check the solver's copy, not the topic. Print the fx, fy, cx, cy and distortion values the plugin stored and compare them against the driver's CameraInfo line by line. In the bench case the wire was correct and the plugin's distortion vector was empty; hard-coding the intrinsics into the solver input, then re-running target detection, made the same geometry solve to a result consistent with the mount. The camera intrinsics for hand-eye calibration guide covers the capture-side half of this.
Factory intrinsics do not exempt a camera from the check: the numbers still have to survive the trip from driver to solver, for the depth and profiler cameras and for the structured-light stereo camera below.
- Type
- Stereo / structured light
How do you test which of the two causes you have?
Export the samples and compute two numbers before collecting again: the smallest pairwise rotation angle across the pose set, which should sit in the 25–50° band on poses driven by both axes, and the distortion vector stored inside the solver, which should match the driver's CameraInfo line for line.
Work down the list:
- Export. The plugin's Save samples control writes each pose's effector_wrt_world and object_wrt_sensor matrices as 4×4 arrays; that gives you a replayable dataset, and the dataset acquisition guide shows the capture workflow.
- Sweep θ_ij. Compute the angle and the axis of every pose pair. A minimum stuck under about 20°, or every axis parallel within a few degrees, indicts the motion, not the solver.
- Echo the intrinsics. Print the fx, fy, cx, cy and D the solver holds. Zeros in D while the CameraInfo shows real barrel terms indict the plumbing, not the optics.
- Run a synthetic control. Plant X at a known value — translation (20, 30, 165) mm, RPY (−104°, 0°, −90°) — synthesize clean A_i, B_i pairs about two axes, and solve. Recovery within 1 mm and 0.5° clears the solver.
| Symptom | Likely cause | Test | Fix |
|---|---|---|---|
| 100 mm and 10° jump from the guess | Single-axis motion | θ_ij sweep: axes parallel | Add torso yaw, re-collect 12–15 poses |
| Result moves on every re-solve | Too few poses | Solve after each added pose | Grow the set to 12–15 |
| Small offsets, poor residuals | Empty distortion in the solver | Print the stored D | Hard-code fx, fy, cx, cy, D, re-run detection |
| Rotation sane, translation wrong | Frame mix-up | Check a known standoff distance | Flip the transform or fix the sensor frame |
Both checks green and the result still odd means you are in the last row of that table — an inputs-and-frames problem. The pose-and-frame-pairs guide covers the timestamp and pairing requirements that a θ_ij sweep cannot see.
What fixes a way-off eye-in-hand result in practice?
Fix the indicted input, then re-solve: re-plan 12 to 15 poses that exercise both rotation axes, hard-code fx, fy, cx, cy and the distortion vector whenever the solver held zeros, and accept a result only after the transform stops moving between successive solves.
- Re-plan the motion. Interleave torso yaw with head pitch so both axes appear in the θ_ij sweep, then collect 12–15 poses. Keep the target 0.3–1 m from the camera so the relative translations carry signal.
- Hard-code the intrinsics. Put the driver's fx, fy, cx, cy and distortion values into the solver input and re-run detection, or re-collect; corners built with empty D stay wrong no matter what the solver does afterwards.
- Grow until it converges. Re-solve after each added pose. When the last pose moved the result by less than 1 mm and 0.2°, stop collecting; more samples stop buying accuracy at that point.
- Cross-check against metal. The solved translation magnitude should sit near the standoff you can measure with a ruler on the mount.
If a single-axis head is all the hardware you have, no pose count rescues the standard solve: add a joint's motion to the capture, or measure the mount directly and treat it as a fixed mechanical dimension.
When does a strange-looking solution still count as correct?
Judge a solve by consistency, not by distance from the URDF guess: reproject the target through base → torso → head → camera for every recorded pose and accept the transform when reprojection error sits near one pixel and keeps falling as poses are added.
Three checks separate correct-but-unfamiliar from genuinely wrong:
- Reprojection. Project the detected corners through the full chain for each pose. A correct X reprojects within about one pixel — corner detection repeats to a few tenths of a pixel, so one pixel leaves margin. The accuracy verification guide turns this into a test procedure.
- Mechanical standoff. If the mount carries a known 40 mm spacer, the solved translation must contain roughly 40 mm along that axis. A few hundred millimetres means a frame got inverted, not that the solve failed.
- Quaternion bookkeeping. q and −q encode the same rotation; a sign flip inside one component does not. Convert to a matrix and compare angles before editing anything — the quaternion-to-matrix guide and the orientation-error guide cover both traps.
What hardware makes a good pose set easy to capture?
Make the physical side boring: bolt the target to a stand with a 700 mm column and a 400 x 450 mm base so it cannot drift between poses, tilt the board once to a fixed angle, and strain-relieve the camera cable so no pose is captured under load.
The target side first. Use a matte board large enough to stay in frame with margin at every pose — the target size guide sizes it — and bolt it down so it cannot rock. Stillness during capture decides image quality on depth sensors the same way it does on 2D rigs; the mounting stillness guide covers the vibration side.
A stand with 700 mm of column and a 400 x 450 mm base holds a board dead still through an afternoon of pose collection.
- Type
- Camera stands
A slope-angle slide under the target holds a fixed tilt, so corners stay detectable at both ends of a wide head sweep.
- Type
- CCD camera stand and positioning stage (LS-MNT-S01)
On the camera side, keep the bracket short and put the strain relief at the camera: a taut cable levers the sensor between poses, and no solver recovers a moved camera. If you are still choosing hardware, the camera stands category and its accessories finder list the rigid options, and the camera finder filters factory-calibrated depth cameras. The stand configurator walks mount, camera and controller in one pass.
Prevention fits in three lines: hard-code the intrinsics once, keep the pose script and its θ_ij sweep under version control, and re-run both checks after any mechanical change. When a solve stays stubborn, the contact page reaches the applications team, and the 3D topic page collects the neighbouring calibration guides.
Questions
How many poses are enough for eye-in-hand calibration?
Treat five or six poses as an unstable preview that can sit tens of degrees inside the solution family. Plan 12 to 15 poses driven by at least two rotation axes, keep most pairwise relative rotations in the 25–50° band, and re-solve as you add: when the transform stops moving between solves, the set is done.
Can I calibrate a head that only nods, with one rotation axis?
One axis is not enough, and no pose count rescues it: rotations about a single axis leave the rotational part of the transform under-determined. Find a second, non-parallel axis elsewhere in the chain — torso or base yaw usually supplies it — and make the pose script actually use both instead of nodding through the same arc.
Why did hard-coding the intrinsics fix the very same pose set?
Because the solver was fitting corners that had been undistorted with empty coefficients. With true k1 = −0.15 left as 0, a corner at half-image radius on a 900 px focal length sits about 17 px from its real position, and that fault enters every target pose. Hard-coded values restore the corners, so the fit lands near the mechanical guess.
What does 'way off' mean in numbers?
In the bench case behind this guide, the URDF guess put the camera at (21, 33, 165) mm and the solver returned (−256, 124, 528) mm: a 466 mm translation jump with about 16° on two Euler angles. Numerical noise lives in the sub-millimetre and sub-degree range, so a jump that large means degenerate input.
How far should the calibration target sit from the camera?
Work at 0.3–1 m on a bench rig. The translation part of the solve comes from the camera's sideways motion relative to the board: too close and the board fills the frame at the pose extremes, too far and each pose's translation change becomes a small fraction of the range, so the flange-to-camera offset fits loosely.
Must I re-collect poses after fixing the intrinsics?
Usually yes. Target poses stored while the distortion vector was empty carry wrong corner geometry, and re-solving cannot repair them; re-run detection on saved images or collect again with the hard-coded values in place. Samples captured with correct intrinsics can be exported and replayed as often as you like without touching the robot.


