Eye in hand calibration inputs: the same robot pose and frame pairs

You already have an eye-to-hand calibration running: robot poses logged, frames captured, pairs built and solved. Now the camera moves to the robot flange and the question is what changes in the data. Nothing does. Eye-in-hand calibration consumes exactly the input eye-to-hand does — one 4×4 robot pose matrix plus the detection taken from one captured frame, wrapped as a single HandEyeInput entry per pose. You keep the capture script, the pose format and the pairing code; you change one solver call and re-home the output transform. What follows is the input structure, the capture steps, a replayed pose dataset with the arithmetic, and the checks that catch a bad result before the robot trusts it.

What changes between the two mountings

The mounting changes; the input stays the same. In eye-to-hand the camera is fixed and the target rides the robot. In eye-in-hand the camera rides the flange and the target is fixed in the cell. Either way the solver receives the same thing: robot pose plus detection, pose after pose. The mounting trade-off itself — stiffness, cable dress, coverage of the cell — is covered in eye-in-hand vs eye-to-hand camera mounting; here only the data matters.

For the fixed camera, rigidity pays twice: every micrometer a stand flexes lands directly in the eye-to-hand transform. A stand from camera stands such as the LS-MNT-0040 holds the camera still for the life of the cell.

One pose, one detection, one input entry

Each entry the solver takes is built the same way whatever the mounting:

HandEyeInput(robot_pose, detection_result)

• robot_pose — the 4×4 matrix the controller reports for that pose: a 3×3 rotation, a 3-element translation, 16 elements in total.
• detection_result — what the detector found in the single frame captured at that pose: the target points, in camera coordinates.

The solver then solves the classic hand-eye equation for the one constant transform:

In code, the whole difference between the two samples — sample_calibrate_eye_to_hand and sample_calibrate_eye_in_hand — is one call:

# Your data, collected once per pose:
robot_poses = []   # list of 4x4 pose matrices, one per robot pose
frames = []        # list of captured calibration frames, in the same order

inputs = []        # one entry per pose: what the solver receives
for pose, frame in zip(robot_poses, frames):
    detection_result = detect_target(frame)   # your detection step
    inputs.append(HandEyeInput(pose, detection_result))

result = calibrate_eye_in_hand(inputs)      # camera on the flange
# result = calibrate_eye_to_hand(inputs)    # camera fixed: the only change

# Pairing test: every pose has its frame, counts and order kept in one pass
assert len(inputs) == len(robot_poses) == len(frames)

Collecting the pairs, step by step

1. Fix the camera before the first frame. Mount it, lock focus and aperture, and leave them locked: intrinsics belong to the camera and lens, not the mounting, and they are only valid at the focus they were captured at — fix the pose first. Most area scan cameras with a global shutter, such as the MV-CH120-11UC, detect target corners cleanly at standstill; when the target is small or the standoff long, the MV-CS200-10UM puts 20 MP behind the same detection.

2. Log the flange frame, not a tool offset. Record the pose as the flange frame — tool0 in ROS-style robot descriptions — exactly as the controller reports it. A TCP with a tool offset baked in shifts the whole result by that offset; match tool0 to the sample shows the failure this produces.

3. Spread the poses. Two rotations about non-parallel axes solve A·X = X·B mathematically; detector noise is why you collect more. Take 15-25 poses for a production cell, varying all three rotation axes while keeping the target in view — pose selection: robot positions that work lists the geometry that conditions the solve well.

4. Capture one frame per pose, at standstill. Same exposure, same focus, robot settled. If the camera streams, one wait call per frame keeps frames and poses in step.

5. Detect and pair in capture order. The detection from a frame pairs with the pose you logged for it, entry for entry, in the same order.

6. Convert the pose if the controller reports quaternions. A pose given as position plus quaternion (x, y, z, w) becomes the 4×4 matrix before it enters the entry — convert a hand-eye quaternion to a transform matrix works the arithmetic.

7. Call the solver for your mounting. calibrate_eye_in_hand(inputs) for the camera on the flange, calibrate_eye_to_hand(inputs) for the fixed camera. The same inputs list either way.

With a 3D sensor doing the capturing — a stereo structured-light unit such as the MV-DLS300P-04 — the frame arrives as a point cloud and the detector works on the target points in 3D rather than image corners, while the pairing stays identical: one detection per pose, in capture order.

Worked example: replaying a 101-pose eye-to-hand set on the flange

The case: an eye-to-hand calibration exists — camera on a stand, target on the robot — and the camera now moves to the flange. A published evaluation of hand-eye calibration techniques acquired 101 robot poses with 1280 × 720 pixel images; use that as the dataset.

Inputs

• 101 saved robot poses, 4×4 matrices, from the earlier acquisition.
• Frame size 1280 × 720 pixels, as captured by the camera then in use.
• The rule: every pose replays unchanged; every frame is captured again, because the old frames were taken by the fixed camera and contain no flange-to-camera motion.

Arithmetic

• Input entries: 101 poses give 101 entries, one per pose.
• Pose data kept: 101 × 16 = 1,616 matrix elements; at 8 bytes each, 1,616 × 8 = 12,928 bytes, about 12.9 kB — the part of the old dataset you keep verbatim.
• Frames to recapture: 1,280 × 720 = 921,600 pixels per frame; × 101 = 93,081,600 pixels, about 93 MB of new image data at one byte per pixel.

Result and decision

Keep the pose file, mount the camera on the flange, and replay the poses one by one while capturing fresh frames. The sample once, replay workflow exists for exactly this, and dataset acquisition with saved robot poses covers storing them so a replay is possible. If flange reach or cable dress blocks some poses, keep at least the 20 with the widest rotation spread and drop the rest; below that, watch the residual rather than trusting the transform.

Verify it before the robot trusts it

Run the check on the held-out pose: command the robot there, detect the target, and transform a target point through the live flange pose plus the result into the robot base. Then put a pointer TCP physically on that same target point and compare. Commit to a number before you start: the gap should sit under a third of your pick tolerance. Above that, recollect with more rotation variety — re-running the solver on the same pairs will not close it. Hand-eye calibration accuracy verification: test before you trust it expands this into a full acceptance test.

Common mistakes

• Empty or failed detections. Target partly out of view, or exposure drifted between poses. Check what the detector actually received: hand-eye calibration target frame missing: what to check.
• Solver crash or refusal. Most often a malformed 4×4 — a quaternion converted wrong leaves a matrix that is not a rotation — or too few usable pairs: diagnose the sudden shutdown.
• Pairs out of order. Two capture sessions merged, or frames and poses logged in different orders. The assertion in the code above catches the count, not the order — keep one list, appended to in capture order.
• Wrong pose frame. A tool-offset TCP logged instead of the flange shifts the result by exactly that offset on every pose alike, so nothing looks wrong until the first pick misses.

When the shared input does not save you

The inputs are shared; the physics is not. Three cases where the shortcut stops:

• You physically moved the camera. The old frames are worthless for the new mounting; only the poses replay. That is the worked example above.
• A fixed-camera dataset fed to calibrate_eye_in_hand. Those frames contain no flange-to-camera motion, so the solve fails or returns a confident wrong answer. Recapture on the flange.

• The task never needed hand-eye. For planar parts picked from one known plane, nine-point alignment is often enough and needs no pose logging at all. An upward-facing camera on the arm is the other special case: the pose set must keep the fixed target in the upper field at every pose.

After the solve: where each result lives

Eye in hand returns the camera-to-flange transform: apply it at runtime with the current robot pose to land camera-frame points in the base — object pose in camera frame at runtime shows the chain. Eye to hand returns the camera-to-base transform, one static value you publish once: publish eye-to-hand calibration as a static camera transform.

Checklist before the solver runs

• Camera mounted where it will stay, focus and aperture locked.
• Every pose logged as the flange frame, one convention throughout.
• Pose set spread over all three rotation axes, target in view at every pose.
• One frame per pose, captured at standstill, exposure unchanged.
• Detection result paired with its pose, in capture order.
• One held-out pose reserved for the acceptance check.
• Solver call matching the mounting: calibrate_eye_in_hand on the flange, calibrate_eye_to_hand on the stand.

The decision, per case: camera on the flange, eye in hand, transform applied per pose. Camera fixed, eye to hand, static transform. Camera moved between the two, replay the poses and recapture every frame. Planar picks from one plane, nine-point alignment and no hand-eye dataset at all.

To build the camera side of either set-up, start at the camera step of the configurator or the camera finder. For a second opinion on a dataset, the contact page takes the pose count and camera model. More integration write-ups sit on the systems topic page.