Hand-eye calibration dataset acquisition with saved robot poses

A hand-eye calibration is only as good as its dataset, and the dataset is where most attempts go wrong: poses jogged by hand are slow to collect, hard to repeat and easy to mix up. The fix is to split the job. First, plan 10-20 camera poses offline on a digital twin of the cell and save every one as a joint target, so the arm re-enters the exact same configuration on every run. Then let a short script step through those targets, capture one 3D frame at each, and record the flange pose at the instant of capture. This guide is for anyone adding a 3D camera to a robot cell. It covers the joint-target rule, the pose layout with a worked tilt-ring example, the acquisition loop, and the residual and touch-test checks that tell you the transform is safe to use.

What you need

Five things on the bench before you plan a single pose:

• A 3D camera that suits the cell, one from each family covering most cases:

• A calibration board, checkerboard or coded markers, sized so it stays fully in frame at every pose. The target-size rule for hand-eye calibration boards gives the sizing math.
• Network details: the controller's IP address, the TCP port its driver listens on, and the pendant switched to remote mode. Put the camera on its own link; one DHCP server for the 3D camera keeps addressing stable.
• An offline robot programming tool holding a digital twin of your exact robot model.
• The camera settings file for the lighting at the calibration station, fixed for the whole run so failed frames stay easy to spot.

Save every pose as a joint target

A cartesian target stores a position and orientation and lets the controller choose any arm configuration that reaches it. Two different configurations can put the flange in the same spot with the wrist flipped over, and a wrist flip moves the camera. A joint target stores axis values instead, so the arm re-enters the identical configuration tonight and next month. That repeatability is the entire reason to save the dataset poses in advance.

Most offline tools treat joint versus cartesian as a per-target property in the target's context menu, and mark the two types differently. Check the mark on every target before you run: a single leftover cartesian target is enough to spoil a pose.

Connect before you plan. Enter the controller's IP and the driver's port in the connect panel, set remote mode on the pendant, and confirm the status reads connected. If the link stays down, suspect the workstation firewall before the robot: robot drivers are blocked by default on many systems, so add an inbound rule that names the driver program and scopes the remote address to the controller's IP on its TCP port. Leave the firewall itself enabled, because an open workstation exposes the whole cell network.

Plan 10-20 poses that cover angles, not just positions

Ten to twenty poses is the working range for a well-conditioned hand-eye solve: below about ten the solver sees too little change in viewing direction, and past twenty the accuracy gains flatten while the run grows. Spend the count on diversity, three elevations such as -30°, 0° and +30° crossed with five azimuths spaced 72° apart, and keep the camera pointed at the board with the board centered and fully in frame at every pose.

Rotate the camera about the board's center rather than translating past it: the standoff then stays constant across the whole dataset. Why that matters, and how to pick the distance band, is the subject of the 3D camera working distance and positioning guide. For pose-by-pose choices, which views add information and which repeat it, see the pose-selection guide for eye-in-hand calibration; at close range, pair it with the short-range target and standoff guide.

Worked example: the tilt ring for a 300 mm board at an 800 mm standoff

Inputs: standoff d = 800 mm, taken as the middle of a typical bench structured-light camera's working band, so substitute the middle of your camera's own range; tilt span ±30°; a 300 mm wide board; 15 targets, three elevations crossed with five azimuths 72° apart.

The rule: point the camera at the board's center and rotate about that point. The camera-to-board distance then stays at d while the camera center traces a circle of radius d·sin θ at height d·cos θ above the board plane.

At θ = 30°: r = 800 mm × sin 30° = 800 × 0.500 = 400 mm, and h = 800 mm × cos 30° = 800 × 0.866 = 693 mm. At θ = 0° the offset is 0 mm and the height 800 mm. The 15-target set therefore lives on a 400 mm ring between 693 mm and 800 mm above the board.

The decision: a cell with ±450 mm of travel around the board takes the ±30° ring with a 50 mm margin. If the arm reaches only ±300 mm, cap the tilt at arcsin(300/800) ≈ 22° and hold the pose count by widening to seven azimuths. Before you store each target, confirm with a live capture that the 300 mm board sits centered and fully inside the frame at that pose.

The acquisition loop

1. Connect in remote mode and prove the link is live: jog one axis a few degrees and watch the twin follow.
2. Slow the arm to about 100 mm/s linear speed and 50 mm/s² linear acceleration. A calibration run buys repeatability, not cycle time, and a slow approach makes the settle window obvious.
3. For each saved target, command the joint move, wait for the controller's in-position signal, or dwell 500 ms if your loop has no such signal, then capture one 3D frame and read the flange pose at that same instant.
4. Save the frame and the pose together under the target's name, in one dated folder per run. The pair must come from the same instant; a frame and a pose written at different moments poison the solution quietly.
5. Re-shoot any failed pose while the cell is still set up: clipped board, occlusion, dropped link.
6. Hand the frame-and-pose pairs to the solver with the mounting mode set.

Eye-in-hand means the camera rides the flange and the solver returns the camera-to-flange transform; eye-to-hand means the camera is fixed in the cell and the solver returns camera-to-base. The dataset itself is identical, only the mounting mode changes. For a fixed overhead camera, that eye-to-hand result feeds directly into computing robot-base picks from the calibration result; for a flange-mounted camera, the next stop is reading the object pose in the camera frame at runtime. Stereo viewing with a laser projector runs through the same loop: a laser galvanometer stereo camera such as the MV-DLS300P-04 captures at each saved pose like any 3D camera in this guide.

Read the residuals before you trust the transform

The solver returns the transform plus a residual per pose. Read uniformity first: the pose whose residual stands well above the rest is the re-shoot candidate, and the usual causes are a clipped board, the arm still settling, or a configuration flip. The absolute level tracks the arm's absolute accuracy and the pose spread: translation errors in the 1-2 cm class are common on uncalibrated industrial arms, as our hand-eye translation error guide sets out, and a field-calibrated robot with a wide tilt spread pushes the figure down.

The decision rule: tight, uniform residuals means go and verify. One outlier means re-shoot that pose. Every residual large means the kinematic model or the pose spread is the problem, not the solver, so re-check the loaded kinematic parameters and add tilt before you re-run.

Verify with a touch test before production

Set the pass threshold backwards from the process. A gripper entering a 40 mm deep bore machined with 5 mm clearance needs the tool point within about 5 mm of nominal, so that is the number the test must show. A glue bead with 1 mm tolerance needs sub-millimetre performance, which usually means calibrating the robot arm itself first, not re-shooting the camera dataset.

Then run the test: place the calibration object in the cell, capture fresh, compute a target point through the hand-eye transform, command the arm to touch it, and measure the miss. Repeat at three or more points spread across the workspace, because a transform that touches well at the board's center can drift at the cell's edge. The full procedure and its failure modes are in the hand-eye accuracy verification guide. For flat conveyor picks, where a camera-to-base transform plus a nine-point check is often enough, how nine-point alignment works covers the simpler path.

Common mistakes that ruin a dataset

• Cartesian targets left in the list: the arm reaches the pose through a different wrist configuration and the camera lands somewhere else. Convert every target to a joint target and re-run.
• Capturing on the move: frame and flange pose come from different instants. Wait for in-position, or dwell 500 ms.
• A clipped board: features cut off at the frame edge bias the detected board pose. Re-center and re-shoot that target.
• A flat spread: twenty near-frontal views solve worse than twelve spread over ±30° of tilt. Add tilt, not count.
• Targets created before the calibrated arm's kinematic parameters were loaded: every stored pose is biased. Load the parameters, recreate the targets, re-run.
• A bumped camera mount after calibration: the transform describes a mount that no longer exists. Re-run the dataset; with saved joint targets it costs minutes.
• A firewall rule that changed between set-up day and run day: scope the inbound rule to the robot's IP and port and leave it fixed for the campaign.

When the touch test passes, write the transform into the pick pipeline and keep the dataset folder: the saved joint targets turn every future re-calibration, a bumped mount, a new gripper, a moved fixture, into a short scripted re-run instead of a day on the floor.

Still choosing hardware for the cell? The camera finder filters the 3D sensors category by type and series.

Questions on a specific cell, mount, board size or pass threshold, reach us through the /contact/ page, and the 3D cameras topic page collects the rest of the pipeline.