Hand-eye calibration pose capture workflow: sample once, replay

The auto-calibration buttons in a robot hand-eye GUI look like a one-click calibration, and on a brand-new cell they deliver nothing: they replay joint states you recorded earlier, so an empty pose list gives them nothing to plan and nothing to execute. The fix is one disciplined manual pass — 12 to 20 poses, each stored with its image — after which every later recalibration is a replay that finishes in minutes. This workflow is for integrators commissioning cells with fixed or wrist-mounted 3D cameras, and it ends with the checks that decide whether a stored pose list can be trusted.

How many poses does a hand-eye calibration need?

Capture 12 to 20 poses the first time you calibrate a cell: the solver must find six transform values, and with fewer than about 10 poses the random error in the result is roughly twice what the same hardware delivers with 20.

Each robot pose contributes one observation of the unknown camera-to-robot transform, and the random part of the estimate shrinks with the square root of the pose count. That scaling is why five poses — the minimum some replay buttons need before they do anything — is not an accuracy target:

Twenty poses cut the random error to half of a five-pose result on identical hardware. Past about 25 poses the curve flattens: five more poses buy roughly another 10%, while capture time grows linearly. Spend the extra effort on spread and tilt instead, because those fix systematic errors that no pose count removes.

Where should the poses sit in the working volume?

Spread the poses over at least half of the volume the camera must actually serve — for a bench cell with a 600 mm square field, that means poses across at least a 300 mm span, on both sides of centre and at two heights.

An orientation error couples into the translation estimate in proportion to distance: at a 500 mm standoff, a 0.5° camera orientation error displaces the apparent position by 500 mm × tan 0.5° = 4.4 mm. Only poses that view the target from widely separated directions give the solver the leverage to pull rotation apart from translation, and a tight cluster cannot. Plan the set geometrically before you jog anywhere:

Two or more working heights matter as much as width: a set sampled on a single plane leaves the height axis coupled to tilt, and the solver compensates with a skewed translation. Cover the full pick width on a conveyor cell; cover at least half the pallet footprint on a palletising cell.

How much should the target tilt at each pose?

Tilt the target roughly 30° off the camera axis in at least four different directions, and include one nearly head-on view, because a planar target that always faces the camera gives the solver almost no leverage to separate target tilt from its position.

A planar target seen head-on projects almost no perspective, so rotations of the target about axes in its own plane stay nearly invisible; tilted views create the foreshortening that pins those rotations down. One head-on view anchors scale and in-plane rotation; the tilted views cover the rest. Mind the depth of field as you tilt, because the board itself gains depth:

Beyond about 45-60° of tilt the foreshortened corners stop resolving cleanly and the depth span can outrun the sensor's depth of field, so keep planned tilts in a 20-45° band unless the camera's depth of field is known to be deep. Vary the tilt direction pose by pose: four tilts leaning the same way are worth one.

What do you need in place before the first capture?

Fix the camera and the target so nothing moves between poses, lock focus and exposure, and confirm the robot's kinematics are calibrated first, because the pose list records the geometry of this exact set-up and replay inherits every looseness you leave in it.

Decide the mount convention before anything is bolted: camera fixed in the cell with the target on the wrist is eye-to-hand, camera on the wrist with the target fixed is eye-in-hand, and the pose bookkeeping differs between them. The eye-in-hand calibration pose selection guide walks through choosing wrist poses, the frame-matching checklist prevents the base-versus-tool mix-up behind rotated point clouds, and the target sizing guide sets how big the board must be for your standoff. A camera stand holds the fixed side rigidly; the LS-MNT-0040 gives a 700 mm column on a 400 x 450 mm base for a bench eye-to-hand rig.

Run the capture GUI on one of the vision controllers built for the job: the MV-VC5525-256G70-NN provides PoE camera ports and 32 GB of memory, which covers the camera link and the stored pose list plus images on the machine that drives the cell.

The capture discipline fits any of the 3D sensors that return one target observation per pose — projected structured-light cameras, galvanometer stereo heads and laser profilers alike.

How do you capture the pose list?

Jog the robot to each planned pose, confirm the target fills the frame with every corner resolved, and save the joint state and the image as one named pair; repeat until 12 to 20 pairs exist, and only then touch the auto buttons.

1. Write the goal poses as joint targets, not bare Cartesian points, so replay reproduces the exact arm posture, and keep every pose at least 5° from a joint limit.
2. Jog to pose 1 and frame the target: every marker corner resolved, no clipped highlights, board inside the frame with margin on all sides.
3. Save one record per pose — pose name, joint state, image — in one folder under one naming scheme.
4. Repeat for the remaining poses and log the tilt direction beside each pose number on the run sheet.
5. Leave Plan and Execute alone until every pose is saved: five samples make replay possible, twelve to twenty make it accurate.
6. Read two or three stored joint states back and compare them digit for digit with the run sheet; a round-trip mismatch becomes a mysteriously off pose during replay.
7. Run the solve on the manual set and note the residuals before you archive anything.
8. Export the pose list together with the cell configuration and a date in the file name.

The saved-robot-pose dataset method automates the pairing step when your stack already logs poses. When the arm is driven from an offline-programming package rather than the pendant, the same loop applies; a hand-eye calibration walkthrough for a collaborative robot in simulation software shows the sequence configured end to end.

Why do Plan and Execute do nothing on a first-time calibration?

Those buttons replay joint states you have already recorded: with an empty list there is no stored pose to plan to and none to execute, which is why they sit enabled but do nothing until about five manual samples exist.

Replay is a convenience for repeating a calibration, not a generator for making a new one. Plan searches a motion path to a stored joint state, Execute drives there and captures, and both depend entirely on what you sampled by hand. They also fail in a second way on older lists: a pose recorded before the cell was retooled can plan into a collision or lose the target. One-click generation of a pose list from a single view of the target is a research direction in the literature, not a button in shipping tools, so plan on one manual capture per cell and treat replay as the payoff.

How do you check a stored pose list before you trust replay?

Open every stored image before replaying: a pose whose target clips the frame edge or blurs will replay exactly as badly as it was captured, and the solver will happily average that pair in with the good ones.

Work the list once, top to bottom:

• Every stored image shows the whole target with roughly 10% of the frame width to spare on each side, so replay's few tenths of a millimetre cannot clip a corner.
• Every joint state sits inside the software limits with the 5° margin from capture step 1.
• No two poses repeat each other: pairs within about 2° in every joint add nothing to the solve, so drop one.
• The sample counter advances exactly once per executed pose; a stuck counter means the capture click never landed.
• A dry-run replay without solving reaches every pose without a fault, and no approach move crawls the way a near-limit posture does.

When should you recapture instead of replaying?

Recapture the whole list whenever the camera has been unbolted, the target swapped, or the robot re-based and re-mastered, because replay then drives to motions that are still valid but calibrate against a changed geometry — a confidently wrong result.

Recapture when the camera bracket moved, the lens refocused or the sensor's internal calibration was updated, the target board was reprinted or replaced, the robot was re-mastered after a collision, or the cell was relocated. Replay is safe when only software changed: a GUI update, a new solver version, a re-run on untouched mechanics. Between those cases one reference check decides — run a single replay pose and compare the live image with the stored one; agreement within about 1 mm says the mechanics are untouched.

How do you confirm the result is accurate?

Verify with one physical test: command the tool centre point onto a target feature and measure the mismatch, because at a 500 mm standoff a rotation error of 0.1° already displaces the touch point by 500 mm × tan 0.1° = 0.87 mm.

Sub-millimetre picking therefore needs rotation accuracy well under 0.1°, and pose spread is what buys it. Test at three standoffs — near, middle and far edge of the volume. Error that grows in proportion to distance means rotation residual; a constant offset at every distance means translation residual. The accuracy verification procedure expands this into a full test sequence, runtime pose transforms cover the frame chain from a detected feature to a base-frame pick point, and the solver's residuals are read against the same average-and-maximum targets as camera calibration residuals. For the pixel-to-base coordinate chain on a cobot cell, a pixel-to-robot coordinate transform walkthrough shows the full path.

The decisions in one pass: a new cell gets about 15 manual captures across half the working volume with four tilt directions; an unchanged cell gets replay; any disturbed bolt gets a recapture. To size the hardware around that, start at the 3D sensor finder or configure a vision system with the pose count and standoff you now know.