Smart camera hand-eye calibration when no wizard exists

You have a smart camera bolted to the robot wrist, the vendor's environment offers no hand-eye calibration wizard, and the deadline did not move. The fix is smaller than it looks: the smart camera already measures a target to a fraction of a pixel and reports the position over TCP or the PLC. Log 12 to 15 robot flange poses at a fixed target, and a short external least-squares solve returns the flange-to-camera transform to picking accuracy. The route below covers the hardware, the framing numbers, the pose set, the solver math with worked arithmetic, and the checks that tell you the answer is good.

Can you calibrate an eye-in-hand smart camera with no vendor wizard?

Yes. The pattern-match centroids the smart camera already reports, paired with 12 to 15 logged flange poses, give you every number an onboard wizard would have collected. A small least-squares solve then returns the flange-to-camera transform and the target's position together. A wizard is only automation: it moves the robot, records the flange pose and the target's image position, and hands both to a solver — three things you can do yourself in an afternoon.

The classic formulation, AX = XB, relates the flange motion A (from the controller) to the camera motion B (from the images). With a single fixed target and tool-level readings, you fold the target's unknown position into the same fit. If your model streams full frames over GigE or USB, take the easier road: recover the target's full pose per image with a PnP solver, then feed pose pairs to a hand-eye solver such as Tsai-Lenz — robot mounted camera calibration: capture and solve walks that route end to end. This page is for the other case: the camera gives you a match position and nothing else.

What hardware do you need on the cell?

A fixed-lens mono smart camera, a matte dot-grid target, a stand the target cannot move on, and a lens focus ring locked after focusing. Four items — and the last one quietly decides the calibration's lifetime. Start with smart cameras in the camera finder: you want a fixed lens, mono output, and a focus ring you can lock. Two models bracket the choice: the MV-SC5060XM-08M-WBN (6 MP mono, 8 mm focal length, M12-mount) and the color MV-SCB007EC-12S-WBN (1.6 MP, 12 mm lens, manual focus).

All 2 share: Humidity 20% RH to 95% RH (no condensation); Certification CE, KC. Values as printed in each product's data sheet; open a part for its drawing and full specifications.

Mono wins this job: black dots on white score higher with no Bayer mosaic in the way, and a rolling shutter costs nothing when every pose is captured at a standstill. If you would rather assemble the whole station from parts, the configurator, starting at the camera step, sequences it for you.

The target is a dot grid on a matte carrier — a symmetric dot gives the pattern-match tool a feature whose center sits still under mild defocus. Calibration targets include boards sized for exactly this; the LS-LXR-0029 carries 0.5 mm dots across a 76 x 76 mm grid, a good match for a wrist-mounted camera at close range.

Fix the board where the robot can orbit it and nothing else can touch it. A stand with a 400 x 450 mm base, like the LS-MNT-0040 at 700 mm height, holds the board rigid through the whole pose set. Camera stands and brackets live in the accessories finder.

Last, focus the lens once at the working distance, check the corners for field curvature, then lock the ring. M12 lens focusing: adjust, check corners and lock covers the drill; a refocus after calibration changes the effective focal length and quietly bends every pick that follows.

How do you frame the target and set the exposure?

Work at about 400 mm from lens to board, where a 76 mm target fills roughly a fifth of the field of view, and set a 15 ms exposure at minimum gain — the robot stands still, so motion blur is off the table. The arithmetic, for a 3072 x 2048 sensor with 2.2 µm pixels behind an 8 mm lens: horizontal field = 3072 × 0.0022 × 400 / 8 = 338 mm. The 76 mm board then covers 76 / 338 × 3072 = 690 px, ample margin for a subpixel centroid. Run the same arithmetic in reverse — from a wanted field of view to a focal length — with the calculator:

Exposure of 15 ms with gain at minimum keeps photon noise low, and because the arm is stopped there is no blur penalty for it. Fixed-aperture M12 lenses run near f/2.0 to f/2.8, so depth of field is not a knob you turn: at f/2.4, focused at 400 mm, the one-pixel-blur band spans 2 × 2.4 × 0.0022 × (400 / 8)² ≈ 26 mm. Keep poses within about ±60 mm of the focus distance anyway, so the match score stays high.

Lighting: a model with built-in illumination removes a cable — the LS-SMC-0063 carries a 16 mm lens and its own polarized white light. Polarization takes the shine off a glossy carrier. Otherwise a small diffuse source beside the lens does the job, and a shroud keeps factory ambient out of the exposure.

Which robot poses does the solver need?

Twelve to fifteen stopped poses, with flange rotations of 30-40° spread over at least two rotation axes, all aimed at the one fixed target. Pure translations, or rotations about a single axis, leave the focal length, the target distance and the camera offset entangled. Rotation is what makes the unknowns observable: when the flange only translates, the solver can trade a longer camera lever arm against a nearer target and fit your pixels equally badly several ways. The full planning rules are in eye-in-hand calibration pose selection: robot positions that work.

Capture the set like this:

1. Fix the board to the stand inside the robot's reach, matte side out, roughly normal to where the camera will sit mid-set.
2. Teach the pattern-match job on the smart camera: region over the dot grid, acceptance score 90, subpixel output on.
3. Wire the reading out as a TCP string or a block of PLC registers, timestamped against the flange pose. One command code per job keeps the parse unambiguous — the pattern in RS232 vision recipe selection: one command code per recipe applies to TCP jobs too.
4. Set exposure to 15 ms and gain to minimum, and switch off any auto-exposure so the score does not wander mid-run.
5. At a reference pose, trigger twice and compare the two readings — that gap is your centroid noise, and it should sit near 0.1 px.
6. For each pose: jog, stop, wait half a second for settle, trigger, and log the flange pose in mm and degrees beside the u, v reading.
7. Work the rotation plan: four to five poses stepping flange Z by 30-40°, four to five with the wrist tilted 30-40° about X or Y, then two or three mixed poses.
8. Return to the reference pose last. Its reading must match the first within the step-5 noise, or the target moved and the set is void.

How do you solve AX=XB from centroids alone?

Fit nine unknowns — six for the flange-to-camera transform X and three for the target point P in base coordinates — by non-linear least squares against the logged pixel readings. Twelve poses supply twenty-four equations, roughly three times the unknowns. The model is four lines:

T_bf(i) is the flange pose the controller reports; fx, fy, cx, cy are the camera intrinsics; u_meas, v_meas are your logged centroids. Any Levenberg-Marquardt routine — scipy's least_squares or the equivalent in your math library — carries the fit. Seed it with the camera offset tape-measured from the flange center (within ±20 mm) and its rotation eyeballed (within ±15°), plus your best guess at the board position. That is close enough, and convergence takes a handful of iterations.

Worked numbers, using the 8 mm mono unit above on a 2.2 µm, 3072 x 2048 sensor:

• Intrinsics: fx = fy = 8 / 0.0022 = 3636 px; cx = 3072 / 2 = 1536; cy = 2048 / 2 = 1024.
• Data: 12 poses give 24 equations against 9 unknowns.
• Fit: converged RMS residual 0.4 px; the typical band is 0.3-0.7 px, and anything above 2 px signals a bad pose or a moved target.
• Accuracy: a 0.2° rotation error at a 600 mm reach moves a pick by 600 × 0.0035 = 2.1 mm; add 0.3-0.5 mm of translation residual and the budget lands at 1-3 mm.

Two refinements earn their keep. Make fx a tenth unknown if the pixel pitch is not published — the rotation-spread pose set makes it observable. And weight each reading by its match score, so one soft pose cannot drag the fit.

At run time, X turns a match reading into a pick point: send the target ray through T_bf · X into base coordinates and intersect it with the plane of the table — the same height-known trick any fixed smart camera uses. The transform chain is laid out for a static camera in using eye-to-hand calibration results to compute robot-base picks; on the wrist, X simply rides along. A worked example of the same pixel-to-robot arithmetic on an arm-mounted camera is in this pixel-to-robot coordinate transform walkthrough.

How do you check the calibration before the first autonomous pick?

Hold three poses out of the fit, demand under 0.5 px RMS on poses the solver never saw, and require two different flange orientations to place the target within 2 mm of each other in base coordinates — ten minutes, three pass/fail gates.

• Repeat-pose noise: the step-5 double trigger should sit near 0.1 px. Above 0.3 px, fix lighting or the acceptance score before blaming the solver.
• Held-out residual: fit on twelve poses, predict the other three. Held-out RMS above twice the fitted RMS means a pose is bad or the target moved mid-run.
• Two-orientation agreement: solve the target's base position from two disjoint pose subsets; the answers should agree within 2 mm.
• End-to-end: drive the tool to the computed target position in two different wrist orientations; the gripper tip should land on the same physical point within 1-2 mm.

If picks repeat well but carry one constant offset in base coordinates, the calibration is sound and a taught offset closes the last millimeter — diagnose an eye-in-hand pick point offset separates the cases.

When is the centroid route the wrong choice?

Take a different road when the camera can stream full frames, when every part lies in one known plane, or when picks must land inside 0.5 mm. Each case has a tool better than a single-point least squares. With frames available, full images let you measure the target's complete pose per shot, feed a classical hand-eye solver, and refine intrinsics inside the loop. With planar parts and a normal viewing angle, robot hand-eye calibration: how nine-point alignment works gets you picking with no solver at all. Below 0.5 mm, a single-point fit bottoms out around 1 mm: teach a final offset on the real part, or move to frame-based calibration. And if your cell already programs robots offline, hand-eye calibration driven from an offline robot tool runs the same capture on that path.