Hand-eye calibration without ROS: use OpenCV and ViSP directly

Your arm's controller speaks its own language and offers no ROS support, and the hand-eye tooling you found wants MoveIt to drive the motions. You can skip that layer entirely. Hand-eye calibration is a solver problem: its inputs are pairs of robot flange poses and camera observations of a fixed target, and OpenCV or ViSP solves them in one function call. This guide is for anyone bolting a camera to a robot whose controller has no ROS. It shows what to record at each pose, how to run cv::calibrateHandEye on your own data, how to size the framing so the target reads cleanly, and how to verify the result to a few millimetres before the first pick.

The problem AX = XB solves

Hand-eye calibration computes the fixed transform between the robot's frame and the camera's frame. Between two stations the robot measures one rigid motion of the flange, and the camera sees the same motion as a shift of the target in the image. One motion, two measurements, and the unknown mount transform X ties them together:

Because only relative motions enter the equation, the method runs on any arm that reports full Cartesian flange poses, ROS or not. What it cannot do is invent poses: you supply them.

Two prerequisites come first. The camera needs intrinsics from an ordinary chessboard calibration, because every target pose comes from solvePnP — with the mean reprojection error below 0.3 px those intrinsics are solid enough to build on. And the target must be rigid, matte and flat, mounted so it cannot creep relative to the bench or the flange. The pose-side details are in camera intrinsics for hand-eye calibration: fix the pose first, and camera calibration residual values: average and max targets sets the quality bar for that first calibration.

The decision: the solver never needed ROS

One-click hand-eye tools are thin wrappers. The calibration half calls solvers already published in OpenCV and ViSP; only the optional automatic-motion half depends on ROS, because it generates trajectories through MoveIt. Drop that half and nothing of value is lost: jog the arm with its own pendant, teach a point list, or script the controller's native API, capture the pose pairs, and solve.

The finished product is X as a 4 × 4 matrix that your pick code multiplies through. In a ROS cell that matrix would be published as a TF frame, and that publishing step is the only part you actually skip — publishing an ArUco marker pose as a TF frame shows the ROS-side hand-off for comparison. The identical function exists in Python as cv2.calibrateHandEye, so the whole loop fits in one script plus the controller's jog commands.

Eye-in-hand or eye-to-hand: same solver, different X

Mounting decides what X means, not how you solve it.

Eye-in-hand: the camera rides the flange and the target is bolted to the bench. The solve returns flange-to-camera, and at pick time base-to-flange × X × target-to-camera lands the part in robot coordinates.

Eye-to-hand: the camera is fixed and the target rides the flange on a pointer or gripper. Feed the same call inverted flange poses and it returns the camera-to-base transform a fixed-camera cell needs.

For the eye-to-hand bench, fix the camera to hardware that cannot creep:

The slide-table stand in the card has a 700 mm column and a 400 × 450 mm base that bolts straight to the bench, and lighter brackets live in the same camera stands range. The trade-offs of the two mountings — reach, occlusion, accuracy at the part — are compared in eye-in-hand vs eye-to-hand camera mounting for robots.

What to record at every pose

Each station yields exactly one pair. Record at standstill, after a short settle, and keep both lists in the same order:

1. Flange pose from the controller — X, Y, Z and orientation in the flange frame, not a tool-offset TCP. Save it as a rotation matrix or quaternion, and write down whether the quaternion is scalar-first or scalar-last; the wrong slot still calibrates and fails silently.
2. One image of the target at a fixed exposure and a locked focus. If the lens can hunt between stations, lock the focus first.
3. Detection output — find the grid with cv::findChessboardCorners, refine the corners, then solvePnP with your intrinsics to produce R_target2cam and t_target2cam.
4. A note of the axis you rotated about, so degenerate motion pairs are visible before the solve rather than after.

How many stations: two motions about different, non-parallel rotation axes are the mathematical floor — three poses — and every accuracy study of AX = XB shows the error falling as motions are added. Plan 10 to 20 stations, rotating 30 to 60° about a fresh axis each move, with translations spread across the volume you will actually pick from. Two habits wreck datasets: pure translation moves carry no rotation information, and consecutive rotations sharing one axis blind the solver in that degree of freedom. The pose-capture workflow and calibration dataset acquisition with saved robot poses articles show this capture loop in detail.

Solve it with cv::calibrateHandEye

With the vectors filled and in matching order, the solve is one call:

// one entry per station; all CV_64F, rotation 3x3, translation 3x1
std::vector<cv::Mat> R_gripper2base, t_gripper2base; // flange poses from the robot log
std::vector<cv::Mat> R_target2cam, t_target2cam; // solvePnP results, same order

cv::Mat R_cam2gripper = cv::Mat::eye(3, 3, CV_64F); // solved here
cv::Mat t_cam2gripper = cv::Mat::zeros(3, 1, CV_64F);

cv::calibrateHandEye(R_gripper2base, t_gripper2base,
 R_target2cam, t_target2cam,
 R_cam2gripper, t_cam2gripper,
 cv::CALIB_HAND_EYE_TSAI);

Feed every translation in one unit — if the controller reports millimetres, scale each solvePnP tvec to millimetres too — and t_cam2gripper comes back in that unit. The method flag selects the published solver: CALIB_HAND_EYE_TSAI, CALIB_HAND_EYE_PARK, CALIB_HAND_EYE_HORAUD, CALIB_HAND_EYE_ANDREFF or CALIB_HAND_EYE_DANIILIDIS. With clean data the five agree to within your measurement noise; run TSAI and PARK and compare, and if they differ by more than a few tenths of a degree or a millimetre, treat it as a data problem rather than a method problem. ViSP wraps the same published methods behind its hand-eye calibration interface, so either library carries the solve.

A 3D camera skips the PnP half. A projected structured-light camera such as the one below returns a pose, or a registered cloud, straight from its SDK, and that feeds the same AX = XB call unchanged. What the data sheet's precision and trueness figures mean for your residual is decoded in 3D camera specs explained: precision, trueness and spatial resolution.

Worked example: framing a 100 mm target at 400 mm with a 1-inch sensor

The inputs, and where each comes from:

• Target: a printed chessboard, a 12 × 9 grid of 5 mm squares on a 100 × 80 mm plate — a standard bench target.
• Capture pose: flange-to-target distance of 400 mm, chosen to keep the camera clear of the fixture.
• Sensor: 1-inch format, 3:2 aspect, so an active area of about 13.2 × 8.8 mm under the 1-inch 3:2 convention.
• Camera: a 20 MP mono USB3 model with a 1-inch-format sensor on a C-mount, such as the card below.

The rule in words: magnification is the share of the scene the sensor covers, and the thin-lens relation converts a wanted field of view plus a working distance into a focal length.

Aim for the plate to fill about half the frame width, so FOV = 100 / 0.5 = 200 mm. Then m = 13.2 / 200 = 0.066 and f = 400 × 0.066 / 1.066 = 24.8 mm. Fit a 25 mm C-mount prime and check it: FOV_w = 13.2 × (400 − 25) / 25 = 198 mm and FOV_h = 8.8 × 15 = 132 mm. The plate then fills 100 / 198 = 51% of the width and 80 / 132 = 61% of the height — inside the one-third to two-thirds window where the grid reads reliably and orientation changes still leave margin.

Pixel scale: a 20 MP sensor in the 3:2 aspect is sqrt(20,000,000 × 1.5) ≈ 5,477 pixels wide, so 5,477 pixels over 198 mm gives 198 / 5,477 = 0.036 mm per pixel on target. Each 5 mm square spans 5 / 0.036 ≈ 139 pixels, far above the roughly 20 pixels at which corner detection starts to degrade. The distance has slack too: FOV_w = 0.528 × (WD − 25), so every 50 mm of extra working distance widens the frame by about 26 mm, and the plate stays inside the window anywhere between about 310 and 590 mm.

Decision: fit a 25 mm C-mount prime, capture at 400 mm, and treat 310-590 mm as the tolerated band before the framing must be recomputed. The focal-length calculator below redoes these two lines for your own target and distance.

Verify the result before the first pick

Hold out two or three stations from the solve. For each, compute the target origin in robot coordinates through the camera chain — base-to-flange from the robot, times X, times target-to-camera from the image — and compare it with an independent reference such as a pointer tip touched to a marked plate corner:

cv::Mat e = p_base_camera_chain - p_base_touch_off; // each 3x1, mm
double residual_mm = cv::norm(e); // example result: 1.4

Bands to work to on a bench cell: a repeatable industrial six-axis arm holds 0.02-0.05 mm, so the robot is not the bottleneck; image noise sets the floor. Agreement under 2 mm passes, fine enough even for small-part assembly. 2-5 mm still supports palletizing and bin picking, but not tight assembly. Above 5 mm, stop and fix the data collection. And if the mismatch lands at 10-20 mm, that is not noise — that magnitude is the signature of a frame or quaternion convention mistake, the case in hand-eye calibration translation error: is 1-2 cm normal?. The full test sequence is walked through in hand-eye calibration accuracy verification: test before you trust it, and using eye-to-hand calibration results to compute robot-base picks covers the multiply-through at pick time.

When the usual answer is wrong

The recorded-pose method assumes the controller reports where the flange is. If it offers only teach-by-jogging with no Cartesian readout, AX = XB has no A to solve with. Check first whether the controller already ships a built-in pixel-to-robot calibration routine — pixel-to-robot workflow built into one collaborative-robot controller shows what those cover — or drive the stations from a hand-eye set-up configured in an offline robot programming tool, which can jog supported arms through a taught list and log the poses.

Four more cases where the standard advice misleads:

• Sloppy arms: backlash or belt drive can push pose noise past half a millimetre; the residual then reflects the arm, and no solver method changes that.
• Rolling shutter plus vibration: shoot only at standstill, a few hundred milliseconds after motion stops, or use a global-shutter camera for capture.
• Focus and thermal drift: a lens that shifts with temperature moves your intrinsics between morning and afternoon — camera focus drift with temperature: diagnose daytime blur gives the diagnosis.
• Re-mounting: X belongs to the mount. A bumped camera, a lens swap or a crash invalidates it even when everything looks straight again.

Checks before you run the solve

• Intrinsics: mean reprojection below 0.3 px on a fresh chessboard calibration.
• Optics: focus locked; exposure and aperture fixed for the whole session.
• Target: rigid, matte, flat; bolted to the bench for eye-in-hand and to the flange for eye-to-hand.
• Robot data: flange poses in one frame, one unit, one quaternion convention, written down.
• Stations: at least 10, rotation axes mixed, no shared-axis pairs, no translation-only moves.
• Solve: two methods compared; held-out residual under 2 mm before anything picks a part.

Camera-side parts come from the area scan cameras range — mono, C-mount, global shutter preferred for captures at unknown moments — narrowed by interface and resolution in the camera finder; stands and brackets are in the accessories finder.

The short version

Record 10-20 flange-plus-image pairs with varied rotations, solve AX = XB with cv::calibrateHandEye in one call, check a held-out residual under 2 mm, and keep X as a matrix in your own pick code. No ROS anywhere in the loop. If the job needs depth rather than a planar target, the 3D sensors range is the starting point, and the 3D vision guides collect the rest of the calibration and integration topics. To spec camera, lens and mount together, run the configurator below.