Object pose in camera frame at runtime after hand-eye calibration

Plate-based hand-eye calibration ends with a set of static transforms, and every worked example then quietly assumes the object pose is already known: it sets ObjInCamPose equal to CalObjInCamPose — the pose measured while calibrating — and composes it with CamInBasePose to reach ObjInBasePose. On a real line, nothing holds a part in that pose, so ObjInCamPose must be measured from every image, at cycle rate. This guide shows the three ways to do that — solvePnP on known object points, a planar 2D match at a fixed height, or a 3D sensor — then sizes the camera and lens for the solve, composes the transforms down to a grasp point, and lists the checks that catch a bad pose before the robot follows it.

What does hand-eye calibration leave unsolved?

Plate-based hand-eye calibration delivers one static transform — CamInBasePose, or BaseInCamPose before you invert it — that stays valid until the camera mount moves; the pose of the actual part in the camera, ObjInCamPose, is not part of it and must be measured from every exposure.

The calibration plate only had a known pose while you were calibrating: CalObjInCamPose records where the plate sat during the run, and it is finished the moment the solve converges. What survives is the intrinsic matrix with its distortion model, and one extrinsic chain. In the stationary-camera arrangement you also get the plate pose in the tool frame, ObjInToolPose, which proves the chain closed but says nothing about production parts.

Two wrong turns are common here. One is to treat the extrinsics as calibration-only: CamInBasePose is used every cycle, forever. The other is the single pixel-to-robot matrix, which bakes intrinsics, plane height and hand-eye into one homography — correct while parts lie on that exact plane, silently wrong the moment they do not.

If the camera rides the wrist instead, the fixed output is the camera pose in the flange frame, and you compose it with the live flange pose the controller reports. The runtime problem is identical: something still has to measure the part in the camera, every image. The full chain is laid out in eye to hand object pose transform for robot picks, and publishing eye-to-hand calibration as a static camera transform shows the clean way to serve it to the robot. If your calibration is not that far yet, a hand-eye calibration configuration walkthrough runs the whole procedure on a real cell.

How do I measure the object pose in the camera frame at runtime?

Run a perspective-n-point solve: detect at least four points whose 3D coordinates you know in an object frame on the part, project them through the intrinsic matrix from the plate calibration, and the solver returns the rotation and translation — the ObjInCamPose — that places the object frame in the camera frame.

1. Model the part once: choose four to ten point features — machined holes, corners, or applied fiducials — and write down their 3D coordinates in one object frame. Spread them wide across the part; never three in a line.
2. Detect those points in each production image, to subpixel accuracy.
3. Solve with the intrinsics and the distortion model from the calibration. Feed the solver raw pixels plus distortion coefficients; undistorting the image first throws away accuracy you paid for.
4. Check the reprojection: mapping the solved pose back onto the image must land within a fraction of a pixel per point. Above 1 px, suspect a mis-detected point or a wrong coordinate in the model.
5. Compose with CamInBasePose and send the pose to the robot.

Coplanar points are allowed — most parts offer nothing else — but a flat point set makes Z the soft axis of the solve, which is why monocular systems quote weaker depth than lateral accuracy. With occlusion or overlapping parts, run the solve through a RANSAC-style variant with extra candidate points so outliers get rejected instead of averaged in.

Which camera and lens does a reliable solvePnP set-up need?

For a 220 mm field of view with a 20 MP 1" sensor you need roughly a 16 mm lens at 285 mm working distance, f/8 for ±11 mm of depth, and about 0.04 mm per pixel — enough that subpixel point detection, not the camera, sets the accuracy.

Work the geometry before ordering anything. Example inputs: a 160 mm stamped bracket lands anywhere in a 220 × 150 mm region of a nest, the grasp is asymmetric so the robot needs full 6D, and the grasp tolerance is ±1 mm. Sensor assumption: a 1" type with 5472 × 3648 pixels and a 2.4 µm pixel pitch, 13.1 × 8.8 mm active area — a common 20 MP industrial format.

• Pixel scale: 220 mm ÷ 5472 px = 0.0402 mm/px. The 8 mm datum hole spans 8 ÷ 0.0402 ≈ 199 px, so a 0.2 px centre fit localises it to about 8 µm. Object space pixel size from magnification derives this scaling.
• Magnification: m = 13.1 mm ÷ 220 mm = 0.0595.
• Working distance with a 16 mm lens: WD = 16 × 1.0595 ÷ 0.0595 ≈ 285 mm, covering 220 × 147 mm on the 3:2 sensor.

• Depth of field: the bracket sits within ±10 mm of the focus plane. With b = 2 px = 4.8 µm, the required stop is N = 20 × 0.00354 ÷ (2 × 1.0595 × 0.0048) ≈ 7, so set f/8 — total depth ≈ 23 mm, about ±11 mm per side.
• Exposure: if the nest stops the part, exposure is free; 5-10 ms is comfortable with ordinary LED lighting. If the part passes at 150 mm/s, hold blur under 0.5 px: t ≤ 0.5 × 0.0402 mm ÷ 150 mm/s = 134 µs. That is strobe territory, and it needs a global shutter, because a rolling shutter exposes rows at different instants and turns belt speed into apparent tilt.

The LS-FFL-0567 fits this geometry: 16 mm focal length, 1.1" format, C-mount, 20 MP resolution, F2.8-22 iris, <0.3% distortion, and a printed 16-22 mm image circle. The 15.8 mm diagonal of the example 1" sensor sits just inside that circle.

For a part that stops in the nest, a 20 MP 1" USB3 mono camera such as the MV-CS200-10UM is in this class.

When the part keeps moving under the camera, switch to a global-shutter model such as the 12 MP 1.1" USB3 MV-CH120-11UC.

The same working-distance arithmetic, derived step by step, is in blister inspection working distance and field of view formula; the camera finder and lens finder both filter on the values computed above.

How accurate will the measured pose be?

With a clean calibration and a 0.04 mm pixel you should plan on 0.1-0.3 mm in X and Y, about 0.2° in rotation, and 0.2-0.5 mm in Z — comfortably inside a ±1 mm grasp tolerance, and limited by calibration residuals rather than by pixel noise.

The budget in the example: subpixel circle centres land within 0.1-0.3 px, which is 4-12 µm at 0.0402 mm/px. The intrinsics and the hand-eye chain typically carry 0.5-1 px of systematic residual, another 20-40 µm. Rotation follows from the lever arm: 0.3 mm of point error across an 80 mm half-length of the bracket is 0.3 ÷ 80 rad ≈ 0.2°. Z is softer than X and Y because a flat point set fixes depth only through scale, so plan on the 0.2-0.5 mm band and treat anything tighter as something to prove, not assume.

Before you trust any of it, run the physical checks in hand-eye calibration accuracy verification; if picks land offset in one consistent direction, diagnosing an eye in hand pick point offset walks the usual causes in order.

Can I match a 2D template at a known height instead of solving 3D?

Yes — when every part rests on one plane and the grasp needs only X, Y and yaw, a 2D match plus a fixed Z is simpler and runs on a basic camera; every millimetre of height error shifts a field-edge point by 0.4 mm at this 285 mm working distance.

The height sensitivity is plain geometry: a point 110 mm off the optical axis moves 110 × 1 ÷ 285 ≈ 0.39 mm for each millimetre the part floats above the calibrated plane. The rule that follows: with a ±1 mm grasp, keep the plane within ±1.5 mm — about 0.6 mm of edge error — or step up to solvePnP. Tilt breaks the 2D route the same way, because a template match assumes the part face lies parallel to the sensor.

Smart cameras and vision sensors do this natively: find the part, output pixel position and angle, and your program converts with the magnification — 0.0595 in the example — plus the known Z. A 1.6 MP smart camera with a 16 mm lens, onboard white light and 60 fps output, such as the LS-SMC-0063, carries the whole measurement.

The pixel-to-robot matrix from the first section is exactly this route with the transforms pre-composed, and it inherits the same one-plane limit; pixel-to-robot coordinate transform shows the shortcut applied to a real cell. Wiring a smart camera into the robot side when no wizard exists is covered in smart camera hand-eye calibration. If you later need tilt as well as yaw, ground plane pose from one image using four known points upgrades the planar case to full pose without 3D hardware.

When is a 3D camera the right answer instead?

Go 3D when parts stack or overlap, when the resting face is unknown, when height varies more than about ±10 mm, or when the robot needs a height map for a collision-free approach — one camera and solvePnP cannot see behind an occlusion or decide which face is up.

Stereo and structured-light sensors hand you the depth directly: reconstruct the surface, match a reference model of the part — taught once from CAD or a scan — and the match returns ObjInCamPose exactly as solvePnP does, so the compose step and the grasp handling do not change. Surface-model matching on the point cloud is the standard route for cast, forged or deformed parts whose 2D features are not repeatable.

The decision is driven by part presentation, not by accuracy: a single layer of flat parts is faster and simpler with one camera. 2D or 3D camera for conveyor pick and place works the same comparison for belt picking, and 3D bin picking for flat parts covers the stacked case.

How do I compose the poses into a robot pick point?

Compose once per cycle: ObjInBasePose = pose_compose(CamInBasePose, ObjInCamPose), then shift the pose origin from the object datum to the grasp point — define that offset once in CAD and every pick after it is a single multiply on the controller.

If the calibration returned BaseInCamPose, invert it first with the pose invert operator; composing from the camera side keeps the chain readable. Define the object frame on the grasp point when you build the point model, and the solved pose is a robot target as-is; otherwise apply the constant datum-to-grasp offset you taught in CAD, once, not per cycle.

Units and angle conventions are the classic silent failure: one library works in metres and radians, the next in millimetres and degrees, and a pose that arrives 90° rotated is a frame bug, not noise — point cloud rotated 90 degrees after hand-eye calibration dissects exactly that signature. Gate every pick: if the match score or reprojection residual crosses its limit, refuse the pose, stop the robot, and log the image, because a solver will return a confident pose from a half-occluded part.

What should I check before the first production pick?

Verify with hardware, not trust: place the part at five known positions and orientations on the fixture, compare the reported ObjInBasePose against the taught robot positions, and accept the system only when every offset lands inside a quarter of your grasp tolerance.

• Five-point fixture check: X/Y error within 0.25 × tolerance, rotation within 0.25 × the allowed angle.
• Reprojection residual under 1 px on every accepted part, logged per cycle.
• Jog check: shift the part a measured 10 mm in X; the reported pose must move 10 mm in the matching axis with the right sign.
• Blur check at your real belt speed; 0.5 px is the ceiling.
• Ambient light pushed to the worst the line sees — ambient light variation in pick and place vision explains why match scores move with the time of day.
• Re-run eye-to-hand calibration verification after anyone touches the camera mount; it catches a moved camera before it bends parts.
• Log ObjInCamPose and ObjInBasePose with every image for the first production week, so drift analysis tells you which side of the compose moved.

Every route ends the same way: ObjInBasePose = pose_compose(CamInBasePose, ObjInCamPose), with the object frame on the grasp point. Size your own set-up in the system configurator, or bring the part, the tolerances and the fixture drawing to our applications desk. More transform-chain and calibration debugging pieces live under vision systems.