Eye-in-hand pixel to world coordinates: chain the transforms

Your camera rides the robot flange, so the same pixel maps to a different base coordinate every time the arm moves. That is the whole difficulty of eye-in-hand guidance, and it collapses into a fixed chain of three transforms with one moving part: the flange pose you read at each capture. If you are wiring a depth camera on a robot arm to vision-guided motion, this walkthrough gets you from any pixel to a base coordinate you can command — the unprojection math, the hand-eye link, the pose read, a worked example you can check by hand, the accuracy to budget, and the checks that catch a wrong chain before the first crash.

What is the transform chain from one pixel to a base-coordinate point?

Three transforms in fixed order: depth plus intrinsics unproject the pixel to a camera-frame point, the hand-eye transform moves that point onto the tool flange, and the flange pose reported by the controller carries it into base coordinates.

Name the frames before the math. The camera frame sits at the depth sensor's optical center with z pointing out of the lens. The flange frame sits at the tool mounting face with z along the tool axis. The base frame is the robot's own coordinate system — the world your controller reports positions in. Defining that world frame is the first integration decision, and everything downstream inherits it.

The chain multiplies three transforms:

T_flange_camera is your hand-eye calibration result, and it never changes until hardware is remounted. T_base_flange is the flange pose the controller reports, and it is different at every robot position. If the 4x4 convention itself is unfamiliar, the homogeneous transformation matrix walkthrough covers the multiplication order used below.

This is also what separates the two mounting styles. An eye-to-hand camera bolted to a stand needs one camera-to-base calibration that then behaves as a static transform — the static eye-to-hand transform workflow shows how little changes per cycle there. Eye-in-hand keeps only the camera-to-flange link fixed and pays for the moving viewpoint with a pose read every capture.

How do you build the pipeline step by step?

Six steps: mount and calibrate once, then for every capture read the depth at the pixel, unproject to the camera frame, apply the hand-eye transform, read the flange pose at the same instant, and multiply the chain.

1. Mount the camera on a rigid bracket at the flange. Flex or backlash between camera and flange shows up later as pick scatter that no software fix removes.
2. Calibrate hand-eye once. Park the flange over a calibration target at several positions and orientations, capture, and solve for the single 4x4 camera-to-flange transform. The robot-mounted capture-and-solve procedure walks the capture, and the calibration pose selection guide explains why the poses must vary in rotation, not just position. Re-run it after any remount, any change that moves the camera, or any knock.
3. Pull the depth intrinsics fx, fy, cx, cy from the camera SDK, at the resolution you actually stream.
4. Per capture, read depth at the target pixel and unproject (next section). If your SDK publishes a point cloud, the unprojection is already done — just confirm which frame the cloud uses, a mix-up the optical frame versus camera frame check untangles.
5. Read the flange pose — position plus rotation, in base frame — sampled at the same instant as the image.
6. Multiply the chain and command the move. Only the hand-eye result survives from cycle to cycle; pixel, depth and pose all refresh.

How do you turn a depth pixel into a camera-frame point?

Multiply the pixel's offset from the image center by the depth and divide by the focal length in pixels: at Z = 350 mm with fx = 600 px, a pixel 40 rows below center sits 40 × 350 / 600 = 23 mm off axis.

The pinhole model needs only the depth at that pixel and the intrinsics:

(u, v) is the pixel, (cx, cy) the principal point — in practice the image center — and fx, fy are focal lengths in pixels. Depth streams come rectified, so no distortion step follows.

The same relation sets your lateral scale: one pixel spans Z / fx at the target. At Z = 350 mm with fx = 600 px that is 0.58 mm per pixel, so a 1280-pixel width covers about 747 mm at that depth. The calculator below pairs working distance, pixel count and focal length when you size the camera:

Reading one pixel's depth, rather than a full cloud, is a technique in its own right; the fixed-image-point depth read uses it to find a table or part surface under the lens.

How do the transforms multiply in a worked example?

Pixel (640, 400) at 0.350 m depth, a 180° bracket twist with 30 mm and 75 mm offsets, and a flange pose of (850, 120, 500) mm yawed 90° land the point at (873, 150, 925) mm in base coordinates — every number below checks by hand.

Camera: 1280 × 720 depth image, fx = fy = 600 px, cx = 640, cy = 360. Target pixel (640, 400), depth Z = 0.350 m. Work in metres.

Step 1 — unproject.

Step 2 — apply the hand-eye transform. Say the bracket twists the camera 180° about z and offsets it 30 mm in flange x, 75 mm toward the work, so R_h = diag(−1, −1, 1) and t_h = (0.030, 0.000, 0.075) m:

Step 3 — apply the flange pose. The controller reports the flange at (0.850, 0.120, 0.500) m, yawed 90° about base z. Rz(90°) sends a flange x to base y and a flange y to base −x:

The pixel lands at (873, 150, 925) mm in base coordinates. Now put the same scene behind a flange yawed to −90°: steps 1 and 2 return identical numbers, but step 3's rotation flips sign, and a pixel that unprojects identically in the camera frame lands at a different base point. That asymmetry is why step 3 has no memory — the pose belongs to this capture only.

Why must the flange pose be re-read at every capture?

The camera is part of the kinematic chain, so its pose in base coordinates changes with every joint move; a flange pose read 50 ms out of sync at 250 mm/s travel puts every converted point about 12 mm off.

A fixed camera earns its simplicity from one permanent rotation. For a moving camera, orientation in base coordinates is a function of all six joints, so no single rotation matrix survives a repositioning. The remedy is timing, not more calibration. Stop-and-shoot is the dependable default: command in-position, wait 50–100 ms for settle, capture, read the pose, compute. Capturing on the fly needs a pose sampled at exposure time, which only some controllers expose — until yours does, stop and shoot.

Controller conventions deserve one check: some flange-position readouts return world coordinates even when the active tool frame suggests otherwise, and note on flange positions returned in world coordinates instead of tool-frame values shows the symptom and the fix. If your program consumes targets in a user frame or work object rather than base coordinates, the camera point to work object conversion is the hop after the chain. The eye-to-hand counterpart computes base targets from a transform that never changes — the eye-to-hand base computation workflow shows that version.

What accuracy should you plan around?

Budget about 1 mm total position error at 350 mm working distance and about 7 mm at 1 m — depth error scales with distance squared — while lateral error stays near 0.6 mm per pixel at 350 mm.

Depth dominates, and for a stereo depth camera with focal length fx, baseline b and disparity noise σ_d it follows:

Take a representative close-range sensor: fx = 600 px, b = 50 mm, σ_d = 0.2 px on a textured scene.

A structured-light projector lowers σ_d on untextured or shiny parts, pulling both numbers down; it does not change the Z² growth. On top of depth error add the hand-eye residual — a fraction of a millimetre with a good target and an arm that repeats to 0.02–0.1 mm — and the robot's own repeatability. The budget at 350 mm is therefore about 1 mm, and that sets the pick tolerance: tight enough for rigid grasping with compliance, and the reason long-reach picks at 1 m want a second, closer look. The 3D camera working distance guide covers how range and resolution trade against each other when you position the sensor.

What hardware carries the eye-in-hand pipeline?

The set-up is a depth sensor light enough for the flange, one Ethernet drop down the arm, and a vision controller with at least two spare GigE ports — one for the camera, one for the robot or PLC.

For the sensor, pick the smallest depth source that resolves your feature at the working distance. Structured light handles untextured or shiny targets because the projector paints in the texture stereo matching needs; a ToF sensing camera such as the MV-DT01WMU trades detail for speed when bin-finding or coarse pose is the job. Both families live in the 3D sensors range, and the 3D camera finder filters them by type and depth range. A projected structured-light camera like the MV-DPS200P-02 is the close-range workhorse of the two.

On the compute side, the controller needs one port for the camera, one for the robot or PLC, and headroom for a second camera or a photometric light later. PoE camera ports put power and image data down the same cable that runs down the arm — check the PoE row in the table below. The vision controllers range runs from compact fanless boxes to GPU inference terminals; both of these fit the eye-in-hand pattern and expose built-in light controller channels:

All 2 share: Humidity 20% RH to 95% RH (no condensation); Certification CE, KC. Values as printed in each product's data sheet; open a part for its drawing and full specifications.

How do you verify the chain before the first pick?

View one fixed target from three flange orientations and recompute each time: a correct chain returns the same base coordinates within about 1 mm at 350 mm working distance, while a stale pose or a wrong rotation scatters it by millimetres to centimetres.

Run these checks on a bench target before anything moves under the robot:

• Three-orientation repeat. One target, three flange yaws spread over 90°, recompute each time. A spread within about 1 mm at 350 mm working distance says the chain is consistent.
• Gauge-block depth. Set a 25.0 mm gauge block on the table and read the depth jump between table and block top; it should return 25 mm within the depth error from the accuracy section.
• Touch-off. Command the computed point with a pointed tip over a shim stack and record the residual and its direction.
• Read the residual pattern. A constant offset in one direction points to a hand-eye or convention error — left-handed image y, row-major versus column-major matrices, or an optical-frame mix-up. Scatter that grows with robot motion is a pose-timing fault. Scatter that grows with depth is the sensor. The eye-in-hand pick offset diagnosis sorts residuals by cause.

Close with the controller hand-off. Where the program consumes targets through a user frame, one pattern writes the vision-computed point into a frame each cycle — feeding vision world coordinates into a dynamic user frame shows the mechanism — and the 3D camera robot integration checklist sweeps the compatibility questions before power-up.

The decision in one line: unproject with depth, keep the hand-eye result fixed, read the flange pose every capture, and trust the chain only after the three-orientation test passes within about 1 mm. To spec the hardware end to end, size the camera in the 3D camera finder, then run the system configurator over the whole set-up; the neighbouring guides live on the 3D vision topic page, and where a set-up needs a drawn spec, contact the applications team with your working distance and target size.