Camera frame to servo frame: one plane, one affine transform

A fixed camera looking down at servos that work in one shared plane is the simplest tracking geometry in machine vision, and it needs the simplest math: one 2×3 affine matrix and one scale number. No projection matrix, no robot in the loop, no hand-eye solver. This guide is for that station: a multi-cup servo picker, a camera-guided feeder, any rig where the camera never moves and the parts stay flat. You will place the world frame, calibrate millimetres per pixel with a coin, fit the matrix from three measured targets, and turn a centroid into a servo angle with atan2. Every number comes from one worked build: a 230 × 170 mm servo array imaged at 0.0625 mm per pixel.

How do I convert camera pixels to servo coordinates on a flat build area?

You fit one 2×3 affine matrix that sends pixel coordinates straight to a world frame in millimetres, then treat each servo as a constant offset from that frame, so a cycle is one matrix multiply and one atan2.

The four-matrix chain — translate, translate, rotate, then convert pixels to millimetres — fails because half of it lives in pixels and half in millimetres, and you cannot know the pixel scale until you measure it. Fit the matrix from measured point pairs instead: scale, rotation, shear and offset all arrive together in one solve, with no unit bookkeeping in between.

This is the planar cousin of the 3D chains used around robots. Those need a full hand-eye calibration because a robot pose has six degrees of freedom; a camera bolted square over a flat plate has none. The 3D machinery still matters when part heights vary or a robot base is involved — the systems topic collects those guides, and the flat-plane case below needs none of them.

Where should the world frame go?

Lock it to a point you can measure with a ruler: the axis of the first servo horn or the centre of the array, at part-top height, because a camera that never moves pairs with a frame that never moves.

One origin, two axes, chosen so that every servo becomes three constants: its position (xk, yk) measured from the datum, and the direction its horn points at zero degrees. Write them into configuration once. The frame owes nothing to the camera — asking where the world origin sits in pixels reverses the dependency, because the calibration is exactly what answers that question.

So do not anchor the world frame to the image with a guess such as 40 mm below the centre pixel. Stating that distance already needs the scale you have not calibrated yet, and it drifts whenever zoom or ROI changes. Punch a physical datum mark on the plate, measure the servo offsets from it with calipers, and let the fit find the camera's relation to it.

What do I need on the bench before I calibrate?

Four things: the camera fixed overhead, one object of known width, three small targets at measured world positions, and a stand that holds the lens parallel to the part plane.

• A camera and lens that cover the array — the next section sizes them; start in the area scan cameras category.
• A rigid mount from the camera stands category or the accessories finder: any height or angle drift after calibration is pure error.
• A 25 mm coin or a gauge block for the scale.
• Three 5 mm paper targets at positions measured from the datum, plus a ruler or calipers.

Stop the lens down until the whole plane is sharp, lock the focus mechanically, and capture a test frame before you measure anything. Refocusing afterwards invalidates the scale and the matrix together; the trade-off between aperture and sharpness is in the depth of field guide.

How many millimetres is one pixel?

Divide a known length by its pixel span: the worked build reads 250 mm across 4000 pixels, which is 0.0625 mm per pixel, and a 25 mm coin lying in the part plane spans 400 pixels.

Put the coin at the height the cups actually touch, capture, threshold, and measure its span in pixels. Scale = coin width ÷ span = 25 mm ÷ 400 px = 0.0625 mm per pixel. Take the span from the thresholded disk, not a hand-drawn box: a two-pixel edge bias on a 400 px span is already 0.5%. This is the same pixel-to-millimetre step used to size coins from pixels, and it is the only absolute scale the build needs — the matrix in the next section absorbs rotation and shear around it.

Which camera and lens cover the whole servo array?

Work backwards from the array footprint: a 250 × 185 mm field needs about 7.4 mm of sensor width, so a 12 MP 1/1.7-inch sensor under an 8 mm lens at 278 mm working distance does it, at 0.0625 mm per pixel.

A 1/1.7-inch optical format is about 7.4 × 5.6 mm of silicon, and the 250 × 185 mm field wraps the 230 × 170 mm array with 7-10 mm of margin on every side. The lines that size the optics:

• Magnification: m = 7.4 / 250 = 0.0296.
• Focal length at a 250 mm working distance: f = 250 × 0.0296 / 1.0296 = 7.19 mm — between stock focals, so take the nearest fixed 8 mm.
• Working distance the 8 mm lens then needs: WD = f × (1 + m) / m = 8 × 1.0296 / 0.0296 = 278 mm.
• Scale: 250 mm ÷ 4000 px = 0.0625 mm per pixel, and the coin check gives 25 ÷ 0.0625 = 400 px.
• Height: the 5.6 mm sensor dimension covers 5.6 ÷ 0.0296 = 189 mm of field, clear of the 185 mm requirement.

The focal length calculator runs the same lines from your own footprint and stand height:

The lens that lands on these numbers is an 8 mm fixed focal with a 6-10 MP rating and an 8-11 mm image circle; the circle clears the 9.3 mm diagonal of the 1/1.7-inch format with margin.

The camera side wants 4000 pixels across: a 12 MP 1/1.7-inch rolling-shutter sensor reading 4000 × 3000. Divide 4000 into the 250 mm field and the 0.0625 mm per pixel reappears. The same sensor ships on USB3 at 30 fps or on GigE at 9 fps, so the interface follows your cable run, not the math.

Rolling shutter suits this station because the scene is still when you capture. The scale lines depend on pixel count, not shutter type: parts that move under the camera while exposed need the global-shutter filter in the camera finder, and the scale then follows from that body's own pixel count.

How do I fit the affine matrix from pixel points?

Print three targets, tape them at world points you measured with a ruler, read their pixel coordinates, and solve the six unknowns of the 2×3 matrix exactly; add more pairs and a least-squares solve when you want a residual number.

Each output coordinate is a weighted sum of the two pixel coordinates:

Six coefficients, so three measured pairs pin them exactly; five or more pairs go through a least-squares solve, which OpenCV returns from one call when you hand it the pixel array and the world array.

1. Measure the three target centres from the datum and write them in millimetres.
2. Threshold a frame and read each target's centroid in pixels.
3. Solve the six coefficients — exactly from three pairs, least squares from five or more.
4. Hold back a fourth target, re-project it through the fitted matrix, and record the residual.
5. Accept the matrix when that residual stays under 0.3 mm — about 5 px at this scale.

If corner targets misfit while centre targets agree, the lens is bending, not the fit: an affine is linear and cannot follow barrel distortion. Correct the distortion first — the distortion vector map guide shows the per-pixel shifts — or keep targets inside the middle 80% of the frame.

What angle does the servo command to reach the dot?

Subtract the servo's world position from the transformed point and take atan2 of the vector: the example dot at (−30, 52) mm from the horn sits at 60.03 mm and commands 120.0°, inside the 5 mm capture band of a 60 mm horn.

Every servo keeps four constants: xk and yk from the datum, its zero-ray direction φk, and a sign for the mounting. The per-cycle code is then fixed:

• Transform the centroid: (X, Y) = A·(u, v) + t.
• Subtract the servo position: dx = X − xk, dy = Y − yk.
• Distance d = sqrt(dx² + dy²); commanded angle θ = atan2(dy, dx) − φk.
• Track while the dot moves; fire suction when |d − L| ≤ 5 mm, with L the horn length.

Worked numbers for the servo whose horn axis is the datum (xk = yk = 0) and whose zero ray runs along +X (φk = 0), with a 60 mm horn:

• d = sqrt(30² + 52²) = sqrt(900 + 2704) = sqrt(3604) = 60.03 mm; |60.03 − 60| = 0.03 mm, inside the band, so the cup fires.
• θ = atan2(52, −30) = 180° − 60.02° = 119.98° ≈ 120.0°.

The 5 mm band is about 8% of a 60 mm horn: wide enough to absorb the height error budgeted below, tight enough that the cup lands on the dot. One pixel of centroid noise moves the angle 0.06° at this radius, far under the band, so raw centroids work; average three frames if the marker is small enough to flicker.

Two conventions cause most first-run failures: image rows count downward while world frames usually count up, and a mirrored mount negates the angle. Confirm both by placing a target on the servo's zero ray — the commanded angle must read zero. The same sign-and-order trap is what makes robot offset corrections shift X and Y in bigger cells.

When does this planar shortcut give wrong numbers?

The scale is only valid in the calibrated plane: at 278 mm working distance, a part whose top sits 10 mm above the plane inflates the scale by 278/268 − 1 ≈ 3.7%, worth about 2.2 mm at a 60 mm horn radius.

Perspective is the mechanism: anything closer to the lens projects larger, and the coin fixed the scale at one height only. The error fraction is close to h ÷ (WD − h), so fixture the parts so their tops sit at the calibrated height, or calibrate at the tallest top you accept. At these numbers the 10 mm error consumes about half of the 5 mm capture band — the reason the band is not 2 mm.

Distortion is the second failure. Lens data sheets quote it as a percentage of image height; corrected machine vision designs print under 0.3%, short wide focals run several times that. At a 125 mm corner radius, 1% is 1.25 mm, so trust the affine near the centre, check your corner target, correct the distortion, or crop the ROI.

Tilt is the third. A few degrees off-normal mostly becomes shear, which the affine absorbs, but an oblique view is genuinely projective. Decision rule: if the held-back target re-projects worse than 0.3 mm, stop fitting affines and fit a homography from four or more points — the ground plane pose guide works that four-point case.

Leave the plane entirely when part heights vary, the gripper needs Z, or a robot base enters the loop: that is the full hand-eye problem, starting from the camera above the robot base setup through computing robot-base picks from the result.

For this build the order of operations is settled: size the field, order the catalogue parts above, mount square, calibrate the scale with the coin, fit from three targets, check the fourth. The camera configurator pairs camera and lens from your footprint, machine vision lenses holds the fixed focals, and contact is the right stop when your geometry is not flat.