Convert image coordinates to robot coordinates for servo moves

You can convert image coordinates to robot coordinates with one reference photograph: a 200 mm bar spanning 250 px gives s = 200 / 250 = 0.80 mm/px, and a centroid 60 px right and 40 px down from your taught origin becomes a servo target of +48.0 mm in X and −32.0 mm in Y. That scale, the origin pixel (u0, v0) and two axis signs are the whole mapping when a fixed overhead camera looks straight down at a flat workspace.

Your detection loop already finds the part — the blob centroid comes out as (160, 120) and the arm has no idea where that is. If segmentation itself is the weak half, start with the color blob tracking and PID control guide and come back; this page assumes a working detector. Every centroid after the one calibration becomes a servo target.

One scale, one origin, two signs

If the camera stays fixed, points straight down at a flat plane, and every part sits at the same height, you do not need a full camera calibration. One reference measurement gives the whole transform:

L is the reference length in millimetres, N is that length measured in pixels, and (u0, v0) is the pixel position of your chosen origin. X and Y are offsets from the origin pose — exactly what a point-to-point move consumes. Add them to the pose you taught at the origin, or feed them to your kinematics. Everything that follows keeps those three numbers honest.

Fix the geometry before the numbers

The scale s holds only at the height where you measured it. The lens images through similar triangles, so a part Δh above the calibration plane enlarges the image, and the relative scale error is about Δh / WD, with WD the working distance. At WD = 400 mm, a part 10 mm proud of the calibration plane runs 2.5 % fast: a commanded 100 mm move lands 2.5 mm long.

Tilt costs more. Tip the camera 5° off perpendicular and the plane rises across a 256 mm field by 256 × tan 5° ≈ 22 mm, so the scale drifts about ±3 % from centre to edge — up to 3 mm on a point 100 mm from the origin. Square the camera to the table, keep part height constant, and both errors disappear. If height cannot be constant, use the four-point method near the end instead.

Hold that geometry with hardware: a rigid column clamped square to a base plate. The camera stands category is built for it; the LS-MNT-0040 has a 700 mm height and a 400 x 450 mm base under a fixed overhead head.

Measure the scale with anything of known length

Lay the reference in the part plane: a machined bar, a gauge block, or a printed target checked with calipers. Photograph it, count the pixels between its edges, and divide. Counting a known object's pixel extent is the same operation as calibrating length measurement from image markers, so that guide doubles as a second opinion on technique.

Use a reference at least a third of your field wide. With a 200 mm bar spanning 250 px, s = 200 / 250 = 0.80 mm/px, and a ±0.5 px edge uncertainty moves s by only ±0.2 %. A 20 mm bar makes the same uncertainty ten times larger in relative terms. Measure an L-shaped pair and compare the axes: if the X and Y scales differ by more than 1 %, the pixels are not square or a bar sits tilted, and no later math repairs that.

Choose the camera for the scale you need

You set s through the optics: s = field width / pixels across. For the example scale, 256 mm / 0.80 mm/px = 320 px across is the minimum; a 5 MP sensor samples the same field about 8 times finer, which is why position accuracy ends up limited by geometry, not pixels. Walk the smart cameras category or the camera finder with your field width and tolerance in hand.

Vision sensors that suit a fixed overhead cell, compact to higher resolution:

The middle card is the no-adjustment option: the MV-SCA002M-03S-WBN-SR includes a 3 mm lens focused at 240 mm, a fixed overhead view for a small workspace.

Fix the origin and the axis signs

Pick a point you can reach and re-find: a dowel hole, a fiducial mark, or the reference bar itself. Jog the tool to it, record the pose or the servo angles, then photograph it and read (u0, v0). Every detection now means: origin pose plus (X, Y). The taught pose and the photographed centroid must be the same physical point; when they are not, every target shifts by the same offset — the signature of a moved pattern matching reference coordinate system.

Image v grows downward. With the camera oriented so the image top is the far side, Y = (v0 − v) · s makes +Y point away from the camera; if your Y runs the other way, drop the sign, and confirm both directions once with a slow manual command. Rotation is the last trap: a camera yawed θ from the robot X axis displaces every target by about r · θ — 100 × 0.087 ≈ 8.7 mm at 5° and a 100 mm reach. Square the camera, or carry the rotation in the mapping:

Make the centroid worth converting

The mapping is exact only if the input centroid is honest. Filter blobs by area, with the window centred on the part's known pixel area, so a glint or a neighbouring part cannot command a move. Take the subpixel centroid your library reports, and check the coordinate convention first — pixel centres and edges are not the same address, and the half-pixel offset in blob centroids explains a whole class of one-pixel errors. If you segment straight out of a raw frame buffer, the index arithmetic for locating image coordinates in a 2D pixel array is the same in every language.

With a stable threshold, centroid repeatability lands near ±0.2 to ±0.5 px. At 0.80 mm/px that is ±0.2 to ±0.4 mm of commanded-position noise — comfortable for millimetre-class picks, and the reason this method holds up in practice.

Worked example: centroid to servo target

Setup: 320 x 240 frame, overhead camera, WD = 400 mm, calibration shot taken in the part plane. Scale: the 200 mm reference spans 250 px, so s = 200 / 250 = 0.80 mm/px. Field: 320 × 0.80 = 256 mm wide, 240 × 0.80 = 192 mm high. Origin: the fiducial centroid reads (100, 80) px and the arm is taught at that pose. Detection: a blob centroid at (160, 120) px.

• X = (160 − 100) × 0.80 = 48.0 mm
• Y = (80 − 120) × 0.80 = −32.0 mm, negative because the part sits nearer the camera than the origin
• Command: taught pose plus (48.0, −32.0) mm, or (48.0, +32.0) if your Y runs opposite

A compact controller closes the loop. The MV-VC2140-128G40-1T pairs an N97 processor and 8 GB of memory with 3-4 LAN ports, 2 USB 3 ports and 4 built-in light channels, so camera, light and robot links live in one box; compare the vision controllers for the port and light-channel counts your cell needs. How the offsets reach the robot over a socket, with framing and acknowledgement, is the subject of sending camera part coordinates to a robot controller over TCP.

Choose the lens before you drill the stand

The scale also fixes the focal length. A 4:3 sensor format spans 16 mm of diagonal per inch of format, so a 1/1.45" sensor is about 8.8 mm wide. For a 256 mm field at WD = 400 mm: m = 8.8 / 256 = 0.0344, then f = 400 × 0.0344 / 1.0344 ≈ 13 mm. Take the nearest standard focal length, shoot the reference again, and derive s from that image — never from the lens label. The calculator below runs the same arithmetic from your sensor and field:

When one scale is not enough

Switch methods when an input breaks. Parts at different heights: the scale error is Δh / WD, so ±10 mm of height at 400 mm WD means ±2.5 % of position — a four-point homography absorbs tilt and height in one plane map. A camera that cannot be squared to the plane: same homography, from four known workspace points. A camera riding on the robot arm: the fixed-frame method does not apply, and the eye-in-hand pixel to world coordinates guide chains the extra transform. Keep measured targets inside the central 80 % of a wide lens's field; beyond that, edge distortion becomes the error source.

Checks before the first live move

• Reference length measured with calipers, lying in the part plane — on a part, not the bare table, when parts carry height.
• X and Y scales agree within 1 %; a square target images as a square, confirming the camera is square to the plane.
• The taught pose and the photographed origin are one physical point: re-shoot the fiducial right after teaching.
• Signs proven with one slow manual command in +X and one in +Y.
• Dry run on paper: command a drawn point, re-image it, and expect agreement within about two pixels — 1.6 mm at this scale.
• Scale re-shot after any bump, refocus or lens change; a two-minute check that protects every later pick.

Set the scale once, prove it on paper, then let the centroid drive. If your field width, working distance or part height does not match the assumptions above, tell us those three numbers through the contact page — the mapping choice follows from them. More worked positioning examples sit on the applications topic page.