Apply camera XY offsets in a robot program with unit conversion

The vision sensor has found the part and read the code, and the result on screen reads 0.1234 m. The robot is parked at a taught pick point measured in millimetres and radians. Between those two facts sits the most common integration error in vision-guided picking: a metre value or a degree value written straight into a controller that wants millimetres and radians. You get the two conversions, a six-step offset pattern that works on any robot controller, a worked example from camera result to commanded position, and the checks that catch a wrong frame or a stale result before it turns a good part into scrap. It is written for whoever wires the camera to the controller over Ethernet: integrator, controls engineer, or the person at the pendant. For the full locate, pick and bin cycle around this offset, see how a 2D vision-guided pick and place cycle works.

What you need

• A smart camera or vision sensor whose calibration maps pixel results into the robot's frame, so the values it returns are X and Y in the robot's coordinate system already
• The taught pick position, recorded as a named pose with Z, tool orientation and yaw
• An Ethernet link to the controller, raw socket or fieldbus, with the result format written down: how many values, in which order, in which units
• The controller's offset or pose-composition instruction, plus two or three spare registers for the offset
• One test part you can place at known positions to about a millimetre

What the camera sends and what the controller expects

A calibrated 2D result is normally a short list of numbers: X and Y in metres, a yaw in degrees, sometimes a Z and a match score. The controller stores positions in millimetres and angles in radians. The code result differs in kind: it is a text string, not a number, so it travels in its own field and drives bin or job selection, never the pose arithmetic.

Calibrations that use a marker or a fixture hide the same trap one level down. A pose comes out in the unit you declared the marker in: teach a 40 mm marker as 40 and translations return in mm; teach it as 0.040 and they return in metres. The marker pose units and tvec article works that case in detail.

Fieldbus mappings can rescale as well: some carry positions as integers where one count is 0.1 mm. Read the scaling once, write it on the drawing, and convert in exactly one place in the program. Unit mismatch is not only a camera problem either; this unit conversion fault that follows a program transfer is a controls-side case with the same signature.

The two conversions

Both conversions are exact definitions, not estimates: 1 m is 1000 mm, and one degree is π/180 = 0.0174533 rad. Miss the first and a true 123.4 mm correction moves the robot 0.12 mm, so the camera might as well be unplugged. Miss the second and 12.5° becomes 12.5 rad, which is 716.2°, nearly two full turns. The failures are not subtle, and the tell is that the error is always the same factor: 1000 for position, 57.3 for angle.

Apply the offset: the six steps

1. Record the taught pick point as a named pose, p_pick: X, Y and Z in mm, orientation and yaw in rad, taken with the part in its nominal position.
2. Read the vision result into three real variables, cam_x_m, cam_y_m and cam_yaw_deg, in a background thread that runs independently of motion. The thread opens the socket, triggers the camera, collects the float array and holds the values; the motion program collects them when it needs them, so the network never stalls a move.
3. Convert the units with the formulas above, into dx_mm, dy_mm and yaw_rad.
4. Build the offset vector [dx_mm, dy_mm, 0, 0, 0, yaw_rad]: translation in mm, rotation in rad, zeros in the axes you are not correcting.
5. Compose, do not accumulate. Apply the offset to the taught point every cycle, p_target = offset(p_pick, offset_vector). Adding it to last cycle's target lets one bad read shift every pick after it.
6. Handle no-read. If the camera does not find the part, zero the offset, raise a fault and stop. Never reuse the previous result.

The composition instruction is spelled differently on each controller; an offset applied to a position register is one common layout, shown end to end in this stacking program built on offset instructions and position registers. In generic form:

// Illustrative names and pseudo-code, not any controller's API
real cam_x_m     = 0.1234                          // vision result, robot frame
real cam_y_m     = -0.0567
real cam_yaw_deg = 12.5
pose p_pick      = [450.0, 210.0, 50.0, 0.0, 0.0, 0.0]   // taught point, mm and rad

real dx_mm   = cam_x_m * 1000                      // 123.4
real dy_mm   = cam_y_m * 1000                      // -56.7
real yaw_rad = cam_yaw_deg * 3.141593 / 180        // 0.2182

pose p_target = pose_offset(p_pick, [dx_mm, dy_mm, 0, 0, 0, yaw_rad])
move_to(p_target)   // approach above it with your own Z lift

The approach point above the part gets the same treatment with a Z lift added; the variable pick approach offsets article covers the approach and retract side.

Worked example: camera result to commanded position

Take a cycle with inputs you can check by hand. The camera, calibrated into the robot frame, reports X = 0.1234 m, Y = -0.0567 m and yaw = 12.5°.

• dx = 0.1234 m × 1000 mm/m = 123.4 mm
• dy = -0.0567 m × 1000 mm/m = -56.7 mm
• yaw = 12.5° × 0.0174533 rad/° = 0.2182 rad

The taught pick point is (450.0, 210.0) mm at yaw 0 rad, so the offset target is (450.0 + 123.4, 210.0 - 56.7) = (573.4, 153.3) mm, yaw 0.2182 rad.

The gripper contacts the part 25 mm from its centre along the part's long axis, and the part sits 12.5° off nominal, so that contact offset rotates with the part:

Final command: (450.0 + 123.4 + 24.41, 210.0 - 56.7 + 5.41) = (597.81, 158.71) mm at yaw 0.2182 rad. One degree of yaw error moves a 25 mm contact point by 25 × sin(1°) = 0.44 mm, which is why the angle conversion deserves the same care as the metres.

Verify before production

Frame and direction test: jog the robot exactly 100.0 mm in X with a part in view. The camera's reported X must change by 0.1000 m. If it changes by 100.0, the camera is already in millimetres and your ×1000 is one step too many. If it changes by -0.1000, an axis is mirrored; fix the calibration, not the program.

Pixel scale sets the ceiling on locate accuracy. A 2448-pixel sensor covering a 245 mm field of view resolves 245 / 2448 = 0.10 mm per pixel, so no calibration will locate better than a few tenths of a millimetre on that hardware. Check your own pixel scale against the accuracy the pick needs:

Repeatability: place one part and run 20 locate cycles, logging the reported X and Y. A calibrated camera on a fixtured part usually stays inside a 0.05 to 0.2 mm band. The robot adds its own data-sheet repeatability, typically ±0.03 to ±0.1 mm on a cobot, so about ±0.5 mm at the part is a defensible budget for a single look.

Stale results: if the part can move between image and grip, through conveyor vibration or a bump from the gripper, the offset is wrong by exactly that displacement. Trigger the image at the moment the part is at rest; the camera trigger delay timing article covers the delay arithmetic.

Tight tolerance, two looks: below about ±0.2 mm at the part, use the first offset to find the part, stop at an approach point, take a second image from closer or with a narrower field of view, and correct the residual before closing. The second look scales the error by the field-of-view ratio: narrowing from 245 mm to 80 mm turns ±0.3 mm into ±0.1 mm (0.3 × 80 / 245 = 0.10).

A part that misses by the same amount at every position, and only after new program files went in, points at a frame or unit mismatch rather than vision. This position error that appears after a program load is one documented case of exactly that shape.

Common mistakes

Ordered by how often each one costs a day:

1. Metres into a millimetre field. The 123.4 mm correction becomes 0.12 mm and the robot hits the taught point as if the camera were disconnected. The link looks healthy; only the part position is wrong.
2. Converted twice. 123400 mm lands in the target and most controllers refuse the move with an out-of-range or soft-limit alarm instead of crashing. Annoying, but it fails safe.
3. Degrees into a radians field. 12.5° becomes 12.5 rad, 716.2°, and a tool that spins nearly two turns. Reading radians as degrees under-rotates by the same 57.3 factor.
4. Offset added in the wrong frame. If the camera's calibrated axes are rotated against the robot base, rotate the offset before adding. At 90°, dx_base = -dy_cam and dy_base = dx_cam.
5. Socket reads inside the motion thread. Every cycle now waits on the network. Keep the read in its own thread and hand values over, as in step 2.
6. Code and coordinates in one field. A parse failure then corrupts the pose or the bin routing. Send the string and the floats as separate fields and route the code with the pattern in route a barcode reader result to robot job selection.

Parts for a locate-and-read station

The hardware only has to do two things here: report a position repeatable to a few tenths of a millimetre, and decode the code every cycle. Three parts that fill the roles:

A 5 MP colour vision sensor runs the locate job and the code read on the same exposure over Fast Ethernet, with a global shutter and an M12 mount: one device, one link, both numbers. Where the code is read at its own station ahead of the pick, a compact reader such as the MV-ID3013PM-12M-WBN, a 1.3 MP reader with a 12 mm focal length on the same interface, keeps the two jobs out of each other's way. For higher throughput, the LS-CRD-0028 decodes up to 60 codes/s at 30 fps through its 8 mm lens on a 2560 x 2048 sensor. Filter the fixed code readers and the camera finder by interface, resolution and mount to finish the selection.

Once the units are converted and the offset composed, the rest is the robot's ordinary job: approach, grip and drop by code value. Size the field of view and the trigger to the part and let the configurator walk the rest of the chain: