Robot picks: apply a vision correction of X, Y and angle in one shot

A gripper camera that returns X, Y and an angle hands you everything needed to pick a displaced part in one move, yet the single-shot correction usually lands with the angle right and the position off by several millimeters. The cause is mechanical, not vision: the reported rotation is meant to swing the tool about the part's trained reference point, while the robot rotates about its tool center point. Whenever those two points differ, the wrist rotation carries the pick point away with it. This guide gives the correction in the right order, the lever-arm term with the arithmetic worked through, and the two test picks that prove the result before you release the cell.

Why a one-shot X, Y and angle correction drifts

The vision tool matches the found part against the trained model and reports three values: the angle θ between part and model, and the X, Y position of the model's reference point in the calibrated frame. Those three numbers describe one rigid planar transform of the taught pick: rotate the taught pose by θ about the reference point, then shift it by the reported X and Y. The order is not stylistic. The rotation is defined about that reference point, and the translation is only valid in the frame the model was taught in.

A robot cannot rotate about a point in the image. An orientation command holds the tool center point (TCP) fixed and swings the tool around it. If the TCP sits on the reference point, the swing moves nothing and the raw X and Y land the pick exactly. If the TCP is offset from it — a suction cup ahead of the part center, jaws closing at the part sides — the rotation carries the pick point with it, and a correction that adds only the raw X and Y misses that carry.

The missed displacement is the chord of the swing:

with r the distance from reference point to TCP. At θ = 12° and r = 40 mm it is 2 × 40 × sin 6° = 80 × 0.1045 = 8.36 mm, directed almost perpendicular to the arm. That reads as a sideways shift that grows with the part angle while the reported angle stays correct — the signature of the wrong-order correction.

What you need

• A taught pick pose, position and orientation, plus the taught photo position where the camera calibration is valid.
• A trained model whose reference point you can name: the feature the reported X and Y point at.
• A camera-to-robot calibration at that photo position. For a camera carried on the gripper this is a hand-eye calibration: the nine-point hand-eye alignment procedure builds it, and the robot-held variant calibrated against a TCP tool frame matches this set-up.
• The values themselves, delivered to the controller over an Ethernet socket connection. The read of X, Y, angle and score from a vision camera over a robot socket shows the exchange.
• The tool's grasp clearance, because it sets the residual you can tolerate.

On the hardware side, the correction assumes a camera that reports a matched pose rather than a bare detection. A 5 MP color vision sensor with a global shutter and an M12 mount, reporting the pose over Fast Ethernet, is the class of camera this job needs.

Step 1: Put the TCP on the reference point or record the lever arm

Remove the lever arm where you can: teach the pick so the TCP lands on the model's reference point. With the two points coincident, r = 0, the rotation carries nothing and the whole correction reduces to offsetting by the reported X and Y and adding θ about the approach axis. No trigonometry.

Grippers often cannot do that: the jaws close on the part's sides and the TCP ends up between them, offset from the reported reference point. Then measure the lever arm once, at the taught pose, in the frame you will offset in: rx = Tx − Px and ry = Ty − Py, where T is the taught TCP position and P the reference point.

Step 2: Rotate about the reference point, then translate

1. Read dX, dY and θ from the vision result. If the calibrated camera axes sit at an angle φ to the frame you offset in, rotate the pair by φ first; a camera yawed 90° swaps the components and negates one.
2. Add θ to the taught orientation about the approach axis, the vertical axis on a top-down pallet pick.
3. Compute the term the TCP-fixed rotation missed:

4. Command the pick at the taught position plus (dX, dY) plus (Δx, Δy), with the corrected orientation.
5. If your controller can rotate a stored position about an arbitrary point in a user frame, steps 3 and 4 collapse into one instruction: rotate the taught pose by θ about the reference point, then translate by the reported X and Y. Same destination, less arithmetic.

Step 3: The arithmetic worked through

Taught pose in the pallet frame: reference point P = (1000.0, 200.0) mm, TCP T = (1040.0, 200.0) mm, so rx = 1040.0 − 1000.0 = 40.0 mm and ry = 0.0 mm. The camera reports θ = 12°, dX = −3.0 mm and dY = +5.0 mm.

cos 12° = 0.9781 and sin 12° = 0.2079, so:

• Δx = (0.9781 − 1) × 40.0 − 0.2079 × 0.0 = −0.88 mm
• Δy = 0.2079 × 40.0 + (0.9781 − 1) × 0.0 = +8.32 mm

Corrected pick point: x = 1040.0 − 3.0 − 0.88 = 1036.1 mm, y = 200.0 + 5.0 + 8.32 = 213.3 mm, orientation raised by 12°. The naive correction commands (1037.0, 205.0) mm and lands 8.36 mm short, almost all of it along y, which matches the chord 2 × 40 × sin 6° = 8.36 mm. With the arm lying along x, the error appears on y: a pure sideways shift, the drift this term exists to remove.

Step 4: Verify with two test picks

Pick a part presented square to the model, θ = 0, ten times. No lever-arm term is in play, so this residual is the floor set by calibration and translation. Then present a part rotated 15–20° and repeat. With the compensation in, the 15° residual should sit near the 0° residual. If it climbs back toward 2 × r × sin(θ/2) — 8.4 mm for the example arm — the lever arm's sign or the angle convention is wrong, not the math.

Budget the result the way you plan vision inspection error rates for the worst case: calibration, correction and repeatability together inside a quarter of the grasp clearance. For a jaw tool with 4 mm of clearance that is 1 mm from all causes, which asks for a calibration residual near 0.5 mm. Run the first compensated picks at reduced speed with the cell safeguards active.

Where the correction should be computed

Three homes work; choose exactly one owner, because a correction applied twice doubles both the rotation and the offset.

• In the camera: a vision tool taught in a calibrated frame can return the corrected point directly, and the robot moves straight to it.
• In a vision controller: it takes the socket role, computes (dX, dY) + (Δx, Δy) and sends finished coordinates to the robot, which keeps the trigonometry out of the robot language and testable against the worked example above.
• In the robot program: the shortest chain and no second device to maintain, at the cost of the transform living in the robot's own language.

If the computation moves off the robot, a vision controller holds both the socket and the math. The MV-VC3301H-128G66 provides five or more LAN ports, four built-in light channels, an Intel Core i3 processor and 8 GB of memory, with the VM6200Pro license included.

Common mistakes

• Rotating about the TCP and adding the raw X and Y: exact at θ = 0, wrong everywhere else, and the miss is the swing chord from the formula above.
• Keeping the two-image method as the permanent fix: it works, but it spends a reposition, a settle, a second acquisition and a second match on every pick.
• Offsetting in the base frame while the calibration lives in a user frame: even square parts then drift by the offset between the frames.
• Trusting the correction away from the taught photo position: an eye-in-hand calibration is valid at that pose, and a different approach height changes the geometry.
• Chasing a moved pallet with more math: the correction absorbs part-to-part variation, not a shifted frame, so re-reference the cell with one reference target after any fixture disturbance.
• Chasing a centimeter-scale residual with finer math: misses of 10–20 mm at every angle point at the calibration, and a 1–2 cm hand-eye error is a known set-up symptom, not noise.

Before you release the cell

• Photo position taught, documented and used for every acquisition.
• Model reference point named; rx and ry stored as constants.
• Angle sign confirmed with a +10° test part.
• Order in the code: orientation about the reference point, or the Δ term, then translation.
• Test picks at 0° and at 15–20° passed with the residual inside budget.
• Pallet re-referenced after any fixture disturbance.

To assemble the chain: the vision sensors and smart cameras category lists cameras that report matched poses, the cameras shop finder filters them by resolution and mount, and the vision controllers category lists the machines that take the socket role, with more detail in the robot vision topic page.

The decision, per case

• TCP on the reference point: offset by the reported X and Y, add θ about the approach axis. Exact at any angle, no compensation.
• TCP offset from the reference point: apply the Δ term, exact at any in-plane angle. Skip it only when 2 × r × sin(θ_max/2) stays under a quarter of the grasp clearance — at θ_max = 3° and r = 20 mm the swing is 2 × 20 × sin 1.5° = 1.05 mm, inside a 4 mm-clear jaw.
• Residual flat across angles: tune the calibration or the frame offset, not the correction.
• Residual growing with angle after compensation: flip the sign of ry or θ and rerun the 15° test.
• Parts that also vary in height or tilt: a planar correction has no axis for them, so absorb the variation in the tool or measure it with a 3D sensor before the pick.

For the planar case the order is the whole answer: rotate about the reference point, then translate, and add the lever-arm term whenever r is not zero. Load your case into the vision system configurator and check the pick chain end to end, or ask the team through the contact page.