Vision pick coordinates on a tilted plane: build your own transform

A vision guidance loop that picks cleanly off a base-parallel table falls apart the day the parts move into a tray tilted at 45°. The camera still finds the part, the calibration still runs, and the robot still drives to the wrong place — every cycle, in the same way. This guide is for anyone wiring an eye-in-hand camera to a robot that must pick from tilted trays or bins. You will teach the tray as its own coordinate plane with three recorded points, square the camera to it, convert pixel offsets to millimetres with one measured scale, and check the whole chain to about 1 mm before the first real pick.

Why does a base-plane calibration fail on a 45° tilted tray?

The calibration projects pixel offsets onto the horizontal plane it was taught, and a point 100 mm up a 45° slope is really 70.7 mm across plus 70.7 mm of height the map never sees, so the robot misses the part every cycle.

A flat calibration is a two-dimensional map: pixel in, base X and Y out, at one fixed height. It has nowhere to put the height component, and at 45° that component is as large as the horizontal one: 100 × cos 45° = 100 × 0.707 = 70.7 mm across and 70.7 mm up the slope. The map reports a point 70.7 mm short along the slope every single cycle. Re-running the calibration, re-teaching the fiducial or tightening exposure cannot fix it, because the map is not inaccurate — it answers a different question than the one the tray asks.

Guidance plug-ins that chain a camera result into base coordinates carry the same assumption: one working plane, parallel to the base. For base-parallel stations that assumption holds and a full camera-to-robot transform calibration is the right tool. A tilted tray breaks the assumption, so the fix is not a better calibration of the same kind. The fix is a different working plane, defined and taught by you.

What does the camera actually give you on a tilted tray?

The image gives you two numbers — the part's row and column offsets in pixels from a reference point — and everything beyond that, millimetres, tray coordinates and the robot target, is math you define; on a tray the pixel scale works out near 0.078 mm/px in the example below.

The eye-in-hand pixel-to-world chain builds that bridge for a base-parallel plane. A tilted tray keeps every step of the logic and swaps in a plane you define. Put a fiducial on the tray at the plane origin P1, move the robot to a fixed imaging pose, and record where that fiducial lands in the image: the reference pixel (col0, row0). Every cycle then re-creates the imaging pose, locates the part, and subtracts — part pixel minus reference pixel is the part's position inside your plane, waiting only for a scale.

The surrounding machinery — transform chains, verification discipline, robot-side integration — is collected on the systems topic page.

What do you need before you teach the plane?

You need an eye-in-hand camera with a global shutter, a lens whose field of view covers the tray pocket with about 20% margin, one part of known width for the scale, pendant access to record three poses, and the tray mounted at its final angle.

• Camera. Global shutter, because the camera rides a moving arm; rolling shutter smears images taken while the robot settles. Any area scan camera between 2 and 6 MP does the job — the pixel budget below shows why more resolution buys little here.
• Lens. Fixed focal length, framing the tray pocket with roughly 20% margin so fiducial and farthest part sit inside the field with contrast to spare.
• Scale part. One part or gauge block of known width, calipered to 0.1 mm.
• Bench. Camera and lens held still while you check focus and field before the arm is involved; a slide-table camera stand keeps the geometry fixed while you measure.
• Tray at final angle. Teach against the tray as mounted in production; re-clamping it afterwards moves all three points.

A 2.5 MP global-shutter GigE camera such as the MV-CU020-90GC clears the pixel budget with margin, and its C-mount accepts the fixed-focal lenses this field needs.

For bench time, a slide-table stand such as the LS-MNT-0040 holds camera and lens steady while you measure focus, field and scale.

Does the camera have to look straight down the tray normal?

Point the optical axis within about 1° of the tray normal: then the sensor plane is parallel to the tray, one millimetre-per-pixel scale is valid across the whole field, and at 3° off you already collect a 1.3 mm bias on a part 80 mm from the image centre.

A single scale is exact only where the tray plane runs parallel to the sensor. Tilt the optical axis by θ and magnification changes across the field by roughly (x / W) × tan θ, with x the part offset from the image centre and W the working distance. This is a first-order estimate, and it sizes the problem: at x = 80 mm, W = 250 mm and θ = 3°, the scale error is 80 / 250 × tan 3° = 0.32 × 0.0524 = 1.7%, which on an 80 mm offset bends the position by 80 × 0.017 = 1.3 mm. At 1° of tilt the same offset moves 0.45 mm — inside a 1 mm budget with room to spare.

Check squareness on the bench before teaching anything: image a rectangular pocket and compare the pixel length of its near and far edges. At 3° of tilt over a 120 mm pocket depth at a 250 mm working distance, the near edge sits at 250 − 60 × tan 3° = 246.9 mm and the far edge at 253.1 mm, so the far edge images 6.2 / 250 = 2.5% short. When the rectangle reads square within about 0.5%, the axis is square to the tray.

With the camera on the arm this costs nothing extra: each tray gets its own imaging pose aimed at its own normal, and two trays at 45° on opposite sides of the robot become two independent sets of taught numbers.

How do you teach the tray plane with three robot points?

Jog the tool to three points on the tray surface and record their base coordinates: one origin, one along the tray's long edge, one square to it — those three poses define the plane frame, and every pick on that tray becomes a two-number offset from the origin.

1. Fix the TCP. Use one TCP, touched off carefully, for the three points and for the picks; taught points inherit TCP error directly.
2. Record P1. Jog the tool tip to the plane origin — a pocket corner or fiducial hole — and record the base-frame pose.
3. Record P2 a good distance along the tray's long edge, 200 mm if the tray allows; a longer baseline dilutes jog error in the edge direction.
4. Record P3 about as far away, roughly square to P2, off the long edge.
5. Compute the frame from the three poses:

u and v span the tray surface, w is the tray normal, and the pick approach runs in along −w. The same construction, checked against a worked plate example, appears in plane orientation from known 3D points.

6. Record the reference pixel. Move to the tray imaging pose, image a fiducial sitting at P1, and store the pixel together with that pose; every cycle re-creates the pose before the shot.
7. Repeat per tray. The second tray is a second P1/P2/P3, its own reference pixel and its own scale.

How do you convert pixel offsets into millimetres and a robot target?

Measure a scale once on a part of known width and every pick after that is two multiplications: millimetres equal pixel offset times the scale, and the robot target is the taught origin plus those millimetres along the tray's two edge directions.

The row term flips sign because image rows count downward; if your library numbers rows the other way, flip it back.

Worked example: a 240 mm field at 0.0781 mm/px

Scale. An 80.0 mm gauge part spans 1024 px in the image: s = 80.0 / 1024 = 0.0781 mm/px. Across the full 3072 px width of a 6 MP 4:3 sensor that is 3072 × 0.0781 = 240 mm of field, exactly what the lens was chosen to cover at a 250 mm working distance.

Position. The fiducial at P1 sits at pixel (1536, 1024) — the image centre. Vision finds the part centre at (1850, 620). Then a = (1850 − 1536) × 0.0781 = 314 × 0.0781 = 24.5 mm along u, and b = (1024 − 620) × 0.0781 = 404 × 0.0781 = 31.6 mm along v.

Target. The robot target is P1 + 24.5 mm along u + 31.6 mm along v, tool Z pointed along −w, approached from 50 mm above the plane. Composing that rotated frame inside the controller is controller-specific; a cobot plane pose math walkthrough for rotated pallet frames shows how positions compose in a rotated frame on a cobot-style controller.

Set the scale budget before buying anything: keep s at or below 0.33 mm/px — three pixels per millimetre — so vision error stays a third of a 1 mm tolerance. A 240 mm field then needs at least 240 / 0.33 = 727 px across the pocket; a 2.5 MP sensor's ~2,000 px delivers about 0.12 mm/px over that field, nearly three times finer than the budget. The resolution calculator below turns a field width and a tolerance into the pixel count to shop for.

How do you verify the chain before the first real pick?

Run ten dry cycles with the computed targets lifted 50 mm, touch off or indicator-check the XY at three pockets, and accept the method when every repeat lands within about 1 mm — the result this three-point chain delivers with the camera square to the tray.

1. Scale stability. Re-measure the known part at a pocket far from the fiducial. If its pixel width differs from the centre measurement by more than about 1%, square up the camera or store a per-region scale for the outer pockets.
2. Dry cycles. Command each computed target lifted 50 mm along the normal, step down in manual mode, and touch off the true part position with a marker or indicator tip.
3. Ten repeats per tray. Log the XY error at each touch-off. With the camera square and one measured scale, picks repeat to about 1 mm — the number to hold this method to.
4. Read the pattern. The same offset in the same direction on every pick incriminates the reference pixel or the TCP; scatter that changes pick to pick points at tray play or threshold drift. The eye-in-hand pick offset diagnosis separates those cases, and the same verification discipline used for hand-eye accuracy applies here unchanged.

When is a taught pixel scale the wrong answer?

The scale is exact only on the taught plane: a part sitting 10 mm above it shifts its computed position by about 3.2 mm at an 80 mm offset (80 × 10 / 250), so varying part heights, steep viewing angles or moving trays call for a 3D sensor instead.

Height is the whole failure mode of a plane method. XY error grows as offset × height error ÷ working distance, so at an 80 mm offset and a 250 mm working distance a part 2 mm proud shifts 80 × 2 / 250 = 0.64 mm, and 3 mm proud consumes the entire budget at 0.96 mm. Pockets machined to one depth stay inside the method; loose parts in a formed tray do not.

• Fixed tray, consistent part height — the case this method is built for: three-point plane, one measured scale, about 1 mm of repeat.
• Several tray angles — one plane, reference pixel and scale per angle; the program selects the frame, not a measurement.
• Heights varying more than about 2 mm — measure each part in 3D rather than assuming a plane. A laser profiler or structured-light snapshot removes the height assumption outright, and the 3D camera working distance guide covers standing one off correctly.

Where part heights genuinely vary, a laser profiler such as the MIS-DP3850-01P measures each part's height before the pick, so the plane assumption leaves the equation.

• Base-parallel stations elsewhere in the cell — keep the plug-in or a published static camera transform for those and run the tray planes alongside; both coexist in one program.

To pick the camera, lens and stand for a tilted-tray station, start in the camera finder or go straight to the configurator; if the cell's geometry turns out stranger than one tilted plane, the contact page reaches the applications team.