ArUco marker pick and place: from marker pose to grasp

Your hand-eye calibration is done and the detector hands you a marker pose every frame. The missing piece is one fixed transform: the offset from the marker to the point the gripper must actually close on. Compose it with the calibration and the detection, in the right order, and the arm lands on the part wherever it presents; skip it and you will chase millimetre pick offsets for a week. This guide is for robot programmers and integrators wiring a fiducial into a pick cell. It gives you the transform chain for eye-to-hand and eye-in-hand set-ups, a touch-off procedure that calibrates the marker-to-grasp offset once per part, a fully worked numeric example with the arithmetic written out, and the pre-pick checks that catch marker-size, glare and blur errors before your first production pick.

What you need

• A calibrated camera and lens. Marker pose inherits every intrinsics error, so calibrate the lens first; Camera intrinsics for hand-eye calibration: fix the pose first shows the procedure.
• The hand-eye result, published once as a fixed transform; Publish eye-to-hand calibration as a static camera transform shows the pattern.
• A printed marker on matte stock, glued flat to one named face of the part, with the side length measured using calipers.
• A rigid camera stand for an eye-to-hand camera, or a bracket on the arm for eye-in-hand.
• A vision controller with spare LAN and USB ports for the camera, the trigger and the gripper I/O.

Any of the area scan cameras detects ArUco-style fiducials once the lens is calibrated. Two that cover the common cases:

Choose by motion and by marker size: global shutter when parts move under the camera, more pixels when the marker is small or the standoff long.

What a marker detection gives you

Each detection returns three things: the marker's ID, its four corners in pixels, and a pose — rotation plus translation — of the marker frame in the camera frame. The marker frame's origin is the marker centre and its z axis points out of the printed face toward the camera. Everything downstream hangs off that pose, so two inputs decide its quality: the marker size you pass to the solver, and the number of pixels the marker covers.

The solver needs the printed side length in millimetres, and the reported range scales directly with it. Enter 50 mm for a true 60 mm marker and a real 300 mm standoff reports as 300 × 50 / 60 = 250 mm, a 17 percent range error with no visual clue. Measure the print itself, because printing often scales files. A wrong size leaves rotation untouched, so the symptom is a pick that lands consistently short or long along the viewing axis — a useful diagnostic.

Pixel budget is the second input. Image scale at the part equals field of view divided by pixel count, and a couple of hundred pixels on the marker side is a workable floor; more when the part tilts. To size the focal length for the field of view your marker needs at the standoff, use this calculator:

Tilt is the other limit. Pose quality holds while the camera stays within about 45 degrees of the marker normal; beyond that the square corners compress and the error climbs quickly, so reject those frames rather than pick from them. A stationary marker should also repeat its pose frame to frame within a few tenths of a millimetre; a larger spread points to glare, flicker or motion blur, not to the algorithm.

Converting the camera-frame pose into a base-frame object pose is its own step: Object pose in camera frame at runtime after hand-eye calibration walks the conversion, and Eye to hand object pose transform for robot picks covers the eye-to-hand case.

The transform chain from detection to grasp

Read T_base_camera as "the camera pose expressed in the base frame". Three transforms, three owners: T_base_camera is static and comes from the hand-eye calibration, T_camera_marker arrives fresh with every detection, and T_marker_grasp is fixed for a given part and marker placement. The third is the one nobody hands you — you build it once, in the next section. Multiply the three 4×4 matrices together and the result is the grasp pose in the base frame, ready for the motion planner.

Compose matrices; never add distances and angles separately. A rotated offset shifts position — in the worked example below, 10 degrees of tilt moves a 40 mm offset about 7 mm sideways. In an eye-in-hand cell the flange pose changes every cycle, so read it from the controller at the moment of detection; the composition order in Eye-in-hand pixel to world coordinates: chain the transforms is the same chain, one link longer.

Calibrate the marker-to-grasp offset once

CAD gives the offset when you have a model. When you do not, a touch-off on the real part works and absorbs the print-and-place tolerance of the marker itself. Fix one part anywhere in the working volume, then:

1. Detect the marker and compute its base-frame pose: T_base_marker = T_base_camera · T_camera_marker.
2. Jog the empty gripper to the exact grasp pose — the point where the fingers close — and record it as T_base_grasp.
3. Compute the offset as T_marker_grasp = inverse(T_base_marker) · T_base_grasp and store it against that marker's ID.
4. Move the part to a second, clearly different pose and command the composed target.
5. If the fingers still close on the grasp point, the offset is good. If the miss rotates with the part, the marker is loose or the offset was recorded in the wrong frame.

If picks still land offset after a clean touch-off, Diagnose an eye in hand pick point offset works through the usual suspects — sign errors, swapped frames, a stale calibration — and the same fault tree applies in eye-to-hand cells.

Worked example: a 60 mm marker seen at 300 mm and 10 degrees

The case: a 60 mm marker printed on the top face of an 80 mm tall box, grasp point at the box centre, 40 mm below that face. The camera hangs above the cell, and its calibration places it at (600, -150, 380) mm in the base frame, looking straight down, camera x along base x, camera y and z flipped. The detector reports the marker at (-50, +30, +300) mm with 10 degrees of tilt, the box leaning so the face normal tips toward -y. Approach standoff: 100 mm. Optics: a 1/1.7-inch sensor, about 7.5 mm wide with 4,000 pixels across, fitted with a 16 mm lens.

• Field of view: 7.5 mm × 300 / 16 = 140.6 mm, so pixel scale is 140.6 / 4000 = 0.035 mm per pixel at the part.
• Marker span: 60 / 0.035 ≈ 1,700 pixels, far above the 200-pixel floor — corner quality is not the limiting factor here.
• Base-frame position: camera y and z are flipped, so (-50, +30, +300) becomes (-50, -30, -300); adding the camera origin puts the marker at (550, -180, 80) mm — 80 mm above the table, exactly the box top. The chain checks itself against the geometry.
• Face normal at 10 degrees: sin 10° = 0.174, cos 10° = 0.985, so the outward normal is (0, -0.174, +0.985) and the inward normal, toward the grasp point, is (0, +0.174, -0.985).
• Grasp point: marker plus 40 mm along the inward normal = (550, -180 + 40 × 0.174, 80 - 40 × 0.985) = (550, -173.0, 40.6) mm, call it (550, -173, 41) mm.
• Approach point: grasp plus 100 mm along the outward normal = (550, -173 - 17.4, 40.6 + 98.5) = (550, -190.4, 139.1) mm, call it (550, -190, 139) mm.

Command the arm to (550, -190, 139) mm, descend to (550, -173, 41) mm, close the gripper. The tilt alone moved the pick 40 × 0.174 ≈ 7 mm along +y; commanding the marker centre with a fixed vertical approach would have missed by exactly that amount. The dominant accuracy term is not the detector but the size you entered: 0.6 mm of print error on a 60 mm marker is 1 percent, and 300 × 0.01 = 3 mm of range error. Measure the print, then trust the chain — and reject any frame showing more than 45 degrees of tilt.

Checks before the first production pick

• The size entered in the solver matches the calipers, not the drawing file.
• Range sanity: at one known pose the reported distance matches a ruler, which catches a wrong marker size in seconds.
• Exposure for moving parts: blur equals exposure times speed. At 100 mm/s with a 0.2 mm blur budget, expose for 2 ms or less; faster belts scale the limit down proportionally.
• Shutter: parts in motion want a global shutter — the MV-CU020-90GC above covers that — while a stationary part is fine on the rolling shutter sensor.
• Matte marker, no glossy tape, no specular reflections across the print; glare shifts detected corners and therefore the pose.
• A rigid mount for an eye-to-hand camera; a 700 mm professional slide-table stand holds the pose fixed through shift changes:

• After any bump to the camera, rerun the verification target before production; Hand-eye calibration accuracy verification: test before you trust it gives pass criteria.
• Compute and I/O in one box: detection loop, robot handshake and lights run together on a controller that ships with 8 GB of memory, 5+ LAN ports and four built-in light channels:

When a marker is the wrong tool

Markers need a flat, visible, matte face and a viewpoint that stays put. Reach for another method when the part has none, when the surface is glossy or washdown-cleaned, or when the part never pauses long enough for a sharp frame; 2D or 3D camera for conveyor pick and place: choosing the system compares the alternatives by cell layout and part presentation. When the marker is right but the surrounding program is the open question — buffering targets, dropping misses, hand-off timing — Structure a vision-guided pick and place program around drop targets picks up where the chain ends.

The whole chain fits in a page of code: detect, compose three matrices, command the pose. Size the camera, stand and controller as one build at the camera step of the configurator, browse the systems topic page for the calibration guides behind this one, or bring the cell drawing to the contact page and we will check the chain with you.