Sort tray picking order by matching objects to cavity positions

Vision-guided tray picking returns detections sorted by match score, so the robot criss-crosses the tray chasing the strongest match and doubling back for the weak ones. A tray deserves the opposite: a fixed sequence, row by row, with empty cavities skipped and doubles flagged before the gripper moves. This guide shows how to assign every detected object to its nearest cavity centre and sort the pick list by row and column index, using only the tray pose your locate step already reports. The math is a rotation, a division and a rounding per object, and the acceptance rule is half the cavity pitch. You end up with a pick order that repeats every cycle and stops loudly on a mis-loaded tray.

Why match score is the wrong order for a tray

A match score measures how well a candidate region fits the taught model, not where the part sits; the score-threshold basics cover what that number does and does not tell you. Two identical parts in adjacent cavities score within a fraction of a point of each other, so highest-score-first degenerates into the vision algorithm's internal search order. The visible symptoms are long empty travel, a gripper that crosses loaded cavities on the way to the next pick, and a cycle time that changes whenever one part goes missing.

Sorting by score also hides loading faults. A mixed or upside-down tray still produces a plausible pick list, just in a nonsense order, and nothing halts the cycle. Detecting a full tray in one snapshot is the first half of the job; turning that detection list into a defined sequence is the second half, which is what this page adds. The batch-picking loop that runs until no parts remain solves continuation between cycles; it leaves the order inside one snapshot to score.

The method: tray pose, cavity grid, nearest-cavity assignment

Two locate steps and one sort do the whole job. The tray locate searches for the tray itself — a square of the tray's outline, a corner notch or the pocket wall — and reports the pose of the first cavity: position (x0, y0) in millimetres and rotation θ. The object locate runs on the same snapshot and returns every part as a centroid (x, y) with its score. Everything after that is arithmetic on the tray drawing: rows, columns and pitch.

The names below are mine; use whatever your software calls them. The relations are what matter:

Two details are worth fixing now. First, assign in tray coordinates, not machine coordinates: rotate every centroid into the tray frame before dividing by the pitch. Second, the acceptance radius is half the pitch — 25 mm on a 50 mm grid — so an object farther than that from every centre is not on a valid cavity and becomes a fault, not a pick. For rectangular cavities divide u and v by their own pitches and use half the smaller pitch as the acceptance radius.

What you need

• One snapshot covering the whole tray, shot from directly overhead. For a tray wider than the field of view, split the table into several snapshot positions and sort within each.
• A tray locate trained on the tray itself: outline, corner notch or pocket pattern. Never train it on a part.
• An object locate trained on the part: a parametric search such as a circle of the part's diameter, or a taught model, whichever your software offers.
• The tray drawing: rows, columns, cavity pitch, and the offset from the located feature to the first cavity centre.
• The camera-to-robot calibration you already pick with, so centroids arrive in robot coordinates.
• A few lines of script in the vision device for the rotation, assignment and sort — no extra hardware.

These vision sensors run both locate steps and the sort on the camera itself:

The MV-SC3050XC-16M-ATN-H pairs a 5 MP colour sensor with a global shutter and an M12 mount for a single overhead snapshot. The MV-SC5060XM-08M-WBN adds a 6 MP mono sensor with an 8 mm M12-mount lens on GigE; its rolling shutter is harmless here because the tray is static during the shot. For smaller trays the HK-TD032-12-WBN is a 1.6 MP mono vision sensor on Fast Ethernet.

If the sort script, logging or extra cameras outgrow a vision sensor, the LS-SMC-0017 is an X86-platform smart camera with 8.8 MP resolution and a C mount:

The steps

1. Locate the tray and store the pose. Save x0 and y0 in millimetres and θ in the angle unit your trigonometry functions expect. If the locate passes in teach and fails at run time, check the stream-versus-snapshot causes before blaming the math.
2. Compute the cavity centres. Loop r from 0 to rows − 1 and c from 0 to columns − 1 through the forward transform above. With different x and y pitch, substitute px and py.
3. Locate every object in the same snapshot. Keep each centroid and its score; the score travels along for logging but never enters the sort.
4. Rotate each centroid into tray coordinates with the inverse transform, giving u and v in millimetres.
5. Assign each object to a cavity. Take c = round(u / p) and r = round(v / p), then compute the distance from (u, v) to that cavity's centre. Accept at 25 mm or less for a 50 mm pitch; anything beyond is unmatched.
6. Resolve claims, then sort. If two objects accept the same cavity, keep the smaller distance and raise a fault for the other. Drop unmatched objects after logging, recompute the order indices, and sort ascending by r × columns + c. For a serpentine route use r × columns + c on even rows and r × columns + (columns − 1 − c) on odd rows, then hand the list to the robot.

Worked example: 6 × 4 tray, 50 mm pitch, tray rotated 0.6°

Inputs. The tray drawing gives 6 columns × 4 rows and a 50 mm pitch on both axes. The tray locate reports the first cavity centre at x0 = 412.3 mm, y0 = 218.7 mm and θ = +0.6°. The object locate returns one part at x = 516.3 mm, y = 314.8 mm, all in robot coordinates. With θ in degrees, cos 0.6° = 0.999945 and sin 0.6° = 0.010472.

Rule in words. Rotate the centroid into the tray frame, divide by the pitch and round to the nearest cavity index, then confirm the leftover distance sits within half the pitch.

Arithmetic.

• u = (516.3 − 412.3) × 0.999945 + (314.8 − 218.7) × 0.010472 = 104.0 × 0.999945 + 96.1 × 0.010472 = 103.994 + 1.006 = 105.0 mm
• v = −104.0 × 0.010472 + 96.1 × 0.999945 = −1.089 + 96.095 = 95.0 mm
• c = round(105.0 / 50) = round(2.10) = 2, and r = round(95.0 / 50) = round(1.90) = 2
• distance = √((105.0 − 100.0)² + (95.0 − 100.0)²) = √(25 + 25) = 7.1 mm

Cross-check with the forward transform: the centre of cavity (2, 2) computes to X = 412.3 + 100.0 × 0.999945 − 100.0 × 0.010472 = 511.2 mm and Y = 218.7 + 100.0 × 0.010472 + 100.0 × 0.999945 = 319.7 mm. The detected part sits √(5.05² + 4.94²) = 7.1 mm from that point — the same answer, so the two transforms agree.

Result and decision. The part maps to cavity row 2, column 2 at 7.1 mm from centre, well inside the 25 mm acceptance radius, and takes pick order 2 × 6 + 2 = 14 (0-based), the 15th of 24. The rotation cannot be skipped: the far column centre sits 5 × 50 = 250 mm from the locate origin, and the tray angle swings it 250 × sin 0.6° = 2.6 mm sideways. Let the tray arrive at 3° and the same swing grows to 250 × 0.052336 = 13.1 mm — half the acceptance budget spent on rotation alone.

Make sure one snapshot separates the parts

The acceptance math is forgiving; the object locate is not. Aim for at least 40–50 pixels across the smallest part dimension so the model match and the centroid stay stable. Spread over a 600 mm tray long side, the common 5 MP sensor format (2448 × 2048 pixels) gives 600 / 2448 = 0.25 mm per pixel, so a 20 mm part spans 80 pixels and a 50 mm cavity pitch spans 200. The common 1.6 MP format (1440 × 1080) gives 600 / 1440 = 0.42 mm per pixel and 48 pixels on the same part — above the minimum, with less margin for glare and edge blur.

To hold 40 pixels on a 20 mm part you can spend at most 20 / 40 = 0.5 mm per pixel, which is 600 / 0.5 = 1200 pixels across the long side. Pixel error then barely touches the assignment: a 2-pixel centroid error is 2 × 0.25 = 0.5 mm, about 2% of the 25 mm radius. The camera resolution calculator runs this the other way, from field of view and part size to the pixels you need.

Assignment faults and what they mean

Half-pitch acceptance turns tray problems into named faults instead of mystery picks. Read the assignment output before the robot moves:

Checks before the first live pick

• Dry-run the sort. Print cavity, distance and order index for every object. Every distance 25 mm or less, and no cavity claimed twice.
• Rotate the tray on purpose by about 1°. The locate should read roughly 1°, the far column shifts 250 × sin 1° = 4.4 mm, and every part should still map to its own cavity.
• Pull one part out. Its cavity disappears from the list while every other order index stays put.
• Double one cavity. The duplicate fault fires and the closer object stays on the list.
• Run one pick. The robot targets the cavity the list names, not the strongest score.

Common mistakes

1. Trigonometry in the wrong angle unit. sin(0.6°) = 0.010472, but the sine of 0.6 radians is 0.565 — a 54× error that scatters every cavity beyond its pitch. One dry-run print catches it.
2. Assigning in machine coordinates. Without the rotation into tray coordinates, the far column lands 2.6 mm off at 0.6° of tray rotation and 13.1 mm off at 3°.
3. Acceptance radius at or above half the pitch. A part sitting between two centres gets claimed by both, and the pick flips between cycles.

4. Training the tray pattern on a part. The pattern degrades as parts are picked; train on the tray outline, notch or pocket wall.
5. Sorting before removing faults. Recompute the order indices after dropping unmatched objects, or the robot seeks a cavity that is no longer on the list.

When nearest-cavity sorting is the wrong tool

Three cases break the cavity model. First, parts loose enough to wander past half the pitch: fit a liner, or locate each part individually as in random-bin picking — choosing a vision approach for bin picking shows when a snapshot-based approach still works for unstructured picking. Second, piles and stacked layers: there is no grid to map to, so treat it as bin picking. Third, a tray larger than one field of view: split it into several snapshot positions and sort within each snapshot, or shoot the tray in tiles and merge the lists in robot coordinates.

The inverse task — filling trays rather than emptying them — reuses the same grid math in reverse: sorting parts by face into tray layers is that problem with an item-type decision added. And if your software prefers teaching models to writing scripts, you can teach the empty cavity itself as a part and detect vacancies directly.

The decision and the next step

Commit to this: acceptance radius at half the smallest cavity pitch — 25 mm on a 50 mm grid; assignment in tray coordinates by rounding u and v to the nearest index; pick order by row and column, serpentine when travel matters. A tray that fails any of those checks stops the cycle with a named fault instead of a wrong pick.

If you also need the hardware, the smart cameras category lists vision sensors and smart cameras that run two locates and a sort on board; the camera finder filters by resolution and interface, and the configurator builds the rest of the station around your choice. The applications topic page gathers the related picking guides, and open questions go through /contact/.