Vision-guided robot pick and place: architecture that works

You have a collaborative robot, an electric gripper and a 3D camera on the bench, and the demo has to pick parts dropped randomly into a bin. The architecture decides whether that happens on the first day or the tenth. This guide fixes the two roles — who computes poses, who moves and grips — shows the pose handoff over TCP, walks the resolution and cycle-time arithmetic with every number shown, and ends with a parts list you can order against. It is written for anyone building a first vision-guided robot pick and place cell, or rescuing one that keeps missing picks.

Who runs the show: the vision software or the robot program?

Let the robot program be the master of every motion and gripper command, and the vision system the servant that answers each request with one calibrated pose, because a single owner of the physical motion is what keeps a pick-and-place cell predictable.

Two architectures compete. In the robot-led one, a plug-in from the vision vendor installs on the teach pendant and adds drag-and-drop nodes for calibration, capture and picking; the pendant program calls capture, receives a pose, then runs its own approach, grip and place moves. In the vision-led one — master-control mode — the vision PC scripts the robot's motion from its own development environment.

The demo killer is the gripper. Master-control modes command the motion but rarely carry a driver for a third-party electric gripper, so the gripper ends up with no owner: the vision program cannot close it and the pendant program is not really in charge. Robot-led costs nothing and fixes this. The gripper's own plug-in then lives in the same pendant program as the vision nodes, grip commands run inside the taught cycle, and error recovery stays in one place. The ownership split is mapped in vision system and robot gripper control architecture: who drives what, and one full cycle is traced in 2D vision guided robot pick place grip: how the cycle works.

Keep the demo on the static-snapshot pattern: the robot presents the bin under the camera, stops, the vision side captures and computes, then the robot moves. On-the-fly picking, with the camera sweeping on the flange, buys under a second per cycle at best and adds encoder tracking; skip it on a first build. Structure the pendant program around explicit drop targets so a missed pose cannot wedge the arm mid-path — the pattern is laid out in structuring a vision-guided pick and place program around drop targets.

How does the pose cross from the 3D camera to the robot controller?

Send one short TCP message per pick — a comma-separated X, Y, Z, roll, pitch, yaw string in robot-base coordinates that stays under about 60 bytes, so any Gigabit link carries it instantly.

As a worked shape, the message the robot receives looks like 6.2, 143.8, 21.5, 0.0, 180.0, -37.4: X, Y and Z in millimetres, then roll, pitch and yaw in degrees, all expressed in robot-base coordinates. That is about 40 characters, so bandwidth is a non-issue; the handshake is the real design work. A minimal pattern that survives a live demo: the robot connects to the vision service, sends TRIG, and waits; the vision side captures, matches, and replies with either the pose string or NO_PART; the robot branches to its pick or its no-part routine, and a timeout guards every wait.

Three conventions cause nearly every first-week failure: units (millimetres versus metres), rotation format (roll-pitch-yaw in degrees versus quaternion) and frame origin (camera frame versus robot base). Agree all three in writing before any code starts, and do the frame conversion on the vision side so the robot receives base-frame numbers only. The socket exchange is built out step by step in sending camera part coordinates to a robot controller over TCP, and the no-part branch — the one that otherwise stalls the cell in front of your audience — is covered in vision no-part error: restart the robot cycle.

If the camera vendor supplies pendant plug-in nodes, they implement this exchange for you. Keep the wire format straight anyway: when a pick lands 4 mm off at 11 p.m., you will be reading raw strings.

How much camera resolution does the pick accuracy need?

Size the camera so one pixel spans about a third of your placement tolerance: for a ±1 mm pick tolerance over a 650 mm field of view that is roughly 2,000 pixels across, which a 5 MP sensor covers with margin.

The rule of thumb: resolve the scene at least three times finer than the placement tolerance, because matching, calibration and gripper compliance each spend part of the pixel budget. In formula form:

Worked example, with illustrative inputs: a bin 600 mm long, imaged with mounting margin on a 650 mm field of view, and a ±1 mm placement tolerance. Target resolution: 1 mm / 3 ≈ 0.33 mm per pixel. Pixels needed across: 650 mm / 0.33 mm ≈ 1,970. A 5 MP sensor with 2,448 pixels across delivers 650 mm / 2,448 ≈ 0.27 mm per pixel — inside target with about a fifth to spare. A 2 MP sensor at 1,920 pixels across delivers 0.34 mm per pixel and misses by a hair. So: 5 MP class for a 650 mm field at ±1 mm.

Depth is the separate axis. Lateral pixel size says nothing about Z accuracy, and Z error becomes tilt and height error at the grasp. Structured-light cameras of this class quote depth accuracy in the tenths-of-a-millimetre range at rated stand-off; read the figure from the product card rather than deriving it from pixel size. SDK down-sampling changes effective resolution at runtime — 3D point cloud X Y Z resolution settings without distortion shows where that hides.

Which 3D camera type fits a bin-picking cell?

For random picks from a tote or bin, choose a projected structured-light or laser galvanometer stereo camera; reserve single-line profilers for conveyor scanning and ToF for coarse volume and presence checks.

Shiny, black and translucent parts break naive structured light; the countermeasures are covered end to end in 3D bin picking for flat parts: choosing a vision approach, and the 2D-versus-3D conveyor decision is made in 2D or 3D camera for conveyor pick and place: choosing the system.

Two series cover most demo cells. The MV-DPS200P-02 is a projected structured-light 3D camera built for random bin picking, and the MV-DLS300P-04 is a laser galvanometer stereo camera for larger volumes and line-side work.

For volume and presence checks where fine depth accuracy is not the point, a ToF sensing camera such as the MV-DT01WMU covers the whole scene in every frame — the right tool when the question is whether the tote is full, not where exactly the part sits.

Galvo and profiler units project laser lines: check the class label at the aperture and keep the scan plane below eye height in a demo cell.

What does the hand-eye calibration have to cover?

Calibrate the transform between the camera frame and the robot base once per mounting, using 9 to 16 distinct robot poses over a calibration board, and expect the residual to sit within a few tenths of a millimetre before you trust any pick.

Two transforms exist and a cell needs exactly one. Eye-to-hand: camera fixed on a stand, calibration solves camera-to-base once and every pick reuses it. Eye-in-hand: camera rides the flange, calibration solves camera-to-flange, and each pick applies the robot's forward kinematics to place the view. The pendant plug-ins for vision systems walk both mountings with a guided board routine; choosing between them is a mounting question, not a software one.

The procedure that survives contact with a real cell: park the vendor board flat in the workspace, then move the robot (eye-to-hand) or the board (eye-in-hand) through 9 to 16 poses that vary rotation and not just position — a set that never tilts hides a degenerate axis inside a comfortable-looking residual. Save the pose positions so the next camera re-mount re-runs the same recipe in minutes; the capture recipe is written up in hand-eye calibration dataset acquisition with saved robot poses. When the board cannot cover a stereo camera's whole field and the routine stalls at the last pose, the fix is in stereo calibration board coverage: fix the stall at the end.

Then prove it before the first pick: command the robot to touch a computed target five times and measure the spread with a dial gauge or feeler stock. Demo-grade repeatability is a few tenths of a millimetre; if the spread exceeds a third of your placement tolerance, suspect board flatness or pose variety before the camera.

How fast will the cycle run and where does the time go?

Budget the cycle as capture plus computation plus robot motion plus gripper actuation: a static-snapshot pick typically lands between 3 and 4 s, which is 15 to 20 picks per minute, and the robot move is usually the biggest block.

Put a number on each block and the total stops being a mystery. Example allocation for a static-snapshot cell with an electric gripper:

Each figure is an example allocation, not a rating: stopwatch the first hour of running and replace them with your own. The ranking — robot move biggest, vision second — holds across most static-snapshot cells, and it tells you where the seconds live.

The cheapest speed-up needs no new hardware. Trigger the capture as the robot clears the workspace on its return move, and the capture-plus-computation block (0.9 s here) hides inside the retreat-and-place move: effective cycle 3.6 s - 0.9 s = 2.7 s, about 22 picks per minute. If capture itself is the drag, the cause is usually settings rather than physics — 3D camera capture time slower in the API than in the viewer walks the usual culprits.

What does the controller and network need to carry it?

Give the cell a fanless vision controller with at least one Gigabit LAN port per camera plus a spare for the PLC, PoE camera ports when the camera mounts away from power, and 16 GB or more of memory for point-cloud work.

A fanless vision controller bolted into the cell beats a tower PC on vibration, dust and power cycling. The port arithmetic decides the model: one Gigabit port per 3D camera, one more for the PLC or factory network, PoE where a separate camera power drop is awkward. Memory deserves respect — point clouds, matching models and the runtime together want 16 GB or more once you go past a single camera and a handful of parts.

Two models bracket the common builds:

All 2 share: Humidity 20% RH to 95% RH (no condensation); Certification CE, KC. Values as printed in each product's data sheet; open a part for its drawing and full specifications.

The entry build runs one 3D camera with traditional matching comfortably and includes a VM6200Pro license out of the box. The headroom build adds PoE camera ports, eight built-in light channels and four times the memory, which is what a second camera, a 2D quality station or a trained model will ask for later. Before ordering, run the compatibility chain end to end — camera SDK, controller operating system, robot interface — as laid out in 3D camera robot integration: check compatibility.

If the camera refuses to appear on the network after wiring, the subnet-and-link checklist in 3D camera not pingable on a Linux controller: fix subnet and link clears most cases in minutes.

When is the usual 'vision drives everything' answer wrong?

Master-control mode — the vision software scripting the robot directly — fails the moment a third-party gripper joins the cell, because it commands motion but has no driver for the gripper, leaving the gripper with no owner in any program.

The mode has legitimate uses in tightly scripted cells, but three symptoms say walk away for a gripper build: the gripper has no driver inside the vision environment; error recovery lives in two places at once; and every motion becomes a script line on the vision PC rather than a pendant program your integrator can audit. Robot-led architecture costs one short message per cycle and buys all three back.

Sometimes the whole 3D stack is the wrong answer, and there are cheaper wins. Parts presented in a nest or on pins need one 2D snapshot and a straight-down pick — the pattern in smart camera socket handoff to a robot: straight down pick. Tiny parts scattered loose belong in a flexible feeder rather than a bin — see vision-guided flexible feeding: architecture for random pick. When the end effector is vacuum, the integration question is signal mapping rather than pose math — vacuum gripper signal mapping for vision-guided picking sorts the wiring.

What goes on the parts list before you order?

Order the cell as six line items — 3D camera, vision controller, robot-side interface, gripper with its plug-in, calibration board and a part-present sensor — so nothing arrives without its software counterpart.

1. 3D camera matched to the task — the projected structured-light MV-DPS200P-02 for random bin picks, the laser galvanometer MV-DLS300P-04 for larger volumes and line-side work.
2. Vision controller with one spare LAN port and, ideally, PoE — the two compared above cover the entry and headroom builds.
3. Robot-side interface — the vision vendor's pendant plug-in, or your own TCP client written in the robot's script language.
4. Electric gripper plus its pendant plug-in, so grip commands live in the same program as the vision nodes.
5. Calibration board from the camera vendor, sized to cover at least half the field of view.
6. Part-present photoelectric sensor for cycle triggering, so the camera stops burning captures on an empty bin.

One choice upstream of all of this matters more than the camera list: axis count. If every pick is planar — part presented flat, gripped from above, placed flat — a 4-axis arm does it faster and cheaper and every rule above still applies; the trade-offs are compared in 4-axis versus industrial 6-axis robots for pick and place.

Total the parts against the cycle target from the section above: if the demo must show 12 or more picks per minute, order the PoE, higher-memory controller now, because the capture-overlap trick is what carries a static-snapshot cell past 20. Browse the 3D sensors range and the vision controllers, start from the system configurator, or send the part photo, bin dimensions and cycle target through the contact page and we will spec the cell with you. More end-to-end builds sit on the applications topic page.