Robot pick and place vision without an external PC

You want the camera and the image processing to live with the robot: no external PC, no server, nothing else on the bench. Three architectures do that, and they are not equally fast. You can run a small vision program as a background process on the robot controller itself, put the processing inside a smart camera, or bolt a fanless vision controller into the cabinet and let it own the cameras. This explainer compares the three on processing power, cycle time and integration effort, shows the transfer and matching arithmetic that decides it, and works a shelf-picking example end to end. The short version: a controller-hosted script suits a proof of concept with a handful of object types, while dozens of products or a tight cycle belong in a smart camera or a vision controller.

The decision, up front

Count your product types and your vision time budget first; everything else follows from those two numbers.

• Proof of concept, up to about five object types, 10 s cycle or looser: run the vision on the robot controller as a background script. You need a USB camera, a vision library and patience; the constraint is CPU speed, not capability.
• Production guidance, one to four camera positions, result inside 200 ms: a smart camera at each position, streaming poses to the robot. Nothing else in the loop.
• Several cameras, strobed lights, an image archive or a PLC in the conversation: a fanless vision controller in the cabinet. This is what no-PC means on a real line: no desktop on a cart, 24 V power, everything bolted in and fanless.
• Detection that has to be learned, not programmed: the model runs where an accelerator exists. Robotics-processor vendors size camera-rate detection at 2-8 TOPS, a budget that lives in a smart camera with on-device inference or in a vision controller, not in the robot controller.

If camera choice, gripping and calibration are still open questions, start with building a robot vision system for object location and gripping and come back here for the processing placement.

Where the image processing can run

On the robot controller

The teach-pendant script language is built for motion: moves, waits, I/O. It has no camera interface and no image operators worth the name. The workable route is a background process, a daemon, that the controller hosts: it opens the camera, runs the vision library, and registers functions the pendant program calls through shared variables or a local socket. The robot program asks for a pose; the daemon answers in millimetres and radians.

The limit is the processor. A robot controller's CPU is an embedded-class part doing real-time servo work and path planning, not a desktop core. Expect OpenCV feature or template matching to run 5-10× slower than on a workstation: a 1.6 MP frame that matches in 30 ms on a modern desktop core takes 150-300 ms on the controller, and every extra product template adds another full pass. Fine for a demo; fatal for a shelf of forty products.

In a smart camera

A smart camera carries sensor, lens, processor and I/O in one housing. You configure it once over the network from a laptop, then it runs alone: trigger in, decision out. Only the result leaves the housing — a pose in tens of bytes — so even a Fast Ethernet interface keeps up easily. The engine inside is its own decision; see AI vs rule-based smart camera inspection: how to choose.

The MV-SCB007EC-12S-WBN is a 1.6 MP color vision sensor with a 12 mm lens and manual focus. Two other self-contained options:

The LS-SMC-0063 delivers 60 fps with a 16 mm lens; the MV-SC5060XM-08M-WBN gives you 6 MP in mono with an 8 mm lens. Either one ends the where-does-it-run question: on the camera.

On a fanless vision controller

A vision controller is an industrial computer built for this job: camera ports, often with PoE so each camera needs one cable, built-in light channels with strobe timing, and a CPU that holds full-frame processing plus archiving. It runs the whole stack and hands poses to the robot exactly as a smart camera does. When the requirement is really no PC beside the cell, this is the production answer.

The numbers that decide it

The vision slice of a cycle has four parts, and only two of them differ much between the architectures:

Transfer. A 1.6 MP mono frame is about 1.6 MB, or roughly 13 Mbit. Fast Ethernet at about 80 Mbit/s effective moves it in roughly 160 ms; GigE in 25-40 ms; a USB 2.0-class port in about 50 ms. A pose of 50-100 bytes takes under 1 ms on any of them, which is why a smart camera on Fast Ethernet is still fast: it never ships the image.

Processing. On one modern desktop core, feature or template matching at 1.6 MP costs 20-60 ms; call it 30 ms to plan with. Robot-controller CPUs pay the 5-10× penalty, so 150-400 ms per frame per template. Twenty product types means twenty passes: 20 × 30 ms = 600 ms on a desktop-class core and 3-6 s on the controller. That arithmetic is the wall that ends the daemon route in production. A trained detector, by contrast, makes one pass over the frame no matter how many types it knows.

Take a 4 s robot cycle with a 500 ms vision budget. The daemon with one template fits at 200-450 ms; three templates risk 500-1,250 ms; twenty never fits. The smart camera sits near 100 ms in every case, and the vision controller lands within 200 ms.

Worked example: tidying a shelf

The scene: a 500 mm section of shelf, grocery items at random angles, robot paused while imaging, camera square to the shelf at a 1,200 mm working distance. Use the 12 mm vision sensor from the card above for this geometry.

Field of view. A 1/2.9-inch sensor format has an active width of about 5.0 mm. With a 12 mm lens at 1,200 mm:

Exactly the shelf section, with no wasted pixels at the edges.

Pixels. 1.6 MP in 4:3 is 1440 × 1080, so scene sampling is 500 / 1440 ≈ 0.35 mm per pixel and the sensor pitch is 5.0 / 1440 ≈ 3.5 µm. A 60 mm box front spans 60 / 0.35 ≈ 173 px, and a ±2 mm pick tolerance is ±6 px — above the 5 px per tolerance band you want before trusting a locate.

Exposure and f-number. The arm is stationary during capture, so blur is zero as long as the shutter closes inside the imaging pause; 15 ms is a realistic exposure under shelf lighting and fits inside a 16.7 ms frame period at 60 fps. Items sit within ±150 mm of the focus plane, so you need about 300 mm of depth of field. The thin-lens estimate:

With the circle of confusion set to one pixel, f/8 covers the shelf depth with margin; f/4 gives about 280 mm and is the wrong setting here. This is a thin-lens estimate, so start at f/8 and confirm on the first live images.

Cycle check. Smart camera: 15 ms exposure + 60 ms on-device locate + 2 ms pose over TCP ≈ 77 ms of vision, and the arm dominates the cycle. Daemon with the store's twenty product types: 20 passes × 150-300 ms = 3-6 s of CPU before the arm even moves. The same daemon with three demo objects, 450-900 ms, is livable inside a 10 s demo cycle. That contrast is the whole decision.

For the vision-controller pairing, area scan cameras with C-mount optics are the natural partners; a 2.5 MP GigE camera with global shutter and C-mount, such as the MV-CU020-90GC, is the class to shortlist in the camera finder.

Smart camera or vision controller

Both keep processing off the robot and out of a desktop. Pick by what the cell needs:

• One to four inspection points, a fixed algorithm, minimal hardware: a smart camera per position. One cable each, poses over TCP, nothing to administer.
• More cameras, strobe timing on lights, an image archive, a model to retrain, or a PLC wired into the cell: the vision controller; machine vision PLC integration on the production line covers that side.

All 2 share: Humidity 20% RH to 95% RH (no condensation); Certification CE, KC. Values as printed in each product's data sheet; open a part for its drawing and full specifications.

The MV-VC3301H-128G66 pairs an Intel Core i3 with 8 GB of memory, four built-in light channels and 5+ LAN ports — single-cell territory. The MV-VC5525-256G70-NN steps up to a 12th-generation i5 with 32 GB of memory, PoE camera ports and eight light channels for a multi-camera cell. For the guidance math once the hardware lands, 2D vision pick and place: selecting a planar guidance system works the pixel-to-millimetre conversion and approach angles.

Handing the pose to the robot

All three paths end the same way: the vision side returns millimetres and radians in camera coordinates, and the robot transforms them into base coordinates through a calibrated fixed relationship. Robot hand-eye calibration: how nine-point alignment works is the procedure; on a sensor with no calibration wizard, smart camera hand-eye calibration when no wizard exists walks the same points by hand. The transport is one short message per result — send camera part coordinates to a robot controller over TCP gives a protocol small enough to debug with a terminal. For grasp quality, plan two locates rather than one generic pick: find the item, then find the grasp feature on it.

Checks before you commit

• Count product types now, then add the ones the customer will request next quarter. Templates multiply daemon time linearly.
• Time one real match on the target hardware — one frame, one template, your lighting — before promising a cycle number.
• Set exposure and gain manually in the daemon; auto-exposure callbacks inside a background process are a common source of dropped frames.
• Run the field-of-view and depth-of-field arithmetic at your real working distance; the calculator above does the lens step in seconds.
• Name the calibration owner and the trigger for redoing it: camera bumped, tool changed, shelf moved.
• Write the no-read and timeout behaviour into the robot program, not into the vision side.
• With a cabinet controller in play, budget 24 V power and mounting space now rather than at commissioning.

When the usual answer is wrong

• The daemon gets dismissed too early. For a funded demo with three objects it is the fastest route to a working cell, and everything you learn — camera control, pose handoff, calibration — transfers to the production hardware unchanged.
• A Fast Ethernet camera is not a slow camera. Archiving full frames caps near 6 fps at 1.6 MP, but results-only traffic is tens of bytes. Choose GigE when you must archive every frame; otherwise the interface does not matter.
• A moving conveyor changes the winner. Encoder tracking and strobe timing push you to a vision controller with light channels; the same question one cell over is whether a delta pick-and-place cell needs conveyor vision.
• Training never happens on the edge. Collect and label on the line, train off-line, deploy the model to the camera or controller. Robot controllers neither train nor retrain.
• A cycle under 300 ms beats both easy answers. Strobed light, global shutter and a sub-100 ms result are a vision-controller design; when processing lags acquisition, acquisition faster than processing: fix the frame rate gap shows the fixes.

When the geometry and cycle are settled, run the system configurator from the camera step, and skim the systems topic page for the integration pieces around this one. If the camera count or the cycle number still does not close, tell us the cell layout and we will size it with you.