Building a robot vision system for object location and gripping

You have a robot arm, a module camera and a PLC or microcontroller, and the goal is direct: the system finds a part and hands the robot its grip position. For parts that lie flat on a tray or belt, the whole machine is one fixed camera, one calibration and one handshake message. This guide walks the build end to end — choosing between a vision sensor and a module camera, sizing the optics to your grip tolerance, calibrating pixels to robot base coordinates, and passing X, Y and theta over Modbus TCP or a serial line. By the end you can order the parts and write the integration with numbers instead of guesses.

Where the detection runs decides the build

Three builds work on a bench, and the first decision is who computes the part position.

Commit to build A when one part family sits in a fixed tray and you want no image code of your own. The MV-SCB007EC-12S-WBN is a 1.6 MP colour vision sensor with a 12 mm M12-mount lens, manual focus and a Fast Ethernet interface. Commit to build B when parts change, when you want to detect on colour or texture yourself, or when the project itself is the algorithm: a module camera feeds a host you program. The MV-CB016-10GM-M-W is a 1.6 MP GigE board-level camera on the IMX296 sensor, 1/2.9" format, global shutter, M12 mount. Build C earns its keep at large fields of view or tolerances under ±0.3 mm. The MV-CS200-10GC puts 20 MP on a 1" IMX183 sensor behind a C-mount, over GigE.

A microcontroller on this bench is a bridge, not a host: it triggers, it reads the answer, and it cannot ingest the pixel stream — the arithmetic sits in the hand-off section below. The board-level cameras finder lists modules for build B; the vision sensors category covers build A. For the system-level view first, read 2D vision pick and place: selecting a planar guidance system.

The shortest route from box to picking is build A:

Match camera pixels to the grip tolerance

Work backwards from the gripper. A parallel-jaw gripper with 1 mm of jaw clearance around the part closes cleanly when the camera reports the centre to ±1 mm anywhere in the grip area, not just near the middle. Size the pixel scale to a third of that tolerance:

For a ±1 mm system that means 0.33 mm per pixel; design at 0.15 mm per pixel and you keep margin for lighting and calibration. A 300 mm wide grip area at 0.15 mm per pixel needs 300 / 0.15 = 2,000 pixels across the sensor. A 3.1 MP sensor resolves about 2,048 × 1,536, so it covers 300 mm × 225 mm at 0.146 mm per pixel — 6.8 pixels to the millimetre. The LS-BCM-0003 is a 3.1 MP USB3 board-level camera: 1/1.8" sensor, IMX252, global shutter, C-mount, 120 fps.

Looser tolerance buys field of view: a vacuum cup forgiving ±3 mm works with a 1.6 MP camera at 0.21 mm per pixel over the same 300 mm field, and the field can stretch toward 450 mm before the scale passes 0.33 mm per pixel. Tighter than ±0.3 mm costs pixels: 0.05 mm per pixel across 300 mm needs about 6,000 columns, which is 20 MP territory. The same arithmetic from the inspection side is covered in configuring camera locate resolution for object location, and choosing the module itself is its own subject in board level camera selection for embedded vision machines.

A module that carries build B:

Size the lens and the working distance

With sensor and field fixed, the lens follows. A 1/1.8" sensor measures about 7.2 mm × 5.4 mm, and the field of view is 300 mm × 225 mm:

The magnification is m = 7.2 / 300 = 0.024. Rearranged for a 12 mm lens, the working distance is WD = f × (1 + m) / m = 12 × 1.024 / 0.024 = 512 mm. So: a 12 mm lens, camera 510 mm above the tray, and a 300 mm × 225 mm field that matches the sensor's 4:3 aspect.

Angular accuracy follows from the same scale. A 60 mm long part whose measured endpoints wobble by one pixel points to atan(0.146 / 60) = 0.14° of angle error, well inside the ±1° a parallel-jaw grip tolerates.

The calculator sizes focal length, working distance and field together, so you can test a 16 mm lens at a longer stand-off in one pass:

Mount the camera once, then stop touching it

For a first build, mount fixed and overhead, looking straight down — the eye-to-hand arrangement. Bolt the camera to a rigid column rather than a cantilevered arm, and keep it outside the robot's envelope or above its reach. Every nudge of the lens or bracket invalidates the pixel-to-robot transform, so mark the mount positions and re-run calibration whenever anything moves.

A wrist-mounted camera works closer and halves the pixel scale when you need it; eye in hand calibration pose selection: robot positions that work covers the poses that make that calibration converge. For the fixed arrangement the transform is one static matrix: publish eye-to-hand calibration as a static camera transform shows how to store it, and use eye-to-hand calibration results to compute robot-base picks together with eye to hand object pose transform for robot picks carry the math through to the pick.

Calibrate pixels to robot base coordinates

Two ways exist to get the transform, and one wins. The printed-target way tapes a 12 × 9 checkerboard of 20 mm squares — 220 mm × 160 mm of grid — into the tray and solves from detected corners; print distortion and paper flatness end up inside the answer. The robot-taught way puts a pin in the gripper, jogs it to known base coordinates and records the pixel at each, so the target is as accurate as the robot itself. Commit to the second.

The fit is affine, six numbers:

1. Fit a pin or pointer in the gripper and set it as the TCP.
2. Jog to nine points: a 3 × 3 grid on 100 mm spacing, centred in the grip area, all inside the field of view.
3. Record the pixel coordinate of the pin tip at each point; a dark dot on a light tray gives a clean centroid.
4. Solve the affine fit by least squares. Nine points give 18 equations for 6 unknowns, and the 12 spare equations average out jog and centroid noise.
5. Read the residuals: root-mean-square should land near half a pixel, about 0.07 mm at this scale. One point above one pixel is a bad teach — redo it, do not average it away.
6. Write the six coefficients where the pick program can use them.

Budget the accuracy honestly: 0.1 mm from calibration, 0.1 mm robot repeatability on a small collaborative arm, 0.3 mm from part-edge contrast, giving ±1 mm across a 300 mm field. That is the number to quote in the report.

The grid math appears again, for the wrist-mounted case, in robot hand-eye calibration: how nine-point alignment works. The printed-target route and its pitfalls are covered in checkerboard camera calibration error: fix the grid pitch, and locate image defects with pixel coordinates and calibration shows the same pixel-to-world conversion in an inspection setting.

Turn the detection into X, Y and theta

Detect the part as a region and take its sub-pixel centroid for X and Y; a centroid over thousands of pixels is stable to a tenth of a pixel, better than any single edge. Take theta from the region's principal axis, or fit a line to the long edge when the outline is interrupted. A vacuum cup tolerates several degrees of theta; a parallel-jaw grip does not, so report theta in 0.01° units and track it through the cycle. Gripper rotation between pick and place: find the full turn catches the 360° mistakes, and vacuum gripper signal mapping for vision-guided picking covers the IO side of the cup.

Hand the answer to the robot

Builds A and B end in the same contract: the PLC or microcontroller triggers, vision answers with one frame's coordinates. Modbus TCP is the common denominator, and a register set both sides can hold to looks like this:

The handshake: the PLC writes 1 to 40001, vision latches the next frame, status goes to 2 and the counter increments; the PLC then reads the values and clears the trigger. A counter that has not moved means stale data — read again. Software trigger image timeout: why one-shot capture misses frames explains why the first trigger after idle is the one that gets lost, and vision controller Modbus communication: a PLC integration guide wires the handshake into a PLC scan.

Over a serial line, one ASCII frame does the same job: X=152.31;Y=88.40;T=-12.5;F=1042 plus a newline is 32 characters, about 3 ms at 115200 baud — instant next to a robot move.

The arithmetic that rules a microcontroller out of the camera path: 3 megapixels at 2 bytes per pixel and 30 frames per second is 3,000,000 × 2 × 30 = 180 MB/s of pixel data. A microcontroller forwards kilobytes. Feed the camera to a PC or single-board computer — any GigE Vision camera works with standard libraries, as camera compatibility with vision software: what GigE Vision guarantees sets out — and keep the microcontroller on trigger-and-reply duty. On a cell with no room for a host, build A removes the computer entirely.

The module for build B when the host is a small computer:

Prove it with a 30-pick test

Before production, measure. Put one marked part at nine positions across the tray, command the pick, and hold the robot's pin over a fixed reference hole or a dial indicator; log the miss each time. Thirty picks all within ±1.5 mm, none past 2.5 mm, passes the system. A worse spread traces to a moved bracket or a lighting change, and the fix is a re-teach rather than more software. Re-run the test after anyone touches the camera, and keep one golden image per part type to catch slow lighting drift.

When one flat camera is the wrong tool

Parts that stack, overlap or arrive in random piles need range, not pixels; 3D bin picking for flat parts: choosing a vision approach marks that line. A moving conveyor needs trigger and strobe: blur equals speed × exposure, so 200 mm/s at 1 ms smears 0.2 mm, more than one pixel — hold exposure under 0.5 ms or the theta you report drifts. Shiny and translucent parts need the right wavelength before a better sensor; start with which LED colour gives the best contrast in machine vision.

Tall parts break the flat-world assumption: a camera 510 mm up sees a 30 mm tall feature 140 mm off axis displaced by 30 × 140 / 510 = 8 mm. Keep the grip area flat, pick on the top surface, or shrink the field. Below ±0.3 mm tolerance, buy pixels: the area scan cameras finder filters by resolution, and the 20 MP camera above is the build C step when the 300 mm field must stay.

Checklist before you order

• Vision tolerance stated in millimetres, pixel scale at a third of it — 0.15 mm per pixel for ±1 mm.
• Camera columns at least FOV width / (tolerance / 3): 2,048 columns for 300 mm at ±1 mm.
• Lens and working distance solved together — 12 mm at 510 mm for a 300 mm field on 1/1.8".
• Global shutter the moment anything moves during exposure.
• Rigid overhead mount, positions marked, outside the robot envelope.
• Controlled light: a diffuse panel, not the ceiling.
• Calibration method fixed: robot-taught nine-point grid.
• Hand-off contract fixed: Modbus registers or a serial string, plus a frame counter.
• Reduced-speed teach and a guarded run mode.

Start at the camera step and carry the numbers above through it:

If a number does not close — tolerance, field, working distance — bring the sheet to the contact page and we will size it with you.