Send camera part coordinates to a robot controller over TCP

A camera that finds the part is only half of a picking cell: the robot still needs the position, in its own units, at the moment the part is actually there. The clean, vendor-independent way to move that data is a TCP socket between the vision controller and the robot controller — calibrated X and Y in millimetres, rotation in degrees, and a one-byte acknowledgement per pick. This guide builds that link end to end: the parts, the calibration that turns pixels into robot coordinates, the message format, the code structure, and the pass that proves the loop before the first live part.

The decision: one socket, the robot as client

Commit to this architecture: the vision application runs on its own vision controller as a TCP server on a fixed port — 5000 is a common choice — and the robot controller connects to it as a client once at cell start-up. Every exchange begins with the robot asking; the vision side never sends unsolicited coordinates. One request, one answer, one acknowledgement.

Choose TCP over UDP. A pick needs every trigger answered exactly once and in order, and TCP guarantees both; on a dedicated cell LAN the round trip costs about 1 ms against a 100 ms answer budget. UDP earns its keep only for continuous streams, such as following a moving belt, where a lost sample is replaced by the next one.

Running the vision software on the robot controller's own Windows partition is possible but is the weaker plan: a camera driver update or a vision crash then shares a platform with motion control, and the support boundary blurs. Keep the detection on a separate computer — the same pattern as connecting a vision camera to a PC without the robot controller — and let the socket be the whole interface. If you are still choosing the wider architecture, read 2D vision pick and place: selecting a planar guidance system first.

The parts that do the job

Three items carry this cell: a fixed-mount area scan camera, a vision controller with spare LAN ports, and a rigid camera stand with a slide table. For a stationary tray, resolution matters more than frame rate, because you take exactly one image per trigger.

Two cameras cover most fixed-mount cells:

The LS-CMR-0053 is a 12 MP GigE camera on an IMX226 1/1.7" rolling-shutter sensor at 9 fps, which is plenty when one part waits for one trigger. The LS-CMR-0166 is the same 12 MP IMX226 sensor over USB3 at 30 fps, 4000 x 3000 px, for cells that also watch a slowly moving tray.

Detection and the socket server run on the vision controller:

The MV-VC3301H-128G66 has an Intel Core i3, 8 GB of memory, 5+ LAN ports, 3-4 USB 3 ports and 4 built-in light channels, so one box carries camera link, strobe power and the robot connection. Bringing a GigE camera onto such a controller step by step is covered in connecting a GenICam camera to an industrial controller over Ethernet.

Hold the camera square over the tray on a slide-table stand:

The LS-MNT-0040 is a professional slide-table type with a 700 mm column on a 400 x 450 mm base. The column height is your working-distance ceiling, and that ceiling drives the lens choice in the worked example below. If your tray is a different size, scale the example through the area scan cameras finder.

The message: trigger, coordinates, acknowledge

Use ASCII rather than binary for a first integration: you can read the traffic in a terminal, and a wrong byte shows itself immediately. Terminate every message with a newline, keep one message per line, and fix the field order once, in writing.

Two decimals are enough: 0.01 mm is a tenth of a pixel at the scale in the worked example. The vision side must hold the last answer until A arrives and then discard it — resending the same coordinates is how a robot picks an empty spot. Design the restart path on day one; vision picking restart handshake: images arrive but picks stop walks through that failure mode. The robot waits at most 2 s for C, re-triggers once, then alarms and stops asking.

Build it in nine steps

1. Mount the camera looking straight down, square to the tray within about a degree; fit the lens, set f/5.6, focus on the tray plane and lock both rings.
2. Wire camera and light to the vision controller. Fix exposure and gain, and switch auto exposure off: a drifting exposure shifts contrast between triggers.
3. Jog the robot's pointed tool to three or more non-collinear points spread across the tray, and at each record the robot's X and Y plus the pixel column and row of the tip.
4. Fit the six affine coefficients by least squares and keep the residual under 0.5 pixel; camera calibration residual values: average and max targets sets the acceptance numbers.
5. Program detection: segment the part against the tray, compute its centre to sub-pixel, and read rotation from two datum features at least a third of the part length apart — robot vision for locating part center points covers the image side.
6. Map centroid and angle through the transform, subtract the taught fixture angle, and reject any answer that lands outside the tray.
7. Open the TCP server on port 5000, accept exactly one connection — the robot's address — and parse T.
8. On trigger, grab one frame, run detection, and reply C with three numbers or E with a code, inside a 100 ms budget.
9. Log every exchange with a timestamp: trigger, pixel values, robot-frame values. That file is what you will read when a pick misses.

The calibration behind step 4 is plain linear algebra:

Here (u, v) are pixel coordinates and (X, Y) are robot-frame millimetres. The fit absorbs scale, rotation, skew and offset in one step, so never scale pixels from the lens formula — the measured scale is the truth, the computed one only picks the lens. At a magnification near 0.02, lens distortion is small enough that the affine fit is the right model.

Worked example: a 400 x 300 mm tray under a 12 MP camera

Inputs: a tray 400 x 300 mm sitting inside the stand's 400 x 450 mm base; a 12 MP camera delivering 4000 x 3000 px on a 1/1.7 in sensor, about 7.5 x 5.6 mm; a column that puts the lens roughly 695 mm above the tray; a bracket with two datum holes 40 mm apart; a pick tolerance of ±0.5 mm and ±0.5°.

Two rules drive the numbers: pixel scale is field of view divided by pixels, and focal length follows from magnification and working distance.

Design the field of view with margin: 430 x 320 mm, so a tray corner is never cropped by mounting error. Magnification is m = 7.5 / 430 = 0.017, and focal length is f = 695 x 0.017 / 1.017 = 11.6 mm — the nearest standard lens is 12 mm.

With the 12 mm lens fitted, m = f / (WD - f) = 12 / 683 = 0.0176, so the field of view is 7.5 / 0.0176 = 427 mm wide and 5.6 / 0.0176 = 319 mm tall. The tray fits with 27 mm and 19 mm to spare.

Scale: 427 mm across 4000 px is 0.107 mm per pixel. Cross-check on the sensor side: 1.85 µm pixels at m = 0.0176 give 1.85 / 0.0176 = 0.105 mm per pixel. Two routes, the same number.

Position accuracy: a 0.5 px calibration residual is 0.05 mm, a 0.1 px centroid is 0.01 mm, and a robot repeating to 0.05 mm adds the rest. Root-sum-square: sqrt(0.05 x 0.05 + 0.01 x 0.01 + 0.05 x 0.05) = 0.07 mm against a ±0.5 mm tolerance — a sevenfold margin.

Rotation: the holes sit 40 / 0.107 = 373 px apart. Centroid noise of 0.2 px at each end gives angle noise of atan(0.2 x 1.41 / 373) = 0.04°, well inside ±0.5°.

Light and optics: a 5 ms strobed exposure keeps factory lighting a minor contributor, so contrast cannot drift between triggers. At f/5.6 the 12 mm lens has a 2.1 mm aperture (12 / 5.6); a part lying 10 mm high blurs by 10 x 2.1 / 695 = 0.03 mm, under a third of a pixel. Depth of field is not the limit at this magnification — f/5.6 buys lens contrast, not depth.

Decision: 12 mm C-mount lens, camera at about 695 mm, measured scale near 0.107 mm per pixel, f/5.6, 5 ms strobe. This layout stays comfortable down to ±0.2 mm of gripper clearance; below that, halve the field of view or double the camera pixels before changing anything else.

Verify the loop before production

• Run 20 picks spread over the tray and log the offset between commanded and landed position. Accept a mean under 0.1 mm and a worst case under 0.3 mm for a ±0.5 mm tolerance: a one-sided mean says the calibration is stale, a wide scatter says the mount or the detection moves.
• Pull the Ethernet cable mid-cycle: the robot must hit its 2 s timeout, stop triggering and alarm, never reuse the last coordinates.
• Power-cycle both controllers: the robot reconnects, the server accepts exactly one client, and one dummy trigger round-trips before the first real part.
• After any knock to the camera or stand, re-teach a single reference point; robot cell vision recalibration with one reference target gives the one-target procedure.

When this design is the wrong one

Coordinates age at belt speed. At 150 mm/s, a 100 ms vision answer is 15 mm of travel by the time the robot reads it, so a stationary camera over a running belt needs an encoder latch, and a camera fixed to the conveyor frame is often the better mount — conveyor tracking camera placement relative to the robot and conveyor vision scene timing: troubleshoot stale pick data cover that timing.

An overhead camera measures one plane. Parts at different heights, tilted parts and overlapping parts all lie about X and Y, and no calibration fixes that; robot vision requires depth to resolve camera coordinates is the next read when Z is not fixed. And if picks must land inside ±0.1 mm, resolution is the wall, not the protocol: shrink the field of view or add pixels.

Checklist before you order

• Camera: the smallest feature you must hold on target spans at least 5 px at the computed scale.
• Lens: covers the sensor format at your working distance, with 10 % field-of-view margin; C-mount for a 1/1.7 in sensor.
• Vision controller: one LAN port per GigE camera plus one for the robot network, fixed IP addresses, camera traffic kept off the office LAN.
• Light: diffuse geometry for shiny blanks, strobed from the controller's light channels on the same trigger as the camera.
• Stand: a column taller than the working distance your lens needs, with a slide table to trim height without redrilling.
• Cabling: one Ethernet run camera to controller, one controller to robot, both routed away from drive wiring.

When the tray size is settled, size the camera in the camera step of the configurator, or send the layout to our applications desk and we will check the numbers before you order.