Smart camera socket handoff to a robot: straight-down pick

A vision pick cell damages itself at one repeatable moment: the robot draws a straight line from the picture position to the part, and the tool runs through a fixture on the way. The fix is architectural, not a waypoint tweak. Let the smart camera own exposure, location and calibration, and publish each part's pose on a TCP socket; let the robot script build its own motion — traverse at clearance height, stop directly above the part, descend straight down. This guide gives the message format, the base-frame calibration, a worked field-of-view calculation for a 400 × 300 mm tray, the clearance and speed rules for the descent, and the parts that fit a single-camera cell.

Why the picture-to-pick straight line fails

Bundled vision-and-motion instructions interpolate linearly between two taught points: where the robot stood when the picture was taken, and the pick offset. Every fixture between those two points sits on that line. Add a gripper that must approach vertically and the path is wrong twice over — a slanted linear move cannot deliver a straight-down grip, and the shortest line to the part is exactly the line through the tray wall, the clamp or the part-present sensor.

The remedy is to never command motion that starts at the picture position. The robot's first approach after a find is a point in the air, directly above the found part, high enough to clear everything measured in the cell. The horizontal travel happens at that height; the vertical travel happens alone.

Split the job: the camera finds, the robot moves

The camera side runs the complete vision job: trigger, exposure, locate the part, transform the pixel result into robot-base millimetres, publish one pose line per part. Once the job is downloaded, the camera runs standalone — set the vision sensor to auto load its job on startup so a power cycle needs no PC and no operator.

The robot side is a thin script: open a client socket, request a pose, parse one line, build waypoints, grip, report a result. That is the same division the 2D vision guided robot pick and place cycle describes, minus the external computer — the whole cell runs robot pick and place vision without an external PC.

The pose message over the socket

Send five fields and nothing else: a sequence number, X, Y, the in-plane rotation, and a status. X and Y are floats in millimetres in the robot base frame, rotation is a float in degrees about the camera axis, and the sequence is an integer that increments per request. The field names below are an example protocol you define yourself — keep every field fixed-width, terminate with a newline, and echo the sequence number so the script can reject a stale reply:

robot  -> camera:  TRG,1041\n
camera -> robot:   POS,1041,231.42,88.07,-15.2\n

The test that proves the parse: teach a part at a corner with known base-frame coordinates, request a pose, and check the printed X and Y against the taught value to within 0.2 mm. Then slide the part 10 mm along one axis and confirm the reported coordinate moves 10 mm on that axis and nowhere else. The message shape is covered in detail by sending camera part coordinates to a robot controller over TCP, and sending string data from a vision system to a robot over TCP shows the string handling on the robot side.

Put the pose in the robot's base frame

With a fixed overhead camera — the eye-to-hand arrangement — the transform is a fit, not a guess. Jog the robot tip to four or more points spread across the field, record the base-frame X-Y at each and the pixel position of the same tip in the image, then fit the mapping. A rigid mount and a square-down camera typically fit to a residual of 0.1-0.2 mm across the field, and that residual is your floor for pick accuracy. The eye to hand object pose transform for robot picks walks the math; the robot-held camera calibration with a TCP tool frame covers the eye-in-hand variant when the camera rides the arm.

Mount the camera normal to the pick plane. A tilted camera turns the in-plane rotation into a scale-and-axis problem the robot script then has to undo; square-down keeps "degrees about the camera axis" equal to degrees about tool Z. The Z of the pick stays yours: teach it once per part type, since the socket carries planar pose only.

Worked example: one 6 MP camera over a 400 × 300 mm tray

Inputs: the fixture measures 400 × 300 mm, and a 20 mm border on every side gives the locate job search margin, so the view must cover 440 × 340 mm. The camera is the 6 MP, 1/2.4" model with an 8 mm M12 lens shown in the parts section below; assume its 3:2 active area is about 5.5 × 3.7 mm, which at 6 MP is about 3072 × 2048 pixels. The working distance WD is the unknown.

The height axis limits first: 340 mm × 8 / 3.7 = 735 mm, so set the stand-off to 750 mm. At 750 mm the view is 750 × 3.7 / 8 = 347 mm tall and 750 × 5.5 / 8 = 516 mm wide — both clear the 440 × 340 mm requirement. The image scale is 516 mm / 3072 px = 0.17 mm per pixel, and three pixels, the usual minimum for a confident fit, span 0.51 mm.

Decision: this single 6 MP shot carries a ±1 mm pick tolerance with margin, since the tolerance window is four three-pixel spans wide. A ±0.5 mm tolerance sits exactly at the three-pixel limit — shorten the stand-off over a smaller window or step up the resolution rather than accept it. Below ±0.3 mm, stop tuning this camera; the tolerance needs a different sensor, not a better threshold.

Build the above-and-down motion

The robot script turns one pose into five moves:

1. From home, joint-move to a fixed safe point at clearance height on the approach side. Joint motion on this leg never aims at the part, so a wrong pose cannot drag the tool into the fixture while travelling in.
2. Linear-move horizontally at clearance height to the X, Y from the socket, tool Z vertical. Z never changes on this leg.
3. Linear-move straight down to the taught pick Z at reduced speed — set the descent at about a quarter of the traverse speed. Grip.
4. Verify the grip, then linear straight up to clearance height before any lateral motion.
5. Linear at clearance height back over the approach side, then joint-move away to place.

Set clearance Z at the tallest measured obstacle between the approach side and the tray, plus 50 mm, measured with the gripper at its widest. That single number is what buys the architecture: every horizontal millimetre of travel happens 50 mm above everything, so no pose, right or wrong, can draw a line through the tooling. Keep the tool orientation fixed through legs 2 to 4 — for round parts that cannot take a rotated grip, keeping tool orientation fixed at every vision pick shows how to hold it.

Cycle budget, timeouts and retries

Against a 1-2 s robot move the vision step is noise, so spend the engineering on the handshake instead. Set the receive timeout at 300 ms, retry the request twice, and fault the cell on the third failure — a blind move is the only unrecoverable outcome. Check sequence continuity on every reply; a gap means a lost line, not a bad part. When a reply never comes at all, the socket no-response send-loop fix lists the causes in order, and the TCP ports and firewall rules between a vision controller and a PLC covers the connection that refuses before it ever times out.

Cameras and controllers for the cell

For a compact cell at short stand-off, the LS-SMC-0063 carries its own illumination: a 1.6 MP GigE smart camera with a 16 mm lens running 60 fps and an integrated white light, so the lighting hardware disappears into the camera. The worked example assumed the bigger sensor and narrower field: the MV-SC5060XM-08M-WBN is that camera, 6 MP mono with an 8 mm M12 lens on GigE. Color parts or a global-shutter requirement move you to the other two cards below.

One camera with a locate job fits the camera alone. Move the job to a vision controller when it grows an AI classification stage, a second camera, or switched light channels for several fixtures. The MV-VC3301H-128G66 is the entry point there: an Intel Core i3 processor, 8 GB of memory, five-plus LAN ports, four built-in light channels, PCIe expansion and a VM license included. The standalone machine vision controller selection guide sizes the bigger units, and the range sits in the vision controllers and smart cameras categories, with the camera finder to filter by resolution and mount.

Checks before the first live pick

• Corner test: request a pose on a part at a taught corner; the reported X-Y must match the taught coordinates within 0.2 mm. This proves calibration and parser in one shot.
• Slide test: move the part 10 mm on one axis; the pose must move 10 mm on that axis only. This catches swapped axes, the classic wired-backwards transform.
• Empty-gripper dry run: cycle legs 2 to 4 twenty times over live parts at reduced speed with nothing in the gripper; zero contact proves the clearance number.
• Dead-link test: block the camera's port once and confirm the cell faults at the 300 ms timeout instead of moving blind.
• Soak: twenty consecutive live picks with continuous sequence numbers; a gap localizes a dropped line before it localizes a dropped part.

When this architecture is not enough

A planar pose assumes the part presents a known face at a known height. Piled or randomly oriented parts, or a pick that must know Z before it grips, need depth: 3D bin picking for flat parts starts that conversation, and 3D versus 2D machine vision settles when the extra sensor pays for itself. A moving belt breaks the stationary-at-trigger assumption instead, which conveyor tracking pick hesitation checks covers. Everything on a static fixture — which is most picking — is the socket handoff above; the systems topic page collects the rest of the robot-vision guides, and if your tray and stand-off are unusual, send the dimensions through the contact page and we will size the field with you.