Connect a camera to a robot with Python: sockets and calibration

You have a camera that finds the part and a robot that must reach it; the missing piece is the Python in between. The answer up front: solve a 4×4 camera-to-robot matrix from nine or more point pairs. Convert pixels to millimetres with the lens scale — a 25 mm lens at a 300 mm working distance on a 2/3 in, 2448-pixel sensor gives 0.038 mm per pixel. Then send each result as a short ASCII line, 7,437.60,110.10, over a TCP socket with a 500 ms timeout and a matching ACK,7 echo before the robot moves. This guide builds each piece with the arithmetic shown, so you can size your own cell as you read.

How does a camera tell a robot where the part is?

Two conversions move the point: pixels become millimetres through the lens scale, and the 4×4 camera-to-robot matrix turns those millimetres into base-frame coordinates with p_base = R * p_cam + t.

Keep the two conversions in separate functions. One turns a pixel into a camera-frame millimetre position; the other applies the matrix from calibration, exactly as in the eye-to-hand object pose transform for robot picks.

When the camera watches a plane — a conveyor top, a fixture nest — Z comes from the tool or a fixed drop height, and the 2D chain above is the whole job. When the part has height or tilt, replace the constant scale with a depth value from a 3D sensor and carry p_cam as a three-component point.

What hardware does the Python link need?

Three devices carry the link: a camera on GigE or USB3, a vision controller that runs the capture and the maths beside the camera, and the robot controller's socket-messaging port on the same network switch.

Put the capture and the maths on a controller in the cell rather than on an office PC across the plant. Handshake timeouts live in the millisecond range, and a shared office network adds jitter you cannot debug from a desk.

The MV-VC3301H-128G66 runs an Intel Core i3 with 8 GB of memory and 128 GB of SSD storage, offers five or more LAN ports and four built-in light channels, and includes a VM6200Pro licence. The MV-VC2140-128G40-1T is the compact variant: N-series processor, 8 GB of memory, 128 GB SSD plus 1 TB HDD, three to four LAN ports and two USB 3 ports. On either box, one port runs the camera and another runs the robot link.

For the camera, a compact global-shutter GigE part is the usual guidance profile:

The MV-CU020-90GC is a 2.5 MP colour GigE camera with a global shutter; the wider range of area scan cameras covers mono sensors and higher resolutions when the part demands it. Sensor format and pixel pitch are on the card above — the pitch sets the millimetre-per-pixel scale in the next section. Size the full chain in the build-a-system configurator.

How do you solve the camera-to-robot transform?

Capture one known point at nine or more robot positions, pair every image point with its base-frame coordinate, and least-squares the 4×4 matrix — nine points average the noise and expose a bad one that four would hide.

The procedure takes an afternoon at the bench:

1. Park the camera (or the robot, if the camera rides the arm) in its production pose; nothing moves from here on but the target.
2. Capture nine or more images of one sharp target spread over the pick area — a plate dot, a tool-point tip, a marked nest corner.
3. Record the matching robot-base coordinate for each capture, from the taught plate position or the touched-off point.
4. Solve for R and t with the hand-eye solver in your robot or vision library; closed-form and iterative methods agree when the points are clean.
5. Re-project every point through the solved matrix and read the residuals: a few tenths of a millimetre is normal, and one point far above the rest marks a bad pair to recapture.

Robot hand-eye calibration: how nine-point alignment works walks the grid in detail, and camera to robot transform calibration: stop measuring by hand retires the tape measure. If you manage the solve in an external tool, the robot calibration tables configured from Python cover the table side. A camera fixed to the frame yields one static matrix, published once as in publishing eye-to-hand calibration as a static camera transform; a camera on the arm yields a hand-eye matrix you multiply with the robot pose at every capture.

How do you convert pixels to millimetres?

Divide the field of view by the pixels across it: a 25 mm lens at a 300 mm working distance on a 5 MP, 2/3 in sensor gives 93 mm over 2448 pixels, or 0.038 mm per pixel.

Worked example, arithmetic shown. Sensor: 2448 × 2048 pixels at 3.45 µm, so 2448 × 3.45 µm = 8.45 mm across. Lens: 25 mm focal length at a 300 mm working distance.

• Magnification: m = 25 / (300 - 25) = 25 / 275 = 0.0909
• Field of view: 8.45 mm / 0.0909 = 93 mm
• Scale: 93 mm / 2448 px = 0.038 mm per pixel
• Inverse, for sizing search boxes: a 1 mm part shift is 1 / 0.038 ≈ 26 pixels

The relation m = f / (WD - f) is the thin-lens estimate for a conventional lens; a telecentric lens prints its fixed magnification instead, and you skip this step. Substitute your own focal length, working distance and pixel count:

The scale carries straight into the matrix step. Say the solve returned R as a 180° camera roll about Z — it flips both axes — and t = (450.0, 80.0) mm. A part at (12.4, -30.1) mm in the camera frame, which is (326, -792) pixels, maps to p_base = (450.0 - 12.4, 80.0 + 30.1) = (437.6, 110.1) mm. Check the signs first in any integration; a flipped axis is the most common first-run error.

What goes in the socket message?

Send a short ASCII line — sequence number, X, Y, then Z and angle if the pick needs them — ended by the terminator the controller's messaging manual specifies, and wait for the controller to echo the sequence number back.

Five steps make the handshake dependable:

1. Open one persistent client connection at program start and set a 500 ms timeout, so a silent controller never stalls the cell.
2. Build the line with the sequence number first and fixed-point values: 7,437.60,110.10 plus the newline.
3. Send, then block on the receive until the echo arrives — ACK,7 matches; anything else or a timeout means hold the part data, re-capture or retry.
4. Log every sent value, echo and cycle time; that log is your evidence when a pick misses.
5. Keep the vision side stateless so a restart resumes cleanly — the sequence number is the only shared state.

import socket

HOST = "192.168.1.50"   # illustrative controller address on the cell network
PORT = 2000             # the messaging port configured in the controller
seq = 7                 # int: increments with every capture
x, y = 437.60, 110.10   # float: base-frame millimetres from the matrix step

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.settimeout(0.5)    # 500 ms
sock.sendall(f"{seq},{x:.2f},{y:.2f}\n".encode("ascii"))
ack = sock.recv(64).decode("ascii").strip()   # expect ACK,7
assert ack == f"ACK,{seq}", f"handshake failed: {ack!r}"

Against a listener that echoes ACK,7 the assert passes; against a silent port it raises within 500 ms, which is the test that proves the timeout. The payload layout in full — field order, scaling, error strings — is in sending camera part coordinates to a robot controller over TCP. When the controller wants the image itself first, the worked image-transfer example from a robot controller into Python over TCP shows the receive side.

How do you test the first calibrated move safely?

Run the first calibrated pick at the controller's reduced-speed manual setting with the safeguarded space active, and give each handshake the 500 ms timeout, because a wrong matrix moves the tool along a path no programmer has checked.

Then add three guards in code. Reject any capture whose target sits outside the calibrated area, because the matrix has no authority beyond the nine points. Refuse any echo that does not match the sequence number on the message. Expire part data: a coordinate pair older than the fixture's part-change time never fires a pick. Watch one full cycle at reduced speed, then hand the cell to production. The whole cycle from trigger to grip — and where each guard sits — is laid out in 2D vision-guided robot pick place grip: how the cycle works.

When is a raw socket the wrong transport?

Change transports when the consumer is not the robot controller: a PLC wants a register handshake with new-data and ready bits, and a pick that needs height and orientation wants a 3D sensor with its own calibrated frame.

The socket stays right for robot-to-vision links: the controller reads a string, the handshake is yours, and no gateway sits between. When the receiving side is a PLC, most integrations map the same coordinates into registers with a new-data bit and a ready bit — the pattern is in sending vision coordinates to a PLC through a register handshake. When the part arrives tilted or comes out of a bin, a pixel scale cannot represent it; start from 3D camera robot integration: check compatibility and plan the sensor calibration before writing the sender.

What do you check before the first production pick?

Six checks stand between the bench and production: grouped calibration residuals, the controller manual's terminator and port, echoed sequence numbers, an expiry timer on part data, a clear production-speed path, and a log file worth reading after a miss.

Every item is cheaper to check at the bench than after a crashed gripper:

• Residuals — re-project all calibration points; the worst one sits with the rest instead of standing alone.
• Terminator and port — from the controller's messaging manual, never assumed from a forum post.
• Handshake — unplug the cable once on purpose and confirm the 500 ms timeout holds instead of hanging the cell.
• Stale data — the expiry timer matches how fast the fixture changes parts.
• Envelope — the 1 mm sign check passed, and the full-speed path clears the fixture with the real gripper mounted.
• Log — every sent value, echo and cycle time lands in a file you can open after a miss.

With the list green, the link is done: Python finds the part, the matrix places it in base coordinates, and the socket moves the robot. When you want the hardware sized alongside the code — camera, vision controllers, light — start there, or send the cell layout through the contact page.