Running a custom CNN for object recognition in robot picking

You designed the network, and the camera on the arm already finds parts, with a recognizer you cannot touch. Closed smart cameras and vision sensors run their recognition in firmware, so the answer is not a new model inside the camera but a pipeline around it: keep the camera as the image source, run your CNN on a vision controller beside the robot, convert the detected pixels into robot coordinates, and hand the result to the pick routine over TCP. This guide walks that path end to end: the capture link, the inference hardware, the pixel-to-millimetre math, a worked two-point calibration, and the socket message the robot side expects.

Can you load your own CNN into a closed smart camera?

No: the recognition inside a closed smart camera is part of its firmware, and there is no supported way to replace it with your own model, so your CNN has to run outside the camera, on the images the camera delivers.

The camera still gives a pipeline everything it needs: every frame it captures, its exposure and gain controls, and usually a strobe output for the light. Treat the camera as the sensor, not the intelligence — the controller finds the part and returns coordinates the robot can act on. Settle who owns which decision before writing code; the vision-versus-robot control split is worth five minutes of design first.

How do you get the camera's images into your own CNN?

Pull frames over the camera's standard interface or its SDK: a GigE Vision or USB3 Vision camera streams into any GenICam-based client, and closed cameras commonly expose a live-image or last-snapshot URL that returns the newest frame.

Then each pick runs five steps:

1. Trigger the shot. Park the arm at a taught snapshot pose, then fire a hardware trigger from a spare robot output or send a snapshot command from your code.
2. Fetch the frame. Grab continuously and keep the newest image, or request one snapshot per cycle if the link is shared.
3. Convert once. Turn the native Bayer or mono stream into the format your model expects; a mismatch silently corrupts the input (see verifying the converted image format).
4. Preprocess. Resize and normalize exactly as in training: same input size, same constants.
5. Stamp it. Record which robot pose each frame belongs to; a result from the previous pose is the classic one-part-off pick.

Which hardware runs the CNN at the pick station?

Run inference on a fanless vision controller with a Core i5-class processor, 16-32 GB of memory and one LAN port for the camera plus one for the robot; a small YOLO-class detector at 640 × 640 input typically lands in the 20-60 ms range there.

Treat that as a planning budget for a small detector: benchmark your own model once on the chosen controller and hold the number. When the model outgrows the CPU, the controller's PCIe expansion takes a GPU card.

From the vision controllers range: the MV-VC3301H-128G66 pairs an Intel Core i3 with 8 GB of memory, five or more LAN ports and four built-in light channels, enough for one camera, one robot link and a strobed light. The MV-VC5525-256G70-NN moves to a 12th-generation Core i5 with 32 GB of memory, PoE camera ports, eight light channels and PCIe expansion.

If the cell has no room for a controller box, an X86-platform smart camera is the class that carries an inference stack on the camera itself: the LS-SMC-0017 combines an 8.8 MP, 1-inch global shutter sensor with a C-mount. Running custom object detection models on embedded smart cameras shows what fits there, and choosing a lightweight CNN backbone sizes the model; bigger networks move to dedicated edge AI hardware for real-time object recognition. Before ordering, check the cable run: GigE reaches 100 m on standard Ethernet cabling, while USB3 stays comfortable at roughly 3-5 m passive.

How many pixels does the part need for recognition?

As a rule of thumb, put at least 100 pixels across the part for detection and 200 or more when the network must resolve small features; with the part size fixed, that decision fixes the field of view.

The optics set it:

A 1/1.7-inch sensor is about 7.5 mm wide. With a 12 mm lens at 300 mm working distance: FOV = 7.5 mm × 300 mm / 12 mm = 187.5 mm. On a 4000-pixel-wide image that is 187.5 / 4000 = 0.047 mm per pixel, so a 50 mm part spans 50 / 0.047 ≈ 1060 pixels — ten times the floor, which means you could raise the camera or lengthen the lens and still pass. Halve the working distance to 150 mm and the same lens covers 93.8 mm, or 0.023 mm per pixel.

The LS-CMR-0166 reads out 4000 × 3000 frames at 30 fps over USB3 — the image width the example assumes. Its rolling shutter wants a stationary part or a strobed flash; a wrist shot while the arm is still settling wants the global shutter of the MV-CU020-90GC on GigE instead.

How do you turn pixel coordinates into robot coordinates?

Teach two points with known robot X-Y values, fit a scale and offset through them, and apply that transform to every detection; at the 0.047 mm per pixel from the lens example, one pixel of detection jitter is about 0.05 mm.

Calibrate in four steps:

1. Jog the gripper tip onto a reference feature inside the camera view — a dowel pin in a fixture works — and record its pixel position together with the robot's X and Y.
2. Move diagonally across the field onto a second feature and repeat.
3. Compute the scale on both axes; if the two disagree by more than a few percent, the camera is rotated or tilted and the transform needs a rotation term, which a third taught point recovers.
4. Store the anchors and the scale; every detection then converts in one multiply-add per axis.

Worked through: the pin taught at pixel (500, 500) sits at robot (100.0 mm, 80.0 mm), and the second pin at pixel (2500, 1500) sits at (200.0 mm, 30.0 mm). The X run gives (200.0 - 100.0) / (2500 - 500) = 100.0 mm / 2000 px = 0.05 mm per pixel, and the Y run gives (80.0 - 30.0) / (1500 - 500) = 50.0 mm / 1000 px = 0.05 mm per pixel — equal, so the mount is square. The network reports a part centre at pixel (1800, 900): X = 100.0 + (1800 - 500) × 0.05 = 165.0 mm, Y = 80.0 - (900 - 500) × 0.05 = 60.0 mm. Y flips sign because image rows count down while the robot's Y counts up.

Prove the chain end to end by placing a part at a robot-measured position and running capture, inference and conversion: at this scale the reported coordinate should land within about ±0.2 mm, and the worst error usually appears at the field corners, where lens distortion peaks.

How does the result reach the robot controller?

Open one TCP socket between the vision controller and the robot controller and exchange one ASCII line per pick — X, Y, angle, status — because every industrial robot controller can speak TCP client or server sockets from within its own language.

Let the robot drive the exchange: it opens the connection, sends a request token when the arm reaches the snapshot pose, and waits on the reply. The vision side answers with a line such as 165.0,60.0,0.0,OK or 0.0,0.0,0.0,NOPART, plus a robot-side timeout of about 500 ms so a missed detection costs half a second instead of a hung cycle. The camera-to-robot TCP handoff shows that socket loop line by line, and the straight-down pick variant covers the station where camera and gripper share one frame.

How much cycle time does the external CNN add?

Budget 50-100 ms from trigger to transmitted coordinates — 20-60 ms of inference, under 10 ms for capture and preprocessing, under 1 ms for the socket send — against the 1-2 s the arm spends approaching, gripping and retracting.

The vision figures are planning numbers for a small detector on the i5-class controller above; time your own model once and hold that budget. At 50-100 ms, vision contributes under a tenth of a 1-2 s cycle, so the pick rate you measure — roughly 30-60 picks per minute on a small arm — is the robot's number, not the network's. If vision ever becomes the constraint, overlap it: fire the capture as the arm starts to retreat from the previous drop-off, and inference runs inside the robot's own motion.

When should you keep the camera's built-in recognizer instead?

Keep the built-in recognizer whenever it passes around 100 consecutive live cycles under production lighting; changing it buys nothing if the closed tool already locates the part reliably, and the external pipeline adds hardware and a calibration to maintain.

Your own CNN earns its keep when appearance varies naturally (food, castings, moulded parts), when classes change often enough that re-teaching a locator costs more than labelling images, or when you need a custom score in the handoff message. The rule-based versus learned inspection trade-off frames that choice, how many images to train an object detection model sizes the data job, and owning your trained model weights keeps the deployed network under your control. An object-picking control comparison for a collaborative arm works through the same onboard-versus-offboard decision end to end.

Build the stack from three shelves — the image from smart cameras or area scan cameras, the inference from a vision controller — then size the optics in the camera finder and the system configurator, or contact us with the part and the target cycle time and we will check the numbers with you.