Wrist camera closed loop picking: iterative re-imaging on approach

Most wrist cameras ship with a one-shot pick routine: take a picture from a snapshot position, detect the part, then descend to the grip point with no further measurement. That is open loop, and it holds up until the part shifts, the nest is loose, or the first estimate is off by a millimetre. This page shows how to close the loop with the same hardware: re-trigger the camera at two approach waypoints, budget the latency of each correction, and decide per station when the loop pays for itself. You get the numbers for loop rate, waypoint placement, field of view and pixel size at the final stop, the checks that prove convergence, and the cases where one shot is still the right answer.

Can a wrist camera pick in closed loop?

Yes — a wrist camera that takes one snapshot can pick in closed loop if the robot program re-triggers it at two or three approach waypoints, turning a single coarse detection into a running correction loop during the descent. The camera does not servo on its own; the pick routine does. A stock routine grabs one image, finds the part, and commits to that grip point, so anything that moves after the shutter closes lands directly as pick error. Closing the loop means treating the first detection as a coarse fix: descend part of the way, stop, re-image, apply the X, Y and rotation correction, and repeat until the residual is inside tolerance. The per-frame math is the same single-shot transform behind robot picks: apply a vision correction of X, Y and angle in one shot, executed two or three times instead of once. When the only question is whether the part is present at all, wrist camera object presence detection without picking is the lighter pattern.

How fast can the correction loop run?

Plan on roughly 10 Hz: a realistic budget of 100 ms per correction cycle covers exposure, readout, transfer, detection and the robot acting on the answer, with command latency the largest single slice. Measure each term in your own cell before trusting the total:

Two terms dominate. A smart camera that detects on the device removes the transfer term and most of the detection term, which is why onboard-detection smart cameras are the default wrist choice for this pattern. The command slice — vision answer to robot, motion starting — is often the largest single term, and image based control latency: budget the full feedback loop breaks it into transport, program update and motion start. Exposure is the term you trade against light: a part that needs a 25 ms exposure pushes the same budget to 120 ms and the loop to 8 Hz. The floor is the sensor's own limit, and maximum frame rate on a hardware trigger: exposure plus readout time shows how to read it off the camera's frame-rate specification.

How many approach waypoints does a closed loop pick need?

Two stops after the snapshot, three at the outside: each waypoint costs about 250 ms — 150 ms of settling plus the 100 ms grab and process — so two waypoints add roughly 0.5 s to a 3 s monitored descent, about 17 percent of cycle. Place the stops where the question changes. The snapshot at about 300 mm standoff answers where the part roughly is, over a wide field. Waypoint one at about 200 mm confirms the part has not moved and tightens X and Y. Waypoint two at about 150 mm, just above the grip stroke, fixes rotation and sets the approach vector — from there, variable pick approach offsets for vision-guided picking decide how the gripper comes in. Teach the pattern at the height where you image it, as teaching pattern matching at final part height to keep scale right shows, or the template scale will not match either waypoint. Skip the last waypoint when the first correction already lands inside half your gripper tolerance; the loop is a tool, not a fixed sequence.

What field of view and pixel size do you get on the wrist?

At the final waypoint plan on a field of view near 100 mm and a pixel size at the part near 0.027 mm, which is fine enough that robot repeatability, not the camera, sets the floor on pick accuracy. Work the numbers for a 1-inch type sensor about 12.8 mm wide behind a 16 mm lens:

At the 300 mm snapshot: m = 16 / (300 - 16) = 0.056, FOV = 12.8 / 0.056 = 227 mm wide — enough to find the part anywhere in a small tray. At the 150 mm waypoint: m = 16 / (150 - 16) = 0.119, FOV = 12.8 / 0.119 = 107 mm. With about 4,000 pixels across the frame that is 107 / 4,000 = 0.027 mm per pixel, so a 0.5 mm edge spans 19 pixels and sub-pixel edge fitting resolves offsets well below 0.1 mm. Run your own standoff through the lens focal length calculator before ordering the lens.

When one snapshot height cannot see the whole table, multiple wrist camera snapshot positions to cover a large table extends the search; the correction loop then starts from the last search position.

Which camera should sit on the wrist?

A smart camera that detects on the device, mounted small at the wrist plate; keep a C-mount camera body as the fallback when the cell needs a different lens or more pixels than the integrated route gives you. Three routes cover most cells. The all-in-one route:

The LS-SMC-0063 is a 1.6 MP smart camera with a 16 mm lens, a white light source and 60 fps on board. Detection runs on the device, which is what removes the transfer term from the latency budget. That is the pattern to copy for any wrist loop: small body, built-in light, no PC in the correction path.

The near-focus presence route:

The MV-SCA002M-03S-WBN-SR is a 0.3 MP mono vision sensor with a 3 mm lens focused at 240 mm — a fixed mid-range focus that suits a confirmation waypoint, not a close-up final stop.

The pick-your-own-lens route:

The LS-SMC-0017 carries 8.8 MP, a C-mount and a global shutter — the body for the worked example above whenever the focal length has to change. Global shutter earns its place on a wrist: a rolling shutter smears whenever the scene moved during readout, and between waypoints the arm is what moved.

Set the three side by side:

Values as printed in each product's data sheet; open a part for its drawing and full specifications.

When is one shot enough without re-imaging?

When the part is fixtured and cannot move between image and grip, one shot plus a straight descent is the right answer, because the loop adds 0.5 s of cycle time and cannot improve a pick that was already inside tolerance. Decide by what can move after the shutter closes:

The conveyor row is where the loop is the wrong tool. Residual error equals part speed times loop latency: 100 mm/s × 0.1 s = 10 mm, and no number of waypoints removes that without syncing robot motion to the conveyor. The comparison still matters: one-shot picking on the same conveyor with 1.5 s between image and grip misses by 100 mm/s × 1.5 s = 150 mm, so the loop cuts the error fifteen-fold — but only tracking closes it.

How do you verify the closed loop converged?

Log the correction magnitude at every waypoint and require it to shrink monotonically toward the arm's repeatability band, typically 0.02 mm to 0.1 mm on a collaborative robot; a correction that stops shrinking means fault and retry, not grip. Four checks before you release the station:

• Corrections fall from waypoint to waypoint, and the last one sits near the repeatability spec rather than near the tolerance.
• The match score at the final waypoint stays above the taught threshold; a score that collapses at close range means the template was taught at the wrong height or the light changed during descent.
• A static repeatability test of 30 picks on a fixed part plots the reported X and Y spread — that spread is the loop's noise floor.
• A deliberate 5 mm part offset is recovered by the first correction within about one pixel at the part, proving the loop measures rather than repeats.

Confirm the image is genuinely fresh each cycle: a stale vision answer every cycle is the classic silent failure of routines that read a buffer instead of re-triggering the camera.

What breaks iterative re-imaging on approach?

Latency spikes and light changes, in that order: a correction that arrives 300 ms late during a 50 mm/s descent is 15 mm stale when it lands, and the gripper's own shadow at close range can drop the match score below threshold. Four failure modes cause most of that:

• Processing jitter. The budget assumes 100 ms; a vision process that stalls for 200 ms turns the loop into a 3 Hz loop exactly when the arm is closest to the part. Detect on the device or on a dedicated process.
• Motion blur. Re-imaging while creeping smears by speed times exposure: 25 mm/s at 5 ms is 0.125 mm, about 5 pixels at 0.027 mm per pixel. Stop at the waypoint or freeze with a strobe.
• The part leaves the frame. The field of view shrinks from about 227 mm at the snapshot height to about 107 mm at the final waypoint; a part near the edge of the first image can fall outside the last one.
• Lighting at close range. Gripper and camera housing shadow the part differently at every height. A camera with its own light source keeps illumination constant through the descent; overhead room light does not.

If stops are unacceptable in the cycle, the alternative is continuous servoing from a fixed camera above the cell on camera stands, kept alive by the overlap technique in two camera closed loop latency: grab and process in parallel.

What do you order to build the loop?

Start from a wrist-mount smart camera with detection on the device, add a C-mount camera body when the lens or the pixel count has to change, and lock both against your standoff in the configurator's camera step before anything else is ordered. Open the system configurator with your working distance and pixel target, compare candidates in the smart cameras finder, and drop to the area scan cameras finder when you need the C-mount route. For a station review, send the part size, the achievable standoff and the grip tolerance through the contact page and the applications team will run the same arithmetic with you.