Tracking a point with a wrist camera and OpenCV

A point marked on a shelf does not sit still in the image when the camera rides the robot wrist: every flange move slides it across the sensor. The fix is not a fixed camera but a fixed chain of transforms. Detect the point in each frame, turn the pixel into a camera-frame ray, carry that ray through the camera-to-flange and flange-to-base transforms, and the point becomes a base coordinate that holds still while the arm moves. If you are wiring vision guidance onto a small arm, this is the whole job in one pass: calibrate, detect, transform, track, and prove it.

What you need

• A wrist camera that streams frames to your PC. Any GigE Vision or USB3 Vision model whose vendor SDK hands you NumPy arrays works; the network side — addressing, packet size, retrieval — is covered in the GigE camera on a robot wrist guide.
• A checkerboard or ChArUco target for intrinsic calibration, sized to the working distance. The checkerboard target size guide gives the sizing rule.
• Ten to twenty flange poses with clearly different orientations, for the hand-eye step.
• Flange pose at exposure time. Position and orientation read from the controller in the same control tick as each frame, through whatever real-time data interface it exposes.
• Python with OpenCV. Calibration, detection and tracking all live in cv2; no extra licensing.

The transform chain: pixel to camera to flange to base

Three transforms, always in this order. The intrinsics turn a pixel into a ray in the camera frame. The hand-eye transform carries that ray into the flange frame, and it never changes unless someone moves the bracket. The flange-to-base transform changes with every pose, so it is read from the controller at the instant each frame is exposed.

Depth comes free from geometry you already know. The point sits on the shelf top, a plane whose height in base coordinates you measured with the calibration target. Scale the pixel ray until its base-frame z equals that height and the point is solved — no depth camera needed.

fx, fy, cx and cy come from cv2.calibrateCamera; u and v are the tracked pixel; Z falls out of the shelf-plane constraint. The full matrix algebra behind the chain is worked through in the eye-in-hand pixel to world coordinates guide, and the same chain reports a whole target pose, not just a point, as the runtime object pose guide shows. With the camera bolted to the cell instead of the wrist the chain changes shape; that eye-to-hand variant is worked in the eye-to-hand results guide.

Calibrate intrinsics, then hand-eye

1. Intrinsics. Grab 15 to 25 target frames across the working and tilt range, find corners with cv2.findChessboardCorners, refine them, and run cv2.calibrateCamera. Aim for a reprojection error under about 0.3 px. If it will not come down, fix the target poses before blaming the lens — the intrinsics-first guide walks that triage.
2. Hand-eye poses. Fix the target in the cell. Move the wrist so every pose sees the target from a different orientation, at least 30° apart, and record target-in-camera and flange-in-base together. The pose selection guide explains which poses carry information; the hand-eye target size guide sizes the board for that pose spread.
3. Solve. Hand the lists to cv2.calibrateHandEye(R_gripper2base, t_gripper2base, R_target2cam, t_target2cam). It returns the camera-to-flange rotation and translation, the constant T_flange_cam of the chain. Ten to twenty poses is the practical range.
4. Prove it. Run the park-and-move test below before anything downstream consumes the result.

Collaborative controllers often hide a nine-point alignment wizard that solves the same pair of transforms; the nine-point alignment guide explains what runs under that dialog.

Detect the point, then track it between frames

Match the detector to the target. A marked dot needs only a threshold, cv2.findContours and the contour moments, which give a sub-pixel centroid. A textured corner is found by cv2.goodFeaturesToTrack and carried by cv2.calcOpticalFlowPyrLK. An unmarked feature such as a hole is tracked by cv2.matchTemplate with a template cropped from a still frame, the peak found by cv2.minMaxLoc.

import cv2
import numpy as np

prev_gray: np.ndarray | None = None
pts: np.ndarray = np.zeros((0, 2), np.float32)   # tracked points, x,y in px
lk = dict(winSize=(21, 21), maxLevel=3,
          criteria=(cv2.TERM_CRITERIA_EPS | cv2.TERM_CRITERIA_COUNT, 30, 0.01))

gray = grab_gray_frame()                          # vendor SDK frame, uint8
if pts.shape[0] == 0:
    pts = cv2.goodFeaturesToTrack(gray, maxCorners=20, qualityLevel=0.2,
                                   minDistance=20)
else:
    pts, st, _ = cv2.calcOpticalFlowPyrLK(prev_gray, gray, pts, None, **lk)
    pts = pts[st == 1]                            # keep this frame's survivors
    if pts.shape[0] < 5:                          # track lost: re-detect
        pts = cv2.goodFeaturesToTrack(gray, maxCorners=20, qualityLevel=0.2,
                                       minDistance=20)
prev_gray = gray

The test that proves the loop: slide the target 20 mm on the bench. Through the pixel scale worked out below, the tracked centroid must move 230 to 260 px, and the base coordinate must read 20 mm within a millimetre. Note that cv2.goodFeaturesToTrack returns None on a frame with no corners, so guard the call in production.

Re-detect on a cadence as well as on loss — every 20 to 30 frames keeps small errors from accumulating. Thresholds that survive one shift fail at the next; the ambient light variation guide shows how to hold them steady, and if the loop itself is the bottleneck, the per-pixel loop fix is the usual culprit. If the arm parks and shoots instead of tracking, teach snapshot positions as the snapshot positions guide describes — the rest of this guide covers the continuous case.

Worked example: pixel scale and exposure for a shelf point at 250 mm

Inputs, each declared: camera class 1/2.9-inch sensor, 4:3 format, about 5.0 mm wide (the 16:9 class runs about 5.6 mm; the optical-format inch runs larger than the physical sensor); lens fixed 12 mm, as on the compact wrist sensors above; working distance 250 mm, lens to shelf, a short approach typical of an insertion cell; stream 1280 px wide; motion at 50 mm/s with frames grabbed at 30 Hz.

The rules, in words: magnification m = f / (WD − f). Field of view = sensor width / m. Pixel scale = field of view / image width. Exposure cap = pixel scale / approach speed. The working distance and field of view formula guide derives the first two in detail.

Decisions: cap the exposure at 1.5 ms while the arm moves, and take the measurement frame at a standstill, where 10 to 20 ms is safe and the centroid reaches sub-pixel precision. At 100 mm/s the cap halves to 0.8 ms and the step grows to about 43 px — still inside a three-level pyramid, but shrink the ROI to keep the loop fast. To frame exactly a 100 mm window at 250 mm: m = 5.0 / 100 = 0.05, f = 250 × 0.05 / (1 + 0.05) = 11.9 mm. The fixed 12 mm sits within 1% of ideal, so crop the ROI rather than change optics.

Pixel scale straight from magnification is the same arithmetic the object-space pixel size guide generalises.

Camera and lens choices for the wrist

Every gram at the flange comes off the payload and feeds vibration into the image. Jitter with no mechanical source can be measured frame to frame, as the camera vibration guide shows. Motion rules out rolling shutter, an integrated lens removes the focus-drift failure mode, and one Ethernet cable carries the frames: 1280 × 960 px at 30 Hz, 8-bit, is 1280 × 960 × 30 ≈ 37 MB/s, well inside a Gigabit link's usable rate.

Two compact M12-mount vision sensors cover the common cases. The MV-SCB007EC-12S-WBN pairs a 1.6 MP color sensor with a fixed 12 mm lens and manual focus. The MV-SC3050XC-16M-ATN-H moves up to 5 MP on a 1/1.45-inch sensor. Both are global-shutter models on Fast Ethernet.

When the pixel budget tightens, take the wide-angle route: pair the 8.8 MP, 1-inch-format C-mount smart camera with the compact 8.5 mm, 10 MP wide-angle below, and the C-mount keeps optics swappable as the cell changes.

Browse the wider range in the smart cameras category or the camera finder; the optics live in the lenses category and its lens finder.

Bench-test the geometry before it goes on the flange

Build the same triangle on a bench first: camera, target at working distance, a ruler for scale. Shoot the ruler and measure the pixel scale — it should land in the 0.077 to 0.087 mm/px band from the example. Set detector thresholds against the shelf's real contrast, not against a printed target. Slide the target 20 mm by hand and watch the loop. If it survives hand speed, it will survive the arm.

The LS-MNT-0040 slide-table stand holds that triangle still: 700 mm of height and a 400 x 450 mm base leave room for the target, the ruler and a laptop beside the camera. The camera stands and brackets category carries the slide tables and wrist-mount brackets; the accessories finder filters them by type. When the loop graduates from your laptop to a PC mounted in the cell, fanless industrial mini PCs skip the fan-filter maintenance.

Prove it: the park-and-move test

Pick the shelf point and keep it fixed. From four or five flange poses with orientations at least 30° apart, capture frame and flange pose together and compute the base coordinate. The answer must come out the same from every pose; that invariance is the whole point of the chain.

Budget the spread before judging the result. Detection repeatability of ±5 px at 0.077 mm/px is ±0.4 mm, and hand-eye plus intrinsic residuals typically add around a millimetre — so a spread within 2 mm across poses says the calibration is good, while more than 5 mm says a pose was degenerate or something moved. The pick point offset guide sorts those causes in order.

Then jog the arm 50 mm along one axis while tracking. Pixels stream, and the base coordinate must stay inside the same band. If it walks as the arm moves, the flange pose is being read late: at 50 mm/s, a pose read 10 ms after the exposure shifts the answer by 50 × 0.010 = 0.5 mm. Read the pose before the frame arrives, or time-stamp both and match them.

Common mistakes

• Standstill exposure used in motion. Ten milliseconds at 50 mm/s smears the point 0.5 mm — five to six pixels at this scale — and the centroid drifts. Cap at 1.5 ms while moving; standstill for the measurement frame.
• Pose read after the frame. The 0.5 mm per 10 ms of lag lands straight in the answer. Time-stamp image and pose, then match them.
• Distortion left in. Stock M12 lenses can bend the image a percent or more; one percent of a 640 px half-width is 6 px, about half a millimetre here. Calibrate intrinsics first and undistort the tracked points, which is far cheaper than warping the whole frame.
• One template across light changes. Templates and thresholds drift with the room. Re-crop the template under production lighting, or control the illumination itself.
• Hand-eye frozen forever. Every bump to the bracket invalidates the camera-to-flange transform. Rerun the park-and-move test after any contact — it costs five poses and a minute — and recalibrate only when the spread fails.

Where to go next

Size the camera and lens from the numbers above in the camera configurator, then check the neighbouring recipes in the systems guides hub — calibration, offsets and network setup for the same wrist. For a second opinion on a station design, the contact page reaches the applications team.