Select an ROI with the mouse and track it in a live video stream

You want to click a box around whatever part lands in view and follow it frame by frame, with no markers and no fixture. Once your camera's frames arrive as OpenCV Mats, the camera side stops mattering: the same mouse selection and the same template trackers run on a USB area scan camera, a GigE camera, or the colour stream of a depth camera. This guide walks the whole loop: converting frames, drawing the ROI with the mouse, choosing between KCF and CSRT with speed numbers, sizing the box, and proving the tracker holds before it drives anything.

How do you get a live camera frame into an OpenCV Mat?

Read the newest frame from your camera SDK or from cv2.VideoCapture, convert it to 8-bit BGR with cv2.cvtColor, and everything downstream accepts it: a 4000 × 3000 colour frame becomes a numpy array with shape (3000, 4000, 3) and type uint8.

Two sources cover almost every set-up. A UVC webcam or an RTSP stream reads through cv2.VideoCapture directly. A vendor SDK frame arrives as a raw buffer, so copy it into a Mat or wrap it zero-copy, as wrapping an SDK frame buffer in a cv::Mat without copying shows. Convert once per frame, because the trackers in this guide accept 8-bit frames only, colour or grayscale.

Any camera that hands you 8-bit frames works. A 2.5 MP global-shutter GigE model such as the MV-CU020-90GC keeps fast-moving targets undistorted, while the 20 MP MV-CS200-10UM over USB3.0 buys pixels when the target is small in the frame.

How do you select an ROI with the mouse?

Call cv2.selectROI once on the frame you want to pick from: it opens a window, lets you drag the rectangle, and returns (x, y, w, h) as four integers in pixels. Press Space or Enter to confirm; press c to cancel, which returns all zeros.

box = cv2.selectROI('pick target', frame, showCrosshair=True)
x, y, w, h = box                  # ints, pixels, origin at top-left
if w == 0 or h == 0:
    raise SystemExit('selection cancelled')

Draw on the frame at native size, or scale the returned box back by the same factor before you use it anywhere else.

For what to do with the region after the click, the guide on finding the region a user clicked picks up where this leaves off.

Which template tracker keeps up with your frame rate?

At a 30 fps stream with a box near 430 pixels wide, KCF typically holds the rate on a desktop CPU while CSRT processes roughly 15-30 frames per second and drops the rest: pick KCF for rate, CSRT when clutter or deformation keeps beating it.

Tracker cost scales with the box area, not the frame size, so the figures below describe a roughly 430-pixel box on a current desktop CPU. The tracking classes ship with the contrib package (opencv-contrib-python), not the base build.

The example camera is the LS-CMR-0166: 4000 × 3000 at 30 fps over USB3, 12 MP on a 1/1.7" rolling shutter sensor.

Suppose the lens and working distance frame a 300 mm wide field on that camera, and the part is a 25 mm wide bracket:

• Pixels per millimetre: 4000 px ÷ 300 mm = 13.3 px/mm
• Bracket width: 25 mm × 13.3 px/mm ≈ 333 px
• Box with 15% margin on each side: 333 px × 1.3 ≈ 433 px, an area of about 188,000 px²
• Frame budget: 1000 ms ÷ 30 = 33.3 ms

The decision follows: KCF at 100-200 fps clears the 33.3 ms budget three to six times over, so it tracks every frame the camera sends. CSRT at 15-30 fps sits at or under the budget, so it processes most but not all frames; run CSRT on a 30 fps stream only when dropped camera frames are acceptable, or shrink the box toward 300 px, which nearly halves the work (300² ÷ 433² ≈ 0.48).

How big should the ROI box be?

Draw the box tight around the target with a 10-20% margin per side, and keep the target itself at least 20-30 pixels across: the 333-pixel bracket from the worked example sits at 433 pixels boxed, comfortably inside both limits.

Every pixel in the box teaches the filter what to follow, and background pixels pull the box toward the background, which is how trackers drift onto a neighbouring clamp or seam. The cost side is quadratic: double the box width and each update takes about four times as long. Below roughly 20-30 pixels of target the template carries too little texture to lock, and the fix is optical, not algorithmic: a lens with more reach or a shorter working distance until the part spans 30 pixels or more.

How do you wire the pieces into one working loop?

Initialize the tracker once on the frame you drew on, then call update once per new frame and draw the box it returns: the whole control flow is select once, update forever, re-select only on failure.

1. Open the stream and read one full frame; stop if the read fails.
2. Call cv2.selectROI on that frame and unpack x, y, w, h; refuse a zero-size box.
3. Create the tracker and call init(frame, box) on the same frame you drew on.
4. Loop: read the newest frame, call update, draw the box when the tracker reports success.
5. Count consecutive failures; after about 30 (one second at 30 fps), pause and re-select instead of letting a lost box wander the scene.
6. Release the camera and close the windows on exit.

import cv2

cap = cv2.VideoCapture(0)                     # any source your OpenCV build reads
ok, frame = cap.read()                        # frame: uint8 array, shape (H, W, 3)
assert ok, 'camera returned no frame'

box = cv2.selectROI('pick target', frame, showCrosshair=True)
x, y, w, h = box                              # ints, pixels
assert w > 0 and h > 0, 'selection cancelled'

tracker = cv2.TrackerKCF_create()             # CSRT variant: cv2.TrackerCSRT_create()
tracker.init(frame, box)

misses = 0
while True:
    ok, frame = cap.read()
    if not ok:
        break
    found, box = tracker.update(frame)        # box: four floats, same order
    if found:
        misses = 0
        x, y, w, h = [int(v) for v in box]
        cv2.rectangle(frame, (x, y), (x + w, y + h), (0, 255, 0), 2)
    else:
        misses += 1
        if misses >= 30:
            break                             # re-select here in your code
    cv2.imshow('track', frame)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

cap.release()
cv2.destroyAllWindows()

What changes on a depth camera stream?

Track on the colour or IR image and apply the same (x, y, w, h) to the registered 16-bit depth map: the coordinates are identical pixels, and keeping depth at 16 bits preserves the millimetre values you will measure later.

A depth camera delivers several streams at once: a 2D image plus a depth map, typically 16-bit millimetre values. Convert the 2D stream to BGR exactly as in the first section, which using a depth camera as a plain RGB camera in OpenCV covers in detail, and select and track there. The box then transfers to depth one-to-one once the streams are registered; if the overlay lands offset by a constant few pixels, that is calibration, and fixing a depth-to-RGB registration offset in pixels walks the correction. A monochrome-looking colour preview means you grabbed the IR stream, as why a depth camera's RGB shows black and white explains.

Keep two copies if you need both: an 8-bit normalized depth image for your eye, the 16-bit original for measurement. To carry the box downstream into a point cloud or volume, crop the depth frame in 2D first, as cropping a depth ROI before the point cloud reaches OpenCV shows.

On a ToF sensing camera such as the MV-DT01WMU, the same loop applies: track on the 2D image, then take the box from the depth map of the same frame.

Why does the stream stall while you draw the box?

cv2.selectROI blocks its thread, so a queued SDK keeps buffering frames and hands you a stale one when you resume; keep only the newest frame in a single slot, or grab-and-discard after selection, and the backlog collapses from seconds to one frame.

The picker runs its own event loop and returns only when you confirm, so any queue behind the camera keeps filling. An SDK with a frame buffer can hand you a one- or two-second-old frame afterwards, and the tracker then initializes on a scene that has already moved. Cheapest fix first: after selection, read and discard until you are current, then init on a fresh frame. For continuous display, run the reader in a thread that overwrites one slot, so the UI always sees the newest frame:

latest = [None]              # one slot, always the newest frame

def reader():                # runs in a background thread
    while running:
        ok, f = cap.read()
        if ok:
            latest[0] = f

With the single-slot pattern, latency after selection is one frame period, 33.3 ms at 30 fps, and the tracker starts on the scene you actually drew on.

How do you prove the tracker is working before it drives anything?

Lock the tracker on a stationary target and log 300 frames at 30 fps: a healthy box wanders under about 5 pixels of jitter, the mean update stays under the 33.3 ms frame budget, and covering the target should fail visibly rather than silently.

The test below prints both numbers and the verdict:

import time

budget_ms = 1000.0 / 30.0           # 33.3 ms at 30 fps
centres = []                        # (cx, cy) tuples of floats
t_total = 0.0

for _ in range(300):                # 10 s on a stationary, textured target
    ok, frame = cap.read()
    t0 = time.perf_counter()
    found, box = tracker.update(frame)
    t_total += (time.perf_counter() - t0) * 1000.0
    if found:
        bx, by, bw, bh = box
        centres.append((bx + bw / 2.0, by + bh / 2.0))

mean_ms = t_total / 300.0
cx = sum(c[0] for c in centres) / len(centres)
cy = sum(c[1] for c in centres) / len(centres)
jitter = max(abs(c[0] - cx) + abs(c[1] - cy) for c in centres)
status = 'PASS' if mean_ms < budget_ms and jitter < 5.0 else 'FAIL'
print(f'update {mean_ms:.1f} ms / budget {budget_ms:.1f} ms, jitter {jitter:.1f} px -> {status}')

A FAIL on jitter means the box is too loose or the target lacks texture; tighten the box or improve the light before touching the tracker settings. A FAIL on timing at KCF means the box is far larger than the 430-pixel assumption, so shrink it. Then run the occlusion test: cover the target for two seconds and watch the box slide off while the miss counter climbs. That visible failure is what you want; silent drift is what the jitter test catches. If the camera side reports nonsense numbers first, reading frame rate reliably in OpenCV sorts that out.

To run this on hardware, pick the camera on the camera finder, compare area scan cameras for 2D tracking or 3D sensors when depth joins the box, or start at the camera configurator. Application support comes free with every order if you want the pixel budget checked against a specific lens, or contact us directly.