ONNX detection on a camera stream in Python: the full loop

The notebook already works: your ONNX detector reads a webcam through cv2.VideoCapture(0) and draws boxes. The production cell is a different machine. The camera is an area scan camera on GigE Vision or USB3, frames arrive several megapixels large at a fixed rate, and a loop that falls behind that rate is a line stoppage, not a cosmetic problem. This guide wires the whole path in Python: grab a frame, letterbox it to 640 × 640, scale by 1/255.0, transpose to CHW float32, run onnxruntime, apply NMS, and map every box back to full-frame pixel coordinates, with the bandwidth and latency arithmetic that decides which camera and link can carry it.

What a vision camera changes about the grab

The webcam path hides three settings a machine-vision camera forces you to choose: pixel format, exposure and frame size. Your SDK or GenTL producer hands each frame to Python as a NumPy array, so everything downstream of the grab looks familiar. What decides feasibility is bytes per second, and that arithmetic comes first.

A 4000 × 3000 frame in Mono8 is 12 MB, one byte per pixel. Thirty frames per second is 360 MB/s, close to 2.9 Gbit/s. A 1 Gbit/s GigE link carries about 115 MB/s of image payload, so full-size frames arrive near 9 fps (115 ÷ 12). USB3 carries about 400 MB/s and holds that stream with headroom. The same arithmetic for the three frame sizes you will most likely quote:

For 12 MP at 30 fps, use USB3. The LS-CMR-0166 is a 12 MP USB3 camera with a 4000 × 3000 resolution at 30 fps, and it is the worked example through the rest of this guide.

On GigE you accept the link ceiling, crop a region of interest, or bin the sensor: 2 × 2 binning turns each 12 MB frame into 3 MB, and 3 MB at 30 fps is 90 MB/s, inside the link. What frame rate really fits GigE Vision on a high-speed camera, and how binning changes what sensor readout allows, each deserve their own read. The MV-CU020-90GC is a 2.5 MP GigE camera with a global shutter, and the MV-CS200-10GC is a 20 MP GigE camera.

At full size a 20 MP Mono8 frame is about 20 MB, so the link holds about five per second, workable whenever the part pitch is slower than 200 ms.

The loop in seven stages

1. Grab a full-resolution frame as a NumPy array.
2. Set the color: demosaic Bayer to BGR, or pass Mono8 through; convert BGR to RGB.
3. Letterbox to the model's 640 × 640 input.
4. Scale by 1/255.0, transpose HWC to CHW, cast to float32.
5. Run the onnxruntime session.
6. Decode the output tensor and apply NMS.
7. Map the kept boxes back to full-frame coordinates and act on them.

Stages 3 to 6 run on every frame, so their total time is your latency budget. The grab side, threading, buffers, wiring the grab loop so acquisition never blocks inference, is the same pattern as any streaming vision task.

Build it, step by step

1. Lock the camera before writing Python

Auto exposure and auto gain belong off, or your gray levels drift between morning and afternoon: turn every auto mode off and cap the maximum exposure time so motion blur cannot creep in. Pick the pixel format deliberately: Mono8 for a mono-trained model, demosaiced Bayer for a color one, and request Bayer explicitly if the host falls back to mono. When a PLC triggers the shot, start from the hardware trigger grab loop pattern.

# acq stands for your acquisition object. Every GigE Vision and
# USB3 Vision SDK delivers the frame as a NumPy array; the call
# name below is a placeholder for your SDK's grab call.
frame = acq.grab_next()  # uint8, shape (3000, 4000) in this example

2. Letterbox, never stretch

The common notebook line, cv2.resize(image, (640, 640)), squashes a 4:3 frame into a square. For 4000 × 3000 it stretches the vertical axis 1.33 times relative to the horizontal one: circles become ellipses and edge features shift. A model trained on letterboxed images then meets a distribution it never saw, and when ONNX confidence comes back lower than in training, this mismatch is the first suspect.

import cv2
import numpy as np

def letterbox(frame, size=640):
    h, w = frame.shape[:2]                    # 3000, 4000 in the example
    scale = min(size / w, size / h)           # min(0.160, 0.213) = 0.160
    nw, nh = int(w * scale), int(h * scale)   # 640, 480
    resized = cv2.resize(frame, (nw, nh))
    canvas = np.zeros((size, size, 3), dtype=np.uint8)
    pad_x, pad_y = (size - nw) // 2, (size - nh) // 2   # 0, 80
    canvas[pad_y:pad_y + nh, pad_x:pad_x + nw] = resized
    return canvas, scale, pad_x, pad_y

3. Build the tensor exactly as the export expects

The same three operations the webcam snippet uses, in the same order. The float32 cast comes last and is not optional: dividing a uint8 array by 255.0 produces float64, and onnxruntime raises a type error on a float64 input.

def to_tensor(canvas):
    rgb = cv2.cvtColor(canvas, cv2.COLOR_BGR2RGB)
    t = rgb / 255.0
    t = np.transpose(t, (2, 0, 1))                       # HWC -> CHW
    return np.expand_dims(t, axis=0).astype(np.float32)   # (1, 3, 640, 640)

4. Run the session once per frame

import onnxruntime as ort

sess = ort.InferenceSession('detector.onnx')
input_name = sess.get_inputs()[0].name
pred = sess.run(None, {input_name: to_tensor(canvas)})[0]
# A YOLO-style export returns shape (1, 4 + num_classes, num_anchors).
# Print it once after loading, then code the decoder to what you see.

5. Decode and apply NMS

Box coordinates sit in the first four rows, class scores in the rest. Threshold on the best class score, then suppress overlaps by intersection over union. Confidence 0.25 and IoU 0.45 are the usual starting pair; tighten IoU to 0.30 when parts lie close together on the belt.

def decode_and_nms(pred, conf=0.25, iou=0.45):
    p = np.asarray(pred[0]).T                  # (num_anchors, 4 + num_classes)
    boxes, scores = [], []
    for row in p:
        c = int(np.argmax(row[4:]))
        s = float(row[4:][c])
        if s < conf:
            continue
        cx, cy, w, h = row[:4]
        boxes.append([cx - w / 2, cy - h / 2, w, h])
        scores.append(s)
    b = np.array(boxes, dtype=np.float32)
    order = np.argsort(scores)[::-1]
    keep = []
    while order.size > 0:
        i = order[0]
        keep.append(i)
        if order.size == 1:
            break
        r = order[1:]
        x1 = np.maximum(b[i, 0], b[r, 0]); y1 = np.maximum(b[i, 1], b[r, 1])
        x2 = np.minimum(b[i, 0] + b[i, 2], b[r, 0] + b[r, 2])
        y2 = np.minimum(b[i, 1] + b[i, 3], b[r, 1] + b[r, 3])
        inter = np.maximum(0, x2 - x1) * np.maximum(0, y2 - y1)
        union = b[i, 2] * b[i, 3] + b[r, 2] * b[r, 3] - inter
        order = r[inter / union < iou]
    return [boxes[i] for i in keep], [scores[i] for i in keep]

6. Map boxes back to full-frame pixels

def to_frame_coords(box, scale, pad_x, pad_y):
    x, y, w, h = box
    return [(x - pad_x) / scale, (y - pad_y) / scale,
            w / scale, h / scale]

# Round-trip test: this box decodes to (1800, 850, 400, 300) on the
# 4000 x 3000 frame, and letterboxing that box again returns it.
assert [round(v) for v in to_frame_coords(
    [288.0, 216.0, 64.0, 48.0], 0.160, 0, 80)] == [1800, 850, 400, 300]

7. Assemble the loop, then keep up or drop

while True:
    frame = acq.grab_next()      # newest frame only, never a backlog
    if frame is None:
        continue
    canvas, scale, pad_x, pad_y = letterbox(frame, 640)
    pred = sess.run(None, {input_name: to_tensor(canvas)})[0]
    boxes, scores = decode_and_nms(pred)
    full = [to_frame_coords(b, scale, pad_x, pad_y) for b in boxes]

When one pass is slower than the frame period, do not queue. Hold a single-slot buffer with the newest frame, process what fits, and count skips as a reported metric. A growing queue turns spare throughput into latency and then into a memory leak, the failure bounded frame queues exist to prevent. Measure the achieved rate where it matters: count frames per second in Python at the point of decision, not at the grab.

Worked example: one frame, every number shown

Input: a 4000 × 3000 Mono8 frame from the LS-CMR-0166 at 30 fps.

• scale = min(640 / 4000, 640 / 3000) = min(0.160, 0.213) = 0.160
• Scaled image: 4000 × 0.160 = 640 wide, 3000 × 0.160 = 480 tall.
• Padding: (640 − 480) / 2 = 80 px top and bottom, 0 px at the sides.
• A detection centered at input pixel (320, 240) maps to x = 320 / 0.160 = 2000 and y = (240 − 80) / 0.160 = 1000, the center of the full frame.
• A 64 px wide box at input scale is 64 / 0.160 = 400 px wide on the sensor.
• Skip the padding and every box lands 80 / 0.160 = 500 px too low; the boxes still look plausible, which is what makes it expensive.
• Smallest part worth detecting: a working rule for YOLO-style detectors is 10 px across in the 640 input, which is 10 / 0.160 = 62 px on this frame. Parts smaller than about 62 px need a cropped region of interest or a second, zoomed pass.

Latency: fit the pass inside the frame period

At 30 fps the frame period is 1000 / 30 = 33.3 ms. Stages 3 to 6, plus the hand-off to your controller, must fit inside it.

The budget that matters runs trigger to decision, so budget the full feedback loop once, including the network write to the controller. On motion, the example camera has a rolling shutter, which skews geometry on a fast conveyor. Freeze it with a short, hard strobe driven from a light controller; the LS-CTL-0322 is a 4-channel digital light controller with an 8 A total output. Otherwise pick a global-shutter camera. Never sight a strobed LED head by eye; align the beam with a card.

Mistakes that cost a day

• Stretching instead of letterboxing: 1.33 to 1 distortion on a 4:3 frame, and confidence drops first on the small parts.
• Skipping cv2.cvtColor(canvas, cv2.COLOR_BGR2RGB): the model sees red and blue swapped.
• Dropping the .astype(np.float32): the tensor arrives as float64 and the session call fails.
• Mapping boxes back without subtracting the padding: a 500 px vertical offset at this scale.
• Queuing frames to keep up: latency grows without bound and the queue eats memory.
• Tuning detector thresholds while auto exposure quietly moves the gray levels underneath.

Checks before the cell accepts it

• Draw the mapped boxes on full-resolution frames; a part you can identify by eye must sit inside its box.
• Keep the round-trip assert from step 6 in the commit; it fails loudly if anyone changes the letterbox.
• Soak for 30 minutes at full rate: processed frames plus reported skips must equal delivered frames.
• On GigE, treat blank frames as GigE Vision packet loss until proven otherwise.
• If a USB3 stream stalls at full resolution, raise the host USB buffer before blaming the model.
• Measure trigger-to-decision latency once, with a timestamped log line or a scope on the strobe.
• On a moving stage, use a drag-chain rated camera data cable: the LS-CBL-0219 is a 3 m USB3.0 high-flex cable built for that duty.

From here

For pass/fail surface inspection rather than object detection, the decision side is covered in ONNX anomaly maps to pass/fail. To size the whole station, camera, lens, light, controller and trigger, start the configurator at the camera step or browse the camera finder. If the numbers above leave the camera choice open, send the frame size, part pitch and smallest feature through the contact page and we will size it with you.