YOLOv5 to NPU integration: keep your OpenCV tracking intact

You have a working detection system: YOLOv5 finds the objects, OpenCV handles the image work, and your own tracker keeps identities frame to frame — all on a host CPU, with inference eating most of the frame budget. The fix is not a rewrite. Cut the pipeline at one call: send the frame to an NPU accelerator, read back its raw output tensor, and keep everything else — decoding, tracking, drawing, signaling — running exactly as it does today. This guide shows where to cut, how to pull frames out of a pipeline callback, what changes in your post-processing, and a frame budget you can check against your own camera before ordering anything.

Where to cut: one call, five stages kept

Your loop has six stages:

1. capture a frame from the area scan camera
2. letterbox and normalize it to the network input size
3. run inference
4. decode: threshold the scores, then non-maximum suppression (NMS)
5. associate boxes into persistent tracks
6. draw overlays, log results, signal the PLC

The accelerator takes stage 3, and some runtimes fold stage 2 in as well. Stages 4 through 6 stay on the host, because the accelerator returns a raw tensor, not finished detections. The standard YOLOv5 export emits a tensor of shape [batch_size, number_of_anchors, 4 + 1 + number_of_classes]: four box values, one objectness score and one class score per row. That is exactly what your existing threshold-plus-NMS code consumes. Swap the call inside stage 3, wire its output into stage 4, and the rest of the loop does not change.

At the default 640 × 640 input, the detection head works on grids of 8, 16 and 32 pixel strides with three anchors each: (80 × 80 + 40 × 40 + 20 × 20) × 3 = 25,200 prediction rows. A single-class barcode model therefore returns 25,200 × 6 values, about 0.6 MB in float32. The tensor is small; the risk is a per-row Python decode loop, so keep thresholding and NMS vectorized in NumPy.

What actually changes in your post-processing

Three things move when inference leaves the CPU.

Coordinates come back in network space. Boxes are expressed in the letterboxed input frame, 640 × 640 pixels here. Convert each box back to full-image pixels with the letterbox scale and padding you logged at pre-processing time — store scale and pad per frame, not globally, or a zooming camera view will drift.

Scores shift slightly. The accelerator's compiler quantizes weights and activations, typically to 8-bit integers. Boxes land within a pixel or two of the float model; scores move more. Keep your pipeline, re-tune the thresholds. The same drift appears whenever a model crosses runtimes — the OpenCV ONNX preprocessing mismatch fix walks through one real case.

Layout is the compiler's decision. Read the converter report for the input layout it expects (NCHW or NHWC) and the output layout it returns, and build the letterbox and decode around exactly that. A fixed input size is normal: your letterbox must emit that size for every frame.

Getting a frame into OpenCV inside a pipeline callback

Framework-driven pipelines invert control: you no longer call the camera in your loop; the pipeline calls you. A callback shaped like app_callback(pad, info, user_data) is typical — the pipeline hands in event objects, and the frame travels attached to one of them, not as a NumPy array. Two habits make this safe:

• Unwrap the buffer yourself. Pull the buffer's pixel data into a NumPy array. Wrapping the buffer without a copy keeps the overhead near zero; the cv::Mat zero-copy pattern works for any buffer you can address as rows × columns × bytes.
• Put your state in user_data. That parameter is your channel: attach the tracker, the frame queue, configuration and a logger at set-up, and the callback reaches them on every call.

Then leave fast. Do the minimum in the callback — unwrap, queue, return — and let a worker thread run tracking and drawing. A callback that blocks stalls the pipeline, and the damage shows up downstream as a frame that never arrives within its timeout or a streaming callback that stops being called. If the host CPU sits near 100 % while frames still queue, replace the spin loop with an event-driven wait — the busy-waiting fix shows the pattern.

Worked example: frame budget for a 4000 × 3000 camera at 30 fps

Take a concrete capture side: the LS-CMR-0166, a 12 MP USB3 camera documented at 4000 × 3000 resolution and 30 fps, with a 1/1.7" rolling-shutter IMX226 sensor.

Your application adds three inputs of its own: 8-bit mono pixels out of the camera (1 byte per pixel — a natural choice for barcode work), a 640 × 640 three-channel network input (the export setting from the first section), and a usable USB3 throughput of 400-500 MB/s out of the link's 5 Gbit/s. That last number is the one estimate in this example; it is the practical range for host controllers.

The rules, in words: the frame budget is the wall-clock time one frame may consume, and the link load is frame bytes times frame rate.

Budget: 1000 / 30 fps = 33.3 ms per frame. Capture, inference, decode, tracking and signaling must all fit inside 33.3 ms, or the camera produces frames faster than you consume them.

Mono stream: 4000 × 3000 px × 1 B = 12,000,000 B = 12 MB per frame, and 12 MB × 30 fps = 360 MB/s = 2.88 Gbit/s — inside the 5 Gbit/s link with headroom. At a practical 500 MB/s the link carries 500 / 12 ≈ 41 fps of this stream, so the link, not the accelerator, is the ceiling.

Full 24-bit color at the same rate: 4000 × 3000 px × 3 B = 36 MB per frame, and 36 MB × 30 fps = 1080 MB/s = 8.6 Gbit/s — above the link's raw 5 Gbit/s. A full-rate 12 MP color stream cannot cross USB3 at all, whatever inference hardware sits behind it.

Network input: 640 × 640 px × 3 B = 1,228,800 B ≈ 1.2 MB — about a tenth of one mono frame and 29× smaller than the color frame.

Decision: request 8-bit mono at 30 fps, letterbox on the host down to the 1.2 MB network input, and hold 30 fps only while inference plus decode plus tracking stays under 33.3 ms. If the sum exceeds the budget, cut input size or frame rate — not the tracking that keeps your identities stable.

What still runs on the host, and which host

The host half is real work: camera control, letterbox, decode, tracking, overlay rendering and the I/O that tells a PLC or robot what happened. The accelerator module rides in that host, so pick the host by stream count and decode load, then add the accelerator to it.

All three controllers below offer at least three camera-side LAN ports and four or more built-in light channels, so capture, lighting and the host loop live in one box.

All 3 share: Certification CE, KC. Values as printed in each product's data sheet; open a part for its drawing and full specifications.

Decision by case. One or two cameras and a lean model: the Intel N-series unit with 8 GB of memory is enough host. A GigE camera fleet on one box: the Core i5 unit — its PoE camera ports power the cameras over the same cable and 32 GB of memory absorbs multi-stream decode queues. If you want a vision runtime preinstalled and a PCIe slot for the accelerator card, the Core i3 unit ships with a VM6200Pro license and PCIe expansion, without PoE. If the whole task collapses to detect-and-decide at the sensor, compare smart cameras: the inspection runs on the device with no host PC. The vision controllers range spans both extremes.

Setting up the accelerator's runtime on that box is a one-time job; a step-by-step NPU runtime walkthrough, from install to a working pipeline, covers the sequence on a small inference host.

Migration checks, in order

Run these before the production line sees the change:

1. Compile and read the report. Note the fixed input size, the expected layout (NCHW or NHWC) and the quantization; make your letterbox emit exactly that size.
2. Parity test on recorded frames. Push 200 saved frames through the float model and the accelerator; compare box overlap and scores. Expect near-identical boxes with slightly shifted scores.
3. Re-tune thresholds from step 2's score distributions before any pass/fail logic goes live.
4. Time stages with wall clock, not CPU time — the deadline is per frame, and the wall-clock versus CPU-time distinction decides whether you meet it.
5. Soak at full rate. Run 30 minutes at the target frame rate and watch for dropped frames and queue growth; if acquisition outruns processing, the frame-rate gap fix lists the levers.
6. Check tracker cadence. Feed detections at the rhythm your tracker was tuned for; faster inference only helps if the tracker consumes boxes at the rate it expects.

When an NPU is not the answer

Commit to the accelerator only when the numbers say so. At a few frames per second, a modern CPU host may already meet its budget with room to spare — CPU inference for inspection covers that path, and the CPU-or-GPU setup check gives the sizing questions. If several streams must run at full rate, a GPU-class host scales past what an edge NPU delivers. And if profiling shows the bottleneck is capture, pre-processing or drawing rather than inference, no accelerator helps — measure per stage first.

Next step: size it with your own numbers

Work the budget with your camera's figures — frame rate, pixels, bytes per pixel — then match hardware: the camera finder filters capture sides by resolution and interface, and the configurator assembles camera, host and lighting as one system. If the split between host and accelerator is unclear, contact us with your frame rate and network input size, and we will work the budget with you. The systems topic page collects the surrounding guides on buffers, callbacks and pipeline timing.