ONNX inference slower in OpenCV DNN: pick the right runtime

A YOLO model that benchmarked fine last month now takes twice the frame time, and the code did not change. That symptom rarely comes from the model itself: it comes from which execution backend actually computed the graph, and a rebuild can switch that path silently. This guide gives you the three checks that identify the active backend, the throughput arithmetic that decides how many frames per second a station can honestly serve, and a decision rule for staying on OpenCV DNN or moving to ONNX Runtime or OpenVINO. It is written for engineers deploying exported .onnx detectors on machine-vision controllers and smart cameras.

Why does the same ONNX file run slower after an OpenCV upgrade?

The slowdown is a backend change: frame time is set by which execution path computes the graph, and a rebuild can move you from a CUDA-enabled build to a prebuilt CPU-only package — one reported case: 5 ms for a YOLOv5m ONNX file on one build, 12 ms on another. Same machine, same GPU, same file, and the newer build reported 13 ms for YOLOv8m too — the model was not the lever. A 2.4× jump with no code change is the signature of a build or code-path change, not of the network.

Ranked by how often we see it:

1. The build lost its CUDA support. The standard prebuilt OpenCV package carries no CUDA backend, so swapping a self-compiled build for a downloaded one drops every net to CPU.
2. The backend assignment disappeared. OpenCV DNN runs on the CPU by default unless the code assigns DNN_BACKEND_CUDA with DNN_TARGET_CUDA; a refactor that drops those two lines moves the whole graph back to the processor.
3. The provider order changed. In ONNX Runtime, the first provider that supports a node wins; a session created with CPU first runs CPU forever.
4. The export changed. A new opset or exporter can restructure the graph enough to move speed; rare, and it shows up in the validation run before it shows up on the line.

Which backend is actually running my model?

Confirm the backend with three checks: read the build-information report for CUDA support, read the backend and target your code assigned to the net, and time warmed-up frames with the GPU synchronized — cv::getBuildInformation() answers the first in one string.

Check the build. cv::getBuildInformation() prints the compile flags as one string; search it for CUDA and cuDNN. Entries that read NO mean DNN_BACKEND_CUDA cannot run in that build, whatever the code asks for. Save that string with the deployment artifacts — it is the fastest way to prove two machines should behave the same.

Check the assignment. The net runs where the code sends it, and the two calls are explicit:

net.setPreferableBackend(cv::dnn::DNN_BACKEND_CUDA);
net.setPreferableTarget(cv::dnn::DNN_TARGET_CUDA);

No assignment means the CPU default. On GPUs with tensor cores, DNN_TARGET_CUDA_FP16 is worth a benchmark pass once accuracy has been validated on your own set.

Check the time. Ten warm-up frames, then a hundred timed frames with the GPU synchronized before the clock stops. A CPU-side timer around an asynchronous GPU call measures launch overhead, not execution, and it will tell you a 2 ms model is fine while the camera starves. Log milliseconds per frame at startup; a slide from 5 ms to 12 ms is invisible on a user interface and unmistakable in a log.

How much does GPU inference add over the CPU fallback?

Plan on a factor of roughly 3× to 16×: OpenCV's published benchmark of the DNN CUDA backend reports 211–1549% faster inference than the CPU path across popular detectors, so the runtime choice moves frame time more than any tuning inside one runtime. Treat 3× as the planning floor and 16× as the ceiling, then measure your model — where it lands decides everything downstream.

Worked example — sizing a line. Eight cameras each need 15 fps, so the model must run 120 times per second (8 × 15 = 120 fps). A timing pass on the target controller gives this breakdown for a 640 × 640 YOLOv8m ONNX model; the 13 ms inference figure is the reported number from the case above, and the surrounding stages are example values for one station:

Capacity per sequential pipeline: 1000 ÷ 22 ≈ 45 fps. Demand 120 ÷ capacity 45 = 2.64, so the line needs three parallel pipelines — 3 × 45 = 136 fps against the 120 fps demand, about 14% headroom. Now swap the runtime and the same model runs at 5 ms inference: 4 + 3 + 5 + 2 = 14 ms per frame, capacity 1000 ÷ 14 ≈ 71 fps, demand 120 ÷ 71 = 1.68 — two pipelines, 143 fps, about 19% headroom. The runtime decision removed an entire pipeline: one fewer thread pool, one fewer copy of the model in memory.

Watch where the bottleneck moved: at 5 ms the model is no longer the longest stage, decode and preprocessing are. That is the point where reuse image buffers instead of per-frame allocation and feeding GPU tensors to a vision model without a CPU round trip buy more than another runtime swap.

Should you stay on OpenCV DNN, or move to ONNX Runtime or OpenVINO?

Stay on OpenCV DNN when measured inference time fits the cycle budget with a 2× margin; move to ONNX Runtime when you need more out of the same GPU, and to OpenVINO when the deployment machine is an Intel-CPU controller with no discrete GPU.

All three read the same .onnx file you already export — no conversion step, no second training run. That is what makes the decision easy to make and easy to reverse: benchmark the identical file in all three on the target machine, keep the fastest that passes your validation set, and record the numbers. If accuracy moved along with speed after an export, that is a separate failure — the OpenCV ONNX confidence fix covers preprocessing mismatches, the usual cause.

Quick decisions by situation:

• One model, latency fits with a 2× margin: stay on DNN and spend the effort on decode and preprocessing.
• Same GPU, need more than the DNN build delivers: ONNX Runtime with the CUDA provider; the measured gain is whatever your file gives, and even 20% can remove a pipeline at the margins.
• Controller with no GPU, and the budget says it stays that way: OpenVINO on the Intel processor; the object detection without a GPU guide covers the CPU path in depth.
• One camera making one simple decision at modest line speed: a smart camera, next section.

How do you switch a session to the CUDA execution provider?

Providers are set once, at session creation, in priority order: append CUDAExecutionProvider first and CPUExecutionProvider last, so supported nodes run on the GPU and the rest falls back without a second build or a code change per model.

1. Install the GPU build of the runtime. The CPU-only package exposes no CUDA provider at all, and no session option adds one back.
2. Append the providers in priority order. In the C API that is OrtSessionOptionsAppendExecutionProvider_CUDA(options, 0); Python bindings take the list directly: providers=['CUDAExecutionProvider', 'CPUExecutionProvider']. First supporting provider wins, so order is everything.
3. Fail loudly when CUDA is missing. Print the session's active providers at startup and raise an error if CUDAExecutionProvider is absent — a silent CPU fallback is exactly the failure mode you are debugging.
4. Warm up and time. Ten frames to ramp the clocks, a hundred timed frames with the GPU synchronized, and milliseconds per frame written into the startup log next to the provider list.
5. Re-benchmark after every runtime update with the same fixed test image, and compare against the logged number rather than against a feeling.

Where should the model run: controller GPU, controller CPU, or on the camera?

Put the model where the arithmetic puts it: a controller with a discrete GPU when one pipeline cannot serve the demand, CPU with OpenVINO when the budget says no GPU, and on the camera itself when one camera makes one simple decision.

On the GPU path the controller needs a free PCIe expansion slot for the add-in card. Controllers with PCIe expansion and PoE camera ports cover this path:

The MV-VC4712-256G10-2TH carries six built-in light channels, the MV-VC5525-256G70-NN eight, and both expose PoE camera ports, so cameras and lights land on the same box as the model. Shortlist by LAN port count, PoE and memory in the vision controllers category, then confirm PCIe expansion on the product page.

On the CPU path, skip the card and let OpenVINO tune for the Intel processor already in the box. Expect less than discrete-GPU speed, let the measured number decide whether the cycle still fits, and size the deployment with the arithmetic above. Most multi-camera CPU deployments in this field run on vision controllers — from Intel Celeron units to Core i7 models with PoE camera ports — for exactly this reason.

When one camera makes one simple decision at modest line speed, the runtime debate disappears: the model runs on the camera. A 6 MP GigE smart camera with an integrated 8 mm M12-mount lens covers the common detection-and-locate cases:

The smart cameras category runs from 0.3 MP vision sensors to devices beyond 25 MP, and the camera finder filters by resolution, shutter and lens mount.

How do you stop inference speed drifting on the next rebuild?

Version-pin the runtime, print the active backend or provider in the startup log, and re-benchmark one fixed test image at every release — the 5 ms regression then appears in the commissioning record instead of three months into production.

• Pin the runtime and exporter versions. Updates move both speed and operator support; treat the version numbers as build inputs like any compiler flag. The SDK version pinning fix for edge inference devices shows the pattern.
• Log the build and the providers. The build-information string, the backend and target constants, the session's active providers, milliseconds per frame — all in the startup record, one line per machine.
• Keep the build recipe with the deployment. The CMake flags (WITH_CUDA, OPENCV_DNN_CUDA, WITH_CUDNN) and the runtime package versions, so the next rebuild reproduces the fast path instead of discovering its absence on the line.
• Benchmark in the release routine. One fixed image, ten warm-up frames, a hundred timed frames, the number compared to the previous release before the machine ships.
• Record the export settings. The opset and exporter version next to the .onnx file, because both speed and operator coverage move with them.

When the deployment spans several machines, the edge vision inference deployment planner splits stages across cameras and hosts before the hardware is frozen.

Next step: run the timing pass and the arithmetic on your own file, then browse the vision controllers with PoE camera ports and PCIe expansion — the shortlist a GPU retrofit starts from. For a retrofit into an existing controller or a cycle budget that will not move, the contact page reaches the applications team, and the systems guides collect the rest of the deployment work.