OpenCL on but processing slow: when GPU offload actually helps

You switched on OpenCV's Transparent API, the console prints OpenCL: ON, and nothing gets faster. One CPU core still runs at full load, the GPU idles, and the frame rate is unchanged. This guide is for anyone running OpenCV-based inspection on a vision controller, an edge box or a bench PC who expected the GPU to absorb per-frame work. It gives you the three causes ranked by how often they explain the symptom, a test for each, the arithmetic that decides whether offload can pay at your frame size, and the hardware to move to when it cannot.

What the ON flag actually reports

OpenCL: ON means the OpenCL module is compiled in and enabled at run time. It is a status flag for the API path, not a report that kernels executed. Since OpenCV 3.0, the Transparent API (T-API) routes qualifying calls to an OpenCL device to save repeated host-device transfers. A kernel reaches the GPU only when two gates pass: the device must qualify, and the data must stay on the device between calls. Fail either gate and every frame runs on the CPU while the console keeps printing ON.

Cause 1: the runtime reports less than OpenCL 1.2

Most ON-but-slow cases end here. OpenCV's OpenCL path requires a device that reports OpenCL 1.2 or newer. A GPU whose installed stack only exposes 1.1 is skipped without an error, and processing falls back to the CPU. The card can sit unused in the machine while the app prints ON.

Two traps hide the cause. First, a CPU-side runtime that reports 1.2 does qualify, so OpenCV can select the CPU as its OpenCL device: kernels then run on the same silicon plus copies and dispatch, which is slower than plain CPU code. Second, compiling your application against newer OpenCL headers changes nothing, because the check reads the version the device runtime reports, not your include files.

Test: run a system OpenCL reporting tool such as clinfo and read the version string printed next to each device. OpenCL 1.1 means skip; 1.2 or higher qualifies.

Fix: install a runtime for the GPU that reports 1.2 or newer on your operating system. If no such stack exists there, the machine has no GPU path for T-API, so optimise the CPU chain or move the stage to other hardware. Once kernels do run and frames are still slow, continue with Slow GPU image processing: diagnose low frame rates.

Cause 2: the pipeline pays the bus once per kernel

T-API operates on cv::UMat buffers. Data stays on the device only while every call in the chain reads and writes UMat. Build a cv::Mat in the middle of the per-frame loop — a copy, a conversion, a draw — and the next call re-uploads the whole image.

The arithmetic: what offload has to beat

Assume an effective transfer rate of 5 GB/s each way. That is a realistic pinned-transfer figure for a desktop PCIe link; measure your own, because it scales every conclusion below.

A 1920 × 1080 mono frame is 1920 × 1080 × 1 B = 2.07 MB. Upload plus download costs 2 × 2.07 MB / 5 GB/s ≈ 0.83 ms. With the chain entirely in UMat you pay that once: 0.83 ms of the 33.3 ms budget at 30 fps, about 2.5%. Ten filter kernels then need only to beat dispatch overhead, tens of microseconds per call on typical stacks. Break-even sits near a 2 ms CPU chain, and clear wins start around 5 ms.

Bounce to cv::Mat at each of the ten kernels and the same pipeline pays 10 × 0.83 ms ≈ 8.3 ms in copies alone, which is slower than the CPU code it replaced.

Scale checks. A 4K mono frame, 3840 × 2160, is 8.3 MB, so the round trip is 2 × 8.3 MB / 5 GB/s ≈ 3.3 ms; only chains of roughly 8 ms or more gain. A 640 × 480 ROI is 0.31 MB with a 0.12 ms round trip, the same order as dispatch overhead, so the GPU cannot win on small ROIs.

Cause 3: the work is too small or exits too early

A single detector call per frame, such as a Haar-cascade face or part detector, is the classic non-winner. The cascade rejects most of the image in its first stages on the CPU, so the saved compute is modest, while T-API adds a full-frame upload and per-stage dispatch. Keep a single detect-per-frame call on the CPU.

The same holds for threshold-and-count logic and anything touching a few thousand pixels. Expect slow first calls regardless: kernels compile on first use and cache afterwards.

The decision table

CPU costs are desktop-class orders of magnitude; measure your own chain, because the verdicts follow from the bus arithmetic above. If acquisition already outruns processing, the bottleneck sits elsewhere — see Acquisition faster than processing: fix the frame rate gap.

Fix it in this order

1. Print the selected platform, device and OpenCL version at startup. Below 1.2 on the GPU, fix the runtime or accept the CPU path.
2. Convert the per-frame chain to cv::UMat end to end: upload once, download results once.
3. Shrink pixels before adding speed: a smaller ROI or a downscale ahead of the chain beats any hardware change.
4. Time the whole chain after ten warm-up frames with the GPU path on and off, and keep the faster one.
5. If the CPU chain still misses the cycle time, move stages to hardware instead of adding threads.

Hardware that moves the work off the CPU

Offloading inside one machine is one option. The other is hardware shaped for the stage.

Rule-based multi-camera lines fit a vision controller: camera ports with PoE, built-in light channels and industrial I/O in a single unit. The MV-VC5525-256G70-NN pairs a 12th-generation Intel Core i5 with 32 GB of memory, 5+ LAN ports, PoE camera ports and eight built-in light channels.

Once a stage is a neural network, T-API is the wrong tool entirely, because inference belongs on dedicated silicon. The MV-VAC5724-256G02 is a dual-card AI inference terminal with PoE camera ports, 5+ LAN ports and eight built-in light channels. Sizing memory for model stacks has its own logic — see Why small neural network models use large GPU memory.

For rule-based chains that outgrow an i5, the MV-VC4712-256G10-2TH steps up to an Ultra 7 255H with 16 GB × 2 of memory, six built-in light channels, PoE camera ports and PCIe expansion.

Compact cells need less: an Intel Core i3 controller with 8 GB of memory and no PoE runs single-camera lines, and an Intel N-series controller with 3-4 LAN ports fits small multi-camera cells.

When one task is the whole inspection, the bus question disappears: a smart camera runs the algorithm on the device itself. The split between on-camera and host-side work is mapped in On-camera image processing: what can you offload?, and the same reasoning carries to depth work in Offload 3D inspection processing to a vision controller.

Where a chain stays under about two milliseconds per frame, plain CPU code wins — the same verdict the embedded-controls side reaches in when an embedded controller needs a GPU and when CPU code is enough.

The decisions in one place

• Runtime reports 1.1 on the GPU: install a 1.2-or-newer stack or drop T-API on that machine; the ON flag will never make it true.
• CPU chain under about 2 ms at 1080p: stay on the CPU and cut pixels instead.
• 1080p chains of 2-10 ms, or 4K chains of 8 ms and up: offload with the whole chain in UMat, measured warm.
• Neural-network stage: move it to an inference terminal rather than pushing it through T-API.
• Single-task cell: put it on a smart camera, and the bus leaves the cycle entirely.

For help drawing the line between camera-side and host-side processing, contact the applications team. More system-level checks live on the systems topic page.