Feeding GPU tensors to a vision model without a CPU round trip

You stream frames from an area scan camera, preprocess them on the GPU, and want to feed the result straight into a detection model. Instead, the framework raises an assertion: it expected a PIL image or an HWC numpy array and received a torch tensor. The usual workaround found in forum threads — convert with .cpu().numpy(), transpose, scale by 255 — works but pushes every frame off the GPU, through host memory, and back again. On a real line this copy, not the model, becomes the bottleneck. This troubleshooting guide explains the layout mismatch, ranks the causes, and shows a pipeline structure that keeps frames resident on the device from capture loop to inference result.

The symptom and what it actually means

The error is an assertion inside the framework's input loader, not inside the network. High-level predict APIs accept many source types — file paths, URLs, PIL images, OpenCV frames, video streams — and normalize all of them into one internal format before inference. The assertion fires when the loader does not recognize your tensor as a supported source in your framework version, or when the tensor branch runs after a check written for image objects.

The key fact: the network itself only ever consumes a 4-dimensional float tensor in BCHW layout, RGB channel order, values normalized to roughly 0-1, sized to a multiple of its stride (often 640). Everything upstream of that — file decoding, resizing, letterboxing, normalization — is convenience code you can replace yourself with tensor operations on the GPU.

Likely causes, in order of probability

1. Unsupported tensor input in your framework version. The tensor branch is incomplete or buggy in some releases, particularly for segmentation variants. Update first; if the release notes still do not claim tensor support, plan to bypass the wrapper.
2. Wrong memory layout. You passed a tensor, but in the wrong shape or range: BCHW float 0-1 versus HWC uint8 0-255 will fail or silently produce garbage even when tensors are accepted.
3. Tensor on the wrong device. A CPU tensor handed to a pipeline that assumes GPU residency, or the reverse, triggers conversion or assertion paths you did not intend.
4. A second, hidden error behind the first. After converting to numpy, some users hit an OpenCV resize assertion because the resized dimensions are empty — a downstream symptom of the model device not resolving correctly, not of your image shape. Check that the model reports the device you expect before debugging shapes.

The layouts, side by side

The pattern is clear: the wrapper accepts host-side image containers and does the HWC-to-BCHW, uint8-to-float conversion for you. If your data is already a GPU tensor in BCHW, the wrapper has nothing to do — which is exactly why feeding it a CPU numpy array "works" and costs you two device transfers per frame.

For background on measuring where your frame time actually goes, see CPU time vs wall-clock time in vision pipeline benchmarking.

The fix: keep preprocessing on the device

Rebuild the preprocessing chain as tensor ops, then call the model's forward pass directly instead of the predict wrapper.

1. Acquire once into host memory. Frames from a GigE or USB3 camera arrive as host buffers; that is fine. Perform any CPU-side decode or rotation now — see camera image rotation without losing acquisition speed for keeping that step efficient.
2. Transfer once. Move the uint8 frame to the GPU with a single .to(device, non_blocking=True) from pinned host memory.
3. Convert layout on device. Permute HWC to CHW, add the batch dimension, cast to float, divide by 255.
4. Resize and letterbox on device. Use torch.nn.functional.interpolate with the same interpolation mode the framework uses, then pad to the stride multiple. Record the scale factor and padding so you can map boxes back to original pixel coordinates.
5. Call the network, not the loader. Invoke the underlying module's forward pass with your BCHW tensor. Post-process its raw output yourself, or feed it to the framework's result class if it accepts one.
6. Copy results, not images. Return boxes, scores and class IDs — kilobytes — to the host. Never copy the frame back.

A worked example shows what the copy costs. A 20 MP color frame is about 5472 × 3648 × 3 ≈ 60 MB. Over PCI Express 3.0 x16, realistic sustained transfer is near 10-12 GB/s, so each direction costs roughly 5-6 ms — and 11-12 ms per frame if you copy both ways. At 30 fps your frame budget is 33 ms; the avoidable round trip eats a third of it, before the model runs a single layer. On an embedded module with slower interconnect the penalty is worse. Use our camera resolution calculator when choosing the sensor size in the first place — fewer needless megapixels shrink both the copy and the inference cost.

Hardware that makes this worthwhile

The on-device pipeline pays off when the inference hardware has enough GPU headroom to absorb preprocessing kernels alongside the model. In our vision controller line, AI-oriented units are built for exactly this role.

For single-model deployments, the MV-VAC3504X-128G01 is an AI inference terminal with 5+ LAN ports, 3-4 USB 3 ports and 4 built-in light channels, so it can host the camera link and the model in one box. For heavier pipelines with room to grow, the MV-VC5525-256G70-NN offers 32 GB of memory, PoE camera ports and 8 light output channels on a 12th-generation Core i5 platform with PCIe expansion for a GPU card. Browse the full vision controllers range, or see industrial mini PCs on our sister site if you also need general-purpose compute hardware.

On the sensor side, match the camera to the model's input size rather than overspecifying: a MV-CS200-10UM, a 20 MP USB3.0 mono camera with a 1" sensor and C-mount, gives plenty of pixels for cropping to a 640-input model while leaving margin for optics. The area scan camera finder filters by interface, resolution and shutter; if the whole system is still on paper, start at the configurator.

How to verify the fix worked

• Profiler check. One inference traced: no device-to-host copy before the forward pass, no host-to-device copy except the single frame upload.
• Numerical check. Run one image through the wrapper path and your on-device path; bounding boxes should agree to within a pixel or two. Larger offsets mean a padding or channel-order mistake, not a float precision issue.
• Throughput check. Frame rate at the result stage should rise by roughly the copy time you eliminated — in the 20 MP example, 5-10 ms per frame.
• Latency check. End-to-end capture-to-result time should drop and, more importantly, stop spiking when the CPU is busy, because the path no longer depends on host scheduling. Related reading: camera acquisition high CPU usage: replace busy waiting.

When the usual answer is wrong

• When the model is not on a GPU at all. If inference runs on CPU, an on-device preprocessing chain is pointless; optimize the CPU path instead.
• When the wrapper does real work you need. Tracking IDs, result plotting, multi-stream fan-out — if you rely on those, calling raw forward means reimplementing them. Weigh the milliseconds saved against the code you now own.
• When frames arrive already decoded on the GPU from a hardware decoder or a capture API that supports device memory. Skip step 2 entirely; the win there is even larger, but buffer lifetime rules get stricter — see CUDA context invalid after camera close: manage GPU buffers.
• When you deploy a new model version. A framework update can change the expected preprocessing (letterbox versus stretch, normalization constants). Re-run the numerical check after every model swap; our guide to swapping a neural network model on edge vision hardware covers the wider update checklist. For model loading failures that appear at import rather than at inference, the notebook GPU model upload fixes knowledgebase article is a useful companion.

Quick checklist

• Confirm the framework version's supported source types before writing conversion code.
• Pin host memory and make exactly one host-to-device copy per frame.
• Permute to BCHW, cast to float, scale to 0-1 — all on device.
• Letterbox with recorded scale and padding so boxes map back to pixels.
• Call the network directly; post-process raw outputs.
• Copy only results to host; verify with a profiler, not a stopwatch.

If your pipeline chains several cameras and models, or you are deciding which controller has the GPU budget for it, tell us the frame rate, resolution and model size and we will size it with you — start on the contact page or request a quote.