JPEG decode for a GPU vision pipeline: where it belongs

A JPEG frame lands in your buffer and every step after it runs on the GPU. The natural question — can the GPU library decode the frame itself — has a firm answer: no. Vision acceleration libraries such as VPI ship kernels for filtering, warping and morphology, and they ship no JPEG codec at all. So cv::imdecode keeps its job, and the craft is in the handoff: decode into a cv::Mat, wrap that Mat as a GPU image, then submit the accelerated steps from there. This page gives you the call sequence with exact function names, the per-frame loop that avoids reallocation, the arithmetic that sizes the decode stage in CPU cores, and the cases where the decode stage should not exist at all.

The decision: decode on the CPU, then wrap

VPI 2.0.14 cannot decode JPEG, and no other release can either: the library targets vision and image-processing algorithms, not codecs. vpiSubmitConvertImageFormat does not rescue you here. It converts an already-decoded image between VPI pixel formats, so a compressed byte stream sitting in a byte vector is not an input it accepts.

The working pattern is three steps, and it uses the CPU decoder you already have:

1. Decode with OpenCV: bgr = cv::imdecode(jpg_bytes, cv::IMREAD_COLOR).
2. Wrap the decoded Mat once: vpiImageCreateWrapperOpenCVMat(bgr, 0, &img).
3. Submit every GPU step with img as the input image.

That is the whole answer: cv::imdecode is not reimplemented inside the GPU library — you hand the library its output. The decode belongs at the very front of the pipeline, one step before the wrap, on the CPU. Everything below is about running that handoff without stalls and without hardware you do not need.

Why GPU vision libraries leave JPEG out

A baseline JPEG decode is a fixed chain: Huffman entropy decoding, dequantization, an inverse DCT on each 8 × 8 block, chroma upsampling and a YCbCr-to-RGB conversion. It is serial, mature and memory-bound at the tail, which makes a CPU core a good home for it. GPU vision libraries sell kernels for operations your algorithm applies to pixels — filters, pyramids, thresholds, warps — and a codec is none of those. OpenCV's imgcodecs module wraps a libjpeg-class decoder with decades of tuning behind it, so the codec work stays where it already performs.

Two exceptions are worth knowing. Some embedded SoCs expose a dedicated hardware JPEG block through the vendor's multimedia API, and at hundreds of megapixels per second it outruns any CPU core. Standalone open-source GPU JPEG decoder libraries also exist, but they add a dependency to save cores that inspection frame rates rarely miss. The arithmetic in a moment tells you when you have crossed that line.

The handoff: imdecode, wrap, submit

The wrap is the bridge between the two worlds — the same trick as wrapping an SDK frame buffer in an OpenCV cv::Mat without copying, run in the other direction. The sequence, with types and initial values:

// once, at start-up
cv::Mat  bgr;                        // decoded frame, CV_8UC3 after the first decode
VPIImage img = nullptr;              // GPU-side image handle, created empty

// first frame, on the decode thread
bgr = cv::imdecode(jpg_bytes, cv::IMREAD_COLOR);  // jpg_bytes: a std::vector of bytes, the compressed frame
if (bgr.empty()) {
    // truncated stream or not a JPEG: reject the frame here
}
vpiImageCreateWrapperOpenCVMat(bgr, 0, &img);     // flags = 0: share the buffer, no copy

// every later frame
bgr = cv::imdecode(next_jpg_bytes, cv::IMREAD_COLOR);
vpiImageSetWrappedOpenCVMat(img, bgr);            // re-point the wrapper, no reallocation
// submit your GPU steps with img as the input image

Two calls carry the design. cv::imdecode proves its own work: a non-empty Mat with bgr.cols and bgr.rows matching the expected geometry is a good frame, and an empty Mat is your corrupt-frame signal. The wrapper call with flags 0 shares the Mat's memory rather than copying, which is why the rebind on the last line is enough to hand a new frame to the GPU.

Prove the wrap on a constant image before you trust it with real frames. Fill a 100 × 100 single-channel Mat with the value 137, wrap it, submit any one GPU step, and read the image back through your library's image-read call. Every returned pixel must be 137: equality shows the wrapper shares the Mat, so real frames arrive intact. Check the status return of every vpi call too — a failure at creation time means a format or flags mismatch, not a broken frame. And if a decode hangs, check what you fed it: raw pixels going into the JPEG decoder is its own trap, covered in JPEG decode hangs on raw pixel data: encode before decoding.

If your source hands you a raw pointer instead of a Mat, vpiImageCreate accepts one, but the OpenCV wrapper is the road built for imdecode output. Match the exact wrapper name against the header your installed SDK ships: some published samples spell it vpiImageCreateOpenCVMatWrapper, the same idea in an older argument order.

The frame loop: rebind, don't recreate

Creating a wrapper per frame turns the pipeline into an allocator benchmark. Create img once at start-up and call vpiImageSetWrappedOpenCVMat at the top of every frame — the wrapper moves a header, not pixels, so the rebind costs almost nothing. Keep the pixel format stable end to end as well: switching between grayscale and color decodes changes every buffer size and kernel input downstream.

The ring costs less than you might expect. A 12 MP BGR frame is 36 MB, so a ring of four Mats is 144 MB per camera stream — carry 150 MB per stream into your memory budget. Multi-channel frames change the byte count, not the method; checking multi-channel image buffer sizes before processing walks through that variant.

Sizing the decode stage: a worked example

One division decides the decode hardware:

The example uses a 4000 × 3000 stream at 30 fps, the geometry of a 12 MP camera class:

• Pixels per frame: 4000 × 3000 = 12,000,000 = 12 MP.
• Decoded BGR frame: 12,000,000 px × 3 bytes = 36,000,000 bytes ≈ 36 MB.
• Decode load: 30 fps × 12 MP = 360 MP/s.
• Per-core rate: open-source turbo decoders on recent desktop cores land in the 50–150 MP/s band; plan on 100 MP/s until you have timed your own build.
• Cores: 360 ÷ 100 = 3.6, so four cores do nothing but decode.
• Time check: the frame budget is 1000 ÷ 30 = 33.3 ms, and one core needs 12 ÷ 100 = 0.12 s = 120 ms per frame — 3.6× over budget, hence four cores.
• RAM traffic: 36 MB × 30 fps = 1.08 GB/s of writes, about 2 % of a dual-channel DDR4-3200 bus (3200 MT/s × 8 bytes × 2 channels = 51.2 GB/s). Bandwidth is not the problem; cores are.
• Compressed side: a quality-85 encode lands near 10:1, so the JPEG is ≈ 3.6 MB and 30 fps pulls 108 MB/s — comfortable on USB3 or 2.5GbE, near the practical ceiling of a 1 GbE link.

Time your own build before committing the core count: decode one representative frame 100 times, then rate = 100 × 12 MP ÷ elapsed seconds. If the measurement lands at 60 MP/s instead of 100, the core count rises from 3.6 to 6 — the arithmetic scales linearly. Measure wall-clock time, the distinction CPU time vs wall-clock time in vision pipeline benchmarking draws, because decode threads sleep on I/O.

When your frames never need a decoder

Area scan cameras on GigE or USB3 deliver raw or Bayer frames, not JPEG, so for them the decode stage does not exist: the camera buffer converts straight into your processing format, a one-step pixel conversion rather than a codec. Bandwidth makes the point — a 4000 × 3000 8-bit mono frame is 12 MB, and at 30 fps that is 360 MB/s, inside USB3's practical 400 MB/s.

JPEG enters only when a source compressed the frame on purpose: file drops from a line-stop archive, saved inspection images, or an MJPEG stream from a network camera. The stream case is its own acquisition topic — acquire an RTSP stream from an IP camera in your vision program — and the raw case starts at convert a raw camera buffer to a processing-ready image.

The camera below carries the workload from the worked example — 12 MP, 4000 × 3000 at 30 fps on USB3:

When the usual answer is wrong

The frames are already uncompressed. Delete the decode stage — no imdecode, no wrap; convert the camera buffer directly. Bolting a JPEG stage on out of habit burns the four cores from the worked example for nothing, and the decode-before-processing rule begins by asking whether any decode exists at all: see image bytes vs pixel data: decode before processing.

Decode is the measured bottleneck. The fix is not a bigger GPU, because the GPU was never decoding. Move decode to the hardware JPEG block through the vendor's multimedia API and hand its output — usually NV12 or another YUV layout — to vpiSubmitConvertImageFormat, which then does real work: turning decoder output into the BGR your color pipeline expects. If frame rates still sag after that, slow GPU image processing: diagnose low frame rates is the next stop.

Two decode builds disagree by a level or two. Inverse DCT implementations differ, so the same JPEG can decode with ±1 intensity-level differences between decoder builds. Never golden-image-compare outputs across decode libraries; compare against the decoder your production line runs.

The frames are small. 1080p MJPEG at 60 fps is 2.07 MP × 60 = 124 MP/s — two cores by the arithmetic above. That is a thread-pool setting, not a hardware-decode project.

Where the pipeline runs

Acquisition threads, the decode pool, the wrapper and the GPU dispatch all live on the processing host, and that host is a vision controller rather than an office PC: the LAN port count has to match the camera list, USB ports have to cover the rest, and memory has to hold frame rings plus the OS and model weights. From the worked example, one stream wants a 150 MB ring, so four streams plus headroom fit inside 16 GB, and 32 GB leaves room for larger frames or more streams.

The three controllers below cover that spread, from a single-camera cell to a multi-camera line:

The middle model carries 32 GB of memory, 5+ LAN ports and 8 built-in light channels. That is the shape of a four-camera line with strobed lighting. The entry model carries 8 GB of memory and fits the single-stream cell; move up a model when the ring math or the model weights say so. Where the GPU stage is an add-in card, the PCIe expansion all three list is its entry point — match the card's slot, power and cooling figures against the product page. Downstream of the wrap, feeding GPU tensors to a vision model without a CPU round trip continues the story.

Checks before you order

• Decode cores first. Cores for decode, plus cores for the library's host code, plus one or two spare. The worked example starts at four decode cores, which makes an 8-core host the practical floor for one stream with real processing behind it.
• Memory. Frame bytes × Mats per ring × streams, plus model weights and OS. The 12 MP example begins at 150 MB per stream.
• Ports. LAN and USB counts against the camera list, PoE if the cameras need it; the filters on the controller range sort by all of these.
• Archive rate. If JPEG frames land as files, 3.6 MB × 30 fps ≈ 108 MB/s ≈ 389 GB per hour — size the SSD for the retention you actually need.
• The GPU itself. Whatever accelerator your library targets, its slot, power and bandwidth join the budget alongside the 1.08 GB/s the decoder already writes.

Where the numbers above land you in the controller range:

If the decode cores outrun the CPU tier, or the rings outgrow the memory tiers, send the frame geometry, frame rate and camera count through the contact page — free expert support comes with every order, and this arithmetic is exactly what that conversation starts from. The systems topic page collects the surrounding pipeline pieces.