System designExplainer

GPU capture and display offload: keep CSI camera pixels off the CPU

Why an OpenCV loop drags CSI camera frames onto the CPU, and how a GPU capture and display offload pipeline keeps conversion, scaling and display in hardware.

LumaSpec Vision11 min read Share on LinkedIn

On this page 11 sections
  1. 01The decision
  2. 02Where the pixels go in the sample loop
  3. 03What one copy costs
  4. 04Capture full resolution, display 720p
  5. 05Three architectures compared
  6. 06How to choose
  7. 07DeepStream or plain GStreamer
  8. 08Cameras for an embedded build
  9. 09Checks before you commit
  10. 10When the usual answer is wrong
  11. 11Questions

An OpenCV loop that reads a CSI camera and shows it with cv2.imshow runs almost entirely on the CPU, even on a SoC with a media engine and a GPU. Every frame crosses from the camera stack into system memory, is converted to BGR in software, then uploads back to the graphics stack for display — and the caps filter in most sample pipelines caps capture at 720p, so full sensor resolution is lost before your code runs. This explainer is for engineers building an embedded preview or inspection station who want capture, color conversion, scaling and display to stay in hardware, with the CPU touching only the frames an algorithm reads. It walks the sample pipeline element by element, prices every copy in MB/s, and gives the pipeline that captures 3280 × 2464 while displaying 1280 × 720.

The decision

Keep the display path in hardware and copy only what an algorithm reads. Four choices, made once at design time:

  • Capture at the sensor's full mode. The source caps after nvarguscamerasrc ask for the sensor's full resolution, not for the window size. Resolution decisions belong at the edges of the pipeline.
  • Scale for display in the hardware converter. nvvidconv takes 3280 × 2464 down to 1280 × 720 and hands NVMM buffers to the platform's hardware video sink. The CPU never touches a displayed frame.
  • Copy to system memory on one branch only. appsink serves the frames an algorithm inspects, at the size that algorithm needs — the pull pattern that keeps that branch healthy is in GStreamer appsink to OpenCV: pull frames without blocking the pipeline.
  • Skip DeepStream until there is inference. One camera, one scaler, one window is plain GStreamer; DeepStream 6.0 pays its setup cost when detectors and trackers ride the same pipeline.

OpenCV stays in the build — as the consumer of the processing branch, never as the display engine.

Where the pixels go in the sample loop

Each element name in the sample says where the pixels sit; the string syntax is covered in GStreamer pipeline strings for on-board camera capture explained. Here only the memory path matters:

nvarguscamerasrc ! video/x-raw(memory:NVMM), NV12, 1280 × 720 ! nvvidconv flip-method=2 ! video/x-raw, BGRx ! videoconvert ! video/x-raw, BGR ! appsink, then cv2.VideoCapture(pipeline, cv2.CAP_GSTREAMER), cap.read(), cv2.imshow.

  1. nvarguscamerasrc delivers NV12 frames in NVMM memory: buffers owned by the media engine, reachable by the display controller without the CPU. The caps filter fixes them at 1280 × 720, so the full sensor resolution is lost at the first link — the camera stack was never asked for more.
  2. nvvidconv runs on the hardware converter, and flip-method=2 flips the image there at no cost. But its output caps request plain video/x-raw BGRx, and that change of memory type is a crossing: the frame is copied out of NVMM into system RAM.
  3. videoconvert turns BGRx into BGR on a CPU core. appsink wraps the buffer, cap.read() copies it into an OpenCV array, and cv2.imshow uploads it back toward the GPU to draw the window.

One displayed frame pays a hardware-to-system copy, a software color conversion, a Python-side copy and an upload back — thirty times a second — and cv2.waitKey(30) waits up to 30 ms per loop, capping the loop at about 33 fps (1000 ÷ 30) whatever the camera delivers. For the path between sensor and source element, see MIPI CSI-2 image capture pipeline: sensor to host step by step.

What one copy costs

frame_MB = width × height × bytes_per_pixel ÷ 1,000,000 NV12 (4:2:0) = 1.5 bytes/pixel, BGR = 3, BGRx = 4 traffic_MB_per_s = frame_MB × fps

A v2-class sensor outputs 3280 × 2464 = 8,081,920 pixels, about 8.1 MP. Carry the units through:

  • One NV12 frame: 8,081,920 × 1.5 = 12,122,880 bytes ≈ 12.1 MB. At 30 fps that is 12.1 × 30 ≈ 364 MB/s — fine, because it stays in NVMM and the CPU is not on that path.
  • The same frame as BGR for OpenCV: 8,081,920 × 3 = 24,245,760 bytes ≈ 24.2 MB, so 24.2 × 30 ≈ 727 MB/s of CPU-side traffic at 30 fps.
  • The BGRx intermediate at full size would be 8,081,920 × 4 = 32,327,680 bytes ≈ 32.3 MB, or 970 MB/s at 30 fps.
  • The 720p BGR frames the sample loop actually reads: 1280 × 720 = 921,600 pixels × 3 ≈ 2.76 MB, about 83 MB/s at 30 fps — survivable, and the reason the sample feels warm rather than dead.

The arithmetic makes the decision: a 720p copy onto the CPU costs roughly 83 MB/s and can be tolerated on a bench; a full-resolution copy costs roughly 727 MB/s before your algorithm runs one instruction.

Capture full resolution, display 720p

Change what the source is asked for, not what the display receives:

  1. Source caps at the sensor's full mode: nvarguscamerasrc ! video/x-raw(memory:NVMM), width=(int)3280, height=(int)2464, format=(string)NV12, framerate=(fraction)30/1. The width and height must match a mode the sensor actually offers — why the mode table, not the window, sets these numbers is in sensor resolution vs video resolution scaling in cameras.
  2. Split the stream, and give each branch its own queue so display and processing run on separate threads. On the processing branch, bound the queue and drop stale frames rather than growing it; the failure mode when capture outruns processing is covered in smart camera image buffers full: match capture to processing rate.
  3. Display branch: nvvidconv flip-method=2 ! video/x-raw(memory:NVMM), width=(int)1280, height=(int)720 into the platform's hardware video sink — the sink that accepts NVMM buffers. No element on this branch names a CPU format, and videoconvert appears nowhere on it. The same full-field idea applied to ISP frames in ROS is in scale the ISP frame for full FOV previews in ROS.
  4. Processing branch, only if an algorithm needs pixels: nvvidconv ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink, with the size set to the algorithm's working resolution. The hardware converter brings frames into system memory; videoconvert finishes BGRx to BGR there. That is the one crossing and the one CPU conversion you accept.

Keep one caveat: the sensor's full-resolution mode often tops out at a lower frame rate than its binned modes. Pick the mode by frame rate first, then scale for the window — the display branch never needs more than 1280 × 720.

Three architectures compared

All-CPU sample loop Hardware scale + processing branch Full hardware display
Color conversion videoconvert, CPU nvvidconv to BGRx; videoconvert finishes BGR on CPU, processing frames only nvvidconv, hardware
Display scaling none — capped at the source nvvidconv to 1280 × 720 nvvidconv to 1280 × 720
CPU work per displayed frame copy, convert, upload none none
Full-sensor capture no yes yes
Frame-rate ceiling about 33 fps from cv2.waitKey(30) the sensor mode the sensor mode
Fits bench debugging inspection with an algorithm monitoring, HMI, recording

How to choose

  • Monitoring or HMI preview: full hardware display path, nothing else. The CPU stays idle; the decoder-side version of the same trick is low resolution preview stream: cut decoder CPU load on small displays.
  • Inspection that reads every frame: full hardware display path plus the appsink branch at the algorithm's input size. The one crossing is the algorithm's input.
  • Inspection that needs all 8 MP: either budget the roughly 727 MB/s of BGR copies at 30 fps, or keep the algorithm on the GPU side of the boundary so pixels never become CPU arrays.
  • Recording instead of a window: the display branch becomes a hardware-encoder branch. Same rule: the CPU stays off it.
  • Bench debugging a sensor: the sample loop is fine. It is a test rig, not a product.

Where the compute itself belongs — ISP, GPU or CPU — is a separate decision: embedded ISP vs GPU for raw image processing on an SoC.

DeepStream or plain GStreamer

The stock answer in most threads is DeepStream. Match the tool to the pipeline instead. DeepStream 6.0 — the version that pairs with JetPack 4.6, with pyds bindings — exists for many streams plus inference: detectors, trackers, batched metadata. Installing it to show one camera window puts weight where the pipeline has no load. Plain GStreamer already owns every element this job needs: nvarguscamerasrc for the CSI stack, nvvidconv for scaling and format on the hardware converter, and the platform's hardware sink for the window. Reach for DeepStream when the frame stops being a picture to show and becomes input to models.

Cameras for an embedded build

CSI modules get the buffer path above. USB3 and GigE board cameras land frames in system memory directly, so the same rule applies with one boundary fewer: convert and scale where the pixels already are, and hand the display its own stream. The compact modules live in board-level cameras; three that suit a small vision node:

Board-level cameras3.1 MP USB3 board-level camera, IMX252, 1/1.8" global shutter, 120 fps (LS-BCM-0003)LS-BCM-0003
Sensor format
1/1.8"
Lens mount
C-mount
$690.00View part
Board-level camerasHikrobot MV-CB016-10GM-M-W 1.6 MP GigE board-level camera, IMX296, Mono, M12 lens mount, WTB GigE portMV-CB016-10GM-M-W
Sensor format
1/2.9"
Lens mount
M12
$559.00View part
Board-level camerasGigE board camera / module, 12 MP, Color/Mono, 1.6 μm×1.6 μm pixels (LS-BCM-0056)LS-BCM-0056
Sensor format
1/2"
Lens mount
C-mount
$209.00View part
  • The LS-BCM-0003 is a 3.1 MP USB3 board-level camera on a 1/1.8" IMX252 with a global shutter at 120 fps — the pick when parts move and every frame is processed.
  • The MV-CB016-10GM-M-W is a 1.6 MP mono GigE board-level camera with an IMX296 and an M12 lens mount, for tight spaces on a network cable.
  • The LS-BCM-0056 is a 12 MP GigE board camera in a C-mount with 1.6 μm × 1.6 μm pixels and a rolling shutter, for static scenes with fine detail.

When the sensor matters more than the interface, the LS-BCM-0020 is a 6.3 MP board-level camera on a 1/1.8" IMX178, rolling shutter, 18 fps, CS-mount on GigE.

Board-level cameras6.3 MP board-level camera, IMX178, 1/1.8" rolling shutter, 18 fps (LS-BCM-0020)LS-BCM-0020
Sensor format
1/1.8"
Lens mount
CS-mount
Resolution
6-10 MP
Shutter
Rolling
$290.00View partBuild a system with it

The whole compact range, filtered by mount and interface:

CamerasBoard-level cameras

Compact cameras to build in

120 models · from $179.00

Checks before you commit

  • The source caps name one real sensor mode exactly. A mode the driver does not offer fails at link time — that is caps negotiation telling you the mode is wrong, not a bug in your code.
  • Count crossings: one NVMM-to-system boundary per processed frame, zero per displayed frame.
  • videoconvert appears only on the processing branch, at the processing size.
  • The display sink's input caps say video/x-raw(memory:NVMM), not a CPU format.
  • cv2.waitKey(30) is gone from any loop that must track the sensor's frame rate.
  • Watch package temperature with the pipeline running. Media engine, GPU compositor and memory controller loaded together is the state that throttles, and throttling announces itself as dropped frames.

When the usual answer is wrong

  • "Add videoconvert and it will link." It links by moving the format change onto a CPU core. The same effort spent on hardware-converter output caps the next element accepts keeps that pass off the CPU.
  • "Drop the capture resolution to reduce load." That reduces load and deletes the resolution you asked about. Scale after capture, in hardware.
  • "Use DeepStream, it is the GPU framework." With no inference in the pipeline it is install weight in the wrong place.
  • "imshow is GPU-accelerated anyway." The window compositor is; your pixels are not — they crossed into CPU arrays and back before the window saw them.
  • "The GPU can run my OpenCV code." The cap.read() loop hands you system-memory BGR arrays; that loop is CPU work by construction. Give the algorithm the GPU-side branch instead. And when the camera is a raw sensor with no ISP behind it, NV12 never appears — the frames arrive as Bayer, which is its own topic: CSI camera with no ISP: what raw sensor output means.

Bring the three numbers that define this pipeline — sensor mode, processing size, display size — to the camera finder or the configurator's camera step. If the interface choice is still open, ask through the contact page; more embedded capture pieces are collected on the systems topic page.

Questions

Does cv2.VideoCapture with CAP_GSTREAMER use the GPU?

It uses GStreamer for transport, but every frame returned by cap.read() is a system-memory BGR array: it has already crossed out of hardware memory and been converted on a CPU core. Hardware steps run only if the pipeline string puts them before appsink. Keep cv2.VideoCapture for the frames an algorithm actually processes, and leave the window to a hardware video sink.

Why does my captured image never exceed 1280 × 720?

Because the caps filter directly after nvarguscamerasrc asks the camera stack for 1280 × 720, and the ISP delivers exactly that. Nothing downstream can add resolution back. Set the source caps to the sensor's full mode — 3280 × 2464 for a v2-class sensor — and move 1280 × 720 onto the branch that feeds the display. Expect the full-resolution mode to offer a lower frame rate than the binned modes; choose the mode by frame rate, then scale for the window.

Is videoconvert ever the right element here?

On the processing branch, yes. Once the hardware converter has brought a frame into system memory at the working size, videoconvert finishing BGRx to BGR is a small, bounded CPU job on a branch you already own. On the display branch, never: the hardware converter with NVMM output caps does the same job with no CPU pass. If videoconvert sits between the scaler and the sink, the pipeline works, but the CPU is doing the display's work.

When is DeepStream the right call over plain GStreamer?

When the pipeline runs inference on video — detectors, classifiers, trackers, many streams batched with metadata flowing downstream. DeepStream 6.0 is the version matching JetPack 4.6, with pyds Python bindings. For one camera, one scaler and one window, plain GStreamer with nvarguscamerasrc, nvvidconv and a hardware sink does the whole job; DeepStream adds complexity the pipeline has no inference to spend it on.

How do I keep full-resolution capture from saturating the CPU?

Do not copy full frames you never process. Display from the NVMM branch scaled to the window, and let the processing branch request only the frames and the resolution the algorithm needs — a region of interest or a hardware-scaled copy. If the algorithm truly needs all 8 MP on the CPU, budget about 727 MB/s at 30 fps for the BGR copies and treat that as the algorithm's input cost, not the display's.