RTSP decode CPU overload: offload decode and conversion to the GPU

Ten IP cameras at 1080p can pin a desktop CPU at 100% while the GPU sits idle. Every stage runs in software: decodebin picks a software decoder, videoconvert turns the decoder's YUV output into BGR pixel by pixel, and videoscale resizes on the CPU again. Teams pulling RTSP streams into Python for analytics hit this wall first as late frames and hot cores. The fix moves decode, scaling and color conversion onto the GPU with an explicitly named element chain, and one single-stream measurement tells you how many streams a host carries.

The symptom: a pegged CPU, an idle GPU, late frames

The signature is consistent. Around ten 1080p streams saturate the CPU completely, cap.read() starts blocking, frames fall behind, and GPU utilization reads near zero. Profiling the default chain — rtspsrc into decodebin, then videoconvert and videoscale ahead of appsink — points at the decode and format stages, which is exactly where a published multi-stream case landed: a 10th-generation Core i7 at 100% CPU for ten streams, with decodebin and appsink named as the heavy elements.

Run one stream alone first and read two numbers: total CPU and GPU utilization. GPU near zero while CPU climbs means decode is running in software. That single measurement also gives the per-stream cost every later decision uses. Measure it the way the pipeline will actually run, because CPU time and wall-clock time are not the same number.

Why the default pipeline saturates the CPU

decodebin builds the chain for you, and that is also its failure mode. With no hardware decoder plug-in installed, it silently selects the software decoder; nothing in the log announces the fallback. After an OS or driver update removes the plug-in set, a pipeline that ran on the GPU quietly returns to full software load.

The second half is the caps. Asking appsink for video/x-raw in BGR forces a full-frame format conversion and a copy into system memory on the CPU, at three bytes per pixel:

That 155 MB/s per stream is conversion output alone; the per-pixel conversion work and the decode itself sit on top of it. This is why ten streams, not sixty, exhaust a desktop CPU — and why adding cores buys less than moving the work to the GPU.

Likely causes, in order of probability

Work down the table in order. The first two causes account for most of the load in a fresh install; the rest matter once decode and conversion are off the CPU.

The fix: name the decoder, convert on the GPU

The slow baseline, as measured in the reference case:

rtspsrc location=<uri> latency=0 ! decodebin ! videoconvert ! video/x-raw,format=BGR ! videoscale ! video/x-raw,width=<W>,height=<H> ! appsink

The chain that resolved it:

rtspsrc location=<uri> latency=0 ! queue ! rtph264depay ! h264parse ! nvv4l2decoder ! nvvideoconvert ! video/x-raw,width=<W>,height=<H>,format=RGB ! appsink

Element by element: rtph264depay unwraps RTP into H.264 access units, h264parse hands the decoder clean, framed data, and nvv4l2decoder runs the GPU's dedicated decode engine (NVDEC), which handles H.264 and HEVC. nvvideoconvert then scales and converts on the GPU — width, height and format live in its output caps, so the resize never touches the CPU. The caps still say video/x-raw, so each frame copies once from GPU memory to system RAM for appsink; that copy is far cheaper than decode plus conversion, and the tensor route above removes it when the model runs on the same GPU. latency=0 holds the RTSP jitterbuffer at its floor for LAN deployments; on longer links, raise it rather than dropping damaged frames.

Hand the whole string to cv2.VideoCapture when your OpenCV build includes GStreamer support, then read with cap.read(); the frame arrives as a numpy array of height × width × 3. Before wiring Python in, run the same chain from gst-launch-1.0: a 'no element' error there means the plug-in set is missing, not your code. If a buffer still arrives with the wrong layout — the reference case saw a two-tuple grayscale frame until the caps were right — fix the format negotiation instead of reshaping in Python.

Naming the elements explicitly, rather than trusting decodebin, is what keeps the pipeline on the GPU after updates. The path from these frames into a running model is traced in GPU accelerated vision streaming: trace the video pipeline.

Worked example: 30 1080p streams on a quad-core host

Inputs, with sources. Cameras: 1080p H.264 at 25 fps, the reference case's stream type. Software baseline, measured in the reference case: ten streams at 100% CPU on a 10th-generation Core i7, so one stream costs a tenth of that CPU. Hardware path, measured in a published single-stream run: one IP-camera feed at roughly 20 fps cost 7-15% CPU through nvv4l2decoder and nvvideoconvert to appsink, and decode alone sits near 1% CPU for 4K on Jetson-class hardware. Planning rule: keep total CPU at or below 75%, leaving a quarter of the host for the model, the OS and disk.

Rule in words: streams per host = cores × 100% × planning load ÷ per-stream CPU. Every input is a percentage point of the same host, so the units cancel.

Result: one quad-core host with decode and conversion on the GPU carries 20-30 1080p streams at 25 fps — 30 at the measured midpoint, 20 if your single-stream number lands at the top of the band. The decision threshold on the software path is about ten streams per desktop-class CPU; beyond it the arithmetic demands more than one machine, and GPU offload buys roughly three times the stream count on the same class of hardware. nvvideoconvert comes only with the DeepStream plug-in set, so the host needs that installed alongside the driver.

Sizing the host and the network around the stream count

The host needs cores for the model and OS, a PCIe slot for the decode GPU, and camera-side network ports. Choosing the CPU-GPU split for the model itself is its own exercise — machine vision deep learning setup: choosing CPU or GPU — but the decode leg belongs on the GPU once you pass about ten streams. Our vision controllers are built as vision hosts, with camera-side LAN ports, light outputs and PCIe expansion on the models below:

The MV-VC3301H-128G66 pairs an i3-8100 with 8 GB of memory and a 128 GB SSD; the MV-VC5525-256G70-NN steps up to a 12th-generation Core i5 with 32 GB of memory, a 256 GB SSD and PoE camera ports. Both take PCIe cards — the slot that carries the decode GPU.

Bandwidth rarely binds at 1080p. A typical H.264 main stream runs 2-8 Mb/s, so thirty cameras at 4 Mb/s make 120 Mb/s, well inside one gigabit link. At 4K, 8-25 Mb/s per camera is typical, and thirty cameras at 25 Mb/s reach 750 Mb/s — close to the roughly 940 Mb/s a gigabit link delivers in practice. Sixty 4K cameras at 25 Mb/s each total 1.5 Gb/s, past one gigabit link entirely. PoE matters here too: the same ports that carry the streams power the cameras and keep their traffic off the office LAN.

A dual-port PoE NIC on GigE covers the 1080p split and powers the cameras:

A quad-port copper 10GbE NIC carries the higher counts with headroom:

Prevention: keep the decode off the CPU

• Name nvv4l2decoder in the chain and keep decodebin out of production pipelines. A missing plug-in then fails loudly at startup with a 'no element' error instead of silently doubling your CPU bill.
• Keep width, height and format in the converter's output caps. A videoscale after the GPU converter re-introduces per-pixel CPU work on every frame.
• Keep frames in GPU memory when the model runs there; when appsink must land frames in system RAM, reuse buffers across frames instead of allocating each one.
• Treat appsink drop=true as a symptom, not a fix — it raises CPU load regardless of the buffer count. If consumption cannot keep up, cut the converter's output size instead: a 640 × 360 BGR frame is 1,866,240 ÷ 2,073,600... more precisely 640 × 360 × 3 = 691,200 bytes, about a ninth of the 1080p frame's 6.2 MB.
• Re-measure the single-stream cost whenever resolution, frame rate, codec or camera count changes; the 30-stream result above moves with that number.
• Check the GPU side as well: decode sessions per engine are finite, and the DeepStream RTSP scaling guide to NVDEC capacity and GPU sizing maps camera counts against one GPU before you order it.

If the model itself, not the decode, becomes the bottleneck once offload is done, that gap is its own diagnosis — see Acquisition faster than processing: fix the frame rate gap.

Next step: write down camera count, resolution, codec and frame rate, measure one stream through the hardware chain, and size the host from the formula above. Compare the frame grabbers pages, start at the system configurator, or contact us with your stream list and we will size the decode budget with you. More acquisition-side fixes are collected on the systems topic page.