High CPU usage with multiple cameras on one embedded board

Eight camera streams on one embedded board, a CPU core pinned for each camera, and the visualization window updating once every few minutes: this pattern shows up on robot platforms, autonomous mobile robots and multi-camera inspection cells of every size. It almost never means the board is too small for the cameras. It means per-frame work — a raw-image viewer, a software decoder, a copy per subscriber — is consuming the cores the acquisition threads need. The driver then misses frame times and drops nearly everything. Here you get the bandwidth math to size the streams, the likely causes ranked with a test for each, and the pipeline layout that holds eight cameras at their commanded frame rate.

Why do multiple cameras pin one CPU core each on an embedded board?

Because the per-frame stages you add on top scale with frames per second, not with camera count: eight streams at 30 fps are 240 frames per second, and any stage costing 4 ms per frame consumes 0.96 of a core at that rate.

The camera itself costs little CPU to run: the sensor streams into memory by DMA, and the driver's share is an interrupt and a copy per frame. Everything after that repeats per frame, per stream. Conversion, compression, decompression, serialization for every subscriber, logging and visualization each run 240 times per second in this cell. One stage costing 10 ms per frame — a typical software JPEG decode of a 1080p image on an embedded-class core — consumes 2.4 cores at that rate. That is most of a small board's budget spent before your algorithm sees one pixel.

When those stages fill the cores, acquisition threads lose their time slices. The receive ring buffer fills while the driver waits, the oldest frames drop, and the effective rate collapses. A viewer showing one frame every few minutes is not a slow camera: it is a queue being flushed whenever the stalled stage briefly yields. The same collapse follows a busy-wait poll loop on each device — the single-camera busy-wait fix covers that pattern. A processing stage that cannot keep pace with acquisition does it too: that is the frame-rate gap case.

How much data do eight 1080p30 camera streams actually move?

About 497.7 MB/s, or 3.98 Gbit/s: at 1920 × 1080 mono8 and 30 fps, each stream carries 62.2 MB/s, so eight streams overrun a single GigE port by a wide margin and sit near 42% of a 10 GbE link.

Run the same arithmetic for your own streams: mono8 and raw Bayer cost 1 byte per pixel, RGB costs 3, and a compressed transport moves roughly a tenth of the raw rate. A GigE link delivers about 118 MB/s once protocol overhead is paid, and a 10 GbE link about 1.18 GB/s, so this load leaves headroom on one 10 GbE port under a 70% utilization ceiling. Whether the streams come from area scan cameras on GigE or from CSI-attached modules, the discipline is the one from multiple WiFi cameras on one embedded board: sum bytes per second per stream and compare against the weakest shared link.

Which stage is eating the CPU, and how do you test each one?

Work down this order: raw-image viewer, software decode, conversion and republishing, acquisition polling, disk logging, then drivers — the viewer is the most common culprit in the field, and each stage has a test that takes minutes.

Viewer. Close the visualization client completely, not minimized. If total CPU falls by two cores or more, you have found the load. Bring it back only on compressed topics — the compressed stream setup covers the topic side — and keep it out of the production launch file.

Decode. A board with a hardware encoder or decoder unit should never see JPEG or H.264 frames touch a CPU core; route each stream to the engine. Why an MJPEG viewer pins the CPU shows the decode cost in detail.

Conversion and republish. Every subscriber to a raw topic pays its own serialize-and-copy per frame, and a republisher that converts mono8 to RGB triples the bytes from 2.07 MB to 6.2 MB per frame. Publish results, not reformatted raw, and cap the algorithm's threads (the setNumThreads approach) so processing cannot take every core.

Polling. CPU at 100% with the streams stopped means a spin loop, not a camera problem; switch to callbacks or a blocking read with a timeout.

Logging. Raw at 497.7 MB/s defeats embedded storage; log compressed, or log raw per triggered event only.

Drivers. Update camera drivers and the middleware stack before any tuning: capture paths move onto hardware engines between releases, and an outdated driver can force CPU copies that the current one does not.

How expensive is a raw-image viewer, really?

A viewer subscribed to eight raw topics at 30 fps triggers 240 conversions, decodes and serialize-copies per second — roughly 2.6 cores on an embedded board before your algorithm runs a single frame.

Decode dominates. Software JPEG decode of a 1080p frame takes roughly 5–20 ms on embedded-class cores, so at 240 frames per second that is 1.2–4.8 cores of pure decode: at 10 ms per frame, 240 × 10 ms = 2,400 ms of CPU per second, or 2.4 cores. Converting mono8 to RGB for display adds about 0.8 ms per frame — 2.07 MB read plus 6.2 MB written is 8.3 MB of memory traffic, and one core sustains roughly 10 GB/s of traffic — another 0.2 core at this rate. Every further subscriber adds a serialize-copy worth about 0.1 core apiece.

The client also hides its own cost. It drops frames to keep the window alive while its cores stay pegged, so a picture that looks almost live and 100% CPU arrive together — while the starved driver's ring buffer overflows and the real stream decays toward one frame every few minutes. The pattern is not unique to vision: embedded HMI view clients pin cores the same way, and the embedded-view CPU fix walks the same diagnosis for dashboard clients.

What architecture keeps eight cameras at full frame rate?

One owner per camera device, exactly one consumer on each raw topic, compressed transport for everything a human watches, and bounded queues that drop old display frames instead of backing up — that layout holds eight streams at the commanded 30 fps.

1. One owner per device. A single acquisition service opens each camera and publishes raw topics with exactly one consumer: your algorithm. The multi-camera session design shows the shape, and the one-camera-per-process rule prevents two processes fighting over one device.
2. Humans watch compressed. Viewer, logger and quality station subscribe to compressed or downsampled topics at a reduced frame rate, never to raw.
3. Decode on engines. Hardware encoder and decoder units take JPEG and H.264 work off the CPU; CPU decode remains only for low-rate previews.
4. Bound every queue. Drop-oldest on display topics; on algorithm topics, block briefly or drop with a counter. A queue that grows without limit becomes the memory failure below.
5. Prioritize acquisition. Raise the capture threads above processing and pin them to cores away from network interrupt handling.
6. Process at the decision rate. If the cell decides at 5 fps, run the algorithm at 5 fps and keep the full 30 fps for triggered events and re-inspection.

What CPU, network and storage budget should you size for?

Budget acquisition plus one processing path at under half the CPU cores, keep link utilization under 70%, and never log raw at the aggregate rate — eight 1080p30 mono8 cameras produce 497.7 MB/s of sustained writes, past what embedded storage holds.

Keep the total under half the cores so acquisition, interrupts and the operating system keep headroom. On an eight-core board, 0.96 cores of algorithm plus one core of capture plus 0.4 cores of compressed previews leaves about five cores free. That margin absorbs bursts without dropping frames.

Storage follows the same rule: 497.7 MB/s of raw writes is past what embedded flash sustains, so log compressed — about 50 MB/s at a typical 10:1 ratio — or log raw per triggered event. The high-rate disk streaming guide covers the buffering this needs.

Queue memory is the third budget. Two seconds of buffering at this aggregate rate is roughly 1 GB of RAM, which is exactly where the per-frame memory leak starts on multi-camera boards.

When that math says the board cannot carry the cell, the hardware built for it is the fanless vision controller: several network ports, PoE camera ports and built-in light channels in one box, with the camera finder matching cameras to the same budget:

The MV-VC5525-256G70-NN carries 5+ LAN ports, 3-4 USB 3 ports, PoE camera ports, 8 built-in light channels and 32 GB of memory on an Intel Core i5. Eight camera links on eight of those ports leave spare networks for PLC traffic and results.

For a capture box that skips PoE and carries a lighter algorithm load, the same family's entry build runs on a Core i3:

The MV-VC3301H-128G66 pairs the Intel Core i3 with 8 GB of memory, 5+ LAN ports, 3-4 USB 3 ports and 4 built-in light channels.

For PoE-powered cameras and a heavier inference or recording load, the PoE build adds an Ultra-class processor and PCIe expansion:

The MV-VC4712-256G10-2TH provides 3-4 LAN ports, 2 USB 3 ports, PoE camera ports, 6 built-in light channels and PCIe expansion.

When is the camera hardware itself the bottleneck?

Only when the driver's own received-frame counter lags the commanded rate with the viewer closed — the causes that remain are shared controller bandwidth, exhausted CSI lanes or thermal throttling, not your software.

USB bandwidth divides per controller, not per port: a 5 Gbps controller moves about 480 MB/s usable, close to seven of the 62.2 MB/s streams above, so keep several cameras on separate controllers rather than one root. A GigE switch that funnels eight ports into one 1 GbE uplink throttles at the uplink, not at the ports — this aggregate needs a 10 GbE uplink. CSI-attached cameras need dedicated lane controllers from the board vendor, and if the kernel driver reports no frames arriving at all, no CPU tuning will help.

Sustained load also tells you what a spec sheet hides: passively cooled boards commonly shed 20–40% of CPU clock during a 30-minute all-core run, so read the clock counters under load. Depth streams obey the same rules — 3D sensors add frames and bytes per stream just as the 2D ones do — and one saturated core from an unrelated service can stall them all; the industrial PC single-core fix is the same class of problem on a cabinet-mounted PC.

How do you verify the fix held?

Log three counters from frame header timestamps across a 30-minute run — frames acquired, frames delivered to the algorithm, frames displayed — and require each within 1 fps of the commanded 30 fps with a flat latency curve.

Record receive time minus header timestamp for every frame at all three points. A flat curve at the commanded rate means the pipeline holds; a rising curve means a stage is backing up; a driver counter below commanded means look at link bandwidth, controller sharing or cooling instead of software. The counter-by-counter method is laid out in ROS 2 image publishing frame rate: find the bottleneck.

Decision, then. With the viewer on compressed previews and decode off the CPU, eight 1080p30 streams hold 30 fps on an eight-core board with fewer than three cores busy. If the driver counter stays low with the viewer closed, fix the link, the controller or the cooling — no software change will. If the aggregate exceeds what the network carries, split the ports, and size that combination in the configurator below; the systems topic page collects the rest of the diagnosis patterns.