Bad frames under heavy multi-camera use: plan host bandwidth

A rig can look healthy at start-up and still fall apart under load: every camera enumerates, every viewer shows a live image, then minutes into full-rate running a handful of frames come back torn, short or stale. Those occasional bad frames are rarely a sensor fault. They are the host reporting that it could not hand off a complete buffer in time. This page shows how to compute the rate your streams actually move, where that rate breaks first on a typical PC, and which settings and hardware give every camera a clean path into memory. The worked example follows a common pair — a 1920×1080 RGB8 area scan camera at 15 fps and a 640×480 16-bit depth sensor at 30 fps — and ends with a per-link budget, a ranked test order and a fix that holds.

Why do frames turn bad only when every camera streams at once?

Frames do not go bad at random: at 1920×1080 RGB8 and 15 fps plus 640×480 16-bit depth at 30 fps, your host must accept about 112 MB/s without a pause, and any buffer handoff that misses that pace tears a frame or drops it outright.

Bad frames have a small set of faces. A torn frame shows the top half of one image and the bottom half of the next. A short frame carries fewer bytes than the pixel format promises. A stale frame repeats the previous buffer, and sometimes a frame arrives all black. Host-bandwidth failures come in bursts: three or four bad frames when the disk, the GUI or a processing spike wakes up, then minutes of clean running. A cable or connector fault behaves differently — steady bad frames at any rate — and that contrast is your first diagnostic.

Checking and discarding protects the inspection step, and you should keep it permanently. It is a symptom mask, not a fix: the lost frames never come back.

How much bandwidth does this two-camera rig actually need?

Add per-frame size times frame rate for every stream: this pair needs 93.3 MB/s for the colour camera and 18.4 MB/s for the depth camera, about 112 MB/s or roughly 0.9 Gbit/s of payload delivered continuously.

One colour frame is 1920 × 1080 × 3 = 6,220,800 bytes, about 6.2 MB. The host path must never stall longer than one frame period — 66.7 ms for the colour camera, 33.3 ms for the depth camera — or a frame is lost. At the 111.7 MB/s aggregate that is a tolerance of roughly 55 ms per 6.2 MB. Protocol overhead rides on top of the payload: 1,500-byte packets add about 5% on a GigE Vision stream, so the colour camera alone puts about 98 MB/s on a 125 MB/s line.

Hold every link at or under 80% of its line rate; the margin absorbs overhead, retries and scheduling gaps. On those numbers the colour stream fits a GigE port only with the port to itself, the two-camera total fits a USB 3.0 controller's budget on paper while still sharing one queue, and a 10GigE port has room for the whole inspection plus growth. For larger rigs, the multi-camera GigE bandwidth budget method applies the same division link by link.

What is failing first: the link, the controller or your buffer code?

Rank the suspects: two cameras on one USB controller is the most common cause, a saturated shared GigE link is second, a buffer pool that is too small or freed too late is third, competing disk and CPU load is fourth, and a cable or connector fault comes last when bad frames appear at any rate.

USB hosts are the usual first suspect because desktop boards wire many ports to one or two root controllers — the port count on the chassis is not the controller count. Fixing USB bandwidth stalls on a depth camera starts at that same topology view. On the GigE side, isolating camera and host timing from frame arrival jitter tells you whether the device or the host is late before you touch hardware. And per-frame allocation in the receive loop is cause 3 in disguise: allocation jitter lands directly in the copy path.

How do you prove which part of the host is dropping the data?

Run three controlled trials and the counters tell you where the loss lives: one camera alone, both cameras with processing paused, and both cameras with processing running.

1. Colour camera alone at 15 fps for ten minutes: expect 150 frames, zero bad. Bad frames here point at cable, port or power, not multi-camera load — isolate intermittent corruption by swapping one element at a time.
2. Both cameras, acquisition only, no processing and no disk writes. Bad frames continuing means the loss is upstream of your code, causes 1 to 3. Bad frames vanishing means your processing loop starves the buffers, cause 4.
3. Validate every arriving frame: exactly 6,220,800 bytes for colour and 614,400 for depth, frame ID advanced by exactly one, timestamp deltas of 66.7 ms and 33.3 ms. Anything else is discarded and requeued — discard, requeue, grab a new one rather than feeding a torn image downstream.
4. Move the depth camera to a different controller or port. Discards stopping confirms the sharing, and the hardware split in the section below is the fix.

Count complete frames over ten-second windows while everything runs: 150 colour frames and 300 depth frames are the targets. Delivered below configured means drops upstream of your code; delivered equal with inspections still failing means torn frames are passing your validator, so tighten the byte-count check.

Which settings keep every link inside its budget?

Set the largest packet the link allows, give the capture API a buffer pool sized for your worst-case loop, and cap each link at 80% of its line rate: for the colour stream this means 93.3 MB/s of payload against the 100 MB/s GigE budget, which jumbo frames make comfortable.

• Packet size. Set the largest the device offers: the transfer packet size in a USB3 camera's stream configuration, or a 9,000-byte frame size on a GigE camera, its NIC and any switch in between.
• Host USB buffers. Raise the host-side buffer before anything else — the two-USB3-camera host buffer fix is exactly this step for a rig of this size.
• Watch the sag. A frame rate that droops under load is the same disease announcing itself early: exposure, USB link and bandwidth interact before frames visibly tear.
• Stream size. Exposure time does not change bandwidth; frame rate and ROI do. Halving depth to 15 fps halves its share to 9.2 MB/s. Full-frame RGB8 at 30 fps would cost 186.6 MB/s, beyond the GigE budget, so a 1280×720 ROI at 30 fps — 82.9 MB/s — is the version that fits with margin.

The buffer pool is the setting engineers get wrong most often, because the average loop time flatters it. Size from the worst case:

The 150 ms loop time is an example input — measure your own worst case with timestamps around the processing pass and recompute. Sixteen colour buffers hold about 100 MB, which is nothing on a modern host.

Which hardware gives each camera its own link?

Give every camera a controller or port of its own: a dual-port GigE NIC separates two colour cameras, a quad-port card separates four, and a quad USB3 adaptor does the same job when the cameras are USB3.

Desktop boards often expose eight ports behind one or two root controllers, so confirm the topology before counting connectors. A PCIe network card ends the sharing: each port is an independent link with its own queue and interrupt, and PoE models carry camera power over the same cable.

The MV-GE2002P dual-port GigE PoE NIC gives two cameras two independent links; the MV-GE1104 four-port GigE frame grabber and the MV-GT2104 quad-port copper 10GbE NIC scale the same split to four cameras. For USB3 cameras, the MV-GU2204 quad USB3.0 adaptor is the equivalent part: four cameras, four ports, one card.

Past about six high-rate cameras, or with CoaXPress sensors, frame grabbers add onboard sequencing and deterministic transfers into host memory, which is why the category sits beside plain network cards. The card below shows the range, and the configurator walks accessory selection once the cameras are chosen.

When is more bandwidth the wrong fix?

More bandwidth fixes saturation only: when one camera alone needs more than a link's 80% budget, or the loss follows processing spikes rather than traffic, adding ports changes nothing and the fix is a smaller stream, a faster interface or better buffer management.

A single stream can outrun any port. Full-frame RGB8 at 3840 × 2160 and 60 fps needs 3840 × 2160 × 3 × 60 ≈ 1.49 GB/s — about 11.9 Gbit/s, past a 10GigE port's 1,000 MB/s planning budget. Extra ports cannot rescue one stream that big: halve the frame rate to 747 MB/s, shrink the ROI, bin the sensor, or move to a camera with a faster interface class such as CoaXPress.

Loss that tracks processing spikes lives in the copy path, not the wire. Pool size, buffer reuse and thread priority come first, and trial 2 from the proving section confirms the fix.

Bad frames at five frames per second with one camera alone are a cable, connector or power problem: replace the cable, check the PoE power budget, and route cable runs away from motor cables.

What do you check before ordering parts?

Order parts only after the stream table adds up: list each camera's width, height, bytes per pixel and frame rate, compute MB/s, add the total per controller, and confirm every controller stays under 80% of its line rate.

• Measure first. The USB topology view tells you how many controllers you really have; utilization counters and delivered-versus-configured frame counts over ten-minute windows tell you how the links behave under load.
• Compute before you buy. The stream table, the per-controller total, and each controller's share against its 80% planning budget with at least 20% headroom left.
• Set before you retest. Maximum packet size on camera, NIC and switch; 8-16 buffers per camera; validation with discard-and-requeue armed with a counter.
• Soak before you trust it. Thirty minutes at production frame rates with processing on; the discard counter must stay at zero after the first seconds.
• Buy with spare capacity. One port per camera plus two spare ports. For the example rig that is one dual-port GigE card and one USB3 controller carrying nothing else.

If the table says the rig fits, order it and soak it before it reaches production. To have the budget checked against specific parts, send the stream table through the contact page, or browse area scan cameras and 3D sensors for the streams themselves. More host-side failure patterns sit on the systems topic page.