Multiple GigE cameras, same trigger: why frames arrive incomplete

Several GigE cameras on one hardware trigger is a standard single-shot layout: the object passes once, every camera exposes at the same instant, and the software gets one chance at each frame. The failure is quiet. The switch reports no errors and the links sit far below their limits, yet frames still arrive incomplete, because the receiver asked for resends until its limit ran out. The exposure is not the problem; the transmission is. This guide sizes one 12-bit frame, shows why simultaneous bursts bury a virtualized receiver, and gives per-camera Frame Transmission Delay values that land eight frames inside a 10 s cycle.

Why do frames go incomplete when the cameras share one trigger?

Because every stream reaches the receiving host in the same few milliseconds: eight cameras transmitting at once put close to 98,000 packets per second onto one receiver, and a 200 ms receive stall inside that window leaves gaps the resend mechanism cannot close.

Alone, each camera is harmless: one gigabit port, one 9.4 MB frame, finished in under a tenth of a second. The failure arrives with the shared trigger, because simultaneous exposures mean simultaneous transmissions. The receiver watches the GVSP packet sequence numbers, sees missing IDs, and asks each camera to resend. Once a frame misses its delivery timeout, or its missing-packet count passes the receiver's resend limit, the SDK hands you that frame incomplete. The switch stays clean because nothing was lost on its ports; the loss is at the host edge. The signature matches GigE Vision stream packet loss: diagnose blank frames, with one difference: here you built the burst yourself.

How much wire time does one 12-bit frame need?

A 12-bit packed 6 MP frame is about 9.4 MB, or 75.5 Mbit, and occupies a gigabit port for roughly 86 ms at a 9 KB packet size — this is the number every delay below is built on.

Work the example: a 3,072 × 2,048 sensor in a 12-bit packed format stores 1.5 bytes per pixel, so 3,072 × 2,048 × 1.5 = 9,437,184 bytes, about 9.4 MB or 75.5 Mbit. A gigabit link with 9 KB packets carries roughly 110 MB/s of payload, a typical figure, so 9.4 ÷ 110 ≈ 0.086 s: 86 ms of wire time. That frame travels as 9,437,184 ÷ 9,000 ≈ 1,049 packets. If your pixel format pads each pixel to 16 bits, double the bytes: 12.6 MB and 114 ms per frame.

A 6 MP GigE camera fits this example's size class:

What Frame Transmission Delay should each camera get?

Set it in steps of about one and a half times the single-frame transfer time: with an 86 ms transfer that is 150 ms per camera — 0, 150, 300, 450, 600, 750, 900 and 1,050 ms.

Frame Transmission Delay holds a captured frame in the camera and postpones only its stream, so the exposures stay simultaneous while the packets leave one camera at a time. You find it in the camera's GigE Vision transport settings, beside packet size and inter-packet delay; the exact feature name follows the camera's own feature list. Many cameras state the delay in timestamp ticks from a 1 GHz clock, a typical value, in which case 150 ms is 150,000,000 ticks — read the tick frequency and convert.

For eight cameras: 7 × 150 ms + 86 ms = 1,136 ms, so the last stream finishes about 1.1 s after the trigger. Allow 0.5 s for resends and the whole set is complete inside 1.6 s, with 8.4 s of the 10 s cycle still spare. Four cameras need only 0, 150, 300 and 450 ms: 3 × 150 + 86 = 536 ms, about half a second plus resend margin.

On the receiving side, set the frame timeout in the SDK's transport settings to about half the cycle — 5 s here — so a stalled frame completes from resends instead of being declared late. The grab loop has to tolerate that wait; Frame didn't arrive within timeout: one wait call per frame shows the structure that does.

Why did throttling each camera to 10 Mb/s not help?

Because 10 Mb/s stretches one 75.5 Mbit frame to 7.55 s: eight parallel streams then finish barely inside the cycle, and one receive stall with its resends pushes them past the next object.

The arithmetic: 75.5 Mbit ÷ 10 Mb/s = 7.55 s per frame. Throttling changes how fast each stream runs, not when it starts. All eight still opened together, so the receiver still held eight half-built frames when its stall hit, and the slow rate removed the headroom the resends needed. The rule generalizes: a rate limit treats congestion on a shared link; it does not separate senders in time. Leave the cameras at full port rate and stagger the starts instead.

What do Inter-Packet Delay and Bandwidth Reserve actually change?

Inter-Packet Delay spaces the packets inside one frame and so sets its average rate; Bandwidth Reserve holds link headroom for resend traffic — neither moves one camera's start relative to another's, which is why neither cures this failure.

Inter-packet delay is a rate control. Stretching the 9.4 MB frame across a full second averages 75.5 Mb/s: 1,049 packets in 1 s is one packet every 953 µs, of which 72 µs is wire time, so the gap is about 880 µs. That is the tool for several cameras streaming continuously through one shared link, where spreading each stream over time lowers the short-term load — the case walked through in GigE camera network planning: switch ports and bandwidth. Bandwidth Reserve belongs to that same continuous case: it keeps part of the allocation free so resend packets fit. In a staggered single-shot system, leave both alone; separation in time does the work.

Can the switch be the cause when it reports no errors?

Rarely in this layout: eight gigabit camera ports peak near 7 Gb/s into a 10 Gb/s uplink — about 70 percent for 90 ms — so the switch buffers comfortably and the loss happens downstream of it.

A switch becomes the suspect when its camera ports sum to more than its uplink. One field-documented pattern: three GigE cameras through a 100 Mbit switch. The switch could not drain, so it sent IEEE 802.3x PAUSE frames, and a PAUSE freezes the whole link, stacking delay on every camera behind it. Two checks settle it: add the camera port rates and compare with the uplink, then read the switch's PAUSE and discard counters. Eight gigabit ports behind a 10 Gb/s uplink passes both; a 1 Gb/s uplink would fail the first, and no per-camera setting would save it.

Why does a virtual machine drop bursts a physical host keeps?

Because the guest's receive path can go quiet for tens to hundreds of milliseconds while the hypervisor runs other work, and paravirtualized NIC drivers such as VMXNET3 are documented to drop packets precisely during very high traffic bursts.

Your capture — the stream silent for about 200 ms, then a clump of resend requests — is that stall seen from the wire. The switch delivered the packets; the guest did not drain them. Three settings keep a VM in the game. Raise Receive Buffers on the physical NIC: at 9,000 bytes per packet, 4,096 buffers hold about 36.9 MB, roughly 335 ms of one gigabit stream, enough to ride out the stall once staggering means only one stream is live at a time. Keep 9 KB packets negotiated end to end, including the virtual switch's MTU, as GigE packet size exceeds MTU: cap auto-negotiation and Set GigE camera packet size before starting acquisition describe. And pin the vCPU that handles the stream interrupts, so the guest is not descheduled mid-burst.

The committed decision: up to four staggered cameras on a VM is workable once buffers and MTU are set; eight simultaneous single-shot cameras belong on a physical multi-port NIC with the grab process on bare metal. On the host side, group the cameras so one trigger's set lands on separate ports, as Assign cameras to ports reliably in a multi-camera system explains, and re-check device order after any card swap, per Multiple frame grabber device order: fix swapped camera channels.

Host-side cards that suit this job:

Values as printed in each product's data sheet; open a part for its drawing and full specifications.

What do you check before changing hardware?

Run this list before buying anything: in most eight-camera systems the delays alone take the incomplete-frame count to zero, and every item here is a setting rather than a part.

• Frame Transmission Delay per camera — 0 through 1,050 ms in 150 ms steps, recorded next to each camera's serial number so a swapped cable cannot hide a missing delay.
• Receiver frame timeout — about half the cycle, 5 s here, so a stalled frame completes from resends.
• Packet size end to end — 9 KB at the camera, on the physical NIC and in the virtual switch's MTU; a mismatch fails the jumbo negotiation and the frame falls back to six times the packet count.
• Physical NIC Receive Buffers — 4,096 as a worked target; re-check after driver updates, which reset advanced settings.
• Port grouping — the cameras sharing one trigger spread across physical ports or cards.
• Counters — the per-camera incomplete-frame counter, logged across a full shift; the target is zero.
• Cabling — a connector that works loose under vibration loses frames on exactly one camera, and locking connectors end that class of failure. High-flex camera data cables with screw locks for anything that moves:

If the system grows past eight cameras, or the pixel format changes to 16 bits, redo the arithmetic above before touching hardware — the step and the timeout both scale with transfer time. Start from GigE frame grabbers and GigE area scan cameras, filter in the accessories finder, run the system configurator, or send the camera count, resolution and pixel format through the contact page and the team will size the delays with you.