Fetch buffer returns empty components: fix GigE stability

Random stalls at maximum frame rate, then buffers that arrive with empty components, look like a software fault. They almost never are. When the frame-time column jumps from 18 ms to 2,000 ms and the next buffers carry no image data, your GenTL producer is telling you it could not assemble complete frames: packets were lost somewhere on the link. This guide is for engineers streaming from a GigE camera through copper, media converters and fiber at full frame rate. You will read the symptom, compute how much headroom the link really has, measure where the loss happens, and apply the fixes in the order that holds: packet size against path MTU, receive buffers, a dedicated port, forced link speed.

The two anomalies are one fault

A fetch_buffer() call that returns a buffer whose payload.components list is empty, and a fetch_buffer() call that blocks for hundreds to thousands of milliseconds, are one event seen from two sides. The producer behind the camera assembles each frame from UDP packets and hands a buffer over only when the frame is complete, or when it has given up waiting. Packets the camera's resend mechanism cannot recover leave a hole the producer cannot fill, so the buffer arrives with no components; while it waits, your fetch blocks. One slow frame of 200–4,000 ms, then a burst of empties, then minutes of normal capture: that sequence is the fingerprint of link-level packet loss under load, not of a race condition in your loop.

Count both outcomes where you already log frames:

frames = 0        # complete frames delivered
empties = 0       # buffers with no components

with ia.fetch_buffer() as buffer:
    if not buffer.payload.components:
        empties += 1
    else:
        frames += 1

Any non-zero empty rate during steady capture is a fault to chase, not noise to tolerate. With resend switched off the same loss shows up earlier and visibly, as horizontal line artifacts in live images; with resend on, recovery hides the loss until it overwhelms the resend window and the empties begin.

Likely causes, ranked

Work from the top; each test takes minutes.

1. Loss on the conversion path. Two media converters and an SFP pair each add a copper PHY and a buffer stage, and a marginal transceiver drops bursts under sustained load. A few kilometres of single-mode fiber is within what 1000BASE-LX optics are rated for, so suspect the hops, not the reach. Test: run the camera straight into the host on a short copper lead.
2. Receive ring too small on the host. At the ~77,500 packets/s a 2 MP stream generates (worked example below), a shallow driver ring or a port shared with other traffic drops receive descriptors. Test: the interface's own drop counters.
3. Packet size larger than a hop carries. Any device between camera and host that caps frame size silently discards every oversized packet. Test: a do-not-fragment probe at each candidate size.
4. Link renegotiating mid-run. Auto-negotiation across converters can flap, and episodes that start and stop over minutes match that signature. Test: force 1000 Mbit/s full duplex and watch the system log for link state changes.
5. The application. Too few buffers, a fetch loop starved by processing, or a blocked GUI produce the same symptom without the stall-first pattern. Test: empties at a steady cadence while loss counters read zero.

Why it shows up only at maximum frame rate

Below maximum frame rate the link has spare bandwidth, so packet resend repairs nearly every loss before you notice. At maximum frame rate the resend traffic competes with new image data for the same area scan camera stream; recovery falls behind, incomplete frames reach the producer, and the buffer arrives empty. GigE Vision resend can only recover packets the camera still holds, so once loss outruns that window the frame is past saving. This is also why dropping the frame rate makes the problem vanish and misleads you toward the code. The producer's resend statistic is the direct measurement; reading the resend ratio and its buffer timeouts shows what a healthy value looks like.

Worked example: a 55 fps stream stalls for 1.9 s

Inputs, with their sources: frame rate 55 fps (the acquisition target); steady frame times 18.01–18.21 ms and stalls of 238.41 ms then 1936.68 ms (the capture log); a 1 Gbit/s copper link end to end; camera payload 1920 × 1080 px at 1 byte per pixel, mono 8-bit — an example camera, so substitute your own width, height and bytes per pixel (12-bit packed counts about 1.5). MB and Mbit below are decimal.

Rule in words: compare what the camera must send with what the link can carry after packet overhead, and measure a stall in frame periods.

• Frame period: 1000 ÷ 55 = 18.18 ms; the log's steady 18.01–18.21 ms says the pipeline ran at target until the fault.
• Frames missed in one stall: 1936.68 ms ÷ 18.18 ms ≈ 106.5 — about 107 frame periods of packets lost or waiting.
• Payload rate: 1920 × 1080 × 1 B × 55 /s = 114,048,000 B/s ≈ 114 MB/s ≈ 912 Mbit/s.
• Capacity at MTU 1500: image data per packet = 1500 − 20 (IP) − 8 (UDP) = 1472 B; wire cost per packet = 8 (preamble) + 14 (Ethernet header) + 1500 + 4 (FCS) + 12 (inter-frame gap) = 1538 B; efficiency = 1472 ÷ 1538 = 95.7%; capacity = 125 MB/s × 0.957 ≈ 120 MB/s ≈ 957 Mbit/s.
• Headroom: 957 − 912 ≈ 45 Mbit/s, or 4.7% of capacity, at a packet rate of 114,048,000 ÷ 1472 ≈ 77,500 packets/s.
• The same stream at MTU 9000: 8972 B of data per packet, 9038 B on the wire, 99.3% efficiency, capacity ≈ 993 Mbit/s, headroom ≈ 80 Mbit/s, packet rate ≈ 12,700 packets/s — about six times fewer interrupts and nearly twice the margin.

Decision: keep computed utilization under about 90% for sustained max-fps streaming, because resends must share the line with new image data. This stream runs at 95% with 45 Mbit/s of slack — one converter hiccup and the resend traffic itself becomes the next loss, which is the stall-then-empties pattern exactly. Fix capacity before touching the loop: jumbo frames end to end, then a dedicated port. If your numbers land at or above 100%, no camera setting holds the frame rate; cut the payload with binning, ROI or a lower frame rate, or change interface — the camera finder filters by interface and resolution.

Diagnose before you change anything

1. Instrument the loop. Log frame count, empty count and fetch duration with timestamps. Empties in bursts after 200–4,000 ms stalls put the fault below the code; a steady empty rate with clean counters puts it in the buffer pool.
2. Read the producer's loss counters. Most producers expose a packet-resend or lost-packet statistic in their node map. A count that climbs during the episodes is direct evidence of link loss.
3. Check the host's drop counters. On Linux, ethtool -S lists per-driver receive statistics (look for missed or dropped columns) and ethtool -g shows the ring limits; Windows exposes discarded-packet counters on the adapter. The overflow signature is common across stacks — this GigE camera buffer-overflow walkthrough lists the counters worth checking — and when frames arrive incomplete with clean loss counters, a driver and NIC MTU mismatch is the usual cause.
4. Probe the path MTU. Send do-not-fragment probes at each candidate size (ping -M do on Linux, ping -f -l on Windows) and find the largest size every hop answers. Oversized frames disappear silently at the hop that cannot carry them.
5. A/B the path. Thirty clean minutes on a direct copper lead, then thirty minutes through the converters with a saturating traffic generator such as iperf3 on the same segment. Episodes here run for minutes, so shorter tests prove nothing.

Fixes, in the order that works

1. Set the packet size to what the whole path carries, before acquisition starts. Set the camera's packet-size control just under the smallest hop's MTU; setting the packet size first walks through the sequence, and if the negotiated size exceeds what the network interface accepts, capping the auto-negotiated packet size is the fix. Confirm with the camera's test-packet mechanism; a test packet that never arrives means a hop still refuses the frame size.

2. Raise the receive ring and give the stream its own port. Set the NIC ring to its maximum (ethtool -g shows the range, ethtool -G sets it) and keep other traffic off the port. On the hardware side, the MV-GE2002P is a PoE NIC with 2 ports.

For several cameras on one host, the MV-GE1104 is a 4-port GigE frame grabber. The accessory finder and the frame grabbers category filter by port count.

3. Force the link speed. Pin every port in the path to 1000 Mbit/s full duplex and switch off energy-efficient Ethernet. A link that renegotiates mid-run produces exactly the minutes-long episodes you are chasing.

4. Use inter-packet delay as a brake, not a fix. A small delay between packets proves a hop's buffers were overflowing: if stability returns, the constraint is real. It also cuts throughput, so remove it once packet size and ring are right.

5. Replace what still drops. If the direct-copper run stays clean and the converter run keeps losing after steps 1–4, the converters or their SFP modules are the fault. No camera setting repairs a marginal transceiver; replace the pair or terminate the fiber in the host.

When the code really is the problem

Some runs do point back at the loop, and the pattern says so: empties at a steady cadence with no stall-first burst, loss and drop counters at zero, worse whenever processing or the GUI competes for the host. Then fix the software: keep the buffer queue moving and copy pixel data out before heavy work (fetch-buffer queue mechanics), drain stale buffers instead of re-reading the newest one (duplicate frames from a full queue), and run fetch off the GUI thread (a frozen interface during capture).

The longer variants differ. Acquisitions that run for hours drift into incompletes; re-time them or throttle the data (incomplete frames in long acquisitions). A fetch that raises a timeout instead of returning empties asks for stream-timeout and buffer-count tuning (stream timeout checks). Acquisition that outruns processing opens a frame-rate gap. And when the camera never reaches its rated rate at all, the link sets the frame rate, not the loop.

Commissioning checklist

Run this before the system ships, and again after any change to the network path.

Before install

• Compute utilization from width × height × bytes per pixel × frame rate and keep it under 90% of effective capacity — capacity, not line rate.
• Probe the path MTU end to end and record the largest frame size every hop answers.
• Baseline the camera on a direct copper lead and keep that log for comparison.

At bring-up

• Set the camera packet size just under the path MTU, before starting acquisition.
• Set the NIC receive ring to its maximum and confirm it with ethtool -g.
• Force 1000 Mbit/s full duplex on every port; switch off energy-efficient Ethernet.
• Put the producer's resend statistic and your empty-buffer counter in the same log as the frame times.

In production

• Soak at maximum frame rate for at least 30 minutes; episodes in this class run for minutes, not seconds.
• Alarm on any non-zero empty-buffer rate; a healthy stream sits at exactly zero.
• Re-check after every switch, converter, SFP or driver change: the path drifts, the camera does not.

Worked in this order, the fault names itself: capacity first (packet size, receive ring, dedicated port), then the path hardware, then the loop. To have the parts picked for your camera and host layout, start the system configurator at the accessories step; the systems topic page gathers the acquisition-side fixes next to this one; or contact us with your camera, host and link diagram and we will work the numbers with you.