Incomplete frames in long GigE acquisitions: check and throttle data

An unattended long GigE capture that passed every short test comes back with a few frames flagged incomplete, or fails at camera open with an incomplete-operation error. The sensor is almost never the problem: packets were dropped on the link, and the odds catch up as the frame count grows. This guide is for anyone scripting unattended acquisitions — parameter sweeps, long recordings, monitoring runs. You will compute the camera's data demand in bytes per second, cap its send rate below what one Gigabit link reliably carries, wire a per-frame status check into the loop so nothing incomplete is ever saved, and match your symptom to the right one of four causes.

Why long runs fail when short tests pass

GigE Vision splits every image into UDP packets. The receiving SDK marks a buffer complete only when the last packet of that frame arrives; one packet lost on the way and the whole frame is flagged incomplete. The bytes that did arrive are not half a picture you can average — they are a tear, and most SDKs hand back a failure object instead of pixels.

Short tests hide this. At a loss rate of 0.1% per frame — realistic on a link running near its ceiling — a 50-frame rehearsal carries a 4.9% chance of a bad frame, while a 5,000-frame sweep running through the night is almost certain to hit one:

Same camera, same cable, same switch; only N changed. That is why the first long run is where this problem surfaces.

A second failure mode appears before any frames arrive: the SDK errors while opening the camera or reading its feature list, returning code -19, which that SDK family defines as an incomplete operation. That points at the control channel — firmware or SDK version mismatch, or another program still holding the camera — not at the stream settings. Both modes are sorted in the table below.

Match the symptom to the cause

Work down this table; the first row covers most multi-hour failures.

Blank frames instead of torn ones are a variant of the same loss; GigE Vision stream packet loss: diagnose blank frames covers that diagnostic path, and torn live images are walked through in GigE Vision packet loss horizontal lines in live images. When the failure surfaces mid-stream as a buffer-incomplete error, the same lost-packet mechanics apply — see GigE camera buffer was incompletely grabbed: fix lost packets.

Size the data rate first

Every fix below hangs on one number: the bytes per second the camera wants to put on the wire.

Bytes per pixel comes from the pixel format: Mono8 is 1 B per pixel, Mono12 held in 16-bit words is 2 B, and Mono12 packed stores two pixels in three bytes, 1.5 B per pixel.

On the receive side, a Gigabit link runs at 1,000 Mbit/s and Ethernet framing takes its cut: each 9,000-byte jumbo packet occupies 9,038 bytes of line time (8 B preamble, 14 B header, 4 B FCS, 12 B inter-frame gap). The payload ceiling is therefore 1,000 Mbit/s × 9000 ÷ 9038 ≈ 995.8 Mbit/s ≈ 124.5 MB/s, before UDP/IP and GigE Vision headers take a few percent — so plan on about 110 MB/s of camera payload per clean, dedicated link. At the default 1,500-byte MTU the ceiling barely moves (1,500 ÷ 1538 gives ≈ 975 Mbit/s ≈ 122 MB/s), but per-packet host load rises steeply, which is exactly what jumbo frames are for.

The same arithmetic budgets extreme cases — the 2,000 fps recording plan works it at scale in high frame rate camera data rate: plan 2000 fps recording — and two cameras sharing one link need caps whose sum stays under the ceiling, the exact failure behind multiple GigE cameras, same trigger: why frames arrive incomplete.

Fix 1: cap the camera's send rate

The direct fix for stream-side loss is the camera feature that limits its transmit rate in bytes per second — in one common SDK's feature list it appears as StreamBytesPerSecond. Set it before the stream starts.

Worked example, with illustrative inputs: a 2448 × 2048 sensor, Mono8, free-running at 24 fps, sweeping 5 gains × 4 exposures × 25 frames across 10 scenes = 5,000 frames.

• Frame size: 2448 × 2048 × 1 B = 5,013,504 B ≈ 5.0 MB.
• Demand: 5.01 MB × 24 fps = 120.3 MB/s ≈ 962 Mbit/s — above the 110 MB/s (880 Mbit/s) planning ceiling, so any host hiccup becomes a dropped packet.
• Cap: set the send-rate limit to 90 MB/s (720 Mbit/s), about 80% of the ceiling.
• Paced result: 90 ÷ 5.01 ≈ 18 fps effective. The sweep simply runs longer, moving 5,000 × 5.0 MB ≈ 25.1 GB in 25.1 GB ÷ 90 MB/s ≈ 280 s of transfer.

The cap trades time for integrity, usually a good trade when a 10 s exposure dominates each cycle anyway. For triggered lines the same arithmetic sets the ceiling trigger rate: trigger rate stays at or below cap ÷ B. If you would rather cut demand than pace, drop the frame rate with timed acquisition (GigE camera timed acquisition: control frame rate and bit depth) or cut the bytes with the format: the same sensor in Mono12 unpacked needs 10.0 MB per frame, so 12 fps already demands 120 MB/s, while Mono12 packed needs 7.5 MB per frame — 12 fps lands at 90 MB/s, right where the example cap sits.

Check this arithmetic against your own width, height and format before the run, not after it.

Fix 2: set the packet size the whole path carries

The cap thins the traffic; the packet size decides whether the traffic fits the path at all. A camera sending 9,000-byte packets into an adapter still at a 1,500-byte MTU loses every stream packet, so frames come back incomplete from the first one — the second row in the table. Enable jumbo frames end to end (camera, adapter, every switch port between), then set the camera's packet size to the largest value the path confirms, before streaming. The procedure is in set GigE camera packet size before starting acquisition, and the driver-level variant where NIC and stack disagree is in GigE incomplete frames from a driver and NIC MTU mismatch.

Test in 100-frame batches and read the counters: if even one of the first hundred frames is incomplete, the path — not the pace — is wrong.

Fix 3: free the host and give the link its own port

Losses that survive correct pacing usually come from the receive side:

• Power management. An adapter that dozes between packets, or a CPU that parks cores during quiet stretches, drops interrupts — and a long run is quiet far more often than a short one. Set the power plan to high performance and switch off the adapter's energy-saving options before a long unattended run.
• A shared port. Built-in Ethernet often shares a controller with other traffic; a backup job or a second camera pushes a near-loaded link over the edge. One camera, one port, nothing else on it.
• Disk stalls. Writing 25 GB while grabbing keeps the storage stack busy. Queue writes off the grab thread — the recording-side fixes are in multi-camera recording to disk without dropped frames, and preview-side stalls have their own checklist in preview dropped frames: diagnose stalls and resource limits.

The structural fix is a port of its own for the stream:

A dedicated PoE network card isolates the camera's interrupts and bandwidth from everything else the machine is doing, and the second port is there for the day the bench grows.

Make the loop self-checking

Pacing removes the drops; the status check catches whatever survives. Grabbing one frame at a time does not buy completeness by itself — the per-frame flag does. In the SDK family where the -19 code came from, the check reads exactly frame.get_status() == FrameStatus.Complete; whatever your SDK calls it, read the completion flag on every buffer before it touches storage:

frames_ok = 0    # completed frames written
frames_bad = 0   # incomplete frames counted, never saved

for frame in camera.get_frame_generator(limit=num_frames, timeout_ms=11000):
    if frame.get_status() == FrameStatus.Complete:
        frames_ok += 1
    else:
        frames_bad += 1   # discard and re-grab, never save

get_frame_generator, get_status and FrameStatus.Complete are that SDK's documented names; frames_ok and frames_bad are illustrative counters that start at zero; timeout_ms = 11000 covers a 10 s exposure with margin. What to do with a flagged frame — discard it and grab a fresh one rather than saving it — is its own pattern, in incomplete frames stay incomplete: discard, requeue, grab a new one.

The test that proves the set-up: rerun the sweep and require frames_ok == 5000 and frames_bad == 0 before you accept the data. For sweeps that vary gain and exposure per frame, carry the settings inside each image so any re-grab reproduces the exact condition — chunk data does this per frame, as in chunk data on camera frames: carry timestamp and exposure per image.

Prevention checklist for the next long unattended run

Run down this list before the real capture:

• Compute demand as width × height × bytes per pixel × frame rate, and hold it at or below 80% of the 110 MB/s planning ceiling — the headroom absorbs host hiccups.
• Set the send-rate cap and the packet size before the stream starts; both lock during acquisition.
• Jumbo frames end to end, camera to adapter, through every switch port.
• One camera per adapter port, no other traffic on the link during the run.
• High-performance power plan, adapter power saving off.
• Grab timeout above the longest exposure plus margin — 11,000 ms for a 10 s exposure.
• Per-frame status check on, counters written to the run log; abort the sweep if frames_bad is nonzero.
• Camera firmware and SDK current and updated together before the run; rehearse 500 frames unattended first.
• Chunk or log the settings into each frame so any re-grab reproduces its condition.

When the link itself is the limit

If the demand is real — Mono12 at high frame rates, or several cameras per machine — pacing turns a broken run into a slow one, and the honest fix is bandwidth. Rerun the framing arithmetic at 10 Gbit/s: 10,000 Mbit/s × 9000 ÷ 9038 ≈ 9,958 Mbit/s ≈ 1,245 MB/s of payload ceiling. That is where 10GigE parts earn their keep: the MV-GT2104 is a quad-port 10GbE NIC, and the MV-GE1104 is a 4-port GigE frame grabber. One card, four independent links, each with its own interrupt line.

Both sit in the frame grabbers category, and the accessories finder filters by interface and port count. For the adapter-versus-frame-grabber decision on the host side, this guide to selecting hardware for GigE Vision acquisition walks the trade-offs, and what the interface carries at speed is worked through in GigE Vision for a high speed camera: how much frame rate fits. If the sensor is the half you are changing instead, the area scan cameras category and its camera finder filter by interface, resolution and shutter.

More stream-side failure patterns live on the systems topic page. If you would rather hand the arithmetic over, send sensor, pixel format and frame rate through the contact page — free expert support with every order.