Incomplete frames stay incomplete: discard, requeue, grab a new one

A camera that keeps saving the same image, with frame status reporting incomplete and a pixel format that reads invalid, is stuck in a loop holding one damaged frame. The data is not late; it is gone. USB packets were dropped before your code ever saw them, and no setting calls them back. It is written for anyone driving a USB3 area scan camera from their own grab loop. You will get the loop that discards a bad frame correctly, the bandwidth arithmetic that shows whether the transport can keep up, and the fixes in the order that pays off.

Why does my camera keep saving the same frame over and over?

Your save routine is reading a variable the grab thread has stopped updating: the thread is stuck waiting on one incomplete frame, current_frame keeps holding the last complete image, and every capture call writes that same buffer to disk until you restart.

The chain is short. The SDK hands your code a frame whose status is not Complete. Refusing to process it is correct — but looping on the same object, sleeping, and checking its status again is not, because that object will never change. The only assignment to current_frame sits after that loop, so the consumer side never sees fresh data. The invalid pixel format is the same fault seen from the conversion call: an incomplete frame's payload is truncated, so its metadata cannot describe a valid Mono8 image. Treat it as a symptom, not a separate bug.

This belongs to the stale-frame family: a consumer reading state that the acquisition side stopped advancing. The camera image latency guide covers the cousin of this fault, where frames are fresh but late; the fix there is scheduling, here it is the loop shape below.

Can an incomplete frame become complete if I keep waiting?

No. A frame is marked incomplete because USB packets were lost in transit — usually after the camera's internal ring buffer was overwritten — and the application receives only what survived. Nothing ever re-sends the missing packets, so waiting cannot repair the frame.

USB 3.0 Vision streams images as bulk packets into buffers the host owns. When the host drains them slower than the camera fills them, the camera's buffer is overwritten mid-frame. The camera reports the loss and delivers the frame incomplete, or drops it outright when whole blocks are gone. From that moment the damage is baked into the bytes sitting in your queue. Read the status for statistics — incompletes per minute is a good health metric — but never as a signal to wait. Return the buffer with one queue call, take the next frame with one get_frame call. That is the whole correct response, and it costs you one frame, not the stream.

What does a correct grab loop look like?

One status check, one copy, one queue call, then the next grab: work with a frame only when its status is Complete, hand every delivered buffer back to the SDK exactly once, and never sleep while you hold a buffer.

running: bool = True
timeout_ms: int = 5000        # grab timeout in ms
incomplete_count: int = 0     # health metric: incompletes per minute
image = None                  # your image type

while running:
    frame = cam.get_frame(timeout_ms=timeout_ms)   # Frame object
    if frame.get_status() == FrameStatus.Complete:
        image = deep_copy_pixels(frame)            # SDK's buffer-to-image copy
    else:
        incomplete_count += 1                      # count it, never wait on it
    cam.queue_frame(frame)                         # exactly once, on every path
    if image is not None:
        process(image)                             # slow work runs on the copy
        image = None

The call names follow one camera SDK's Python API; your library spells them differently, but the shape — check, copy, requeue, grab — is the same everywhere. deep_copy_pixels stands in for the SDK's frame-to-image conversion, the call that copies pixels into your own object; a plain reference would follow the buffer as it gets refilled. The requeue comes before the slow work, so the transport layer gets its buffer back while you compute. In a callback-driven SDK the same rule applies inside the callback, which is exactly what the torn frames in a camera callback guide shows.

Prove the fix instead of trusting it: run 100 consecutive grabs of a running clock or a rotating part, hash the pixels of every saved frame, and require 100 distinct hashes.

For a triggered line the loop shape does not change per exposure; the hardware trigger grab loop in Python shows the trigger-driven variant.

Why didn't sleeping, resetting or requeueing fix it?

Because none of those actions puts a missing packet back: sleeping wastes cycle time while you hold a buffer hostage, resetting sacrifices the whole stream for one bad frame, and queueing the same delivered object twice corrupts the SDK's buffer bookkeeping.

• Sleeps inside the incomplete branch. A 0.5 s wait per bad frame caps recovery at 2 frames/s while the buffer stays out of the queue. The ring gets shallower, so the next frames drop more easily — which is why the delays seem to need growing over time. The camera is not overheating its way into this; the loop is starving itself.
• Requeue, then keep polling. After a queue call the buffer belongs to the transport layer again. Poll its status as if it were still yours and the next incoming frame can be written into memory you are still reading. That is how a stuck image turns into torn or duplicated data — the classic double-queue fault.
• Reset as a reflex. Reopening the device clears the state, which is why the unplug ritual appears to work, but it costs seconds per event and hides the cause. When the reset path itself throws — an IndexError from a stream reset, for example — the recovery kills the thread it was meant to save.

Changing the SDK's allocation mode (one API exposes it as AllocationMode.AllocAndAnnounceFrame) changes where buffers come from, not whether damaged frames heal; the default allocation is fine.

How much USB bandwidth does my camera really use?

Frame size times frame rate is the number that decides everything: a 4000 × 3000 Mono8 sensor at 30 fps produces 12 MB frames, 360 MB/s of payload, which fills most of one USB 3.0 controller's practical budget of 350-400 MB/s.

The 12 MP USB3 area scan camera reads out 4000 × 3000 pixels at 30 fps — exactly the size of camera this arithmetic is built on.

The budget has a basis, not a guess: USB 3.0's 5 Gbit/s line rate drops to 4 Gbit/s after 8b/10b encoding, and packet overhead takes the usable payload into the 350-400 MB/s zone — hence the USB3 Vision planning figure of about 350 MB/s per camera. Every interface lives by this arithmetic; the 5GBASE-T bandwidth, cables and distance guide runs the same numbers for high-speed GigE.

The same math runs backwards when you cap DeviceLinkThroughputLimit. Take a 5.3 MP Mono8 sensor at 5328 × 4608: each frame is 5328 × 4608 × 1 B = 24,551,424 B ≈ 24.6 MB. Cap the limit at 360,905,933 B/s and the best possible rate is 360,905,933 ÷ 24,551,424 ≈ 14.7 fps. If the application needs 20 fps from that sensor, it needs 20 × 24.6 MB ≈ 491 MB/s — more than one USB 3.0 controller delivers, whatever the software settings say. Decision: at 12 MB frames and 30 fps you get one camera per controller, or a lower frame rate.

Watch the fallback too: on a port that negotiates USB 2.0 — front-panel headers are the usual trap — the ceiling drops to 480 Mbit/s, about 40 MB/s usable. The 12 MB frame then tops out near 40 ÷ 12 ≈ 3.3 fps, and everything faster arrives as incompletes.

How do I prove the problem is bandwidth, not my code?

Cut the payload and count incompletes: if they vanish at reduced frame rate, a smaller ROI or a capped DeviceLinkThroughputLimit, the transport was saturated; if they continue unchanged at a third of the rate, the grab loop or the buffer pool is still wrong.

Run the tests in this order and write down the incomplete count after each:

Ranked by how often each is the cause: transport saturation first, by a wide margin on multi-camera or high-fps hosts; then the Linux usbfs pool at its 16 MB default, which cannot hold even two 12 MB frames; then cable and port faults, including silent USB 2.0 fallback; then grab-loop faults, the double queue above; and last, host silicon shared with a display or other traffic. That last one is real: one documented camera setup lost frame-rate stability whenever an external monitor was attached to the same machine.

What do I change first: settings, cable, port or controller card?

Work from the host outward and re-count incompletes after every step: raise the usbfs pool, fit a certified cable, take a verified SuperSpeed port, cap DeviceLinkThroughputLimit to a payload the bus sustains, then give each camera its own controller.

1. Raise the Linux usbfs budget. The kernel caps USB buffer memory at 16 MB across all devices by default, so two 12 MB frames already exceed it. Set 256 MB for the session with sudo sh -c 'echo 256 > /sys/module/usbcore/parameters/usbfs_memory_mb', and make it permanent with the boot parameter usbcore.usbfs_memory_mb=256. Five 12 MB buffers need about 60 MB, so 256 leaves room for a second camera or a deeper queue.
2. Fit a certified cable. Shielded, screw-locked, 3 m or less for full USB 3.0 speed. For a fixed camera the 3 m high-flex USB 3.0 camera cable suits the bench, and its drag-chain-rated jacket suits moving axes.

The camera data cables category holds the other lengths and connectors, and the accessories finder filters them by length and type.

3. Take a port you have verified. Rear-panel SuperSpeed ports first; if the camera enumerates at USB 2.0, its ceiling collapses to about 40 MB/s and no software setting recovers the frames. On Linux, lsusb -t shows which bus and root hub the camera actually sits on.
4. Cap the camera to the bus. Set DeviceLinkThroughputLimit to frame rate × frame size: 20 fps of the 12 MB frame is 20 × 12 = 240 MB/s, comfortably inside one controller's budget. Cap further when the host also drives disks, displays or a second camera.
5. Split controllers when the sum exceeds the budget. Two cameras at 360 MB/s each ask one controller for 720 MB/s against 350-400 MB/s of budget; they cannot share. Add a PCIe USB 3.0 controller card per camera, or choose a host built with independent controllers — the industrial mini PCs category is that kind of hardware.

Working up a whole chain at once? The camera step of the system configurator pairs the camera, cable and interface so the bandwidth sum is checked before you order.

Decision: a single camera dropping frames at full rate is steps 1-4 in almost every case; incompletes that appear only when the second camera starts is step 5.

How do I stop incomplete frames from coming back?

Make the discard loop and the bandwidth budget permanent: status-checked grabs with one queue call per frame, an incomplete counter with an alarm threshold, and the payload-per-controller sum written into the machine file so the next camera addition respects it.

• Check the frame status on every grab and never process an incomplete frame; count it instead.
• Queue every delivered buffer exactly once, on every code path, exceptions included.
• Copy pixels out before the queue call and run slow processing on the copy; the reuse image buffers instead of per-frame allocation guide shows the buffer pattern behind this.
• Set usbcore.usbfs_memory_mb at boot, not per session — a reboot quietly restores the 16 MB kernel default.
• Keep the arithmetic on the cabinet door: frame size × fps per camera, summed per controller, against the 350 MB/s planning figure.
• Use certified, screw-locked cables with strain relief, routed away from motor drives and drive wiring.
• Watch the SDK's throughput statistics where it exposes them; a rate that sags over a shift is an early warning before frames go incomplete.

Sizing the camera side from scratch? Browse the area scan cameras category, filter the camera finder by interface and frame rate, and read the rest of the acquisition guides on the systems topic page. If you want the bandwidth sum checked before you order, the team on the contact page will run it with you.