USB3 camera callbacks missing images with multiple cameras

Your trigger counter says 10,000 pulses; your image handler delivered 9,850. With one USB3 camera the counts match every run. With two or more cameras triggered at full frame rate, frames vanish — and a 10 ms pause between triggers makes them reappear. The handler code is almost never guilty. There are two causes: grab threads racing a shared host path, and a USB3 controller whose practical bandwidth is already spent. Here is the signature that separates them, the bandwidth arithmetic to run before you touch code, and the three fixes that hold at full rate.

The failure signature, and what each clue rules out

Write the observations down before you change anything. Each one eliminates a class of causes.

1. The triggered count is higher than the handler's received count.
2. One camera alone: counts match, every run.
3. Two or more cameras at full trigger rate: counts fall behind.
4. About 10 ms between triggers: counts match again.
5. One lock around the capture method: counts match again.

Clues 4 and 5 together prove the race: pacing removes the collisions, the lock removes the shared resource, and both restore the counts. Bandwidth is the second suspect, and no amount of threading fixes a pipe that is too small — so run the arithmetic next.

Two roots: a raced capture path and a saturated controller

SDK event handlers do not run on your threads. Every camera's grab loop calls your handler code on the SDK's own thread, so with N cameras you have N threads entering camera code and your code at the same time. Any object two of those threads touch is a race: a shared buffer pool, a decode or colour-convert workspace, a register write through one connection object, a plain counter increment. The symptoms are dropped notifications, torn or stale buffers, and counters that drift behind.

One USB3 controller is one pipe, no matter how many ports it has. The USB 3.0 line rate is 5 Gb/s — about 640 MB/s on paper — but hosts sustain roughly 350-400 MB/s across all ports of one controller, and many run out of breath near 437 MB/s (3.5 Gb/s) of measured traffic. A bulk stream cannot slow down gracefully: USB3 Vision defines no resend, so when demand passes the ceiling, whole frames disappear. Your code never sees them and no callback fires.

We plan multi-camera USB hosts with these same numbers in multiple USB cameras on one host: bandwidth and power budget.

Worked example: two 5 MP cameras at 75 fps on one USB3 controller

The inputs, with their sources:

• Frame: 2448 × 2048 pixels, 8-bit mono — the 5 MP camera class this failure is reported on, 1 byte per pixel.
• Frame rate: 75 fps, the full trigger rate of the case.
• Controller budget: 350-400 MB/s practical per USB 3.0 host controller (5 Gb/s line rate).
• Design margin: 80% of the conservative end, 350 MB/s.

The rule: multiply frame size by frame rate for each camera, add the cameras that share one controller, and compare the sum with the budget.

The arithmetic:

1. Frame size: 2448 × 2048 = 5,013,504 bytes ≈ 5.01 MB.
2. One camera at full rate: 5.01 MB × 75 fps = 376 MB/s.
3. Against the budget: 376 ÷ 400 = 94%; 376 ÷ 350 = 107%. One camera alone sits at the ceiling — which is why the single-camera test passes and tells you nothing.
4. Two cameras: 2 × 376 = 752 MB/s of demand. The controller delivers at most 400 ÷ 752 = 53% of it; the other frames never arrive.
5. The pacing test: 75 fps is a 13.3 ms period (1000 ÷ 75). Adding 10 ms gives a 23.3 ms period, so the stream drops to 1000 ÷ 23.3 ≈ 43 fps and 5.01 × 43 ≈ 215 MB/s — under budget and collision-free. Pacing fixes both causes at once, and charges you cycle time for it.

Decision: size every controller to 280 MB/s or less (0.80 × 350 MB/s). Two of these cameras share one controller only at 280 ÷ 2 ÷ 5.01 ≈ 28 fps each. At 75 fps, one camera per controller — or cut pixels, pixel depth or rate until the sum fits.

Fix the race: one lock around every shared call

The minimal fix that the counts confirmed: serialize the capture path with one lock. Give each camera its own grab thread, and put a single mutex around every call two threads can enter at once:

• buffer pool acquire and release;
• decode, colour-convert or any step over a shared workspace;
• feature or register writes that share one connection object;
• your counters, unless they are atomic.

Hold the lock for the push only — a queue insert and a tick add microseconds against a 13.3 ms frame period. Processing stays outside the lock, or you have built a single-camera system with two cameras attached.

The pass condition is numeric: 10,000 triggers, both counters logged, equality at full rate with no pacing.

If your wait-for-trigger gate is shared across cameras, it serializes triggering too; give each frame its own wait, as in frame didn't arrive within timeout: one wait call per frame.

Scale past the lock: per-camera queues instead of callbacks

Callbacks are where sharing hides, because the SDK picks the threads. Invert the flow: each camera gets a grab thread — the SDK's loop or your own — and its own queue; your worker pool pulls. The only cross-camera objects left are queues with one producer and one consumer each.

Three rules make it hold:

• Copy the buffer before you queue it. The SDK can recycle the buffer the moment the handler returns, which is the mechanism behind torn frames in torn frames in a camera callback: copy the buffer before you queue it.
• Take buffers from a pre-allocated pool, not from the allocator, per reuse image buffers instead of per-frame allocation.
• Let grabs and processing run in parallel, the pattern in two camera closed loop latency: grab and process in parallel.

Split the bus: one camera per USB3 controller

Find the topology before assigning ports. On Linux, lspci lists the xHCI controllers; on Windows, the host controller view shows them. Two ports on the same controller share its 350-400 MB/s; ports on different controllers each carry their own budget. Connect cameras straight to host ports — a hub funnels everything into one upstream link. Where the sum still runs hot, cap each camera: USB3 Vision cameras expose a bandwidth or throughput limit in their feature tree, so you can reserve shares that add to 280 MB/s on a shared controller.

The arithmetic also tells you when one camera is already too much for one controller. The LS-CMR-0166 streams 4000 × 3000 at 30 fps over USB3. At 8 bits per pixel that is 12.0 MB per frame, and 30 fps turns it into 360 MB/s — a full budget for a single camera. Design such a camera a dedicated controller and the bus stays clean.

Vision controllers make the port count easy to plan around, because the cards print it. The MV-VC3301H-128G66 lists 3-4 USB 3 ports; the MV-VC2140-128G40-1T lists 2 USB 3 ports. The vision controllers range puts camera ports, light channels and industrial power in one box.

If the host itself is a compact PC, count its root controllers in the datasheet before you commit cameras to ports. Machines built for always-on machine duty are listed under industrial mini PCs.

Cut the demand before you add hardware

Work down the list, cheapest first:

• Pixel depth: run 8-bit mono unless the algorithm needs more. A 2-byte format doubles the stream — the example camera becomes 752 MB/s on its own.
• Region of interest: crop to what you inspect. Half the rows is half the bytes; the decision does not change.
• Frame rate: trigger at the speed the line moves. A slow cycle does not need a 75 fps stream.
• Sharing: throughput caps per camera keep one fast camera from starving the rest.

A marginal cable turns a saturated bus into a dropping one. Use short, shielded cables with locking connectors, and drag-chain-rated stock where the camera moves: the LS-CBL-0219 is a 3 m USB3.0 camera cable, high-flex rated for drag chains. More lengths and connectors sit in the camera cables category.

On the camera side, pick a USB3 model whose frame size and frame rate you can multiply straight into the controller budget. Compare interface and resolution across the area scan cameras category, and size each controller with the same arithmetic as the example.

When one camera alone drops at full rate

Then threading is not the story. Work through these in order of likelihood:

1. The camera saturates the controller by itself — the 376 MB/s case. Cut rate, crop, or move to an interface with headroom.
2. On Linux, the kernel's usbfs memory cap throttles buffers in flight; the raise-and-test procedure is in USB3 camera low frame rate and latency: fix usbfs memory and exposure.
3. The handler cannot keep up and the SDK's internal queue overflows — same medicine: queues, pooled buffers, processing off the grab path. The wider view is in acquisition faster than processing: fix the frame rate gap.
4. A timeout shorter than the slowest frame period swallows frames at burst peaks.

Checks before you call it fixed

1. Count both sides — triggers issued, frames dequeued — for at least 10,000 triggers at production rate with no pacing delays. Equality is the pass.
2. Run 30 minutes under real load: processing on, storage writing, every camera streaming.
3. Re-run the arithmetic per controller and confirm each stays at or under 280 MB/s.
4. Remove the pacing delays. If counts break again, the race is still there — bandwidth was never the whole story.
5. Log the camera's embedded frame counter across a full shift; gaps point at the exact frames to chase.

If the sum says your cameras cannot share the bus, re-plan before you debug more code: the camera finder filters by interface and resolution, and the camera configurator pairs cameras with mounts and controllers. For a second pair of eyes on a port map, send frame sizes and the controller layout through the contact page. More host-side fixes live on the systems topic page.