Long waits for frames in a multi-camera rig: switch to polling

Three depth-and-color cameras stream over USB, each in its own process, and every few seconds one pipeline's blocking frame wait returns after 150 to 220 ms instead of 67 ms. The cameras are healthy — the frame numbers prove it. What you have is a host-side queuing artifact: one blocking call per camera, each waiting on a complete color-plus-depth set, sits on the slowest stream while three cameras share one USB link. This walkthrough reads the evidence from a 1280×720, 15 fps RGB-D rig, sizes the bus with the arithmetic shown, and gives the fix in order: poll every camera non-blocking, assemble frame sets by timestamp, and split controllers so no root hub carries the whole rig.

Why does one camera's frame wait block for 150 ms with three running?

Because a blocking wait cannot return until every enabled stream has a frame ready: at 15 fps with color and depth both on, one stalled depth delivery holds the return while color keeps arriving, and three cameras sharing USB make that stall routine.

The blocking call is a single-camera construct, and the SDK documentation says so: wait_for_frames() is meant for one camera, and projects running several cameras should use poll_for_frames() instead. The more cameras stream at once, the worse it gets, so three RGB-D pipelines is precisely the case the guidance excludes. For one camera the blocking pattern is correct — frame timeouts and the one-wait-per-frame rule covers that simpler case.

One process per camera does not change the physics. Processes isolate your code and the SDK state, but they cannot partition a shared USB root hub or the host's frame dispatch. Each blocking call still demands a complete set before it returns, so the slowest of six streams sets the pace for every process.

What do the frame numbers say when a wait runs long?

Color advancing while depth stands still — color 661 to 663 while depth held at 659 — means one stream stopped delivering for three frame periods, not that your code or the camera stalled.

The two log rows tell the whole story:

Depth delivered nothing new during either wait. Color advanced two frames in one row and three in the other while depth sat still, and the call still had to come back with a set, so it paired fresh color with the same depth frame a second time. Alignment cost 5.6 ms, a rounding error next to 213.9 ms inside the call — the alignment block is a victim of the stall, not its cause (raw versus rectified frames covers what alignment itself does). Exposure is equally innocent: at these rates exposure sits far inside the 66.7 ms frame period, as trigger timing versus exposure time breaks down.

How much USB bandwidth do three RGB-D cameras need?

About 553 Mb/s per camera and 1.66 Gb/s for the trio at 1280×720, 15 fps, RGB8 plus Z16 — barely a third of a 5 Gb/s link, which is why the failure is intermittent scheduling and retries rather than steady overload.

That total explains the pattern you already measured: one camera with both streams stayed clean, three cameras color-only stayed clean, and only three cameras with both streams stalled. The combined payload is what pushes host dispatch into contention. A 5 Gb/s USB 3 controller sustains roughly 3.5 to 4 Gb/s once protocol overhead, other devices and retries take their share, so 1.66 Gb/s fits on paper with thin margin — and bursts spend it. Sizing a four-camera host works the same budget for one more camera.

The queue lives host-side. Each camera lands frames in its own buffers, the SDK dispatch thread hands them to your pipeline, and a blocking wait holds your thread until the set completes. A few milliseconds of jitter on each of six streams adds up, and the wait reports the sum.

Is switching to poll_for_frames the real fix?

Polling is the documented fix for multi-camera capture: poll_for_frames() returns immediately with whatever has arrived and never blocks on the slowest stream, so a stalled depth frame costs one dropped set instead of a 150 ms stall.

Converting one pipeline takes minutes:

1. Keep the process layout: one process, one pipeline, one device serial, both streams enabled exactly as now.
2. Replace the blocking wait with the non-blocking poll, called in a loop with a 1 to 2 ms sleep between empty returns.
3. Pull each arriving frame into a small per-stream buffer keyed by frame number, newest overwriting older.
4. Assemble complete sets by timestamp, as the next section describes, and process only those.
5. Count dropped sets per minute as the new health metric. With metadata-only logging, a drop costs one sample, not an inspection.

The blocking call has a second documented failure mode worth knowing while you are in here: it also drops frames during file logging, as frame drops in recorded captures shows. Polling fixes that path too.

How do you pair color and depth frames after switching?

Pair the newest color and depth frames whose timestamps differ by less than half a frame period — 33 ms at 15 fps — and drop anything older than two periods rather than reusing stale depth.

Timestamps are the one clock both streams share. Hardware sync tightens pairing to a millisecond or two, but delivery still lands in separate buffers, so the matching window stays yours to enforce — multi depth camera sync configuration covers the wiring side, and frame timing in camera buffers covers reading the timestamps correctly. Gate each accepted set on frame numbers as well: every stream's number must advance by exactly one from the previous set. A repeat or a jump means a drop; log it and move on, and never interpolate a missing depth frame.

The window is one line of arithmetic. At 15 fps the frame period is 1000/15 = 66.7 ms, half is 33 ms, and the stale cutoff is two periods, 133 ms. At 30 fps those become 16.7 ms and 66.7 ms.

When is the stall a cable or controller problem?

When the stall survives the switch to polling: that points at delivery — a shared root hub, a hub in the path, or a cable re-negotiating under motion — so map the USB tree before buying hardware.

Run lsusb -t and note which controller each camera hangs from. Three cameras behind one root hub is the default layout on most industrial motherboards, and any external hub makes it worse because every port behind it shares one upstream link. Then test in this order:

1. Move one camera to an add-in USB 3 controller card. If the long waits collapse, the link was the constraint, not the SDK.
2. Watch dmesg during a stall for reset or re-enumeration lines: those name a cable or connector, not scheduling. Scrambled USB frames and color stream timeouts across host ports follow this same split.
3. Replace long or moved cables with short, shielded, drag-chain-rated ones and re-run the 60 s capture.

When the harness flexes with the fixture, order drag-chain-grade data cables:

One bench habit worth keeping: the depth projector is a laser, so leave its housing intact and keep eyes away from the aperture while you swap cables.

If controller slots run out, move part of the rig to area scan cameras on switched Ethernet and the shared-link question disappears, because every camera gets its own link. Multi-camera frame drops over GigE runs the same diagnostic order for that interface. On the sensor side you can stay put: a USB3 camera such as the LS-CMR-0166 and its GigE sibling, the LS-CMR-0053, share one 12 MP IMX226 sensor across two interfaces, so pick per station by link, not by sensor:

All 2 share: Sensor size 1/1.7". Values as printed in each product's data sheet; open a part for its drawing and full specifications.

How do you prove the fix worked?

Log the monotonic return interval and both frame numbers for a 60 s run per camera, and accept the change only when the 99th percentile sits under two frame periods — 133 ms at 15 fps — with zero repeated frame numbers inside any set.

Your baseline is the two rows above: 152.8 ms and 219.7 ms worst waits, depth timestamp steps of 0 ms. After the change, check five numbers:

• Median return interval close to 66.7 ms, one frame period.
• 99th percentile under 133 ms.
• Color and depth frame numbers each advance by exactly one per accepted set.
• Dropped sets under 1 percent of frames per camera.
• Alignment running after pairing, at its measured 5.6 ms, not inside the wait.

How do you keep the next multi-camera build stall-free?

Design to a bus budget from day one: hold sustained payload under about 60 percent of the practical link, give each camera its own controller or a switched Ethernet link, and poll every stream non-blocking.

The checklist for the next quoting round:

• One pipeline and one device serial per process; never share a pipeline across cameras.
• Non-blocking polls on a fixed 2 ms cadence, with no blocking waits anywhere in the capture loop.
• Timestamp pairing with a half-frame window, gated by frame-number advance.
• Payload budget per controller computed before the hardware order, using the 60 percent rule.
• No hubs between camera and host; drag-chain cables wherever the harness moves.
• Hardware sync lines when frames must correspond across cameras, with the matching window kept regardless.

When the budget will not close, move depth out of the host and into 3D laser profilers:

A laser profiler ships processed profile data instead of raw video, so the host receives measurements rather than depth frames it must pair, align and buffer. You trade away pixel-level access to the raw depth image, and the stall problem leaves with it.

To spec the cameras and links for a build like this, start from the camera finder or the system configurator; more capture-timing fixes sit on the systems topic page. If a stream plan needs a second opinion, contact us with your camera count, streams and frame rates.