You asked one stereo depth camera for depth, color and motion at the same time, and the frame wait that returned cleanly with any two streams now throws a timeout with all three — in one field case even a 10 second budget did not save it. The camera is not faulty: this is a frame-set architecture problem in the SDK's Python wrapper, and it has a repeatable fix. This guide walks it in the order you meet it: why three stream types stall one composite wait, why the motion data moves to a second pipeline, the two timeout numbers that make the split stable (10000 ms first, 200 ms after), the warm-up frames to discard while auto-exposure settles, and a 60 second count test that proves the result before you ship.
Why does wait_for_frames time out with RGB, depth and IMU together?
Because one pipeline assembles a single frame set from every enabled stream, the call cannot return until depth at 30 fps, color at 30 fps and motion samples at 63 and 200 Hz have all delivered — and at start-up, auto-exposure settling makes color the straggler.
The symptom is precise. Enable any two of the three stream groups and frames flow; enable depth, color and motion together and the first wait_for_frames call raises a timeout. A 10 second budget on the single pipeline did not clear it in the field case, which rules out a too-short timeout as the whole story.
The mechanism is in the frame set. The wait returns one composite set — the field code pulls accel and gyro out of the same returned object with first_or_default — so the wait covers the slowest enabled stream, every time. Motion streams at 63 and 200 Hz also multiply the matching work the SDK does to build one coherent set, and the first seconds of streaming are exactly when color auto-exposure hunts for its operating point. Three stream families, one composite wait, worst-case timing at start-up: the first call loses the race.
After start-up the failure changes character. With motion split out, the wrapper's 5000 ms default wait still dropped the occasional first frame and nothing else. That is why the fix has two parts: architecture and budgets.
Should the IMU run in its own pipeline?
Yes: keep depth and color on one pipeline and run accel plus gyro on a second pipeline object against the same device, with a 10000 ms first-frame timeout and 200 ms after — that combination ran stable where a single pipeline timed out even at 10 s.
Splitting removes motion from the composite wait. The image pipeline's frame sets now contain only depth and color, so a late gyro sample can no longer hold your 33 ms frame cadence hostage, and the motion data arrives on its own schedule for whichever consumer wants it.
The split alone is not the whole fix. In the field case the failure became intermittent — first frame only — under the default 5000 ms wait, and short steady budgets traded stability for latency. The next section sets the two numbers that closed it out.
While you rebuild the configs, enable every stream before start. A color stream that never appears at all is a different failure with its own order dependency, covered in Depth camera color stream missing: the enable-order fix.
How long should the first-frame and steady-state timeouts be?
10000 ms for the first frame after start, then 200 ms for every wait after: that pair ran stable where a single pipeline timed out at 10 s and the split alone still dropped first frames under the 5000 ms default.
Why does the first frame need 10 s?
Start-up is the expensive moment: streams negotiate, buffers allocate, and color auto-exposure converges from a dark first exposure. The field case showed a first frame that could outlast 5000 ms with nothing wrong, and never a second failure once the first frame landed. Give the first call 10000 ms and the first call only. If the first frame still has not arrived in 10 s, stop raising the budget and read the host-link section below.
Why 200 ms after the first frame?
Steady-state waits are a host-scheduling question, not a camera question:
At 100 ms, three frame periods, the field case dropped frames whenever the operator moved the mouse; at 200 ms, six periods, it held. Two hundred milliseconds absorbs a host hiccup and still surfaces a genuinely stalled stream as an error within a fifth of a second instead of hanging your loop.
If a blocking wait still costs you processing time, the polling alternative in Long waits for frames in a multi-camera rig: switch to polling covers the same trade-off for several cameras on one host.
How many frames do you skip while auto-exposure settles?
Skip 10 frames — 0.33 s at 30 fps — before you trust image content: the guidance that closed the field case was to discard the first several frames while auto-exposure converges, and 10 frames at 30 fps delivers that with margin.
The arithmetic is one line: 10 frames ÷ 30 fps = 0.33 s. During that window the color image is dark or unevenly exposed, and depth from an active projector can wobble with it. Skipping is about data quality, not the timeout, but do it in the same loop: the first frames you successfully wait for are exactly the ones you should not use. The convergence behavior is covered in Dark frames under auto exposure: wait for convergence.
How do you write the two-pipeline code?
Create two pipeline and config pairs, enable depth and color on one and accel and gyro on the other, start the image pipeline and land one frame set, then start the motion pipeline — the complete pattern with the field stream parameters is below.
- Create the image pipeline and its config; enable depth at 848 × 480, z16, 30 fps and color at 424 × 240, bgr8, 30 fps.
- Create the motion pipeline and its config; enable accel and gyro, both as motion_xyz32f, at 63 Hz and 200 Hz.
- Start the image pipeline and wait for the first frame set with the 10000 ms budget. Image first means a hang at the second start points at the motion path alone.
- Start the motion pipeline; give its first wait the same 10000 ms budget.
- In the loop, wait image frames at 200 ms, discard the first 10, then read one motion set per image frame.
- Stop both pipelines in a finally block.
The SDK's Python module is aliased as rs here, as the field script imports it:
# illustrative settings (int)
first_timeout_ms = 10000 # first frame after each start
steady_timeout_ms = 200 # every wait after the first
warmup_frames = 10 # discarded while auto-exposure settles
image_pipe = rs.pipeline()
image_cfg = rs.config()
image_cfg.enable_stream(rs.stream.depth, 848, 480, rs.format.z16, 30)
image_cfg.enable_stream(rs.stream.color, 424, 240, rs.format.bgr8, 30)
motion_pipe = rs.pipeline()
motion_cfg = rs.config()
motion_cfg.enable_stream(rs.stream.accel, rs.format.motion_xyz32f, 63)
motion_cfg.enable_stream(rs.stream.gyro, rs.format.motion_xyz32f, 200)
image_profile = image_pipe.start(image_cfg)
_ = image_pipe.wait_for_frames() # pass first_timeout_ms here
motion_profile = motion_pipe.start(motion_cfg)
kept = 0
try:
while True:
frames = image_pipe.wait_for_frames() # steady_timeout_ms
kept += 1
if kept <= warmup_frames:
continue
motion = motion_pipe.wait_for_frames() # steady_timeout_ms
gyro = motion.first_or_default(rs.stream.gyro,
rs.format.motion_xyz32f)
accel = motion.first_or_default(rs.stream.accel,
rs.format.motion_xyz32f)
g = gyro.as_motion_frame().get_motion_data()
a = accel.as_motion_frame().get_motion_data()
finally:
image_pipe.stop()
motion_pipe.stop()
If you need depth in the color perspective, align after the wait: create align = rs.align(rs.stream.color), call align.process(frames), and read the depth scale from image_profile.get_device().first_depth_sensor() with get_depth_scale(), all as the field script does. Aligned output is still two frames from two sensors, so pair them by device timestamp rather than arrival order, as Sync RGB and depth frames by device timestamp, not queue order shows.
How do you keep every gyro sample at 30 fps?
Drain the motion queue every image cycle: the gyro produces 200 samples per second against 30 image frames, so one read per cycle keeps roughly 30 of every 200 samples and silently drops the rest.
The ratio tells you how much you leave behind:
The loop above reads the newest motion set once per image frame. That is correct when you need current orientation per image, and wasteful when you fuse full-rate inertial data. For full rate, poll the motion pipeline between image frames and queue everything you get; Capture all IMU samples between frames works that pattern in detail. If the merged streams feed odometry or a dataset, pair camera and IMU samples by timestamp at merge time — synchronizing camera and IMU data for vehicle datasets covers the pairing rules end to end.
When is the timeout not the pipeline at all?
When the failure breaks the two-of-three rule — a single stream times out on its own, or the second pipeline start hangs forever — you are looking at the host link or the SDK stack, not at frame-set architecture.
Three patterns sit outside this fix:
- A color or depth timeout inside a virtual machine. USB passthrough is the usual culprit; Depth camera color stream timeout in a VM: fix the USB link walks the link settings. Test on bare metal before touching pipeline code.
- A motion stream that dies on the second run. That is the USB autosuspend signature on Linux hosts; Depth camera accelerometer stream fails second run: USB autosuspend covers disabling it for the camera's port.
- The vendor viewer streams all three, your code does not. When the same camera and cable deliver in the viewer but time out in your build, compare capture settings and SDK build first — 3D camera capture time slower in the API than in the viewer covers that gap.
A hang at the second pipeline's start also occurs in practice and stayed unresolved in the field case. Starting image-first, as in step 3 above, isolates it: if the motion start still blocks on a clean host, treat it as a link fault rather than a code fault.
Do you need different hardware, or is the fix purely software?
Purely software: the split with tiered timeouts fixed the three-stream stall on the camera already in hand, so buy nothing — new hardware enters only when the station itself is new or the depth modality must change.
For a new station, the 3D sensors category carries the depth modalities: a ToF sensing camera such as the MV-DT01WMU, projected structured light, and stereo configurations.
- Type
- Depth / ToF camera
- Type
- Stereo / structured light
- Type
- Stereo / structured light
Every one of these carries an active emitter or illuminator: leave emission settings at factory defaults and do not view the source at close range.
What do you check before shipping the fix?
Run a 60 second count: expect about 1800 image frame sets (30 × 60), 12000 gyro frames (200 × 60) and 3780 accel frames (63 × 60), with zero timeout exceptions logged.
| Stream | Rate | Frames expected in 60 s |
|---|---|---|
| Image frame sets (depth + color) | 30 fps | 30 × 60 = 1800 |
| Gyro | 200 Hz | 200 × 60 = 12000 |
| Accel | 63 Hz | 63 × 60 = 3780 |
Accept counts within a few percent: USB scheduling shifts the last digit, not the order of magnitude. Then run the checklist:
- First wait after each start carries the 10000 ms budget; every later wait carries 200 ms.
- 10 frames discarded after start, before any image content is used.
- Image pipeline started and confirmed before the motion pipeline.
- Both pipelines stopped in the finally block — a stream that fails on the second run is usually the port, not the code.
- Motion drained per cycle, or polled to a queue, so 200 Hz data does not shrink to 30 Hz.
If the counts hold, the fix is done and the camera you own stays in service. If a stream still starves on a clean host, the camera finder and the system configurator are the next step for a hardware path, and the contact page reaches the applications team with your SDK version and stream rates.
Questions
Does a longer timeout harm the camera or the data?
No. The timeout only bounds how long one call waits before raising an error; nothing in the camera wears or corrupts. The real cost is latency: a 10 s budget on every frame would let a stalled stream hold your loop for seconds instead of 200 ms. Keep the long budget on the first call only, and treat repeated steady-state timeouts as a signal to fix the host side, never as a reason to raise the budget.
Can the motion pipeline run on its own thread?
Yes. The two-pipeline split does not require both waits in one loop: a common layout runs the motion pipeline on a worker thread that queues samples, while the main thread handles image frames and fusion. Apply the same budgets on each thread, 10000 ms for the first frame and 200 ms after, and stop both pipelines at shutdown so the worker does not wait on a stopped device.
Why 63 Hz accel and 200 Hz gyro specifically?
Those are the rates the field configuration requested and achieved: 200 gyro samples and 63 accel samples per second alongside 30 fps images. They are not universal values. You choose a rate when you enable the stream, from the rates the SDK lists for your device, and the ratio arithmetic follows you: at 30 fps, a 200 Hz gyro delivers about 6.7 samples per image frame, so plan the drain accordingly.
Do you still need the split with only one motion stream enabled?
Keep it. The field failure appeared with accel and gyro both enabled, and it was never tested with a single motion stream alone, so there is no evidence the stall shrinks. The split also solves the rate mismatch on its own: motion streams deliver 2 to 7 samples per image frame, and a separate pipeline is where you drain them. The cost is a few lines of setup code.
What if starting the second pipeline hangs?
Start the image pipeline first and confirm one frame set, then start the motion pipeline in a minimal script on its own. If a motion-only pipeline starts cleanly, the fault is in your loop ordering; if it blocks even alone, the fault is the port or power path, so move the camera to a rear host USB port or a powered hub and retest before touching the streaming code.