Capture all IMU samples between frames: poll and queue, don't wait

A depth camera that carries its own IMU produces two streams on two clocks: depth and color frames at 15-90 fps, inertial samples at 100-400 Hz. Drive both from one blocking grab loop and the image rate decides what you keep. At 30 fps you visit the gyro 30 times per second while it produces 200 samples, so about 85% of the motion data never reaches your code. This guide sets out the poll-and-queue pattern that keeps every sample: one non-blocking poll per stream, one bounded queue per sensor, a host timestamp on arrival, and consumers running in their own threads.

Why one grab per frame drops most of the IMU data

The blocking wait, named wait_for_frames in most SDK documents, returns one combined frameset per image frame. Your loop therefore runs at the image rate, and whatever IMU handling sits inside it runs at the image rate too. The sensor keeps sampling at its configured rate regardless, and every sample that lands while you are busy with the previous frame waits in the SDK or never gets read.

The arithmetic is blunt:

At 30 fps the frame interval is 1/30 s = 33.3 ms. A gyro at 200 Hz delivers 200 / 30 = 6.7 samples in that window, and an accelerometer at 250 Hz delivers 250 / 30 = 8.3. Read one IMU sample per frameset and you collect 30 samples per second: 30 / 200 = 15% of the gyro stream and 30 / 250 = 12% of the accelerometer stream. Switch the gyro to its 400 Hz option and the kept share falls to 30 / 400 = 7.5%.

Processing tightens the same screw. Thirty milliseconds of per-frame work caps the loop at 1 / 0.030 = 33 fps, and any stall starves the IMU reads along with the images. Exposure does the same from the other side: when it limits the camera to 15 fps (see depth camera frame rate drops: exposure settings), the interval doubles to 66.7 ms, each frameset spans 200 / 15 = 13.3 gyro samples, and you still collect at best 30 per second.

What you need

• A depth camera with a built-in IMU. Shortlist from the 3D sensors category and read the listed IMU rates alongside depth range and frame rate.
• An SDK that exposes the IMU as its own sensor with its own rate control. Typical depth cameras list accelerometer options from roughly 60 to 250 Hz and gyro options at 200 or 400 Hz; configure the highest your model lists.
• Python or C++ on an ordinary PC. The queues in this design hold kilobytes, not gigabytes.
• A destination for each stream: live fusion, disk, or both. Writing images without stalls is its own pattern; see streaming camera images to disk at a high frame rate.

Step by step: poll, queue, consume

1. Open each stream separately. Take a handle for depth, color, gyro and accelerometer. Do not pull the IMU out of the image frameset: that path runs at the image rate and is the whole problem.
2. Set and confirm the rates. Query the options, set the gyro to 200 Hz and the accelerometer to 250 Hz, then print what the SDK reports after the write. The configured value is what you will verify against, not the brochure value.
3. Poll instead of waiting. In the grab thread, call the non-blocking poll on each stream; poll_for_frames and try_wait_for_frames with a short timeout are the two common names for it. A poll returns immediately when nothing is ready, so an idle stream never blocks the others.
4. Timestamp on arrival. Read the host monotonic clock (time.monotonic_ns() in Python) in the producer, the moment the sample comes back. Timestamping in the consumer bakes the queue delay into your fusion.
5. Push to one queue per stream. Put timestamp-and-sample tuples into bounded queues: gyro, accelerometer and images each get their own. Bounded, so a slow consumer shows up as a full queue instead of as memory loss.
6. Consume in dedicated threads. One consumer per queue, and the image consumer carries the heavy processing. Keep the fetch off any UI thread; the GUI freezes during image capture fix is the same discipline applied to windows.
7. Shut down cleanly. Clear the running flag, join the consumers, then drain whatever remains. The stop streaming hangs fix walks that drain order.

Sketch of the gyro path, with placeholder names where your SDK's calls differ:

import queue, threading, time

gyro_q: queue.Queue = queue.Queue(maxsize=400)   # 400 samples = 2 s at 200 Hz
running: bool = True

def grab_gyro(gyro_sensor) -> None:              # producer thread
    while running:
        sample = gyro_sensor.poll()              # placeholder: your SDK's non-blocking poll
        if sample is not None:
            gyro_q.put((time.monotonic_ns(), sample))

def use_gyro() -> None:                          # consumer thread
    while running:
        t_host, sample = gyro_q.get(timeout=1.0)
        log_sample(t_host, sample)

Size the queues for your worst stall

Decide how long a consumer may stall, through a garbage-collection pause, a disk flush or a debugger break, and size each queue from that budget:

Worked example at 30 fps images, 200 Hz gyro, 250 Hz accelerometer and a 0.5 s stall budget:

• Gyro queue: 200 samples/s x 0.5 s = 100 samples.
• Accelerometer queue: 250 samples/s x 0.5 s = 125 samples.
• Image queue: 30 frames/s x 0.5 s = 15 frames.

Memory never becomes the problem. An IMU sample is three floats plus a timestamp, about 20 bytes, so 100 samples occupy roughly 2 KB. A 16-bit depth frame at 800 x 400 pixels is 800 x 400 x 2 B = 640,000 B, about 0.64 MB, so 15 frames sit in about 9.6 MB.

Latency is the real trade. A gyro queue that runs 100 deep adds up to 100 / 200 = 0.5 s between measurement and use. Robot guidance wants freshness, so size for about 100 ms (200 Hz x 0.1 s = 20 samples) and make the consumer fast instead. Recording for calibration can take the deep queue; the H5 recording route keeps image and motion streams together for offline work.

How to verify nothing was dropped

Run the rig for 60 s and count what arrived:

• Gyro: 200 Hz x 60 s = 12,000 samples expected.
• Accelerometer: 250 Hz x 60 s = 15,000 samples expected.
• Images: 30 fps x 60 s = 1,800 frames expected.

Achieved rate is counted samples divided by 60 s. Sensor and host clocks differ by a few tens of parts per million, so a shortfall under about 0.5%, which is 60 gyro samples per minute, is clock disagreement; beyond that you are counting real drops.

Then check spacing. Sort the host timestamps and scan the gaps. The nominal gyro period is 1 / 200 = 5 ms: treat a gap above 10 ms (two periods) as a stall and one above 20 ms (four periods) as a probable loss. Samples arriving in tight bursts with regular gaps are scheduler jitter, not loss, and the IMU sample rate jitter walkthrough shows how to separate the two. Track the queue high-water mark while you are at it: a mark that creeps upward means the consumer is slower than the producer and drops are next.

Common mistakes that still drop samples

• Reading the IMU from the image frameset. That caps capture at the image rate, 30 of 200 gyro samples per second, which is the 15% trap this pattern exists to escape.
• Processing inside the grab loop. Thirty milliseconds of per-frame work caps the loop near 33 fps and stalls every stream behind it.
• Sleeping between IMU reads on a timer. A 33 ms sleep collects 30 samples per second no matter what the sensor produced; only the SDK's poll or its callback knows when data is actually ready.
• Timestamping in the consumer. The queue delay lands inside your fusion. Timestamp at enqueue, in the producer.
• Leaving a queue unbounded. At 30 fps with 0.64 MB depth frames, ten minutes with no consumer is 30 x 600 x 0.64 MB = about 11.5 GB, and the queue hides the fault until the machine stops.
• Blaming the queue for a different fault. If the IMU read rate collapses only while video is actively streaming, work through the IMU read rate drops during video streaming fix instead.

When a callback beats a poll

If the SDK offers per-sensor callbacks, register one per IMU stream and push into the same bounded queues. The SDK hands you each sample as it lands, so latency is the lowest on offer and there is no polling jitter to smooth out. The contract is the one in the tip above: the callback body must stay far below the sample period, 5 ms at 200 Hz, so enqueue and return, with no logging and no heavy allocation inside the callback. Polling keeps one edge: you own the backpressure. When a bounded queue fills, you decide whether to block, drop the oldest or raise a flag, and you can join and drain deterministically at shutdown. Choose callbacks for lowest latency, polling for controlled overflow and clean teardown.

Match the hardware to the streams

With the software pattern settled, three questions pick the camera: which IMU rates the model lists, what frame rate it sustains at your exposure, and whether the interface carries depth, color and inertial data together without drops. Filter candidates in the camera finder, start a build at the camera configurator step, and browse the rest of the depth-camera fixes on the 3D vision topic page. If a station needs a fused image-and-motion design reviewed end to end, send the details through the contact page.