IMU read rate drops during video streaming: why and the fix

When you stream video and poll an IMU over the same USB link, raising the frame rate can quietly cut the IMU rate instead of helping your odometry filter. In the worked case below, doubling video from 30 to 60 fps pushed polled IMU reads from about 140 Hz down to 42-46 Hz. Here you get the mechanism behind that coupling, the arithmetic on a measured 640 x 480 MJPG camera, and a clear separation between what each timestamp marks and what it does not. You leave with four fixes in priority order and a decision for each rig you might build.

One link, two streams

On cameras that expose an inertial sensor through a UVC extension unit, the host drives the IMU by polling. Every read_imu() call is a control request, and the camera firmware answers it with the latest available measurement. There is no FIFO delivery through that path, and the sensor's output rate is managed by the firmware with no SDK setting to change it. Whatever the sensor produced between two polls is gone before the host sees it.

The video stream rides a different USB transfer class. Isochronous packets hold reserved slots on every 125 µs microframe of the high-speed bus, whether or not their payload turns up on time. The 480 Mbps signaling rate is not what bites here: 640 x 480 in MJPG runs tens of kilobytes per frame, so roughly 2-4 MB/s at 60 fps, a fraction of what the bus can carry. What the read waits on is the firmware servicing control requests between frame tasks. Same host, same cable, same bus — the contention is scheduling, not bytes. The same budgeting discipline matters whenever a hub splits one USB link across cameras, and the stall signature looks similar when frames stop updating on a saturated USB link.

Why doubling the frame rate cut the IMU rate threefold

Watch the read durations, not just the average. At 30 fps the median polled read came back in 4.85 ms and the 95th percentile in 19.9 ms: most reads slip into the gaps between frames, and the long ones queue behind a frame. The count of long reads tells you how often that happens — about 1,800 reads of 16 ms or longer per 60 s, which is 1,800 ÷ 60 = 30 per second, exactly the frame rate. One blocked read per frame.

At 60 fps the median stretches to 17.5 ms, one 16.7 ms frame period, and the 95th percentile reaches 32 ms, about two frame periods. When every read costs a frame period, reads can only emerge at roughly one per frame, and delivery falls to 42-46 per second. Twice the frame events, three times the IMU interval: the turnaround is what moved.

Worked example: 640 x 480 MJPG at 30 fps versus 60 fps on one USB 2.0 link

Inputs, and where each comes from:

• Video: 640 x 480 pixels, MJPG, streamed at 30 fps and then at 60 fps (the measured camera profile)
• Link: USB 2.0 high speed, 480 Mbps signaling (the bus's rated rate)
• IMU path: polled through a UVC extension unit; one call returns one latest sample, no FIFO
• Measured at 30 fps: about 140 reads/s, median read 4.85 ms, 95th percentile 19.9 ms
• Measured at 60 fps: 42-46 reads/s, median read 17.5 ms, 95th percentile 32 ms
• Timestamp counter: 1 MHz in the reported test, 1 µs per tick
• Calibration diagnostic at the 30 fps profile: average interval 7.12 ms ± 5.27 ms

Rule in words: a polled latest-sample read costs the firmware's turnaround time, and once that turnaround reaches the frame period, delivery collapses to roughly one read per frame. The estimate that follows fits both measured points:

Arithmetic at 30 fps: T_frame = 1000 / 30 = 33.3 ms. The median read, 4.85 ms, is shorter, so ceiling = 1000 / 4.85 = 206 reads/s. The realized rate was about 140 reads/s, which is 140 / 206 = 0.68 of the ceiling.

Arithmetic at 60 fps: T_frame = 1000 / 60 = 16.7 ms. The median read, 17.5 ms, is longer, so ceiling = 1000 / 17.5 = 57 reads/s. The realized mid-range rate of 44 reads/s is 44 / 57 = 0.77 of the ceiling. The 0.68 to 0.77 spread across the two points is why the estimate uses 0.7.

Cross-check from an independent direction: the calibration diagnostic's average interval of 7.12 ms implies 1000 / 7.12 = 140 Hz, landing on the same 30 fps profile from the other side. Its jitter, 5.27 ms, is 5.27 / 7.12 = 74 % of the mean interval, so a filter expecting near-constant intervals will mis-weight most samples.

Result: doubling fps from 30 to 60 cut the delivered IMU rate by 140 / 44 = 3.2x, from about 140 Hz to 42-46 Hz.

Decision, with the threshold built in: compute 0.7 × 1000 ms / max(median read, frame period) for your own profile before designing around the IMU. If your odometry needs 100 Hz, 42-46 Hz at 60 fps fails and about 140 Hz at 30 fps passes. If the application needs 200 Hz, neither profile passes, and the fix has to change the acquisition path, not the settings.

What the two timestamps mark

imu_timestamp_raw is the camera-side counter, 1 MHz in the reported test, so 1 µs per tick. It rides inside the same polled read that carries the sample, and the 3-5 ms and 17-20 ms clusters in the stamp intervals are the turnaround pattern wearing a timestamp: delivery times, not samples on a fixed sensor grid. Between two delivered stamps, everything the sensor produced is missing, and no later read brings it back.

host_timestamp_ns is the host clock read at the call. In the reported test it matched the midpoint between the start and the end of the read call, which makes it the time of the read, not of the measurement. That is expected — the host clock cannot reach back into the sensor's past — but it means the value must not go to a filter as a sampling instant. With a 17.5 ms median wait at 60 fps, a stamp can lag its measurement by most of a frame period.

Decision: treat both stamps as bounds on the sample instant, calibrate the tick rate once against the host clock over a minute of idle capture, and reconstruct per-sample times by interpolating between frame-anchored stamps — the same move as when IMU and depth image timestamps need alignment. Keep the whole rig in one clock domain; mixing counter domains is its own failure mode, covered in IMU timestamp clock domains at lower frame rates.

Four fixes, in the order to try them

1. Cut the frame-event rate first. The fps knob dominates: 30 fps delivered about three times the IMU rate of 60 fps in the worked case. Drop fps to the lowest your application tolerates, then trim resolution or hold the compressed format if contention persists. Frame rate also interacts with exposure — when a camera cannot hold fps because exposure eats the frame time, the cause is different and covered in camera fps drops from exposure and USB link limits.

2. Re-time the sparse stream instead of discarding it. Interpolation between frame-anchored stamps is the zero-hardware fix. It creates no samples, but it puts honest times on the ones you have, which is what the filter actually consumes.

3. Split the IMU onto its own link for rig builds. The classic odometry rig keeps the inertial sensor off the camera: a small IMU module on its own serial or HID link, camera frames triggered in hardware, both streams stamped in one clock domain. The contention disappears because nothing shares the camera's firmware anymore, and triggering both devices together anchors the two streams. When the video side then moves to Ethernet, the trade-offs change class; the page on selecting USB webcams versus IP cameras for live video streaming walks that comparison.

4. Change the camera when the IMU must stay on it. Order hardware whose sensor FIFO is burst-readable and whose frame capture carries a hardware timestamp. A polled camera with firmware-set output rate and no FIFO access cannot be fixed in software, and burst paths still follow the rule that when capture runs faster than the link, the buffer has to absorb the burst.

With the IMU off the camera, the camera's only job is frames, and the right purchase is link headroom:

The LS-BCM-0003 is a 3.1 MP USB3 board camera with a global shutter and a 120 fps ceiling.

USB3 signaling runs at 5 Gbps, about ten times the 480 Mbps high-speed rate, and on a separate host controller it is a separate bus. The walkthrough on MIPI CSI-2 to USB bridge bandwidth shows how to size a link when the raw sensor stream, not compressed MJPG, has to fit. If the split goes to Ethernet instead, the MV-CB016-10GM-M-W is a 1.6 MP GigE board camera with a global shutter; its stream rides the network card and leaves every USB bus free.

Why the usual answers do not help

Separate threads or processes do not move the rate. In the measured case, video in one process with IMU polling in another, and video plus IMU in threads of one process, delivered the same interval clusters, because the wait lives in the camera firmware rather than in host scheduling.

A C++ build of the same SDK does not either. Native code trims host-side overhead measured in microseconds, while the median wait is 17.5 ms; the ceiling is the turnaround, whatever language submits the request.

Raising the sensor's output rate is not available on a polled camera of this type. The firmware manages it, the SDK exposes no setting, and a faster sensor behind the same polled delivery would simply drop more samples between polls. Depth pipelines hit the same wall through their data streams, as in 3D camera frame rate low on software trigger: mode and data streams.

Which fix fits which case

• 30 fps odometry, moderate dynamics: stay polled. Expect about 140 Hz delivered, interpolate the stamps, and keep the interval histogram healthy. Decision: no new hardware; re-check the profile after any fps or resolution change.
• 60 fps video required: the polled stream fails at 42-46 Hz. Either accept sparse inertial data with interpolated stamps for slow motion, or split the IMU onto its own link. Decision: change the rig, not the settings.
• Fast rotation at 200 Hz-class gyro rates: polled latest-sample delivery on a shared bus cannot reach it, because the ceiling is one read per frame period. Decision: external IMU on its own link, or a camera whose FIFO and hardware timestamps are part of the product.
• Choosing a camera now: make FIFO access, a visible output rate and hardware frame timestamping selection criteria, not questions for the debugging week. The camera finder filters by interface and shutter type, and depth rigs should also browse the 3D sensors category.

Checks before you commit

• Record 60 s of IMU reads at your target fps and plot the interval histogram. Healthy is one tight cluster; turnaround failure shows as clusters at the firmware turnaround and at the frame period.
• Log median and 95th-percentile read durations. If the median reaches the frame period, budget one read per frame and nothing better.
• Compare host_timestamp_ns against a loop-timed read once. If it tracks the call midpoint, it stamps the read, not the sample.
• Calibrate the timestamp tick rate once; 1 MHz, or 1 µs per tick, was an assumption in the reported test, and a 2x tick error doubles every interval you derive.
• Run your calibration toolbox's rate diagnostic before field trials. Jitter above a third of the mean interval flags a stream the filter will fight.
• Confirm the camera exposes FIFO burst reads and an output-rate setting before ordering. If neither exists, plan for sparse data or an external IMU from day one.

Building the rig

The short version: polled IMU reads and video frames share one firmware turnaround, so the IMU rate is budgeted by the frame period the moment turnaround reaches it. Keep fps low enough that reads fit between frames, interpolate the stamps you get, and when the numbers still miss the filter's floor, take the IMU off the camera or buy one that exposes the timing path. Among board-level cameras, filter for the interface that splits the traffic on your host, or scope the whole chain in the camera configurator. More on the depth side sits in the 3D camera guides, and if you want a second pair of eyes on a bill of materials, ask through the contact page.