IMU and depth image timestamps: interpolation for alignment

You log a depth session, line the streams up, and find that no IMU sample carries a depth frame's timestamp. Nothing is broken. The depth stream runs at 30 Hz, the IMU at 400 Hz, both are stamped from the device clock, and each stream arrives through its own callback. The fix is arithmetic, not configuration: bracket every frame time between two IMU samples, interpolate gyro and accel to the exact frame stamp, and reserve the nearest sample for scenes where its error budget clears. The numbers below use a 30 Hz depth stream and a 400 Hz IMU; scale each interval to your own rates.

Why no IMU sample ever lands on a frame timestamp

The two rates never meet. A 30 Hz depth stream produces one frame every 1/30 = 33.3 ms; a 400 Hz IMU produces one sample every 1/400 = 2.5 ms. The chance of an exact stamp tie is essentially zero, and the driver does not merge the streams: images and motion packets are delivered through separate callbacks, each when its own packet lands.

What the streams share is the clock. On depth cameras with a built-in IMU, the IMU packets are normally stamped from the same hardware clock as the depth sensor, so the numbers are directly comparable even though nothing pairs them. That gives the working rule: treat every frame time as falling somewhere inside a 2.5 ms gap between two IMU samples. The nearest sample is at most 1.25 ms away — half the IMU interval — and interpolation pins the value to the frame time itself. Everything below turns that rule into code, an error budget and a test.

What you need before you align

• Device stamps, not host receive times. If your driver fills the headers with receive time, fix that first: receive jitter on USB and GigE is milliseconds, the same size as the error you are about to remove. The camera IMU timestamp clock domains guide shows how the mismatch hides at low frame rates.
• The stamp unit for each stream — µs, ns or ticks — and each stream's rate.
• A buffer of IMU samples reaching at least one interval ahead of the newest frame: 2.5 ms at 400 Hz, or 5-10 samples in practice.
• Your worst-case rotation rate, measured or estimated, for the error budget in Step 4.

If you are still choosing the camera side, the 3D sensor category below lists depth cameras, profilers and stereo sensors with their stream types.

Step 1: Put both streams on one clock basis

1. Confirm the stamp source: frame and IMU metadata should report a device or hardware timestamp, not a receive time.
2. Convert every stamp to one unit and one epoch before you subtract anything.
3. Handle wraparound: a 32-bit microsecond counter wraps after 2^32 µs = 4,294.97 s ≈ 71.6 min, so any long session rolls over mid-record. The timestamp rollover and monotonic time mapping guide walks the wrap-safe mapping.
4. Sanity-check deltas: consecutive depth stamps should sit at 33,333 µs ± one scan time; consecutive IMU stamps at 2,500 µs ± jitter. Deltas bouncing by milliseconds mean you are still reading receive times.

Step 2: Bracket every frame timestamp between two IMU samples

Keep IMU samples in a sorted queue. For each frame, advance through the queue to the last sample at or before the frame time; that sample and the next one are the bracket. A frame older than the queue head arrived too late — widen the buffer or drop it. A frame newer than the newest sample must be held for one IMU interval, 2.5 ms at 400 Hz, until the closing sample lands. Never extrapolate.

# Illustrative pseudo-code: these names are ours, not an SDK's.
imu = []                # list of (t_us: int, gx, gy, gz: float, rad/s), oldest first
frame_t = 43_777_373    # int, depth frame stamp in microseconds
i = 0                   # int, index of last sample with t_us <= frame_t
while i + 1 < len(imu) and imu[i + 1][0] <= frame_t:
    i += 1
if i + 1 >= len(imu) or imu[i + 1][0] <= frame_t or imu[i][0] > frame_t:
    raise ValueError("no bracket: hold the frame or widen the buffer")
t0, t1 = imu[i][0], imu[i + 1][0]   # ints, now guaranteed t0 <= frame_t < t1

The same bracket-and-match pattern joins any asynchronous streams by time. If you also align exposure events rather than whole frames, aligning camera and IMU events with exposure applies the identical logic, and matching asynchronous vision results with timestamps uses it for results against trigger records.

Step 3: Interpolate gyro and accel to the frame time

Linear interpolation between the bracketing samples:

The same form gives each accelerometer axis, and the stamp units cancel inside the ratio, so µs, ns or ticks all work. Over a 2.5 ms gap a gyro's change is effectively linear, so the interpolation error is set by sensor noise, not by the method.

Worked example — stamps illustrative, in µs, during a fast pan:

• Frame: t = 43,777,373 µs.
• Samples: t0 = 43,776,250 µs with gz0 = 0.50 rad/s; t1 = 43,778,750 µs with gz1 = 0.90 rad/s. The interval is 2,500 µs, and the frame sits 1,123 µs after t0, so the nearest sample is t0.
• Fraction: (43,777,373 − 43,776,250) / (43,778,750 − 43,776,250) = 1,123 / 2,500 = 0.4492.
• Interpolated: gz = 0.50 + (0.90 − 0.50) × 0.4492 = 0.50 + 0.180 = 0.68 rad/s (0.67968 unrounded).

The nearest sample would have returned 0.50 rad/s — 26% low on a number your fusion or measurement consumes directly. That gap is why Step 4 tells you when nearest is safe anyway.

Step 4: Budget the nearest-sample error against your rotation rate

The angular error of the nearest sample is rotation rate times offset:

Convert rates with 1°/s = 0.01745 rad/s. At a 60°/s inspection pan, ω = 1.047 rad/s: Δθ = 1.047 × 0.00125 = 0.00131 rad = 0.075°. At a 300°/s flick, ω = 5.236 rad/s: Δθ = 5.236 × 0.00125 = 0.00655 rad = 0.375°.

Translate to pixels on an illustrative depth image of 640 pixels across an 87° field of view: 87 / 640 = 0.136° per pixel. The 60°/s error is 0.075 / 0.136 ≈ 0.55 pixels — invisible. The 300°/s error is 0.375 / 0.136 ≈ 2.8 pixels — enough to smear an overlay or shift a projected edge.

Decision: interpolate by default; nearest is acceptable only while ω × 1.25 ms stays below your tolerance in radians. On the reference camera, one pixel equals 0.136°, and nearest crosses that at ω = 0.136° / 1.25 ms ≈ 109°/s.

How to verify the alignment worked

Integrate the interpolated gyro across one frame interval and compare with rotation measured from the images. Take the 60°/s pan: two frames sit 33.3 ms apart, so Δθ = 1.047 rad/s × 0.0333 s = 0.0349 rad = 2.0°. Track features between the same two depth or RGB frames and measure the same rotation. The two numbers should agree within budget — 0.1° is a fair bar at 60°/s, because alignment alone contributes at most 0.075°. A consistent excess near 0.3° is gyro bias (0.005°/s × 60 s = 0.3°), not alignment.

Then sweep the offset: shift the IMU stream by candidate offsets from −5 ms to +5 ms in 0.5 ms steps and cross-correlate gyro rate against image-derived rate. A peak at 0 ms confirms the shared clock; a peak elsewhere reports residual skew in milliseconds, and a sawtooth or step pattern means stamps are re-based mid-run, which the IMU timestamp jumps guide helps diagnose.

A 60 s test run produces 1,800 frames and 24,000 IMU samples at 400 Hz. Alignment passes when every frame found a bracket and the integrated-versus-image residual stayed inside budget.

Common mistakes that look like alignment faults

• Reading host receive times instead of device stamps — millisecond jitter swamps the 1.25 ms budget.
• Extrapolating past the newest sample instead of waiting 2.5 ms; the error is unbounded and grows with the gap.
• Mixing units across threads: a µs-versus-ms slip shifts every stamp by a factor of 1,000 and mimics total desync.
• Applying gyro values in the IMU's own axes; rotate them into the camera or world frame first — the IMU optical frame transform guide covers the check.
• Blaming alignment for drift that is bias: 0.01°/s of uncorrected bias is 0.6° per minute of integration.
• Fusing streams whose rates or resolutions do not match; registration breaks the same way depth-to-color pairing breaks on a resolution mismatch — match the streams before you fuse.
• Letting the driver resample the IMU to the frame rate and treating the result as measured data; grouping callbacks is not synchronization.

When the usual answer is wrong

• Fast motion with overlays or metrology: always interpolate. Nearest crosses one pixel at about 109°/s on the reference camera, and hand-held scans exceed that routinely.
• Filter-based fusion (EKF, SLAM): feed interpolated values with the frame stamp. Never stamp a nearest sample as if it were measured at frame time — the filter integrates the inconsistency.
• Hardware triggering: a shared trigger line locks frames to events, as the stereo paired-capture trigger guide shows, but the IMU still free-runs, so bracket-and-interpolate remains the tool for IMU-to-frame alignment.
• Long recordings: interpolate at replay time from raw pairs. Storing pre-interpolated data freezes today's clock offsets into every future analysis. For stereo depth rigs, timing residue shows up as depth jitter between frames; the depth jitter and IMU drift troubleshooting guide walks those stereo-specific symptoms.

Alignment checklist and next step

• Both streams stamped by the device clock; deltas read 33,333 µs and 2,500 µs.
• One unit and one epoch, wrap-safe; the offset sweep peaks at 0 ms.
• Buffer reaches one IMU interval past the newest frame; zero unbracketed frames per run.
• Gyro and accel interpolated, never extrapolated; the 2.5 ms hold-back is accepted.
• Integrated-versus-image residual inside budget at your fastest rotation.

When you are specifying the hardware side, the camera finder filters depth and stereo sensors by stream type and resolution, and the configurator below assembles camera, mount and controller into one build:

Other depth-timing guides sit on the 3D vision topic page, and questions on a specific integration go through the contact page.