IMU camera time synchronization for visual SLAM

An IMU feeds a SLAM pipeline a 200-400 Hz stream of angular rates and accelerations while the camera delivers 30 images a second. Fuse them with the wrong time relationship and the map smears on every turn, the trajectory jumps, and the filter eventually gives up on the IMU entirely. The cause is rarely the sensor: it is which clock stamped each stream, and what your code does with those stamps. This guide walks the diagnosis in the order it bites: force one clock domain, interpolate the IMU to each frame's exposure-start timestamp, prove the alignment with a rotation test, and get off the virtual machine before blaming the camera. Every number below is worked for 30 fps video against a 200 Hz IMU, a common stereo depth camera configuration.

Why is the IMU not synchronized with my camera images?

Two clock domains are the usual cause: the IMU is stamped on the camera's hardware clock while your code stamps images on the host clock, and a constant offset plus a few milliseconds of jitter is enough to break visual-inertial SLAM. "The IMU data is wrong" on a stereo depth camera almost always means "the gyro disagrees with the images during rotation" — a time-alignment problem, not a sensor fault.

Four causes cover nearly every case:

1. Arrival stamping. The timestamp is written when a frame or IMU packet reaches your process, not when the measurement happened. USB delivery and driver queues add latency that changes from packet to packet.
2. A virtualized host. A virtual machine passes USB through in batches and virtualizes the clock, turning a fixed offset into wandering jitter.
3. A driver or queue fault. An IMU stream publishing at a wandering rate, or a frame queue delivering old packets late, scrambles the time relationship even with good stamps.
4. The wrong reference instant. Frame arrival, end of readout and exposure start are three different times, as much as one full frame apart at 30 fps.

Each cause has its own test below. If the pipeline instead stalls waiting for IMU messages at all, that is a stream-configuration fault rather than sync — see RGB-D odometry stuck waiting for IMU.

What timestamp belongs on every frame and IMU sample?

Every stream must carry the time the measurement happened, not the time your process read it: a frame gets its exposure start, an IMU sample gets its sample time, both on one clock — 33.3 ms apart at 30 fps, 5.0 ms apart at 200 Hz. Enable hardware (device-side) timestamping in the camera API, the option that makes the sensor stamp its own packets, and the alignment survives queues, host load and transport. Hardware time stamping aligns independent sensors at the microsecond level; host arrival stamps are good to a few milliseconds at best.

Two conventions to fix once and keep:

• Exposure start or midpoint, never both. With a 10 ms exposure the midpoint sits 5.0 ms after start, which is 0.225° of rotation at 45 °/s. Pick one instant and use it for every frame. Aligning measurement events to exposure is the whole subject of camera IMU synchronization: aligning events with exposure, and the traps at lower frame rates are laid out in camera IMU timestamp clock domains at lower frame rates.
• Know what the driver means by "timestamp". Exposure start is scheduled inside the camera; frame arrival is not. Per-frame chunk data is the clean way to receive a timestamp and the exposure value together — chunk data on camera frames: timestamp and exposure per image.

A rolling shutter complicates the instant: each line exposes at a different time, so a single stamp can only be a convention, first line being the common one. A global shutter makes the stamp exact, which is one reason SLAM builds favor it. How exposure and readout are scheduled, and therefore when the driver can know the exposure instant, is the subject of maximum frame rate on a hardware trigger: exposure plus readout time.

If you hang other instruments off the same rig, the same rule applies at bench level: one trigger or one clock for everything, the discipline behind synchronizing DAQ tasks and a camera.

How do I interpolate IMU samples to an exposure timestamp?

Linear interpolation between the two IMU samples that bracket the frame timestamp is the standard method, and at 200 Hz it costs one multiply: with samples 5.0 ms apart you never query the gyro more than 2.5 ms from a measurement. Do it inside the preintegration, the summed gyro increment between consecutive frame timestamps, rather than resampling the IMU onto an invented grid.

Worked example, one gyro axis, 30 fps frames, 200 Hz IMU:

• IMU sample 1: t = 1.030 s, omega = -12.4 °/s
• IMU sample 2: t = 1.035 s, omega = -8.9 °/s
• Frame exposure start: t = 1.032 s

alpha = (1.032 - 1.030) / (1.035 - 1.030) = 0.002 / 0.005 = 0.40

omega at exposure start = -12.4 + 0.40 × (-8.9 - (-12.4)) = -12.4 + 0.40 × 3.5 = -11.0 °/s

That rate feeds the preintegral for this frame. Across one 33.3 ms frame period you integrate 6.7 gyro samples at 200 Hz, or 13.3 at 400 Hz; interpolate at the two frame boundaries and integrate between them. The same pattern applied to depth streams is walked through in IMU and depth image timestamps: interpolation for alignment.

How do I run a rotation test that proves the sync?

Rotate the camera at a steady, known rate and compare the frame-to-frame rotation measured from the images with the gyro angle integrated between exposure timestamps: the mismatch in degrees equals rate times the time offset, so 45 °/s times 33.3 ms is 1.5°. This is the one test that separates a time offset from a calibration error, because a calibration error does not scale with speed.

1. Log every stream raw, with its device timestamps, into a file you can replay — recording 3D camera data to H5 files for offline processing is one format that preserves them.
2. Mount the camera on a bench stand. Hold still 2 s, rotate smoothly about one axis at about 45 °/s for 2 s — a 90° swing, 60 frames at 30 fps — then hold still again.
3. Integrate the gyro between consecutive exposure-start timestamps: at a steady 45 °/s, each frame-to-frame increment is 45 × 0.0333 = 1.5°.
4. Measure the same increment from the images by tracking features between consecutive frames.
5. Divide the mean mismatch by the rate: 1.5° at 45 °/s gives 1.5 / 45 = 0.033 s, one frame period — the classic signature of arrival stamping.

Repeat the sweep at 15 °/s. A true constant offset halves with the rate; queueing artifacts do not scale that cleanly, which is how you tell them apart.

Why does synchronization break inside a virtual machine?

A virtual machine passes USB through with variable latency and virtualizes the host clock, so frames and IMU packets arrive late by an amount that changes packet to packet — commonly tens of milliseconds of wander on a guest that would hold a few milliseconds on bare metal. The fatal part is that the error is not constant: an estimator can absorb a fixed offset, but wander smears the map whatever you calibrate.

The typical setup is a Linux guest running the whole SLAM stack on a Linux host. Everything looks healthy in the vendor viewer, and the pipeline still runs — it just drifts on rotation. Two things go wrong at once:

• USB passthrough batches transfers. The guest receives bursts of frames and IMU packets, so arrival stamps jitter, and any host clock the driver reads jitters with them.
• The guest clock is virtualized. Even a driver that stamps in its receive thread reads a clock that stops and starts with the machine.

The fix is to leave the VM for measurement: dual-boot a native install, or move capture to a bare-metal host and ship the data out. Host software can still add latency of its own — 3D camera capture time slower in the API than in the viewer dissects that — but on native hardware it is small and stable, and hardware timestamps bypass it entirely.

Keep the physical layer boring while you are at it: a shielded USB3 cable of the right length removes the dropouts and re-enumerations that look like timestamp faults on a busy host.

The LS-CBL-0219 is a 3 m USB3.0 camera cable, high-flex, built for drag-chain use, so it also survives a moving rig. More lengths and connectors are under USB3 camera cables.

Which settings break IMU timestamps before the hardware does?

Three settings cause most bad IMU timestamps: host-side (arrival) stamping left enabled, a driver publishing the IMU at a wandering rate instead of the configured 200 or 400 Hz, and frame queues that deliver old packets behind new ones. Check them in that order; all three are visible in a five-minute log.

• Timestamp source. Enable hardware or device time, then prove it took: frame and IMU timestamps must advance monotonically with no jumps or freezes — camera image timestamps frozen or jumping: fix the clock source covers the failure modes.
• IMU publishing rate. Log the gaps between consecutive IMU timestamps. At a configured 200 Hz they must sit at 5.0 ms with small scatter. One stereo depth camera's ROS driver shipped an IMU stream publishing at an unsteady rate, fixed in its 2.2.11 release — a reminder that a driver update is a five-minute check.
• Queue behavior. Set queue depth to 1 and drop late frames during the test. If timestamps arrive out of order, the queue is handing you stale packets: image timestamps out of order: frames and IMU data scramble.

Start every test from a known configuration — leftover settings from an earlier session are a classic way to chase a sync ghost that is really a mode mismatch: camera test flaky on leftover settings.

How much sync error can visual SLAM tolerate?

Aim for better than 1 ms between the IMU and image timebases; a constant offset of a few tens of milliseconds can be estimated online by the estimator, but jitter of the same size smears the map and no parameter absorbs it. The tolerance scales with angular rate, because a time error becomes an angle error the moment the rig turns: 1 ms at 90 °/s is 0.09°, while 33.3 ms at a gentle 5 °/s is only 0.17°.

That gives one decision per use case:

• Handheld or mobile robot, turning at 45-90 °/s: hardware timestamps, no compromise. Interpolation to exposure start, target under 1 ms.
• Fixed conveyor or rail inspection, under 5 °/s: a measured constant offset of tens of milliseconds is survivable. Interpolate to exposure start anyway; it is free.
• Multi-stream rigs, color plus depth plus IMU: every pair needs its own offset. Depth color timestamp drift that grows while running and depth and color synchronization: check the right timestamps cover the pair-by-pair method, and whole-frame color lag usually traces to matching by sequence number — color lags depth by one frame.

The gyro is the short-term truth between frames, and the filter weights streams by how well they agree in time. The mechanics of that weighting are worth understanding before you tune, as in this walk-through of a Kalman filter for IMU and camera fusion.

What should I check before rewiring or ordering parts?

Before you change hardware, force one clock domain and log it: hardware timestamps on, exposure metadata enabled, a fixed 200 or 400 Hz IMU rate, and a native host — that combination addresses every cause in the list above. Then work the checklist:

• Hardware timestamping enabled, proven by a monotonic-timestamp log.
• Exposure metadata received with every frame.
• IMU gaps logged: 5.0 ms at 200 Hz, 2.5 ms at 400 Hz, low scatter.
• Native host, with no virtual machine between camera and driver.
• Rotation test passed at two speeds, the offset scaling with rate.
• Raw logging that preserves timestamps, for replay after any change.

If the diagnosis lands on the camera itself — say the rig needs stereo structured-light sensing at the SLAM front end — compare the stereo options first:

The MV-DLS300P-04 and MV-DPS200P-02 are both stereo structured-light cameras, and every clock-domain rule above applies to them unchanged. For the 2D side, the area scan cameras finder filters on shutter type; the wider stereo selection sits under stereo 3D cameras.

When an offset survives all of this, send the log — both streams' timestamps around one rotation sweep — via the contact page, compare sensors in the camera finder, and browse the 3D vision topic for set-up guides that assume sync is already solved.