A gyro-and-accelerometer track at hundreds of hertz and image streams at 30 fps never line up sample for sample, so a frameset will not hand you a matched IMU reading. The fix is bookkeeping, not hardware: queue the inertial samples by timestamp, interpolate them onto each image stamp, and integrate the gyro between consecutive frames. The result is an orientation for every color and depth frameset, expressed relative to your starting pose, which is what per-frame extrinsics need. This guide gives the procedure for a 30 fps camera with a 200 Hz IMU, the verdict on the 10 ms stamp gap inside one frameset, and the reason a static accelerometer still reads 9.81 m/s².
Decide the time base before you write code
Take image and IMU stamps from the device clock the SDK reports, never from the moment data lands on the host; transfer and batching delay arrival by milliseconds that have nothing to do with motion (see frame timestamp vs system time: why clocks differ). Nothing else about the streams is wrong: color and depth of one frameset are stamped up to about 10 ms apart, and a 200 Hz IMU runs 6.7× faster than 30 fps images. You build exactly three things: a per-sample queue on device time, linear interpolation onto the stamps you care about, and trapezoidal integration between frames.
The companion guide on interpolating IMU stamps onto depth frames works the depth-only case; the same interpolation carries the color stream unchanged.
What you need
- A depth camera whose SDK delivers gyroscope and accelerometer samples beside the image streams; the camera finder filters the range by interface and resolution.
- The SDK's frameset call and its inertial callback or queue, both reporting device time.
- A bench with 60 s of stillness for the bias measurement and one known turn, 90° being ideal, for the trueness check.
Two 3D cameras of this class, as the shop lists them:
- Type
- Depth / ToF camera
- Type
- Stereo / structured light
The procedure
- Stamp on device time. Read the timestamp of every image and IMU sample from the device-clock field the SDK provides. If you must log on host time, record the device-to-host offset once and apply it to every stamp.
- Queue IMU samples by timestamp. The inertial callback pushes each sample into a buffer as it arrives. Arrivals come in bursts; that is transport, not motion — the sample-rate jitter guide shows why the batches appear.
- Guard the stamp order. Discard any record whose stamp does not advance. Long recordings can scramble order, and one stale stamp poisons the integration.
- Interpolate onto the frame stamps. For each frame stamp t_k, find the IMU samples i and i+1 with t_i ≤ t_k < t_{i+1} and apply the first formula above.
- Integrate between frames. Feed the interpolated rates at t_{k−1} and t_k through the second formula and accumulate θ into a quaternion or rotation matrix. Gyro goes in rad/s, angle comes out in rad.
- Seed roll and pitch from the first accelerometer sample. Normalize it: that direction is gravity, so it fixes roll and pitch at start. Yaw starts at zero, which is what makes the pose relative to your starting point.
- Attach the pose to each frameset. Store (t_k, R_k) alongside the color and depth frames. If you merge color and depth geometry while the rig moves, first rotate depth by ω·(t_d − t_c): at the 10 ms stamp gap and a 30°/s pan that is 0.3°.
The queue-and-drain skeleton, sitting inside your acquisition loop, with example variable names:
double t_prev = 0.0; // previous frameset stamp, s, device clock
double theta = 0.0; // accumulated yaw since start, rad
struct ImuSample { double t; double w; }; // t: stamp in s; w: yaw rate in rad/s
std::deque<ImuSample> imu_q; // the inertial callback fills this
auto frameSet = pipeline.waitForFrames(1000);
if (frameSet == nullptr) { /* timeout: no frameset within 1000 ms */ }
auto colorFrame = frameSet->colorFrame();
auto depthFrame = frameSet->depthFrame();
double t_k = /* frameset stamp from the SDK's device-time field, s */;
if (t_k <= t_prev) continue; // test: frame stamps must advance
while (imu_q.size() >= 2 && imu_q[1].t <= t_k) {
theta += 0.5 * (imu_q[0].w + imu_q[1].w) * (imu_q[1].t - imu_q[0].t);
imu_q.pop_front();
}
t_prev = t_k; // theta now spans start to t_k
Worked example: 30 fps camera, 200 Hz IMU
Frame period T = 1/30 s = 33.3 ms. IMU period = 1/200 Hz = 5 ms, so 33.3 / 5 = 6.7 samples sit in every frame window.
Interpolation: an image is stamped 10.007 s; the bracketing IMU samples sit at 10.005 s and 10.010 s with yaw rates 0.520 rad/s and 0.528 rad/s. The weight is α = (10.007 − 10.005) / 0.005 s = 0.4, so ω(10.007) = 0.520 + 0.4 × (0.528 − 0.520) = 0.5232 rad/s, which is 30.0°/s.
Integration: a steady pan of 30°/s = 0.524 rad/s across one 33.3 ms frame gives Δθ = 0.524 rad/s × 0.0333 s = 0.0174 rad = 1.0° per frame, so a 1 s pan accumulates 30°. The enemy is bias: an uncorrected 0.1°/s adds 0.1°/s × 60 s = 6° of yaw error per minute, which is why the bench test below measures bias first.
The 10 ms color-depth gap inside one frameset is normal
Ask for a frameset and the color and depth frames carry system stamps about 10 ms apart. The two sensors do not start exposure and readout at the same instant, and the pipeline stamps each stream as its data lands, then groups them into one frameset anyway. Read the gap as metadata, not a fault: pair by frameset membership, and use the two stamps whenever the motion math needs them. Discarding framesets over the gap, or waiting for a zero gap, only starves your pipeline. The same rule holds if you ingest point clouds and RGB separately — pair frames by timestamp, never by arrival order.
Why a static accelerometer reads 9.81 m/s² and why Y is not zero
An accelerometer measures specific force: held still, it reports the reaction to gravity, about 9.81 m/s² on the axis pointing up. A level camera reading roughly 0, 0, 9.81 m/s² on x, y, z is exactly right. A nonzero Y with the camera stationary means gravity leans onto that axis: the mount is tilted.
A static Y of 0.50 m/s² gives φ = asin(0.50 / 9.81) = asin(0.0510) = 2.9°, so the mount sits 2.9° off level about the X axis. Whatever tilt does not explain is bias. Park the rig on a machined level surface: the leftover steady reading on each horizontal axis is the bias to subtract before integrating.
Verify the alignment on the bench
- Stillness test. Integrate for 60 s with the rig untouched. Whatever wander you see is your bias: 3° in a minute means 0.05°/s; subtract it and rerun.
- Trueness test. Turn the rig exactly 90° about one axis. The integrated angle should land on 90° within the bias drift and interpolation error you measured in the stillness test; a larger miss points to a stamping fault, not the sensor.
- Stamp audit. Log one minute of stamps and check three things: frame stamps strictly increase, every frame finds bracketing IMU samples (the queue never runs dry), and no stamp steps backward or resets. A reset mid-run is a clock fault, not motion — the IMU timestamp jump guide walks the diagnosis.
Common mistakes
- Mixing host arrival time into the math — it adds milliseconds of pure transfer delay to every sample.
- Demanding equal color and depth stamps inside one frameset; the 10 ms gap is normal.
- Integrating raw gyro output with no bias measurement: 0.1°/s becomes 6° per minute.
- Taking the nearest IMU sample during fast motion: at 200 Hz the worst pick is 2.5 ms off, which at 300°/s is 0.75° of orientation error.
- Treating batched IMU arrivals as bursts of motion; batches describe the bus, not the scene.
When to hand off to a visual-inertial estimator
Gyro integration gives orientation relative to start and nothing more: drift grows without bound, and position from double-integrated acceleration is unusable within seconds. For absolute, metric trajectories — navigation, mobile robotics, moving-base scanning — run a visual-inertial odometry estimator that fuses these same IMU samples with the images themselves. Also check whether your camera already delivers a fused pose stream, because a minority do and it would replace this whole pipeline (which depth cameras output a pose stream). For per-frame extrinsics relative to your starting pose, the seven steps above are the whole job.
Choose the camera side of the rig from the 3D sensor range below, then configure the camera stack if you want lens, lighting and controller matched to it. More timing guides sit under the 3D camera topic.
Laser profilers and stereo 3D cameras
170 models · from $699.00Questions
What IMU rate should I set for a 30 fps camera?
Aim for 5 to 10 times the frame rate: 150–300 Hz for 30 fps. At 200 Hz the 33.3 ms frame window holds 6.7 samples, so linear interpolation error stays small next to gyro noise at ordinary pan speeds. Below about 100 Hz the window holds fewer than 4 samples; interpolation still works for slow motion but degrades fast during quick turns, so use the highest rate the camera offers.
Why do IMU samples arrive in batches rather than one at a time?
The USB or Ethernet bridge packs several inertial samples into each transfer, so they land on the host in bursts while every sample keeps its own device timestamp. That is transport behavior, not motion. Drain the queue strictly by timestamp, and never read arrival time or batch size as a measurement of what the camera was doing.
Does the accelerometer give me position?
No. Double-integrating acceleration drifts within seconds even with good bias correction, because each integration multiplies the error. Here the accelerometer has one job: the gravity direction at startup, which fixes roll and pitch. Yaw comes from the gyro integration, and any position estimate must come from a visual-inertial estimator that also uses the images.
How long can I integrate before drift ruins the pose?
Your measured bias sets it, not a fixed number. The 60 s stillness test gives the bias directly: at 0.05°/s yaw error grows 3° per minute, at 0.1°/s it grows 6°. Re-zero at every stop, fold the bias into your filter if you run one, and size your runs so a few degrees of accumulated yaw still leaves the inspection inside tolerance.

