Playback IMU data from a bag file: use a second pipeline

A bag file should make the offline half of a vision project easy: capture once, then replay depth, color and motion data as often as you like. Many engineers meet the opposite the first time they enable accel and gyro next to depth and color in one playback pipeline. The display stalls, the wait times out, and the file replays its first few seconds in a loop. This guide names the symptom, states the two-stream-type rule behind it, shows the two-pipeline fix that leaves the recording untouched, and closes with the frame-count check that proves the whole file played.

Why does bag playback loop back to the start after a few seconds?

The playback restarts because the one pipeline you started is waiting for a single frameset that carries depth, color, accel and gyro together. When that combined wait times out, the playback stops and starts over from the first recorded frame.

The reset typically lands inside the first five seconds of the file, and it only appears once accel and gyro are enabled next to depth and color. Switch the config back to depth and color alone and the same file plays end to end. That is the quickest proof that the recording itself is fine.

It reads like a corrupt file or a viewer bug, and it is neither: the recording is intact, and the identical file plays cleanly once the streams are grouped differently. The grouping, not the data, is what the next section pins down.

Which stream combinations play back together, and which stall?

In recorded playback the pattern is consistent: any two stream types play together, but all three stream types at once make the combined wait fail. The pipeline never returns the frameset your loop needs.

Depth with color plays. Depth with the motion module plays. Color with the motion module plays. Depth, color and the motion module together do not, and that is exactly what a depth-plus-color-plus-motion config requests. The motion module counts as one stream type even though it delivers both an accel and a gyro stream, so depth with gyro and accel is still two types and plays.

That gives the decision rule for any config: keep each playback pipeline to two stream types at most. Image types, depth and color, go in one pipeline; motion types, accel and gyro, go in another. An enable-all-streams call guarantees three types whenever the file carries both image and motion data, so it recreates the stall by construction.

How do you confirm the bag file itself is not the problem?

Play the recording with the motion streams disabled: if depth and color alone run from the first frame to the last, the file and its recorded streams are intact. The fault is in how the playback pipeline was configured.

Run that pass headless, with no display, and judge it by frame count rather than by eye:

Take duration in seconds and fps in frames per second, so the result is a frame count. For a 60 s recording captured at the 30 fps the config requests, the depth stream owes you 60 × 30 = 1,800 frames, and playback should take about 60 s of wall time. A looped pass delivers 5 × 30 = 150 frames and then returns to the start; three clean-looking passes still leave you at 450 of the 1,800 frames. A count that reaches 1,800 with wall time matching the file length clears the recording, and the stall is config-side.

Frames that go missing during a clean depth-and-color pass have their own causes and fixes; the missing frames in recorded bag playback guide walks through them. A loop that resets cleanly to the first frame is the grouping fault, not dropped data.

How do you play motion data without restarting the file?

Start a second pipeline on the same bag file and enable only the accel and gyro streams in it. The image pipeline waits on depth and color, the motion pipeline waits on accel and gyro, and each wait can be satisfied.

Both configs point at the same file through the same enable-from-file call, so both pipelines read the identical recording. The image config enables depth and color at their recorded resolution and rate. The motion config enables accel and gyro with the motion format for their frames. Each pipeline starts on its own config and returns its own profile; keep both profiles, because they list the streams and rates you will check later.

From there the main loop simply reads two pipelines instead of one. The image wait returns image frames only, the motion wait returns motion frames only, and neither wait ever has to satisfy three stream types at once. That is the whole fix.

What does the two-pipeline playback code look like?

The working script keeps two pipeline objects, points each config at the same input file, and reads both in one loop. Every frameset you receive contains only stream types that are known to arrive together.

# One pipeline for images, one for motion; both read the same file
rgbd_pipeline = rs.pipeline()
rgbd_config = rs.config()
rs.config.enable_device_from_file(rgbd_config, args.input)
rgbd_config.enable_stream(rs.stream.depth, 640, 480, rs.format.z16, 30)
rgbd_config.enable_stream(rs.stream.color, 640, 480, rs.format.rgb8, 30)
rgbd_profile = rgbd_pipeline.start(rgbd_config)

imu_pipeline = rs.pipeline()
imu_config = rs.config()
rs.config.enable_device_from_file(imu_config, args.input)
imu_config.enable_stream(rs.stream.accel, rs.format.motion_xyz32f)
imu_config.enable_stream(rs.stream.gyro, rs.format.motion_xyz32f)
imu_profile = imu_pipeline.start(imu_config)

def motion_xyz(frame):
    motion = frame.as_motion_frame().get_motion_data()
    return np.asarray([motion.x, motion.y, motion.z])

while True:
    frameset = rgbd_pipeline.wait_for_frames()
    depth_image = np.asanyarray(frameset.get_depth_frame().get_data())
    color_image = np.asanyarray(frameset.get_color_frame().get_data())

    imu_frameset = imu_pipeline.wait_for_frames()
    samples = [motion_xyz(imu_frameset[i]) for i in range(imu_frameset.size())]
    # tag each sample by its stream, not by its position, before you store it

In the listing, rgbd_pipeline and imu_pipeline are pipeline objects, args.input is the file path string your command line already carries, and samples is a list of three-element arrays. Every call comes from the SDK's own playback interface: the file-enabled config, the stream and format names, the motion frame conversion, and the frameset size query. To prove it works, run the loop headless and print the depth frame count and elapsed wall time at the end; the numbers to expect come from the check below.

Framesets from file playback keep the layout you recorded. If you also need the color frame lined up with the depth frame pixel for pixel, the RGB and depth alignment method for bag playback covers that as its own step.

Does the same split apply to a live camera?

Live capture hits the same wall: with all three stream types enabled, image frames keep arriving while accel and gyro freeze after a few seconds. Run the same split and keep the 10 s first-frame timeout that file playback does not need.

The timeout discipline for live capture waits up to 10 s for the first frameset after start, then treats any single wait longer than 200 ms as a miss and moves on instead of blocking. Recorded files usually play with the default wait, because the frames arrive on demand. Keep the long timeouts for live streaming, where the first combined frameset can genuinely take seconds.

A live motion stream that stops updating can also have hardware causes: power delivery, hub bandwidth, driver buffers. If the split does not cure a freeze on a live camera, work through the live IMU stream freeze checks before rewriting the config again.

Once motion data plays reliably it becomes an input in its own right, for example triggering the depth capture from IMU motion when a part moves into view.

How do you verify the whole file played end to end?

Let the playback run with no display at all and count frames: a 60 s recording captured at 30 fps must deliver about 1,800 depth frames in about 60 s of wall time. A loop that restarts shows the same early timestamps again and again.

The check reuses the formula from the file test, expected frames equals duration times frame rate, with three comparisons: frames seen against 1,800, wall time against 60 s, and the last timestamp against the file's duration. All three matching means every recorded frame crossed your parser.

Prevent the reset instead of debugging it. Write the capture script with the two-pipeline layout from day one, enable the motion module at record time so the samples exist to replay, and re-run the frame count right after every recording. The capture side has habits worth reading too — recording depth data losslessly for offline analysis and why depth and color belong in a bag file rather than a video container.

The decisions for each case, in one place:

• File loops within seconds once motion is enabled: split into an image pipeline (depth, color) and a motion pipeline (accel, gyro) on the same file, and keep the default wait.
• Depth and color play cleanly but motion never arrives: the grouping is wrong, not the data; check that no pipeline carries three stream types.
• Live camera freezes accel and gyro after a few seconds: same split, plus the 10 s first-frame wait and the 200 ms per-wait drop.

The fix is config-side, so it works with any depth camera that records image and motion streams into one file. When you are building the capture station itself, structured-light 3D cameras are the usual hardware side of recorded depth datasets; two from the 3D sensors range:

For everything downstream of the camera, the camera finder filters 3D sensors by type, the camera step of the configurator pairs one with the rest of the station, and the 3D topic page collects the playback and capture fixes like this one. Questions about a specific dataset go to the team through the contact page.