Bag file playback dropped frames: read every frame offline

You recorded depth and color to a bag file, the clip played complete in the viewer, and the playback loop hands back a fraction of it: about 100 framesets from a 15-second recording, with frame numbers that skip and never match the viewer's count. The recording itself is almost never where frames go missing. They are on disk, and the read path is discarding them. This guide names the three causes in the order they strike, gives a test for each, and builds a read loop with the arithmetic to show it recovers all 450 framesets a 15-second, 1280×720, 30 fps recording holds.

What the symptom tells you

Three observations travel together: a frameset count far below recording length times frame rate, frame numbers that start above zero and skip values, and a loop that exits early. Frame numbers are counters the capture side assigns at record time, so gaps already exist in the file; they are neither playback losses nor file positions. The viewer's counter is its own tally again. None of these numbers measures what you care about, which is framesets consumed against framesets stored.

Frames missing from the file itself are a different fault with its own causes, covered in Missing frames in recorded depth bag playback: causes and fixes. This page handles the other half: getting every stored frameset out.

Cause 1: real-time pacing discards what your loop cannot take

By default the player paces delivery on the timestamps stored in the file: depth and color leave the player at capture rate, 30 fps here, matching the wall clock. Your loop is the other end of the pipe, and the pacing layer discards anything the loop fails to accept in time. So measure one iteration end to end: the frame wait, the two frame extractions, every conversion or display call inside the loop. If an iteration costs 120 ms, the loop consumes 1000 / 120 ≈ 8.3 framesets per second, and 15 seconds of paced playback yields 15 × 8.3 ≈ 125 framesets — close to the roughly 100 a failing run reports, with the pairing layer accounting for the difference.

The fix is the playback device's real-time switch. Take the playback handle from the started profile and call set_real_time(False) on it. Delivery then waits for your loop instead of the clock, so the loop's appetite sets the pace. Pacing off removes the clock, not the queue: a loop that stalls on a display call can still overflow the pipeline's internal buffer. That is why the read pass below dumps raw frames and pushes every bit of processing into a second pass.

Cause 2: repeat playback restarts the file under your loop

The call that loads a file as the pipeline's source takes a repeat flag, and it defaults to looping: the player rewinds at the end and starts again. A loop that terminates when the frame number stops increasing then quits early, because the restart pushes the numbers back below the values already seen and the comparison fires on the spot. The run ends with a fraction of the file consumed and looks exactly like dropped frames.

Test: print every frame number and watch for values that fall back to the file's first numbers partway through. Fix: open the file with repeat playback off — in the from-file loader that is the repeat_playback=False argument — and never make a frame-number comparison the loop's exit condition.

Cause 3: frame numbers are capture counters, not file positions

Frame numbers are counters the capture side assigns at record time. They start wherever the counter was, they skip when frames were lost before recording ever wrote them, and the depth and color values inside one frameset can differ. The viewer's count is its own tally again. No comparison against these numbers measures playback health. The reliable measure is the file's frameset total: playback interfaces expose current position and total frames on the playback device handle, so read the total once, consume exactly that many wait_for_frames results, and compare the two figures at the end. If your SDK exposes no total, loop until the wait times out and treat the timeout as end of file. The timeout, not a frame number, is the stop signal.

The read loop that cannot drop frames

Work through the set-up in this order; each step removes one way to lose frames.

1. Load the file as the pipeline source with repeat playback off (repeat_playback=False), so it plays once.
2. Start the pipeline, take the playback device handle, and set real time off (set_real_time(False)).
3. Enable only the streams you need, at the rate and format recorded: 1280×720 z16 depth and 1280×720 rgb8 color here. Every extra stream joins the pairing and can starve it.
4. Read the frameset total from the playback status interface once, before the loop.
5. Loop exactly that many times, pull one frameset per pass with wait_for_frames, and write the raw depth array, the raw color array and the frameset's timestamp to disk.
6. Stop the pipeline, then print the consumed count against the total. With no total available, loop until the wait times out and count what arrived.

# Read pass: dump raw frames only, process in a second pass.
# total and saved are illustrative variables; the calls follow the SDK's playback interface.
total = 450                          # int, frameset count read once from the playback status
saved = 0                            # int, framesets written to disk
playback.set_real_time(False)        # pacing off; the file was opened with repeat off
while saved < total:
    frames = pipeline.wait_for_frames()
    depth = frames.get_depth_frame()   # z16, 1280x720
    color = frames.get_color_frame()   # rgb8, 1280x720
    save_raw(depth, color, saved)      # illustrative writer; store the frameset timestamp too
    saved += 1
pipeline.stop()
print(f"saved {saved} of {total}")

The pass is proven when the printed line reads saved 450 of 450: consumed equals total on every run, on every file.

Worked example: what a 15 s recording holds

The arithmetic behind every number above, at 1280×720 and 30 fps:

Depth in z16 is 2 bytes per pixel: 1280 × 720 × 2 = 1,843,200 B ≈ 1.8 MB per frame, and 1,843,200 × 30 = 55.3 MB/s. Color in rgb8 is 3 bytes per pixel: 1280 × 720 × 3 = 2,764,800 B ≈ 2.8 MB per frame, 82.9 MB/s at 30 fps. A 15 s recording stores 30 × 15 = 450 framesets per stream and carries about 138 × 15 ≈ 2.1 GB of payload before compression.

The same arithmetic explains the symptom. A loop at 120 ms per frameset consumes 8.3 fps, so paced playback over 15 s delivers 15 × 8.3 ≈ 125 framesets — close to the roughly 100 in a failing run, with the pairing layer's losses taking the rest.

If pairs still vanish, read one stream at a time

A frameset delivers only when every enabled stream has a frame ready to pair. With pacing off, a slow color decode or an alignment call that reprojects depth into the color image can still stall the pair and cost depth frames. Split the pass: run the whole loop with depth enabled only, then again with color enabled only, and join the two dumps on the timestamps you saved. At 30 fps the frame period is 33.3 ms, so a one-frame join window is ±16.7 ms. This is the same one-stream-at-a-time logic that rescues three depth cameras dropping to low FPS on one host. Alignment belongs in the second pass either way; the playback method for aligning RGB and depth keeps the reproject out of the capture loop.

Prevention at record time, checks before you trust the result

Before recording:

• Uncompressed where the disk allows it. The 138 MB/s from the worked example sits well inside what a SATA SSD sustains (several hundred MB/s typical), and 2.1 GB per 15 s clip is 2.1 × 4 = 8.3 GB/min, so size free space before a long session.
• Keep depth and color in one recording, not side files: see Record depth and color to a bag file, not an AVI. If your camera line writes H5 instead, the loop logic is identical — see 3D camera recording to H5 files for offline processing.
• If the recorder compresses, budget the decode: once per-frameset read cost passes 33 ms, reading a 15 s file takes longer than 15 s. Acceptable offline, impossible live.
• Record both streams at the same rate; a mismatched rate starves the pairing all recording long.

Before you trust any output:

• The pass/fail test is one line: saved 450 of 450, consumed equal to total, on every run.
• A conversion tool must pass the same count test. Offline converters have dropped frames silently, so count their output images against the frameset total before building on them.
• Keep timestamps in every dump row. The timestamp, not the frame number, is the join key between streams and between cameras.

When the hardware is the bottleneck

If the count test fails at record time — the file itself holds fewer frames than frame rate × seconds — the capture chain, not the player, is dropping them. Sustained 138 MB/s wants a disk that never hesitates and a host with margin; add displays or more cameras and the margin disappears. The 3D sensors range covers laser profilers and stereo depth cameras, the camera finder filters them by type and series, and the build-a-system configurator matches camera, controller and cables to your resolution and frame rate. For a pipeline that still loses frames after every check above, send us the frame rate, resolution and host specs and we will help size the chain. The rest of the capture-and-playback fixes sit on the 3D topic page.

The decision, case by case: loop quits early or counts come back short — pacing and repeat flags off, loop driven by the frameset total, processing moved out of the read pass. Pairs still missing with pacing off — two single-stream passes joined on timestamps. File short on disk — the recording chain needs a faster host, not a different playback loop.