Missing frames in recorded depth bag playback: causes and fixes

A five-minute depth recording that converts into a two-minute video has lost about six frames in every ten. The loss is not in the encoder and not in the recording itself: it happens in the read loop, where the file reader hands frames to your processing code faster than the code can take them, and the queue between the two overflows. This guide is for anyone who records depth and color streams to a file and processes them offline — filtering, measuring, annotating, archiving. You will count the frames you actually have, find the hand-off where they vanish, pace the reader so every frame survives, and prove the fixed conversion with a frame count instead of a stopwatch.

Count frames, not minutes

The two-minute output is not a clue; it is the answer. Whether the container is a bag file or an MP4, a constant-frame-rate file holds duration × fps frames, so an output that plays for 120 s at 30 fps contains 120 × 30 = 3,600 frames. The recording ran 300 s at the same 30 fps, so it should have yielded 300 × 30 = 9,000 frames. Exactly 5,400 frames — three in every five — never reached a writer.

The fps value you pass the writer is metadata only. It tells a player how fast to step through the frames it was given; it cannot create a frame and it cannot save one. That is why the short file plays as slow motion rather than as a compressed five minutes.

Diagnose with a counter, not a stopwatch. Increment one integer every time a frameset is saved and print it when the loop ends. Compare it with the file's true total: the playback-device object your SDK hands you usually reports the frame count or playback status, and if yours does not, a still-per-frame export gives the number exactly.

Then apply one rule. A written count within a frame or two of expected means nothing was lost and the loop is merely slow. A written count below expected means frames died in the read path; the causes below are ordered by how often each one turns out to be guilty.

Where the frames go: three causes in order of probability

Cause 1 — real-time playback against a slower loop (most common). With real time on, the reader releases framesets at their recorded timestamps: one every 33.3 ms at 30 fps. If an iteration takes longer, the pipeline's frame queue fills, overflows and discards the surplus, and the output keeps only the frames you happened to be ready for. This is the recorded-file twin of acquisition faster than processing: fix the frame rate gap on live streams, and it feels the same as capture time slower in the API than in the viewer.

Cause 2 — resume-and-pause pacing that floods the queue. The tempting fix, resume() then one wait_for_frames() then pause(), makes it worse. With real time off, resume() streams the file at disk speed into the queue; while your loop spends tens of milliseconds processing one frameset, the queue overflows, wait_for_frames() hands you a single frameset, and pause() abandons everything still buffered. A 60% loss — three frames in five — is exactly this signature. The pause throttles the reader only after the overflow has already happened.

Cause 3 — per-frame object churn. Building the aligner, filters and colorizer inside the loop costs an allocation on every frame, and a stateful temporal filter rebuilt each frame forgets its smoothing history. The churn adds tens of milliseconds per iteration and deepens causes 1 and 2.

Time one iteration around the loop body to see which cause is live. Above 33.3 ms per iteration with real time on is cause 1. Any resume() call left in the loop is cause 2, whatever the timing. A loop that speeds up sharply once the objects move out was cause 3 feeding cause 1.

Fix the read loop: pace the reader, not the writer

Three moves, in this order.

1. Set real time off once, at set-up. Take the playback device from the started profile — profile.get_device().as_playback() — and call set_real_time(False) before the loop. The file is then read as fast as you consume it, not as fast as it was recorded.
2. Delete every resume() and pause(). Loop on wait_for_frames() and let the SDK's timeout or end-of-file error terminate the loop. The reader now applies backpressure and waits for you, so nothing overflows. If the wait instead spins or times out while frames should exist, the getdata timeout: no frames available fix covers the loop habits behind that.
3. Hoist the heavy objects out of the loop. Aligner, filters, colorizer and both writers are created once and reused on every frame.

The pattern, with my counter and error class marked as illustrative:

frames_expected = 9000    # int, computed before the run: 300 s * 30 fps
frames_written = 0        # int, counts framesets actually saved

playback = profile.get_device().as_playback()
playback.set_real_time(False)            # SDK call: read at consumer speed

while True:
    try:
        frames = pipeline.wait_for_frames()   # SDK call: next frameset
    except TimeoutError:                      # illustrative: use your SDK's end-of-file error
        break
    frames_written += 1
    # align, filter, colorize and write with objects created above the loop
    writer.write(color_image)
    writer.write(depth_image)

print(frames_written)
assert frames_written == frames_expected, (
    f'conversion lost {frames_expected - frames_written} frames')

The assert is the test that proves the fix: it passes only when every frameset in the file reached a writer. After pacing, loop cost decides how long the conversion takes, never how many frames survive — 9,000 framesets at 100 ms each is 900 s of batch time, and the assert still expects 9,000.

Worked example: the 5-minute recording that came back as 2 minutes

Inputs and where each comes from: the recording lasted 5 min = 300 s at a configured 30 fps (the failed conversion); the output plays 2 min = 120 s, written by a writer told 30 fps (the output file); the loop cost is 100 ms per frameset, an illustrative assumption for the pacing arithmetic — time your own loop once and substitute the number you measure.

The rule in words: a constant-frame-rate file holds duration × fps frames, and frames lost = expected − written.

Arithmetic:

• Expected frames: 300 s × 30 frames/s = 9,000 frames
• Written frames: 120 s × 30 frames/s = 3,600 frames
• Lost frames: 9,000 − 3,600 = 5,400 frames, a loss of 5,400 / 9,000 × 100 = 60%
• Frame interval at 30 fps: 1 / 30 s ≈ 33.3 ms, so a 100 ms loop is 3.0× the budget and real-time playback must drop frames
• Paced conversion time: 9,000 × 0.1 s = 900 s = 15 min, about 3× the recording's duration

Decision: convert paced, with real time off, and budget a 15-minute batch window for this file. Accept the output only when the counter reads 9,000 of 9,000. If the file must be reviewed live instead, get the loop under 33.3 ms per frameset first; above that line, dropping frames is arithmetic, not luck.

Prove it before you trust the output

Duration proves nothing once real time is off: a paced conversion spends 15 minutes on a 5-minute file, and a stopwatch comparison would report a false failure. The frame count is the proof. Assert written frames against the file's total on every conversion and leave the assertion in the script permanently — it is a two-line regression test that catches the next person who reintroduces a resume().

When the count matters more than a video file, skip the AVI: save one still image per frameset under a sequential index and count the files afterwards. A lost frame then shows up as a gap in the numbering, and you can build a video from the complete still set later if a reviewer wants one.

Record-side habits prevent the recut: record at a rate the disk sustains for the whole session, keep timestamps intact, and prefer a complete format over a compact one, as covered in recording depth data losslessly for offline analysis and reducing depth recording file size without losing usable data. If the recording itself outran the disk at capture time, the timing fixes are in camera recording faster than real time.

When the loop is not the culprit

Three situations survive the fix above.

All frames present, motion wrong. If written frames equal expected but the video runs fast or slow, the fps label mismatches the true rate. Re-mux with the correct value; no frames are missing, so nothing needs recovering.

Frames missing before your loop runs. Export stills for the whole file first and count them. If the count is already short, the recording dropped frames at capture time because disk or CPU could not sustain 30 fps. Fix the recorder — lower resolution or rate, faster disk — not the reader.

Stream pairing halves the count. If you save only framesets where color and depth both arrive, a file recorded without one of the two streams cuts your output silently. Count each stream separately and the absent one shows up as an exact half or a flat zero.

Where to go next on the bench

For the hardware side of a recording station, start at the 3D sensors category or the camera finder; the pacing habits here apply to whatever depth camera records the files. The rest of the depth-recording fixes sit on the 3D topic page. If a conversion still loses frames after the counter says it should not, send the loop timings and the two counts through the contact page and we will read them with you.