Align RGB and depth frames from a bag file: playback method

You recorded color and depth from a 3D camera, and the file on disk holds two separate streams — not aligned pairs. That is not a recording fault: alignment is a computation, and a capture stores streams, not results. This guide walks through the fix: replay the file as a virtual device, pair frames by device timestamp, run depth-to-color alignment on every pair as it plays, and size the frame queue so the replay loses no more than the recording did. You get the steps, the timing numbers, a worked example and a verification list. It assumes a stereo, structured-light or ToF camera SDK with record and playback support and two synchronized streams in the file.

What a capture file actually stores

A capture file, often called a bag file, is not a video of aligned pairs. It stores each stream separately, exactly as delivered: raw depth frames, raw color frames, the hardware timestamp of every frame, and the calibration active at record time — each stream's intrinsics plus the transform between the two sensors. Alignment lives nowhere in the file.

The alignment computation takes a depth frame, reprojects each pixel through the depth intrinsics into space, applies the depth-to-color extrinsic, and samples the result onto the color pixel grid — or the reverse for color-to-depth. The output is indexed by the target stream: align to color and you get one result per color frame; align to depth and you get one per depth frame. When either stream lost frames at record time, the two directions report different aligned-frame counts. That difference is expected arithmetic, not a fault, and it is the first symptom to recognize when a replay looks short of pairs.

What you need

• The capture file, played back by the same SDK family that recorded it
• The SDK's playback mode, which presents the file to your program as a virtual device
• Per-frame device timestamps on both streams
• The SDK's depth-to-color alignment function
• A settable frame queue size
• A disk that keeps up with the replay bitrate

The bitrate arithmetic tells you how much disk and link the replay needs. Assume an uncompressed 848 × 480 depth stream at 30 fps with 16-bit values, plus 1080p color at 30 fps with 3 bytes per pixel:

Depth: 848 × 480 × 2 × 30 = 24.4 MB/s. Color: 1920 × 1080 × 3 × 30 = 186.6 MB/s. Together about 211 MB/s, roughly 1.7 Gbit/s. A dedicated USB 3 link carries that — 5 Gbit/s nominal, about 4 Gbit/s usable — while a hub shared with other devices does not, and the frames it loses are gone from the file permanently. If your recorder compresses streams, divide the file size by the recording duration to get the true rate, and size the disk from that.

Replay, sync and align: the steps

1. Open the file in the SDK's playback mode instead of enumerating live cameras. The player then acts as a virtual device, and the rest of your pipeline code stays the same.
2. Enable both streams at the profiles stored in the file — the recorded resolutions and frame rates. Asking for anything else either fails or silently resamples.
3. Turn on hardware (device) timestamps for both streams before the first frame arrives, and pair frames by device timestamp, never by arrival order.
4. Raise the frame queue size above its default. The queue absorbs the bursts that happen when the OS briefly starves your process; every frame it holds is latency you have accepted.
5. For each frame of the reference stream — color if you align to color — take the partner frame with the nearest device timestamp, and accept the pair only when the difference is within half a frame period.
6. Run the alignment function on the pair and write the aligned result out together with both timestamps.
7. Log three counters while the file plays: frames seen per stream, pairs formed, pairs rejected for a missing partner. Those numbers are the verification data in a moment.

The numbers: pairing window, queue latency, expected pair rate

At 30 fps the frame period is 1000 / 30 = 33.3 ms, so the pairing window is ±16.7 ms; a candidate pair outside that window is a drop or a timing fault, not a match.

Worked example — a 60 s recording at 30 fps on both streams:

• Nominal frames per stream: 30 × 60 = 1,800
• Example replay log: 1,762 color frames and 1,791 depth frames in the file; 1,745 pairs form inside the 16.7 ms window
• Pair rate: 1,745 / 1,800 ≈ 97% of nominal

Decision: around 97% is what a clean two-stream capture on one dedicated link typically delivers, because the streams contend for the same bandwidth. For dataset building that is fine — you keep 1,745 aligned pairs and record the gap. Metrology over a known duration is different: the missing frames were lost at record time and no playback setting recovers them, so the capture has to be redone.

Align to color or to depth?

Choose by where the data must live. Overlays, annotated datasets and anything a person reviews belong on the color grid: align depth to color. Measurement, volume and robot guidance need undisturbed depth values: align color to depth, and the depth grid stays metric.

Resampling costs either way. Depth pushed onto the color grid gets holes at depth edges and occlusions, because color pixels the depth sensor never observed have no value, and the interpolated result is softer than the original — the same resolution-versus-accuracy trade as any upsampled depth frame. Color pushed onto the depth grid lands on a coarser grid and loses sharpness. If the aligned image sits a constant pixel or two off the color image, suspect the calibration or the stream format: raw and rectified frames use different pixel grids, and a checkerboard calibration you run yourself shows which one is at fault.

How to verify the alignment worked

Run these checks on the first minute of replay:

• Timestamp check. The worst pair difference stays under 16.7 ms at 30 fps. Differences clustered near 33.3 ms mean you are one frame off — something still pairs by order.
• Count check. Pairs formed divided by nominal frames. About 97% in the example above; a rate far below that points at record-time drops, not at alignment.
• Edge check. Aim at a high-contrast straight edge, replay a few seconds, and confirm the aligned depth edge lands on the color edge within about 1–2 pixels.
• Flatness check. A flat wall still fits a plane after alignment. Curvature that was not in the raw depth means the resampling or the depth units are wrong.
• Offset check. An offset that grows with distance is an extrinsic problem; an offset fixed in pixels is usually a raw-versus-rectified mix-up.

Common mistakes that break sync

• Pairing by queue position. Two streams rarely arrive interleaved in step; the symptom is a constant one-frame color lag and ghosted edges on every moving boundary — the classic color lags depth by one frame failure.
• Using host receive time. The OS stamps frames when the driver delivers them, with milliseconds of jitter. The device timestamp is written in hardware and is the only stable clock; to line a capture up with other logs, map device time to PC time once and reuse that mapping.
• Aligning twice. Feeding an already-aligned frame back through alignment resamples depth a second time and softens edges. Keep raw depth in the pipeline and align each raw pair.
• Heavy work inside the frame callback. With a small queue, processing inside the callback overflows it and manufactures drops during replay; copy the buffer first and return, then process outside.
• Trusting the file before counting it. Recorded depth bag playback can be missing frames for reasons that have nothing to do with your replay code. Run the count check before debugging anything else.

When the recording cannot be saved, fix the capture

Drops baked in at record time are permanent, so the highest-leverage settings live on the capture side:

• Record through one dedicated port straight into the host. A hub shared with other devices is the classic source of lost frames on depth cameras.
• Keep write speed above the stream rate. For the 211 MB/s example above, test-record a minute to a drive with clear headroom and check the file's frame counts before the real capture.
• Record both streams at native rate. Downsampling one stream in software during capture desynchronizes the frame clocks, and it shows later as a sawtooth in the pairing differences.
• Where the SDK offers a hardware sync mode that slaves one sensor's frame start to the other, enable it: sync at the source beats any replay-time repair.
• Keep processing out of the record callback, exactly as during replay.
• Log device timestamps for both streams, so replay-time pairing has clean data and verification has ground truth.

If the next capture is yours to set up, the 3D sensors range spans stereo, structured-light, ToF and laser profiler types:

Not every capture needs a color pair at all. Where the measurement is a profile — a seam, a groove, a cross-section — a laser profiler records height per scan position, and pairing its frames with a color camera follows the same nearest-timestamp rule:

Replay once, then judge

Replay the file one time with the counters running before you change anything. The three numbers decide the diagnosis: short frames in one stream means capture damage; full frames but few pairs means sync or pairing; full pairs but a shifted image means calibration or stream format. Fix in that order — capture, sync, calibration — because work downstream of a broken step is wasted.

For the next capture, start at the camera finder, configure the station in the build-a-system configurator, or browse the replay, calibration and sync guides on the 3D topic page. The machine-vision knowledge hub holds the rest, from triggering to streaming diagnostics.