Replay recorded camera streams: mapping files to the right sensor

You record a multi-stream session from a 3D camera rig — color, the stereo pair, an aligned depth stream — and the folder on disk is complete. Replay it the next morning and the node you built as the color camera renders depth frames. No error, no warning; the pipeline just runs on the wrong pixels, and the same folder replays correctly on a colleague's PC.

That combination — intermittent across machines, silent at runtime — is the signature of a stream-mapping bug. Here is the failure end to end: why the loader swaps streams, why the result changes between computers, how to prove which file a node actually opened, and the mapping discipline that makes replay deterministic everywhere. If you are building record-and-replay from scratch, read test 3D camera code without hardware: record and replay fixtures first and come back for the mapping rules.

Why does replay show depth frames on the color camera?

Because the replay loader matched your color camera to whichever stream first claimed the color socket, and a depth stream with an unparsed file name claims that default, so the wrong file wins purely by the order the operating system listed the recording folder.

The tell is silence. The visualization opens, the frame counter ticks at your recorded frame rate, and the image under the color label is unmistakably depth — smooth grayscale banding instead of the scene. Nothing in the log complains, because as far as the pipeline knows, every node received a stream that claims to be the color camera. The behavior also follows the folder rather than your code: re-copy the recording, rename a file or move it to another drive, and the outcome can flip. That points at load order, not decoding.

Why does the same recording replay correctly on one PC and fail on another?

The stream-to-camera match takes the first hit from a dictionary built while walking the folder, and dictionary order follows the directory listing, which differs between file systems and operating systems, so the real color file can be first on one machine and the depth file first on another.

The loader opens a recording by walking the folder once, filtering video extensions, and filing every file in a dictionary keyed by its lowercased name prefix. The replay component then matches its camera's socket against the streams in that dictionary and takes the first hit. Dictionary iteration follows the order the directory listing returned the files, and listing order is a property of the file system: NTFS enumerates names in sorted order, ext4 in hash order, other systems in raw directory-entry order. Modern runtimes preserve dictionary insertion order, so each machine repeats its own result faithfully — the machines simply disagree with each other. On the PC that works, the real color file was read first and won the match; on yours, the depth file was read first. Your script never changed; the folder's iteration order did.

How does a depth stream end up claiming the color socket?

The loader parses a camera socket out of each file name, and a depth stream's name contains no socket, so the parse stores nothing; instead of rejecting that stream, the lookup answers left or right from the name and otherwise hands back the default color socket.

Recordings keep one file per stream. Camera streams carry a board-socket prefix in the file name; derived streams — the aligned depth, the disparity map — carry a stream name instead, so there is no socket to parse and the parser stores nothing. The damage happens in the lookup that follows. Asked for its socket, such a stream answers left if its name contains left, right if it contains right — and otherwise the default color socket, never unknown. The depth file now claims to be the color camera, the match loop takes the first stream making that claim, and your color node binds to the depth file. Two ingredients, both present: a stream that guesses at its own identity, and a matcher that accepts the first guess.

How do you test which file a replay node actually opened?

Read frame 0 from the file the color node actually opened: a 1920 x 1080 RGB frame carries 6,220,800 bytes and a 1280 x 720 depth frame carries 1,843,200 bytes at 16-bit, so one reported width separates the two with certainty.

Take a one-minute session at 30 fps, so 1,800 frames per stream. The color stream records 1920 x 1080 at 8-bit RGB: 1920 x 1080 = 2,073,600 pixels, x 3 bytes = 6,220,800 bytes per frame, x 1800 frames = 11,197,440,000 bytes, about 11.2 GB. The depth stream records 1280 x 720 at 16-bit: 1280 x 720 = 921,600 pixels, x 2 bytes = 1,843,200 bytes per frame, x 1800 = 3,317,760,000 bytes, about 3.3 GB. The ratio, 6,220,800 / 1,843,200 = 3.4, separates the two files even in a plain directory listing. Frame count cannot: both streams wrote 1,800 frames.

Where recordings are encoded and file sizes shrink, read frame 0 from the file the color node opened and check its reported width and height — 1280 x 720 under a color node is the depth file, identified in seconds.

What is the correct fix for stream mapping during replay?

Key each stream by the camera socket it declared at record time and refuse to start when any stream has none: the mapping must come from the recording's metadata, never from file order, name guessing or a fallback default.

1. Declare at record time. Write one metadata entry per stream when the recording opens: stream name, camera socket, resolution, pixel format, frame rate. A derived stream takes the socket of the camera it is aligned to.
2. Load from metadata, not the directory. Build the socket-to-file map from the declarations first; sort file names only so runs repeat on one machine.
3. Fail on unmapped. If a stream has no declared socket, raise before the pipeline builds. If your capture library exposes a specify-socket call for replay streams, call it for every stream you create rather than leaving the match to the first hit.
4. Prove it before start. Print stream, file, socket and first-frame size, and read the table: the color line must show 1920 x 1080, not 1280 x 720.
5. Pin it with a test. Load one fixture recording twice and assert the two maps are identical, then run the same test on a second file system — that is where order dependence hides.

How do you stop the swap happening in future recordings?

Prevention is three habits: name every file, including derived depth and disparity files, with its board socket; sort directory entries before you iterate; and print the stream mapping before the pipeline starts, so a bad mapping dies at load time and not on the line.

• Put the board socket in every file-name prefix, including derived depth and disparity files, so a parse can never come back empty.
• Write one metadata file per recording, after the last stream opens, listing socket, resolution, pixel format and frame rate for each stream.
• Sort directory entries wherever a folder is walked: hygiene for repeatability, never the source of truth.
• Compare the pre-start mapping printout against the session you believe you recorded, every time.
• Keep a CI test that loads one recording twice, on two file systems, and asserts identical maps.

Each habit moves the failure from the production floor to load time, where it costs seconds instead of a morning of debugging.

Which cameras and rigs run into this?

Any rig that records more than one stream per session can hit this — color plus stereo pair plus aligned depth is the common three-file case — and it is the standard workflow on structured-light and logistics-volume 3D cameras, where you record parcel flows and tune algorithms offline.

The bug needs at least two recorded streams and one unparsable prefix, which is exactly the layout of a color-plus-depth or stereo-plus-depth session; a single-stream recording cannot be crossed with anything. Among the shop's 3D sensors, the multi-stream cases are structured-light stereo cameras and logistics-volume depth cameras — rigs built to record runs of parcels and have their algorithms tuned against the files afterwards. Sessions with three or more recorded streams are the richest case, and the match loop is the first suspect on any of them.

Two examples of those two types, straight from the range:

When is the wrong image not a mapping problem?

Before you blame the loader, rule out the three look-alikes: the color stream was never written because of capture-side enable order, the replay points at the wrong session folder, or the file decodes to black frames — each has a one-minute test.

• Color stream never written. Capture-side enable order can leave the color file out of the folder entirely; list the folder and count the files against the streams you asked to record. That failure mode has its own fix in depth camera color stream missing: the enable-order fix.
• Wrong session folder. Recordings nest by timestamp and device serial, so pointing the replay at the parent folder can load a different device's session. Check the metadata inside the folder, not the folder name.
• Decodes to black. The file opens and frames advance, but every pixel is zero — a decode or pixel-format problem, not a mapping one. If the streams sit on the wrong geometry instead, that is the resolution mismatch covered in depth to color alignment breaks up: match stream resolution first.

For trimming what you keep on disk, see reduce depth recording file size without losing usable data; for a container that carries its own metadata, see 3D camera recording to H5 files for offline processing. Once the mapping is right, the layout of the aligned depth itself — stride, encoding, which intrinsics apply — is covered in aligned depth streams: data layout and which intrinsics apply.

If you are standardizing a record-replay flow across a fleet, pick the rig in the camera finder, size it in the camera configurator, or bring your recording layout to the applications team through the contact page. The 3D topic page collects the rest of the replay and capture guides, including recorded video replay for machine vision pipelines.