Record depth and color to a bag file, not an AVI

You recorded a depth camera's output to an AVI, and the file plays a nice rainbow video — with no depth in it. The conversion to an 8-bit colormap happens inside the recording script, before anything reaches the disk, so the 16-bit distance values were not lost in a codec; they were never stored. This guide is for anyone building offline datasets, debugging a vision station or re-running inspections on captured scenes. You will record depth and color together to the SDK's native bag file, play the file back as if the camera were still connected, and know what ten minutes of recording costs in disk space.

Why the depth AVI never contained depth

A depth camera hands each frame to your code as a 16-bit array: one distance value per pixel, 65,536 levels, stored in millimeters on most sensors. The classic recording script converts that array for display before writing anything: each pixel is multiplied by a scale factor, clipped to the 0–255 range of an 8-bit image, and pushed through a color palette such as Jet. Only then does the video writer encode the AVI.

Take the scale factor a typical script applies to millimeter values, alpha = 0.03:

Every pixel keeps at most 33 mm of depth resolution, and nothing at all beyond 8.5 m. The palette then maps each of the 256 surviving levels to a color, and the AVI codec compresses those colors lossily, so even a perfect palette inversion returns coarse, noisy guesses. The millimeter-level precision the sensor delivered is gone. A repair tool that rebuilds a broken AVI index hands back the same rainbow, because the values were never in the file.

Decision: if the recording path passed through a colormap, treat the depth as lost and re-record. Display with a colormap; record raw. If your point cloud comes back wildly out of scale instead, that is a different fault — the depth scale behind point-cloud cut-offs.

What you need

A depth camera that streams native 16-bit depth plus a color or intensity image, the SDK that ships with it, and disk space budgeted from the table further down. Recording is a pipeline feature of these SDKs, not an add-on: the same configuration object that starts the streams writes the file. The shop's 3D sensor category covers laser profilers and stereo 3D cameras, and the camera finder filters them by type and series.

A projected structured-light stereo camera is the usual choice for this workflow:

Record depth and color to a bag file in seven steps

The call names differ slightly between SDKs; the sequence does not. In a Python wrapper these are settings on the pipeline configuration before start; in the desktop viewer they are the stream panel and the record button.

1. Set resolution and frame rate first. The frame rate belongs to the stream definition: it is read-only once the pipeline runs, and it cannot be changed in the finished file either. Pick from the size table below, and if saved settings refuse to stick, re-apply and read them back as in startup frame rate verification.
2. Enable recording on the configuration before start. The call takes the output path and writes the SDK's native stream container — a bag file (.bag, the ROS bag format) in this class of SDK; the same steps apply to whatever raw-stream container your SDK offers.
3. Record depth in z16. Add the depth stream in its native 16-bit pixel format and make sure the recorder captures the stream, not the display path. The container stores each frame with its metadata: timestamps, camera intrinsics and the depth scale.
4. Record color in the same file. One container keeps depth and color frames paired by timestamp, so the matching RGB frame for any depth frame comes back automatically at playback.
5. Keep the colormap out of the record path. Apply the palette after the frame reaches the recorder, on a separate variable, for the on-screen preview only.
6. Run the capture with disk headroom. The rate table below sets the expectation; a full disk truncates the file mid-run.
7. Stop cleanly. Stop the pipeline or the record session so the container closes its index. Killing the process can leave a file that playback tools refuse to open.

Verify the recording before you walk away

Point the pipeline's start call at the file instead of the device and check four things on an early frame. The depth array is 16-bit and its maximum sits in the thousands of millimeters, not at 255 or below. The frame count matches fps × seconds: a 30 fps capture of 60 s holds about 1,800 frames. Depth and color frames carry matching timestamps or frame numbers. The file size lands within a factor of two of the estimate; far under usually means one stream never recorded.

Playback behaves like a live camera

Playback is the payoff. Swap the device descriptor for the file in the start call and the rest of the script runs unchanged: frames arrive through the same interface, depth as a 16-bit millimeter array, color as 8-bit BGR, paired and time-stamped. Copy each frame out of the SDK buffer into your own array before you queue or store it — copying frames from the SDK buffer is the same trap at playback as on a live camera.

Two details matter. A recording plays at the rate it was captured: you cannot re-time the file, but you can sample it, taking every 5th frame of a 30 fps recording as 6 fps of processing while the rest stay in the file. And a model consumes playback frames exactly like live frames, so a scene captured once can be re-run against every model revision. Where you fuse streams, the same pairing is what makes depth, color and infrared fusion reproducible offline.

What ten minutes of recording costs

Depth dominates the file. A 16-bit frame is width × height × 2 bytes, and the cost is linear in frame rate:

Raw 8-bit BGR color adds 0.9 MB per frame at this resolution, 27.6 MB/s at 30 fps, but real recorders compress the color stream, so treat the depth column as the planning floor. A measured 10-minute capture at 640 × 480 came out around 1 GB: an average of 1 GB / 600 s ≈ 1.7 MB/s, which is about 2.7 fps of raw depth plus compressed color. The observed file and the table agree once frame rate is accounted for.

The levers, in order: frame rate is linear, so 30 fps to 6 fps cuts the depth payload sixfold; resolution is linear in pixel count, so 320 × 240 costs a quarter of 640 × 480 — 153,600 bytes per frame. Converting depth to 8 bits to save space is the colormap mistake again, and it saves less than one fps step. For what to drop without losing usable frames, see reducing depth recording file size; for a host recording several streams at once, multi-camera RGB-depth bandwidth.

Decision: choose fps from the fastest motion the analysis must resolve — 30 fps for a fast conveyor, 6 fps for a static station — and let resolution follow the depth camera pixel size the smallest feature demands. Never trade bit depth.

When an AVI is still the right container

For a review clip, a training video or a shareable file, record the bag first and export afterwards: play the recording back, apply the colormap inside the export loop, and write the AVI from that. The master keeps the values; the AVI is a disposable view. Single frames export as 16-bit PNG, the format that keeps Z intact, and packing a 16-bit stereo frame into two 8-bit halves is reversible if a viewer needs it — see splitting a packed 16-bit frame.

The general recipe for camera video files in Python is its own guide, recording video from a machine vision camera in Python; depth adds the 16-bit constraint that guide cannot cover. For display, normalize a copy and leave the recorded array untouched, as in normalizing a depth map for Jet colormap display. When only relative depth is needed in 8-bit color, disparity often encodes it better than Z: hue colormap depth encoding.

Common mistakes

• Writing the colormap image to the depth file. Keep the preview on a separate variable and never let the palette touch the recorded frame.
• Changing fps after the stream starts, or expecting a finished recording to play at another rate. The rate is fixed at stream definition and survives into the file; sample every nth frame instead.
• Scaling depth into 8 bits temporarily for a recording. Temporary becomes permanent the first time a dataset is used.
• Killing the capture process instead of stopping it. The container index does not close and playback tools may reject the file.
• Budgeting disk from the compressed size you hope for instead of the raw rate you can compute.

Pick the camera, then press record

The workflow needs a camera with native 16-bit depth, its SDK and disk. Choose hardware by range, accuracy and field of view, because recording rides along. When the surface moves faster than a static capture tolerates, a laser galvanometer stereo camera steers the scan across the part:

Assemble the rest of the station in the configurator, and the 3D topic hub collects the capture, calibration and playback guides around this one. Questions the guides do not answer go to the support desk.