Recorded color images come back null: set the playback conversion

You open a recorded MKV file from a 3D camera, walk the capture loop, and k4a_capture_get_color_image returns null while the depth data plays back perfectly. The recording is usually fine; the playback contract is the problem. The SDK documentation states that when you never call k4a_playback_set_color_conversion, color comes back in the format it was recorded in. A field report on SDK v1.3.0 shows the other outcome: a default-settings recording on Windows 10 Pro x64 returned a null color image until the conversion was set explicitly. This guide gives the call order that restores color, the byte-level arithmetic for picking a target format, and an ordered set of checks for the cases where the image stays null.

What you see: depth plays back, color returns null

The sequence is always the same. You open the file with k4a_playback_open, step through it with k4a_playback_get_next_capture, and ask each capture for color with k4a_capture_get_color_image. The depth side of every capture arrives intact. The color pointer is null. Nothing errors and nothing lands in a log; the loop runs to the end of the file with nothing to show.

The report that pinned this down ran SDK v1.3.0 on Windows 10 Pro x64, with RGB camera firmware 1.6.98 and depth camera firmware 1.6.70 (depth config file 6109.7). The MKV was recorded with the recorder's default settings, color included. On the SDK team's own test machine the same path returned color, and the team could not reproduce the null. Same version, same stated default, two outcomes.

That is the reason to stop relying on the default. It is not wrong in a way you can patch; it is unreliable in a way your code cannot see. The call that removes the ambiguity costs one line.

Why it happens: recording stores, playback converts

Live capture and playback are not symmetric, and the asymmetry sits exactly where the null appears. When you stream from the camera, you choose the color format while configuring the stream, and frames arrive already decoded. The recorder writes the color track in the compressed format the camera streamed, MJPG in the default case, because storing raw frames would multiply the file by roughly the compression ratio.

Playback reverses that. The SDK decompresses and converts each stored frame before your code sees it, and one setting drives the stage: the playback color conversion. The stated default returns color in the recorded format, as though you had asked for exactly that. On the affected build, the unset path hands back a null image instead.

The working discipline is the one behind querying before setting any camera parameter: read the state, then set the value. Query a 3D camera exposure range from the API before you set it covers that habit for exposure; the conversion setting deserves the same treatment. The recording-side counterpart is the container: record depth and color to a bag file, not an AVI is what gives each stream its own track to play back.

The fix: one call, before the first capture read

1. Open the recording and keep the playback handle that k4a_playback_open returns.
2. Call k4a_playback_set_color_conversion on that handle, passing the format constant for your target, a 4-byte BGRA32-class format for display and RGB work.
3. Read k4a_playback_get_record_configuration once, to confirm the file holds a color track before you blame the conversion.
4. Iterate with k4a_playback_get_next_capture and take color from each capture with k4a_capture_get_color_image.
5. Release the image and the capture before the next iteration, every iteration.
6. Verify the fix: the color pointer is non-null, the image dimensions match the recorded color resolution, and the color frame count matches the track.

// API names below are spelled exactly as the SDK defines them:
//   k4a_playback_open, k4a_playback_set_color_conversion,
//   k4a_playback_get_next_capture, k4a_capture_get_color_image.
// Every other identifier is an illustrative variable for this example.

playback = k4a_playback_open(file_path)       // playback handle to the MKV
target_format = the SDK constant for BGRA32   // 4 bytes per pixel

k4a_playback_set_color_conversion(playback, target_format)  // the fix, once

capture = null                                // one capture at a time
color = null                                  // color image pointer
while (k4a_playback_get_next_capture(playback, capture) succeeds)
{
    color = k4a_capture_get_color_image(capture)
    // Proof the fix worked: color is non-null, and its dimensions equal
    // the recorded color resolution from the record configuration.
    release color, then the capture, before the next iteration
}

Worked example: 1280×720 color at 30 fps through a BGRA32 target

Take a recording that holds 1280×720 color at 30 fps, typical of a default color recording, and set the conversion to a 4-byte BGRA32 target. For comparison, take the stored compressed frames at 250 kB each; that is the example's one estimate, and the arithmetic below shows where to redo it with your real file's numbers.

• Pixels per frame: 1280 × 720 = 921,600 pixels
• Expanded frame: 921,600 pixels × 4 bytes per pixel = 3,686,400 bytes ≈ 3.52 MiB
• Expanded data rate: 3,686,400 bytes × 30 frames per second = 110,592,000 bytes/s ≈ 110.6 MB/s
• Frame budget: 1 / 30 = 0.0333 s = 33.3 ms per capture
• Expansion over the stored track: 3,686,400 / 250,000 ≈ 14.7 times per frame
• A 60 s clip: stored ≈ 250,000 bytes × 30 × 60 = 450 MB; expanded ≈ 110.6 MB/s × 60 s ≈ 6.6 GB

Two decisions fall out of the last lines. The file keeps the compressed track, so the conversion runs on playback whether you choose its target or not; choose it, once, inside the SDK, in the one format your algorithm consumes. And if your loop cannot hold the 33.3 ms cadence, the first cost to cut is a second conversion in your own code, because at about 110 MB/s of expanded color you pay the copy twice. The same keep-the-disk-copy-compressed logic sits behind reducing depth recording file size without losing usable data.

Picking the target format

For inspection code that displays frames or runs RGB algorithms, set the BGRA32 target and stop thinking about it. Move to packed YUV only when downstream code already runs on YUV; the YUYV handling in extracting color frames from a recorded file is that case. Ask for the recorded compressed format only when you decode it yourself, otherwise you move the null risk one call downstream and save nothing.

Still null after setting the conversion: checks in order

1. Confirm the color track exists. Read k4a_playback_get_record_configuration before the loop. A depth-only recording yields no color image under any conversion setting; re-record with color on. When you re-record with the depth projector running, keep eyes out of its beam and treat it like any laser source.
2. Set the conversion on the handle you actually read. Wrappers own their playback handles: the body-tracking wrapper exposes k4abt_playback_set_color_conversion for exactly this, and in the field report color appeared once that call ran on the wrapper's handle. A handle you never configure keeps the unreliable default.
3. Align SDK and camera firmware before debugging further. Playback quirks cluster around version mismatches, and 3D camera header version mismatch: fix firmware and SDK alignment gives the matching procedure.
4. Play a freshly recorded file on the same machine. One new file separates file-side damage from SDK-side behavior faster than any other single test.
5. Count which captures lack color. Color on some captures and not others is a frame-pacing problem, not a conversion problem; read missing frames in recorded depth bag playback and bag file playback dropped frames: read every frame offline.

Prevention: make the call part of opening the file

The fix costs one line; the habit saves more than that. Set the conversion inside the same function that opens the file, so no code path can iterate an unconfigured handle. Log the record configuration beside every recording so playback code knows what the file holds. When you consume both streams, align them deliberately with the method in aligning RGB and depth frames from a bag file instead of assuming the two tracks index together. If color lands one frame behind depth, syncing RGBD frames by sequence number removes the off-by-one.

Pre-flight checklist:

• Conversion set on the playback handle before the first read
• Target format matches what the algorithm consumes, chosen once
• Record configuration read, color track confirmed, logged with the file
• Wrapper handles carry their own conversion call
• SDK and camera firmware drawn from one aligned set
• Every iteration releases the image and the capture

Cameras that make these recordings

Recordings like the one in this failure come from depth-plus-color 3D cameras: ToF units and structured-light stereo units that write synchronized tracks to one file. For a new capture station, browse 3D cameras and the camera finder; the 3D vision topic page collects the playback and alignment guides. Sensors of this class include the MV-DT01WMU, the MV-DLS300P-04 and the MV-DPS200P-02:

If a recording from your station fails in a way this page does not cover, send the file, the SDK version and the camera firmware versions through the contact page; reproducing the failure on a like-for-like machine is the fastest route to a root cause.