Extract color frames from a bag file: handle YUYV color

You recorded a bag file, the player shows it in color, and every color frame you export comes out greyscale. Nothing is lost. The bag stores the color stream in its capture format, packed YUYV (4:2:2), and the export wrote the raw buffer as single-channel images. This guide is for anyone post-processing recorded depth-and-color data. You will confirm what the stream really is, decode YUYV to RGB with a short function, check the result frame by frame, and decide what format to record next time.

Why a bag color export comes out greyscale

A bag file is a container of raw stream frames, each stored exactly as the camera delivered it. The stream profile inside the recording states width, height and pixel format. Depth frames are plain 16-bit values, so any converter can write them out as images. Color is different: it is usually captured as YUYV, a packed 4:2:2 layout with two bytes per pixel, where one shared U and one shared V serve every two pixels. An export that writes those bytes as a single-channel image keeps the luma and drops the interleaved chroma, and mono frames are exactly what that looks like.

The color data is all still there: full luma plus half-resolution chroma. A stream buffer is bytes, not pixels, and the same bytes read as grey, striped or correct color depending on the format you declare — that trap is the subject of Image bytes vs pixel data: decode before processing. YUYV is not exotic either: under its other name YUY2 it is the layout Microsoft recommends as the standard 8-bit 4:2:2 format for video rendering (Microsoft's 8-bit YUV format reference).

What you need

• The bag file itself. No re-recording is needed: YUYV carries full luma plus half-resolution chroma, so the color is fully recoverable.
• A reader that hands you raw buffers: your camera SDK's playback API opened on the bag for offline playback. If live playback drops frames during extraction, read every frame offline instead, as in Bag file playback dropped frames: read every frame offline.
• Python 3 with NumPy, or any language with byte arrays and 16-bit integer math.
• An image writer for PNG or JPEG output, or a video encoder if you want a clip.

Worked example: one 1920 × 1080 YUYV frame

The inputs come from the recording's stream profile: width 1920 pixels, height 1080 pixels (the common color resolution for these recordings) and pixel format YUYV, packed 4:2:2 at 2 bytes per pixel. 4:2:2 packing needs an even width; camera streams always deliver one.

The rule in words: every 4 bytes hold two luma values (Y0, Y1) plus one shared U and one shared V, which works out at 2 bytes per pixel. Luma alone is 1 byte per pixel; each chroma plane is 1 byte per column pair.

The arithmetic:

• Pixels per frame: 1920 × 1080 = 2,073,600 pixels.
• Raw color buffer: 2,073,600 px × 2 B/px = 4,147,200 bytes, about 4.15 MB.
• Cross-check by plane: luma 2,073,600 B, U plane 960 × 1080 = 1,036,800 B, V plane 1,036,800 B; 2,073,600 + 1,036,800 + 1,036,800 = 4,147,200 B.
• Decoded RGB24: 2,073,600 px × 3 B/px = 6,220,800 bytes, about 6.22 MB.

The decision, with its threshold: a color frame from this stream must read as exactly 4,147,200 bytes. A 2,073,600-byte buffer is luma only; 6,220,800 bytes means the frame already is RGB; anything else means the stream is not YUYV, so re-read the reported format before decoding anything.

Decode YUYV to RGB, step by step

1. Open the bag for playback and read the color stream profile: width, height, pixel format, frame count. For the example stream: 1920 × 1080, YUYV.
2. Pull each frame's raw color buffer and check its length against width × height × 2 for your stream. Reject mismatches with an error; they mean a format you are not decoding.
3. Split the byte planes by stride. YUYV byte order per 4 bytes is Y0, U, Y1, V, so luma sits at stride 2, U at offset 1 step 4, V at offset 3 step 4.
4. Expand chroma: repeat each U and V value across its two-pixel pair.
5. Convert with the standard 8-bit BT.601 coefficients about the 128 chroma midpoint: R = Y + 1.402 × (V - 128), B = Y + 1.772 × (U - 128), G = Y - 0.344136 × (U - 128) - 0.714136 × (V - 128).
6. Clip to 0-255, cast to 8-bit unsigned, and stack R, G, B into an image.
7. Save PNG or JPEG frames, or feed the RGB frames to a video encoder. Index depth and color by the same frame number so pairs stay together; the playback-side method for keeping them aligned is in Align RGB and depth frames from a bag file: playback method.

The decoder function:

import numpy as np

def yuyv_to_rgb(buf: bytes, width: int, height: int) -> np.ndarray:
    """Decode one packed YUYV (4:2:2) frame to RGB.

    buf: raw frame, length must equal width * height * 2;
    byte order per 4 bytes: Y0, U, Y1, V.
    Returns a uint8 array of shape (height, width, 3).
    """
    expected = width * height * 2
    if len(buf) != expected:
        raise ValueError(f"frame is {len(buf)} bytes, expected {expected}")
    raw = np.frombuffer(buf, dtype=np.uint8).reshape(height, width * 2)
    y = raw[:, 0::2].astype(np.int16)      # luma, (height, width)
    u = raw[:, 1::4].astype(np.int16)      # U, (height, width // 2)
    v = raw[:, 3::4].astype(np.int16)      # V, (height, width // 2)
    u = np.repeat(u, 2, axis=1)            # chroma shared by each pixel pair
    v = np.repeat(v, 2, axis=1)
    r = y + 1.402 * (v - 128)
    b = y + 1.772 * (u - 128)
    g = y - 0.344136 * (u - 128) - 0.714136 * (v - 128)
    rgb = np.stack([r, g, b], axis=2)
    return np.clip(rgb, 0, 255).astype(np.uint8)

Working in MATLAB instead? The layout and the coefficients are identical; the matrix version of this decode is in YUYV to RGB conversion in MATLAB: decode raw camera frames.

Check the decode before you trust it

First run it on a synthetic frame whose right answer you know: set every luma byte to 255 and every chroma byte to 128. Both chroma offsets are then zero, so the whole frame must decode to pure white (255, 255, 255).

w, h = 1920, 1080
raw = np.full(h * w * 2, 128, dtype=np.uint8)
raw[0::2] = 255                    # luma bytes white, chroma neutral
rgb = yuyv_to_rgb(raw.tobytes(), w, h)
assert rgb.shape == (h, w, 3)
assert rgb.min() == 255 and rgb.max() == 255

Then check real frames. On an actual scene the R, G and B histograms must differ from one another; three identical channels mean luma leaked into all of them and a stride is wrong. Decode frame 0 and compare it with the same moment in the player: colors should match, and object edges should sit at the same pixel columns.

Common mistakes and the symptom each leaves

• Chroma-first byte order. Some devices pack each 4 bytes as U, Y0, V, Y1. The decode then runs without errors but every frame shows a strong pink-green cast. Move the luma slice and swap the two chroma offsets; the white-frame test passes either way, so catch this one on real footage.
• Processing before decoding. Resize, annotation and encode steps assume three bytes per pixel, so on a raw YUYV buffer they either fail on length or drop chroma. Decode first; the greyscale-after-resize version of this failure has its own write-up at Resized frames come out greyscale: fix colour loss.
• Studio-swing luma. Some sources store luma over 16-235 instead of the full 0-255. Symptom: washed-out frames with no true black or white. Stretch levels once, before any color judgment.
• Chroma shifted by one pixel. Repeating chroma with an off-by-one shift tints every vertical edge while flat areas stay correct. Anchor the expansion to the pair boundary, not the pixel boundary.

Keep YUYV on disk or convert at the source

Numbers first: YUYV stores 4,147,200 bytes per 1080p color frame against 6,220,800 for RGB24, so RGB costs 50% more disk and bandwidth per frame. At 30 frames per second, for example, that is 4,147,200 × 30 = 124,416,000 B/s, about 124 MB/s or 448 GB per hour, against 186,624,000 B/s, about 187 MB/s or 672 GB per hour, for RGB24.

Decode on demand when you annotate, review, feed an RGB-input model or share frames with tools that expect RGB. Skip color entirely when a task needs brightness only: slicing luma at stride 2 yields the 2,073,600-byte greyscale frame with no color math at all. And when you plan the next capture, set depth and color to record into one file with matching frame counts — the setup is in Record depth and color to a bag file, not an AVI.

The one case that argues for recording RGB: color judged on fine vertical detail. YUYV carries half the horizontal chroma resolution of its luma, and no decoder restores it. If an acceptance criterion measures color on features a few pixels wide and the camera offers an RGB capture format, take it and budget the 50% extra bytes and bandwidth.

Decision by case: storing and post-processing — keep YUYV and decode offline, deterministically. Color acceptance on fine vertical features — record RGB at the source when bandwidth allows. Brightness-only analysis — extract luma and never run the color math.

Where to go next

Decode, check against the player, then work from the RGB exports. If you are specifying hardware for the next capture round, the 3D sensors range covers laser profilers, structured-light and stereo cameras and ToF depth cameras, the camera finder filters by the specs that matter for your recording rate, and more recorded-data workflows sit on the 3D vision topic page. For help matching a camera to a bandwidth budget, reach us through the contact page.