Depth stream green screen in DirectShow: 16-bit data, not greyscale

You wired a depth camera into a DirectShow graph. The RGB pin renders normally, the depth pin fills the window with green, and GraphEdit refuses the formats that look relevant with 'this format is not currently available'. The camera is fine and so is your graph: the depth pin is delivering 16-bit range numbers, and the renderer is painting them as if they were 8-bit YUY2 colour. This page shows why that exact tint appears, how to prove the data is healthy with two quick tests, how to grab, unpack and scale the samples into a grey image you can display and process, and what your virtual camera should publish so the downstream application receives pictures or measurements rather than green noise.

Why a depth pin renders green, not grey

A depth value is an unsigned 16-bit integer: 0 where the camera could not measure, up to 65,535 where it could. Real scenes land between a few hundred and a few thousand units. Your RGB pin carries genuine YUY2 colour, 8 bits per channel. The depth pin carries no colour at all, but when the driver exposes its buffer under a YUY2 media type, DirectShow believes the label and hands the bytes to a video renderer.

Most depth cameras pair an infrared depth engine with a normal RGB module, which is why the two pins behave so differently inside one graph; the RGB side is an ordinary camera of the kind listed under area scan cameras.

YUY2 packs two pixels into four bytes in the order Y0, U, Y1, V. Depth samples arrive little-endian, low byte first, so each sample's low byte lands on a luma position and its high byte lands on a chroma position. Chroma is a signed offset around the neutral value 128. A scene value of 1,200 has high byte 4, and any reading up to 4,095 keeps the high byte at 15 or less, so the chroma byte always reads far below 128: an extreme colour offset, not a subtle one.

The renderer converts with the standard BT.601 full-range equations:

Take depth value 1,200 = 0x04B0. Its little-endian bytes are B0 04, so Y0 = 176 and U = 4. The next sample, 1,180 = 0x049C, gives Y1 = 156 and V = 4. For the first pixel:

• R = 176 + 1.402 * (4 - 128) = 176 - 173.8 = 2.2, about 2
• G = 176 + 0.344 * 124 + 0.714 * 124 = 176 + 42.7 + 88.5 = 307.2, clamped to 255
• B = 176 - 1.772 * 124 = 176 - 219.7 = -43.7, clamped to 0

Every pixel lands near RGB(2, 255, 0). Chroma bytes pinned close to zero push red and blue below zero while green saturates, and that is the whole green screen. Studio-range coefficients change the intermediate numbers, not the clamped result.

Invalid pixels explain the darker green speckle. The camera writes 0 where it cannot measure, so those bytes are 00 00: Y = 0, U = V = 0, and G = 0.344 * 128 + 0.714 * 128 = 44.0 + 91.4 = 135. Holes in the depth map therefore render as mid-green RGB(0, 135, 0), not black.

The connection succeeds because the frame sizes match

A renderer never checks what bytes mean, only how many arrive. At 640 x 480 a frame holds 307,200 pixels. 16-bit depth needs 2 bytes per pixel: 307,200 * 2 = 614,400 bytes. YUY2 also stores 2 bytes per pixel, so it claims the same 614,400-byte sample. The sizes match exactly, the graph connects, and the renderer produces steady, plausible-looking nonsense. At 30 fps the stream moves 614,400 * 30 = 18,432,000 bytes per second, about 18.4 MB/s, which the bus carries without complaint. Nothing in the graph's behaviour hints at the misread; only the colour says so.

Two tests before you change any code

Render the RGB pin first. If it works, the driver, the graph and the bus are healthy and the fault is interpretive. No filter is missing, and rebuilding the graph will not help.

Then capture one depth buffer and read it as unsigned 16-bit little-endian. The buffer must measure width x height x 2 bytes, 614,400 at 640 x 480. Cover the lens: every value should read 0. Point the camera at a plain wall, measure the distance with a tape, and check that the median value times the device's depth unit lands within a centimetre or two of the tape at bench distance. If it does, the data path is sound and the only problem is that you are watching raw numbers.

If the depth pin refuses every format rather than rendering green, that is a different failure: the no matching media type fix for Windows stream setup covers that case step by step.

The fix: grab raw samples and unpack them

Stop trying to render the depth pin and start reading it. GraphEdit's 'not currently available' message about formats such as INVZ, INZI and INZC is correct: those are vendor-defined FOURCCs that only the camera vendor's decoder understands, and a standard renderer will never connect to them. The YUY2 entry on the depth pin is your data path, not colour but a container whose size happens to fit. When a graph cannot interpret a format at all, the same diagnostic order applies as in unknown pixel format: diagnose missing depth streams.

Everything below is standard DirectShow, documented in the DirectShow documentation.

1. Insert a Sample Grabber on the depth branch and set its media type to the negotiated YUY2 subtype, so the connection cannot fall back to another format.
2. Enable sample buffering with ISampleGrabber::SetBufferSamples or register a callback, and end the branch without a video window.
3. In the callback, read the IMediaSample buffer as uint16_t little-endian, after asserting that its length equals width x height x 2.
4. Convert raw values to millimetres using the device's depth unit from the SDK, and log the median against the tape-measured wall.
5. Copy the values into a buffer declared CV_16UC1 if you use OpenCV; an 8-bit Mat misreads the same bytes, which is the mismatch behind Mat type versus pixel format failures.
6. Scale to 8-bit grey for display only, keeping invalid zeros out of your statistics.

/* Illustrative names; 640x480, buffer of 614400 bytes. */
uint8_t*  bytes = (uint8_t*)sample_buffer;   /* YUY2-carried depth */
uint16_t* depth = (uint16_t*)bytes;          /* 307200 samples, little-endian */
for (uint32_t i = 0; i < 307200; ++i) {
    uint16_t d = depth[i];                   /* raw value; 0 = invalid */
    uint8_t  g = (d >= 300 && d <= 2000)
               ? (uint8_t)((d - 300) * 255 / 1700)
               : 0;                          /* mask: invalid or out of window */
    grey_image[i] = g;
}

The test that proves the conversion: d = 1,200 gives (1,200 - 300) * 255 / 1,700 = 135, and a covered lens gives all zeros.

Scale the 16-bit values into a visible grey image

The linear map is one line:

Set d_min and d_max to the working range of the station: the nearest surface minus margin at the low end, the farthest surface you must see at the high end. With a window of 300 to 2,000 mm and a reading of 1,200 mm: (1,200 - 300) * 255 = 229,500, then 229,500 / 1,700 = 135, so grey level 135. Readings below 300 mm clamp to 0 and above 2,000 mm to 255. The resolution cost is 1,700 / 255 = 6.7 mm per grey level, comfortable for viewing and segmentation and useless for measurement. A narrower window sharpens the display: 300 to 900 mm spreads 600 mm over 255 levels, 2.4 mm each.

A shift is the simpler alternative: d >> 4 spreads the full 0 to 65,535 input evenly, which leaves a 300 to 2,000 mm station occupying only grey levels 18 to 125 and looking flat. The windowed map costs one multiply and uses the whole range.

Two details sit next to the scaling. First, raw values are units, not millimetres, until multiplied by the device's depth unit from the SDK; guessing that factor shifts every distance, the classic depth scale mismatch behind point clouds that cut off or drift. Second, send invalid pixels to a mask rather than a grey value so they never enter a histogram or median; the bit-level decoding of those flags is in 3D camera confidence image values: decode bitflags and mask bad pixels.

What your virtual camera should publish

The green screen happened because a display format carried non-display data. Do not ship the same trap downstream. Decide by what the other application does with the stream:

• It shows the depth image or runs 2D processing on it: publish 8-bit grey already scaled with your validated window. Every consumer renders it correctly on the first try.
• It measures distances: never round-trip values through an 8-bit display format, because 255 levels cannot carry them. Have it consume the SDK's 16-bit stream directly, or publish the raw buffer with its unit scale, stride and frame size documented.
• It needs both: publish two streams, one scaled grey for viewing, one raw for measurement.

Decision: Mono8 for viewers, raw 16-bit for measurers, both when you have both. Formats such as INVZ and INZI exist because depth is not video; a virtual camera that only speaks video formats must convert at its boundary. The same trade in another guise, why sensors with more bits still ship 8-bit images, is explained in ADC bit depth vs pixel format. If the downstream application must also align depth with RGB, the buffer layout and which intrinsics apply are covered in aligned depth streams.

Other green screens that are not this problem

Match the symptom before applying the fix above:

• Green on the RGB pin too: the capture path is at fault, with wrong stride, half-filled buffers or a source that stops mid-frame. On embedded boards with serial camera links the same symptom traces to capture timeouts, worked through in this Jetson Orin NX GMSL green-screen and timeout fix.
• Diagonal green noise that scrolls: a stride mismatch, where the source writes lines of width x 2 bytes and the renderer reads a different line pitch. Align the buffer length to width x height x 2 and the noise disappears.
• Green speckle over an otherwise grey image: invalid depth pixels reaching your already-scaled output. Mask zeros before scaling.
• A converted image that comes out nearly black: the viewer is scaling a higher-bit-depth image badly, as explained in why 10-bit PNG images look black in image viewers.

Checks that keep it away

• Log the negotiated subtype and bit count before connecting; never assume a depth pin's YUY2 label means colour.
• Assert every frame's length equals width x height x 2; a silent size drift shows up as noise, not as an error.
• Validate one buffer per session against a covered-lens zero buffer and a tape-measured wall.
• Keep the scaling window in configuration and log it with each capture set.
• Treat any preview, DirectShow included, as unverified until decoded values match physical distances.
• When wrong colours come from a format default rather than a misread buffer, work from the negotiated format as in the GigE camera pixel-format default fix.

If this sends you back to equipment choice rather than graph wiring, the 3D sensors category lists depth cameras with documented SDK streams, the camera finder filters them by type and interface, and the camera configurator step pairs one with the rest of the station. More capture-pipeline fixes are collected on the 3D topic page, and we are reachable through /contact/ if you want a second opinion on a depth integration.