Depth camera RGB shows black and white: you are on the IR stream

You connected an RGB-D camera, opened what you thought was the colour stream, and the image updates — in black and white. The camera works, reinstalling the driver changes nothing, and white-balance controls do nothing, because nothing is broken. On this class of camera the depth sensor and the colour sensor are two separate devices in one housing, and the depth device's own image is an infrared picture: one intensity channel with no colour in it. This guide shows you how to confirm which device you opened, open the colour endpoint as a second capture, synchronise the two streams and budget the USB bandwidth that makes them drop frames, with the checks that stop the same surprise on the next machine.

The symptom, and what it is not

The frames arrive at full rate and exposure responds to light, but every pixel is gray. Look closely at a textured surface and a faint dot pattern moves across it: that is the projector's speckle, the fastest visual confirmation that you are watching the infrared sensor.

Two other faults also produce gray pictures, and they look different:

• A misread pixel format leaves colour data present but decoded wrong — the image shows a tint or a mosaic. See Camera preview shows a gray image: fix the format mismatch.
• A camera defaulting to a mono pixel format shows gray although colour silicon sits behind it. See GigE camera shows no color image: fix the pixel format default.

The infrared image is clean grayscale with no mosaic and no tint. That difference tells you the problem sits upstream of any decode: the data itself is single-channel.

Two devices in one housing

An RGB-D camera is two or three imagers behind one cable. The depth engine — an infrared camera paired with a pattern projector, or two infrared cameras in stereo — reports depth plus an infrared intensity image. The colour sensor is a separate imaging device, and on many models it is exposed only as a standard USB Video Class (UVC) device rather than through the vendor SDK's stream list. Your capture asked the depth device for its image, and the depth device's image is infrared.

That picture cannot be colour. The sensor has one channel, it sits behind an infrared-pass filter, and the scene is lit by a monochromatic near-infrared pattern. Red, green and blue never reach it, so no setting recovers them.

Confirm it in one minute

One physical camera appears as two or more device nodes, and picking the right node is its own task. The guide to Linux camera device selection: find the RGB stream walks through telling the nodes apart; the same two-device logic applies on every operating system.

The fix: open the colour endpoint as a second capture

1. Enumerate every video device the host sees before opening any. Expect at least two nodes for one RGB-D camera.
2. Read the formats each node advertises. Packed colour formats mark the colour sensor; single-channel 8- or 16-bit formats mark depth or infrared.
3. Open the colour node as a second capture and keep both handles for the session — one for depth, one for colour.
4. Set exposure on each stream separately. The two auto-exposure loops are independent.
5. Match frames by timestamp, not arrival order. Two devices run on separate clocks, and queueing can swap adjacent frames. At 30 fps, hold the measured skew under one frame period, 33 ms.
6. Register colour onto depth with the extrinsics, then check the result on a flat target marked at your working distance.

The registration offset you will see

Colour and infrared sit a short baseline apart, so their views differ by parallax. Assume a 25 mm baseline and a 1920-pixel-wide colour sensor with a 70° horizontal field of view: the sensor resolves about 1920 / 70 ≈ 27 pixels per degree, and the baseline subtends atan(25 / 500) ≈ 2.9° at 500 mm, roughly 78 pixels of offset. Double the distance and the offset halves. Plan for tens of pixels of correction at close range, applied once through the extrinsics.

Two guides cover this stage directly: Align depth to a third RGB camera: fix missing extrinsics for getting the extrinsics when the pair was never calibrated together, and Depth to RGB registration offset in pixels: fix the rectified stream for the residual offset left after alignment.

The bandwidth arithmetic

Colour is the expensive stream. Its raw payload follows one line:

Worked example for a typical 1080p colour sensor and a VGA depth sensor, both at 30 fps:

• Colour, 1920 × 1080 pixels, YUYV at 2 bytes per pixel: 1920 × 1080 = 2,073,600 pixels, × 2 B = 4,147,200 B per frame, × 30 fps = 124,416,000 B/s ≈ 124 MB/s ≈ 995 Mbit/s.
• Depth, 640 × 480 at 16 bits: 640 × 480 = 307,200 pixels, × 2 B = 614,400 B per frame, × 30 fps = 18,432,000 B/s ≈ 18 MB/s ≈ 147 Mbit/s.
• Sum ≈ 143 MB/s ≈ 1.1 Gbit/s before protocol overhead.

USB 2.0 moves 480 Mbit/s on the wire and sustains about 40 MB/s of payload, so raw 1080p colour cannot cross it — that is why colour on a USB 2.0 link arrives compressed as MJPEG. USB 3.2 Gen 1 moves 5 Gbit/s and carries the sum with margin. Adding the infrared stream on top of depth plus colour is the heaviest combination and a common cause of dropped colour frames; Depth and color stream timeouts: the USB bandwidth check measures it, and Depth camera black image: fix USB bandwidth and power issues covers the collapse when power sags alongside it.

The decision: run depth plus colour on a USB 3 link, request the separate infrared stream only when you use it, and accept MJPEG colour when a USB 2.0 controller is unavoidable.

When the grayscale stream is the one you want

The infrared picture is not a consolation prize. It ignores printed colour and labels, it works with the room lights off, and it shares the depth sensor's viewpoint and clock, so it needs no registration. Fusing it with colour and depth in one point cloud is covered in Point cloud with RGB and IR intensity: fuse depth, color and infrared.

The two-camera alternative

When the colour endpoint stays unreliable on your host, or you need colour quality the embedded sensor cannot reach, split the job and register the pair once: geometry from a 3D sensor, appearance from a colour area scan camera. The link trade-offs for that kind of set-up are compared in RGB depth stream interface selection for vision cameras.

On the appearance side, the MV-CS200-10GC is a 20 MP colour area scan camera on GigE with a C-mount and a rolling shutter; for moving parts, filter the same finder for a global shutter instead. On the geometry side, choose by task: a laser profiler such as the MIS-DP3850-01P for height profiles across a moving line, or a time-of-flight camera such as the MV-DT01WMU for full-frame depth in one shot.

Filter the camera finder by interface and mount, or start the system configurator at the camera step.

Checklist before you change hardware

• Enumerated two or more device nodes for one physical camera before writing capture code.
• Read the advertised formats per node and matched colour to packed formats.
• Saw the projector's dot pattern, confirming the gray image is infrared.
• Computed the bandwidth sum at your resolution and frame rate; chose MJPEG colour on a USB 2.0 link.
• Matched frames by timestamp and measured the skew under one frame period.
• Registered the pair and confirmed near-zero offset on a marked target at working distance.
• Still gray after all of the above: the unit may be a mono variant, or your decode drops channels — work through the two format guides linked earlier.

If the part choice is the blocker, the contact page reaches our applications team, and more integration wiring guides sit on the systems topic page.