RGB depth stream interface selection for vision cameras

An RGB-D camera is two sensors behind one shell: color and depth leave it on different logical interfaces, and no single capture library reaches both by default. The classic symptom is color previewing in any video player while your depth application lists no color stream, or the reverse. This workflow gives you the interface map, the link budget and the pass numbers: 101.4 MB/s for a 720p-plus-VGA pair, a 60-second acceptance run of about 1,800 frames, and a timestamp skew limit of 16.7 ms. Work through it before reinstalling drivers or swapping hardware; these faults almost always fix in software.

Name the symptom, then pick the row

A stream that never opens and a stream that opens but delivers unusable frames are different faults with different tests. Match the observation to a row and run that test first:

None of these rows starts with a cable or a camera swap. Start at the interface.

Why color and depth split at all

The color sensor enumerates as a USB Video Class device: a standard device class for which the operating system supplies the driver, so every capture API and every video player can open the node. Depth rides a vendor-specific interface that only the manufacturer's SDK opens, because no operating system ships a driver for it. One shell, two interfaces, two access paths — that is the whole mystery behind a color stream that works in one application and a depth stream that works in another.

Industrial area cameras invert the picture because they follow interface standards that RGB-D depth channels skip. GigE Vision requires a camera to announce itself to network discovery, accept control commands on a standard channel, and stream video in its packet format with resend on loss. USB3 Vision requires a machine-readable device descriptor carrying the camera's identity and its GenICam feature list, streamed over bulk endpoints. GenICam, the EMVA generic camera interface, turns that feature list into one programming model across brands.

Buffer contents are pinned by GenICam's pixel format naming convention (PFNC): RGB8 is 3 bytes per pixel, YUV 4:2:2 is 2 bytes per pixel, and Coord3D_C16 packs a 16-bit depth value into 2 bytes per pixel. When both interfaces report PFNC names, you can size links and allocate buffers without guessing.

Map streams to interfaces before writing code

Inventory first, code second. For each physical camera, record the logical device, its interface, the backend that lists it, the profiles it offers, and the evidence you saw. A working map for a typical RGB-D unit:

With several units attached, resolve USB device path conflicts between depth cameras before interpreting any result, or a passing color test may belong to a different camera. For finding the color node at the operating-system level, the RGB stream device selection workflow covers enumeration; this guide takes the next step and attaches each node to the backend that serves it.

Test the causes in order of probability

1. Open color through the video backend

Point a standard capture API at the color node and pull 5 seconds of frames — about 150 frames at 30 fps. Confirm the image changes with the scene and that the channel order matches the reported format; a BGR buffer displayed as RGB is the most common color fault. If color never appears in any generic backend, the camera exposes no video-class color node: capture color through the SDK instead. The boundary between generic video APIs and depth backends is the same one walked, depth-side, in the depth stream capture troubleshooting guide.

2. Open depth through the SDK

Depth on an RGB-D camera does not arrive through a generic video API. Use the manufacturer's SDK, obtained from the manufacturer or an authorized distributor, and pull 5 seconds at the profile it offers; 640 × 480 at 30 fps is the common depth format on this class of device. Then convert with the depth scale the SDK reports:

raw_value is the integer in the depth buffer, depth_scale is the meters- or millimeters-per-count the SDK reports, and distance_mm is the converted range. At a 1 mm scale a 16-bit buffer spans 65.5 m, and the invalid-pixel code (often 0) is a flag, not a distance.

3. Clear competing owners

One interface, one process — the USB stack enforces that on the depth node. Close viewers, tray tools and earlier instances of your application before testing. When a stream starts only after another application closes, that application held the interface; log which node each process opens and keep it out of the startup sequence. When several consumers genuinely need the same stream, add a host-side fan-out after acquisition is stable, the pattern shown in the multi-consumer stream fan-out design.

4. Request a profile the interface lists

Set resolution, frame rate and pixel format to values the backend itself enumerates — never an application default carried over from another camera. Log requested and negotiated values side by side, because a backend can accept a request and deliver a different mode. Allocate from the delivered format, stride included: a 1,280-pixel RGB8 row is 3,840 bytes, and any padding the driver adds on top of that shifts every row after the first.

5. Align firmware and SDK last

Only after interfaces, ownership and profiles pass, match operating system, driver, SDK and firmware to the manufacturer's supported combinations, one component at a time. When startup logs report a protocol or header mismatch, follow the firmware and SDK alignment procedure; the fix there is version alignment, not a driver hunt.

Size the link before running both streams

Uncompressed payload sets the floor. For a color profile of 1,280 × 720 RGB8 at 30 fps and a depth profile of 640 × 480 at 16 bits per pixel, also at 30 fps:

width_px and height_px are the negotiated image size, bytes_per_pixel comes from the PFNC format (RGB8 = 3, Coord3D_C16 = 2), and frame_rate is the negotiated rate. The result covers uncompressed payload only; transport overhead, resend traffic and metadata ride on top.

Compare the total with the link. A GigE Vision camera on a 1 Gbit/s line carries about 100 MB/s of payload after protocol overhead, so 101.4 MB/s does not fit on one: compress color, drop a profile, or move the pair to USB 3, whose 5 Gbit/s link budgets roughly 350 MB/s of payload. USB 2 at 480 Mbit/s delivers about 40 MB/s — enough for depth alone at 18.4 MB/s, far short of the pair. This arithmetic is a screening floor; measure live traffic with both streams running. For larger hosts, the multi-camera RGB and depth bandwidth plan extends the same calculation to four cameras.

Prove each stream, then the pair

Gate each stream alone, then together, with numbers at every gate.

Color gate: capture 60 seconds at the negotiated 30 fps — about 1,800 frames — and log received against nominal. A drop allowance of 0.1 percent, 2 frames in 1,800, is the starting threshold; tighten it to your line's cycle time from there.

Depth gate: the same 60-second run, plus a flat target at two positions a measured 500 mm apart. After scale conversion, the delta must read 500 mm, with invalid pixels excluded.

Pair gate: run both streams with no registration or overlay, and judge timestamps, not arrival order — host buffering reorders arrivals freely. Hold skew to half a frame period, 16.7 ms at 30 fps. Matching frame counts says nothing about synchronization; the multi-camera exposure timing method shows why equal frame rates are still not synchronized capture.

Spatial pairing comes last: load the intrinsics and the color-to-depth transform from the SDK, then run the RGB and depth registration checks before assigning color to depth points.

Choose the architecture from the map

Two backends is the normal case for RGB-D units, not a workaround. Keep each acquisition in its own adapter that outputs frame, format, timestamp and validity, so processing code never touches an interface.

Replace hardware only when a requirement has no answer

Replace the camera when a hard requirement has no answer in the device's own mode list: no simultaneous color-plus-depth mode, no depth format your software can convert, or no SDK build for your operating system. Interface-selection faults are not on that list; they fix in software.

For a new station, browse 3D sensors for the depth half and area scan cameras for a dedicated color unit; the camera finder filters both.

The MV-DPS200P-02 is a projected structured-light camera: it supplies depth and leaves color to a second device, which is exactly the two-interface split this workflow tests.

The MV-CU020-90GC is a color area scan camera on a GigE interface, so color runs on its own cable and its own link budget.

Two separate devices are the wrong answer when the application needs the factory-calibrated color-to-depth relationship and you cannot recalibrate it on the line. Forcing both streams through one API is equally wrong when the maker splits the interfaces on purpose.

Close the issue with the checklist

• Each stream tied to a physical camera, its interface and its backend, by serial number.
• Requested and negotiated profiles logged at startup.
• One owner per interface during tests; fan-out added deliberately afterward.
• Color gate passed: about 1,800 frames in 60 s at 30 fps, channel order checked.
• Depth gate passed: a measured 500 mm offset reads 500 mm after scale conversion.
• Pair gate passed: timestamp skew at or under 16.7 ms, registration checked against SDK intrinsics.
• The passing configuration still selects the same devices after a cold restart and a reconnect through the maker's procedure.

Save the passing configuration and its software versions as the baseline every future change is compared against. For a new station, carry the verified stream list into the camera configurator. When an interface stays silent, send its enumeration results, profiles and startup logs through application support before ordering replacement hardware.