RGB stream timeout on a depth camera: SDK support vs hardware fault

Depth frames arrive at full rate; the first color read blocks, waits out its timeout and returns nothing, and the sample viewer shows an empty window the moment you switch it to image mode. If that is your 3D camera, the likely diagnosis is good news: the camera is fine and the middleware never opened the color sensor. On this camera class the color imager hangs off a separate standard video interface that depth-first middleware does not touch. This guide ranks the causes by probability, gives a separating test for each, works the USB bandwidth arithmetic on a 480 Mbit/s link, and lists the four software routes that deliver color. You will know within ten minutes whether to change one line of code, one setting, or nothing at all.

The decision, up front

Depth works, RGB times out: in most cases you are asking the wrong client. The depth device and the color device are two separate enumerations on one cable, and middleware built on the depth protocol talks only to the first. Work it in this order.

1. Open the vendor's own viewer on the same port and switch to color.
2. If color appears, the hardware is good: move the color stream to the vendor SDK, a ROS wrapper, libuvc or a GenTL producer.
3. If color is missing there too, walk the ranked causes below — a second client holding the interface, USB bandwidth, a virtual-machine layer, a depth-only variant — with hardware last.

The empty viewer window fits this reading: the sample displays whatever streams it holds, and it holds only depth. An absent stream produces no error, just silence until the read gives up.

Two devices, one cable: why depth arrives and color never does

A structured-light or stereo depth camera is an IR projector plus one or two mono sensors. Their frames are packetized under a vendor protocol and reassembled by the SDK into depth and infrared images. The color imager is not on that path. It is exposed to the host as a standard USB Video Class (UVC) device — a second interface on the same physical link. The migration documentation for this camera generation states the split plainly: depth and infrared travel as proprietary packets, and color is a standard video stream on its own interface.

Middleware such as OpenNI2 contains a client for the depth protocol and no UVC client at all. A color stream request therefore cannot be satisfied: it either finds no device or blocks on a queue nothing fills, and the read times out. Nothing was lost in transit — nothing was ever delivered. That is why reseating cables never fixes this one.

Rule out one possibility early: whole product lines of this generation shipped depth-only, with no color imager. If your exact model is one of them, no client will ever produce RGB, and the split-the-job route below is the answer.

Causes in order of probability

Work this table top to bottom; each row's test separates it from every row below it.

Rank 2 is more common than it looks. Depth and color are grabbed separately, so a viewer you forgot about can hold the color interface while your code holds depth — the same one-client rule behind a camera that works in the vendor tool but not in ROS, showing up here as a color-only timeout.

Rank 3 has a public history: an embedded board ran depth beside 640 × 480 color and failed, then ran the moment color dropped to 320 × 240 with a lighter pixel format. That is a bandwidth signature, and the next section works the arithmetic.

Rank 4 earns its own test because a hypervisor forwards the USB interfaces it was configured for, and the depth interface is the one every setup guide configures. The pattern and its fix are covered in depth camera color stream timeout in a VM; the short version is to prove the code on metal before changing it.

Worked example: 640 × 480 color beside depth on one USB 2.0 link

Inputs, each with its source: color at 640 × 480, 30 fps, uncompressed YUY2 at 2 bytes per pixel — the default uncompressed UVC format; a 16-bit depth map at the same 640 × 480 and 30 fps, the matching depth mode this camera class ships; and a USB 2.0 high-speed link. Color usually outruns depth in resolution on these modules — one current module pairs 800 × 600 RGB with 320 × 240 depth at 30 fps — so color is the expensive stream on the bus. The link signals at 480 Mbit/s, which is 60 MB/s on paper; once protocol overhead and isochronous scheduling take their share, plan on about 35 MB/s sustained for mixed traffic. Treat 35 MB/s as the working budget.

Color: 640 × 480 = 307,200 pixels. At 2 bytes each that is 614,400 bytes per frame, and at 30 fps it is 614,400 × 30 = 18,432,000 bytes/s ≈ 18.4 MB/s.

Depth: the depth map carries the same 640 × 480 at 2 bytes (16-bit) and 30 fps, so it costs another 18.4 MB/s.

Combined: 18.4 + 18.4 = 36.8 MB/s against a 35 MB/s budget — already over before the infrared stream you probably also enabled. The read times out.

The same link with color at 320 × 240: 320 × 240 = 76,800 pixels × 2 bytes = 153,600 bytes per frame × 30 fps = 4,608,000 bytes/s ≈ 4.6 MB/s. Combined with depth that is 18.4 + 4.6 = 23.0 MB/s, or 66% of the budget — it runs with margin, which is exactly the resolution drop that rescued the embedded case above.

Decision rule: on a 480 Mbit/s link keep the stream sum under 28 MB/s, which is the 0.8 factor applied to 35 MB/s. This pair, at 36.8 MB/s, misses that rule by 31%, so the fix is bandwidth — compressed color (MJPG), lower resolution, or USB 3 — and not longer timeouts or cable swaps. Stretching the timeout only moves the failure later. With several cameras on one controller, test one stream at a time; three depth cameras dropping to low FPS is the same wall at larger scale.

Four software routes that deliver color

Once the viewer test clears the hardware, pick the route by what your host already runs.

The ROS drivers for this camera class grew color and infrared access precisely because the stock middleware could not fetch them; the wrapper adds the second client the samples lack. When both streams do arrive but drift apart, alignment and latency become their own problem — depth and RGB out of sync covers that fix. On the GenTL route, the producer file must be found and accepted by your viewer first; validating a CTI file before you enumerate saves an hour at the wrong layer. Which transport should carry which stream is a wiring decision of its own — RGB depth stream interface selection compares the options.

Split the job when the middleware will not move

Sometimes the platform decides for you: a locked OS image, middleware you cannot replace, or a host with no room for a second client. Then stop forcing one camera to carry both streams. Use a depth-first 3D sensor for Z and a standard color camera for RGB, triggered together — hardware sync for two cameras covers the wiring, and combining or splitting RGB and the point cloud covers what the split costs you.

On the color side, an area scan camera with a trigger input does the job. The MV-CS200-10GC is a 20 MP GigE color camera. On a GigE link you also get up to 100 m of cable, which USB does not offer.

For the depth half, three sensors whose whole job is Z:

Size the color camera before ordering: two pixels across the smallest feature you must see is the working floor. The camera resolution calculator turns your field of view and feature size into a pixel count and filters the camera finder to the models that clear it.

Checks before you change anything

1. Run the vendor's native viewer first; it is the reference client and answers the hardware question in one step.
2. Count the devices the OS lists on that cable. Two entries — vendor depth device plus standard video device — mean the color path is intact.
3. Confirm your exact model carries a color imager; depth-only variants of this generation exist.
4. Close every other viewer, SDK instance and browser demo before blaming the camera. The color interface serves one client at a time.
5. Budget the bus with the formula above and keep the stream sum under 28 MB/s on a 480 Mbit/s link.
6. Check your SDK version against the camera's supported list. A camera your SDK does not recognize fails loudly; old middleware in between fails quietly.

When it really is the hardware

Three results point at hardware rather than software: no video device enumerates on a second host with a second cable; the vendor viewer cannot show color anywhere; and the OS never listed a color interface even on first install. Then the return path is correct, and our team starts it from the contact page. Every other outcome on this page is cheaper than that call.

Two neighboring symptoms are different faults. Color opens but the image is black: an exposure or pipeline problem, covered in a black RGB stream from a depth camera. Color opens but arrives grayscale: you are watching the infrared stream — depth camera RGB showing black and white names the switch. And a unit that previews infrared but produces no depth is in passive IR mode; the passive IR mode fix is one setting.

The ten-minute checklist

1. Note which middleware holds the camera; depth running proves the depth device.
2. Close every other client and retry the color read once.
3. Open the vendor's native viewer and switch to color.
4. Color shows: move the color stream to the vendor SDK, ROS, libuvc or GenTL. Done.
5. Color missing: replug, second USB port, then a second host.
6. Count the listed devices and confirm the model is not depth-only.
7. Test bandwidth: color at 320 × 240 and 30 fps. If it runs, rework the bus budget.
8. In a VM? Repeat on bare metal before changing code.
9. Still nothing on two hosts: take the return path.

If the viewer test clears the camera and you would rather split depth and color for good, start at the 3D sensors category for the depth half and the camera finder for the color half, or run the system configurator for both in one pass. More integration guides sit on the 3D topic page.