Depth camera color stream fails in OpenCV: one owner per USB device

Depth arrives cleanly through the OpenNI2 SDK, then the color stream runs into a wall: VideoCapture reports false, the console prints "Failed to set USB interface!", or the first read() freezes while isOpened() still says true. Sometimes the message is "This stream is open by other components." This guide is for you if you are integrating a stereo or structured-light depth camera on Linux and adding OpenCV for color. You will learn which client owns the device, what each error string means, and the two-capture split that lets depth and color run side by side.

Why does the color stream fail when the depth stream already works?

Because the depth middleware still owns the device: a USB interface accepts exactly one claim, your depth application holds it, and OpenCV's CAP_OPENNI2 backend is a second client that cannot claim what is already taken.

A depth camera is a composite USB device. A vendor interface carries depth and infrared, driven from userspace by the middleware; a standard UVC interface usually carries color, driven by the kernel video stack. USB ownership is blunt - one driver or one userspace handle per interface, enforced at claim time. While your depth viewer or SDK application runs, it holds the vendor interface, so the OpenNI2 backend inside OpenCV fails its claim and the open aborts.

Reverse the order and the fault flips: OpenCV holds the device and your depth application gets the refusal. The exclusivity is not an OpenNI2 quirk. A USB3 Vision camera refuses a second client at the interface level - one client owns a USB3 Vision camera - and a GigE camera refuses it on the control channel, where opening a camera in a second tool fails on exclusive control. Add a second camera and the rule applies per device: two depth cameras hitting the interface claim error is the same wall, twice.

What do the three error messages actually mean?

They are three stages of one conflict: the first fails while claiming the interface, the second fails while configuring a stream someone else already started, and the third means the color node went to a backend it does not belong to.

The fourth line people see - "USB events thread - failed to set priority. This might cause loss of data..." - is not part of the failure. It is a scheduling warning printed at open, and it appears in healthy sessions too; ignore it while you hunt the owner. If you have ever watched a V4L2 node answer "device or resource busy" when two tools opened one camera, that is the kernel-side face of the same rule: two nodes opening one camera.

Which process is holding the camera right now?

In order of likelihood: the vendor's depth viewer still open, a service from the camera's OS image that starts a depth pipeline at boot, and a stale process left behind by a crashed run.

1. Close every depth application you can see: the viewer, SDK demos, and your own earlier test binary still running in another terminal.
2. Ask the kernel who holds the video nodes. List the processes with the /dev/video* nodes open and kill any leftover from a crashed run - an old node still holding the camera after a restart is the classic invisible owner.
3. Check what the OS image auto-starts at boot. Some camera images launch a depth or streaming service before you log in; disable it and the conflict disappears for every application you write.
4. Re-run your open alone and require isOpened() to return true before the first read.

Nine times out of ten the holder is step 1: a viewer left open beside the terminal. One layer up, a camera that is detected but shows no image has the same diagnosis, and a depth viewer stuck on opening live usually traces back to a second client as well.

How should depth and color be opened instead?

Give each stream to the backend that owns its interface: one VideoCapture(CAP_OPENNI2) for depth and infrared, and one index-based VideoCapture for the color node - never two middleware clients on the same device.

The split follows the hardware. On this class of camera, color enumerates as an ordinary UVC device while depth stays behind the vendor interface - why some depth camera color streams show up on UVC or V4L2 covers the enumeration side. Some vendor drivers go further and list color as invalid for the middleware on purpose, which makes the index-based capture the only supported route to those frames.

1. Close every other client of the camera, as diagnosed above.
2. Open the depth capture once and keep it for the life of the process.
3. Find the color node's index by trying 0, then 1, and keep the one whose isOpened() is true.
4. If an index opens but prints "GStreamer: unable to start pipeline", request the V4L2 backend explicitly for that capture instead of accepting the build's default backend order - VideoCapture takes the backend as its second argument.

import cv2  # built with the OpenNI2 and V4L2 videoio backends

depth_ok = False   # bool: true only when the depth capture reads a frame
color_ok = False   # bool: true only when the color capture reads a frame
frame = None       # image returned by the last read

depth = cv2.VideoCapture(cv2.CAP_OPENNI2)  # the only middleware client
color = cv2.VideoCapture(0)                # the UVC color node, by index

depth_ok = depth.isOpened()
color_ok = color.isOpened()

if depth_ok:                  # proof: each read returns instead of blocking
    depth_ok, frame = depth.read()
if color_ok:
    color_ok, frame = color.read()

print("depth:", depth_ok, "color:", color_ok)

With no other client running, the expected output is "depth: True color: True" and both reads return immediately. If index 0 grabs the wrong node, step it to 1.

Why does the read freeze instead of returning false?

Because the capture half-opened: the device accepted the open, but the stream was never started - the other client already had it running in a mode your open could not change - so the blocking read waits forever on frames that will never arrive.

That is the classic signature: isOpened() returns true, the first cap.read() never comes back, and the console from the failed middleware attempt carries the "open by other components" text. It is one conflict wearing two faces; OpenCV stuck at cap read on a depth camera walks the same symptom from the middleware side.

Once ownership is fixed, two leftovers still stall a healthy setup. USB autosuspend powers the camera down between runs, so the second run hangs - the pattern behind a second stream failing on the run after the first. And a USB 2 port cannot carry depth plus color once both streams run, failing the bandwidth check exactly as in no data in the depth stream on a USB 2.0 port. Use a USB 3 port and disable autosuspend for the device.

To clear a freeze: stop acquisition, release the frozen capture, close the other client, then reopen once with the watchdog in place. That order matters at exit too - stop acquisition before you close keeps the next open working on any USB camera. If the frames feed a pipeline that then reports an internal data stream error, the ownership question comes first there as well: USB camera internal data stream errors start with the same check.

What if the color node still refuses to open?

Then the driver, not your code, is the limit: with no other client running and the node still failing, some vendor drivers list color as invalid for the middleware and serve it only through the UVC stack, so the index-based capture is the supported path.

Test the node's life apart from OpenCV: if no tool on the host can pull a frame from it, the node has no working driver on that kernel, and backend selection cannot fix a missing driver. At that point you stop debugging and re-architect - color from a standard camera, depth from the sensor. A 2.5 MP GigE color camera such as the MV-CU020-90GC moves color off the USB tree entirely, so the depth device keeps its single owner and color arrives on its own cable.

For a station you are specifying fresh, depth can come from a dedicated sensor of the same class:

Compare the depth and ToF options in the 3D sensors finder, and color sensors in the area scan cameras finder or the full camera finder.

How do you keep the conflict from coming back?

Make one owner per interface a written rule of the station: one process opens the depth device, nothing else touches it while that process lives, and every capture is stopped and released before the process exits.

• Close the vendor viewer before running your application. Never run both, not even to look.
• Stop acquisition, then release each capture, then exit - in that order, every time.
• After any crash, list the holders of the video nodes and kill the leftovers before the next open.
• If a second tool must watch the stream, serve the frames from the owning process over your network or shared memory; do not open the device twice.
• Keep depth and color on a USB 3 port when both run; a USB 2 port fails the bandwidth check under the combined load.
• Disable USB autosuspend for the camera so the second run starts as fast as the first.

If the embedded color node is the dead end on your kernel, the decision is made for you: a standard color camera beside the depth sensor. Start from the color end with the camera configurator, or lay out the station on the contact page and we will match the depth side to it.