Depth camera busy on startup: detect and reset it programmatically

A vision system that runs headless cannot have someone reseat a USB plug. Yet one crashed Python script is enough to leave a depth camera busy for every start afterwards: the dead process never released the device, and the camera refuses the next pipeline open. Below is the startup sequence that replaces the plug cycle. You will fix the expected camera set, query the SDK device list, hardware-reset cameras that are present but refuse to start, port-reset cameras missing from the list, and alarm out when the bus cannot be recovered. It is written for USB depth cameras under Linux, driven from a supervisor script, with the numbers to size the retries and the decision rule for each branch.

What "busy" means, and why disabling a stream does not clear it

The camera is not damaged. Its firmware still holds the stream endpoints the crashed process opened, so the device enumerates normally but refuses a new client. A fresh process that disables streams in its own configuration only changes what that new session requests; it does not reach the stale claim at the device. That is why the same camera can fail to start while still showing up correctly in the device list.

The same refusal appears when two live processes both try to own one camera; that variant, between two running streams, has its own diagnosis in two depth cameras USB busy: fix the interface claim error. Prevention is cheaper than recovery: close every pipeline in a finally block, as laid out in camera handle cleanup after Python script errors.

What you need

• Root or sudo on the host, for the port-level reset.
• Your SDK's Python API with a device-list query and a hardware reset, written below as query_devices() and hardware_reset(); substitute your SDK's exact names.
• The expected serial numbers in your station configuration file, not hardcoded in the script.
• lsusb from the usbutils package, and a small userspace USB reset helper called through Python's subprocess module.
• A powered hub for the cameras: a re-enumeration surge lands on top of your normal load.

The startup sequence, step by step

1. Fix the expected set. List the serials each station must see. A count alone (EXPECTED = 3) catches a missing camera; the serial list also catches a swapped one.
2. Query and compare. Call the device-list query once at process start. detected = len(devices); missing = EXPECTED - detected.
3. Reset what you can reach. Start each detected camera's stream inside a try-except. A busy or claim error means the stale client survived: call hardware_reset() on that one device. Never reset healthy cameras — another consumer may be mid-stream on them.
4. Port-reset what you cannot reach. If missing > 0, no SDK call exists for the absent camera. Reset at the port level (next section), then settle and re-query.
5. Settle and re-check. Sleep SETTLE_S after any reset, then query again; two seconds covers a normal USB re-enumeration.
6. Alarm instead of looping. Cap recovery at ATTEMPTS rounds. A camera still missing after three rounds has a physical cause — power, cable or hub — and the cell should stop safely, not spin.

EXPECTED  = 3      # int,   cameras that must be present
ATTEMPTS  = 3      # int,   recovery rounds before alarm
SETTLE_S  = 2.0    # float, seconds between rounds

for attempt in range(1, ATTEMPTS + 1):
    devices = context.query_devices()        # SDK call
    detected = len(devices)                  # int
    if detected == EXPECTED and starts_ok(devices):
        break                                # clear to run
    if detected < EXPECTED:
        usb_port_reset(EXPECTED - detected)  # your helper, via subprocess
    else:
        for dev in devices:
            try:
                start_stream(dev)            # your start wrapper
            except Exception:                # the SDK's busy / claim error
                dev.hardware_reset()         # SDK call
    time.sleep(SETTLE_S)
else:
    raise AlarmError("depth cameras not recovered")

start_stream(), starts_ok(), usb_port_reset() and AlarmError are helpers you write; query_devices() and hardware_reset() are the SDK's calls. When several cameras share one host, boot order and per-camera timing matter too — the startup troubleshooting for multiple USB depth cameras on Linux walks the enumeration side.

Worked example: three cameras on one host, one missing from the list

The inputs, and where each comes from: EXPECTED = 3 cameras, because the cell runs three USB depth cameras from one host, the case this routine exists for. detected = 2 cameras on the first query, the representative failure: the wedged unit stopped enumerating after its owner crashed mid-stream. SETTLE_S = 2 s between rounds, because a USB device re-enumerates within about a second of a port reset and two seconds leaves margin for the hub. ATTEMPTS = 3 rounds, enough to ride out a slow hub and short enough to keep the line's stop time bounded.

The rule in words: a camera missing from the device list is unreachable by any SDK call, so it gets a port reset; a camera present but refusing to start gets the SDK hardware reset; the supervisor proceeds only when the detected count equals the expected count and every camera starts.

The arithmetic: missing = 3 - 2 = 1 camera. Round 1 at t = 0 s: detected = 2 < 3, so the supervisor port-resets the absent camera's port and sleeps 2 s. Round 2 at t = 2 s: the query returns 3 devices and all three streams start, so the loop breaks. Recovery cost: one port reset and roughly 2-3 s of wall time. Worst case, if round 2 also comes up short: reset again, round 3 at t = 4-5 s, and if the count is still short the loop exhausts and raises the alarm at about 5 s elapsed — long before anyone could have walked to the cell and pulled a plug.

The decision threshold: proceed at detected = 3 with zero busy errors; alarm at three failed rounds. A camera that needs its port reset on every startup is not a software fault — treat it as bus hardware, as below.

The USB port reset, and when it is safe to run

lsusb prints one line per device, and the Bus and Device numbers on that line are what the reset helper needs. Grab them from each line with a compiled pattern:

import re
device_re = re.compile(b"Bus\s+(?P<bus>\d+)\s+Device\s+(?P<device>\d+)")

Feed bus and device to a userspace reset helper — a small utility compiled once and kept with the station tools — through subprocess. On hosts without the helper, the kernel's bind and unbind files under /sys/bus/usb/drivers/usb do the same job: write the device's port name (the 1-2.1-style string that /sys/bus/usb/devices shows) to unbind, then to bind, about a second apart. Either path is the programmatic version of the plug cycle.

A camera that vanishes from the list every week is usually a hardware story: a bus-powered hub sagging under three cameras' current draw, a cable flexed past its rating at a moving axis, or a front-panel port sharing a controller with a keyboard. Power the hub and replace the cable before rewriting recovery code. The camera cables range includes purpose-built USB cabling; the LS-CBL-0219 is a 3 m USB3.0 camera cable built for high-flex, drag-chain runs between controller and camera.

Verify the recovery before you trust the data

A reset camera is not automatically a good camera. Before the supervisor resumes production:

• Confirm the detected serial set equals the expected set. Ports and device indexes shuffle; serials do not.
• Stream a few frames and check depth coverage and exposure. A camera that returns with degraded coverage has an illumination problem, walked through in depth coverage changes after startup: check IR exposure.
• Log the recovery action per serial with a timestamp. Three port resets on one serial in a week is a maintenance ticket; resets spread across all serials point at power or the hub.
• If the SDK itself throws while reopening after a crash, camera reinitialization after a crash covers the error order and what to clear first.

Common mistakes

• Disabling streams in the new configuration and expecting the stale claim to clear — the device holds it until a reset.
• Hardware-resetting every device when one is wedged, interrupting cameras that were healthy.
• Treating a line in lsusb as a working camera. The OS can enumerate a device the SDK cannot open; that split is diagnosed in USB camera in lsusb but not detected by the vision SDK on Linux.
• Looping resets without a cap, so the cell never reports its true state.
• Blaming software when the camera drops mid-stream rather than at startup. A device that disappears while streaming is a cable, hub or power story, traced in VIDIOC_DQBUF error 19 no such device: diagnose USB camera drops.

When resetting is the wrong fix

If the busy state returns at every startup, the camera has an owner you did not plan: a leftover autostart script, a vendor tool left running, a second supervisor instance. Find the owner, kill it, and enforce one process per camera. Recovery code then stays a rare path instead of a daily routine.

If the port reset fires weekly, fix the bus instead: a powered hub, one camera per host controller where bandwidth is tight, and cabling rated for the motion in the cell. And if you are specifying hardware rather than rescuing it, the 3D sensors range covers the two depth-camera families behind most inline volume and sensing cells, ToF and projected structured light, which you can filter further in the 3D camera finder:

Before you deploy

• Expected serials in the station configuration, not just a count.
• A try-except around every stream start, with the reset targeted by serial.
• The port-reset helper installed and tested once at the bench, with sudo, before it is needed for real.
• SETTLE_S = 2 s, ATTEMPTS = 3, and the alarm wired to the cell controller.
• Recovery events logged per serial, so a pattern shows up before it becomes downtime.
• One clean shutdown path — the finally block — so recovery stays rare.

If you are building the cell rather than repairing it, choose cameras, cabling and mounts in one pass:

For a multi-camera bus layout — hub power, cable lengths, controller placement — send the camera count and the mounting sketch through the contact page and we will size the bus with you. The other recovery and calibration write-ups sit on the 3D vision topic page.