Two MIPI cameras timeout after hours: fixing long CSI runs

You have two MIPI CSI-2 cameras streaming on a Jetson AGX Orin. Both run clean for a while, half an hour to four hours in one logged case, and then nvarguscamerasrc raises an Argus timeout, the pipelines drain, and every later attempt to open the sensor returns 'Device or resource busy'. A single sensor on the same host ran more than eight hours without an error, and that difference decides the whole investigation. This guide walks the failure in the order you meet it: what the error pair means, which clocks to lock, why the firmware update must be a matched set, how to size recovery, and when to change interface.

Why do two MIPI cameras fail only after hours of streaming?

Both sensors share one capture chain, meaning the VI, ISP, CSI and memory clocks plus a single camera daemon, so the second stream pushes a shared resource past its stability limit: one sensor ran past 8 hours where two failed between 30 minutes and 4 hours.

Your GStreamer graph is not at fault. The same drop appeared whether the streams went to a udpsink over the network or to a local display sink, so the sink and the network are cleared. What the second sensor adds is shared: the VI capture engine, the ISP, the CSI-2 receiver, the memory controller and the clock domains that serve them. For where those blocks sit between the sensor and your application, read MIPI CSI-2 image capture pipeline: sensor to host step by step.

Put numbers on the shared load. NV12 packs 1.5 bytes per pixel, so the two streams in the logged case moved:

That is before ISP processing, conversions and buffer copies, so treat it as a floor. It is not throughput starvation: the fps counter in the failing run held a 59.95 average with zero dropped frames until the second the error fired.

It is continuous traffic that keeps the vi, isp, nvcsi and emc clock domains from ever idling. Every re-rate of those domains under two active capture sessions is a chance for a missed arbitration window. Heat compounds it, because two sensors plus a busy ISP put the SoC at hour three in a thermal state it never saw at minute one.

A camera that drops only after a working period is a known pattern on single-board hosts, not a defect in your code. This CSI camera timeout after a working period write-up shows the same shape on a smaller board.

What do the Argus timeout and a busy video node tell you?

That pair of errors is the signature of a host-side lockup: the camera daemon stopped answering its RPC socket, and the kernel capture node is still held by the dead client, so every v4l2-ctl call returns 'Device or resource busy'.

On the daemon side the trace reads Error Timeout propagating from the RPC socket client, and the Argus client then exits with four outstanding threads. The daemon is no longer answering; your pipeline is only the messenger. The follow-up test localizes it. Run a one-frame capture straight against the video node:

v4l2-ctl -d /dev/video0 --set-fmt-video=width=1500,height=1500,pixelformat=BG10 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=1 --stream-skip=0 --stream-to=frame.raw

Every call fails: VIDIOC_S_EXT_CTRLS on bypass_mode, VIDIOC_S_FMT and VIDIOC_REQBUFS all return 'Device or resource busy'. The node is held by a client that will never release it. That is why unplugging the sensor or reseating the ribbon changes nothing. Contrast the alternative: a sensor or cabling fault shows CRC errors, short frames or an I2C failure within the first seconds of streaming, as covered in MIPI CRC errors after the first stream: pixel clock and mode timing. For how a stale holder keeps a node busy across restarts, V4L2 device busy after restart: the old node still holds the camera works the same mechanism from the recovery side.

Which clocks do you lock, and how far does it get you?

Lock the vi, isp, nvcsi and emc clocks to their maximum rate through the bpmp debug interface; in the logged case that moved the failure from inside 4 hours to roughly 1.5 hours at 60 fps, which is better but not fixed.

The writes go through the bpmp debug clock interface, one lock flag and one rate write per domain:

echo 1 > /sys/kernel/debug/bpmp/debug/clk/vi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/isp/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/nvcsi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/emc/mrq_rate_locked
cat /sys/kernel/debug/bpmp/debug/clk/vi/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/vi/rate
cat /sys/kernel/debug/bpmp/debug/clk/isp/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/isp/rate
cat /sys/kernel/debug/bpmp/debug/clk/nvcsi/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/nvcsi/rate
cat /sys/kernel/debug/bpmp/debug/clk/emc/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/emc/rate

mrq_rate_locked tells the power-management processor to stop re-rating that clock, and writing max_rate into rate parks it at the top. After these writes the two-sensor run survived to about 1.5 hours at 60 fps before the same timeout: the interval moved, the mechanism did not. Read that as diagnosis, not cure. If freezing the clocks changes the failure, the clock-manager interaction is in the fault chain, and what remains sits in the camera firmware stack, which is the next section.

Why did the camera firmware update stop the cameras launching?

The RCE camera firmware and the user-space camera libraries must come from one matched set; in the logged case the first mismatched library drop made nvarguscamerasrc exit at launch, at execute:759 with 'No cameras available', instead of streaming.

The update path on the R35.5 release replaces the camera real-time-controller image, camera-rtcpu-t234-rce.img, together with the user-space camera libraries libnvscf.so and libnvfusacap.so. Mix an old component with a new one and the stack fails at bring-up rather than mid-run: no pipeline starts, which blocks testing but tells you the set is mismatched. A corrected, matched set restored streaming on the same hardware with no wiring change. Confirm the running release in /etc/nv_tegra_release before and after, so you know which component set you actually have. Sensor drivers and host-side tuning behave as a pair, and the checklist for that pairing is in MIPI CSI camera bring-up: sensor and host tuning must match as a pair; launch-time no-camera failures are worked through in IMX477 not detected on Jetson Orin Nano: CSI bring-up checks.

How do you size recovery so unattended runs survive?

Detect the timeout, tear down both pipelines, restart the camera daemon and relaunch, sized by arithmetic: against a 1.5-hour mean failure interval, a 90-second recovery keeps uptime at 98.4 percent and a 20-second daemon restart at 99.6 percent.

At the observed interval those restarts still total 112 events per week, which is why the firmware fix comes first and the watchdog second. The watchdog is a small supervisor per camera: run the stream under it, treat an Argus timeout, or a rendered-frame counter that has not advanced in three frame periods, as the fault signal. It then kills the pipeline tree, restarts the camera daemon, waits until the single-frame v4l2-ctl capture succeeds, and relaunches. A soak test with fpsdisplaysink into fakesink gives you the counter with no display or network in the way:

gst-launch-1.0 -vvv nvarguscamerasrc sensor_id=1 ! video/x-raw(memory:NVMM), width=(int)800, height=(int)800, framerate=30/1, format=(string)NV12 ! nvvidconv ! fpsdisplaysink video-sink=fakesink --verbose

Before you blame the clocks, rule out the slow leak: a capture process that grows memory or file descriptors fails the same way at hour two every night, and Long-running capture pipeline crash: isolate the leak first is the procedure for separating it. And when two of your own clients touch one node, the same busy error can be yours rather than the daemon's, so check for a second open handle on the node before you restart anything.

When should you switch away from MIPI to a cabled camera?

Switch when the station must run unattended for days, when you need more than two sensors on one host, or when the camera sits beyond ribbon-cable reach; a USB3 or GigE board camera gives you a finished stack and cable runs measured in meters.

MIPI CSI-2 is a board-to-board link: short flex, no locking in most builds, and every stability fix is a firmware and clock-tuning exercise on the host. For unattended production, an interface whose whole stack ships finished removes that class of work. The board-level cameras keep the compact form factor, and full area scan cameras cover machine-vision builds where GigE reaches 100 m on Cat5e against a ribbon's centimetres. Browse the camera finder by sensor and mount, or start at the camera step of the system configurator to match sensor, lens and cable in one pass.

The LS-BCM-0003 is a 3.1 MP USB3 board-level camera with an IMX252 sensor, a 1/1.8" global shutter and 120 fps.

For runs over a network, the MV-CB016-10GM-M-W is a 1.6 MP GigE board-level camera with an IMX296 sensor, a 1/2.9" format and an M12 lens mount.

Cable them for movement: the LS-CBL-0219 is a 3 m USB3.0 camera cable rated for drag-chain use, and the LS-CBL-0117 is a 15 m RJ45 camera data cable with a locking stud.

A third route keeps the SoC: GMSL2 cameras on coax, two of them through one deserializer, as in wiring two GMSL2 cameras through a MAX9296A deserializer.

What do you check before calling the fix good?

Re-run the two-sensor soak for at least three times the worst old failure interval, meaning 12 hours against the 4-hour worst case and 24 to 72 hours if the station runs around the clock, logging frame rate, temperatures and daemon errors to one file.

• Baseline first: one sensor into fakesink for 8 hours or more, with the clock locks applied. If the single-camera baseline now fails too, the fault has moved into hardware or power and the two-camera theory is void.
• Run the pair for the full 12-hour minimum with the locks applied from boot, not from a manual shell you forgot on Tuesday.
• Log everything to one timestamped file: the fps counter, SoC temperatures, dmesg and the daemon log, so the moment of failure can be reconstructed.
• Drill the watchdog on purpose: kill a pipeline mid-stream and measure recovery time against the MTTR you assumed in the arithmetic.
• Confirm the node returns: the single-frame v4l2-ctl capture must succeed after a forced recovery before anything relaunches.
• Re-read /etc/nv_tegra_release after any update, so the soak you trust ran on the stack you think it ran on.

The decision, case by case: a bench rig or a supervised demo lives with locked clocks plus the watchdog and a soak test; an unattended line takes the matched R35.5 camera stack, and if the pair still drops on that stack, it takes the cabled interface instead. If you are between those cases, capture the daemon log around one failure and bring it to the contact page with your two pipelines and clock settings, and we will work it with you.