Multi-camera stream failure recovery: contain the crash in userspace

Two cameras streaming through an embedded Linux capture stack, one disrupted link, and the whole controller reboots. First the kernel log fills with warnings from the capture driver's control path, then a paging fault arrives, then the board is gone. The failure is not in your application code; it sits in the error path of the kernel-side capture pipeline, and it bites hardest when captures run concurrently. This guide gives you the log signature to confirm it, the causes ranked by probability, a test for each, a worked watchdog calculation, and the containment pattern that keeps every recovery in userspace.

What the crash looks like

The event is intermittent by nature. Two or more captures run side by side, something disrupts one stream, and nothing happens. Or the machine dies an hour later. Or it dies on the very first disruption. The log signature is the reliable part.

Run journalctl -kf and read two stages. First come WARNING lines from the capture driver's control-transport module, reporting that a control callback is unregistered or notified while captures are still active. Then the next capture start faults with Unable to handle kernel paging request, and the call trace runs through vb2_ioctl_streamon and vb2_core_streamon into the capture driver's channel-start code and down the DMA allocation path: dma_alloc_attrs, __iommu_dma_alloc, sg_alloc_table_from_pages_segment. Then the board reboots.

That trace points at a race, not at your code: the teardown of a dead capture and the startup of a new one touch the same driver structures with no lock between them. The single-camera side, a device that vanishes and returns, is covered in Diagnose single camera disconnects during capture; the multi-camera case is the one that reboots the machine.

Causes, ranked by probability

Work the list top down; each cause has a test in the next section.

1. Teardown/start race in the capture driver's error path. Most likely when the trace lands in DMA allocation during streamon, right after another stream died. One capture tears down while another registers a control callback, nothing serializes the two, and the paging fault follows.
2. Sensor reset while frames are in flight. A register write that resets a streaming sensor stops its clocking mid-frame. The pipeline holds buffers that will never fill, and the error path starts exactly when a sibling capture is starting up.
3. An unstable electrical link. Long flat-flex cables, connectors near vibration, routing alongside motor drives or switched power. Every repeat loss re-enters the error path, so the odds of hitting the race compound run over run.
4. Resource pressure. A full destination filesystem or exhausted memory makes restarts fail in new ways, and several USB cameras on one host controller add bandwidth errors, as in Acquire from multiple USB cameras: bandwidth and sync.

Test each cause before you change code

Run the four tests in order and let the log pick the cause, not a hunch.

1. Reproduce with a scripted failure. Start your capture loops, then a third loop that kills one capture mid-run or cuts one camera's power through a relay. Watch journalctl -kf: WARNING lines followed by the paging fault confirm cause 1.
2. Take I2C out of the equation. Repeat the test with no register writes at all, then allow resets only between capture runs. A crash count that goes to zero hands the case to cause 2.
3. Stress the link. Shorten the failing camera's cable, reroute it away from drives and power wiring, reseat the connector. A fault rate that falls with the cable work points to cause 3.
4. Check resources. Capture to a tmpfs mount with headroom, confirm free memory, spread USB cameras across host controllers. Faults that track disk or memory pressure point to cause 4.

Sizing the watchdog for a 200-frame capture at a 10-second timeout

The containment pattern needs a per-capture timeout, and that timeout needs arithmetic, because a badly sized watchdog makes things worse: it kills healthy captures and churns the very restart path you are trying to keep quiet.

Inputs. From the reproduction case: a capture of 200 frames per run, supervised by a 10 s timeout, at a format of 2592 × 1944 pixels. The frame rate comes from your driver's mode table; this example assumes 30 fps for that format.

Rule. The longest legitimate run must stay within about 80% of the timeout, because the watchdog must outlast every healthy run with margin. The timeout itself then sets your worst-case detection time for a stall.

Arithmetic. A legitimate run lasts 200 ÷ 30 fps = 6.7 s, which is 67% of the 10 s window and inside the 80% ceiling. The loop tolerates any mode at 20 fps or faster, because 200 ÷ 10 s = 20 fps is the break-even. A slower mode fails: at 15 fps, 200 ÷ 15 = 13.3 s, longer than the whole window. Fix that by cutting the capture to 0.8 × 10 s × 15 fps = 120 frames, or by raising the timeout to 13.3 ÷ 0.8 ≈ 17 s.

Decision. Keep every capture run inside 80% of its timeout. A stalled channel is then detected within one timeout window (10 s here) plus the restart, so recovery is bounded in seconds. A kernel panic is not bounded at all.

Contain each capture in its own supervised process

With the timeout sized, the fix is mechanical: one capture process per camera, each run under timeout, each failure restarted from scratch. Per-process capture also removes the shared-state problems of driving several devices from one program, as laid out in Multiple vision software instances: one camera per process.

# one loop per camera; /mnt/tmpfs is a tmpfs mount
while true; do
  timeout 10 v4l2-ctl -d /dev/video0 \
    --set-fmt-video=width=2592,height=1944 \
    --stream-mmap --stream-count=200 \
    --stream-to /mnt/tmpfs/cam0.raw
  echo "$(date -Is) cam0 restart" >> /var/log/cam0-restarts.log
  sleep 0.5
done

Three details make the loop survive. Write the raw dump to a tmpfs mount, so a hung write cannot wedge the capture. If the loop dies instantly after an edit to the format line, read the height you set: a raw V4L2 capture fails outright when the height is undersized, per this raw V4L2 capture format note. Configure the transport layer to hand you fresh frames instead of stale ones, per Stream buffer handling mode: set newest only on the transport layer. And when you reopen a device after a failure, stop acquisition cleanly before closing, per USB camera timeout on reopen: stop acquisition before you close.

Never reset a sensor over I2C while it streams

That reset write, the one from cause 2, is worth its own section because it is the trigger you control. The link goes silent with buffers armed, the driver's error path starts, and if any sibling capture is in streamon at that moment you are back in the race from the log trace.

The discipline is a fixed sequence: stop acquisition on that sensor, close the device, write the registers with i2ctransfer, then open and stream again. Runtime changes such as exposure or gain belong in the driver's documented controls, which sequence the writes for you, not in raw register writes, as in Apply new camera settings without replug: set, stop or reset.

Take capture-driver fixes through the vendor's channel

The faulting error path lives in the kernel capture driver, so the durable fix is a driver update, and the only good source is the board vendor's own BSP release. Fixes for this class of bug are small and specific: a lock around the control-channel registration so teardown and startup cannot interleave. Ask the vendor for exactly that, the error-handling update for concurrent multi-camera capture, and check the release notes name it before you build.

Forum tarballs are the wrong channel. They lag the current BSP, they conflict with vendor support, and a mismatched driver stack hides the real fault. Pin the vendor release, rerun the soak below, and keep kernel, driver and capture library versions in your deployment notes so the next board matches.

Prove it with a randomized-failure soak

A fix is not a fix until a machine has failed on schedule for hours. Run your supervised capture loops, plus a chaos script that disrupts one camera at random intervals between 0.5 s and 5 s, either by killing one capture process or by cutting one camera's power through a relay. Let it run for at least 8 hours, unattended if your test bench allows it.

Pass criteria, all three: zero reboots; zero kernel WARNING lines in the journal; and every recovery completes within one timeout window (10 s in the worked example) plus the restart time, which you measure once during the soak. Keep a restart counter per camera: a channel that restarts more often than its siblings has an electrical or driver problem of its own, and the counter finds it before production does.

When the embedded pipeline is the wrong tool

If your platform keeps adding cameras to the same fragile capture path, move the cameras instead of patching the path. GigE and USB3 cameras deliver frames over standard links, so a dead stream is a socket or USB error inside your process: the kernel capture pipeline with its control channel never enters the picture. Watchdogs are still worth running, but a failure costs a stream restart, not a board reset.

For a bench of mixed-resolution cameras, the area scan cameras category covers GigE and USB3 models from 2.5 to 20 MP:

On the host side, a PoE network card carries data and camera power on one cable per camera. The MV-GE2002P is a dual-port PoE GigE NIC, and the LS-FGB-0004 is a PCIe x4 card with four PoE Gigabit ports on an Intel i210AT controller. Where several cameras need one machine, the MV-VC3301H-128G66 vision controller pairs an Intel Core i3 with 8 GB of memory, five or more LAN ports and four built-in light channels.

For deterministic capture on Camera Link or CoaXPress cameras, frame grabbers take the acquisition off the CPU path entirely. Whichever route you take, give each camera a fixed port assignment at boot, per Assign cameras to ports reliably in a multi-camera system. The camera finder filters by interface, resolution and shutter in one pass if you are speccing replacements.

Checks before the next deployment

• One capture process per camera, each under its own timeout, with run time inside 80% of it. This is what bounds recovery.
• Raw dumps on a tmpfs mount with headroom, and restart counters logged per camera. Disk-full must never look like a camera fault.
• No register writes to a streaming sensor; register scripts gate on the capture PID. This removes cause 2 entirely.
• Driver stack at the board vendor's current release, with the multi-camera error-handling fix in its release notes.
• Cable routes short and away from drives and power wiring, connectors seated, strain-relieved on moving axes.
• Randomized-failure soak of at least 8 hours passed: zero reboots, zero kernel warnings, recovery inside one timeout window every time.

If you are speccing the host side, start from the vision controllers category and the camera configurator. More containment and recovery patterns sit on the systems topic page, and you can bring a journal log to the contact page for a read-through.