Long-running capture pipeline crash: isolate the leak first

A pipeline that captures, processes and ships results without complaint at commissioning, then dies with a SIGSEGV at hour one or hour ten, is the hardest class of vision failure to triage: the crash site tells you almost nothing about the cause. This guide gives you the three measurements that separate your bug from the vendor stack's — memory growth per minute, outstanding-buffer counts, and a sanitizer verdict on the processing path — plus the queue-sizing and soak-test numbers that keep the failure off a production shift. The arithmetic works the same whether your frames arrive from area scan cameras over GigE or board-level cameras on an embedded board.

Why does a capture pipeline that runs for minutes crash after hours?

A failure that appears only after 30 minutes to 10 hours of continuous capture is accumulated damage, not a start-up misconfiguration: memory grows until the allocator fails, or a buffer is freed while another thread still writes it. Start-up faults show on frame one and have their own diagnostic path for capture pipeline startup failures; endurance faults need measurement instead. Rank the suspects in this order:

1. An unbounded queue. Frames arrive faster than the consumer drains them; memory climbs until allocation fails.
2. A per-frame leak. A buffer is allocated for each frame, or for each error, and never released.
3. A buffer-lifetime bug. Memory is freed or recycled while another thread still holds a pointer; the heap corrupts and detonates far from the cause.
4. A defect in the vendor capture stack or camera firmware. Last, and only by elimination.

What to measure first when the crash repeats every few hours?

Log the process RSS (the VmRSS line in the per-process status file under /proc) and the length of every frame queue once a minute. A straight RSS slope names a leak; a flat RSS with a growing buffer count names a handle or descriptor leak. Write the samples to a small ring file on disk — two hours of log convicts or acquits the leak hypothesis, and it survives the crash that a debugger session may not. Read the signature from the table:

The slope converts directly into a crash prediction:

Worked example: a 4000 × 3000 mono 8-bit sensor produces 4000 × 3000 × 1 = 12,000,000 bytes = 11.4 MiB per frame. Your error path allocates a fresh buffer for every dropped frame, and one frame in 300 is dropped — once every 10 s at 30 fps. The leak rate is 12,000,000 × 6 = 72,000,000 bytes/min = 68.7 MiB/min. The board has 8 GiB of RAM and the pipeline sits at 4 GiB steady state, so headroom is 4096 MiB, and 4096 ÷ 68.7 ≈ 60 min to crash. Halve the drop rate and the crash moves to roughly two hours — which is exactly why these failures scatter across a 30-minute to 10-hour window.

How to prove heap corruption instead of a plain leak?

Rebuild your processing path with the capture layer stubbed out — record an hour of frames once, replay them from disk — and run it under AddressSanitizer. A use-after-free or double-free report convicts your code before you touch the vendor stack. The vendor's capture libraries are closed and uninstrumented, so a sanitizer only watches code you compile; stubbing gives it the whole processing path while keeping the hardware out of the loop.

Read the crash signature too: a SIGSEGV raised inside malloc's own free-list bookkeeping is corruption speaking — the fault lands where the damaged list is walked, not where the damage was done. Run Valgrind on the same replay afterwards; leaks that Valgrind flags can go unreported under AddressSanitizer's replacement allocator. The ownership rules that prevent this class of bug are in the guide to fixing heap corruption from image buffer lifetime, and the cross-thread case is the torn-frame callback rule: copy the buffer before you queue it.

How large should frame queues be so the pipeline survives bursts?

Bound every queue and size its depth from frame bytes: 30 frames of a 4000 × 3000 mono 8-bit sensor is 30 × 11.4 MiB = 342 MiB per queue. Three queues cost about 1 GiB and can never grow beyond it. Depth then follows from the slowest consumer.

At 30 fps in and 25 ms of inference per frame, the consumer sustains 40 fps and a shallow queue rides out bursts. At 40 ms per frame it sustains 25 fps, so 5 frames per second accumulate: 5 × 11.4 MiB = 57 MiB/min, and 4096 MiB of headroom lasts 4096 ÷ 57 ≈ 72 min. Size the depth from the burst you must absorb, set an overflow policy up front, and drop the oldest frames rather than the newest. The implementation pattern is walked through in the guide to bounding the frame queue and reusing frames; with several cameras, apply the callbacks-and-buffer-queues checks per camera, not per process.

Which patterns in your own code leak or corrupt buffers?

Three patterns account for most endurance failures: a fresh buffer allocated per frame and never released, a callback that hands a pointer to another thread without transferring ownership, and a queue that grows without a cap. Fix them in place.

For the leak: allocate a fixed ring of buffers once and recycle them for the life of the process; steady-state allocations per frame should be zero, so put a counter on your allocator and treat any per-frame increment as a regression. For ownership: every buffer has exactly one owner at any instant, hand-off transfers it, and the receiving module releases it. For queues: a cap and an overflow policy, never an unbounded append. The rewrite is shown step by step in the guide to reusing image buffers instead of per-frame allocation, and the same bug at video-loop scale is the per-frame image leak that exhausts memory.

When is the crash actually the vendor stack's fault?

Blame the vendor capture stack only when three signatures agree: your processing path passes a sanitizer run, a vendor sample application crashes unchanged over the same duration, and RSS and buffer counts stay flat until the fault. A backtrace that ends inside vendor libraries is where corruption detonates, not where it started: malloc's free list is walked millions of times per second, so a corrupted list crashes there no matter which module damaged it.

When all three signatures hold, report to the vendor with the full backtrace, the stack and firmware version strings, the RSS log and the shortest reproduction you measured. That package turns a crash report into a fix on the first reply; a backtrace alone turns it into a guessing game.

What to update once your own code is clean?

Update the capture stack and camera firmware to the current stable release through the vendor's own procedure, and record the running version string before and after, because endurance fixes ride along with such releases. Two details make the update stick.

First, match the pair: vendors pin camera firmware to a matching capture-stack release, so read the running version string before flashing and record both values in your logs. Second, flash only from the vendor's own script or recovery flow, never from an image found elsewhere. Expect a sequencing effect: the memory crash disappears and a lesser error surfaces — log it with timestamps rather than assuming the update failed. Sensor-side errors that survive a clean software bill usually trace to the physical link, so swap the cable and reseat connectors before rewriting code.

How to keep an endurance failure from reaching production?

Soak the exact production build for at least twice the longest shift it will run — 16 hours for an 8-hour line — with RSS and queue lengths logged every 60 seconds and an alarm on any sustained growth. Acceptance is arithmetic: require the projected time-to-crash from the RSS slope to exceed five times the longest shift. With 4096 MiB of headroom and 8-hour shifts, that means a leak below 4096 ÷ 40 ≈ 102 MiB/h ≈ 1.7 MiB/min — a pipeline leaking 68.7 MiB/min fails this gate inside the first hour of the soak.

Keep a watchdog that restarts the pipeline with a short backoff, preserves the log ring and a core dump, and counts every restart as a failed run: recovery buys uptime, never silence. Bringing the capture device back after a fault has its own steps; the guide to releasing a camera after a crash on Linux covers the GigE case.

If you are sizing a new capture chain for continuous duty, start with the camera finder and the system configurator, both of which filter by the interface and resolution your soak numbers assume. The systems guides hub collects the rest of the pipeline-stability articles.