GPU memory not freed after a camera stream disconnects: fixes

A vision server that runs detection and classification on live camera streams keeps decoded frames, batched inference buffers and the model engine in GPU memory. When a camera goes away — an operator removes it, or the link drops — that memory has to go back. On servers where cameras come and go all day, a pipeline that gets paused instead of destroyed holds its reservation forever, and the card fills until the process crashes. This guide follows the failure in the order you meet it: what nvidia-smi is really showing, how to size the leak in five cycles, what a clean teardown looks like, and the capacity math that keeps a twelve-stream server alive. Numbers are for an 8 GB-class card; substitute your own readings from the same commands.

Why is GPU memory still reserved after the camera disconnects?

Because the disconnect handler paused the camera's pipeline instead of destroying it: paused pipelines keep their decode buffers, inference batch and frame pools on the card, so nvidia-smi keeps showing memory that no frame is using.

In a multi-stream inference pipeline, the stream muxer and the inference engine allocate their GPU buffers when the pipeline is built, and they release those buffers only when the pipeline is fully destroyed. Stopping just the camera source while the decode and inference branches stay alive keeps every one of those pools resident.

The trigger is easy to miss. The transport throws a websocket close exception when the management side drops, the handler logs it, and the pipeline sits in its last good state. One documented deployment saw the build-up both when an operator deleted the camera and when the management server went down — two different causes, one shared pause path. Unstable links multiply the events: every GigE Vision stream timeout is another disconnect your handler has to survive.

Which failure do you actually have?

Five failures share this symptom, and each has a distinct signature in nvidia-smi and torch.cuda's memory_summary(): a paused pipeline, allocator caching mistaken for a leak, per-stream engine copies, slow asynchronous teardown, or a bug in the camera-management plugin itself.

Work down the table in that order. The paused pipeline is the most common by far; engine copies and plugin bugs follow. Every test in the third column takes minutes and needs no extra tooling.

Is the held memory allocated or just reserved?

Most of it is reserved-but-unused: the caching allocator returns freed tensors to its own pool rather than to the driver, and the CUDA context adds roughly 300 to 500 MB per process that never appears as an allocation at all.

memory_summary() under torch.cuda prints the two numbers that matter: allocated, the bytes in live tensors, and reserved, the bytes the allocator's pool holds. nvidia-smi's per-process figure is reserved plus that context, documented alongside the allocator in the PyTorch issue tracker. A process showing 5 GB in nvidia-smi may hold 2 GB of live tensors, so "the number didn't move" after a first fix tells you nothing until you know which line you expected to drop.

How do you measure the leak per disconnect?

Take three nvidia-smi readings — idle baseline, steady state with every stream running, and 60 seconds after one disconnect — the drop from steady state back toward baseline is the reclaim, and the shortfall is your leak per cycle.

1. Record the idle baseline with no streams connected. Example: 1.3 GB.
2. Connect all twelve cameras, run inference for two minutes, record the steady state. Example: 4.9 GB.
3. Disconnect one camera, wait 60 seconds, record again. A healthy system returns that stream's 0.3 GB share toward baseline.
4. Reconnect and repeat on the same camera five times. Leak per cycle = (fifth reading − steady state) ÷ 5.
5. Read the verdict: below 0.03 GB per cycle is noise, a tenth of one stream's share. A 0.25 GB per-cycle leak is the case worked through below, and it fills the card in about twelve disconnects.

What must be torn down when a stream ends?

Four things, in this order: stop accepting frames, flush the frame queues, send the decode and inference chain to its null state, and release the stream's slot in the muxer — anything left alive keeps its GPU buffers.

Treat any close as final, clean or exception. Hold no pipeline open waiting for a reconnect; build a fresh one when the camera returns.

Order matters twice over. Teardown while buffers are still mapped elsewhere produces the CUDA context invalid after camera close crash, and unmapping planes in the wrong order trips illegal memory access in GPU frame mapping. Both cost more debugging than the teardown itself.

Idle callbacks that keep firing after a disconnect belong in the same teardown — disconnect callback handling after idle periods covers unregistering them. The full SDK exit sequence is in shutting down camera streams in the right order.

Why does memory not come back even after teardown?

Two legitimate reasons survive a correct teardown: the caching allocator keeps freed blocks reserved for reuse, and the CUDA context stays loaded until the process exits — only monotonic growth across cycles is a true leak.

So expect a floor, not zero. After correct teardown the per-process figure lands on context plus engine — 1.3 GB in this example — and stays there. The allocator's pool-release call under torch.cuda drops reserved toward allocated; run it after each removal event, not per frame, because it stalls synchronisation. The context loads once per process and unloads only at exit, which is why twelve cameras served by one process is the memory-cheapest design and why a process restart reclaims everything.

How much GPU memory should the server plan for?

Budget the steady state plus a 30 percent margin: for twelve streams on an 8 GB card that is 4.9 GB steady plus 1.5 GB margin, and the alert threshold sits at 6.4 GB — early enough to act long before the card fills.

Worked example, using the measurements from the section above:

• context 0.4 GB, engine 0.9 GB, twelve streams × 0.3 GB share: steady = 0.4 + 0.9 + 3.6 = 4.9 GB
• headroom = 8.0 − 4.9 = 3.1 GB
• margin = 0.3 × 4.9 ≈ 1.5 GB, so the alert threshold = 4.9 + 1.5 = 6.4 GB
• measured leak 0.25 GB per disconnect: the alert trips after 1.5 ÷ 0.25 = 6 disconnects, and the card fills after 3.1 ÷ 0.25 ≈ 12

Decision: alert at 6.4 GB, fix or recycle the same day, and never commission a server whose steady state exceeds 70 percent of the card.

Disconnect events are usually cable and port events, so the host decides how often this whole loop runs. One camera per port keeps a fault local, and locking high-flex connections stop vibration re-creating the test every hour. The link-level causes are covered in diagnosing single camera disconnects during capture, and the frame grabbers and machine vision cables categories carry the parts.

For the host side, these parts fit that job:

The MV-GE1104 is a 4-port GigE frame grabber. One camera per port keeps a single link fault to a single stream. The MV-GE2002P is a dual-port PoE NIC for smaller cells and the management link. The LS-CBL-0199 is a 3 m high-flex GigE camera cable with screw locks, built for drag-chain use.

Two ways to give cameras their own ports:

All 2 share: Camera type supported GigE industrial camera; Dimension 181 mm × 126.3 mm × 21.6 mm; Humidity 10% RH to 95% RH (no condensation). Values as printed in each product's data sheet; open a part for its drawing and full specifications.

A typical GigE source for a detection station on that server:

The LS-CMR-0053 is a 12 MP GigE area scan camera with a rolling shutter. At roughly 0.3 GB of GPU reservation per stream, the formula above holds. Pick further sources in the camera finder or start the system configurator at the camera step.

When is the leak not in your code?

When your teardown is provably correct and the growth persists: in one documented case the camera-management plugin itself paused per-camera pipelines on every removal, the vendor confirmed the bug, and only the fixed plugin release cleared the card.

Prove yours in an afternoon. Reproduce the disconnect with the vendor's own sample pipeline — same removal action, same per-cycle nvidia-smi log. If the sample leaks too, the bug sits upstream of your code: send the cycle log to the vendor, move to the fixed release, and record the plugin version in the commissioning file. Until the fix lands, contain it: run the plugin in its own process, so a scheduled restart between cycles reclaims the whole context.

What checks prove the fix before production?

Run a 500-cycle soak: connect and disconnect one camera on a timer, log the per-process nvidia-smi figure every cycle, and pass only if it returns to within 5 percent of its floor each time and never ratchets upward across the run.

• Baseline, steady-state and post-disconnect readings recorded, with the stream count noted beside each.
• Disconnect handler destroys, never pauses, the per-camera pipeline — on the clean close and on the exception path alike.
• Allocator pool released after every removal event; the release shows as reserved dropping in memory_summary().
• Alert threshold at steady state plus 30 percent; watchdog kill gated between production cycles.
• Plugin version recorded, and the vendor's fixed release installed if that bug was implicated.

For the server around the fix, multi-camera system design for reliable capture covers the architecture, assigning cameras to ports reliably keeps discovery clean as cameras come and go, and the systems topic page collects the rest. Part-level questions go through the contact page.