Python frame queue memory leak: bound the queue and reuse frames

A hardware-triggered capture loop in Python works perfectly for one trial, then memory climbs trial after trial until the machine can no longer allocate. The reflex — more gc.collect() calls — changes nothing, because every frame is still referenced by something the loop itself created: an unbounded queue, a writer thread that never exits, or a camera buffer the callback kept. This article walks the three holders in the order they usually occur, with the arithmetic for a 1456 × 1088 Mono8 stream, the bounded producer–consumer pattern that fixes them, and a per-trial measurement that proves the leak is gone.

Symptom: memory climbs across trials and gc.collect() changes nothing

Per trial everything works: the trigger fires, frames arrive, the stack saves to disk. Between trials the process's resident memory never drops back, and after enough trials allocation fails mid-run. Scattering gc.collect() through the loop cannot help: Python's collector frees only objects nothing references, and every candidate here is referenced. The queue holds each tuple you put in it, a blocked thread's stack holds its local arrays, and the driver keeps its buffer pool for as long as streaming ran.

Two questions split the cases before you touch any code. Does memory grow only while streaming runs, or does it step up once per trial even while the camera sits idle? And does the live thread count rise by one per trial? The three causes below follow in the order they usually appear: the queue, the threads, then the driver buffers.

First test: log RSS, queue depth and thread count per trial

Add one log line per trial and the cause usually names itself:

import os
import threading
import psutil                      # pip install psutil

proc = psutil.Process(os.getpid())
rss_mb = proc.memory_info().rss / 1e6      # float: resident memory in MB
threads = len(threading.enumerate())        # int: live Python and SDK threads
print(f'trial {i}: rss={rss_mb:.1f} MB, threads={threads}, queued={frame_queue.qsize()}')

Run ten trials with the camera idle between captures. Growth only during streaming, with the queued count climbing, is the unbounded queue in Cause 1. A step at the end of each trial that persists while idle, with the thread count rising one per trial, is Cause 2. Flat memory with missing frames points away from leaks entirely, toward the checks at the end. The same log doubles as the proof after the fix, so keep it in place. More acquisition-side patterns sit on the systems topic page if your loop outgrows this one.

Cause 1: the frame queue is unbounded

queue.Queue() with no maxsize accepts every frame the callback hands it, without limit. The camera side is the fast producer: hardware trigger rates run from a few frames per second into the hundreds. The consumer side is slow by design — it fills a numpy stack and writes it to disk — so any rate gap becomes a backlog of frames in RAM.

Frame size sets the stakes. A 1456 × 1088 Mono8 frame is 1456 × 1088 × 1 byte = 1,584,128 bytes, about 1.6 MB. At a 60 fps trigger with the consumer keeping only 20 fps, the backlog grows at 40 frames per second: 40 × 1.6 MB ≈ 63 MB per second, roughly 3.8 GB in a one-minute run. Nothing in the standard library slows that down.

The fix is one constructor argument: queue.Queue(maxsize=8) caps the queue at 8 × 1.6 MB ≈ 12.7 MB no matter what the producer does. Put with put_nowait() inside the callback and choose the overflow behaviour deliberately — drop the frame and count the drop, or block for a bounded time you can afford. Print frame_queue.qsize() after each trial: a large value after the writer has finished means the backlog outlived the run.

Cause 2: one more writer thread per trial, and none of them exits

The per-trial step in the RSS trace comes from a loop shape like this: each capture starts a fresh writer thread, and the writer wraps a fixed-length for loop inside while True:. After its last frame the outer while restarts, the thread blocks on get() forever, and nothing ever joins it. The thread itself is small — what it keeps alive is not. Its locals still reference the preallocated result stack, so one complete stack stays resident per trial, and the SDK callback thread joins the pile if streaming is not stopped cleanly either.

The stack is 280 frames × 1,584,128 bytes = 443,555,840 bytes, about 444 MB. Twenty orphaned trials therefore hold 20 × 444 MB ≈ 8.9 GB. On a 16 GB machine with roughly 12 GB free to the process, allocation failures start around trial 27 — exactly the 'works fine, then out of memory' pattern.

Confirm it with the thread count from the first test, then fix the shape: let the writer exit after its frame count and join it before the next trial starts.

def writer(frame_queue, data, ticks, n_frames):
    # data: (n_frames, H, W) uint8 array, preallocated before streaming starts
    for i in range(n_frames):            # exits when the stack is full
        img, ts = frame_queue.get()      # img: (H, W) uint8 copy
        data[i, :, :] = img
        ticks[i] = ts
        frame_queue.task_done()

t = threading.Thread(target=writer, args=(frame_queue, data, ticks, n_frames))
t.start()
recorder.run_session()                   # start streaming, wait for the trigger window, stop
t.join(timeout=10)
assert not t.is_alive(), 'writer thread did not exit'   # the check that proves it

Where the application allows it, start streaming once per session and make each trial a trigger window instead of a new stream plus a new thread. If teardown per trial is unavoidable, stop streaming and join both directions before the next one begins.

Cause 3: camera buffers never go back to the driver

Callback streaming hands you frames from a fixed pool of driver buffers. The contract is simple: copy the pixels out inside the callback, then return the buffer so the driver can fill it with the next exposure. If any code path keeps the frame object alive, or skips the buffer-return call after an exception, the usable pool shrinks a little more each time it happens. The visible effect depends on the SDK: a stall after roughly the buffer count of frames, a rising dropped-frame counter, or fresh allocations that do show in RSS. The copy-out and buffer-return call names differ between SDKs, so take both from your SDK's Python reference — the pattern itself is identical everywhere.

Whatever the calls are named, make the return unconditional: on the success path and on every drop and exception path, before the callback returns. And keep the slow work out of the callback; a pixel copy costs microseconds, a TIFF write does not.

The fix: bound the queue, return the buffers, keep one writer

Assembled, the bounded pattern looks like this:

import queue
import threading

MAX_IN_FLIGHT = 8                        # int: frames allowed in the queue at once
frame_queue: queue.Queue = queue.Queue(maxsize=MAX_IN_FLIGHT)

def on_frame(cam, frame):                # callback signature per your SDK reference
    img = frame.copy_pixels()            # generic name: the documented copy-out call
    try:
        frame_queue.put_nowait((img, frame.get_timestamp()))
    except queue.Full:
        pass                             # deliberate drop; count it if frames matter
    cam.requeue_frame(frame)             # generic name: the documented buffer-return call

Run the ten-trial log again after the change. Three results prove the fix: the queued count never exceeds 8, the thread count is identical every trial, and RSS plateaus within about 50 MB of its trial-1 value instead of stepping up by roughly 444 MB per trial.

If you would rather not carry a queue at all, the hardware trigger grab loop in Python builds the same capture single-threaded. The ownership discipline carries across runtimes too: the frame memory lifetime article for managed code covers the same rules in C#, and in LabVIEW the queue reuse notes with malleable VIs configure the equivalent buffer reuse.

The numbers, worked end to end

Worked end to end for the example stream: one frame is 1456 × 1088 × 1 B = 1,584,128 B ≈ 1.6 MB. The 280-frame stack is 280 × 1,584,128 = 443,555,840 B ≈ 444 MB, so an orphaned writer leaks 444 MB per trial and a 12 GB budget runs out near trial 27. With maxsize 8 the ceiling is 8 × 1.6 MB ≈ 12.7 MB, and nothing scales with trial count any more.

Resolution moves the first row, not the logic: a 4000 × 3000 Mono8 frame is 4000 × 3000 × 1 B = 12.0 MB, so the same eight-slot queue holds about 96 MB. Still bounded — just four times higher.

Checks before you call it fixed

Run each check at the stage where it catches the failure:

While writing the loop:

• Construct the queue with maxsize from the first line — unbounded is the default, and the default is the bug.
• Copy pixels inside the callback and return the buffer on every path — the driver pool is finite.
• Preallocate the result stack once per trial and write frames into it in place — 280 × 1.6 MB ≈ 444 MB allocated once, not 280 separate allocations.

Per trial:

• Log RSS, queued and thread count on one line — three numbers, three signatures, one grep.
• join() the writer with a timeout and assert it exited — a thread blocked on get() leaks its locals even with a bounded queue.
• Read the SDK's dropped-frame counter alongside RSS — rising drops with flat memory is a throughput problem, not a leak.

Before you ship:

• Run twenty or more idle trials between captures — the per-trial step only shows across trials.
• Trigger at the production rate, not a bench rate — every rate figure above scales with fps.

When it is not the queue, and what to order next

Flat memory with missing frames means the writer cannot keep pace: the frame-rate gap article shows how to measure acquisition against processing, and the hardware trigger frame rate article explains the ceiling an external trigger imposes. A stream that stops after exactly one frame is trigger configuration, not memory — see the one-frame trigger fix. Frames that arrive slower than expected are best measured directly with the frame rate measurement how-to, and the set-up that makes each trigger window deterministic — trigger source, activation, the free-run default — is covered in the per-camera trigger configuration guide.

The queue discipline also matters more as resolution rises, because every ceiling above scales with frame bytes. For bench work this loop typically runs on a USB3 area scan camera; the LS-CMR-0166 is a 12 MP USB3 model with a 1/1.7" rolling-shutter sensor, 4000 × 3000 resolution and 30 fps.

On a triggered line, where every frame must correspond to exactly one exposure, a global-shutter GigE model is the standard choice: the MV-CU020-90GC is a 2.5 MP color camera with a 1/1.7" sensor, global shutter and a C-mount.

Next step: shortlist the camera in the area scan cameras category or the camera finder, then step through the camera configurator for sensor, interface and mount. If a memory or trigger symptom does not match anything above, send the per-trial log from the first test through the contact page and we will read it with you.