Retrieve multiple images from the camera buffer for later processing

Your camera delivers 50 frames per second and your processing delivers 40; the frames in between die in the transport layer unless you give them somewhere to wait. This guide is for anyone whose grab loop fetches and converts one image at a time. You will size the stream buffer from your worst processing stall, raise the buffer count on the transport layer, split the loop so grabbing never waits on math, and prove with a stall test that no frames were lost. The arithmetic uses a 640 × 512, 16-bit, 50 Hz camera as its worked example — a typical radiometric format in our thermal cameras range — and the same method covers any GigE or USB3 camera, including the area scan cameras range.

Why do frames drop when processing cannot keep up?

Because the transport layer holds only a few frames in flight by default: ten buffers at 50 Hz is 200 ms of cover, so any processing stall longer than that makes the camera reuse a buffer you have not read yet.

The loop in a typical first implementation is serial. The grab call blocks until the next frame arrives, your thread converts it, and only then does it ask for the next one. At 50 Hz each frame must be handed off within 20 ms, so a conversion that runs longer than 20 ms guarantees a growing backlog.

Default pools are small — four to ten buffers is typical — which is 80 to 200 ms of cover at this rate. That absorbs operating-system jitter and nothing else. The cure is a producer-consumer split: the camera keeps delivering into a deep buffer while slower workers drain it at their own pace. The rate side of that split is worked through in acquisition faster than processing: fix the frame rate gap; this guide is about the queue that bridges it.

How deep should the stream buffer be?

Size it from your worst processing stall, not from habit: buffer_frames = frame_rate × stall_seconds × 1.5, so a 50 Hz camera that must survive a 10 s stall needs 750 buffered frames.

Measure the stall rather than guessing it. Time every processing iteration for a full production run and take the longest gap between finished frames — a disk hiccup, a model reload, a dashboard redraw. That worst gap is your stall seconds.

50 frames/s × 10 s × 1.5 = 750 frames. The 1.5 margin absorbs timing jitter around the worst gap; it cannot rescue a permanent rate mismatch, which no depth can — that case has its own section below.

How much RAM does a buffered burst need?

For a 640 × 512, 16-bit camera, one frame is 640 × 512 × 2 B = 655,360 B, so the 750-frame buffer needs 750 × 655,360 = 491,520,000 B ≈ 469 MiB — a 512 MiB allocation covers it.

Worked through step by step:

• Frame size: 640 × 512 px × 2 B/px = 655,360 B, about 0.64 MiB
• Stream rate: 655,360 B × 50 Hz = 32,768,000 B/s ≈ 32.8 MB/s, about 262 Mbit/s — roughly a quarter of a 1 Gbit/s link
• Stall cover: a 10 s stall banks 50 frames/s × 10 s = 500 frames, and 500 × 655,360 B = 312.5 MiB
• Sized pool: 750 frames × 655,360 B ≈ 469 MiB, which fits a 512 MiB pool with headroom for SDK overhead
• A full 512 MiB pool holds 536,870,912 B ÷ 655,360 B ≈ 819 frames, or 16.4 s of stream at 50 Hz

The LS-IRC-0044 delivers this exact stream: 640 × 512 frames at 50 Hz over GigE.

How do you raise the buffer count on the transport layer?

Open your capture tool's stream parameters, switch the buffer-count mode from automatic to manual, and set the count to the 750 frames your stall calculation produced; the same setting block holds the overflow handling mode.

The buffer pool belongs to the transport layer — the GenICam side of the driver — so the setting sits with the other stream parameters such as pixel format and packet size, not among acquisition settings like exposure. Automatic mode sizes a handful of buffers; manual mode lets you enter your computed count, and each buffer costs its full 655,360 B of RAM.

The handling mode next door decides what overflow means: a ring mode overwrites the oldest queued frame, a newest-only mode keeps just the most recent. Newest-only is the setting behind saved recordings that repeat frames — stream buffer handling mode: set newest only on the transport layer walks through it. For later analysis you want the ring.

How do you set up buffered capture, step by step?

Seven steps: measure the stall, compute the depth, set the buffer count, preallocate a bounded queue of that size, grab-copy-release-queue in the acquisition thread, convert in the workers, and verify with a stall test.

1. Time the processing step across a full run; the longest gap between finished frames is your stall seconds.
2. Compute buffer_frames and buffer_bytes from the formula above.
3. Switch the transport layer's buffer-count mode to manual and set the count to buffer_frames.
4. Create the application queue bounded at buffer_frames — in Python, queue.Queue(maxsize=750).
5. Acquisition thread: grab with GetNextImage(), return incomplete frames with an immediate Release(), copy the payload from GetNDArray(), Release(), put the copy on the queue.
6. Worker threads: take a frame, apply the counts-to-temperature conversion, store or publish the result.
7. Run the stall test at the end of this guide and record how many frames the queue absorbed.

import queue

frame_queue = queue.Queue(maxsize=750)   # buffer_frames from the formula
drop_count  = 0                          # frames lost to a full queue
grabbed     = 0                          # frames pulled from the camera

# acquisition thread: grab, copy, release, queue — nothing else
while recording:                         # bool flag, set False to stop
    image = cam.GetNextImage()           # blocks until the next frame
    if image.IsIncomplete():
        image.Release()
        continue
    raw = image.GetNDArray().copy()      # 655,360-byte copy: you own it now
    image.Release()                      # buffer returns to the pool here
    try:
        frame_queue.put_nowait(raw)
        grabbed += 1
    except queue.Full:
        drop_count += 1                  # sustained deficit: see below

# worker thread: convert at consumption time
while not frame_queue.empty():
    counts  = frame_queue.get()          # uint16 counts, 0 to 65535
    celsius = counts * 0.1 - 273.15      # documented 0.1 °C/count scale

The four grab calls keep the get-next-image form used by common thermal SDKs; your SDK's names will be close. For a finite burst — record everything, process after the last frame — the simpler pattern is appending each image object to a list and releasing the whole list when acquisition ends; the bounded queue is for continuous running.

Should you copy the frame before you release it?

Yes, always: the array GetNDArray() hands you is a view over the driver's buffer, so once you call Release() the memory goes back to the pool and the next incoming frame can overwrite it mid-copy.

The discipline is copy, then release, immediately. The copy costs 0.64 MiB and well under a millisecond; losing one frame in a hundred costs a rerun of the whole batch. Converting the counts does allocate a fresh array, but make the explicit copy the habit — it keeps the release point unambiguous.

Saved sequences that mix half-old and half-new content, or repeat frames, are almost always this bug. Torn frames in a camera callback: copy the buffer before you queue it dissects the symptom, and stereo frames saved wrong: copy Mat data from the SDK buffer shows the same trap in handle-based SDKs of other families.

When are user-supplied buffers worth the extra code?

When the count runs into the hundreds or allocation jitter shows up in your timing: SetUserBuffers(...) hands the transport layer a ring of buffers you allocated, so count, size and memory location become your variables.

Allocate once at startup — 750 buffers × 655,360 B is the same 469 MiB as before, paid once instead of by a per-frame allocator — and hand the block to the transport. What you gain:

• No per-frame allocation, so no allocation jitter in the grab loop
• Free-list bookkeeping that tells you exactly how many frames are in flight
• Control over memory location; some transports ask for page-locked memory for user buffers, often for direct-memory-access reasons — the SDK's user-buffer example states the requirement, so follow it

When a dozen default buffers already cover your jitter, skip the extra code — default buffering is simpler and the gain is nil. The allocation discipline either way is in reuse image buffers instead of per-frame allocation.

What if processing never catches up?

No buffer depth fixes a sustained deficit: at 50 fps in and 40 fps out, the 750-frame buffer fills in 75 s and drops start, so you have to cut the input rate or move the work.

The arithmetic: a 10 frames/s deficit drains 750 frames of margin in 750 ÷ 10 = 75 s. After that, every second loses 10 frames forever. Ranked fixes:

• Set the camera's frame rate to the processing rate — 40 Hz here. Input matches output, nothing accumulates; the cost is 20% of frames, accepted by design.
• Shrink the region of interest: 320 × 512 instead of 640 × 512 halves frame bytes to 327,680 B, and the processing load scales down with the pixels.
• Drop to an 8-bit format for presence and pass/fail checks — it halves the stream the same way, but it ends counts-to-temperature conversion, so radiometric work must stay 16-bit. Changing pixel format halves your frame rate: check linked attributes lists the side effects to expect.
• Stream raw counts to disk and process offline: stream camera images to disk at high frame rate is the pattern, and saved 16-bit counts keep the radiometry intact for later conversion.
• Move the processing itself: feeding camera images to external processing tools shows frames routed to a separate processor, which takes the deficit off the acquisition host.

If the deficit is permanent, set the frame rate just above the processing rate and keep the deep buffer for genuine spikes. A buffer is a shock absorber, not a gearbox.

How do you verify the buffer saved every frame?

Pause the worker threads for a known interval — 10 s at 50 Hz should leave roughly 500 frames queued — then resume, drain the queue, and confirm the frame IDs run consecutively with drop_count still at zero.

The test: start acquisition with the queue live, hold the workers for exactly 10 s, resume, and wait for the queue to empty. During the pause, watch the SDK's dropped-frame statistic — if it moves, the buffer count or the queue size is too small for the stall you measured. After the drain, compare frame IDs: most transports stamp every buffer with a frame counter, and consecutive IDs with a step of 1 prove no gaps. Repeated IDs point to duplication — the latest-image-only behaviour dissected in latest image only returns duplicate frames: drain the buffer queue.

Common mistakes, in order of how often they cost a day:

1. Converting inside the grab loop — the original failure; keep the loop grab-copy-release-queue.
2. An unbounded queue with float conversion — RAM grows until the host swaps; bound it and keep counts.
3. Holding the image object past Release() — torn or duplicated content in saved frames.
4. Trusting wall-clock timing instead of frame IDs — the IDs are the evidence that survives.

The same arithmetic applies when you choose the camera: frame rate × frame size gives the link load, and the stall test sizes the RAM. The camera finder filters GigE and USB3 thermal cameras by resolution and interface, and the systems topic page collects the acquisition-side guides around this one. Cameras in the range that carry this pattern:

If the numbers say your link or your RAM cannot carry the stream, send them to the applications team before ordering — the usual fix is a smaller region of interest or a lower frame rate, not a bigger computer.