Reuse image buffers instead of per-frame allocation

Vision pipelines are full of temporary images: a smoothed copy, a gradient map, a binary mask, an overlay canvas. The easy habit is to create each one inside the function that needs it and destroy it on return. For large frames that habit costs real time: the allocator must find, fault in, and zero megabytes per buffer, every cycle, and a long production run pays again in heap fragmentation. What follows allocates every temporary image once at startup, reuses the same memory every frame, and keeps the code readable by passing a small pool struct instead of a wall of globals. The examples are plain C; the pattern maps onto C#, Python, and any library that hands you writable buffers.

The decision, stated plainly

Allocate temporary image buffers once, at startup, and reuse them every frame. Keep the exception small rather than the rule. Buffers under about 100 KB cost almost nothing to create per frame, because a modern allocator serves them from per-thread caches in well under a microsecond. Anything above about 1 MB — roughly 250 pages of 4 KB that the kernel may have to hand over, zero, and fault in — belongs in a pool. Full camera frames always qualify.

Why per-frame allocation costs you

A create-image call for a multi-megabyte buffer does not come out of the allocator's small-block cache. The allocator asks the operating system for pages, and the kernel zeroes each page before your process may read it. First touch of 12 MB at an effective 6 GB/s costs on the order of 2 ms — per buffer, on every frame where the allocator hands back fresh pages.

Freed blocks are often recycled, so the average frame gets cheaper after warm-up. What stays is jitter — the occasional frame that pays the full first-touch price — and fragmentation over hours: blocks come back scattered, the working set grows, and cache locality decays. Some vision libraries offer a create variant that reserves a maximum-size backing store, able to grow but never shrink. An image that once reached a given size then always fits the same space, whatever the heap does around it; use that variant for buffers whose size can change.

The pool, without global variables

The usual objection to startup allocation is a wall of global variables. The standard fix is one small struct per pipeline, passed by pointer like any other state. Nothing is global, functions stay testable because a test builds its own pool, and adding a temporary image means widening one array.

/* Plain C, our own types. Swap malloc for your library's buffer
   creation call; keep the shape. POOL_N = overlapping stages + 1. */

#define POOL_N 4

typedef struct {
    unsigned char *slot[POOL_N];   /* buffers, NULL until pool_init runs */
    int next;                      /* next slot to hand out, starts at 0 */
} image_pool_t;

int pool_init(image_pool_t *p, long w, long h, long bpp)
{
    p->next = 0;
    for (int i = 0; i < POOL_N; i++) {
        p->slot[i] = (unsigned char *)malloc((size_t)w * (size_t)h * (size_t)bpp);
        if (p->slot[i] == NULL) return -1;   /* fail fast at startup */
    }
    return 0;
}

unsigned char *pool_get(image_pool_t *p)
{
    unsigned char *b = p->slot[p->next];
    p->next = (p->next + 1) % POOL_N;        /* round-robin over the ring */
    return b;
}

void pool_destroy(image_pool_t *p)
{
    for (int i = 0; i < POOL_N; i++) { free(p->slot[i]); p->slot[i] = NULL; }
    p->next = 0;
}

Own it the way you own any other state: a member of your pipeline context, a local in main, or per-thread storage. Declare it with image_pool_t tmp = { { NULL }, 0 }; so the initial state is explicit. Round-robin handout is safe for one thread; read the threading notes below before you share it.

Worked example: four 12 MP temporaries at 30 fps

The inputs, with the source of each. The frame source first.

The frame source is the LS-CMR-0166: a 12 MP USB3 area scan camera with a 4000 × 3000 resolution running at 30 fps.

The pipeline works on the mono plane, so bytes per pixel is 1. Four temporaries overlap in one frame — smoothed copy, gradient, mask, overlay — a count you take by walking your own functions, exactly as in step 1 below.

• One buffer: 4000 × 3000 × 1 B = 12,000,000 B ≈ 12 MB.
• Pool: 4 × 12 MB = 48 MB, allocated once at startup.
• Churn avoided: 4 × 12 MB × 30 fps = 1,440 MB/s ≈ 1.4 GB/s of allocator traffic.
• Frame budget: 1 ÷ 30 fps = 33 ms per frame.
• Worst-case first touch: 12 MB ÷ 6 GB/s ≈ 2 ms per buffer, so 8 ms for four — a quarter of the 33 ms budget on any frame that pays it.

Decision: pool these buffers. The pool lives on the host that runs your per-frame code; a vision controller such as the MV-VC3301H-128G66 carries 8 GB of memory.

Your 48 MB pool is 0.6% of that. The same arithmetic transfers to any area scan camera: a wider sensor scales the buffer, never the pattern. Stations built on smart cameras run inspection on the device, which manages its own buffers; the pattern still applies to the host-side code you write around them, and to the vision controllers that host it.

Step by step: move the allocation out of the frame loop

1. Count your temporaries. Walk every processing function and list each image it creates per frame, with width, height, and pixel depth. Mark which lifetimes overlap.
2. Fix slot sizes. Size each pool slot for the largest resolution and pixel depth any recipe uses on that station.
3. Define the pool. One image_pool_t per pipeline — or per thread — with POOL_N set to overlapping stages plus one spare.
4. Allocate after the camera opens and before the first trigger. Treat a NULL return as a startup failure: log it and refuse to run. A failure to allocate camera-side memory is a different fault; frame grabber buffer allocation errors covers that diagnosis.
5. Replace the create calls with pool_get. Delete the per-frame destroy calls; the pool owns the memory until shutdown.
6. Rebuild only at change boundaries. A recipe or ROI change stops acquisition, destroys, re-initialises at the new size, and restarts — never inside the frame loop.
7. Free at shutdown with pool_destroy. Explicit freeing keeps leak detectors quiet, which matters when you audit the wrappers later.

How to know it worked

The ring behaviour is testable in a few lines: the fifth pool_get must return the first buffer again, and no two of the first four may match.

int test_pool_wraps_after_four_gets(void)
{
    image_pool_t p = { { NULL }, 0 };
    if (pool_init(&p, 4000, 3000, 1) != 0) return 1;   /* 4 × 12 MB, once */

    unsigned char *a = pool_get(&p);   /* slot 0 */
    unsigned char *b = pool_get(&p);   /* slot 1 */
    unsigned char *c = pool_get(&p);   /* slot 2 */
    unsigned char *d = pool_get(&p);   /* slot 3 */
    unsigned char *e = pool_get(&p);   /* slot 0 again */

    int ok = (a != b) && (a != c) && (a != d) && (e == a);
    pool_destroy(&p);
    return ok ? 0 : 2;                 /* 0 = pass */
}

Then measure. Run 10,000 frames both ways and record minimum, average, and worst frame time: the pooled build's worst case should sit near its average, while the per-frame build shows occasional spikes toward the first-touch cost. Watch resident memory as well — flat after warm-up means the pool is doing its job. If it grows instead, the leak is elsewhere: ownership bugs in wrappers are covered in managing image buffer ownership in C#, and unbounded queues in bounding the frame queue in Python.

When pooling is the wrong default

Threads. pool_get mutates a shared index, so two threads on one pool can receive the same slot and corrupt each other's frames. Give each thread its own pool, guard the index with a mutex, or use a ring with an in-use flag per slot that hands out only free buffers.

Camera callbacks. Never call pool_get from an acquisition callback. Queue the frame and process it on your own thread; multi-camera acquisition freezes shows what happens to callbacks that do too much work.

Stages that truly overlap. When one stage reads a buffer that a later stage still writes, round-robin hands out the wrong slot. Two buffers per stage is the minimum; frame differencing with two image buffers is the smallest working example of that discipline.

Library-managed pooling. Some libraries already pool buffers behind reference-counted image handles, so your create call is a lightweight handle grab rather than a memory allocation. The 10,000-frame timing run tells you in minutes; spend your effort where the measurement points.

A short checklist before you commit

• Every temporary over 1 MB listed, with size and overlapping lifetime.
• Slots sized for the largest recipe; POOL_N = overlapping stages + 1.
• Pool allocated after the camera opens; a NULL return stops the station.
• Wrap test passes; no create or destroy calls left inside the frame loop.
• The 10,000-frame run shows flat worst-case time and flat resident memory.
• No allocation or pool access from acquisition callbacks.
• Recipe changes rebuild the pool at a stopped-acquisition boundary.

Once the frame loop is clean, size the rest of the station: start from the camera finder or the system configurator, and read wider integration notes on the systems topic page. Questions about a specific frame budget belong on our contact page.