Out of fast frame buffer memory: fit algorithms to embedded RAM

An embedded Python smart camera that raises "Out of fast frame buffer stack memory" is telling you one thing: the frame plus the algorithm's working memory no longer fit the module's fast on-chip RAM. The error hits on the first call, repeats on every run, and has nothing to do with the rest of your script. If you program this class of module for detection or a small quantized network, the sections below give you the byte budget behind the message, the order of fixes that keeps the job on the module, and the point at which the honest answer is a smart camera or a vision controller with gigabytes instead of kilobytes.

What the error means, and what it does not

The message reports an allocation failure inside the fast RAM that serves the frame buffer, the detector's working arrays and the Python heap. On this class of module that RAM is a fixed budget of a few hundred kilobytes: not expandable, not swapped, and shared by everything that runs. A frame at QVGA in RGB565 already claims 153,600 B of it before your first line of detection code executes.

The signature is deterministic. Same framesize, same pixel format, same algorithm: the failure lands on the same line in every run. That separates it from a leak, where free memory decays across iterations and a call that passed at boot starts failing minutes later. If your failure grows over time, you are reading the wrong guide — the per-frame image leak in video loops and inspection-loop memory growth cover that PC-side pattern.

The frame sets the floor: width × height × bytes per pixel

Two pixel formats matter here. RGB565 stores a pixel in 16 bits — 5 red, 6 green, 5 blue — so a pixel costs 2 bytes. Grayscale stores 8 bits, so 1 byte per pixel. The frame's claim on RAM is then plain arithmetic:

Read the table as the non-negotiable floor. At QVGA RGB565 the image alone holds 153,600 B — exactly 150 KiB — and every detector array, model weight and heap object competes with it for what is left.

Why the algorithm adds its own demand on top

Detection routines allocate intermediate arrays that scale with pixel count, which is why a single frame-size step flips a detector between pass and fail. Quantized networks add their file size, and the family's model loader lets you choose where the weights sit: load_to_fb=True places them in the frame buffer region, load_to_fb=False places them in the Python heap. The pattern the firmware family documents compares the file size against free heap minus a 64 KiB reserve:

# gc.mem_free(), uos.stat() and tf.load() are this module family's calls;
# free, size and "model.tflite" are this example's variables (both int)
free = gc.mem_free()                 # bytes reported free in the heap
size = uos.stat("model.tflite")[6]   # file size in bytes
net = tf.load("model.tflite", load_to_fb=size > (free - 64 * 1024))

Worked example with real numbers. gc.mem_free() reports 300,000 B free. The reserve is 64 × 1024 = 65,536 B, so the decision threshold is 300,000 − 65,536 = 234,464 B. A 250,000 B model exceeds it, load_to_fb evaluates True, and the weights go to the frame buffer region while the heap keeps all 300,000 B for working arrays. Had the heap reported 400,000 B, the threshold would be 334,464 B, the model would fit the heap, and load_to_fb would stay False, leaving the frame buffer untouched for images.

The worked budget: why QVGA fails and QQVGA runs

The family's documented behaviour for the rectangle detector: at QVGA RGB565 on the small-memory variant the error fires, and at QQVGA the same call runs. The arithmetic shows why. QVGA is 76,800 pixels and QQVGA is 19,200, so the step down removes 57,600 pixels — 115,200 B in RGB565 (57,600 × 2). That is exactly the working room a proportional detector was missing.

The order of moves matters. Grayscale first: it halves the bytes and keeps the field of view, taking QVGA from 153,600 B to 76,800 B. Frame size second: one step down quarters the pixel count, taking QQVGA grayscale to 19,200 B — one eighth of where the failing script started. Most detectors in this family accept gray, so color is usually the cheapest thing to give up.

sensor.reset()
sensor.set_pixformat(sensor.GRAYSCALE)  # 1 B/px
sensor.set_framesize(sensor.QQVGA)      # 160 x 120 -> 19,200 B frame

When shrinking the frame breaks the inspection

The sampling rule the rest of machine vision works by: the smallest feature you must act on needs at least 3 pixels across to measure it, 2 to detect it, and anything at 1 pixel is noise. Frame-size steps scale each dimension linearly, so pixels-per-feature scale with them: QQVGA halves both width and height against QVGA.

Run the check on your own target. A scratch spanning 12 px at QVGA spans 6 px at QQVGA — still comfortably measurable, so shrink without fear. A 5 px marking falls to 2.5 px: the memory error is gone and so is the defect. Decision rule: if the largest frame the RAM accepts leaves your feature under 3 px, stop shrinking and move the workload up a class.

When the right fix is more RAM, not a smaller frame

If the task needs QVGA-or-better resolution, a quantized model and double buffering at the same time, the module class is out of budget by design; the fast RAM is fixed at manufacture and no storage card changes it. Two catalog paths do this job. Smart cameras run the whole inspection on the device with no PC. Vision controllers give the algorithm a Linux user space with gigabytes of memory, several camera ports and built-in light channels. Bare modules remain fine for learning the API on a desk — small form factor USB camera modules for embedded Linux covers that middle step — but a production cell needs the headroom.

The MV-VC2140-128G40-1T carries 8 GB of memory; the MV-VC5525-256G70-NN carries 32 GB plus PoE camera ports. Set those against the module budget: 8 GB is 8,589,934,592 B, which alone holds about 447,000 QQVGA grayscale frames of 19,200 B each (8,589,934,592 ÷ 19,200 ≈ 447,392). At that scale a model measured in hundreds of kilobytes stops being a budget line at all.

If the cell has no room for a box, an on-device camera keeps the loop in one housing. The MV-SC5060XM-08M-WBN is a 6 MP mono GigE smart camera with an M12 mount; match its shutter class to line speed with the finder's shutter filter, which splits global and rolling models across the range.

The platform choice changes the software question too. AI versus rule-based inspection on smart cameras sets out which approach belongs on-device, and adding AI surface inspection to cameras you already run covers the retrofit path. To browse the hardware, the smart camera finder filters by resolution, shutter and mount.

Diagnose first: a fixed order of checks

1. Time the failure. On the first call it is a budget error; after minutes of running it is a leak, and the leak guides above are the place to start.
2. Print gc.mem_free() on the line before the failing call and write the number down.
3. Compute the frame bytes from the table in the frame-floor section; the difference to the free-heap figure is the room the detector has to live in.
4. Confirm the firmware build matches the board variant. Some module families ship a larger-RAM variant with its own firmware build; a board flashed with the smaller variant's build throws this exact error on algorithms the larger variant fits. Reload the build made for your board, taken from the manufacturer's own downloads, then re-test before changing any code.
5. Switch the pixel format to grayscale and re-run.
6. Drop one frame-size step and re-run.
7. Recheck pixels-per-feature at the frame that now fits; below 3 px, escalate to a smart camera or vision controller.

The decision, case by case

If the table sends you up a class, shortlist on the smart camera and vision controller pages above, then send the camera count, frame sizes and model file sizes with a quote request — expert support comes with every order, and those are exactly the numbers the quote needs. More systems-side fixes, from trigger timing to buffer handling, live on the systems topic page.