Why small neural network models use large GPU memory

You export a nano-scale detection model, check the file, and it is a few megabytes. Then the inference process reserves a gigabyte of GPU memory and stays there. The instinct is to shrink something: half precision, a smaller input size, fewer channels. None of it moves the number much, because the memory was never mostly yours. This guide explains what actually occupies the GPU during inference, gives a rough budget you can apply before you buy hardware, and lists the changes that reduce the footprint and the ones that only feel like they should.

The weights are the smallest line item

A model file stores the arithmetic: layer definitions and trained coefficients. A nano-scale object detector stores a few million parameters. At 4 bytes each in FP32, that is roughly 20 MB; at 2 bytes in FP16, roughly 10 MB. Even a dozen such models together would fit in a fraction of a gigabyte.

What the runtime must hold to execute that arithmetic is a different budget:

• CUDA context. First contact with the GPU creates a context: command queues, page tables, the driver's kernel module images and fixed allocations. On a typical x86 system this alone is on the order of 200-500 MB, and it exists whether your model has one layer or a hundred.
• Framework runtime. The inference library loads its own kernels and metadata onto the device, often another 100-300 MB depending on version and backend.
• Workspace memory. Convolution and matrix-multiply algorithms request scratch space to run fast. Backends grab a generous workspace by default and keep it allocated.
• Activations. The intermediate feature maps your input produces as it flows through the network, plus the input and output tensors themselves.

The first three lines barely depend on your model. That is why the file size and the reserved memory are almost unrelated at the small end: the fixed cost dominates, and your model sits on top of it.

A related symptom is covered in CUDA context invalid after camera close: manage GPU buffers, which looks at the lifetime of that context from the application side rather than the size of it.

Why half precision and smaller inputs barely help

Half precision halves the weights and, if the backend runs the network in FP16, the activations too. On a nano model the weights are ~20 MB, so you save perhaps 10-30 MB of a 1,000 MB footprint. The context, the runtime and the workspace are untouched.

Reducing the input size scales the activations with the image area. Dropping from 640 to 160 pixels on a side divides activation memory by 16. If activations were 80 MB, they become 5 MB: a real saving in relative terms, invisible in the total. This matches what the raw numbers predict, and it is worth verifying yourself before spending an afternoon on it.

A worked example: where 1 GB goes

Take a nano-scale detector at 640 pixels input, FP16 weights, on a desktop-class GPU with a current driver:

Add the columns and you land near the observed gigabyte. Halve the input size and you save a few tens of megabytes. Convert to FP16 and you save a few more. The three fixed rows, around 800 MB, are the baseline every model on that system pays, once per process.

One practical consequence: two small models in two processes each pay their own context. Running both models in one process shares the context and the runtime, which is often the single largest saving available without touching hardware.

How to measure where your memory goes

1. Baseline the process without the GPU. Start the application, run one inference on CPU, and record total process memory. This is your non-GPU floor.
2. Record reserved memory after first inference. Most runtimes expose reserved (not just allocated) device memory; drivers show per-process usage in their own tools. Reserved is the number that matters, because caching allocators hold what they grabbed.
3. Change one variable at a time. Input size first, then precision, then backend. Re-measure after each change with a warm run, because the first inference allocates the workspace.
4. Compare one process versus two. If two models double the footprint, they are paying two contexts. Merging them into one process or one inference session recovers most of it.

The same discipline applies to timing, and CPU time vs wall-clock time in vision pipeline benchmarking covers the measurement traps that mirror these memory ones.

What actually reduces the footprint

Once you know the baseline is fixed overhead, the levers change:

• One process, one context. Consolidate models and cameras into a single inference process instead of one process per camera. This is the cheapest multi-hundred-megabyte saving available. For multi-camera set-ups, the memory behaviour interacts with acquisition design; camera acquisition high CPU usage: replace busy waiting covers the acquisition-side pattern that pairs with it.
• Right-size the workspace. Many backends let you cap or disable algorithm workspace search. Trading a little throughput for a fixed cap is often worth it when several processes share one GPU.
• Free between inferences. Some frameworks keep intermediate tensors alive unless you explicitly release the session or call the runtime's empty-cache call. If memory grows frame over frame, you have a leak, not a baseline; the buffer-ownership patterns in the CUDA context article above apply directly.
• Buy headroom, not heroics. If the baseline is 800 MB and your budget is one model per camera, hardware with enough memory removes the whole class of problem. Vision controllers exist for exactly this: industrial machines with the LAN ports, trigger I/O and light-controller channels a vision cell needs, sized so a gigabyte baseline is not a crisis.
• Run on CPU where the frame rate allows. A nano model at modest input sizes runs at tens of milliseconds per frame on a current multi-core CPU. If your line allows it, CPU inference has zero GPU footprint and one less driver in the failure chain.

Both controllers above carry 8 GB of system memory with four built-in light-controller channels, which is comfortable for one or two small models plus camera acquisition. For cells that run several models, several GigE cameras through PoE, or larger networks, more memory is the honest answer.

When the usual answer is wrong

• "Use a smaller model." The file size is not the problem; the context is. Moving from nano to small changes activations slightly and the baseline not at all.
• "Reduce the input resolution." Do it for latency and bandwidth reasons if you like, but do not expect the memory number to follow.
• "The GPU is leaking." A caching allocator holding reserved memory is normal. A footprint that grows every frame is a leak. Distinguish the two before you file anything.
• "Add a GPU with more memory." It works, and sometimes it is right, but check the one-process-versus-two comparison first: you may be paying two contexts for one workload.

Before you order hardware

• Sum your models, not your files: baseline context and runtime once per process, plus weights and activations per model.
• Decide how many processes must run in parallel, and try to make it one.
• Leave headroom for the workspace allocator; plan roughly 1 GB fixed cost per GPU process plus your models, then double it as a margin.
• Check the I/O you need in the same breath: camera ports, trigger lines and light channels decide the controller as much as memory does. Browse the vision controller category for available controller specifications, and use the system configurator to plan the rest of the vision system.
• If a smart camera can run the inspection on the device, the GPU question disappears entirely; see the smart cameras range for inspection without a PC.

A short checklist

• Measure reserved GPU memory after a warm inference, before changing anything.
• Test input size and precision once each; expect small movements, and stop there.
• Consolidate models into one process.
• Cap or empty the allocator between batches if memory creeps.
• Compare CPU inference timing against your cycle budget.
• Size the controller for baseline plus margin, not for the model file.

If you are sizing a cell and want the memory, camera and light decisions checked together, contact us with your frame rates and model list, and we will walk the numbers with you.