Low GPU utilization with memory nearly full is the classic starved-trainer signature: the GPU has claimed its memory, but not enough work arrives. Performance modes on the CPU and the card change nothing, because compute was never the limit — the image pipeline in front of the GPU is. This troubleshooting guide gives the diagnosis in the order that pays: which counter to read (the CUDA graph, not the overview), the tests that isolate the data loader, and the fixes — prefetching workers, one-time decode to the model's input size, pinned-memory batches. A worked example with 20,000 images from a 12 MP camera shows the arithmetic end to end, and what the same card does once the loader keeps up.
Read the CUDA counter, not the overview graph
The GPU overview graph most tools open first samples the 3D engine. Training kernels run under the compute engine, which Windows Task Manager shows as the Cuda graph on the GPU page; on the driver side, nvidia-smi reports the same utilization per card. Watch that graph across one full training step and commit to one of three readings:
- CUDA above 90% and epoch time already acceptable. The overview number is cosmetic. Stop; there is nothing to fix.
- CUDA sawtoothing between 20% and 40%. The card alternates between computing and waiting. Starved input; keep reading.
- CUDA flat at zero. The GPU backend is not engaged. Open your vision software's compute-device or runtime setting and select the GPU before touching anything else.
What full memory plus low utilization tells you
Memory and utilization answer different questions. Weights, gradients, optimizer state and cached activations claim their memory in the first steps and hold it; the utilization trace says only whether kernels ran in each sample window. Near-full memory with a low trace therefore means the allocations are fine and the work is not arriving — the opposite of a GPU problem. For how a small network can still reserve a large block of memory, see Why small neural network models use large GPU memory.
The fastest confirmation costs one run: train the same job twice, once with the compute device set to CPU and once set to GPU. When the input side gates both runs, wall-clock times come out nearly identical, because the compute difference hides under the load time. Identical times are the tell; no performance mode, governor or driver setting will move them.
The causes, in order of probability
| Rank | Cause | Signature | Ten-minute test | Fix |
|---|---|---|---|---|
| 1 | Loader too slow: decode, resize, augmentation on the CPU | One CPU core pinned at 100%, CUDA trace sawtoothes per batch | Loop the loader alone, count images per second | 2-4 workers, prefetch, decode to input size |
| 2 | Batches built at full sensor resolution | Batch shape equals sensor pixels; copy engine spikes each step | Print the batch tensor's shape | Resize inside the decoder, never after |
| 3 | Blocking host-to-device copy | Utilization dips while the copy engine works | Copy bytes ÷ bus bandwidth versus step time | Pinned memory, copy issued before the step |
| 4 | Batch size or model too small | Loader rate above demand, CUDA still under ~50% | Feed generated tensors, no disk or decode | Raise the batch; or move to CPU or an inference box |
| 5 | Storage I/O | First epoch far slower than the second | Time epoch 1 against epoch 2 | Local NVMe, cache the decoded set in RAM |
| 6 | GPU backend not selected | CUDA graph flat at zero | Watch the CUDA graph during one step | Set the device or runtime to GPU |
| 7 | A second process on the card | Another process in the card's process list | Open the per-process list in the driver monitor | Move the display or inference job off the card |
Rank 1 covers most vision-training cases, because camera images are large and JPEG decode is pure CPU work. The worked example next puts numbers on exactly that case.
Worked example: 20,000 frames from a 12 MP camera
Take a defect dataset of 20,000 JPEGs from a 12 MP area scan camera. The LS-CMR-0166 is typical of that source: 4000 × 3000 frames over USB3. The detector takes a 640 × 640 input at batch size 32, and one training step costs 0.25 s on the card.
Now the loader side. A mainstream JPEG decoder moves roughly 100 megapixels per second per core, so one 12 MP frame decodes in 12 ÷ 100 ≈ 0.12 s. With four workers the loader delivers 4 ÷ 0.12 ≈ 33 images per second, so a batch of 32 takes 32 ÷ 33 ≈ 0.97 s to assemble against 0.25 s of compute:
Duty is 0.25 ÷ 0.97 ≈ 26%: the card runs one step, then waits three step-times for images. The epoch takes 20,000 ÷ 33 ≈ 600 s, about 10 minutes, and the CUDA trace sawtoothes — the exact complaint. Memory sits full the whole time, because the model, optimizer state and reserved batch buffers never leave.
Fix move one: decode straight to the input size. Most JPEG decoders can decode at 1/2, 1/4 or 1/8 scale directly from the compressed stream. At 1/8 the 4000 × 3000 frame arrives as 500 × 375 in about 0.012 s — roughly a tenth of the full decode, because the inverse transform and color conversion run on 1/64 of the pixels while the stream is still parsed once. Four workers now deliver 4 ÷ 0.012 ≈ 333 images per second, and a batch takes 32 ÷ 333 ≈ 0.10 s, under the 0.25 s step. The loader has stopped setting the pace.
Fix move two: pin the batch memory and overlap the copy. One batch at model size is 32 × 640 × 640 × 3 ≈ 39 MB. Pinned (page-locked) host memory lets the copy run asynchronously at bus speed: 39 MB ÷ 12 GB/s over PCIe 3.0 ×16 is about 3 ms, issued while the previous step computes. The mechanics of keeping tensors on the device are covered in Feeding GPU tensors to a vision model without a CPU round trip.
With both moves in, the epoch is GPU-bound: 625 steps × 0.25 s ≈ 156 s, about 2.6 minutes. Same model, same card, 600 ÷ 156 ≈ 3.8× faster, and the CUDA trace holds above 90%.
Two details keep the example honest. First, 4000 × 3000 is a 4:3 aspect ratio; scaling straight onto a square 640 × 640 stretches circles into ellipses. Letterbox with padding or train a rectangular input — the trade-offs are laid out in Rectangular image input in deep learning training: letterbox or rect.
Second, never assemble batches at sensor resolution: one decoded RGB frame is 4000 × 3000 × 3 = 36 MB, so a batch of 32 is 1.15 GB of host memory that then crosses the bus. Decoded once at 640 × 640, the entire set is 20,000 × 1.2 MB ≈ 25 GB and fits in RAM on a 32 GB host, which makes every epoch after the first disk-free. The size-the-data-first habit pays at capture time too: Full resolution image capture is slow or stalls: size the data first.
The fixes, in the order they pay
- Measure the CUDA graph across one step, as above. Everything downstream depends on that reading.
- Set 2-4 loader workers and a prefetch of 2-4 batches. Starvation is a queueing problem; more workers put decode on parallel cores while the GPU computes.
- Decode once, at the model's input size. Use the decoder's scaled-decode option, or pre-resize the whole dataset offline and train from the small copies.
- Pin batch memory and issue the copy before the step, so it overlaps compute instead of stalling it.
- Cache the decoded set in RAM when it fits, as in the 25 GB example, and point the loader at the cache after the first epoch.
- Re-measure. Loader rate must beat demand with margin: at least 1.5 × batch ÷ step time. In the example, 333 images per second against a demand of 32 ÷ 0.25 = 128.
When the batch size is the real limit
Point the trainer at generated tensors of the same shape — no disk, no decode. If the CUDA trace still sits under about 50%, the step itself is too light for the card. Raise the batch size until roughly 10% of memory stays free; step time should grow near-linearly while steps per second hold. When memory caps the batch below about 8 — common with very large inputs from 16-25 MP cameras — launch and synchronization overhead takes a growing share of each step, so plan on the 60-80% band on the CUDA graph rather than 90%. A second card with data-parallel training restores throughput in that case.
If the model itself is small — a classifier well under ten million parameters — a training card may never fill. Either accept the idle capacity or move that workload to CPU inference and an inference terminal; the decision path is in Object detection without a GPU: CPU inference for inspection and Machine vision deep learning setup: choosing CPU or GPU. And if a larger batch trips an out-of-memory abort, the recovery sequence is in YOLO training segmentation fault: GPU memory and CUDA fixes.
Hardware that does not starve the card
The loader fixes are free, so make them first; hardware pays only after the pipeline outruns the card. Cores: after scaled decode, the example needs 128 images per second × 0.012 s ≈ 1.5 cores of decode — any modern CPU has that. Before the fix, the same demand is 128 × 0.12 ≈ 15 cores, which is why you fix decode before buying processors. RAM should hold the decoded set, 25 GB in the example. The raw JPEGs, at 3-5 MB per frame, are 60-100 GB for 20,000 images and belong on local NVMe, not a network share.
For central training runs, the shop carries a machine built for exactly this stage:
- LAN ports
- 2 LAN ports
- USB 3 ports
- 3-4 USB 3 ports
For a station that trains and inspects in one box, look at vision controllers with expansion slots for a GPU card:
- LAN ports
- 5+ LAN ports
- USB 3 ports
- 3-4 USB 3 ports
- LAN ports
- 3-4 LAN ports
- USB 3 ports
- 2 USB 3 ports
The MV-VC5525-256G70-NN is a VC5000 controller with 32 GB of RAM and PCIe expansion for the card. The MV-VC4712-256G10-2TH pairs an Ultra 7 processor with 16 GB × 2 memory and the same PCIe expansion. Whichever host you pick, driver and VRAM planning for the training environment are covered in GPU deep learning environment for machine vision: drivers and VRAM.
On the capture side, the training set is only as good as the camera that made it: browse area scan cameras or the camera finder when the dataset has to grow.
Vision computers and fanless industrial PCs
190 models · from $1,119.00When low utilization is fine
Two cases need no fix. If the CUDA graph is high and only the overview reads low, nothing is wrong — the 3D engine simply has no work during training. And if retraining runs through the night against a schedule, an idle card between steps costs nothing; utilization is a means, and the number that pays is epoch time against your retraining cadence.
Checklist before you blame the hardware
- [ ] Read the CUDA graph, not the overview — the 3D engine idles during training by design.
- [ ] Time the loader alone — images per second against batch ÷ step time.
- [ ] Set 2-4 workers with prefetch 2-4 — starvation is a queueing problem.
- [ ] Decode once at input size — scaled decode costs a tenth of a full decode.
- [ ] Pin batches and overlap the copy — 3 ms overlapped beats a stall every step.
- [ ] Compare a CPU run against a GPU run — identical times mean the input side gates both.
- [ ] Check the card's process list — a display or inference job shares it silently.
Bring the numbers — dataset size, image resolution, batch, step time — to the contact page and the arithmetic above turns into a host and card recommendation in one pass. To build the capture side instead, start at the system configurator.
Questions
Why does GPU memory stay full after utilization is fixed?
Training frameworks cache freed blocks instead of returning them to the operating system, so reserved memory stays near the ceiling even after your changes make the card busier. Reserved is not in use: the framework's memory report separates the two, and restarting the process releases everything. Treat memory as a capacity budget, not a health signal — the readings to watch after a fix are the CUDA trace and images per second.
How many data-loader workers should I set?
Start at 2-4 with a prefetch of 2-4 batches, then tune by arithmetic: workers divided by seconds per image must beat batch size divided by step time by about 1.5 times. Gains flatten past the physical core count as decode threads contend for memory bandwidth. In Python-based loaders the workers must be processes, not threads, because one interpreter serializes threads — a wrong setting here looks exactly like a slow disk.
Is near-full GPU memory a problem during training?
No. Idle memory buys nothing, and a trainer that has claimed most of the card is sized as it should be. The failure mode is an allocation failure when you raise the batch size or start a second job, and that aborts the run rather than slowing it. Keep about 10% headroom for that moment, and be ready to step the batch down one notch when it happens.
My CPU and GPU training times match — is the card being used?
Usually yes, and it is starving. When the input pipeline gates both runs, compute time hides under load time and the two readings converge. Swap the dataset for synthetic tensors of the batch shape and rerun — that removes disk and decode entirely. A healthy GPU pipeline then finishes steps on convolutional backbones on the order of ten times faster than the host CPU, so if the gap opens up, the card was fine and the loader was the gate.
Will a faster GPU raise utilization?
Usually the opposite. A faster card finishes each step sooner and then waits longer for the next batch, so the utilization trace drops further while the loader is unchanged. Lift the loader above the demand rate first — 1.5 times batch size over step time — and only then does a bigger card convert into shorter epochs. Buying compute before fixing input is the costliest version of this mistake.


